Skip to content

Recommended settings for Request Timeout and Maximum Website Download?

710fla710fla ★ #1 GSA SER VERIFIED LIST
edited August 2016 in GSA Platform Identifier
I'm running my raw Scrapebox scrapes through Platform Identifier at 5000 threads and want to make sure I'm getting the best settings.

What do you recommend for "Request Timeout (seconds)" setting and the "Maximum size of a website to download (MB)" option?

Right now I have the timeout at 120 seconds and the max download size at 25MB

Also my download speed rarely goes over 1MB, is that normal? Even at 5000 threads?


  • 1linklist1linklist FREE TRIAL Linklists - VPM of 150+ -
    25MB is insane. All thats going to do is waste bandwidth and memory.

    What page is over 25mb? Certainly not one google is ever going to crawl.

    I set mine at 180 Timeout, 4MB page-size when I'm running contextuals, and 2mb when I'm running comments, etc.
  • s4nt0ss4nt0s Houston, Texas
    @710fla - What 1linklist said :)

    Also, you can adjust the bandwidth limit on the main screen right above the pie graphs on the right side. Increasing that should improve speed a bit but CPU/mem resources will also increase too.
  • 710fla710fla ★ #1 GSA SER VERIFIED LIST
    @1linklist @s4nt0s thanks guys I adjusted my page-size to 5MB. I'm only getting 126 URL/min despite running 5000 threads and leaving the "Limit bandwidth" unchecked.

    Is this normal? What are you guys getting as your URL/min if you don't mind me asking
  • s4nt0ss4nt0s Houston, Texas
    edited August 2016
    Hmm, 5,000 threads is quite a bit and you most likely won't see much benefit running it that high.

    Even at 100 threads I'm averaging at least 1,300+ URL/min and that's on a slower VPS.


    Do you have any specific settings you checked in the project when setting it up? Are you running it on a VPS or home machine?
  • 710fla710fla ★ #1 GSA SER VERIFIED LIST
    edited August 2016
    I'm running it on my laptop with 8gb of ram and i5 processor. I lowered the threads down to 100 and it jumped up to 700 URL/min

    I have retry count set to 1, should I switch it to 0?  And I'm not using any proxies. My home internet connection runs at 14 Mbps so I'm thinking of switching to a VPS. 

    Any suggestions for a good company? How much RAM / cores should I aim for if I want to run GSA Search Engine Ranker and Platform Identifier side by side?

    Thanks a lot by the way I've been trying to set this up for days now.
  • s4nt0ss4nt0s Houston, Texas
    I personally use Solidseovps for my servers. Solidseovps and poweruphosting are two very popular VPS providers.

    For ram, I would try to go with at least 4 gigs to be on the safe side. Pi can use a lot of CPU/ram depending on how hard you want to run it. So maybe 4 cores/4 gigs or something along those lines if you can. Anything better then that is a plus :)
  • 710fla710fla ★ #1 GSA SER VERIFIED LIST
    Thanks a lot! I'm gonna go ahead and try out the $20 Windows VPS to start with.

    Do you always limit your bandwidth when running it in your VPS?

    Do you also set a bandwidth limit in GSA Search Engine Ranker?
  • 710fla710fla ★ #1 GSA SER VERIFIED LIST
    @s4nt0s one more question. Almost got it running smoothly

    If I'm running platform identifier on my home 15 Mbps connection would it be a good idea to set the bandwidth limit to 512kbps?

    Or would that cause problems
  • s4nt0ss4nt0s Houston, Texas
    For me, I always leave the bandwidth limit on just to keep resources down when running it with my other tools.

    I don't set a bandwidth limit in SER, its mainly used in Pi to help lower CPU. 

    You can set the bandwidth limit to whatever you want. If its running slow, increase, if its using up too many resources (CPU) then lower it a bit. It's really just a matter of tweaking it until you find the sweet spot for the computer you're running it on.
  • vkp1988vkp1988
    Go for a VPS an you will immediately see the difference.

    I highly recommended TBNrocks vpss as well (greenclouds). They are in here too. you will get instant support as well. (no waiting times at all :) )

    I have been using their dedis almost 4years (3 1/2) with zero issues. 

  • 710fla710fla ★ #1 GSA SER VERIFIED LIST
    @s4nt0s thanks so much man I finally got it running smoothly on my home computer! I owe you one you saved me a lot of man hours
  • s4nt0ss4nt0s Houston, Texas
    @710fla - No problem. Good to hear you got it working for you now. :)
  • s4nt0s I would suggest you to not suggest/mention poweruphosting to anyone. They recently wiped all my data and said they cant recover it,
  • s4nt0ss4nt0s Houston, Texas
    @SagarTolani - I've been using Solidseovps for the past few years, but before that I used poweruphosting and never had problems.

    It sucks to hear that happened to you. 
  • 1linklist1linklist FREE TRIAL Linklists - VPM of 150+ -

    I second what s4nt0s said; I've never had any issue with powerup. I'm definitely sorry to hear about your trouble with them - I've only ever had the opposite experience, which is why they stand out in my mind.

    I've had problems with just about every OTHER host. For me, it comes down to them, and Soldseovps.
Sign In or Register to comment.