Recommended settings for Request Timeout and Maximum Website Download?

710fla

I'm running my raw Scrapebox scrapes through Platform Identifier at 5000 threads and want to make sure I'm getting the best settings.

What do you recommend for "Request Timeout (seconds)" setting and the "Maximum size of a website to download (MB)" option?

Right now I have the timeout at 120 seconds and the max download size at 25MB

Also my download speed rarely goes over 1MB, is that normal? Even at 5000 threads?

1linklist

25MB is insane. All thats going to do is waste bandwidth and memory.

What page is over 25mb? Certainly not one google is ever going to crawl.

I set mine at 180 Timeout, 4MB page-size when I'm running contextuals, and 2mb when I'm running comments, etc.

s4nt0s

@710fla - What 1linklist said

Also, you can adjust the bandwidth limit on the main screen right above the pie graphs on the right side. Increasing that should improve speed a bit but CPU/mem resources will also increase too.

710fla

@1linklist @s4nt0s thanks guys I adjusted my page-size to 5MB. I'm only getting 126 URL/min despite running 5000 threads and leaving the "Limit bandwidth" unchecked.

Is this normal? What are you guys getting as your URL/min if you don't mind me asking

s4nt0s

Hmm, 5,000 threads is quite a bit and you most likely won't see much benefit running it that high.

Even at 100 threads I'm averaging at least 1,300+ URL/min and that's on a slower VPS.

Do you have any specific settings you checked in the project when setting it up? Are you running it on a VPS or home machine?

710fla

I'm running it on my laptop with 8gb of ram and i5 processor. I lowered the threads down to 100 and it jumped up to 700 URL/min

I have retry count set to 1, should I switch it to 0?  And I'm not using any proxies. My home internet connection runs at 14 Mbps so I'm thinking of switching to a VPS. 

Any suggestions for a good company? How much RAM / cores should I aim for if I want to run GSA Search Engine Ranker and Platform Identifier side by side?

Thanks a lot by the way I've been trying to set this up for days now.

s4nt0s

I personally use Solidseovps for my servers. Solidseovps and poweruphosting are two very popular VPS providers.

For ram, I would try to go with at least 4 gigs to be on the safe side. Pi can use a lot of CPU/ram depending on how hard you want to run it. So maybe 4 cores/4 gigs or something along those lines if you can. Anything better then that is a plus

710fla

Thanks a lot! I'm gonna go ahead and try out the $20 Windows VPS to start with.

Do you always limit your bandwidth when running it in your VPS?

Do you also set a bandwidth limit in GSA Search Engine Ranker?

710fla

@s4nt0s one more question. Almost got it running smoothly

If I'm running platform identifier on my home 15 Mbps connection would it be a good idea to set the bandwidth limit to 512kbps?

Or would that cause problems

s4nt0s

For me, I always leave the bandwidth limit on just to keep resources down when running it with my other tools.

I don't set a bandwidth limit in SER, its mainly used in Pi to help lower CPU. 

You can set the bandwidth limit to whatever you want. If its running slow, increase, if its using up too many resources (CPU) then lower it a bit. It's really just a matter of tweaking it until you find the sweet spot for the computer you're running it on.

vkp1988

Go for a VPS an you will immediately see the difference.

I highly recommended TBNrocks vpss as well (greenclouds). They are in here too. you will get instant support as well. (no waiting times at all )


I have been using their dedis almost 4years (3 1/2) with zero issues. 

710fla

@s4nt0s thanks so much man I finally got it running smoothly on my home computer! I owe you one you saved me a lot of man hours

s4nt0s

@710fla - No problem. Good to hear you got it working for you now.

SagarTolani

s4nt0s I would suggest you to not suggest/mention poweruphosting to anyone. They recently wiped all my data and said they cant recover it,

s4nt0s

@SagarTolani - I've been using Solidseovps for the past few years, but before that I used poweruphosting and never had problems.

It sucks to hear that happened to you. 

1linklist

@SagarTolani

I second what s4nt0s said; I've never had any issue with powerup. I'm definitely sorry to hear about your trouble with them - I've only ever had the opposite experience, which is why they stand out in my mind.

I've had problems with just about every OTHER host. For me, it comes down to them, and Soldseovps.