Proxy v Threads balance

bnaimy

Hi,

This is my server spec

xeon hex processor 2.7 ghz
32gig ddr3 ram
200MBIT dedicated bandwith just for me

I also have 50 high quality dedicated proxy, that can run 100 threads each

There are two settings "Threads to use" and HTML timeout, from what i figure you would need to increase
HTML timeout if you increase "Threads to use"

With my specs, how many threads should i be running to get the maximum from server+private proxy's ???
Is 200 "Threads to use" and HTML timeout of 120 seconds ok ?

Would love to hear what you veterans are using for your settings, i have also
checked the online tutorial vidoes, but just want to make sure i am squezzing the server for
all it got. Without proxy's getting blocked

Thanks for taking the time to read this

bnaimy

I have checked around on various forums and some are using 200 and some are using 400-1000 threads, so its a huge difference between what people say. So hard to gauge anything from just listening to random people on the internet. They could be doing the wrong way for all i know.

Waiting with excitement for response

strovolo

you have to play with your settings and figure out the best combination.

apart from that i will advice you to max out your HTML timeout and disable the log if not required (after completing the fine tuning)

....start with 100 threads per proxy, that will be 5000 simultaneous website interactions (50 proxies)... monitor and fine tune 

bnaimy

Hi Strovolo,

Thanks for the advice, was not aware i could use that many.
I will try it out and monitor and adjust as i move along

Thanks for the advice

Cheers

bnaimy

Ill start with 1000 threads and slowly move it up

davbel

Sorry, but @strovolo's advice about 100 threads per proxy is slightly innacurate.  You should be aiming to run no more than 10 threads per proxy so I'd aim for about 400-500

Your real limit is dependent on how many projects you plan on running.  SER is a 32bit app and can only address 2gb of memory, so the more projects and threads you run, the quicker SER will run out of memory, which is where the scheduler comes in.

To give you an idea of what to expect, I'm running 29 projects concurrently at 150 threads (used to do 200, but proxies got burnt too quickly) and I'm averaging about 1.5gb of memory.

If I'm running anymore than this, then I will start using the scheduler to spread the load

bnaimy

Thanks davbel for sounds suggestions, 10 pr proxy does sound allot more natural
I was also slightly surprised when i heard i could do 1000+ threads

Its shame SER is only 32 bit, was not aware of this, shame to waste all that memorie

Thanks Davbel

AlexR

@davbel - have you found that with/without memory issues using the scheduler has improved your LPM? I've seen a few reports of this and wondered if you've found the same. (I think Ron uses it but not to solve memory issues but to increase LPM) I've been using it not for memory problems but increase LPM and it's increased it, but I've done other changes as well so can't say for certain!

strovolo

davbel  is right i don't know what i was thinking while typing that 100 it was suppose to be 10

Next time i have to make sure  having my coffee first

davbel

@AlexR I've had the same suspicion too, but I haven't been able to reliably prove it one way or another.

ron

When I had less projects (like 30-35) my world was fine without scheduler. But as soon as I added more projects, I tried running everything at once and ran into freezes and other obvious memory issues. I even went into dividing my most important projects first, and less important for the final 12 hours. Too much babysitting.

I eventually just started to use scheduler out of practicality. But do keep in mind that it will greatly reduce memory demands. I have it at 10 projects every 20 minutes, and average 400MB memory. When I was not using scheduler, my total memory (for just SER) was approaching 2GB which is pretty fatal.

Even after 1 year of using SER, I can't say with 100% certainty that scheduler helps or hurts LPM. I have had it both ways, so for me it's a tossup.

AlexR

@Metster - "CPU usage monitor in GSA is inaccurate" - I'm no specialist but I recall Sven saying that it can only measure the CPU % of the module that the VPS allocated to SER, while task manager checks across all modules. So it might be at 99% but on task manager at 65%. It sounded like there was no solution around this constraint but to use task manager. 

How do you optimise your internet connection on a VPS???

LeeG

I use task manager to set up the amount of threads

Altering the threads used to be an instant reaction, until someone asked for that to be changed and only take effect after you saved the settings

You need to check the cpu usage after every update and then try and keep it so the cpu peaks at 100% from time to time

Recent update I did, I went from running 210 to 240 threads, which gave a LpM change of about 20

I do an update, then drop to 190 threads, and increase by ten threads at a time, watch and wait until I get an optimal running without running at max cpu all the time

A method I have used for over a year to get optimal performance from ser

 

eliquid

I know with scrapebox back in the day the best ratio was for 1 thread every 4 proxies since SB had a bad way of handling the threads/proxy use. I have learned that I can do 4 threads for every 1 proxy pretty reliable with GSA SER.

When I try to go more then 4 ( say 10 ) I run into issues with AskMeBot and other captcha services. I am using 50 proxies.

bnaimy

I am using 10 thread pr proxy, and its going like smooth as butter

I am getting about 100 verified backlinks pr hour, and its mostly PR 2 and above.
I dont backlink to sites were its below PR2

Keep in mind i am using private proxyies

Running 300 threads for GSA SER and 100 threads with GSA indexer, that also uses private proxies

AlexR

@Metster - thanks!