Proxies Vs amount of Threads

avomoon

Hello.

If i have 10 proxies (private/paid ones), is it enough to use with 100 threads? 500 threads? etc

 

Thanks

mmtj

Using your own list = you can go up as high as 15-20 connection per proxy (depending on your proxy provider and if they cap the bandwith) - split test to find the optimum.

Scraping = Keep it at 2-4 connections per proxy. 

sootedninjas

"Scraping = Keep it at 2-4 connections per proxy."

I used 250% of my proxy.

But I did not think about that when you have your own list. to bump it up way high since it's just going to be used to post. which makes sense.

mmtj

You gotta test and fine tune accordingly, maybe the high thread count will kill your proxies, maybe not. You gotta find a sweetspot for your proxies.

We use at least 200-300 private proxies per server and we run those with 300-400 threads no problem (they run best on our dedi servers and on our virtual private servers with 1GB Uplink) - and that's keeping it low.

With 10 proxies you should be fine with 100-150 threads using your own lists and about 40-50 threads for scraping.

sootedninjas

"With 10 proxies you should be fine with 100-150 threads using your own lists and about 40-50 threads for scraping."

i'll take that into consideration. I'll def try it and see if there are any performance degradation in the system as a whole.

cmo140

is there away to just set a certain number of threads for just scraping and a certain number for posting and ect?

kilauea999

I can't seem to find a setting in GSA for separate thread counts, but anyway I think he meant when you're scraping outside of GSA.  I.e. with SB or Hrefer.