Skip to content

Proxies Vs amount of Threads

edited November 2012 in Need Help

Hello.

If i have 10 proxies (private/paid ones), is it enough to use with 100 threads? 500 threads? etc

 

Thanks

Tagged:

Comments

  • Using your own list = you can go up as high as 15-20 connection per proxy (depending on your proxy provider and if they cap the bandwith) - split test to find the optimum.
    Scraping = Keep it at 2-4 connections per proxy. 
  • "Scraping = Keep it at 2-4 connections per proxy."

    I used 250% of my proxy.

    But I did not think about that when you have your own list. to bump it up way high since it's just going to be used to post. which makes sense.
  • edited November 2012
    You gotta test and fine tune accordingly, maybe the high thread count will kill your proxies, maybe not. You gotta find a sweetspot for your proxies.

    We use at least 200-300 private proxies per server and we run those with 300-400 threads no problem (they run best on our dedi servers and on our virtual private servers with 1GB Uplink) - and that's keeping it low.

    With 10 proxies you should be fine with 100-150 threads using your own lists and about 40-50 threads for scraping.


  • "With 10 proxies you should be fine with 100-150 threads using your own lists and about 40-50 threads for scraping."

    i'll take that into consideration. I'll def try it and see if there are any performance degradation in the system as a whole.
  • is there away to just set a certain number of threads for just scraping and a certain number for posting and ect?
  • I can't seem to find a setting in GSA for separate thread counts, but anyway I think he meant when you're scraping outside of GSA.  I.e. with SB or Hrefer.
Sign In or Register to comment.