SET-UP GSA search engines with 300 threads and GSA indexer with 200 threads

richardbesson

HI,

I have a quad core CPU with 8 threads (i7 2700k), i did setup GSA search engines with 300 threads and GSA indexer with 200 threads and it's barely scratch the CPU (lower than 10%, on average 7%) it seems that everything is running perfect with a high LPM.

Is that a good idea to stay with this crazy number of threads, to try to set-up a more higher value of threads (400 or 500)?

I don't know if i can go and push GSA more than that, i need advice on this.

seowizzard

Number of threads depends more on your internet line, router and network adapter, you can have even a 16 core CPU, if your router and network adapters are crap, you will see a lot of fails.

I recently got a Asus-N66U router, and swapped to a PCI express network adapter.. I have also a i7 CPU (I rarely see it go above 20% when doing 200 threads) and everything runs smoothly.

I've done a test a few nights ago, I created 20 campaigns poitning to some of the tier 1 links. I created almost 60k verifieds overnight.

But yea, if you see a lot of download failed, it probably means that the request from the thread didin't get through and/or it timed out (this is related to your internet line/router/network adapter spec). If you only see ocassionaly the downlaod failed, it's ok ( some sites might be off or are loading very very slow).

richardbesson

Thanks seowizzard make sense

GSAguy123

Download failed could have more to do with html timeout being set too low?

richardbesson

well, i guess one go with the other, if you have a slow connection that can't handle a high number of threads it will fail to download, setting a too high HTML timeout is not something that i want to do as it will significantly slow GSA, the thing is to find the correct GSA config to my internet connection capability.

If the CPU is not an issue here, i would want to have the maximum of threads open but still have GSA work without errors and as fast as i can.