Increase proxy thread loading speed (proxy test)
edited January 2021 in GSA Proxy Scraper
I find myself in a case where I would like to test millions of proxies (test only the anon, no other additional test), the file is 25mb.
Having a powerfull fiber optic (full 1gbs) I would like to use around 700 threads but I find myself facing a problem.
Threads aren't starting fast enough to reach the number of concurrent tests my connection can handle (I could go higher without a problem).
The time that test threads increase over the time that previous tests complete does not allow me to exceed about 250 concurrent connections.
I think the charging thread is not fast enough or should be multiplied to compensate for this. For example 2 or 4 loading threads by sharing the sum of the proxies to be tested.
What do you think ? is this feasible?
I imagine many of us would find ourselves in this kind of situation (only the scratch free test) and the performance would surely be greatly accelerated.
Sorry for my extreme optimization fanaticism. I see a probable gain of x2 x3 minimum.
EDIT: It would be possible to automate the number of load threads directly by the number of actual threads through the number of threads desired. If the actual thread count is less than 10, an additional load thread is added.
This implies at the very beginning of the process not to divide (and therefore allocate) the total sum of proxy by the number of load threads, but rather a fixed amount of proxy. For example 500 proxy per loading thread.
Sorry for time i no longer receive notification, no approximatively 250 threads (700 in setting).
This is why I was talking about multiplying the loading queues if you want to leave it as it is (control).
Just out of curiosity and to think in the right direction, I imagine you increased the speed of the control but by how much? x2 x3 ...?
The more I test the proxy simultaneously, the more difficult it is for PS to start threads. Does that have anything to do with the GUI?
There is a bottleneck :