Search Engine Scraping Performance
I'm confused with low performance while scraping the search engines.
GSA SER is provided with 1000 proxies. They are all mine and have the following format: same_ip:different_port.
The same IP is because my proxy server rotates a pool of IPs on its own, and I'm actually testing it while playing with GSA SER.
I set "Custom time to wait between search queries" = 1 second! And expect 1 query through each proxy every few seconds. But I have T:0..1 instead.
I've attached the log file. It looks like GSA considers one IP = one proxy, ignoring the different port number, therefore it waits 1 second.
UPDATE: GSA SER uses target IP rather than domain while making CONNECT request to the proxy Is there any way to disable such behavior? I need domains in CONNECT requests.