GSA Proxy Scraper - Optimal Settings
I am using GSA PS to bulk check TF/CF using the URL metrics scanner.
Not to feed into GSA automatically, I feed in my URLs into GSA manually - so this is for my own list creation and filtering.
Just wondering if people could chime in on the following settings:
GSA PS SETTINGS:
- Testing against Majestic - I guess this makes the most sense given the single purpose I am using this tool? This reduces the number of working proxies considerably - to under 200 right off the bat, for example.
- Automatically search every 5 min or when less than 200 proxies are available - sound reasonable?
- Automatically remove proxies when down for 3 minutes - too low?
- Using 100 threads maximum. I heard GSA PS doesn't use much system resources. But I also heard you have to be careful and not get your own IP banned - is this reasonable?
- Otherwise everything the same.
URL METRIC SCANNER SETTINGS:
- Max threads 5. I guess this should be lowish to not burn out the proxies?
- Max retries 10. I figured that you can always export the ones that didn't work and retest them later...
- Otherwise everything the same.
Anyone else using it for these purposes? Let me know your thoughts!