[FEATURE REQUEST] Proxy Improvement

AlexR

FEATURE 1: I'd like to get the use of the private ones prioritised. 

Sometimes your private proxies go dead, and in this instance it would be neat to have the program check for public proxies. BUT if you use 20 private proxies, and enable it to search for proxies when number falls below xx, you will then get a situation where you will have 3000 public and 15 private. Chances of private being used in this ratio are small!

FEATURE 2:prioritise the use of the fastest ones automatically (not manually). 

If you have a list of 3000 public proxies, would be nice for it to prioritise the use of the fastest ones automatically (not manually). 

FEATURE 3: Would be nice to be able to have it automatically retest them periodically. AND then disable after xxx retests and still dead. 

If I am travelling and I leave my VPS's running, I'd like it to autotest my private proxies from time to time and disable them if they are blocked, but then RETEST them later to see if they have been unblocked. Currently we manually have to test proxies regularly. So I have 20 private working, and I run a campaign, but during the night 10 get blocked, well, it should retest the dead ones every hour, those that fail for 5 retests in a row should be removed. 

Currently, it just keeps using the blocked ones (since I don't use the auto deselect blocked proxies)...so I am missing out on some many URL's. This means I need to clear caches and rescrape everything to get those missed sites. Could just be avoided if the proxies were working. 

FEATURE 4: Test my private proxies, and if if they are getting quickly blocked, I'd like it to autoadjust the global thread

There are so many variables (HTML timeout, SE's selected, SE Query Time Search Footprints, etc), and it would be neat if the program did this. Basically, as soon as proxy working % drops, it could lower threads by xx% and if proxy working = 100% it could increase thread count to find the correct balance. (It could also do the same with the custom time between search queries)

FEATURE 5: "automatically disable bad proxies" but I'd like a proviso here "unless active proxies is below xx" 

I'd like to have an option "automatically disable bad proxies" but I'd like a proviso here "unless active proxies is below xx" (i.e. you limit it so your IP doesn't show. A few users have had a bad experience with their IP getting shown when all proxies where deselected. 

Brandon

Thread count has no bearing on your private proxies getting blocked by SE's.  If your private proxies are getting blocked, you need to set the search time higher.  I have 20 private proxies and 600 threads.  My proxies are never blocked and I use a delay of 20 seconds and use my proxies for "Everything".

It sounds like you are having issues with your proxies since you need to test them all the time.  I've been there before, the problem is your proxy provider.  Once you find a good, stable, provider, you won't have the same concerns.  I use buyproxies.org and check my proxies once a week or so and they're always good.

AlexR

@brandon - it's not that the proxies are going dead. They are getting 403'd and then come alive again.

Provider is stable. Why don't you run at 300 threads rather and reduce delay to 10s...20s seems very high!

AlexR

Also - if you use a workaround by getting it to harvest proxies when it falls below a certain level, you get a massive ratio of public to private proxies. 

So currently it's 2500 public to 10 private (working)...the chances of these getting used are very slim...This is why I'd like to see it prioritise the private ones or the fast public ones. 

GiorgosK

from Ozz's discussion back in October
https://forum.gsa-online.de/discussion/740/public-proxies-best-practice-suggestions

Suggesetions for improvements

- seperate private proxies from public which may be used for scrapebox for instance

- better proxy management:

- only use the fastest ~100 proxies in random order 

- and/or using proxies we are knowing that they are working at the moment

- retest dead but formerly fast proxies (< 2 seconds) automatically up to X times. 

- auto-delete old proxies and 

-> option to retest them before banning

-> option to blacklist the IP so they don't get added anymore

- "already parsed" for proxy forum threads and blog posts so the scraper don't parse already

known/old proxy sources 

- option to test proxy right before it get used for posting

My own feature request
https://forum.gsa-online.de/discussion/1512/private-and-public-proxies-together
summary
Private and public should be working together with different settings for each one
My proposition was to get ability to TAG/Classify for each proxy by right clicking
(private/public/good/bad/other) default would be public

Then we could have seperate settings for each tag
or later on within the each project one could decide which proxies to user (by way of TAG)
then some of GlobalGoogler's concerns would go away

It would be nice to get Sven small pieces of changes/additions that would get us to the desired result
otherwise public/private proxies will never work together
and globalgoogler has done a great start
https://forum.gsa-online.de/discussion/1439/feature-requests-please-discuss-add-your-thoughts


If private and public  don't work together prioritized then I really don't see a point of using PRIVATE proxies ALONE, just bought 10 private proxies a few days a go and for 3 days my verified almost tripled (could be other changes I made though) and yesterday it seems that my verfied had dropped to even worse levels than when I was working with public proxies, checking the email spam reports I can see all the proxies are right there in the reports, I will give it another try but I doubt it will change anything ...

AlexR

@sven - are any of these features possible?

Sven

yes but please don't expect me to add everything in 0sec. You all keep asking for this and that and it's getting a bit too much lately. I bookmark things, write things to the to do list but don't push me please.

AlexR

There's no rush. Really not expecting them to be added quickly. Just noting them down when I come across them. 

You're an excellent developer and this program is great as it is. These are just some ideas. :-)

CreditDoc

Would it be better to use an outside program like Proxy Multiply to grab public proxies so GSA can continue to do what it does best :-)

Proxy Multiply will continually scrape and retest proxies at any interval and amount of threads you choose and it will also automate creating reports of the good proxies.  You can filter by practically anything (trans, anonymous, google "safe", speed of proxy, etc).

Maybe one of the options could simply be to pull proxies from a file like GSA does for other fields.

I'm definitely not a coder, but thought an option like this might be easier to implement and it might remove GSA from doing tasks unrelated to submissions.  

If I can help in anyway with this, let me know

GiorgosK

@CreditDoc would prefer if those are handled by GSA internally
the basic functionality is there anyway
and the changes we are asking can only be implemented internally anyway

but @Sven sorry if it seems that we rushing you
you do them when you think its right but we don't want such requests to be lost

AlexR

Neatened the thread so it's easier to see what's requested.

My order of preference is for FEATURE 5 & FEATURE 3 first as they will have the most impact if you're using private proxies.