Skip to content

[FEATURE REQUEST] Proxy Improvement

AlexRAlexR Cape Town
edited January 2013 in Feature Requests
FEATURE 1: I'd like to get the use of the private ones prioritised. 
Sometimes your private proxies go dead, and in this instance it would be neat to have the program check for public proxies. BUT if you use 20 private proxies, and enable it to search for proxies when number falls below xx, you will then get a situation where you will have 3000 public and 15 private. Chances of private being used in this ratio are small!

FEATURE 2:prioritise the use of the fastest ones automatically (not manually). 
If you have a list of 3000 public proxies, would be nice for it to prioritise the use of the fastest ones automatically (not manually). 

FEATURE 3: Would be nice to be able to have it automatically retest them periodically. AND then disable after xxx retests and still dead. 
If I am travelling and I leave my VPS's running, I'd like it to autotest my private proxies from time to time and disable them if they are blocked, but then RETEST them later to see if they have been unblocked. Currently we manually have to test proxies regularly. So I have 20 private working, and I run a campaign, but during the night 10 get blocked, well, it should retest the dead ones every hour, those that fail for 5 retests in a row should be removed. 

Currently, it just keeps using the blocked ones (since I don't use the auto deselect blocked proxies)...so I am missing out on some many URL's. This means I need to clear caches and rescrape everything to get those missed sites. Could just be avoided if the proxies were working. 

FEATURE 4: Test my private proxies, and if if they are getting quickly blocked, I'd like it to autoadjust the global thread
There are so many variables (HTML timeout, SE's selected, SE Query Time Search Footprints, etc), and it would be neat if the program did this. Basically, as soon as proxy working % drops, it could lower threads by xx% and if proxy working = 100% it could increase thread count to find the correct balance. (It could also do the same with the custom time between search queries)

FEATURE 5: "automatically disable bad proxies" but I'd like a proviso here "unless active proxies is below xx" 
I'd like to have an option "automatically disable bad proxies" but I'd like a proviso here "unless active proxies is below xx" (i.e. you limit it so your IP doesn't show. A few users have had a bad experience with their IP getting shown when all proxies where deselected. 
Tagged:

Comments

  • BrandonBrandon Reputation Management Pro
    Thread count has no bearing on your private proxies getting blocked by SE's.  If your private proxies are getting blocked, you need to set the search time higher.  I have 20 private proxies and 600 threads.  My proxies are never blocked and I use a delay of 20 seconds and use my proxies for "Everything".

    It sounds like you are having issues with your proxies since you need to test them all the time.  I've been there before, the problem is your proxy provider.  Once you find a good, stable, provider, you won't have the same concerns.  I use buyproxies.org and check my proxies once a week or so and they're always good.
  • AlexRAlexR Cape Town
    @brandon - it's not that the proxies are going dead. They are getting 403'd and then come alive again.

    Provider is stable. Why don't you run at 300 threads rather and reduce delay to 10s...20s seems very high!
  • AlexRAlexR Cape Town
    Also - if you use a workaround by getting it to harvest proxies when it falls below a certain level, you get a massive ratio of public to private proxies. 

    So currently it's 2500 public to 10 private (working)...the chances of these getting used are very slim...This is why I'd like to see it prioritise the private ones or the fast public ones. 
  • from Ozz's discussion back in October
    https://forum.gsa-online.de/discussion/740/public-proxies-best-practice-suggestions

    Suggesetions for improvements
    - seperate private proxies from public which may be used for scrapebox for instance
    - better proxy management:
    - only use the fastest ~100 proxies in random order 
    - and/or using proxies we are knowing that they are working at the moment
    - retest dead but formerly fast proxies (< 2 seconds) automatically up to X times. 
    - auto-delete old proxies and 
    -> option to retest them before banning
    -> option to blacklist the IP so they don't get added anymore
    - "already parsed" for proxy forum threads and blog posts so the scraper don't parse already
    known/old proxy sources 
    - option to test proxy right before it get used for posting

    My own feature request
    https://forum.gsa-online.de/discussion/1512/private-and-public-proxies-together
    summary
    Private and public should be working together with different settings for each one
    My proposition was to get ability to TAG/Classify for each proxy by right clicking
    (private/public/good/bad/other) default would be public

    Then we could have seperate settings for each tag
    or later on within the each project one could decide which proxies to user (by way of TAG)
    then some of GlobalGoogler's concerns would go away

    It would be nice to get Sven small pieces of changes/additions that would get us to the desired result
    otherwise public/private proxies will never work together
    and globalgoogler has done a great start
    https://forum.gsa-online.de/discussion/1439/feature-requests-please-discuss-add-your-thoughts


    If private and public  don't work together prioritized then I really don't see a point of using PRIVATE proxies ALONE, just bought 10 private proxies a few days a go and for 3 days my verified almost tripled (could be other changes I made though) and yesterday it seems that my verfied had dropped to even worse levels than when I was working with public proxies, checking the email spam reports I can see all the proxies are right there in the reports, I will give it another try but I doubt it will change anything ...
  • AlexRAlexR Cape Town
    @sven - are any of these features possible?
  • SvenSven www.GSA-Online.de
    yes but please don't expect me to add everything in 0sec. You all keep asking for this and that and it's getting a bit too much lately. I bookmark things, write things to the to do list but don't push me please.
  • AlexRAlexR Cape Town
    There's no rush. Really not expecting them to be added quickly. Just noting them down when I come across them. 

    You're an excellent developer and this program is great as it is. These are just some ideas. :-)
  • edited January 2013
    Would it be better to use an outside program like Proxy Multiply to grab public proxies so GSA can continue to do what it does best :-)

    Proxy Multiply will continually scrape and retest proxies at any interval and amount of threads you choose and it will also automate creating reports of the good proxies.  You can filter by practically anything (trans, anonymous, google "safe", speed of proxy, etc).

    Maybe one of the options could simply be to pull proxies from a file like GSA does for other fields.
    I'm definitely not a coder, but thought an option like this might be easier to implement and it might remove GSA from doing tasks unrelated to submissions.  

    If I can help in anyway with this, let me know
  • GiorgosKGiorgosK Greece
    edited January 2013
    @CreditDoc would prefer if those are handled by GSA internally
    the basic functionality is there anyway
    and the changes we are asking can only be implemented internally anyway

    but @Sven sorry if it seems that we rushing you
    you do them when you think its right but we don't want such requests to be lost
  • AlexRAlexR Cape Town
    Neatened the thread so it's easier to see what's requested.

    My order of preference is for FEATURE 5 & FEATURE 3 first as they will have the most impact if you're using private proxies. 
Sign In or Register to comment.