Skip to content

Best practice for using multiple GSA PS installs?

seoaddictseoaddict
edited August 2018 in GSA Proxy Scraper
@Sven curious to hear your thoughts on this:

I have a ton of proxies that are in GSA PS, using various sources,  Proxy Buddy, scraping a ton of my own sources. Sometimes I can have 30k proxies that are tested and working by GSA PS. The beautiful thing is that Scrapebox test shows 80% of these SOCKS proxies don't work, but with GSA PS they work like a charm. It's amazing really.

The problem is that, when I use the same list across multiple PCs/servers, I imagine the proxies are getting hit at the same time, and being detected as down even though they are not. For example, I just checked a server that was running 800 threads between 4x installs, it had only 100 valid proxies left!

Edit:

I think I found a solution Sven. Could you add a feature that is "Automatically re-test 'Not Working' Proxies after X minutes?" I just did a re-test of not working proxies and it worked perfectly, now I have 26k again, although going down of as they're being used. I'm pretty sure this feature would solve the problem...

Edit 2:

Uhh... so after I re-tested the Not Working proxies, I check back and within 1 minute it went from 26,000 -> 14,000...?

Wow, after re-testing Not Working, the proxies almost INSTANTLY went to 0 alive... It has to be that I'm using the same list between 2 servers or something... because these proxies are up when tested by GSA PS, but then detected as down instantly when used for SB.

Edit 3:
What does "Test error: No buffer space available" mean? I was looking into this last night and tried some stuff in Windows 7 like increasing the max ports etc... hmm well I think the timeouts are too high, trying to change them now. Just downloaded TCPView and like 5 minutes after not running into connections, there were thousands open.

Maybe I should install this but I don't have a "Multiprocessor PC" so I didn't install it:
https://support.microsoft.com/en-us/help/2577795/kernel-sockets-leak-on-a-multiprocessor-computer-that-is-running-windo

But, does GSA PS create "loopback sockets"?

Comments

  • seoaddictseoaddict
    edited August 2018
    Testing some windows settings now to try and clear connections faster. Okay I think I was hitting the max port cap on Windows 7 Pro... I decreased the TIME_WAIT timeout to 30, apparently the lowest Win7 will allow, should be better now. This is likely why the proxies were moving down so quickly as well...
  • seoaddictseoaddict
    I think I managed to fix it by tweaking some Windows registry settings... Pretty sure the problem was old connections hanging in TIME_WAIT too long, and not enough sockets were available, hence the "No buffer space". The proxies also seem to be going down dramatically slower in GSA PS.

    I'm thinking that when the buffer space ran out, GSA PS just basically started canceling the proxies at a rapid rate, because it had no choice.

    For anyone else who has this problem: Here are the registry settings:

    HKLM > SYSTEM > CurrentControlSet > services > Tcpip > Parameters

    TcpTimedWaitDelay - 30 (decimal)

    MaxUserPort - 65534 (decimal)

  • seoaddictseoaddict
    FYI the post above 100% fixed the issue. If anyone else is using a lot of threads with SB  + GSA PS,  use that method!
  • SvenSven www.GSA-Online.de
    thanks for the feedback. I should add it to FAQ
  • seoaddictseoaddict
    edited August 2018
    Yes also a warning

    TcpTimedWaitDelay cannot be set lower than 30. Apparently if you try it, it will default to 120-240.

    The MaxUserPort number I used is also the highest the OS will allow. I’ve had no problems out of these settings yet.
Sign In or Register to comment.