Skip to content

Weird side-effect from "down detection" update?

edited August 2018 in GSA Proxy Scraper
The down detection is working well, but it seems to have had some sort of unintended side-effect of making the internal server "lag out" after hours of running. The program seems to quit feeding SB proxies, and the Link Extractor instances just come to a halt. This is to be expected on public proxies (without GSA PS), as they go down and SB can't pull any working ones. However, one of the major benefits of using GSA PS + SB is to avoid this issue.

While investigating, I found a weird thing in the TCPView. Usually "localhost" is never shown in the connection manager from what I've seen (it's not a "remote address"). I think whatever happened here, is that GSA PS is "lagging out" somehow. It could also possibly be SB lagging out, but I ran this exact same setup weeks ago and never saw this issue with GSA PS. It only happened with public proxies without GSA PS.

When I logged into the server today, all instances of SB Link Extractor had stopped moving, as if there were no working proxies.. but there were 50k working in GSA PS. I often check the connections to clear them out using TCPView, but I've never seen "localhost" actually be listed in the connections.


  • edited August 2018
    May actually be SB causing this issue...  I'm really not sure. But when I checked this morning, I had each SB running 100 threads only. The amount of open ports were quite small, only a dozen of so.

    And actually EDIT: I just checked TCPView while everything is still moving, and those "localhost" entries are indeed there, I just never saw them before because too many ports were open and hard to see. But I sorted and they are there. (Makes sense because of the Server, but I just never saw them).

    Hmm... I wonder what's making everything stop after 80,000 URLs scanned. Possibly something to do with the time-running instead of the threads etc. I tested 2000 threads tonight and although GSA PS proxies flew down quickly, I got 200k URLs scanned without any slowdowns at all.
  • SvenSven
    hmm no idea really...i haven't made any deeper change except that "set down if detected as down 3+ times".
  • SvenSven
    what about latest update @seoaddict
  • edited August 2018
    The proxies are going down much slower now. I’m at 1000 threads (2x at 500 each testing) and not even having to search at the moment. Definitely huge improvements.

    One thing though, no matter what, this one server of mine will never get the update. It just never sees there’s an update available, even if I click check for update, it says I have the latest one but it’s not. No firewall enabled or anything. Both others receive all updates fine... same Windows, same updates, same TCP settings, etc. What could cause this? (Note that GSA server would be seeing these 3 installs on the same IP).

    Far as the “stall” thing I’ll have to see in a bit (testing a large list now), I think maybe it’s an issue with Scrapebox, or possibly the TCP Windows time-outs I’m using. But the down detection is definitely way better now, I can leave everything on overnight and not worry about.

    Thanks for the updates!!
  • SvenSven
    update: if no reply is received, the update tells the user there is nothing so maybe you are hammering your network too much to get any data? ;)
  • edited August 2018
    Well the “stall” on SB still exists, all my campaigns stalled out at ~200k links and everything just quit moving.

    Also it seems that 2.64 and 2.65 go down at about the same speed... was 2.64 a bit slower to go down (more retries) or should 2.65 be a bit slower?

    Re: not getting update — Nope even if I turn everything else off... stop all programs, shut down all PCs, restart router and turn only the 1 on... I won’t receive the update. But all others will even if running at 100mbps there’s never a problem receiving updates.
  • SvenSven
    no clue really
  • edited August 2018
    @Sven I sent in a bug report on that server, see if you see anything. Proxies are moving down about the same as 2.64, but it does indeed seem to be performing better. Also somehow as a by-product, it seems to be using slightly less bandwidth somehow. I'd say, maybe 5-10mbps lower... mainly just fluctuating more than before (in a good way). It's going up higher than before, then lower than before... on average, I'd say at least 5mbps drop. Could just be coincidence idk.

    Tonight I'm testing removing all TCPIP registry settings I added from a server to see if it does anything with the stall issue. I really don't remember it doing this before I added these settings... the settings actually helped a lot with errors and stuff though. It used to do this with public proxies in SB, but never when using GSA PS.
Sign In or Register to comment.