Option to make Internal Server more lenient with "Detect to be down on use"

seoaddict

When I'm scanning links with SB, the proxies are "detected to be down" and the list moves down really quickly because, I think, I'm using many threads with the proxies at the same time causing them to be "detected as down," when they're actually not.

For example, if I re-test the not working proxies while my link checking is going on, they're all still working, but that requires me to re-test every X minutes because they're flying down quickly.

Is it possible to add an option for the Internal server... "re-try proxy X times before marking as down"? This should pretty much solve this issue, as if GSA PS re-tried the proxies it would see that they were indeed still up and working properly.


- - - - - - - -

Edit:

Actually it seems this isn't anything to do with multiple servers. When you're running a lot of threads, GSA PS will just automatically zoom down the list and mark them all as "down" even when they're all working. "re-try proxy X times before marking as down" option should be perfect for avoiding running down to 0 working proxies in under 30 minutes.

Edit 2:

Yes I can confirm 100% that it's using many threads that cause GSA to mark the proxies as being down when they're not. It seems that ~500 threads is the limit otherwise the list flies down and you go from 30k -> 0 in under 30 minutes. "re-try proxy X times before marking as down" would be perfect and allow for many more threads to be run.

seoaddict

Meh I found the main problem. I was testing running lower amount of Link Extractor instances with higher threads. GSA PS did not like this. Link Extractor has a really bad CPU usage issue right now that they're working on fixing it, but 100 threads uses the exact same CPU as 300 threads, so I was testing.

But it does seem that the option I mention above would make Internal Server 100x more powerful. As it stands now, even running 6x instances at 100 threads per for a total of 600 threads is making the proxies move down fairly quickly (nowhere near as quickly as before when I was testing more threads though), when they're actually all working great... Just need an option to let GSA PS know they're not actually down.

Sven

next update will improve the down-detection.

seoaddict

Whatever you did @Sven, helped a bit (originally I thought a lot, but after a few minutes of running ~700 threads, the proxies are moving down wayy too fast still - re-testing = 100% of them are still working).

Maybe make this the default and add an option for advanced users to change. Or just make an option to completely disable the "down detection" and just leave the proxies running forever (I'm assuming this would be the easiest to implement - this function exists in SER already), that would work a lot better for people running a lot of threads.

Still going down too fast to leave it running overnight. Oddly, I don't remember the proxies going down this fast a few weeks ago... I think it's because a large majority of the URL's I'm scanning are being successful, whereas a few weeks ago, I was getting more errors (cloudflare URLs etc that were erroring, but now I've filtered them).

Edit: Well after 15-30 minutes of running, the proxies are still going down just as fast as yesterday. Hopefully in the future there will be an option to disable down detection entirely or make it "re-try proxy X times before marking as down". Because Internal Server is 1000x superior than running proxies directly in SB, as SB can't even confirm that 90% of the proxies even work.

Sven

ok im improving it further in next update.

seoaddict

Any idea why one of my servers refuses to offer to update? Both others work fine... and offer the update. This one even if I press “check for update” it says “your version is up to date”.

I have to manually transfer the exe / root folder from another server and I’m worried doing this is going to miss some part of the update.

seoaddict

I just re-installed hopefully it's fixed. It seems to be a bit better now, although I'm getting a lot of fails on the latter threads. The proxies are still going down, but nowhere near as fast.

Question though, what happens if GSA PS gets to 0 proxies? I've even seen stuff like "-215" proxies before. Is it then using my real IP? Or will it just stop processing the data until is has available proxies to use?

Sven

if there are no proxies, it will fail and not use your real ip

seoaddict

Alright last question -- how often does the scheduler re-test the not working proxies automatically? I have the search function set to every 30 minutes.  I unchecked "re-test working proxies" and only having it re-test not working proxies.

I just logged in to a server and to my surprise it's back at a high number, which tells me that the scheduler re-test the not working automatically and now it's back. If I can be sure of when that will happen, I think everything will be great.

seoaddict

So if I run all SB instances at 100 threads each, GSA PS actually responds really well. I'm thinking that because I was running 2x at 200 threads each (then the others at 100 threads each), GSA PS was having trouble because 2 instances were trying to scan too fast. Also the updates you made are helping dramatically, I can easily tell the difference.

But I have 8x at 100 threads each right now, and everything seems to be working perfectly.

Still interested in the question above when you get a chance though!

Thanks as always Sven!!

Sven

the none working proxies get tested each time after the proxy scan and when working proxies get checked.