GSA PS + SB = Recommended
So I'm finally getting the hang of using GSA PS and SB together. And I just wanted to say, the program is simply amazing for using Link Extractor, but you have to do it in the right way. Here's what I've learned:
1) GSA PS allows SB to use "connect" proxies, which results in dramatically higher success / completed URLs than just importing + checking proxies in SB.
2) Using 4x SB instances each at 100 threads only + GSA PS Internal server, results in a whopping 5x more completed URLs processed at 1/5th the threads, 100 threads vs 500 threads, in the same timeframe. For example, my 500 thread non-GSA PS SB's at 500 threads completed only 50-75k URLs before the public proxies died, whereas my SB's with GSA PS and only 100 threads completed 250k+ URLs in the same timeframe.
3) Using GSA PS with SB never "lags out" because it's constantly finding new proxies and importing them, opposed to just importing into SB and having proxies die mid-run.
- - - - - - -
There is unfortunately one issue. If you run GSA PS Internal server + SB for a while, GSA PS will be unable to close. Not sure what causes it, but it simply refuses to close the program, even when the SB's have been stopped for a long time. You're forced to ctrl+alt+delete, which removes all new proxies you've added.
The only way to combat this is to do auto-export, AND export all proxies before ctrl+alt+delete. I wish this issue could be fixed, I'm trying to send bug reports every time it happens.
1) GSA PS allows SB to use "connect" proxies, which results in dramatically higher success / completed URLs than just importing + checking proxies in SB.
2) Using 4x SB instances each at 100 threads only + GSA PS Internal server, results in a whopping 5x more completed URLs processed at 1/5th the threads, 100 threads vs 500 threads, in the same timeframe. For example, my 500 thread non-GSA PS SB's at 500 threads completed only 50-75k URLs before the public proxies died, whereas my SB's with GSA PS and only 100 threads completed 250k+ URLs in the same timeframe.
3) Using GSA PS with SB never "lags out" because it's constantly finding new proxies and importing them, opposed to just importing into SB and having proxies die mid-run.
- - - - - - -
There is unfortunately one issue. If you run GSA PS Internal server + SB for a while, GSA PS will be unable to close. Not sure what causes it, but it simply refuses to close the program, even when the SB's have been stopped for a long time. You're forced to ctrl+alt+delete, which removes all new proxies you've added.
The only way to combat this is to do auto-export, AND export all proxies before ctrl+alt+delete. I wish this issue could be fixed, I'm trying to send bug reports every time it happens.
Thanked by 1nycdude
Comments
I actually DLed and installed the demo of GSA PS just this morning!!!!!!!
@Sven
On this note, I'm running 6x SB's with 100 threads each all going through Internal Server. Now I know the Internal Server seems to not work well with a lot of connections, but it's running fine... however, I just got a "Application Error" popup from GSA PS, and it said "continue running" or "close". I pressed Continue Running and my SB's still seem to be processing.
I sent a bug report about the issue, please look! On that note, are the bug reports not saved anywhere that I can look at? I'd be interested to see what causes this issue, or rather, what the bug report shows.
Is there any way you could tweak the Internal Server to work with more connections... even if it requires more CPU? Because as of now, it doesn't really use much CPU... but if it could use more when there's a lot of active threads, that would be great...
Thanks!
Basically I was running 6x SB's at 100 threads each, and after a while GSA PS threw the "GSA Error" where the popup comes, and says Continue Running or Close Application. I pressed Continue Running, then submitting a bug report. It surprisingly kept working somehow... but, as with the memory leak/error I described when I first got the program...
After this error. I decided to stop the SB's and restart GSA PS, and when I restarted (even though it closed properly) the "state" was reverted back to 10k proxies in the list, when there should've been only 7k.
Now I'm trying 4x SB's at 100 threads, and 2x at 50 threads to see if it happens again. I also tried running the program as administrator to see if it makes a difference. However this is on Windows 7, which usually it doesn't matter either way.
Second one, I changed the search function from 60 -> 1 (just so it would start) -> then back to 60.
Third one I didn't touch any settings at all. I just opened GSA PS, started 1 SB campaign, and it instantly crashed. I usually run my SB campaigns with GSA PS scraping proxies at only 5 threads, just to have a small drip of them coming in.
Note that I do have auto-export set to every 30 minutes, mainly because I'm afraid it's going to crash and I'll lose all of the proxies that it's harvested while I'm asleep. But in either of these 2 cases, I did not try to export anything.
Whatever the updated changed though (reading the changelog it seems insignificant) made the crashes happen instantly, within seconds of starting a SB campaign. I am back on the old version now, which at least doesn't crash instantly.
That does make me wonder if the auto-export is causing the crash now that you mentioned it above... but how else can I ensure the proxies are saved in case it crashes while I'm away?
I am testing 800 threads between 4x SB's now (200 each) and no crashes... It may have very well been the auto export causing the crashes the whole time.
Update: I am fairly certain that it was the auto-export every 30 minutes causing the crash. Running at 800 threads on 2 different machines, haven't seen a single crash. We will see tomorrow when I wake up if it holds! But nice catch Sven and thanks a lot for the help!