GSA SER stops to take a breath at high LPM
Hi,
I noticed that GSA SER is slowing down a lot after 10-20 minutes, complaining that it's out of targets. It then pauses for a few minutes, and then picks up the pace again.
I'm using a global site list of 300k+ identified guestbooks and image sites in a project. I imported this site list from Options -> Advanced -> Tools and let GSA SER identify the targets. Search engines are disabled in the project and I'm not using proxies (posting directly from my IP).
A couple more details:
- This is the only project that is in active state.
- Verify links is turned off
- The project doesn't using search engines for scraping. It uses identified sites from the global list (300k+ sites were imported).
- Before running the test, I cleared URL cache and URL history on the project. All URL filters are off, including "avoid posting URL on same domain twice".
- During the submission process, CPU usage remains under 50% and there's bandwidth to spare on the server. I'm not using proxies.
- Captcha Sniper is used to solve captchas.
- The project uses verified URLs from a sub-project that is inactive.
- 200 threads, monitor PC resources and automatically lower threads is turned off
Here's the log file (URLs removed):
Start:
[10:28:49] test project: Setting up project
[10:28:49] test project: Starting project
[10:28:49] test project: [ ] Attention! No more targets, but also no search engines selected in project.
[10:28:50] test project: [ ] Loaded 538/538 URLs from site lists
[10:28:50] test project: [ ] Attention! Option "Verify submitted links" is disabled in project options.
[10:28:50] test project: [ ] 001/538 previously successful - hxxp://domain.com
[10:28:50] test project: [ ] Loaded 99/99 URLs from site lists
[10:28:51] test project: [+] 002/538 matches engine Icybook - hxxp://domain.com
[10:28:51] test project: [-] 002/538 text captcha found and skipped hxxp://domain.com
[10:28:51] test project: [-] 003/538 no engine matches - hxxp://domain.com
[10:28:51] test project: [+] 004/538 matches engine Icybook - hxxp://domain.com
[...] (edited out, lots of posting)
Throughout the log file, there are messages such as "Loaded 3/3 URLs from site lists" which maybe are not enough to keep GSA SER well fed at the speed it's going (200 threads, 100+ LPM)? I think the problem is easier to reproduce with more threads.
[10:41:58] test project: [ ] Loaded 2/3 URLs from site lists
[10:41:58] test project: [ ] Loaded 1/1 URLs from site lists
[10:41:58] test project: [ ] Loaded 3/4 URLs from site lists
[...] cut to the moment the pause happens:
At this point, the number of active threads is slowly going down to 0. Once it reaches 0, notice GSA SER stops and stays idle for a few minutes, complaining it's out of targets. Then it starts again, loading only a couple hundred targets from the list of 300k+, and the thread count goes back to 200.
[10:44:49] test project: [-] 402/402 required variable "url" was not used in form. hxxp://domain.com
[10:44:51] test project: [ ] Attention! No more targets, but also no search engines selected in project.
[10:44:51] test project: [ ] Attention! Option "Verify submitted links" is disabled in project options.
[10:44:54] test project: [+] 271/400 matches engine Trackback - hxxp://domain.com
[10:44:54] test project: [+] 271/400 new URL - hxxp://domain.com
[10:44:54] test project: [-] 271/400 unknown submission status - hxxp://domain.com
[10:44:54] test project: [+] 271/400 matches engine BellaBook - hxxp://domain.com
[10:44:54] test project: [-] 271/400 unable to find suitable URL
[10:44:54] test project: [+] 271/400 matches engine Phoca Guestbook - hxxp://domain.com
[10:44:54] test project: [-] 271/400 required variable "url" was not used in form. hxxp://domain.com
[10:45:03] test project: [+] 272/400 matches engine Trackback-Format2 - hxxp://domain.com
[10:45:03] test project: [+] 272/400 new URL - hxxp://domain.com
[10:45:03] test project: [+] 272/400 submission successful (1642 submitted - AVG: 6071.96/h) - hxxp://domain.com
[10:45:03] test project: [+] 399/399 matches engine Pixelpost - hxxp://domain.com
[10:45:03] test project: [ ] 399/399 waiting 5 seconds to not spam. hxxp://domain.com
[10:45:03] test project: [+] 399/399 submission successful (1643 submitted - AVG: 6073.81/h) - hxxp://domain.com
[10:45:51] test project: [ ] Attention! No more targets, but also no search engines selected in project.
[10:45:51] test project: [ ] Attention! Option "Verify submitted links" is disabled in project options.
[10:46:52] test project: [ ] Attention! No more targets, but also no search engines selected in project.
[10:46:52] test project: [ ] Attention! Option "Verify submitted links" is disabled in project options.
[10:47:52] test project: [ ] Attention! No more targets, but also no search engines selected in project.
[10:47:52] test project: [ ] Attention! Option "Verify submitted links" is disabled in project options.
[10:48:51] test project: [ ] Loaded 241/286 URLs from site lists
[10:48:51] test project: [+] 273/400 matches engine mygb.nl - hxxp://domain.com
[10:48:51] test project: [-] 273/400 no form at all - hxxp://domain.com
[10:48:51] test project: [+] 274/400 matches engine Guestbook - hxxp://domain.com
[10:48:51] test project: [-] 274/400 unable to find suitable URL
[10:48:51] test project: [+] 001/274 matches engine Burning Book - hxxp://domain.com
[10:48:51] test project: [-] 001/274 unable to find suitable URL
... and it goes on at great speed of 100+ LPM, until it's out of breath again. Any idea?
Tagged:
Comments