Skip to content

Frozen threads?

fakenickahlfakenickahl
edited January 2014 in Bugs
Hi,
Recently I've been unable to run gsa ser at speeds which I'd like. Ser seems to run just fine and very fast at first when I start all the projects, but after some time (30 minutes - 1-2 hours) it is only using a very small amount of threads. I've set it to 2k threads at the moment, but only 1-2/8 projects (I only see these in the log for several minutes) will use threads and only 300-400 at most. However when I stop all the projects with the big red stop button, the projects which were running just fine will quickly finish what they were doing and stop, as expected. But the last 6-7/8 projects will not finish. I have sometimes seen that the projects which were halted will finish as expected, yet it's first after 5-20 minutes if they ever finish. Also before these halted projects will finish as expected, ser says it's only using something like 5-20 threads, yet the stopping project will post a LOT into the log.

I've also seen this behavior happen and resolve itself after a few hours. Some projects will not submit, search, or anything for a long time then suddenly start going crazy. I'd like to note that all my projects have lists of 3 million urls imported, 168 search engines enabled, 100k keyword lists, and lots of public proxies along with some private ones though it usually is plenty for me with just public proxies.

The screenshot below shows how 2/8 projects have stopped after pressing the stop button, yet 6 projects are still hanging around only using 6 threads. The screenshot is taken 7 minutes after stopping all the projects, yet they're not stopping and still aren't 35 minutes later. They're behaving the same way while running as they're not doing anything.
image



Please tell me if I can provide more information if my problem is unclear @Sven and I'm sorry for the wall of text, but I wanted to make my problem as clear as possible. I'm also not sure if this is intended, but it just seems very weird that projects will not do a single thing for a looong time, also even after all projects have been stopped and it's really killing my LpM.

Comments

  • SvenSven www.GSA-Online.de
    Your proxies seem to be the problem. 1000+ public proxies...that seems way too much and many will be down or dead already. Testing them all might freez the program I don't know. Maybe just use your private proxies and skip the public once or at least only trim to the sources that deliver some good/stable once.
  • fakenickahlfakenickahl
    edited January 2014
    I appreciate your reply, but I don't really think it's because of proxies. A few of my projects are still running fine when I run into the problem described above, yet the majority of projects doesn't run at all. Right now I'm testing proxies with 300 threads, every 15 minute, and proxies which are detected to be down on use are disabled and removed after 5 minutes. It used to be just fine to use this setup for me when I was using a vps 2 weeks ago. What also makes me believe it isn't because of proxies is that the issue only starts after a few hours usually.

    However I've started to use a source which is unknown to most, which gives about 1-2k google passed proxies of high quality every 30 minute. I'll use these and test less often (like every 60 minutes) and see if it makes a difference and report back.
    Edit: also I'm using Captcha Breaker, and I've tried cranking up the speed to see if that can possibly fix it along with lowering my threads from 2k.
  • SvenSven www.GSA-Online.de
    OK, yes try to optimize your proxy config. I think thats the bottelbneck here.
  • namdasnamdas
    edited January 2014
    To be honest I think there is an inherent limit to the number of threads SER / Windows can handle. I was having the same issue at 1000 threads on my dedicated server and hitting 100% CPU, so I tried running it on a 32 core beast of a server. Even though the CPU usage fell to just 15% and I was only using a fraction of the unrestricted 1gbps bandwidth I couldn't push SER far past 1200-1400 threads without the same thing happening.

    I'm a bit surprised as that was obviously a server class cpu that would be used for sites that get hundreds of thousands of visits a day so assumed it would take a lot more of a beating, but there is obviously a software bottleneck somewhere.
  • fakenickahlfakenickahl
    I tried changing my proxy source around. I used to scrape around 20-30k proxies at a time, and I can see how that could interfere with GSA. I'm still experiencing the same problem as mentioned in the op, but it actually seems to sort itself out after an hour or two now. However when all my projects do "freeze" (nothing new shows in the log) I can see that lots of captcha are being sent to captcha breaker and being solved. Though I'd like to note that at one point I opened the window with ticked and unticked proxies, where it had scraped proxies and should have finished testing a rather long time ago, yet it had 2 threads running which didn't stop after 15 minutes and pressing abort. While this was going on, the majority of my projects were also frozen.

    It could also be due to the fact that I changed the settings in captcha breaker to solve captchas much faster, or the fact that I've reduced my threads from 2k to 1,6k.

    I appreciate both your input though. I'll try further optimizing my settings in gsa.
  • tsaimllctsaimllc
    I actually have the same problem as OP and I have mentioned it before, but no solution. I had this happen to me today as well. I dont really understand it. I dont have it set that high, like 230 threads now, and sometimes it will run single digits even though there are projects that should and can be running. I use all private proxies and they are working.

    In fact  right now it is using 8 threads
  • tsaimllctsaimllc
    https://forum.gsa-online.de/discussion/6175/thread-usage-low#latest never got a real answer.
  • tsaimllctsaimllc
    fakenickahl did you find a solution? This really needs to be looked into... no one else having this issue? I have 50 private proxies, and set threads to 200 and right now it says it is using 18 and with a LPM of 2.88 which is SHIT.

    I am running a complete churn and burn so its every kind of link checked so it isnt stopping because of filters. Still should be using all 200 threads.
  • fakenickahlfakenickahl
    @tsaimllc I thought I had found a solution to it and that it would just fix itself after some time. But turns out it doesn't. The majority of the domains I was spamming have burnt now though, so right now I've only got 2 projects running however it seems one project is more likely to freeze than the other. Though I just woke up to both projects having done nothing for the last 12 hours with still 1179 threads running.

    I'll try to reduce the amount of threads even further to 900.
Sign In or Register to comment.