Skip to content

GSA works fine, but eventually gets very slow and never recovers

piscopisco
edited February 2013 in Bugs
Don't know if there is a better place for this, maybe a ticket service, but since i didn't found it, i'll just do it here.

I've been using GSA for a while, but only recently, inspired by the lpm thread, i've been tweaking around, and used the global list (just wasn't using it), but now i've hit a wall.

If i delete url cache and url history for all my projects (i do that if i change settings all around or add new kw, just to make sure it's all ok), it goes fast, around the 20lpm mark (hey its fast for me atm :p) and keeps up that speed for about a day or so, using all threads (usually 200). The problem is that eventually something happens and the whole thing gets very sluggish, cpu at 99% threads drop to aroud the 30 mark, and lpm goes below 1.

Restared VPS, restarted CS, restarted SEO Indexer, restarted GSA, lowered threads, but even at 10 threads, still sluggish, and cpu bottoms out, tried isolating projects also but nothing, the only thing that works is clearing cache and url history, but just for a day or so, after that, all the same.

The VPS has 2g Ram, and it's always at 50% usage, so don't think its related, private proxies are ok (30 of them), run about 12 projects, tried sheduling also but same result, so basically i'm a bit frustrated atm and looking for help.

Thanks
«1345

Comments

  • LeeGLeeG Eating your first bourne
    How many keywords do you have?
  • rodolrodol
    are you importing any list?
  • BrandonBrandon Reputation Management Pro
    Sounds like they're verifying during that time.  Your LPM is going to drop while they're verifying since nothing is being submitted.  I tend to restart my VPS every day or two.
  • LeeGLeeG Eating your first bourne

    If threads are dropping like you say, its normally keyword related

    Try scraping about 10k keywords

    Niche related upper tiers, general keywords lower tiers

  • piscopisco
    i thought about the projects beeing verified, but i think that would account for temporary drop in performance, but it should rise after this, problem is it never does rise. Tried restarting the vps but it has no effect, only thing that works is clearing url history (isolated the cache clear, and by itself doesn't fix this), about imported lists, not really these are just running keywords and the global lists

    Most of the projects are using keywords from a txt that someone here supplied of the most searched kw (about 3k) in some projects i added around 1k of some more niche related.

    What really puzzles me is that the threads drop but the cpu usage does not, for instants yesterday i cleared cache and url, and managed to up the theads to 400, and still didn't hit consistent cpu max usage.

    Those that are running high lpm, do you guys manage to do that consistently, or what maintenance do you have to perform in the projects to keep it up ?
  • LeeGLeeG Eating your first bourne

    Try increasing your keywords

    I can kill 300k + duplicate urls each each day

    Something I personally do after each restart of ser is kill duplicate urls

    Main options > advanced > tools > remove duplicate urls

    On your sitelists, only save submitted and verified urls under the advanced options on the main options tab

    And again, on each project only use submitted and verified site lists

  • piscopisco
    Just added a massive amount of kw (100k) to each project, and shortened a bit the verification time (12h) but the result is basically the same, it starts to grind and eventually goes below 1lpm in about 24h, what can i analyse in terms of the log to find out what is wrong.

    Also on another note, there a column in the keywords grid , "Found", but in all my projects that always shows 0, anyone has values on that column ?
  • rodolrodol
    the found is not a big deal, but your verification time its too low, you should use at leas 1440 minutes, you go to 1lpm because SER spend so much time verifying, also i believe 100k its too much keywords, 5 - 10k its enough
  • sonic81sonic81
    I"ve been running tests all morning I have the same problem as pisco.  I can hit 100lpm early but after 12-24 hours it slows to a grinding halt to like less than 1lpm.

    I find if I delete my target url history and everything is fine and it goes back to normal to > 100lpm.  I have no idea what's causing this when I run two projects by themselves (after 12-24 hours when gsa has "frozen") the log seems find it searches fine but it runs really slow on a few threads and doesn't save any targets in the url cache.

    Anyone else have the same issues?
  • sonic81sonic81
    edited February 2013
    my thread count also drops drastically like @pisco too after 12-24 hours.

    the only way I can get it to speed up again is delete my target url history.

    I'm also using 100k+ keywords.  

    The only thing I can think of is some of the 100k keywords are in foreign language not sure if this will cause a problem with GSA, it imports OK...
  • rodolrodol
    test with 5k-10k keywords and let me know
  • sonic81sonic81
    edited February 2013
    @rodol.  thanks for your input i appreciate it.  did that still doens't work.  i'm trying to debug by manually starting single projects and seeing which ones don't work.  here is an example log of one which has "frozen"

    it runs as per normal but target urls are always empty even after watching it for 15 mins.  here is the log:

    Keeps going loading 0/414 urls from sitelist

  • DunceDunce
    My guess, when this happens next time, check your left target url list, it should be short, less than 100 maybe.
    I am thinking, the lpm droped because of run out of target list to post, so ser could not start more threads. But the CPU goes up that's something I can't figure out, any possible you select too many search engines and keywords to find target urls?
  • sonic81sonic81
    I think I've found it after debugging each project manually.  it appears i've been posting heavily and google have banned some of my proxies without me realizing it. (i run 6 google search engines)  i always test my proxies first but when I selected to test the proxies I was only testing www.bing.com (the default) instead www.google.com or whatever local google i had selected.

    i've changed my search engines for now and it appears to have fixed it.
  • piscopisco
    Glad i'm not the only one with this issue, was starting to think i was beeing paranoid, i actually test the proxies with gmail, but either way if it were the proxies, how would clearing target url history solve the matter, about the kw, 100k or 10k the result is the same really.

    About the target list, since we are using the global lists this really shouldn't be an issue, as the app always has the global to work on..
  • sonic81sonic81
    @pisco.  i ahven't fixed the problem even after getting new proxies and selecting new SEs.  i've tested it down to 1 single project and I can confirm that it stops running and the threads dwindle down after just 5-10 mins.  weird other people aren't getting this there must be something specific to our settings.
  • sonic81sonic81
    @pisco also i'm using captcha sniper not captca breaker what are you using?

    i doubt it's related but after upgrading captcha sniper to x3 i've had some problems when the captchas aren't fed thru to captcha sniper (don't think captcha sniper can handle the speed), need to reboot to fix it.  i doubt it's related to the above issue but just some extra info.
  • piscopisco
    CS also, but don't think it has anything to do with it.

    That problem you are having with it have it also (feeding the captchas), what i do is delete the CS license file, re-appl the file they sent to activate on the email, and it works again, i'm waiting for the next version to see if it is fixed.
  • sonic81sonic81
    edited February 2013
    @Sven,

    I finally managed to find a consistent way to reproduce the problem.  In a nutshell I've tested v5.08 and v5.16 and it's consistent in both.  I've isolated to one project only which has a number of engines selected (ones based on my data identified / verified > 10%).

    It runs for about 5-10 mins at a high LPM then consistently drops off to basically nothing (the number of threads goes from 188 to 1).  The project is using 6 googles SEs, 4k keywords, 25 buyproxies private proxies.  When I click stop and start again to restart the project (WITHOUT a SER shutdown) it runs correctly at 188 threads for 5-10 mins.  This happens to all projects.

    Here is the log I've put in paste bin I've gone over it 3x myself can't find anything wrong with it.

    I'm not sure but my best is guess is there is some condition which is causing the thread count to reduce automatically.  I've even tried unselecting monitor CPU usage and reduce threads and the problem stil happens.  I can consistently reproduce this over and over again.

    log #2 for the same issue appearing in a few mins after i restarted the project

    thanks in advance @sven
  • sonic81sonic81
    @sven.  i've done a jing video of the issue occuring but it was hard to catch it exactly when it did.

    the recording starts when the threads are already down to 1 thread / 0.  What's interesting is around the 38 second mark for some reason the threads jump automatically back to 188....  i'm using SER v5.06 with 1 project only in this video.

    click play flash fullscreen link to see it larger


    hope this helps you debug let me know what more info you need.  thanks in advance.
  • SvenSven www.GSA-Online.de
    @sonic81...good news. Thanks to your logs, I was able to locate the problem. A new version will be released in some minutes. Though as you said, this problem was only happening to a few people.
  • sonic81sonic81
    @Sven.  Awesome!!!  you are seriously the best with the response time on making dev changes!!!
  • rodolrodol
    @sven rocks! =D>
  • ronron SERLists.com

    @sven, You have my curiosity peaked that it was only affecting a few people. I would have thought it was systemic and would have affected evrybody. 

    I just want to know what the issue was so we can all learn.

     

  • sonic81sonic81
    @sven update i found the issue still exists for 2 projects even though it fixed it for some others i've sent u the log. thc
  • SvenSven www.GSA-Online.de
    @ron I have mixed a check for keyword when using macros or keyword extraction and the reload of verified URLs from a main project.
  • divatzdivatz
    This has been happening to me for a couple of weeks now, thought i was doing something wrong. Kept tweaking my settings but LPM didn't improve at all (it actually dropped even lower during the past few days).

    PS: The latest update didn't fix things, at least for me.
  • SvenSven www.GSA-Online.de
    @divatz please provide the project backup to me than.
  • sonic81sonic81
    edited February 2013
    @divatz i've been having the same issues i always thought it was me as everyone was running fine for most people.  my LpM has been dropping as well and still very low even in v5.17 so i dont think it's fixed either.

    Does your behaviour match mine shot in the above video where running 1 project goes to 0 threads?

    @sven sent u my project file.
  • divatzdivatz
    @Sven sent you a message with 2 of my projects

    Thanks
Sign In or Register to comment.