Skip to content

GSA+CS crashing servers?

mmtjmmtj
edited November 2012 in Bugs
We recently have some weird issues with either GSA or Captcha Sniper (or both) on our private servers.

We found that our VPS and our Dedi Servers crash over night. They seem to be running, but we can't connect via RDP. After talking to PowerUp (one of our providers and well recommended), he gave us some insight and the issue may have to do with the use of unused RAM resources, subsequently resulting in a system crash - but we are not 100% sure.

We had this issue over more than 5 servers for the last 4-5 days. We are sure we are not the only ones with that problem, but it's rather frustrating to restart servers all the time.

Would be great if any of you could confirm our problem.

Setup:
330 Proxies per server
~200-300 threads per server
CSX
No scraping, just using our own link database.

Cheers,
mmtj

Comments

  • chaplin35chaplin35
    I actually had similar problem connecting to my VPS via RDP for a month or so, but it seems to have stopped in the last week or so. My problem was that I couldn't connect via RDP for a certain time, but if I waited long enough (sometimes an hour, sometimes 5) I could connect again.

    I have started using proper CS so not sure if that's the reason why it's OK now but before I had a cracked version of CS installed and it was happening nearly every day.
  • mikie46mikie46
    edited November 2012
    i have the EXACT issue as the OP.

    Unable to access the server via remote access in the AM.

    GSA is crashing my server for the past 2 nights.

    It has never done this before. 2 days ago i upgraded from 4.58 -> 4.61. By this morning i could not access the remote server which meant it was hung up or crashed. I had to do a reset for the 2nd time.

    Finally when i logged in GSA stated it was back at v4.58? How is that possible? I did an upgrade 2 days ago. Then i ran though the upgrade again.

    Whats going on with this?

    Also, you state this in the changelog;  4.60 - new: added a basic default badword filter list that gets added to newly created projects. You still need to enable that option though.

    But where is this option? When i click NEW i dont see this option anywhere.

    Thinking about it, i should not have upgraded to 4.61 again this morning. I should have left it alone and continued to run 4.58 until such time that this issue is fixed. I have no idea how it reverted back after this crash anyway.
  • OzzOzz
    edited November 2012
    can all of you do some test with RamMap? i have a suspicion which i observed some time ago on my vps where my "page table" was maxed out after a while (= danger of running out of memory).

    1) restart all your tools once and make a screenshot of RamMap after restart
    2) let it run for a few hours, stop SER and make sreenshot of RamMap again
    -> compare the "Page Table" stats.

    what else you can test.
    after all your tools have run for a couple of hours take a look into your task manager -> performance, stop all tools and close CS or SER first and monitor the changes in RAM usage. if turning off one of the tools free up unusual high amunt of memory than you have at least a trace if one of the tools could cause a memory failure.

    however, @sven will be back again tomorrow and give you better advice.


  • poweruphostingpoweruphosting Los Angeles
    edited November 2012
    I did a lot of testings and I can assure you that a combination of GSA and CS is causing a memory leak issue.

    For testing purpose, I setup a brand new server, I decided to run only 1 Project @300 threads on the dedicated server. Here are the initial stats:

    CPU : 13%
    RAM: 23%

    I left them running for 24 hours and then the consumption of the resources increased after 24 hours. Here are the stats:

    CPU : 15%
    RAM: 50%

    I decided to leave them running for another 24 hours and the RAM consumption further increased. Here are the stats:

    CPU: 15%
    RAM: 80%

    After this, I decided to close both the programs one by one. Here are the stats after closing GSA.

    CPU: 15%
    RAM: 80%

    So with only CS running (it wasn't even breaking any captchas), above were the stats. So based on these tests, I can make an assumption that GSA is leaving a bad port open for CS which is resulting into a memory leak issue and causing servers to crash.

    I have PMed Sven about this issue and I can setup some more test servers for him to ensure that issue gets fixed very soon.
  • dabdab Wa
    From looking at that I would say CS is the one with the memory leak.
  • poweruphostingpoweruphosting Los Angeles
    @dab, I thought the same and I did contact Mike, the developer of CS and he said this could be due to the bad port which is being left open by GSA for CS.

    Either way, we gotta find out the root cause and get it fixed :)
  • OzzOzz
    edited November 2012
    @poweruphosting:  thx for testing.

    those screenshots i posted at October, 27.

    After fresh start:
    image

    After some hours:
    image

    Page table is increasing massively within some hours and will be cleared when closing CS. So there is something up either with CS or SER.
  • poweruphostingpoweruphosting Los Angeles
    @Ozz Yes, there is some issue with either of the software for sure. Whatever it is, we gotta make sure that they get resolved soon.
  • HunarHunar
    You guys are awesome.   Thank you both very much for doing the testing.  and hope this issue gets resolved quickly. :)
  • SvenSven www.GSA-Online.de
    Im doing a profile on my end now. No mem leaks so far or any open ports/handles. 
  • HunarHunar
    I'm currently managing 7 VPS's/Dedi's  And I also notice this issue across the board on all server.  I have to restart GSA + CS  every 12-24 hours.
  • piscopisco
    Also having this problem (that caused a number of tickets to powerup), don't really know what to do know, i've dropped threads to half, and still got this.

    I'm thinking maybe a batch file to kill and restart CS, or something like that.
  • poweruphostingpoweruphosting Los Angeles
    @pisco Yes, After receiving tickets from you and few other GSA + CS users, I started to investigate the issue to find out why are all these random crashes happening, wanted to make sure nothing is wrong with the server and finally concluded that the issue was actually with the softwares and not with the servers! :)
  • AlexRAlexR Cape Town
    Me too. But on my side when I view the processes, it always shows CS as not responding while GSA still runs. 
  • BrumnickBrumnick
    I have been having this same issue for a while.

    I leave my task manager performance tab open constantly and close and reopen both programmes just before I go to bed and then again when I wake up.  It starts off as both progs using less then 500mb mem and just before I shut them down it can be up to > 1.90gb. (between 6 - 12 hrs).  Shutting them down and restarting always works but in the last few days I have had CS stop responding too.

    It gets all blacked out so you can't see it and then you have to kill the process.

    I am running about 40 projects but using the scheduler for 5 at a time, 20 min duration.  I have 10 private proxies and am running 30 threads.
  • CaptchaSniperCaptchaSniper
    @all - I am also looking into it and trying to isolate where in the stack this is coming from.
  • sootedninjassootedninjas
    now that is what I call support :)
  • SvenSven www.GSA-Online.de
    @CaptchaSniper the only thing I changed when sending captchas to CS is that it now sends recaptcha as well. Maybe the problem is the function to solve unknown captchas?
  • OzzOzz
    No, I don't think that this is the issue as I've noticed this in CSX already IIRC.
  • darman82darman82
    i'm not a good at this technical issue but i really hope that Sven(GSA) and mike(CSX) can works together to solve this crashing issue.
  • SvenSven www.GSA-Online.de

    I had GSA running the whole night together with a "captcha-solving-app" (other than CS).

    No memory leaks on both sides. All running stable without any issues. So as CS was not involved here I can only assume that CS is the problem. Just a guess though.

    I can't run CS on my system as it keeps crashing almost immediately (tessact error). Maybe I should give it a try with that popup closing app that Ozz mentioned once.

  • OzzOzz
    edited November 2012
    I can confirm that the "other tool" works without memory issues on my machine in a steady flow and no significant memory in- or decreases at all. I had this running for up to 3 days in a row without any issues before I finally restarted it by hand.
  • cherubcherub SERnuke.com
    Sven and Ozz - tag-team tantalizers ;)
  • poweruphostingpoweruphosting Los Angeles
    edited November 2012
    Found a temporary solution to the issue.
    http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/agent_add_task.mspx?mfr=true

    You can Schedule a restart task for your VPS/Server inside Windows every 6-12 hours. This is a nice lil decent solution until the everything gets sorted out.
  • CaptchaSniperCaptchaSniper
    @Sven - you mean you're saying you cant even run CS at all without a tesseract pop-up?  So EVERY single captcha you send it results in a pop-up?  Please clarify.  I would hope have that you have a working version so that you are able to properly unit-test GSA with CS.
  • SvenSven www.GSA-Online.de

    Well I got this tesseract pop-up almost on every captcha that is analysed by CS. And that was from the start. Only solution for me was the exchange of the exe with a newer version. But even that was crashing sometimes. But you mentioned once that this is not a good idea as you tested CS only with that older one and so I didn't run CS anymore.

    A test with just exchanging CS with another captcha-breaking-executable was not showing any errors either crash or memory leaks as Ozz pointed out as well. And the way GSA-SER is sending the captchas is the same here.

  • CaptchaSniperCaptchaSniper
    lol @Sven - what are you trying to accomplish with the 'trying to be covert' but mentioning my upcoming tool comments?  yeah .. we all get it :) nudge nudge wink wink!

    Good news - we have beta testers currently testing a fixed version addressing the leaks.  Our intial tests indicate no more leaking :)
  • SvenSven www.GSA-Online.de

    @CaptchaSniper naw don't get me wrong, I only tried to see what the problem is. As I can't run CS for some odd reason, I was using another one and as this one communicate in a same way with SER than CS and everything was OK, I thought I would through it in here to get this sorted out.

    Anyway good you got it fixed on your end :)

  • mozzermozzer
    I don't know if someone has noticed that but when GSA SER threads are lowering (several times during a blast), CS slows it down.

    GSA SER seems to wait for captcha answers, and even hangs up during a few seconds, resulting in threads blocking.

    I can tell that it's related to CS because I've done tests with 3rd captcha services only and this issue did not occur.
  • SvenSven www.GSA-Online.de

    @mozzer, thats right. The program will wait till it gets an answer from CS before sending a new captcha to analyse. Thats done to not cause any conflicts or hammering CS.

    This however is not slowing things down in the end as CS can only work on one captcha at a time and would queue things anyway (Im not sure if it does though). You might see threads hanging but thats due to the wait of CS to answer the captchas.

  • poweruphostingpoweruphosting Los Angeles
    edited November 2012
    @Sven Yes, you are correct, CS does queues them and this is why the thread is usually a bit fluctuating. But this shouldn't be an issue really
  • AlexRAlexR Cape Town
    @Sven & @captchasniper - am I correct in saying that CS probably slows down GSA a bit. In that it has to throttle the captchas it sends and then wait for CS to solve it. 

    I have no idea if it is possible, but what if you had 2 or more CS programs on your VPS and the GSA just the captchas to whichever one is free? This could really speed it up and resolve the bottleneck a bit. 

    Anyways...I try and use higher PR, so not an issue really for me, but just an idea. 
  • OzzOzz
    edited November 2012
    I don't think that your suggestion works because the CPU power is already used by a single CS session to solve captchas. The CPU power doesn't double up if you are using 2 sessions of CS ;)
  • piscopisco
    Glad to see you guys are working towards this, as far as myself i made a batch file that kills CS every 3h (tried 6h but it wasnt enough for me), since GSA SER launches it as needed there is no need to launch it manually; it has survived to the first night, so it's good for now.

    But yesterday i was thinking about something curious, i don't know this for a fact, but i think CS is older than GSA SER, and is used for a lot more software than just SER, do other programs also have these compatibility issues using CS ?

  • aymenbnraymenbnr
    yea , i'm having this problem too !
    seems the problem is from CS  because when i closed CS the memory usage dropped to 45% , when it was around 91%
    i think the CS is causing the server crashes , and hopefully this gets fixed soon !
  • mmtjmmtj
    Right now we do the same as pisco - works fine with one restart a day.
  • SvenSven www.GSA-Online.de
    next version will restart CS every 30 minutes if wanted (optional)
  • CaptchaSniperCaptchaSniper
    Its been fixed and will be relesaed in the next 48 hours :)
  • AlexRAlexR Cape Town
    @captchasniper - will it have all the definitions that have been shared here included or must we still update them manually?
  • CaptchaSniperCaptchaSniper
    I will release a patch, so it wont affect this
  • theorbitaltheorbital
    I know this goes without saying but this is really great that this forum has vetted these problems and things are getting resolved. Thanks to all.
  • aymenbnraymenbnr
    cool , today's update seems to be solving the problem , thanks Sven
  • sootedninjassootedninjas
    edited November 2012
    CaptchaSniper

    Do you send eMail to your user base when an update of CS is available ?

    Because I don't see an option on CS for an update check.

    regards...
  • CaptchaSniperCaptchaSniper
    edited November 2012
    @ ninjas - yes I email my customers and post on forums

    @ Sven - already getting support tickets about problem with GSA shutting down CS.  Please remove this, as Ill have a fix for it out shortly.  Also, please don't work on things like so much 'in your own box'.  Something like this I would appreciate a direct discussion on :) Im sure you would want the same.

    And here is the the patch for all concered ... leak fixed and significant tess pop up reduction ... you can use mine now sven instead of yours :)


    Happy Thanksgiving! :)
  • SvenSven www.GSA-Online.de
    hmm so people enable the option to restart CS every 30 minutes and wonder why it restarts? Seriously, what customers do we have :-B 
  • BrumnickBrumnick
    I certainly didn't complain about the new restart feature but when I checked this morning, GSA had restarted 6 versions of CS and it created a "port 80, another device is currently using this port" error.

    Gonna try the new CS version now.  Thank you both for the continuing work you do for support.  Always appreciated!
  • dcc24496dcc24496
    @ captchasniper - how do I install the patch ? Because link you posted above ask for license as I would install the software for the first time, doesn't look like a patch to me and I'm afraid to continue , it might invalidate the current version and I would have no tool anymore , at least for a while.
  • SvenSven www.GSA-Online.de
    thats not a patch but a patched executable. So you must exchange your original exe with that one and it should work.
  • sootedninjassootedninjas
    this is the way I did it and it is working just fine.

    backup the CS program folder just in case.

    extract the zip file

    copy over and replace the exe file in the CS program folder.

    that's it.
  • dcc24496dcc24496
    ah, I see. 
    Yup, made the replacement as suggested and it works, thanks for the tip.
  • mikie46mikie46
    edited November 2012
    @captchasniper 
    Thanks for the update. Lets see how this goes.
  • CaptchaSniperCaptchaSniper
    @mikie - yeah i havent had a chance to look at your problem yet, but it wont be long.. today some time. but I initially tested it here and its ok.  Im assumiing something is off with your db and file structure.  Perhaps from using the GSA tool.. im not sure, but ill let ya know as soon as I login and take a look.
  • mikie46mikie46
    edited November 2012
    The annoying Tesseract.exe issue is still unresolved. @CaptchaSniper stated that with the new .exe replacement this issue would happen less often but for me, after running the new .exe for the past 2 days, its happening even more than ever!!

    Now, every time i log in to my server i see the Tesseract box open. This used to happen sometimes and not always on Win2008 R2. Its always happening now. And it randomly appears daily, more than 3 times (Arrrggggggggggg) a day.

    So this annoying thing is still around. Please dont ask me to run another program to counter act it. I ll simply switch back to the old version regardless of the memory leak. My updated consisted of a rename so i can still do that without a problem.
  • LWDLWD
    @mikie46
    +1 here for annoying Tesseract.exe error (Windows 7 server), problem is that you can't rely on software to run overnight, during the day I do check it but when I leave it to work unmonitored it pops up that error and after a while whole thing crashes...
  • darman82darman82
    Does anyone face the problem like mine? the CSX is not responding after running a while... yes, my VPS doesn't shut down but the CSX is not responding cause no captcha being solve by CSX.

    Anyone face the same problem?
  • LWDLWD
    @darman82 something similar happened to me today. No teseract error, but CS was not responding and GSA SER started new CS instance so there were 2 active CS and everything was so slow... I think that CS went to not responding and when GSA needed captcha to resolve it simply started new instance...
  • ronron SERLists.com
    edited November 2012

    @mikie46 I agree. I now have 5X the number of Tesseract errors and things slowed down, so I went back and replaced the old .exe file back in the CS folder. I haven't been experiencing the leakage issue like some of the others, so for me I will wait until the fix gets a little better tuned up by CS.

    @darman82 I saw that I was getting a bunch of Terreract errors and things weren't running as smoothly, so I shut everything down and replaced the old .exe file. I sensed I was heading in the direction you described, and decided to roll back.

  • mikie46mikie46
    I counted 32 times Tesseract.exe needed closing today. Ridiculous.
  • HunarHunar
    I'm noticing that CS is freezing a lot,  I mean a ton,   I did the initial beta testing with this new CS update  and for me personally Yes I have seen a huge drop in tesseract errors,  It's wierd that it would drop for me but not you guys.   

    but with how much it is freezing lately is making it very hard for me to run,  I have 7 servers  so I have to check each one like 3-4 times a day.  Does anyone have a link to the old .exe  i'd like to roll back as well.  I'd rather put up with the tesseract errors than it freezing up all the time.
  • LWDLWD
    I have been testing CS older version with new (memory leak fixed patch).

    After running Fixed patch, it crashes a lot no matter how many threads I put in, I tried with like 17 threads and so... It's all the same... Now after testing:
    • Running new CS version overnight, keeps crashing, GSA keeps restarting it, Port 80 issues, lot of CS instances not responding and opened with Port 80 error (this actually makes sense because when ONE CS is started it takes up port 80, then when GSA starts new one, port 80 is taken etc... I have to say that I think NEW version works faster then previous one on same settings. Not to mention tesseract.exe error...
    • Running OLD version (just before patch) does indeed run a bit slower (or I think so), but it is stable so far and does not crash server...

    I do have to recommend something to all users, I think there is absolutely no point of setting CS to do retries if Captcha is wrong, as I think that effects stability of the software and I'm not even sure if GSA knows that captcha that CS sent is wrong or right... 

    Hm, does this even makes sense? As I've read somewhere that CS has no idea if captcha is right or wrong... So how does GSA knows?

    In that case does GSA actually keeps sending SAME captcha to CS to read it again for 3 times (or how many retries we set)? Does this makes sense?

    Just my observation on the issue...

Sign In or Register to comment.