Out Of Memory problem

Scaramanga

I have been running SER for a while with no problems. 

Recently I had a problem with my dedicated server, which required a reinstallation, and had to do a fresh install of SER. I restored all my projects to it as well, which total about 60.

I am running the exact same number of threads, 600, as I was before, with more proxies than I had before, 100.

My dedicated server has 24 GB RAM and SER was previously running absolutely fine on it, as expected.

I've even reduced the number of projects I am running, but I am frequently maxing out the CPU % that is shown on SER, and getting out of memory messages.

I've seen other people mention this problem on here before, and I'm wondering if anyone knows why this happens and what to do about it?

There appears to be no reason why SER should not be running smoothly as before, but clearly there is a problem.

What can I do about it?

Desire_

I'm also getting a lot of "out of memory" error messages lately on a dedicated server with 16gb of ram and E3-1240v2 CPU, I've tried lowering my threads down to 200 and it helped a bit but after some time I still got the error. When I import any list (directly from site lists or .sl files) that has around 100k links or more to any project, memory usage will spike quickly and I'll get an out of memory error within minutes of starting SER. So I stopped importing URLs into my projects and it helped a bit but in the end I still got an out of memmory error. When I first got my dedicated server around a week ago, I was able to run all my projects smoothly with 1000 threads, 100+ LPM and both memory usage and CPU usage were a lot lower. And I only run around 20 projects... I've submitted a bug report in SER but does somebody check them?

@sven

Scaramanga

Is there a way to move this thread? I'd like to move it to the GSA SER help section instead, or else I'll have to repost it in there.

s4nt0s

 @scaramanga - I moved the thread for you.

Have you tried any of the "automatically decrease threads on CPU usage above x % and the decrease threads on memory usage above x %? 

Scaramanga

Thanks Santos.

While those features may help, and I may have to use them, SER was working absolutely fine without needing them before. When I decided to get 100 proxies I was hoping that I could increase the number of threads I was using. Using those features is going to mean that the number of threads I can use is constantly being throttled.

There is obviously a problem somewhere, because with the specifications of my dedicated server I really shouldn't be having this problem, and indeed wasn't having it before.

As Desire's reply indicates, I'm not the only person experiencing this problem. The difficulty is going to be finding out what causes it.

nawshale

Yes same issue.I used to run 400 Threads and 800LPM plus 600 - 650 VPM but latest updates i had to decrease it to 200 threads and my vpm has decreased to 200 - 250 .

Trying memory above 2500mb and my cpu have never gone than 50%

Desire_

@Scaramanga

Do you remember what was the last version that worked well for you?

RYKsebastino

Over the last few days I have began receiving out of memory errors about every 4 hours on a dedicated server with 32GB of memory, of which 28.5GB is available. This makes absolutely no sense. I'm running 150 proxies with 62 projects and 50 threads. This is an 8-core system. My CPU usage never goes above 25%, even with GSA, SEnuke and other supporting programs running at the same time.

botman

He thought that my LPM decreased because my list gets old, but if yo guys reporting you have same problem, I can add that my LPM is half down. I had 500 LPM and now same projects shows me 200+ LPM and CPU eats lot more....

claus10

I run 2 servers with 8 and 32GB respectively. They have been getting out of memory for about 6 months every day now. It doesn't matter if I run 50 or 200 threads. The memory leak or whatever it is keeps happening.

Hunar

Yeah, I agree I've been getting the out of memory error for months now.  I think i'm just so used to it now though. 

Desire_

Now that I have pretty much stopped importing lists directly into my projects, lowered the threads, lowered the maximum size of a website to download and stopped using lists with more than 100k+ links, the "Out of Memory" error doesn't show up after starting my projects, but when I just let my dedi with SER run for 4-12 hours, I'll get an out of memory error sooner or later. And again: the first time I started to having issues with this was around 1-1,5 weeks ago. From what I remember version 8.55 didn't have this problem.

Scaramanga

I don't remember the number of the last version that worked fine for me, but it was about 1 - 1.5 weeks ago.

Prior to that it was working absolutely fine, so something must have caused it. However, other than the fact that I restored saved projects of mine to a fresh install, everything is the same as it was before. 

Desire_

Has anyone looked into this yet?

ron

If you fiddle with SER like stop projects, turn to Active(V), hit Stop, and then turn back on - you really should be closing out SER, and restarting it. Or you are simply stopping it and re-starting it for whatever reasons. The RAM cache stays active, and then when you switch tasks, or you hit start/stop/start, etc., it layers the RAM on top of what you were doing previously. It just keeps getting bigger. So I always close out after I have made a bunch of changes like I described.

Now if you start it, and simply run it without doing the stuff I am talking about - and then run into memory issues just from normal running - then that is something I never really experienced.

Desire_

@ron

I do that, I also tried using a program called memory cleaner. The most important thing is that as I said before, 2 weeks ago I didn't have ANY problems with memory at all. It's strange. Right now I loaded your newest Red List straight into my projects and global list too, it's running nicely and I'm getting 300-600 LPM with 450 threads. Memory usage is around 1-2gb, so it still isn't even that high but I will still get an "Out of Memory" error after 20-60 minutes of running it but SER still works smoothly...

Desire_

I'm going to bump this thread, because I can see @Sven answering many threads each day but somehow this one always gets ignored. I never had this problem until 2 weeks ago and I've changed nothing so I dont think it's a problem on my side.

ron

@Desire_ which version of SER are you using? I am using v8.59, and I am so pleased with its performance I really haven't upgraded. But with that version I am definitely having no issues - very smooth sailing.

Desire_

@ron

I'm using the newest one and even though I'm getting that error, it's still running smoothly too... or at least that's what it looks like because I'll usually get this error when SER is running for a few hours, so when I login through rdp I'll see the error but memory and cpu usage looks fine, LPM too.

I started to think that it could be because SER eventually runs into some very heavy website that cause a very high memory usage or something. So I ran a test with your old list (blue 5 or 6 I think) that I was using a few weeks ago with 1000 threads, getting over 200 LPM (and no errors!). Now I used 400 threads, my LPM was even higher but after a few hours I still got the out of memory error but performance was still ok.

Desire_

Another thing I noticed, when I let SER run for 12+ hours I'll obviously get the out of memory error even though SER is running fine when I check it, but now I also got "an error occured in the application" crash as soon as I close the out of memory error box and click "Stop".

It looks like I can spam this thread all I want but @Sven still won't give us any answers...

@Scaramanga do you still have this problem?

nawshale

Bump! to @sven

Scaramanga

Yes I still have this problem.

cefege

Yep, have this problem as well. With 128GB Server.

davbel

It doesn't matter how much memory your server / PC has, SER is an a 32 bit app and as such will only work reliably when it is using less than 2Gb Ram.

Yes it will run taking up more Ram and mine frequently does, but anything above 2Gb and the Out of memory messages will start appearing.

From my experience (and I'm sure @sven will correct me if I'm wrong) you need to find a balance between # of projects running, # of threads, size of link lists per project, size of project files.

coneh34d

The size of your project files is perhaps the biggest memory consumer. To handle this issue, begin using spinfolder macros. 

Check out the macro guide here:

http://docu.gsa-online.de/search_engine_ranker/macro_guide

Desire_

Bump

JudderMan

@coneh34d and when your SER is as optimised as possible and you still get out of memory errors....lower threads

coneh34d

JudderMan How large are your .prj files? Ar eyou running the scheduler? How many projects are you running per server?

gooner

I had this problem a lot, i switched dedi suppliers a couple months back and i have not seen the error again even once across 7 servers. I don't know the technical reasons for that, but it seems that the server used plays a bigger part in this then any SER settings. At least that's my experience.

davbel

@gooner I think it's more to do with how lean your SER is, how many projects, how many threads and what gets loaded into memory per project. Remember it's only a 32bit app, so if your projects take up 4gb of memory, you will have warnings.

I think you chaps have optimised / tweaked so much and reading between the lines, you seem to run your set ups so lean, so you get more from them than most of the rest of us.

gooner

@davbel - I agree optimising it defo improves things, but it couldn't solve the error completely before no matter what i did. Now i can turn SER to 1500 threads and no error - Exactly the same settings as before... Strange but true. Either it's the server or Sven's magical coding