Version 15.74 GSA SER - Memory and CPU are both being used excessively

sickseo

Hi,

Seems to be an issue with recent version of GSA SER. I normally can run 300 threads continuously and the cpu will be at about 10%-20%.

Currently it is hitting high cpu at less than 100 threads. I've set the software to automatically reduce the threads at 75%. Normally this option was never utilised, as the cpu usage is normally very low, but now, it's reducing the threads to below 100 as the cpu is hitting 75%. 

Also the memory use is excessive. I have only 130 projects on one install (I used to be able to run 300 projects) and the memory has already hit 3gb ram and I'm getting out of memory errors. I've only got 80k verified links on the install, so there is no reason for it to hit 3gb ram that I can see. I've seen over 3 million verified links in one install before. So something seems very wrong here also.

Hope you can figure it out. I'll try reverting back to previous version for now and see if that runs better,

Thanks

Sven

In such situations please try to run it with as less threads as possible and if you see the memory/cpu increase, try hitting HELP->Create Bugreport and leave a message in the notes about this thread so I can investigate and reply here.

sickseo

OK thanks. Just sent you a bug report referencing this thread and high cpu usage.

Sven

got it but nothing strange happening there...most of the time threads are busy downloading in inner libs...no code of mine actually

sickseo

Oh thats a shame. It's happenning on all the vps I have. They are decent specs so they should be able to handle the threads. They certainly could run at 300 threads comfortably before with 20-30% cpu max. At 400 threads they would start to max out, but the speed of submission/verification goes crazy fast at 400 threads on the vps.

Currently, those installs are struggling to go beyond 100 threads without the cpu maxing out. I think I've optimised the servers too much and they are running so fast, that it causes the higher cpu. The internet line is 1gbps and the proxies are 10gbps.

On the dedi I have it seems fine, but that's got a crazy amount of processors so it can handle the higher threads without maxing out.

Thanks anyway for checking.

Sven

Can you verify with the previous version (linked to start menu)?

sickseo

I've tried on previous versions 15.71, 15.72 and current version also. I need to test it on 15.73. It's the same for me on all of them with regards to the cpu maxing out at lower threads.

The other thing with the memory issue, it may well be because of my link sources as I'm using a lot of exploits and blog comments and retry failed sites=1. Those platforms seems to consume more memory than article/forum/social network /wiki links.

I've noticed it when exporting the 2 different groups of verified urls. File sizes of exploits/blog comments are much much bigger than the other types of sites. 100k urls of blog comments/exploits equates to file size of 100mb. Where as I've had over 200k urls of the other types of sites and the file size has been less than 30mb.

I don't know if the memory/cpu can be further optimised to handle larger volumes of links?

Sven

Indeed spammed sites like blog comments take a lot time to download and analyze.

AliTab

I had memory issues in the last 3 versions as well Sven. I'm always running 1000 threads without any issues but recently I can see memory usage passing over 3 GB. I have no clues about this though. That's why I didn't report
btw, Thanks for being active in the community @sickseo

sickseo

Well i'm glad it's not just me then that's having issues with the memory. lol

@AliTab I used to be active many many years ago. The good old days when we had a lot of experienced users like @ron here willing to share successful strategies and those guys/gals that made site lists that would make your heart skip a beat lol The days of when we had over 10k joomla k2 contextual link sources for T1 links lol Those were the days.

But when I got slapped by Google from outsourcing my link building to pbn sellers on bhw, things kinda changed lol I abandoned google and just focued on bing/yahoo for 4 years. then couple years ago I got dragged back into Google SEO by an old contact from 7 years ago who is now my biggest client.

So I'm back now and ranking nicely in google. It's nice to see you here with your link list service. I hope you can make that list of yours bigger and bigger every month lol

We need those new link sources!

AliTab

Thank you @sickseo
Nice story and I'm really glad you're back. Sharing thoughts and helping others is what I respect the most and I see you are doing well here. The community needs people like you

sickseo

I've been doing some more testing as the current version still has these issues with memory and cpu and slow speed. Out of all the versions I've tested, 15.68, 15.71, 15.72, 15.73, 15.74, only 15.71 works great with high vpm and low memory/cpu usage. I've switched all my installs back to 15.71.

@Sven it seems something has changed since 15.72 onwards. No idea what has changed, but I'm happy I've got a stable fast version running now.

sickseo

If anyone needs it, you can download 15.71 here: https://1drv.ms/u/s!AvaPxZBKuUdaiYlSszQzsEAlznDxGg?e=Y8Aeot

Just replace this file with the search_engine_ranker.exe file in your program folder for the software.

Sven

If you have a look at the change log, you see that most changes have been done on parts not related to the inner core. On 15.72 e.g. I simply improved some tools and some engines. Nothing special. But I will have a look on this later this week.

AliTab

Thanks for investigation @sickseo
I'm sure @Sven will fix the issue ASAP

Sven

latest update should fix some major issues on memory/cpu usage as well as on the growing target url list.

AliTab

I can see a huge difference in memory usage and stable threads count.
Thank you @Sven

sickseo

Yes, same here. Memory and cpu use is much better than previous versions. Thanks @Sven

bolaplay

Disabled Log can help reduce CPU, is it ok?

draculax

Hello,

I notice that reCAPTCHA levels via xevil we have passed to 50% of success following the new version.

We were at 96/97% before and 60% for classic captcha.

Can you verify @sven thank you 

AliTab

draculax said:

Hello,

I notice that reCAPTCHA levels via xevil we have passed to 50% of success following the new version.

We were at 96/97% before and 60% for classic captcha.

Can you verify @sven thank you 

I have noticed that it's not just Recaptcha. captcha images have also been lowered. It's like there is some threads limit for captcha solving. I'm not sure if I'm right though.

draculax

100 thread 20 projects and memory bad out .

draculax

good evening,

I tested the old version and there is a problem with memory and cpu performance with few projects 10/20 max and 100 threads gsa and in the red all the time.

I am all alone in this case or maybe there is a solution?

MrGreen

Latest version fixed a lot of those issues for me.

CPU usage dropped at least 50% and I can run 400T without OOM (before 300).

Thank you @Sven

Tim89

Hey @sickseo nice to see you active again in Google! I hope it has been treating you well It's still a rollercoaster mate!

sickseo

Hey @Tim89 How you been mate? Yeah it's been a while since I played with google. So far so good. Ranking in google quite nicely these days. Just not happening as quickly as I'd like. But shouldn't complain.