Memory Errors Galore
Since one of the more recent versions of GSA I am getting a crazy amount of memory errors on my projects. It's the "Out of Memory!" popup error.
It seems to be worse when I run projects that have large linklists imported to run. (200k-600k urls)
Any advice on how to avoid this?
It happens with just a couple of projects, server is a an absolute beast with 2 x 8 Core CPUs and 32GB of RAM so it can't be to blame.
I've taken to using the scheduler to run projects now and am just running 2 projects at once (used to be able to run dozens) but its still happening with just 2 projects!
Any advice would be much appreciated.
It seems to be worse when I run projects that have large linklists imported to run. (200k-600k urls)
Any advice on how to avoid this?
It happens with just a couple of projects, server is a an absolute beast with 2 x 8 Core CPUs and 32GB of RAM so it can't be to blame.
I've taken to using the scheduler to run projects now and am just running 2 projects at once (used to be able to run dozens) but its still happening with just 2 projects!
Any advice would be much appreciated.
Tagged:
Comments
Over the last few days the whole thing has gone very very laggy. Now running 20 projects at 200 threads and it's been lagged totally out for the last 3 hours.
Not sure what the problem is, but i suspect something has changed on a recent update.
Maybe @sven can shed some light.
Since I started using SB for harvesting and GSA posting-only, I also thought about getting a 2nd vps just dedicated to scraping + importing + identifying...consumes resources like nothing else.
That is a lot different to 400k submitted links in 45 mins of course.
The best performance i have ever got is 10,000 verified links per hour... Consistently hour after hour. That was with a Hetzner dedicated server which was truly amazing. Too bad they banned me after 1 day for spam abuse reports.
I use a separate VPS for processing scraped lists and it works well, but i have 3 other VPS to feed. I wouldn't necessary recommend 1 VPS to feed only 1 other VPS.
Unless you are already making decent money of course... I'm more than happy to spend money on something that makes me more money lol