no, @sammy262 that criteria is telling SER what the largest size of a site it should download. If you have it set at 50MB, it won't download a site larger than 50MB and hence the post will fail.
Ok so gooner has no problem with 50 so I should have no problem with it. And it would make sense if the memory raises the more threads SER uses but it does not. It just increases with the number of projects. But since gooner seems to have 150 active and running projects and no problems it is weird.
@Gooner: Do you have these 150 active and running at the same time or with scheduler by let's say 10-30 every 2 hours or so? And what VPS or dedi do you use?
@Sven: Then why can gooner with his weaker server and the same filter set to 50MB like I have it set to 50MB run 150 projects at a time whereas I can only run 10 and once I start number 11 then I get out of memory.
I mean I would try it but when I see someone has the same setting (I think even more threads) and there it is no problem at all then I doubt that it's that download filter.
So there must be something triggering the memory usage on my SER to skyrocket because obviously a lot have the Out of Memory issue. Maybe not as dramatic with over 3GB memory usage that I have but still.
I said yesterday that 10 projects with 500 threads were fine, but when I checked the server this morning I saw the annoying Out of Memory again. What I will do now is backup all my projects running projects and then I will upload it to the cloud and send you a link - via PM message ok.
But what is also funny is when I get this message I can click Yes or No it doesn't matter - the popup always comes back and the threads never really stop, sometimes 2-7 are still running and the message keeps coming back. So I have to end the task manually in the taskmanager and this leads to a loss of all links that were created since the last restart.
Is there something we can do about that. How often does GSA SER autosave the submitted and verified stats of each project?
I think it autosaves every 5 mins, but when that out of memory happens it always seems to reset stats to zero. Maybe that's just stats but actual links are saved i don't know.
@sammy262 - I notice if you stop pressing yes or no, run memory cleaner app (Goggle it), turn down threads and leave that message alone for like 20 mins then you can successfully press no and it will go away.
It's not a solution to out of memory, but it saves you having to restart at least.
@sven - I also noticed 8.03 is more memory hungry. I had to drop from 800 to 600 threads to prevent out of memory messages.
I noticed something that might help to fix at least one of the causes of the Out of Memory.
@gooner: Do you import a lot of target urls per project and if you STOP SER select all actively running projects and click "Show remaining URLs" - what happens?
When I do it with 5 of my 10 projects that each have 400k Remaining URLs everything is fine and it shows me round about 2 Million URLs in total and the memory usage for that in SER is 800 MB.
But when I check the 5 other projects individually I see each has more than 3 Million URLs. So when select all 5 projects of this campaign and click "Show remaining URLs" (which should be round about 15 Million - 5 projects with 3 Million URLs each) then I see the meory usage increase quickly and when it hits 3.8x GB I get the Out of Memory usage PopUp. So the list of remaining urls seems to be to big to be processed which triggers the popup.
SO I have 2 options now:
1.) I delete the remaining URls of each of these campaigns and feed just a a couple hundred thousand per project
or
2.) maybe this information (and the video that I send directly) helps @Sven to fix that problem.
Because not sure how this affects SER when it is running. I guess it only loads chunks of each target url list but maybe these chunks add up to quickly.
Maybe you can try that as well @gooner. Increase threads back to 800 but reduce the target URL list size for each of the active projects.
@sammy262 - There's your problem... target URL cache is too big. No doubt about it. If it's a verified URL list you are using, put in one of SER's folders like "identified" and let it post from there. Memory usage will drop instantly
@gooner: It is an identified list that I merged into one text file to be able to feed via "Import Target URLs by file" because I have the feeling this is faster then the use of global site lists, but I guess this really is the problem. I send you another PM.
SER can store a surprisingly large amount of URLs in the target URL cache without getting bogged down. On one of my processing servers, I've got 25 projects with 290k unique urls per project in cache at any given time. This was on a server with only 8GB of ram too.
@Justin: But 25 x 290k is only 7.25 Million URLs ;-) I had 3.6 Million per project. I have 5 of these projects plus another 5 projects with 400k URLs each. So the total is somewhere at 19 to 20 Million URLs spread across these 10 projects.
Not sure if that is the problem or trigger but I documented it and I am currently writing back and forth with Sven to see what the problem is.
IN the meantime, when my verification is done, I will just delete the Target URL cache of the 5 projects that have more than 3.5 Million URLs each and just add 200k URLs for each project and then I check if the Out of Memory still comes up.
And if you have 6GB, 8GB or 16GB or like me 64GB RAM does not matter for SER because it only uses max 4GB. Once the memory usage in SER hits the 3.8x or 3.9 GB mark you get the out of memory message. It's because it is an 32bit app.
Comments
@Gooner: Do you have these 150 active and running at the same time or with scheduler by let's say 10-30 every 2 hours or so? And what VPS or dedi do you use?
I mean I would try it but when I see someone has the same setting (I think even more threads) and there it is no problem at all then I doubt that it's that download filter.
he uses 550 threads and the memory usage is below 600MB.
So there must be something triggering the memory usage on my SER to skyrocket because obviously a lot have the Out of Memory issue. Maybe not as dramatic with over 3GB memory usage that I have but still.
@sammy262 - I notice if you stop pressing yes or no, run memory cleaner app (Goggle it), turn down threads and leave that message alone for like 20 mins then you can successfully press no and it will go away.
It's not a solution to out of memory, but it saves you having to restart at least.
@sven - I also noticed 8.03 is more memory hungry. I had to drop from 800 to 600 threads to prevent out of memory messages.
When I do it with 5 of my 10 projects that each have 400k Remaining URLs everything is fine and it shows me round about 2 Million URLs in total and the memory usage for that in SER is 800 MB.
SO I have 2 options now:
Because not sure how this affects SER when it is running. I guess it only loads chunks of each target url list but maybe these chunks add up to quickly.
Maybe you can try that as well @gooner. Increase threads back to 800 but reduce the target URL list size for each of the active projects.
No doubt about it. If it's a verified URL list you are using, put in one of SER's folders like "identified" and let it post from there. Memory usage will drop instantly
Not sure if that is the problem or trigger but I documented it and I am currently writing back and forth with Sven to see what the problem is.
IN the meantime, when my verification is done, I will just delete the Target URL cache of the 5 projects that have more than 3.5 Million URLs each and just add 200k URLs for each project and then I check if the Out of Memory still comes up.
And if you have 6GB, 8GB or 16GB or like me 64GB RAM does not matter for SER because it only uses max 4GB. Once the memory usage in SER hits the 3.8x or 3.9 GB mark you get the out of memory message. It's because it is an 32bit app.