Still Getting Out of Memory Errors

JudderMan

I know I sound like a broken record, but this is getting insane. I have one project running at the moment, 150 threads, running through my verified lists only......and getting out of memory errors. HTML timeout is 180sec, 60 proxies (55 working). It's running at 200LPM.

I know from reading and previous threads that people have suggested to delete projects, delete cache, delete target URLs but I'm running global lists only so what is up? What am I doing wrong? 150 threads isn't a lot, if I'm cleaning lists or running one project at a time, I used to be able to run 800-1000 threads.

Also, I've mentioned this before but SER looks says my CPU is running at 99% but when checking Task Manager it's actually <20%, sometimes 4% and still getting out of memory errors. When the RAM on SER goes to 1GB that is usually when it stops working....even though my server has 32GB to play with.

gooner

@judderman - I've been looking at this closely over the last few weeks and i think your problem may be the size of your verified list, if you are posting from it.

Example: 400k verified list i can run at 600 threads before i get "out of memory".

35k verified list i can run at 1500 threads.

Loading these lists must consume a lot of memory, What size is your verified?

Also, i remember reading a very old thread that said the out of memory issue is triggered at 1.2GB usage - Maybe @sven can confirm that?

JudderMan

Hmm, not sure what my verified list is TBH. I lost all of my old data from swtiching from my old VPS to dedi, so really my verified list on my dedi is only 3 weeks old....so it can't be that crazy big.

So, you're saying as you load lists in that it would be better to trim them to say 50k and burn through them faster?

I don't normally load lists in, well there is one that I'm running through a dummy project and it's got 1.8million left......maybe that's the problem then?

Usually, I just let SER search without global lists checked, and am constantly getting 2-20LPM most days.As you can imagine, I'm going out of my mind.

gooner

@judderman - Yea i've been there mate, drives you crazy i know.

Yea i'm working with smaller lists only now. Myself and a few other users have been testing that method and it works really well.

The problem is the 1.8 million list i reckon. On the server i use for scraping/testing i import max 100k links per project to keep it lightweight and fast.

Anything that uses memory and you don't need, get rid... Even de-duping domains and url's daily keeps lists to a minimum.

JudderMan

Currently at 1.5mn now...I did split the lists up originally, but at first it was churning through them quickly so I just dumped them all in. I still think I had problems before that, though.

Although I have no doubt the way you're running your SER(s) works but I'm after a more hands-off approach ie load a 1 million list and let it get on with it. 

ps. did you get your dedi set up yet?

gooner

There's 2 ways to do that:

1) Load 1 million and split it over multiple projects.

2) Assuming you are importing into test projects: Run more projects so SER processes each project more slowly.

I only need to import lists once per day, so not so bad.

JudderMan

Cool, makes sense across a few different projects - can I just duplicate the dummy/test project a few times? Will that work?

Then I assume you change all of the 'real' projects to run off global verified only (or do you check submitted/failed as well?)

Thanks, as always.

chela1980

I got the same problem, am getting this error couple of times a day, and its pretty annoying as i have to stop everything and close gsa and run again, which means i have to babysit the program. Am running just 100 threads, 10 private proxys. in program got max 900MB memory used (i got 8GB ddr3)

JudderMan

From lots of reading and lots of help, mainly from gooner, I think the best way is to make SER run as thin/light as possible. I use %spin folder% for pulling data from files ie. titles, articles and for the keyword list, but now I'm thinking of trimming back the keyword list to a lot less than 500k (it is split into 50 files) as I don't think it's doing much good for my progress.

I'm filtering through the lists I create with 5 dummy projects and it's clearing through the lists quickly. I plan on only running my other projects off the global list but I need to build up the global list to be better than what it is right now.

I've been tweaking lots of things lately to stop this, and currently I've gone back to 20 projects by 20 minutes on the scheduler, but making T1 inactive (after getting to a set figure, say 1000 verified) and running more on the T2+ tiers. It's running at 85LPM at the moment which is OK for now, as I'm still letting SER search until the 2mil list has been cleared through by the dummy projects, then I'll unleash the rest of the projects onto the global list.

I've found that unchecking continuously post even if failed before has quickened up the LPM - if SER can't do it at first then just move on. I think my emails are causing problems now and my proxies - I've cancelled my 20 private proxies as some had speeds of 9+ seconds but am keeping the semi-dedicated proxies which are faster (but obviously used more). I am tempted to add another 40 semis from a different provider, but am waiting to see the difference in removing the private ones.

Proxyhub said to me that you should only run double the threads of the amount of proxies (even though their site states that it can be x100), so in your case, you should only run 20 threads and I should only run 80...which is BS as I can run 800 without problems 'sometimes'. I'm currently running at 500 and that seems to be a nice figure for my server keeping under the out of memory errors and stalls.

The out of memory errors could be down to PR checking and letting SER search with too many projects/threads running - so if you have PR3 for your T1s for instance, this will cause problems. I've stopped filtering so tightly now, I removed the bad words list, PR filters and have OBL at 100 for some projects and removed completely for others. A link is a link, it just needs indexed and f-k loads of more links thrown at it.

gooner

@judderman - Sorry i didn't see your previous reply. Yes i agree keep it as light as possible.

I have a dedi that processes scraped lists and then i import them into small batches into my main dedi as "submitted". I also check "verified" so it can use verified links across all projects, about once per month i save that verified list and remove it from SER and start a new one. Keeps the lists small and the links fresh.

For keywords, i put just "1" - No need to waste memory saving keywords that are not used because SER doesn't scrape. I check the option to save PR to verified lists so that SER doesn't have to check PR when it uses those links again.

De-dup all lists regularly, every little memory saver helps. Good lists are really important for speed and low memory usage. If i couldn't scrape and produce my own sitelists i would buy them every month.

A couple of months ago i was the steady 70 - 100 LPM guy, day in day out. With all those modifications my stats look like this now:



That's with 130 projects running 24/7 - No scheduler.

JudderMan

Awesome reply. I'm sold. I'm gonna scale up the scraping and will buy another VPS/dedi/SER license to do this. I'm running a mini version of what you have going but with one license, and it's working well so far. I can 'see' that this will be better if the filtering and posting are on separate copies of SER.

gooner

Good luck mate - Give me a shout if you need anything.

chela1980

as @gooner know am still learning ser and seo, so guys maybe shed some light on those "keywords from folder" or how to build those lists? Only thing i made with SB was to put some KWs from my niche, mixed them with footprints i found online (ultimate footprints list - package) and srapped some articles / social networks, but clearly i dont know how to build those global list to not bother SER for it and not getting those errors

JudderMan

In the keywords section, you can right click and add %spin folder% macro. This pulls data from a folder on your desktop or Dropbox (I do this to share from my local computer to my dedi). You can use Text Wedge/Scrapebox to split lists. It saves having tonnes of keyword (target keywords) in SER as you can see above, having a 'heavy' SER with lots of data can slow it down.

You can build lists using GScraper (which is better I think but I don't own it) or Scrapebox. Use your footprint list and harvest potential targets. Run those targets through SER in a dummy project(s) with all Engines ticked. SER will identify the target URLs and see if it can post to the sites in the list. Those successful and verified go to your global lists which you can run on 'real' projects at lightning speed as SER has already sorted the list, so you can run higher threads and thus higher LPM and therefore links. 

Check out the video series from Donald Beck who shows you how to scrape targets for SER in detail - I still keep on meaning to buy this, apparently it's very good.

chela1980

thx for post. So its doeasnt matter what keyword to use to scrape them? as its global list its goin for every project anyway? so it is a good idea to scrape keyword for lets say "weigl loss" use it for some YT video, and then those links will be used for "dog training" niche? or should i transfer them to other file and use for "weight loss" blog later? cuz am little confused about sorting things up with list building

JudderMan

If you're just using global lists then just don't pay much attention to the keyword part (just write keyword), as it just needs something/anything in the field.

I'm not sure about how 'good' a list needs to be ie. does it make sense to have the same global lists for various niches, as they will all have the same link profile (pretty much anyway), and if it works then it works, but in my mind not all niches should have the same link profile....but then the alternative is to have SER scrape relevant sites/urls to post to.