Skip to content

Out of Memory

HunarHunar
edited December 2012 in Need Help
Running into a issue that I've never really seen or never really seen anyone post before.   I've got some pretty powerful dedi's running GSA pretty hard,  On one of the server I have something like 50 Projects all running at one time.  Submitting about 40k - 50k Links a day after 4-5 hours of constant running It will pop up and say GSA is out of Memory while I check the dedi  It's barely using the CPU Or the Ram. 

Can GSA only run so many projects at one time before it won't work any longer?   Also, How come this is a issue that is just coming up now while i've had GSA for a long time and this issue has never once arose till now.

Comments

  • SvenSven www.GSA-Online.de
    for "now" you mean what version? Also keep in mind that RAM no matter how many you have is limited to 2.1GB of it.
  • HunarHunar
    Ahh that could be it than.  a few of the server i'm running have around 16g Ram and i'm barely touching it.  but I just started seeing this pop up  after the last 3-4 versions. 

    But at the same time GSA is running like a beast lately.
  • ewandyewandy
    @Hunar
    Use the scheduler function and run about 10 projects at the same time.
    I had the same issue and since I use the scheduler running only 5-10 projects at the same this issue disappeared.
    You're probably creating really spun content like me (I'm using KM).

    btw I have about the same server specs. 8GB Ram (so a lot of Unused CPU and Ram power)
    Submitting like 50-200K links daily :)
  • KayKayKayKay
    nice tip with the scheduler .. i kinda overlooked that ... i got ~ 50 projects with 4 tiers .. this setup runs not realy good because i loaded alot of urls and so it uses too much ram ..so i will try to scheduele them too ,)
  • adventurousloveradventurouslover
    What scheduler are you guys talking about? Does GSA has one?
    If so, how do I access it?
  • SvenSven www.GSA-Online.de
    click the down arrow on the Start/Stop button.
  • darman82darman82
    Strange... Does anyone have the same problem like me? Cause my GSA always freeze if i use this scheduler. I use 10 project with 30 minute setting.

    I have around 50 project with 2 tier.

  • LeeGLeeG Eating your first bourne

    Try clearing your duplicate url list.

    Dont ask why, but thats what I do when it happens and it seems to cure it

  • senty4lovesenty4love
    Try to reduce project per schedule..I calculate my setting by dividing Todal number of project by 'time SER takes to complete one project'..

    with my settings and proxy speed, SER usually finishes one project in 30 minutes. So if i have total 150 projects i just need 5 projects/30 minutes .. this will keep SER always busy and wont take too much resources at a time
  • darman82darman82
    done it, and i didn't use collect keyword also. Bellow is my setting :

    - The content i use is from ACW and i use 100K generic keyword from BHW.
    - Use 30 dedicated proxies from buyproxies with 100 thread 180HTML and 10 search queries.
    - Use dedicated server from mydedis with 16GB memory with 2 tiers - 50 project on the GSA
    - I only run CSX3, GSA SER and GSA SEO indexer with 25 thread.

    anything that i miss???
  • senty4lovesenty4love
    I only run SER on VPS with 3GB ram and 3 processors ..still manage to run it fast and no hangs..Only thing i am doing different than you, is I only use 3-4k keywords per project [always use different keywords for each project, this helps increase my sitelist] .. and I use content from local hard disk..i don't import content in to SER [not sure this affects efficacy of SER]
  • darman82darman82
    OK @hyde suggest me to change the verify link from automatic to custom time 7200 for tier1.

    I will try his suggestion. at the mean time i have set 5 project 30 minute for the scheduler.
  • senty4lovesenty4love
    i would say set it at 3000 ..somewhere Ozz said that SER deletes submitted backlinks after 5 days [I am not sure if SER deletes submitted backlinks even if its not verified]..so my understanding is, if links are not verified within 5 days may be they get deleted. Either way 3000 minutes [or 2.5 days] is more than enough to identify links which are dead
  • OzzOzz
    edited January 2013
    yes, general time range to check verifications is 7200 minutes (= 5 days) which is also configurable in the engine files. search for 'verify timeout=' within the script(s) to modify.
  • senty4lovesenty4love
    thanks Ozz for conforming it..it means if you are using option 'custom verify' time ..its alwasy good to check 'don't delete urls' option [to keep your submitted links]
  • OzzOzz
    edited January 2013
    yes, "don't remove urls" will keep your submitted urls entirely. but keep in mind that your submitted link list will grow huge by time if you have 'no-limit' projects with low filter settings. this will cause more memory usage i believe as all submitted links needs to be cached and the verification process lasts longer as well when trying to find your link again and again.

    "don't remove urls" is only recommended for article- and link-directories IMO because some of those platforms/sites may need some time before your post will be approved.
  • AlexRAlexR Cape Town
    What's the advantage of using the scheduler? Unless you have too many projects for your VPS's resources?
  • darman82darman82
    I still have no luck with the scheduler... Is this because to low of switch for next project time. I use 10. Now i'm pause around 10 project and only run 37 -  2 tier project with tier1 200 verify each day and tier2 custom time 3000 with submission each day 1000.

    Let's see how it goes...


    image
  • SvenSven www.GSA-Online.de
    edited January 2013

    @darman82 disable the option to monitor memory usage.

    Also note that latest versions can use up to 4gb of ram on 64bit systems (didn't noticed that in change log).

    Next version will not go nuts on a 64bit system with that message on 1gb but 3+gb ... hope it fixes it for you.

  • darman82darman82
    i don't know, i'm using mydedis PRS Mini with 16gb memory

    ok i will do that.. thanks...
  • AlexRAlexR Cape Town
    @sven - just to confirm - So does GSA SER now run on 64bit with 4gb ram? I thought there was a big discussion about it staying at 2gb...when did this change?
  • SvenSven www.GSA-Online.de
    I said I will not compile a 64bit version. Though with some tricks on the memory manager I made it to use up to 4GB on a 64Bit system.
  • AlexRAlexR Cape Town
    Sneaky! So is there an advantage to running it on 64bit over 32bit? 

    What would you advise?
  • SvenSven www.GSA-Online.de
    You don't need 64bit...all thats needed is 16bit ;P
  • LeeGLeeG Eating your first bourne
    Wahooooo bigger submissions totals, no loosing an hour doing vps reboots :D
  • royalmiceroyalmice WEBSITE: ---> https://asiavirtualsolutions.com | SKYPE:---> asiavirtualsolutions
    edited January 2013
    @Sven

    You mentioned : "Though with some tricks on the memory manager I made it to use up to 4GB on a 64Bit system."

    I am on version 4.96 and it does not want to use more than 1GB RAM

    Would you mind telling how to get it to use 4GB.

    On the moment the memory is driving me nuts, because even i stop everything and make all inactive, i still get an Out of Memory message with GSA ser when importing 50 Kontent Machine data files for a new project.

    I know you said no for the 64bit version - but darn it would be really cool if i can use all my available ram


    EDIT: I notice it goes up to 2GB when running projects But instead of importing 50 Kontant machine data files, i had to reduce it to only 8 otherwise i get out of memory ( even when everything else is stopped and inactive )
  • darman82darman82
    Same problem here... how to solve this problem, i have to stop half of my project now.
  • SvenSven www.GSA-Online.de
    You get that message with just importing these 50 files or when running a project with that much data imported?
  • royalmiceroyalmice WEBSITE: ---> https://asiavirtualsolutions.com | SKYPE:---> asiavirtualsolutions
    @Sven

    I get it with all projects set to inactive and everything stopped, 0 threads running -- Then I create new project select all but 2 platforms ( video & doc sharing), Then Tools \  Autofill \ Use Kontent Machine \ Import \ -- Select all the datafiles i exported with Kontent Machine ( 50 is max)  -- I then get the Out of Memory message.

     
  • darman82darman82
    @sven, from your calculation. Can you know how many project that we can run in GSA SER if we use this setting :

    - select use all the engine
    - didn't collect keyword.
    - 100 Submission a day
    - didn't use global list
    - didn't use PR or OBL

    Just the basic project. Can you know how many those project that we can run? cause when i run with scheduler i have a freeze problem.
  • SvenSven www.GSA-Online.de

    @royalmice can you send me these 50 files? I would like to get the same exception here and maybe improve it's performance.

    @darman82

    - select use all the engine  << doesn't influence the memory usage
    - didn't collect keyword. << influences memory usage a tiny bit
    - 100 Submission a day << doesn't influence memory usage
    - didn't use global list
    << doesn't influence memory usage

    - didn't use PR or OBL << doesn't influence memory usage 


    It's more depending on the data you have filled int he project (especially keywords).

  • darman82darman82
    - Keyword i use to scrape is 100K generic keyword
    - keyword for anchor text is only 1-2
    - LSI keyword around 50 spin keyword {key1|key2|key3|etc}
    - Thread 100-150
    - HTML timeout - 120
    - search engine queries - 10
    - dedicated proxies
    - Article i use is ACW articles

    @Sven, how abut that settings?

  • royalmiceroyalmice WEBSITE: ---> https://asiavirtualsolutions.com | SKYPE:---> asiavirtualsolutions
    @Sven

    Have zipped the Kontent Machine exported files and busy uploading to Dropbox now, will send you a message as soon as the upload is done


  • royalmiceroyalmice WEBSITE: ---> https://asiavirtualsolutions.com | SKYPE:---> asiavirtualsolutions
    Message send with the zip file
  • SvenSven www.GSA-Online.de
    @royalmice thanks. I will debugging it tomorrow.
  • darman82darman82
    OK, here is what i found recently...

    If we run the GSA SER at the windows 32bit then the max memory usage of GSA SER will be 1GB, but if we run the GSA SER at windows 64 bit the max memory usage will increase to 2GB.

    This is the main reason why only couple of people here who having the "out of memory" problem with GSA SER.

    I have ask my vps provider to re-install my vps with windows 64bit. I hope this will solve my "out of memory" problem.

    For those not having "out of memory" issue, i assume your windows is 64bit. I'm i correct?

  • OzzOzz
    if thats the case than it sounds like a bug to me as 32bit should use up to 2GB and 64bit up to 4GB
  • SvenSven www.GSA-Online.de

    @Ozz, it's actually true. A 32bit OS does only provide up to 2gb to the program if it is modified as in here: 

    http://www.maxi-pedia.com/3GB+switch+Windows+boot.ini+3+GB

    A 64bit program provides this only if the 32bit tells the OS it can actually handle that kind of address allocation.

    GSA programs allow that special use of more memory but on 32bit systems only if the boot.ini is modified.

  • AlexRAlexR Cape Town
    So I am running Win Server 2008 RS2. 

    1) Does this mean the whole system can only use 1gb RAM or just a single program running?
    2) If so, what's the point of everyone having 8gb of RAM on their VPS's if only 1 is used?
    3) I'm having issues with CPU use being very very high. I have 2gb RAM, but it's normally only using 100mb of it, but CPU is at 98% all the time. It's a 2 core, 6.80HZ CPU. Is anyone else having issues with SER using a lot of CPU?
  • BytefakerBytefaker
    Be carefull when SER measures the CPU usage. Check it via taskmanager or other windows tools. You'll see that it very seldom use 99% uf the CPU ressources.

    But CPU is my bottleneck too. I'm currently use a VPS with 4 cores and 4 GB ram. Ram is more than enough available but CPU is the problem.

    What you can do with the ressources: For example let Scrapebox run and harvest a nice "little" list of new targets you can import later.
  • AlexRAlexR Cape Town
    @Bytefaker - issues on both SER measure as well as task manager. I've even asked hosting if they can put a little VPS package together with a bit more CPU, but they won't. The next package has more disk space, RAM and other things I don't need. 

    That's why I was wondering if others are finding GSA SER is using a lot of CPU. I know Sven codes a lean program, but wondering if there is a feature or something that is eating up the CPU. 
  • HunarHunar
    edited January 2013
    I'm bottled by the 2G as well.

    On my server I have a lot of projects running.  If I load up the scheduler at 40 Projects and a switch of 30 mins and threads at 200.  Within 20 Mins Ser will say it's at 99% CPU and 2G Mem.  but If I look at my task manager I'm only at like 40% cpu usage and 25% Ram Usage.  So, I am barely using my Dedi to it's full potential and thus wasting a lot of money per month on the server.

    So my workaround to this is only loading up 20 Projects at a time with a switch of 30 mins and i stay around the 1g mem and 99% cpu usage according to SER.
  • AlexRAlexR Cape Town
    @Hunar - are your threads staying around 200 or is that just your setting? 

    The reason I ask is that many assume it's running at full threads. My swings from 5 to 100 just while I'm watching it.

    That's why I asked for Feature 10:


  • HunarHunar
    it stays constant at 200 threads because I have global site list enabled.  As soon as I disable it.  It will do that swing from 1-100 even though I have 200 threads enabled.
  • sonic81sonic81
    @hunar.  i notice this as well.  what's the cause of the drop when you turn off global lists i'm guessing because it doesn't have as many targets to post to?
  • HunarHunar
    If I remember right it's because the extra threads have nothing to do.  So it only uses the threads that have something to do.  I'd imagine that it could use the extra threads to do more searching or what not but sven, lee or ozz would know more about that than I would. :)
  • LeeGLeeG Eating your first bourne

    I notice threads drop when it does searches and also when it shuts down one project and starts another

    I run 250 threads, 10 projects on a 30 minute rotation

     

  • darman82darman82
    OK, i'm using trustvm vps and the os was windows 7 - 32bit. I have ask Carlos about to re-install my vps with windows 64bit but he answer me back by saying he doesn't have a windows 64bit os available. What a bad support system they have.I think i will have to move two of my vps from them to other vps service.

    @sven, about this page -> http://www.maxi-pedia.com/3GB switch Windows boot.ini 3 GB

    It only support this os :
    • Windows Server 2003 Standard Edition
    • Windows Server 2003 Enterprise Edition
    • Windows Server 2003 Datacenter Edition
    • Windows 2000 Advanced Server
    • Windows 2000 Datacenter Server

    My vps was Windows 7 and it does mention that if we edit it wrong it can cause Windows not to start. But is ok, now i know the problem is not at the GSA SER but instead because my VPS OS. I guess i have to move it then.

    OK i think this problem is solve, so the solution for the "Out Of Memory" problem is, please use windows 64bit for your OS and have minimum 4GB memory.


    Thanks All...

  • SvenSven www.GSA-Online.de
    Well I read it's working for XP as well...so I would guess for Win7 as well. Not tried it though.
  • grindgrind
    One install running three new projects with ~5k urls and 15k keywords (plus Collect Keywords From Target Sites and Use Collected Keywords To Find New Targets selected) would hit the 1GB (32bit) memory limit in about 3 minutes.

    Knowing I'd been running the same project with diff URLs and keywords for three weeks without instance, I tried to figure out what I'd done differently. Each URL has specific keywords assigned to it along with a fair number of raw URLs and random anchors (click here, click this, etc) set up in a deep nested spintax configuration to give me the %s I want. I had to do it this way instead of using the checkboxes below the anchor text area since the keywords were specific to URL.

    Anyway, for this run, I used a bunch of %Website% variables instead of the actual URL list in my set up file, as it made for quicker work in excel and as soon as it fired, memory over limit. Rebooted, ran one project only, same thing. Tried another project, same thing. Went back and fixed my data to use the actual URL instead of the %website% variable, fired it up, been running like a champ for 20+ hours now, current memory usage is 221MB.

    So if you're using a ton of nested spintax and %variables% and running into memory issues, try replacing the variable with the actual data. Takes longer to set up with excel, text editor of choice and some regex than using the variables but seems to run a lot cleaner. Hope that helps somebody.

    Grind
  • darman82darman82
    @Grind, the article i use is from ACW and i only use %anchor_text% variables and no other things.

    I will test to edit my scrape keyword, now i'm using 100K generic keyword for scraping. I will edit it to around 1000keyword only.

    I have 2VPS and both my vps was out of memory. :((
  • royalmiceroyalmice WEBSITE: ---> https://asiavirtualsolutions.com | SKYPE:---> asiavirtualsolutions
    edited February 2013
    For some reason my GSA SER dont want to use more than 1GB of RAM, even when it indicated 99% cpu usuage.

    I get Out of Mem warings when i start GSA SER, and then after about 10 warnings it runs fine and no more warnings.


    Here is my set up:

    WIN 7 - 64 bit
    CPU : Core I7 - 2006 k Sandy Bridge - over clocked to 4.2 Ghz
    RAM : 16 GB DDR3 (usually only using around 50%)

    GSA SER : V.5.04
    Projects: 43 ( 10 projects each with about 3 tiers )
    Threads : 100
    HTML Time Out : 130
    Use proxies Everywhere
    Time to wait between serp queries : 20
    Automatically search proxies every 0 min.
    Check newly added proxies + retest previous
    I use 40 of my own semi shared ( buy proxies) + the scrapped.


    Why dont want my SER want to use more than 1 GB RAM ?
    I read somewhere that we can get it to use up to 4 GB RAM on a win 7 64 bit system but no one seems to be able to tell me how


  • darman82darman82
    hi royalmice,

    from my experience GSA SER will use 3GB max memory if we run the GSA SER at windows x64 bit. Also, from my testing - GSA SER can only run 40 campaign max, for better result just use 30 campaign.

    I keep having the same problem until i stop some project and only run my highest priority campaign.
  • royalmiceroyalmice WEBSITE: ---> https://asiavirtualsolutions.com | SKYPE:---> asiavirtualsolutions
    HI darman

    Thanks for the feedback. My problem is my GSA SER dont want to go over 1 GB, even if i run 500 threads.
  • darman82darman82
    GSA SER don't go to 1GB if you using 32bit windows, but it really strange since your windows is 64bit.

    Make sure you uncheck monitor pc resource and automatically lower thread at options. Don't use those options.

    try it
  • royalmiceroyalmice WEBSITE: ---> https://asiavirtualsolutions.com | SKYPE:---> asiavirtualsolutions
    @darman82 yes i did try with both monitor pc resources on and off but it did not mke any difference.

    I think i have just managed to get it to use 2GB RAM.

    I stopped all projects and edited each project as follow
    Schedule on a 50 verified submissions per day.
    Dont collect keywords from target sites
    Dont select use domain url as abchor

    I noticed that when i clicked the main stop, some projects were not stopping, only showing stopping for more than a hour. After i edited each project i restarted it and this time all was ok.
    So either the editing i done fixed something that was causing it not to go over 1 GB.

    Anyways all ok now and it is cranking along at 2 GB. Now i just need to wait for the cows to grow horns then Sven might do a 64bit version (with optemised code ), so i can make full use of my available RAM
  • bretty113bretty113
    Schedule on a 50 verified submissions per day.
    royalmice how to schedule 50 verified submission per day?
  • mamadoumamadou
    edited July 2014
    @Sven

    You said in this thread :

    >> I said I will not compile a 64bit version. Though with some tricks on the memory manager I made it to use up to 4GB on a 64Bit system.

    My questions is , On all my 64bit servers once SER hit 1.9 / 2.1 GB of ram ..I get the annoying "Out of memory" message forever!

    Any ideas why my SER installations are not using 4 GB Ram ??
Sign In or Register to comment.