Skip to content

What Uses More CPU/RAM: Processing Raw Lists or Submission/Verifying

JudderManJudderMan UK
in Other / Mixed
Simple question, but one I want to hear from the pros:

I'm moving some projects to a new server and proxies as I want to localise them to the EU but wondered how hard SER runs just processing lists (raw ones from Gscraper/Scrapebox/ScraperBandit) compared to posting and verifying them. I've been using SER for a long time but I'm not 100% sure on the answer.

Thanks in advance

Comments

  • sysco32sysco32 Skopje
    I think if you use the same amount of thread than the cpu/ram costs will be the same.It just makes difference on your LPM.
    So far i didn't see that SER would use a lot of ram,rather CPU.
    Scrapebox is using the RAM,when you finish with the scraping.
  • NikodimNikodim
    Im not a pro by any means, however in my earlier scraping experience, when using all engines in processing of raw list, things can be as ugly for CPU as they would be on normal runs with selected engines.

    So for example I did tests like:
    Comments + All engines run, monitoring CPU load
    vs
    Comments + Selected engines, monitoring CPU...
    vs
    No Comments + Selected engines, etc...

    The most loaded test (Comments + All engines) loaded CPU similarly to the Selected Engines normal run. (same server Xeon E3 1230 v2)

    But it was quite some time ago (summer 2014, so old SER versions) and also, multitude of settings can affect CPU/RAM usage, so it should be taken with a grain of salt.
  • JudderManJudderMan UK
    Thanks man, I run very quick dedis but am tempted to move down a few grades as they are completely overkill and with the savings I can run more smaller dedis or even VPSs seem better than they used to be these days.
  • NikodimNikodim
    edited March 2015
    Yeah, I'm personally obsessed with optimizing (minimizing ;) ) costs by any means necessary so I totally understand you. 

    Maybe you could just grab one weaker dedi, say the one that you think will suffice based on your current CPU/RAM/Bandwidth usage, test your common setups (scraping/posting, your preferred threads, global and project settings, etc.) on it for a bit and then migrate accordingly. 

    Personally, I think VPS is a bit risky (being shared with other users), but its just me, Im not tech-savvy in that kind of thing, just prejudice really, so maybe VPS will also suffice.
  • JudderManJudderMan UK
    Changed my mind, gonna run the super fast dedis and have Scrapebox/Gscraper feeding lists to each set of projects on each SER install.  I could do a master scraper server but I find that passing the lists between servers with Dropbox is a pain and has so much potential to go wrong. 
  • IdentifiedURLsIdentifiedURLs
    What could go potentially wrong with sharing list via Dropbox @JudderMan ?
  • JudderManJudderMan UK
    edited March 2015
    @IdentifiedURLs I have seen some weird things with Dropbox, probably just the way I set it up but I no longer use it for letting SER pull files from. I just use it to share files between servers but then I upload onto the Desktop (as this I find lets SER post faster than having to go into DB when other servers are accessing the same files.

    SERLists did a great write up on how to stop this from happening but can't remember if it's in the forum or as part of their zip file when I bought some lists from them.
  • IdentifiedURLsIdentifiedURLs
    @JudderMan

    Ah, got it. Thanks
Sign In or Register to comment.