Skip to content

[HELP!] Number of Threads

Please suggest me the exact number of threads that I should use.
* No scraping. I just import lists.
* CPU: 6.8 GHZ
* RAM: 2 GB

Thanks in advance!
«1

Comments

  • edited April 2014
    Start at 100 and increase by 10. When your CPU Usage (by task manager not GSA SER) reaches 95+% or your lpm does not increase anymore with additional threads then you've found the thread number you are capable of.
  • edited April 2014
    @bencrabara: Thanks! But my CPU is always at 99% even I already decreased it :(
    Do you know why? Do I still have to reduce the number of threads?
  • Just increase your threads until you see no increase in performance and dont get out of memory errors.
  • With 30 proxies you should be able to happily run 300 threads with that set up if you are just posting from lists, although the way that your projects are set up can also make a big difference to cpu usage.

    I'd also contact your VPS supplier and get them to check that all's well their end.
  • ronron SERLists.com

    I used to run at 99% for months on end with old versions of SER. That doesn't hurt anything.

    But I have to say that more recent versions tend to use a lot lower CPU, typically averaging at 50%. So I do find that unusual.

    That RAM is very close to the limit - I would be more concerned with that as SER can use 1.2GB by itself. Then add the Windows platform, CB, etc. and you are walking a very fine line.

  • edited April 2014
    @fakenickahl @2Take2 @ron Hello everyone! Thanks for your replies.
    I just upgraded my VPS to a more powerful one.
    Here's the new specs:
    * CPU: 20.4 GHZ
    * RAM: 6GB
    * Proxies: 30 semi-dedicated
    * No scraping. I just import lists.

    Please suggest me the number of threads the I should use. In this new VPS, whether it's 300 threads or 50 threads, CPU at task manager is always at 99% :(
    What is wrong? I can't really see difference in speed.
    Could someone please help me? I'm using solidseovps. :|
  • ronron SERLists.com
    edited April 2014

    Usually we recommend about 10 threads per proxy, so you are fine at 300. However, I think it is worthwhile to test in both directions (higher/lower) to see what happens. Some VPS's react differently.

    At face value it looks very strong, and you should be pleased with that unit.

    I'm not sure why you think something is wrong? Are you making an acceptable number of links? How many projects are you running? Are you using the scheduler? If yes, how many projects are you running at once? (You can only run so many at once on a VPS before the CPU starts gagging). What are your project limits (# of links) that you have set on T1's? On lower tiers?

    As you can see, there are so many variables. And I think you are making an assumption you are having a problem. 99% still doesn't mean you have a problem. If you have any issues, it may lie with your overall settings.

    Sometimes I lower the threads by 50% and still make the same # of links. Not that unusual. That is why you should actually count the verified links for something like 6 hours on one setting (like 100 threads), and then count the verified links for another 6 hours at 300 threads. Etc. I think you are looking at the controls, and expect to see something that isn't happening (and you think it is wrong when it isn't). 

  • edited April 2014
    Thanks for your reply @ron!
    My only problem is, whether it's only 50 threads, 300 threads or 500 threads, my CPU Usage at task manager is always 99%. I can't seem to find any difference at all :(
    I already contacted solidseovps but they said that I should get some help from the GSA forum.
    I'm only running 2 projects every time.
    By the way, I'm currently using your blue list (April release) and it's only running around 20 LpM. With my VPS specs and with your blue list, do you think my LpM is normal? Or is it too slow? :|
  • Update: I just run a new project and used my verified instead. The CPU Usage is now below 99%. I think the problem is with the blue list :(
  • goonergooner SERLists.com
    @ohohraii - I'm not sure that putting a list in identified folder can make your CPU usage go crazy, it's not like the list has millions of URLs. Maybe @sven can clarify that?
  • edited April 2014
    @ron @gooner: Now I'm running around 150 LpM using my verified list and I can now increase the number of threads without reaching 90%+ CPU Usage.
    But whenever I use your blue list from SER Lists, the CPU usage (at the task manager) always becomes 99% whatever number of threads I put (whether it's only 50 threads, 100 threads or 500 threads).
    Now I think the problem is in the blue list. It was the first time I bought your list. I thought it was a verified list :|
  • molchomolcho Germany
    Have you imported the list into the project,
    or read your project the list from a folder.
    I have problems with with my CPU usage when i import big lists directly in several projects, but when SER the same reads from a folder, he runs normal.
  • 2Take22Take2 UK
    edited April 2014
    Hi @ohohraii, before it was released, I personally tested the blue list twice on 2 different rigs and I can assure you that it is not the cause of your CPU problems.

    As above, if you import it into 'identified' and run from that, then you should be fine.

    Also, are you using the latest version of SER as it appears that Sven has just improved CPU usage a bit?


  • edited April 2014
    @2Take2: Thanks for your reply. Yes, I imported my list from the identified. And yes, i'm also using the latest version of GSA SER. In fact, I already tried to uninstall and install it again but i'm still having the same problem :(
  • PM Sent.
  • Could you do a little test like this:

    1. Make one dummy project with all engines checked except maybe videos and some other that require Document and image from you. Load it with only this blue list via Import Tagret URLs - From Sitelist. Set around 300 threads and measure CPU load. And also measure it for low threads count, like 50.

    2. Make another dummy project with same settings, load it with your list instead and measure performance. 

    Does it really different then?

    I have no Serlists but I have also some CPU problems and it would be interesting to know that maybe some engines put a lot of strain on CPU? Maybe you could also show stats from both lists (if its okay) via Tools to see what engines are most common?

     

  • I have this issue with cpu too..And i don't use the blue list right now, so the problem is not from the list. 
    If anyone had this issue and solved, please post here!
  • I have smothing simliar. When I right click-import from site list-verified, CPU is ok 40-50%. But if i set it up to use verified list CPU is always at 99%. Could be a bug?
  • I tried all methods, same result. Cpu over 95%. Don't know what to do anymore.
  • KaineKaine thebestindexer.com
    edited April 2014
    Suggestion

    Is that SER consumes the CPU or CB ?

    It is possible that if SER go faster, you spend more time to break captcha.

    The CPU works more on a check list for this reason, all threads have captcha to resolve.
  • SER consumes the CPU.
  • edited April 2014
    Yes, GSA SER has the problem, not CB. CB's CPU usage seems fine with me. But as I have observed, there are lots of captchas that can't be solved by CB. From 65% success rate, it suddenly plunged to 48% while using the blue list.
    I think GSA SER really has a bug right now. When I import using my site list, it works smoothly 50% of the time (You heard it right.. SER will still have 50% chance that it will mess up).. But when I import my target list through "import file", the CPU usage becomes crazy every time.

    I also tried to import my verified list. In the first 5 minutes, everything seems working well. But after 10-20 minutes, GSA SER suddenly becomes crazy. My CPU usage is very low, the number of threads is also low.
    I think GSA SER is not using all my VPS's resources or something.
    Already sent 2 bug reports.
  • goonergooner SERLists.com
    @ohohRaii - Please send me a PM and i will look into it for you, it's late here so i will reply in the morning.
    Thanks.
  • edited April 2014
    I have to add that I'm also seeing crazy RAM and CPU usage while I'm using a list with a lot of verified urls. It seems to have nothing to do with lots of submissions, but rather posting to a lot of very good targets. Allow me to explain how I came to this conclusion. 

    On two projects I imported a very recently created verified list of about 80k urls and the RAM usage quickly rose to 3GB resulting in out of memory errors on 500 threads, I had to lower my threads to 250 to avoid getting errors. After running that verified list I imported a list of 2 million unique domains I had just scraped into both of these projects, and I was able to run 1000 threads just fine again. This scraped list gave me a lpm of 400-500 while the verified list was just at about 200. I had verification completely turned off on both projects, so I'm positive it does not have anything to do with verifying a lot of successful submissions.

    Are any of you guys seeing something similar?
  • goonergooner SERLists.com
    Verifying usually uses less resources i believe. I also notice that if for example you import a new list with lots of nice fresh targets for your projects. SER goes into a feeding frenzy posting links in super quick time and then yes you can see increased use of resources.

    I guess SER uses more resources posting than it does verifying or looking for suitable URLs.
    Those are only my observations but it seems to be true.

    When SER is going crazy posting like mad i usually reduce the threads a little if it is causing problems. After some time SER will slow down and you can up the threads again.
  • @gooner, my point was that SER used a lot more resources posting to a verified list of very high quality compared to a freshly scraped list, even though SER was doing a lot more submissions per minute to the freshly scraped list.
  • goonergooner SERLists.com
    edited April 2014
    @fakenickahl - Ahhh i thought you meant because the verified list has more good targets = more time posting
    A scraped list doesn't have that many viable targets = less time posting.

    But now you say you got more submissions per minute on the scraped list, maybe that is not the answer.
  • @fakenickahl I d notice the same, albeit not on the same grand scale :@) . What about targets structure? Does "easy" one - scraped list contain more "easy" targets like shorteners while Verified contain more "hard" ones like forums, maybe?
  • MrXMrX Germany
    I would like to hear a word of @Sven as he is the IT pro here. As far as I know threads also depend a lot on your internet connection as running 500 threads on a 5mbit wire will let you wait for hours for results ...
  • @magicmastermind: I think so. But in my case, it's not because of internet connection because I have 500mb/s+ result on speedtest.
Sign In or Register to comment.