Skip to content

Server recommedations for GSA PI

Devender_GargDevender_Garg India
in GSA Platform Identifier
Hello Guys, 

I see many screen shots here where someone using GSA PI with 300+ threads.

My server dies with 50 threads.

I would like to know if someone can share their Server source and perfoemce with GSA PI please.

Thanks 

Comments

  • Devender_GargDevender_Garg India
    @AliTab Can you help, please?


  • TheGypsyTheGypsy Madrid
    solidseo vps comes to my mind and maybe asia virtual solutions is still selling vps packages. If you want performance then get ready to reach deep into your pocket though.
  • AliTabAliTab https://gsaserlists.com
    Devender_Garg said:
    @AliTab Can you help, please?


    Hi
    First of all choose dedicated server not VPS.
    I recommend AMD Ryzen series because they are more powerful and cheaper than intel ones.
    If you are running GSA SER + GSA CB beside PI, 16 GB ram Is enough and you even can run other SEO tools simultaneously.


  • Devender_GargDevender_Garg India
    edited June 2021
    Thank You for your comments guys.

    I am actually using the same AMD Ryzen beast dedicated server with tons of DDR4 RAM and Nvme harddisk

    So I believe it is not an issue with the Server. Here are my testing results-


    1. I am running it with 200 threads and it is running fine with 50-60% CPU. But suddenly, it crosses 100% CPU.
    2. Sometimes it even stuck there then I have to restart PI.

    I am not using any "CPU extensive" marked settings.


    I wish there could be a setting that will automatically reduce threads according to CPU usages. Like in GSA SER (I don't use the GSA setting but it is more relevant in PI) @Sven can you look into it and make it working nonstop like SER?


    Suggestions and tips are welcome. All I want is 

    1. Make maximum use of CPU ... I don't want to run 100 threads and 30% CPU Use.
    2. Want something to prevent CPU from hitting 90%+ Mark

    Thanks, @Sven awesome tools and @TheGypsy @AliTab to build the community better and better.
  • Devender_GargDevender_Garg India
    I appreciate it if I get an answer on this from Sven, 

    What happens in the background when GSA PI hits the "Limit Bandwidth" threshold?

    Does it reduce threads untill Bandwidth consumption cool down 
    OR 
    It kills the threads 

    or anything else?

    Thanks Again

  • SvenSven www.GSA-Online.de
    it will try to pause the socks for a set interval before continuing to rad from them to stay under the bandwidth limit. It will not kill threads or in any way lose data.
  • Devender_GargDevender_Garg India
    There is a tremendous performance gap when checking and not checking "Limit Bandwidth"

    If I don't check it, I am unable to run even 100 threads.
    After checking it, I am running 400 Easily. 

    If the output is the same in both cases, then better to keep it checked. 

    I am working to find a sweet spot for my server 

    Thanks for the help.
  • AliTabAliTab https://gsaserlists.com
    Hi
    You definitely need to check "Limit Bandwidth". If you don't your PI will crash. My connection is around 300 Mbps and I have limited that to 80,000 kb/s
    with your server spec you should be able to have three projects with 300 threads/each easily.
    Also as you said, don't use CPU intensive options.
    You also can check "Add processed URLs to the black list database" and also "skip blacklisted URLs" and delete your database each 10 days or 1 month(the size will become big quickly).
  • Devender_GargDevender_Garg India
    edited June 2021
    Thanks, @AliTab for best practices. It helps a lot.

    I do keep all settings as you suggested. I was not aware of deleting the database. So I believe this may be causing trouble for me... (I am not able to run that many threads so far even with the powerful server.)

    MyDatabase is 20M URLs and 2GB in size (with 1 week scraping). What is your standard size before we should start over again?

    I will test with the PI database and how many resources hungry the process is (GSA PI filtering URLs against Blacklist DB.
  • AliTabAliTab https://gsaserlists.com
    My pleasure.
    I myself go for 10GB each time. It really helps to not process urls which were processed before but when the db file becomes big, It reduces PI's performance. You may experience many lags and not respondings.
  • Devender_GargDevender_Garg India
    Agree with you @AliTab, I am able to run 500+ threads easily UNTILL I hit 100% 

    Once I hit 100%, I believe many threads stuck there and it won't let the server goes below 100%.

    Even I close PI, the server stuck at 100% CPU. then I have to close it through the task manager and start over.

    I run it for processing files. It gives me a better idea OR how much is remaining and I also get a chance to remove duplicates before I run PI. So better work.
    I keep running scraping in the background.

    So I just have to come back and replace the processing file every 12-18 hrs but it worth the time.

     Does CPU run consistently for you?
  • AliTabAliTab https://gsaserlists.com
    Devender_Garg said:
    Agree with you @AliTab, I am able to run 500+ threads easily UNTILL I hit 100% 

    Once I hit 100%, I believe many threads stuck there and it won't let the server goes below 100%.

    Even I close PI, the server stuck at 100% CPU. then I have to close it through the task manager and start over.

    I run it for processing files. It gives me a better idea OR how much is remaining and I also get a chance to remove duplicates before I run PI. So better work.
    I keep running scraping in the background.

    So I just have to come back and replace the processing file every 12-18 hrs but it worth the time.

     Does CPU run consistently for you?
    Hi
    what a mess. how long does it take to reach 100% CPU after running PI?
    Also if you are scraping yourself, it's better to stick with monitor folder. if you are using scrapebox you can buy their Automator and let your Scrapebox run 24/7 and you can remove duplicates through Scrapebox automatically after scraping session is completed.
    these files are picked by PI and processed automatically.
    I'm running PI with 800 threads + 80.000 kb/s bandwidth limit and it's eating 20% of cpu.
    Best wishes
  • Devender_GargDevender_Garg India
    edited June 2021
    I am on 32 Core Server, Nvme Disk, 12GB DDR4 RAM 

    Running SB and Automator.

    I was monitoring the folder with PI previously but now monitoring files ... does not make any difference so for now, leave it.

    If I run 200+ threads and higher the 5Mbps limit, it hit the 100% CPU within 10 minutes and never comes back to normal.

    I also tested With OR without using blacklist.

    Next, I am going to do is, reinstall PI ... And I am also going to test on my smaller 8Core dedicated server,.


    WTH ... You are running PI with 800 threads + 80.000 kb/s bandwidth limit and it's eating 20% of CPU.
    I will be killing it if I managed to run that much ... definitely, be processing 10k URLs per minutes 
    Which server is that, if it is ok to ask?.

    This thread might help many others to configure PI correctly ... Thanks to you.




  • AliTabAliTab https://gsaserlists.com
    Devender_Garg said:
    I am on 32 Core Server, Nvme Disk, 12GB DDR4 RAM 

    Running SB and Automator.

    I was monitoring the folder with PI previously but now monitoring files ... does not make any difference so for now, leave it.

    If I run 200+ threads and higher the 5Mbps limit, it hit the 100% CPU within 10 minutes and never comes back to normal.

    I also tested With OR without using blacklist.

    Next, I am going to do is, reinstall PI ... And I am also going to test on my smaller 8Core dedicated server,.


    WTH ... You are running PI with 800 threads + 80.000 kb/s bandwidth limit and it's eating 20% of CPU.
    I will be killing it if I managed to run that much ... definitely, be processing 10k URLs per minutes 
    Which server is that, if it is ok to ask?.

    This thread might help many others to configure PI correctly ... Thanks to you.




    Oh, I see your network speed is too slow! you should put far smaller number in "limit bandwidth" input. start with 5000 and increase it until you find the sweet spot.
  • Devender_GargDevender_Garg India
    edited June 2021
    I am using BuyProxies semi dedicated 400 proxies but I am using same proxies for other works as well.

    dedicated server speed is too good, no problem in that.

    Looks like it is a proxies issue. Let me think about switching proxies and testing... Thanks for idea.
  • AliTabAliTab https://gsaserlists.com
    Devender_Garg said:
    I am using BuyProxies semi dedicated 400 proxies but I am using same proxies for other works as well.

    dedicated server speed is too good, no problem in that.

    Looks like it is a proxies issue. Let me think about switching proxies and testing... Thanks for idea.
    My pleasure. Let me know if the problem solved.
    Best wishes
  • Devender_GargDevender_Garg India
    Finalllllllyyyyyy ...........!!!!!!!!!

    Everything solved 

    Can not thank enough to the @AliTab .... Priceless support.

    Cheers to the great software and community.

  • AliTabAliTab https://gsaserlists.com
    Devender_Garg said:
    Finalllllllyyyyyy ...........!!!!!!!!!

    Everything solved 

    Can not thank enough to the @AliTab .... Priceless support.

    Cheers to the great software and community.

    Happy about that. It's always my pleasure to help.
    Wish you the best
  • newbienewbie
    Hi @AliTab
    I would like to ask you i use your service Gsaserlist.com  
    so im still need to use GSA PI or not for checked link ??? 

    I use Verified TopTier Targets [minDA-15] 

    I would like to ask how is there a way to not do new backlink everythime like  
    bypass what account ever do that before

    because every 12 hours run time my vpm is reduced to less than 5.00 or 2.00 
    So I had to delete the data in order to clear it. Everything and then come back to start again. My Vpm will came back to run at 25-30 

    Thank you for your spents times for help 




  • AliTabAliTab https://gsaserlists.com
    newbie said:
    Hi @AliTab
    I would like to ask you i use your service Gsaserlist.com  
    so im still need to use GSA PI or not for checked link ??? 

    I use Verified TopTier Targets [minDA-15] 

    I would like to ask how is there a way to not do new backlink everythime like  
    bypass what account ever do that before

    because every 12 hours run time my vpm is reduced to less than 5.00 or 2.00 
    So I had to delete the data in order to clear it. Everything and then come back to start again. My Vpm will came back to run at 25-30 

    Thank you for your spents times for help 




    Hello
    Nope. You won't need GSA PI when you're using our service.

    Use our verified targets folder and submitted targets in your Global options -> advanced -> identified and failed site lists.
    And for your Tier1 links, right-click on your project and then import target URLs. Go to our verified top tier targets folder and choose all of the files inside that.

    Best wishes

    Thanked by 1newbie
  • googlealchemistgooglealchemist Anywhere I want
    AliTab said:
    newbie said:
    Hi @AliTab
    I would like to ask you i use your service Gsaserlist.com  
    so im still need to use GSA PI or not for checked link ??? 

    I use Verified TopTier Targets [minDA-15] 

    I would like to ask how is there a way to not do new backlink everythime like  
    bypass what account ever do that before

    because every 12 hours run time my vpm is reduced to less than 5.00 or 2.00 
    So I had to delete the data in order to clear it. Everything and then come back to start again. My Vpm will came back to run at 25-30 

    Thank you for your spents times for help 




    Hello
    Nope. You won't need GSA PI when you're using our service.

    Use our verified targets folder and submitted targets in your Global options -> advanced -> identified and failed site lists.
    And for your Tier1 links, right-click on your project and then import target URLs. Go to our verified top tier targets folder and choose all of the files inside that.

    Best wishes

    why add them to the failed site list?
  • AliTabAliTab https://gsaserlists.com
    googlealchemist said:
    AliTab said:
    newbie said:
    Hi @AliTab
    I would like to ask you i use your service Gsaserlist.com  
    so im still need to use GSA PI or not for checked link ??? 

    I use Verified TopTier Targets [minDA-15] 

    I would like to ask how is there a way to not do new backlink everythime like  
    bypass what account ever do that before

    because every 12 hours run time my vpm is reduced to less than 5.00 or 2.00 
    So I had to delete the data in order to clear it. Everything and then come back to start again. My Vpm will came back to run at 25-30 

    Thank you for your spents times for help 




    Hello
    Nope. You won't need GSA PI when you're using our service.

    Use our verified targets folder and submitted targets in your Global options -> advanced -> identified and failed site lists.
    And for your Tier1 links, right-click on your project and then import target URLs. Go to our verified top tier targets folder and choose all of the files inside that.

    Best wishes

    why add them to the failed site list?
    Hello
    Because there are just 4 inputs (identified, submitted, verified, failed)
    You'd better save your submitted and verified targets and make your own lists from our database. so, there will be 2 free inputs that are failed and identified and you can use them to add our Dropbox folders and receive our real-time targets.

Sign In or Register to comment.