Skip to content

Threads limit for the project

AngryRedPandaAngryRedPanda
in Feature Requests
I have a lot of projects in the works, and it often happens that I start a new project in the list of sites, and it starts collecting and processing information. For this case I have a general limit of 70-100 threads. 

But because the load on projects at startup is uneven, I run into the problem that I have an overloaded queue for recaptcha resolution and PR filtering. As the result some domains are skipped before PR check and some don't wait for captcha (I use xevil on the same server and there are up to 10 captchas being processed simultaneously).

As a result, because of the uneven load, I am missing a lot of sites from the full analysis. 

Maybe if I could set a thread limit for a specific project, I could solve this timeout problem. 
Tagged:

Comments

  • rastarrrastarr Thailand
    If you click the dropdown on the 'Start/Stop' button, you can enable the scheduler.
    It's common (and I do it too) to run 20 or whatever you want projects at any one time for a 30 minutes at a time. This cuts down on the load, memory usage and captcha issues.
    Give that a try
  • AngryRedPandaAngryRedPanda
    rastarr said:
    If you click the dropdown on the 'Start/Stop' button, you can enable the scheduler.
    It's common (and I do it too) to run 20 or whatever you want projects at any one time for a 30 minutes at a time. This cuts down on the load, memory usage and captcha issues.
    Give that a try
    Yes, I sometimes use this solution too, but I think it is not very convenient. So I'm hoping for a more convenient solution, that's why I wrote this post in ideas.
Sign In or Register to comment.