Skip to content

Global Site Lists: Better Threading

BytefakerBytefaker
edited September 2012 in Feature Requests
I really like the idea of a global site list (3 big "pots" of urls which are used for scraped url's and to share between the different projects). What I think is suboptimal is the use of the available ressources. Wouldn't it be possible if there are not processed URL's to use the not used threads to get them done?

I know I work at the upper borders, but for my examples 800 threads work nice (takes about 80 mbit/second) if I force SER to do it (like identifying sites and so on). But with normal operations there aren't very often more than 10 threads used (and only less bandwith). In my eyes is this a waste of ressources. Why not using free threads to handle the global list beside the "normal" searching/posting.

I don't see any disadvantages with this strategy and a HUGE performance boost for people which really use their global site list and feed SER with a lot nice urls :)
Tagged:

Comments

  • SiNeXSiNeX
    800 threads for import of Global Site List? Are you kidding - what kind of super computer are you running. Anything more than 200 on a i2600k 16GB ram and CPU is 100% maxed even less on a VPS.
  • BytefakerBytefaker
    Not only for global lists, but for stuff which uses multiple threads. It's a dedicated server, Core i7 2600k, 16GB RAM and with a strong backbone connection. I don't have the slightest problems with CPU (max about 60-70%, uses only 2.3 GB RAM)  and - perhaps this is the difference - on a fast SSD. That's the reason why I think it's a little bit pitty to have such resources and only use them for about 10% of the time. But on this server only runs SER and CS - if you have a lot other CPU consuming tools running, then of course your VPS can't handle it.

    I know nobody else uses that much threads, but I think it would be a huge step forward to use 200-300 threads the most time.
  • AlexRAlexR Cape Town
    +1 for this!
  • BytefakerBytefaker
    edited September 2012
    (just read your other (https://forum.gsa-online.de/discussion/214/increased-actual-threads#Item_12) thread - you see that's also my problem... don't have nice/fast results when I'm not forcing the project to only use the global list).
  • AlexRAlexR Cape Town
    Raised the issue a few times:

    And another thread:

    As per what Sven says "To make it short, if you want to use all your threads on submission, you need to import target urls or use site lists."
  • BytefakerBytefaker
    Thanks for linking the relevant information. That's exactly my point, I'm doing it over target urls/global site lists, but when I don't force SER via status "Active (use global lists only" I only get a really low thread count and can't use the resources.

    Of course there shouldn't now new way of using SER implemented, but I don't see any disadvantage in using the defined threads exactly for this important purpose.
  • OzzOzz
    >Thanks for linking the relevant information. That's exactly my point, I'm doing it over target urls/global site lists, but when I don't force SER via status "Active (use global lists only" I only get a really low thread count and can't use the resources

    so what you are saying is if you uncheck all SEs and check "use global lists" than GSA won't make use of all threads? only if you change the status to "global"?
  • BytefakerBytefaker
    edited September 2012
    That's my experience... Am I wrong? Or at least very seldom...
Sign In or Register to comment.