Skip to content

How Does It Allocate Resources to Scraping & Submission?

AlexRAlexR Cape Town
in GSA Website Contact
Hi

I have a project that has 10000 keywords & 250k URLS I loaded into the queue.

I.e. if it processes the URLS in the queue, I want it to then roll over and do scrapes based on keywords. 

How does it currently handle it if we have both keywords & load urls to submit to.

Many thanks
Alex

Comments

  • SvenSven www.GSA-Online.de
    1. starting search tasks up to a certain amount
    2. on free resources it starts check/Send threads
  • AlexRAlexR Cape Town
    So it's best I remove all the keywords as I want it to process the submit URL queue first!

    Any way to have it do it the other way around? (i.e. have keywords as a backup in case it's finished the submitted urls) - this means projects won't sit idle when URLS are processed.
  • SvenSven www.GSA-Online.de
    don't you see the checking/sending happening?
  • AlexRAlexR Cape Town
    The preview screen has too many urls so takes forever to render. (would be great if it could be disabled to make it faster to switch between projects)

    I see it now.

    Is there any chance you could add a feature that rather submits imported URLS first and then scrapes keywords if it runs out? 
  • SvenSven www.GSA-Online.de
    why? thats why you have the option to set the threads for search/check to prioritize the one or the other.
  • AlexRAlexR Cape Town
    That's a global setting. If some projects have keywords and no urls in queue I want it to run full threads on it.

    If you're loading urls, it should do those before keywords as it's faster. 
Sign In or Register to comment.