Skip to content

Filter whole domain by time

Hello again,
i tried a first project with GSA Spider, and while overall im satisfied, I ran into a problem: on some domains the program spends  A LOT of time. I know there is a checkbox to skip whole domain if nothing found after a long time, but even so, with some domains the program wastes a lot of time. I guess it finds 1 or 2 emails, then spends an eternity searching in the history database.

Is there an option to set a maximum ammout of time the program spends analisyng a domain ?

Thank You.


  • SvenSven
    Sorry there is no value you can set. But feel free to use better filters to skip certain URLs/Links.
  • Tx for the reply. Can maximum ammout of time /link or /domain be taken into consideration for a sugested feature?

    Also, what can cause the very slow parsing of some pages ? and what can i do to speed up the process?

    Thank You for your answer.
  • SvenSven
    The check is done on a single domain only not across many domains so thats not a problem. The slow parsing speed is unknown to me. What type of parsing do you use? By search engines or by parsing from a starting URL?
  • By search engines
  • any ideea about very slow parsing of some links?
  • SvenSven
    edited June 2014

    OK then it's due to the search engines the program has to wait for to not spam them.

    You can try to use different/more than one search engines and it would speed things up.

  • when its parsing a link, does it no meant it allready arrived at a destination and is analysing that page ?
  • SvenSven
    Yes but it might have analyzed all search engine results and is now not able to ask the search engine again before a certain time passed.
  • as i see, if this is the case, queue should be empty. If every time parsing is on hold and queue is epty you are right. I will check this in the following days.

    However, what if queue is not empty and it has allready harvested some urls ?
  • SvenSven
    It will take URLs from the queue not being search engines and parse them.
Sign In or Register to comment.