Filter whole domain by time

lone_promoter

Hello again,
i tried a first project with GSA Spider, and while overall im satisfied, I ran into a problem: on some domains the program spends  A LOT of time. I know there is a checkbox to skip whole domain if nothing found after a long time, but even so, with some domains the program wastes a lot of time. I guess it finds 1 or 2 emails, then spends an eternity searching in the history database.

Is there an option to set a maximum ammout of time the program spends analisyng a domain ?

Thank You.

Sven

Sorry there is no value you can set. But feel free to use better filters to skip certain URLs/Links.

lone_promoter

Tx for the reply. Can maximum ammout of time /link or /domain be taken into consideration for a sugested feature?

Also, what can cause the very slow parsing of some pages ? and what can i do to speed up the process?

Thank You for your answer.

Sven

The check is done on a single domain only not across many domains so thats not a problem. The slow parsing speed is unknown to me. What type of parsing do you use? By search engines or by parsing from a starting URL?

lone_promoter

By search engines

lone_promoter

any ideea about very slow parsing of some links?

Sven

OK then it's due to the search engines the program has to wait for to not spam them.

You can try to use different/more than one search engines and it would speed things up.

lone_promoter

when its parsing a link, does it no meant it allready arrived at a destination and is analysing that page ?

Sven

Yes but it might have analyzed all search engine results and is now not able to ask the search engine again before a certain time passed.

lone_promoter

as i see, if this is the case, queue should be empty. If every time parsing is on hold and queue is epty you are right. I will check this in the following days.

However, what if queue is not empty and it has allready harvested some urls ?

Sven

It will take URLs from the queue not being search engines and parse them.