Crawl Delay (?)

daviddig1

Hi @sven is there a way to add a delay in crawling sites just like I can in sending? I cannot seem to find that setting if it exists

The reason why is because I get false negatives given that the software opens a webpage with a bot-vs-human verification page for about 5 seconds and immediately writes the page off as "no contact form" incorrectly

Sven

well the content is downloaded and parsed for a contact form. If there is none of that, it will not magically change the content in like 5 seconds...thats done by javascript e.g. and usually this can not be executed to see the real content. Show me a sample URL so I can have a look though.

daviddig1

Sure @sven here's an example:
https://www.superioroverheaddoors.com/contact-us/

Sven

I don't see any contact form here.

daviddig1

Oh you're right that's weird ! It used to pop up

Well at least here's one:
https://winerlawoffice.com/contact-us

It finds it crawling with proxies but then cannot without

daviddig1

Here's another example:
https://remodelercorpuschristi-tx.com/contact/

Sven

Sorry for long delay....both work for me...