Skip to content

Crawl Delay (?)

daviddig1daviddig1 San Antonio, TX
in GSA Website Contact
Hi @sven is there a way to add a delay in crawling sites just like I can in sending? I cannot seem to find that setting if it exists

The reason why is because I get false negatives given that the software opens a webpage with a bot-vs-human verification page for about 5 seconds and immediately writes the page off as "no contact form" incorrectly

Comments

  • SvenSven www.GSA-Online.de
    well the content is downloaded and parsed for a contact form. If there is none of that, it will not magically change the content in like 5 seconds...thats done by javascript e.g. and usually this can not be executed to see the real content. Show me a sample URL so I can have a look though.
  • daviddig1daviddig1 San Antonio, TX
    Sure @sven here's an example:
    https://www.superioroverheaddoors.com/contact-us/
  • SvenSven www.GSA-Online.de
    I don't see any contact form here.
  • daviddig1daviddig1 San Antonio, TX
    edited May 28
    Oh you're right that's weird ! It used to pop up

    Well at least here's one:
    https://winerlawoffice.com/contact-us

    It finds it crawling with proxies but then cannot without
Sign In or Register to comment.