Skip to content

crawl site for url option

martymarty UK
in Feature Requests
when i was looking on the backend of gsa before i thought i saw a crawl url option, so you could crawl all of your sites urls & it would put all the urls in a file,
main gui>options>advanced

but i cant find it now, does anyone know where it is ?

i think they are talking about it here




Comments

  • cherubcherub SERnuke.com
    It's in the project settings. So click on your project > edit > data tab > edit next to url box > crawl urls/anchors
    Thanked by 1marty
  • martymarty UK
    thanks - that worked, it got all the main urls, but it also added loads of urls like
    wp-includes/css/dashicons.min.css?ver=5.6.4

    is there a way to filter them out ?

    i did it twice, by using the sitemap & not using the sitemap & it got all those extra urls both times.

  • cherubcherub SERnuke.com
    Probably something like right click on the found urls, uncheck by mask and then use something like *wp-includes* or *wp-content* as the mask. But yeah I wouldn't think it should crawl css and such.
Sign In or Register to comment.