Log: Same two urls

system0102 · March 2014

Hi @sven

Was wondering why in the log each line displays the same url twice? Is this a bug or how it's supposed to be?

[10:15:09] [+] 0911/1867 unknown extra submission status (188/4300 submitted - AVG: 149.39/h) - http://daxue.tywiki.com/index.php?title=Gay_Rights_Movement_History_Uk http://daxue.tywiki.com/index.php?title=Gay_Rights_Movement_History_Uk

[10:15:09] [-] 0866/3340 download failed - http://ymcaleadersclub.org/social/blog/view/363098/say-no-to-short-term-loans http://ymcaleadersclub.org/social/blog/view/363098/say-no-to-short-term-loans

[10:15:09] [-] 0836/4808 no engine matches - http://host280.hostmonster.com/suspended.page/disabled.cgi/allenshay.com http://host280.hostmonster.com/suspended.page/disabled.cgi/allenshay.com

[10:15:09] [+] 0837/4808 matches engine MacOSWiki - https://webtools.sau16.org/groups/ https://webtools.sau16.org/groups/

Sven · March 2014

Thats no bug but on some log lines you have an url supplied to it, on some not. That line with two urls is a log line with url in message and the url behind it.

cusabio1 · March 2014

The site under http://foo.com can be the same as under http://www.foo.com, but it can be a totally different site, due to web server configuration. It depends on the DNS config too, which IP kit points to www and which one to without www.

If you want compare two sites, you need to fetch the content, and compare key parts (using nokogiri for example) about similarities.

Nowadays due to sidebars and news, two consequent request to the same url, gives slight different html responses.

Log: Same two urls

Comments