About Rechecking Failed Statuses
Here what I found:
I run software on a 4GB + 2CPU + 100Mbps VPS with 900 checking threads. I made a check over 15.000 web pages.
- Atfirst run it couldnt find contact pages on 8800 pages!
- Then I selected all the failed 8800 pages, reset their statuses and only run a check again only on failed pages(that does not have contact page).
- And interestingly it found around new 1000 contact pages among failed ones. So new number for failed pages reduced to 7800!!
--> So this is my result: since I use 900 thread for checking, they make checking in paralled with 900 threads. Thus software tries to open an http connection between pages. And since the total page number is high, i think, some of the pages return http tmeout, because my bandwidth is divided among 900 connection. This is why I get new pages for the second run on failed pages.
Sven am I wrong? I think this is very important point.
Also I believe same is valid for search threads. I use 10 proxies and total 150 connections. And now I strongly believe that my internet connection doesnt support this much connection! May be most searches giving timeout.
Sven for the next update, could you please add "TIMEOUT" log at the right of the current logs if the software stops waiting page results because of timeout.
Looking forward to hear you.
I run software on a 4GB + 2CPU + 100Mbps VPS with 900 checking threads. I made a check over 15.000 web pages.
- Atfirst run it couldnt find contact pages on 8800 pages!
- Then I selected all the failed 8800 pages, reset their statuses and only run a check again only on failed pages(that does not have contact page).
- And interestingly it found around new 1000 contact pages among failed ones. So new number for failed pages reduced to 7800!!
--> So this is my result: since I use 900 thread for checking, they make checking in paralled with 900 threads. Thus software tries to open an http connection between pages. And since the total page number is high, i think, some of the pages return http tmeout, because my bandwidth is divided among 900 connection. This is why I get new pages for the second run on failed pages.
Sven am I wrong? I think this is very important point.
Also I believe same is valid for search threads. I use 10 proxies and total 150 connections. And now I strongly believe that my internet connection doesnt support this much connection! May be most searches giving timeout.
Sven for the next update, could you please add "TIMEOUT" log at the right of the current logs if the software stops waiting page results because of timeout.
Looking forward to hear you.
Comments