Skip to content

Does Remove Duplicate URL's + Remove Duplicate Domains Speed Things Up?

nicericenicerice
edited January 2013 in Need Help
Curious to know this as to my mind it would only be sensible to assume that it does.

I am wondering then if we can create a setting for each of Remove Duplicate Url's every XX days and Remove Duplicate Domains every XX days also?

Might be another bottle neck that could help?

Thanks for readings
NR

Comments

  • SvenSven www.GSA-Online.de
    This is not influencing speed as the program picks up random URLs from the file. It doesn't load the whole file into memory or alike and would not wast time/memory. It jumps to a random location in the file and reads some lines out of it. Cleaning the file is not influencing anything.
  • nicericenicerice
    Thanks Sven, that is enlightening.
Sign In or Register to comment.