Skip to content

Remove Duplicate Domains/URLs

BrandonBrandon Reputation Management Pro
edited February 2013 in Feature Requests
With some platforms, having the domain in their twice doesn't help, but it definitely hurts.  Sven said that my "Already Parsed" error was due to the fact that I had domains in my "success" list multiple times.  I checked and removed dupe domains with SB.  For Drupal I had 285,000 domains, dupe removed I only had 26,301 so basically 90% of my list was the same duplicate domain.

Is there a way to have GSA remove duplicate domains from certain platforms?  Obviously this wouldn't be good for blog commenting or trackbacks, but for most domains it would be a huge speed increase.

Comments

  • LeeGLeeG Eating your first bourne

    Thats an easy one

    Options > Advanced > Tools and the option is there

    I delete duplicate urls every day when I restart ser

  • BrandonBrandon Reputation Management Pro
    Thanks @LeeG my files are too large and it's crashing SER.  I'll try to do them a few at a time and see if I can get it under control.
Sign In or Register to comment.