Skip to content

Remove Duplicate URLs Question

mirenmiren Macedonia
in Other / Mixed
I have used Scrapebox to scrape for new urls. Now I have a new list of 5M urls but I know that a lot of them are already imported into GSA SER. Is there any way that I can remove those urls from the new list that are already into the submitted, verified, identified or failed list because this way SER will just import a lot of duplicate URLs and that is a waste of time and resources.

Thanks

Comments

  • SvenSven www.GSA-Online.de
    Sorry but this is not going to work as it would require to parse all site lists files and compare it with your urls. A big memory waste.
  • mirenmiren Macedonia
    Ok @Sven and thanks.
    Anyone knows some other program that can do this?
Sign In or Register to comment.