Does Remove Duplicates Compare All Files in a Folder?

spiritfly

I know that Remove Duplicates checks all files in the specified folder for duplicate urls. However does it compare the urls from all files inside that folder to each other and then remove duplicates, or does it compare for duplicates for each separately?

Example: Two files with urls exist in folder A. Both files contain the same set of urls. If folder A gets scanned with Remove Duplicates will it remove all urls from one of the files? If yes, how will PI select which file will get it's urls removed?

s4nt0s

It compares them like SER does (compare duplicates for each file separately)

This is because some identified URLS, might work for 2 different engines, so if it compared all .txt files and removed dups, it would remove URL's that could potentially be more backlinks. 

spiritfly

I understand. I was trying to use remove duplicates on my raw lists from scrapebox and I get to choose whether scrapebox should save all urls in one file or each session in separate files. I went with the second option, but I guess I will have to revert to one file for all urls so PI can process the duplicates before they go into the identifying process.

Thanks for the anwer!

s4nt0s

We can probably add an option for that in a future update to compare all files or single. I'll put it on the to-do list.