Dedup for MORE than 100k dup URLs only works after restarting SER

hans51

the funciton to remove duplicate URLs

> options > tools > remove duplicate

seems NOT to work if number of dupicate URLs greater than 100;000
= all inactive
= all stopped

only after shutting down SER and restarting SER = de-dup works  as usual

without shutting down SER
the dedup progress bar appears, the dedup function seems to run, but very fast = may be 2-4 seconds (normal would be on same system some 30-60 seconds per dedup run)

at the end is the message 0 dup removed

dedup without shut down of SER however works on stopped system if number of dup URLs only a few or several ten thousand URLs

the critical value seems to be a 6-digit number of URLs = once a day

donaldbeck

You on a VPS or something like that? Sounds like a hardware limitation, because I dedup much larger numbers without doing anything special.

hans51

on regular laptop acer aspire one
all laptop ONLY for SER and SB
de-dup done when all halted and all resources free

donaldbeck

I can tell you it works for me with 10's of millions removed.

cherub

I've experienced one situation similar to hans51's where after stopping SER after 8 or so hours to dedupe files, the usual 2-3 minutes it took to dedupe took an apparant 10 seconds. After checking a few sitelist files I noticed many duplicates still present. The problem went away after restarting SER, and unfortunately I can offer no other information that may replicate the problem

hans51

so we have 2 opposite experiences
and may have to live with that
needless to say = I am always running the very newest SER version
several weeks ago,
that situation did NOT exist and dedup even was working on running SER
now dedup seems to work only on stopped SER or all inactive
but several weeks ago I also never had more than xxk dups, never xxxk