Skip to content

Dedup for MORE than 100k dup URLs only works after restarting SER

the funciton to remove duplicate URLs

> options > tools > remove duplicate

seems NOT to work if number of dupicate URLs greater than 100;000
= all inactive
= all stopped

only after shutting down SER and restarting SER = de-dup works  as usual

without shutting down SER
the dedup progress bar appears, the dedup function seems to run, but very fast = may be 2-4 seconds (normal would be on same system some 30-60 seconds per dedup run)

at the end is the message 0 dup removed

dedup without shut down of SER however works on stopped system if number of dup URLs only a few or several ten thousand URLs

the critical value seems to be a 6-digit number of URLs = once a day


  • donaldbeckdonaldbeck Advanced SER Videos -> | Learn SER For Free Step By Step ->
    You on a VPS or something like that? Sounds like a hardware limitation, because I dedup much larger numbers without doing anything special.
  • on regular laptop acer aspire one
    all laptop ONLY for SER and SB
    de-dup done when all halted and all resources free
  • donaldbeckdonaldbeck Advanced SER Videos -> | Learn SER For Free Step By Step ->
    I can tell you it works for me with 10's of millions removed.
  • I've experienced one situation similar to hans51's where after stopping SER after 8 or so hours to dedupe files, the usual 2-3 minutes it took to dedupe took an apparant 10 seconds. After checking a few sitelist files I noticed many duplicates still present. The problem went away after restarting SER, and unfortunately I can offer no other information that may replicate the problem :(
  • edited October 2013
    so we have 2 opposite experiences
    and may have to live with that
    needless to say = I am always running the very newest SER version
    several weeks ago,
    that situation did NOT exist and dedup even was working on running SER
    now dedup seems to work only on stopped SER or all inactive
    but several weeks ago I also never had more than xxk dups, never xxxk
Sign In or Register to comment.