Skip to content

0 URLS/minute Stuck at 100%

senginessengines us
in GSA Platform Identifier
I click on tools and tick remove duplicate domains option only.

I let it run and get back four million urls and am not able to save or export to project.

v1.26 windows server 2008 r2image



Comments

  • s4nt0ss4nt0s Houston, Texas
    edited July 2015
    Checking this now with a big file, will report back in a minute.

    *Edit* - Just tested on my end and saving worked fine and export to project also worked: 

    image

    I'm assuming that file you removed duplicates from is pretty big? Maybe its the size of the file you're using. Keep in mind it stores the file in memory when removing duplicates so depending on how much ram your VPS has, might consider splitting the file first.

    Can you try on a smaller file to test and see if it makes any difference? What happens when you click save? Anything?


  • kp55kp55
    Same problem. First I ran out of memory, then I tried with 5 files ~7mb each, which worked, after I loaded a second set of 5 files then this error happened. 

    Try deduping on a subsequent run, the Save button will not enable.
  • s4nt0ss4nt0s Houston, Texas
    Ok, I will do more testing on this to try and replicate the bug. I just tried it again and got back 1.9 million urls and its able to save fine .. :/

    You guys are loading in .txt files, right? 
  • kp55kp55
    Yes, text files. I'm using v1.26
  • s4nt0ss4nt0s Houston, Texas
    Ok, thanks do you think you can send me one of those 7 mb files that's not saving? I'll shoot you a PM. 
Sign In or Register to comment.