Skip to content

Import Lists

@Sven or anyone else who me be able to help - Is there a way to set up SER so that when it has finished processing one list of potential sites, it automatically loads the next one?

The reason I ask is that I've been v busy scraping lately and I've got something like 40million *potential* pages I can drop a link with SER.  I've split them into lists of 500k, but even so that's 80 files I've got to import and when I've been importing these into a couple of campaigns I've created purely to process these links, it seems to slow everything down.

It might not make a difference to the speed, but it would be good if from within a project you could set a directory where it imported link lists from and then I could spilt them into smaller files of about 10k sites.

Thanks


Tagged:

Comments

  • SvenSven www.GSA-Online.de
    There is no speed decrease if you import as many URLs as you feel like. The URLs are read in 1MB batches from the imported file, nothing is slowing down here.
  • Cheers @sven so even if it's a 50Mb file, it only loads 1mb into memory per project?
  • SvenSven www.GSA-Online.de
    yes
Sign In or Register to comment.