Skip to content

optimizing settings for Import URLs

TheStigTheStig
edited February 2013 in Need Help

Can anything be done to help speed up the Import URLs process?

I have a rather large scraped list of ~20M urls, so far it has processed ~30k in one hour.

Comments

  • OzzOzz
    edited February 2013
    more threads.

    however, you could just import those urls directly to your projects or split them up for different projects and import to those.
    doing that will safe you much time as you post directly to the links of your list and sort them in the lists at the same time.
  • TankTank
    You should probably start by removing most duplicate sites, because with a list that big, I assume most are duplicate.
  • TheStigTheStig
    thanks Tank, I should have thought of that before importing them all. the logs are producing tons of "already parsed" messages.
Sign In or Register to comment.