How To Keep Track Of Total Scraped List For Gsa!

keith88keith88
in Other / Mixed
I'm sure this question is not real relevant to GSA but maybe someone can help!..

(Important) Is there a way you can only import new urls in GSA when you scrape?

For Instance....

so if you have 100 urls that i scraped and uploaded to gsa last week. I want to scrape again this week and only give gsa the new domains. So lets say I scraped 20 new domains I would have 120 but I need gscraper to pull out those new 20 and leave the master list of 120 to compare my next scrape.

So GSA wont run though duplicate domains wasting resources!

Is there a way I can do that?
Tagged:

Comments

  • jpvr90jpvr90
    edited December 2013
    Scrapebox can this but if your machine is low spec it will make it crash. There is also a pretty good software that can clean list called Once is Enough (free)

    It can handle large list.
  • keith88keith88
    jpvr90   Do you know the feature in scrapebox?
  • mmtjmmtj
    edited December 2013
    First you import the 120 URLs and then you use "Use Import URL List" -> "Select the URL list to compare on a domain level." and Import your old lists. Subsequently you will be left with the new domains that were not present in your old list.

    Important: This is on a domain level - so if you also want inner page to stay present, then use "Select the URL List to compare" - then Scrapebox will compare the lists on a URL level.
  • JamPackedSpamJamPackedSpam
    @mmtj, where is this option found? I don't see it in the global options.
  • donaldbeckdonaldbeck Advanced SER Videos -> http://bit.ly/1ySrbwu | Learn SER For Free Step By Step -> http://sertips.com
    He's talking about scrapebox.
  • JamPackedSpamJamPackedSpam
    Oh I gotcha, this is nice if you're lists are small enough but using anything too big takes way too long.
  • JamPackedSpamJamPackedSpam
    Thanks
  • jpvr90jpvr90
    You can clean list at domain level and url level with scrapebox. I have never had problems with Scrapebox handling huge lists.

    There are free tools that can do the same also.
  • JamPackedSpamJamPackedSpam
    I see that GSA has a build in function called clean site lists. Does this do a global alive check or something? @jpvr90
  • BrandonBrandon Reputation Management Pro
    If you're actually talking about doing 100-5000 URLs, SER will process them very quickly. If you're talking about doing 500,000 then a better option is to use scrapebox to compare lists.
  • keith88keith88
    @Brandon ;  yea I will need SB... I do have some big boy lists!!
  • BrandonBrandon Reputation Management Pro
    SB is definitely worth the price. I've owned it for years and use it at least a few times per week. Not many programs you can say that about.
Sign In or Register to comment.