How To Keep Track Of Total Scraped List For Gsa!

keith88

I'm sure this question is not real relevant to GSA but maybe someone can help!..

(Important) Is there a way you can only import new urls in GSA when you scrape?

For Instance....

so if you have 100 urls that i scraped and uploaded to gsa last week. I want to scrape again this week and only give gsa the new domains. So lets say I scraped 20 new domains I would have 120 but I need gscraper to pull out those new 20 and leave the master list of 120 to compare my next scrape.

So GSA wont run though duplicate domains wasting resources!

Is there a way I can do that?

jpvr90

Scrapebox can this but if your machine is low spec it will make it crash. There is also a pretty good software that can clean list called Once is Enough (free)

It can handle large list.

keith88

jpvr90   Do you know the feature in scrapebox?

mmtj

First you import the 120 URLs and then you use "Use Import URL List" -> "Select the URL list to compare on a domain level." and Import your old lists. Subsequently you will be left with the new domains that were not present in your old list.

Important: This is on a domain level - so if you also want inner page to stay present, then use "Select the URL List to compare" - then Scrapebox will compare the lists on a URL level.

JamPackedSpam

@mmtj, where is this option found? I don't see it in the global options.

donaldbeck

He's talking about scrapebox.

JamPackedSpam

Oh I gotcha, this is nice if you're lists are small enough but using anything too big takes way too long.

JamPackedSpam

Thanks

jpvr90

You can clean list at domain level and url level with scrapebox. I have never had problems with Scrapebox handling huge lists.

There are free tools that can do the same also.

JamPackedSpam

I see that GSA has a build in function called clean site lists. Does this do a global alive check or something? @jpvr90

Brandon

If you're actually talking about doing 100-5000 URLs, SER will process them very quickly. If you're talking about doing 500,000 then a better option is to use scrapebox to compare lists.

keith88

@Brandon ;  yea I will need SB... I do have some big boy lists!!

Brandon

SB is definitely worth the price. I've owned it for years and use it at least a few times per week. Not many programs you can say that about.