I've scrape millions and millions of url with Gscraper, what's next?

coin666

I got my gscraper running 24 hours and i got millions of dedup url, from varies gsa footprint and general keywords. I don't know what is the proper way to manage this: 

1. Import to GSA SER in Options - Advanced - Tools - Import URLs (Identify platform and sort in) or

2. Create project point to bing.com and import target urls

If i use method 1, while i have 50 projects running, it took months for me to finish 1 millions of url with my dedi 8GB ram 1Gbps line, and i got millions of urls every single day with gscraper.

@ron any good ideas how to proper manage this process? 

ron

#2 But make a bunch of those projects and split up the processing.

coin666

speedy and precise @ron , thanks so much! will do as what you suggest! 

fakenickahl

Unless you have a boner for blog and image comments, you might as well dedupe by domains as that'll save you a ton of time.

coin666

should i trim to root only dedupe domain? or just dedupe without trimming? @fakenickahl

fakenickahl

The best option would be to not trim in my opinion. SER will navigate to whatever is appropiate itself.

coin666

thanks for the tips @fakenickahl