When Importing URL's to project do you need to trim to root?

kymdog

When importing a list of scraped target URL's into a project (scrape box) do they need to be trimmed to root and then de-duped or just de-duped?
It would make sense to trim to root as there would be no dupe URL's but your assistance would be appreciated.

Bob

Sven

You do not need that but you can safely remove dupes of domain if you have certain engines unchecked (unchecked by default).

kymdog

Thank you Sven - So to he clear (I'm not the sharpest tool in the box) if I have

http://a.b.c/page1
http://a.b.c/page2
http://a.b.c/page3
http://a.b.c/page4 etc

If I trim to root and import just

http://a.b.c/

That's OK and the correct engine for the submission will be processed?

Sven

yes

kymdog

Thank you Sven much appreciated

donaldbeck

For the sake of completeness, you could post to:

1) 1 URL

2) 1 URL trimmed to last folder

3) Root domain

So:

1) http://www.reallygoodwebsite.com/folder/my-website-is-good

2) http://www.reallygoodwebsite.com/folder/

3) http://www.reallygoodwebsite.com/

This would be a quite thorough test for each domain.

Just food for thought.

kymdog

Thank you Donald.

DonCorleone

@donaldbeck

how do you trim from url to folder ?

donaldbeck

@DonCorleone

If you were using scrapebox, you would click Remove/Filter -> Trim URLs to last folder.

If you were using Gscraper you would click "Trim to last path" in the filtering options next to where you import URLs.