Import 22million list can make gsa slower?

maulik2425

Guys i just scrape 22million list and i just upload that same list to 2 of my project and after that i restarted it but i feel like its became slower ..

so am i right ???

Can You Guys Suggest me the best option for that ..

Hope i got answer from some pro person who did this kind of things before.

Thanks

Trevor_Bandura

That's actually only 2 million not 22 million.

But yes there seems to be a problem with huge left target lists. I sent in a bug report last night to @Sven about this. Not sure if he will be able to see what the problem is with having big lists like that imported.

I have some projects with 10 million in the left target list.

maulik2425

Oh yes sorry for that 22million..

Ii also sending the bug report from last 2-3 days.

So this big list can create the problem right ?

randomusername

Duplicate your project 20 times, and split your list into 20 parts, then it works awesome

fakenickahl

I'm always importing lists in 2 million chunks and I'm not seeing any decrease in performance. I'm only running unique domains, so I'm curious if you're deduping by urls?

Ferryman

as @fakenickahl said - Remove duplicate domains done ? Remove duplicate URL's done? 

Brandon

Remove dupe domains and you'll be live with under 100k I would guess.

Xhat

Removing dupe domain isn't a bad idea ?

fakenickahl

@Xhat it only makes sense to dedupe blog and image comments by urls. Personally, I'm seperating my scrapes and doing only comments or all other platforms except comments. Sure I'm missing a good amount of the image and blog comments, but who cares? Processing lists will be a hell of a lot quicker.

Xhat

Hm I see personnally I didn't made any tiered linking strategy until now so I never used tier 3 but I'm going to now as ppl seems to have nice result with so I'll scrape blogs comment now.

GSAHKing

More threads the better cause GSA has to verify the platforms ect, "if you want the list done faster" then basically yes more threads.