The right way to import verified site lists...

Satans_Apprentice

Here is my opinion, based on experience - do not use the import site list function. It works just fine, but it could cause issues later. Over time many links go bad, and they sit in your verified list until you clean them through one of several means in the SER software (Reverify, Cleanup, etc). 

Here is the problem with using the import site list function - if some of the links in the verified site list are bad (and they will be), you can't build a link to them. Furthermore, you can't purge them from your verified list since you never built a link with it, a requirement for any cleanup. Over time, your verified list will accumulate more bad links. If you delete projects, and those links go bad, they will pollute your verified list as well.  If you are running off of your Verified site list, bad links will kill your LPM. I get over 100 LPM with a clean list, and under 20 with a dirty list.

Here is my recommended way to import a sitelist:

1. Save the new sitelist to its own folder (duh)

2. In Global Options=>Advanced, point your "Submitted" folder via the dropdown menu at your new sitelist. Be sure the "submitted" box remains unchecked when you are done. Otherwise, you will pollute your nice new list.

3. Make sure the "verified" box is checked.

4. In your projects, click the checkbox "On" for options=>submitted (under the search engine selections). Identified, Verified and Failed should be unchecked.

5. Uncheck all of your search engines. You only want to run off of your list.

Start SER and run it until your  LPM collapses. You will be amazed at your LPM. When your LPM finally dies, go back and undo the steps above.

SER will read and build links from your submitted list (your new list), and save them to your standard verified list.

ron

+1 I hope people pay attention to this - it's brilliant.

Satans_Apprentice

As per Sven you can also select projects, right click, import targets=>from site lists. I am not in front of my computer, so you get the idea. Turn off your search engines, and the identified, failed, etc should all be unchecked..

gooner

Also, with the dreaded "important messages" you'll now get a "no more targets" message on your projects when your list is exhausted. So you'll know it's time to import a new list to submitted.

Nice one @satans_apprentice - Good method.

namdas

That's a neat solution to a problem thats been nagging me for a while - thank you

tsaimllc

I dont understand this at all.
Why dont you just import>sort in and identify then it removes bad links?

Satans_Apprentice

For 200,000 links?

donaldbeck

Nice post @Satans_Apprentice - and I liked your post about the dropbox stuff as well.

tsaimllc

200K is nothing to sort in/identify.. Ive done 2 mil before. it takes about 2 days, but it eliminates whatever problem you are having i think

Satans_Apprentice

The lists are already sorted. The trick is getting rid of the bad ones.

gooner

@tsaimllc - It's not a "problem you are having". It's a method to import lists in a way that improves LPM and total verified.

Today is my first day using this technique and i'll end up with 140k verified maybe, that would never be possible if i spent hours identifying and sorting the list first - That function is a waste of time IMHO.

tsaimllc

well i guess i still dont understand it
:-?

tommy99

"Submitted" folder this means the same as successful correct.

ron

@tsaimllc - While you were taking 2 days to sort and identify, the process described above was already making links out of the scrape, and then feeding all other projects with pure verified links. This process works, and you can hit unbelievable numbers of verified links in minutes.

I didn't understand the problem until the post from @Satans_Apprentice. I give it a +1 as the most important post I have read in a year on this forum (for those who really want to make a ton of links).

tsaimllc

okay after re reading it a few times and looking at the steps involved in My gsa, I understand the technique. But I still dont understand why he says "if some of the links in the verified site list are bad (and they will be), you can't build a link to them. Furthermore, you can't purge them from your verified list since you never built a link with it,"
if you identify and sort in, if the link is bad and it cant build a link there, then it will be disregarded, so the same result occurs, just slower. ( i can see that now)

ron

@tommy99 - The answer is yes. Think of those 4 sitelist areas (identified, successful, verified and failed) as generic 'ports'. You could use the 'failed' one if you wish. It makes no difference. You are assigning a function to an open port that has a name. You could use any of those ports to process sitelists, and you can use any of those ports to write your verified file. I obviously want verified to go into verified, but the other three are all equal in my eyes. 

2Take2

Another vote from me for this method, have been running a list that has been "cleaned" like this and the LPM is more than double what I would get from just running my "usual" verified list. >-

tsaimllc

Also, what is the difference between this drawn out method and just starting a project with import targets, wouldnt that do the same thing or no

gooner

Yes they would. But with this method all your projects share that list and you get the benefit of super high LPM.

And because your projects are not holding thousands of targets you get decreased memory usage, which means you can increase threads and get even higher LPM.

Satans_Apprentice

@tommy99: yes. In projects it says submitted. In global options it's successful. It's the same list. It's a typo by @sven. Just make sure that the only box you have checked in global options is "Verified". The check box in projects determines which lists SER will read to build links. The check box in global options determines which list SER will write to. It's input/output. If you aren't writing to a list, it's just a data source. Failed, identified, etc are just labels. @tsaimllc: we are dealing with pre-sorted verified site lists that have bad links in them. Yes, your method will work, but you have to shut down production to do it.

tommy99

Thanks Ron & Satans_Apprentice

fengli

I'm always looking for a way to clean my verified list! Every time I launch a new project with pull url from my verified list only and the "re-post" is disabled, the links generated are only 20% of my verified list.

So from my understanding,

1) I move my existing "verified" list out of GSA and saved in under other name like "to be verified again"
2) The "verified" list in GSA is empty now. Redirect the "submitted" port to my "to be verified again" list
3) "Submitted" is unchecked so that the list won't polluted when GSA running, and "Verified" is checked so that GSA can write data into it.
4) Create a project and set it to pull url from "Submitted" only.
5) Wait for it to finish and now I have a filtered verified list.

Correct me if I'm wrong. Thanks!

FengLi

gooner

@fengli - You can do that to clean your current verified list but of course you will be getting an ever decreasing number of good links every time you go through that process.

This system really works best if you are getting fresh verified lists from somewhere, either buying them or producing them yourself using a second installation.

How you do that is up to you but the morale to the story is old verified lists = low LPM.

I had a 500k unique domains verified lists, cleaned as far as SER could clean it and my LPM was around 70. I switched this for a new verified list of 30k and my LPM shot up to 280.

Satans_Apprentice

@fengli You have it perfect. Once you understand this concept, there are a lot of creative ways to use these lists.

genghiskhan

@Satans_Apprentice

I am also having a little problem to figure out the point and workflow of this procedure.

1. If i have understood correctly, we can put any (even and unverified) list in newly created folder for which will be used for submission, right?? If yes, is there any limitation how big that list can be, and how it should be named? And can we put more lists here in separate files as we let say scrape them from time to time??

2. When we finish with "list cleaning" (all links from file used) we then redo the whole procedure, but now with link lists which are placed in "verified" folder??

Thanks in advance!!

ldiaz117

I am sooo confused. I have a custom list in a custom folder on my computer. That folder contains all the text files I imported from a .SL file. On a specific project, how do i ONLY use that site list and NOT the global one???

Satans_Apprentice

1. Empty your "failed" list. 2. Using global options=>advanced=>tools Import the.sl to identified. 3. Make sure the only box checked in global options=>advanced is "verified". 4. In projects uncheck all of the search engines. Also underneath the search engines, the only box that should be checked is "identified" There are better ways to do this, but it will get the job done.

ldiaz117

"In projects uncheck all of the search engines. Also underneath the search engines, the only box that should be checked is "identified""

Do I uncheck them all or leave identified check? You say both.

ron

He is saying that in your Main Options, you 'uncheck' Identified. The only thing in Main Options that should be checked is Verified.

What this means: When you check a box for verified in Main Options, it means that all the links you build will be 'written' to whatever folder you check. You don't want two folders checked - ever. There is not a reason to 'write' out identified, submitted, or failed URLs. Absolutely pointless. I refer to those boxes (Identified, Successful, Verified and Failed) as 'ports'. My own term, but basically you use various ports on SER for processing - not to write information.

At the Project Level: Check only Identified. This means that at the project level, the only source of links will be the sitelist in Identified. Search engines are unchecked. You haven't imported any URLs directly. Therefore, the only place SER can find links to process is in the Identified folder.

Summary:

1) Understand that checking the Verified box in Main Options is simply an instruction to SER to write your verified to some file.

2) The other three 'ports' in Main Options can be used to set-up processing for Sitelists. You simply need to create an empty folder, and use SER to import the sitelist into that folder.

3) Once the new Sitelist is in the right folder, and you have the path correct in Identified (and unchecked), you are set up to process a sitelist.

4) To process that sitelist, turn off all search engines, make sure to clear target URL cache to remove the old queue of URLs, and check Identified in your sitelist to process all URLs in that sitelist.

Hope that helps. 

Satans_Apprentice

I made a mistake I thought I had corrected. "Failed" should be "identified". The box checked in projects needs to match the folder where you imported the .sl file

ldiaz117

Ok that is a little more detail which helps. To confirm:

Under Global Options...only thing checked is VERIFIED. The folder with my premium lists is in the IDENTIFIED section but that section is NOT checked.

Then under my project, i deselect all search engines but check IDENTIFIED so the project will check whatever is in that location.

Correct?