2) I am asking for a system to more correctly allocate lists so we don't have to reparse everything. Currently many users use the "Clear Target URL History" - but this just means it redoes ALL the searching again and again...what a resource waste! Surely it would be much better to have a failed list or submitted but not verified list with the reason per project. This way, we can just focus on making sure we get the links that we've spent so many resources finding, rather than just resetting and reparsing everything! I think this is a MASSIVE shortfall in GSA, that can hopefully be remedied...by giving us a better management of data.
Clearing the duplicate url list from the sitelist can help speed things up.
I killed over 1.5 million duplicates the other day while trying to find a cure for an intermittent fault on gsa at present, which only a handful of people are experiencing
Something Im going to start doing each time I reboot my vps or do a gsa update from now on.
Be nice to have an option to do this automatically ( a lazy person button )
At the same time, you can free up a lot of memory
The only time I personally clear the target url history is when Sven makes big changes to the engines that you post to or there is a big batch of new sites added, or if I have altered or added my captchas to CSX. My theory being (it might well be wrong) first hit gsa might have been configured differently, going back again you might get a link
Search engines, its your choice. Do your research, I know Ron has been in touch about this and has found what I was saying after a bit of research. He was left bewildered about my own approach to choosing engines. Nice name, sounds cool, good page layout etc
Joking apart. If you ever use a rank checker and choose multiple googles for example. Google us, google uk etc, you will find most of your rank checking is about one or two places apart. Thank to Ron for the next bit. Bing powers Yahoo, Lycos, Ecosia, HotBot. Google powers => Ask, Startpage, info.com
Which gives some idea about using two engines instead of nine. There are a lot more.
If you choose all English engines, think how many times you will be seeing the same results repeated time and time again. Even by just using the UK engines, you will cut a lot of the repeat results out. Which gives the end result of more time posting, less time sorting through sites already parsed
I never give straight answers as you notice, only pointers as to what to look into.
That way, you make the changes, and feel the benefits of doing the research
I will add that I cut a number of redundant search engines, and my efficiency improved (more linkbuilding).
But the biggest change which just blew me away is my captcha count was cut by over 50%. Even though I only use CSX, I found that very, very interesting.
Well...Bing powers Yahoo, right? And Google powers Startpage, right? There are many, many more redundacies. You should go to these search engines and put in a few search terms, side by side. You will see what I mean.
Captcha retries = 0. You can thank LeeG for that one .
I was just winding some bloke called Santos up on his sales thread on BHW
Last heard gibbering the word "insane"
Someone over there pmed me about my keywords list.
This is the trick to a good way of building your own keywords
Everyone's keywords will be niche specific
I spent a day scraping mine
The trick is to use a niche finder program. Scrape keywords for your niche Delete the duplicates and scrape with the new bigger list Keep doing that until you get 10k to 20k keywords Mines about 22k keywords for my niche
Did you ever try to sort your keyword list and spin them accordingly?
Example:
Best Football Club
Awesome Soccer Club
Awesome Football Team
Best Soccer Team
{{Best|Awesome} {Soccer|Football} {Team|Club}}
I know you like to tweak a lot so this could be another way to safe RAM as SER don't need to cache that many characters (= less memory usage). Of course there is no way to do this for all 20k keywords, but it might be worth to dig into this and save some rescources.
I had to step out and run some errands, so let me elaborate on a few things regarding the whole search engine thing.
Redundancy Issues - One of the things you need to figure out is which search engines power which search engines. A lot of time it is very easy to tell because it will say "Enhanced by Google" or something like that. But the biggest reveal is to open up Bing and Google side by side, together with a new search engine. Use the same keyword phrase across all and hit search. You will definitely see who powers what. Aside from what LeeG and I mentioned above, there is a company called Infospace that powers the metacrawlers/compilers like Dogpile.com, Metacrawler.com, Excite and 100 others. So you will get the same results in these SE's.
Why Is This Important? - You are literally crawling the same results (with maybe a few websites as an exception). You are literally bogging down GSA SER with the task of going through the same targets twice. That's insane. For example, if you only used two search engines, Bing US and Yahoo US, and fed it the same keyword list, you would have achieved 50% inefficiency in using GSA SER.
Captcha - I trimmed redundant search engines, and my captcha calls were literally cut in half. I can give you an example: At this moment I have 14,895 links submitted, and 3891 captcha calls. Check your stats and see if you have this type of ratio.
Before I ventured into this whole thing, @LeeG inspired me to cut captcha retries on CSX from 6 to 0. One of the smartest things I ever did. I must have eliminated 70% of all my captcha calls - and the number of links increased by 20%! Why? Because instead of retrying a captcha six times at a rate of 1-2 seconds each attempt, GSA SER instead found new targets to post links. Seriously, on all those retries, how many correct solves do you really think occurred? I say probably 5 - 10% AT BEST.
Which Search Engines Should I Use? - There is no right answer for everyone. We are all from different countries, and we are all marketing to different countries. As a guy from the US, I will focus on the biggest English speaking markets, like the US, UK, Canada, Australia, New Zealand, and South Africa. If I was in Germany, I think I would want to make sure I had Austria, Belgium, the Netherlands, Switzerland and a few others.
But does that mean I should only use search engines from those countries? No, at least I don't think so. I believe you should have your 'country pack' thoroughly covered by as many different and unique search engines that those countries have available, and you should have international search engines looking for additional targets. Will I target Africa and Asia? Probably not. But I do have an entire planet where I can selectively pick countries to diversify my linkbuilding.
Keywords -@LeeG is giving you some very good advice. If you have scrapebox, you just keep rerunning the keywords through that scraper, eliminating duplicates until you have a big list. Myself, I typically just go with 1000 and change them up once every month or two. Either way, you'll find plenty of targets. What @Ozz said above has a lot of merit as you will encumber the memory with a very large list, but it is worth testing to see if there is a noticeable difference. I haven't tested it yet, but it makes a lot of sense.
Search Engines: You need to know that the SEs with the most results per page are the fastest. Google has 100 results/page, Bing 50/page. Every SE with 10/page will be very slow compared to Google and Bing. Some sites like ecosia.org have 20 results/page but on good servers and have good speed also. As ecosia is powered by bing it may give you the same results as Bing, but could be usefull when IPs are banned for searching.
Captchas: Simple math to get a feeling for percentages (hopefully i don't embarrass myself not with that, haha).
But the numbers LeeG are showing that it should be better to post to another target instead of waiting for another captcha (+ solving time). To delete target history URLs from time to time will give you another shot to post to missed sites.
There are other ways of finding which search engines can give a diverse set of results and not test with gsa
A free rank tracker like Traffic Travis that on the free version checks the top 100 positions. If you have a keyword or two, you rank for on google and know your position, ie pgae 2.4 3.7 etc
Check as many engines as your allowed and track that keyword
See which ones give the most diverse selection of results and also the ones that give the same repedative positions
@Ozz that is interesting on how you are calculating the captchas. Mathematically you are correct. But what I am seeing when I watch the captcha screen is captchas appear that are not solvable - meaning there is no template for that captcha. And I see a lot of those. They will have a (100% failure rate x 6). I do agree on the retries where there is a known platform.
Yeah, this thing with @LeeG really taught me something. I am experiencing much better link building results by setting retries=0.
@LeeG - One thing @Ozz brought up to me that is brilliant is to use scrapebox to scrape Bing for a keyword, and scrape Yahoo for the same keyword. Merge the results, eliminate duplicates and see what you have. In the case of Bing and Yahoo you will see almost the same results.
A lot of these search engines that are powered by Bing and Google overlay a tiny algorithm on top of the host engine search results, and more than anything, it just slightly re-orders the results, but it doesn't seem to provide new unique results.
@ron: you are right with the "6 x 0 %" of course, but I bet there will be a solution to that in the near future which can handle the low percentage captchas in different way (a 2nd captcha service may be needed though)
yeah, I have been following this thread with great interest and have been learning a lot.
I only ask about the proxies as I can't really tell the difference between dedicated and semi dedicated.
I have just started using proxy hub (following your recommendation) but can't get them to work with scrapebox rank checker for some reason. Will probably have to revert to buyproxies again as their worked in all 3 progs I need.
Thanks again for all the great info. Still trying to decipher what search engines you use from your cryptic clues a few posts above!
Up until a day or two ago, all I was using was the 13 UK search engines
That could and should be cut back in all honesty.
Ron has done a lot of detective work along with Ozz on the engines which are clones and which give maximum results per page
So if you use the UK engines that are listed, you can cut back even more on what gives cloned results.
So if you select bing, you dont need yahoo ect
Once you have a good list of engines, you can then right click on the engines window and export the list and then do the same on your other projects and import them.
Its time saving on getting all projects cloned in certain areas
It is very dry British humor. It has taken me hours to recover in my penthouse.
Now that I took the time to sort through the search engines, it really has made a difference. I am getting near 30,000 submitted in a day.
But...I am using SER straight up without feeding it any lists. It would be nice to compare to others how many submitted they are doing without feeding SER any scraped lists. Of course, feeding it scraped lists will squeeze a bunch more productivity out of it.
Sven gave us a lot of food with SER. Basically a whole buffet table. And I think we were choking trying to down all the food. It's up to us to figure out the proper helping.
@GG - I understand what you are saying, but I honestly don't know how SER sequences its tasks. I just don't know enough of what happens behind the curtain as I am not a programmer.
But one thing you mentioned in your write-up still bothers me. It's the whole OBL and PR check thing. I think these features are just killing people's efficiency in linkbuilding.
Again, in this new era since Penguin, I have found it incredibly easy to rank using GSA with no OBL and PR filters. And if I really did care about that stuff (which I did prior to Penguin), I would set up a separate project so I wouldn't choke my productivity on the 95% of linkbuilding that I need to get done.
@ron what about your tier 1's do you filter those?
And just out of curiosity, did any of you keywords for your site drop a few positions? I've been doing almost the same method you are doing and it seems I lost a few positions with the update google processed 12/11/2013.
No filters on moneysite links or tier links. I am just very choosy on what I link to my moneysites, and I make sure they are built at an appropriate velocity given the age of the site.
I didn't lose any positions on my important keywords, but did see some small movements on some secondary keywords, but no big shakes.
Comments
Clearing the duplicate url list from the sitelist can help speed things up.
I killed over 1.5 million duplicates the other day while trying to find a cure for an intermittent fault on gsa at present, which only a handful of people are experiencing
Something Im going to start doing each time I reboot my vps or do a gsa update from now on.
Be nice to have an option to do this automatically ( a lazy person button )
At the same time, you can free up a lot of memory
The only time I personally clear the target url history is when Sven makes big changes to the engines that you post to or there is a big batch of new sites added, or if I have altered or added my captchas to CSX. My theory being (it might well be wrong) first hit gsa might have been configured differently, going back again you might get a link
Search engines, its your choice. Do your research, I know Ron has been in touch about this and has found what I was saying after a bit of research. He was left bewildered about my own approach to choosing engines. Nice name, sounds cool, good page layout etc
Joking apart. If you ever use a rank checker and choose multiple googles for example. Google us, google uk etc, you will find most of your rank checking is about one or two places apart. Thank to Ron for the next bit. Bing powers Yahoo, Lycos, Ecosia, HotBot. Google powers => Ask, Startpage, info.com
Which gives some idea about using two engines instead of nine. There are a lot more.
If you choose all English engines, think how many times you will be seeing the same results repeated time and time again. Even by just using the UK engines, you will cut a lot of the repeat results out. Which gives the end result of more time posting, less time sorting through sites already parsed
I never give straight answers as you notice, only pointers as to what to look into.
That way, you make the changes, and feel the benefits of doing the research
I will add that I cut a number of redundant search engines, and my efficiency improved (more linkbuilding).
But the biggest change which just blew me away is my captcha count was cut by over 50%. Even though I only use CSX, I found that very, very interesting.
which search engines are you using ? the Ozz recommended 37 search engines ?
did you set your retries to 0 ?
So your getting less captcha calls and more submissions
Not a bad result
Wonk be long until your a regular member of the 40K club Ron
Well...Bing powers Yahoo, right? And Google powers Startpage, right? There are many, many more redundacies. You should go to these search engines and put in a few search terms, side by side. You will see what I mean.
Captcha retries = 0. You can thank LeeG for that one
.
I was just winding some bloke called Santos up on his sales thread on BHW
Last heard gibbering the word "insane"
Someone over there pmed me about my keywords list.
This is the trick to a good way of building your own keywords
Everyone's keywords will be niche specific
I spent a day scraping mine
The trick is to use a niche finder program.
Scrape keywords for your niche
Delete the duplicates and scrape with the new bigger list
Keep doing that until you get 10k to 20k keywords
Mines about 22k keywords for my niche
I had to step out and run some errands, so let me elaborate on a few things regarding the whole search engine thing.
Redundancy Issues - One of the things you need to figure out is which search engines power which search engines. A lot of time it is very easy to tell because it will say "Enhanced by Google" or something like that. But the biggest reveal is to open up Bing and Google side by side, together with a new search engine. Use the same keyword phrase across all and hit search. You will definitely see who powers what. Aside from what LeeG and I mentioned above, there is a company called Infospace that powers the metacrawlers/compilers like Dogpile.com, Metacrawler.com, Excite and 100 others. So you will get the same results in these SE's.
Why Is This Important? - You are literally crawling the same results (with maybe a few websites as an exception). You are literally bogging down GSA SER with the task of going through the same targets twice. That's insane. For example, if you only used two search engines, Bing US and Yahoo US, and fed it the same keyword list, you would have achieved 50% inefficiency in using GSA SER.
Captcha - I trimmed redundant search engines, and my captcha calls were literally cut in half. I can give you an example: At this moment I have 14,895 links submitted, and 3891 captcha calls. Check your stats and see if you have this type of ratio.
Before I ventured into this whole thing, @LeeG inspired me to cut captcha retries on CSX from 6 to 0. One of the smartest things I ever did. I must have eliminated 70% of all my captcha calls - and the number of links increased by 20%! Why? Because instead of retrying a captcha six times at a rate of 1-2 seconds each attempt, GSA SER instead found new targets to post links. Seriously, on all those retries, how many correct solves do you really think occurred? I say probably 5 - 10% AT BEST.
Which Search Engines Should I Use? - There is no right answer for everyone. We are all from different countries, and we are all marketing to different countries. As a guy from the US, I will focus on the biggest English speaking markets, like the US, UK, Canada, Australia, New Zealand, and South Africa. If I was in Germany, I think I would want to make sure I had Austria, Belgium, the Netherlands, Switzerland and a few others.
But does that mean I should only use search engines from those countries? No, at least I don't think so. I believe you should have your 'country pack' thoroughly covered by as many different and unique search engines that those countries have available, and you should have international search engines looking for additional targets. Will I target Africa and Asia? Probably not. But I do have an entire planet where I can selectively pick countries to diversify my linkbuilding.
Keywords - @LeeG is giving you some very good advice. If you have scrapebox, you just keep rerunning the keywords through that scraper, eliminating duplicates until you have a big list. Myself, I typically just go with 1000 and change them up once every month or two. Either way, you'll find plenty of targets. What @Ozz said above has a lot of merit as you will encumber the memory with a very large list, but it is worth testing to see if there is a noticeable difference. I haven't tested it yet, but it makes a lot of sense.
There are other ways of finding which search engines can give a diverse set of results and not test with gsa
A free rank tracker like Traffic Travis that on the free version checks the top 100 positions. If you have a keyword or two, you rank for on google and know your position, ie pgae 2.4 3.7 etc
Check as many engines as your allowed and track that keyword
See which ones give the most diverse selection of results and also the ones that give the same repedative positions
@Ozz that is interesting on how you are calculating the captchas. Mathematically you are correct. But what I am seeing when I watch the captcha screen is captchas appear that are not solvable - meaning there is no template for that captcha. And I see a lot of those. They will have a (100% failure rate x 6). I do agree on the retries where there is a known platform.
Yeah, this thing with @LeeG really taught me something. I am experiencing much better link building results by setting retries=0.
@LeeG - One thing @Ozz brought up to me that is brilliant is to use scrapebox to scrape Bing for a keyword, and scrape Yahoo for the same keyword. Merge the results, eliminate duplicates and see what you have. In the case of Bing and Yahoo you will see almost the same results.
A lot of these search engines that are powered by Bing and Google overlay a tiny algorithm on top of the host engine search results, and more than anything, it just slightly re-orders the results, but it doesn't seem to provide new unique results.
Ozz, you can soon go off people that are playing with Svens new toy :P
Especially if all they do is tell you how good it is
Another speed boost is the blacklist checking
Enabled that yesterday and forgot about it
Noticed my mojo had gone missing today, thought my proxies were being over shared. It slowed that much to just over a 2k an hour average
Just disabled it and the speed is back to almost no proxie speed
Something that needs more looking into to get a good balance of reliable blacklist checking along with speed
No tried dedicated proxies in all honesty
I use proxy hub. If you search on google you can find discount codes for them. Anywhere between 10% and 15% discounts
I use 40 shared, which costs a few $ more than thirty from the last company I used
A lot of tricks I use for making GSA run faster, is reducing quiries, taking out the middle man etc
One quiry less on a searching and posting, over the course of a day can soon add up
My niche is UK related.
Up until a day or two ago, all I was using was the 13 UK search engines
That could and should be cut back in all honesty.
Ron has done a lot of detective work along with Ozz on the engines which are clones and which give maximum results per page
So if you use the UK engines that are listed, you can cut back even more on what gives cloned results.
So if you select bing, you dont need yahoo ect
Once you have a good list of engines, you can then right click on the engines window and export the list and then do the same on your other projects and import them.
Its time saving on getting all projects cloned in certain areas
Ron was even more bewildered
Its just my dry British humor taking unwitting victims at times
When you read through the tips, add the ideas one at a time.
Get one working right, then onto the next rather than trying to do everything at once.
More time spent tweaking, can produce better results in the long run
It is very dry British humor. It has taken me hours to recover in my penthouse.
Now that I took the time to sort through the search engines, it really has made a difference. I am getting near 30,000 submitted in a day.
But...I am using SER straight up without feeding it any lists. It would be nice to compare to others how many submitted they are doing without feeding SER any scraped lists. Of course, feeding it scraped lists will squeeze a bunch more productivity out of it.
Sven gave us a lot of food with SER. Basically a whole buffet table. And I think we were choking trying to down all the food. It's up to us to figure out the proper helping.
@GG - I understand what you are saying, but I honestly don't know how SER sequences its tasks. I just don't know enough of what happens behind the curtain as I am not a programmer.
But one thing you mentioned in your write-up still bothers me. It's the whole OBL and PR check thing. I think these features are just killing people's efficiency in linkbuilding.
Again, in this new era since Penguin, I have found it incredibly easy to rank using GSA with no OBL and PR filters. And if I really did care about that stuff (which I did prior to Penguin), I would set up a separate project so I wouldn't choke my productivity on the 95% of linkbuilding that I need to get done.
No filters on moneysite links or tier links. I am just very choosy on what I link to my moneysites, and I make sure they are built at an appropriate velocity given the age of the site.
I didn't lose any positions on my important keywords, but did see some small movements on some secondary keywords, but no big shakes.