The REAL Search Engine Optimisation Thread

Ozz

Sorry for the title, no pun intended

This thread is all about search engines:

- How to add/fix SEs to GSA

- Recommendations of SEs in general

- Which SE to use for what purposes (scraping,  project searching)

- Results of tests (search results, speed,...)

I think that there are many misconceptions regarding the use of search engines. Too many user are just hitting the "use all english language" button and oversee that not all search engines give you good results for your purpose or might even slowing down GSA. Because of that I want to share my thoughts and experience about this topic.

First of all let me say that if you are using Google only you will miss a lot of targets. Google may be the best SE to use because it is fast, give good results and supports all advanced operators like "inurl:" or "intitle:". But many SEs support the most common operators too and give different but also good results for your search terms. This means that you will find more targets to post to altogether.

To get an idea what "advanced operators" are and which effect they have in your footprints, you should take a look at the Google Guide.

To know which operators are supported by GSA and the search engines you have to take a look into the "se.dat"-file you'll find in your main directory of GSA (...\Program Files (x86)\GSA Search Engine Ranker). You can open that file with a text editor of your choice and will see which SE is supported by which operator ("site=", "inurl="...).

Ozz

How to add a new search engine?

Adding a search engine to GSA is pretty easy if you know how to do it properly. 

To add a SE with a GUI of GSA you just have to click "Add"-button.

All of your added SEs are stored in user-SE-file called "user_se.dat" which you find in your appdata folder (...\AppData\Roaming\GSA Search Engine Ranker).

Because the GUI doesn't define the advanced operators by now you have to add them manually with an editor. If you have done it right the code looks something like this:

To illustrate how to add a SE by an example I've picked out a SE called mojeek which is not supported by GSA so far. Once you've opened the site you will notice that you can search "The Web" or "UK only". This means that we can add an "International" and an "UK" search engine but we have to figure out how to define them.

Many of the search sites have a link to "Advanced Search" on their page. If you find something like this you should open those links first. In mojeek the link to advanced search is: http://www.mojeek.co.uk/advanced.html

There are some nice feature this engine have. We can change "results per page" to 40 and display "no more than ... result per site" to 1.

Next step to take is to do a simple search for "vbulletin" and compare the results:

Page 1 (results 1 - 40): http://www.mojeek.co.uk/search?q=vbulletin&s=1&t=40&si=1

Page 2 (41 - 80): http://www.mojeek.co.uk/search?q=vbulletin&s=41&t=40&si=1

Page 3 (81-120): http://www.mojeek.co.uk/search?q=vbulletin&s=81&t=40&si=1

vbulletin = %search%

1/41/81 = %page%

40 = results per page

si=1 <-- the operator for only one domain per search query

&reg=uk <-- this operator is used for "UK only"

Once you figuered all out you can transfer it to the GUI.

The variables %search% and %page% are to used within the url of the SE.

"Links on page", "Start page" and "Increase" vary from SE to SE. Compare the urls of each search page (1,2,3,...) to know these variables.

"Ignore:" is used to filter ad sites and other links that are not usefull.

[mojeek]

desc=

country=International

url=http://www.mojeek.co.uk/search?q=%search%&amp;s=%page%&amp;t=40&amp;si=1

ignore=mojeek.co.uk

must_have=

links_on_page=40

start_page=1

inc_page=40

site=site:

[mojeek UK]

desc=

country=United Kingdom

url=http://www.mojeek.co.uk/search?q=%search%&amp;s=%page%&amp;t=40&amp;si=1&amp;reg=UK

ignore=mojeek.co.uk

must_have=

links_on_page=40

start_page=1

inc_page=40

site=site:

mojeek is a fast SE with good results I can recommend.

Sven

Wow excellent work again. And I didn't even tell you a single thing about how this all works. You figured all out yourself. Very nice! Anyway I keep this new search engines out for now to let people play with it.

Next release will also have improved search engines for China in case you care.

m3ownz

Great Post Ozz, i havent really used the search aspect of GSA SER apart from a few default campaigns i run when i have nothing else to do.

Now that you can save your selections for other projects, i think i will spend some time testing, and your posts are a fantastic start.

That tiny little 'no more than x results per domain' is an amazing little operator than i wish more SEs used. Do you know of any other SEs that use something similar?

I am also going to test some of the better ones (starting with your recommendations in the other thread) for IP ban thresholds. I bet there are plenty of decent SEs that dont have such a low tolerance as the big three.

s4nt0s

@ozz

Fantastic post. This is a great learning experience for me as well. I have no idea how you figured all that out by yourself as well as a lot of your other tutorials.

You're definitely on another level with this software.

Keep up the good work.

Ozz

Thanks to all of you.

>
I keep this new search engines out for now to let people play with it. 

No problem. Once we figured out which SE makes sense you can sort out the "bad apples" and add some new instead.

>That tiny little 'no more than x results per domain' is an amazing little operator than i wish more SEs used. Do you know of any other SEs that use something similar?

No, I've seen this the first time. This is a fantastic feature I like to see a lot more often.

>I am also going to test some of the better ones (starting with your recommendations in the other thread) for IP ban thresholds.

That would be fantastic. The more people do some tests, the better results we can give for recommendations.

About the "bad apples" I've mentioned. I suppose that some SE are useless and will only block valuable rescources of GSA. I'll give you an example. SE like yandex.com (not .ru!) or the excellent duckduckgo.com (which I use in private along with ixquick) don't use "pages". So GSA only read the first page and get only 10 results. Some SEs have slow servers or some other issues.

(As a side note: I've sent Sven some of these "bad apples" like yandex.com )

Because of that I think that if would benefit for everyone if we are doing some tests with the "Search online for URLs"-Tool to get an idea which SE are usefull and which are not.

I did some research about the international SEs a few weeks ago with the "search online for url"-tool and keywords "powered by vbulletin" and"leave a comment" website. Not all results are accurate but thats a good start, I think.

Here are my results without dupes (KW1|KW2):

-google (453|394) -> very, very fast

-msn/bing (145|17) -> wtf??. good speed though 

-yahoo (855|825) -> very fast

-hotbot (850|807) -> moderate speed

-ecosia (849|792) -> very fast

-info (677|607) -> moderate speed

-lexxe (573|454) -> moderate speed

-metacrawler (416|416) -> moderate speed

-sky (406|305) -> moderate speed

-euroseek (430|287*) -> slow. *idle mode after some time (aborted)

-exalead (94*|599) -> moderate speed. *idle mode after some time (aborted)

-mamma (241|209) -> moderate speed

-ask (192*|129*) -> *idle mode (aborted)

crappy results: search, searchhippo, jayde

only page 1 results: blekko, dogpile, exite, gigablast, ixquick, scrirus, yandex

not working: aol, charter, clusty, lycos, scrubtheweb, teoma, thunderstone, verizon, volunia

Sven

added the mentioned search engines to upcoming version

Ozz

[google ME]

country=Montenegro

url=http://www.google.me/search?q=%search%&amp;as_qdr=all&amp;filter=0&amp;num=100&amp;start=%page%

links_on_page=100

start_page=0

inc_page=100

enabled=1

ignore=&oi=translate&|*&h=*&prev=*&zoom=1&q=*|*&sa=X&ei=*&usg=*|atdmt.com|google.|googleusercontent.|googleusercontent.|youtube.

site=site:

inurl=inurl:

intitle=intitle:

link=link:

Ozz

I did another "Identifier"-test with search term "powered by vbulletin". As a result I use this selection of SEs. The results might differ from someone else when tested, because of proxies, SE query time or whatever, but I feel that this is a good selection for most.

Download: http://mediafire.com/?ju4ub6ajjsd7rkb (once dowloaded -> right click SEs in GSA project -> load selection)

Included are 37 SEs from these countries: AU, CA, IE, INT, NZ, UK and US.

With this SEs: Google, MSN, Yahoo, Ecosia, Exite, Hotbot, Info, Lexxe, Metacrawler, Sky

Possible additions to this selection: 

mediocre results - aol, ask, euroseek, mamma, teoma

not tested - mojeek (uk/int), zapmeta (uk/us), izito (uk/us)

Sven

Nice Thanks for this analysis.

AlexR

@Ozz- Thanks! Will try those out. 

oil

i am going thru that now, just a little heads up thou, mojeek doesnt support the advanced search anymore, i ll get 403 error with those queries, so please double check yourself, but it seems the mojeek advanced SE in SER doesnt work anymore since they dont support those params anymore

Sven

correct settings for them (fixed in next release as well)

[mojeek]

desc=fast SE with good results
country=International
url=http://www.mojeek.co.uk/search?q=%search%&amp;s=%page%&amp;si=1
ignore=mojeek.|w3.org
links_on_page=10
start_page=1
inc_page=10
site=site:

[mojeek UK]
desc=fast SE with good results
country=United Kingdom
url=http://www.mojeek.co.uk/search?q=%search%&amp;s=%page%&amp;si=1&amp;reg=UK
ignore=mojeek.|w3.org
links_on_page=10
start_page=1
inc_page=10
site=site:

Username

So which are the best SEs to use? Ozz above tested all the International SEs so I thought maybe I should use those. Then a few posts down he said Google Montenegro is also good. Do I select just all the Google ones or what? Does anyone have an updated list?

EDIT: Oops, sorry! I missed the mediafire link as it's not a clickable link but just plain text! Thanks, Ozz for doing the Lord's work! Please let us know if you have updated your SE list again...

Ozz

You could do the tests also. Why not give us an update about your results?

davbel

@Ozz - thanks for the SE list - Made a decent difference to speed :-)

eLeSlash

But when i look in the GSA there are 1000 search engines. I allways select all of them. Shouldnt i? I also added the 37 search engines but i didnt saw them added....

Sven

selecting all search engines is not very clever. A lot are dependent on special countries or languages. And if you e.g. have only English keywords in your project, what sense does it make to use e.g. German search engines. 

eLeSlash

Isnt GSA searches for new target websites to post to? If it finds a new URL that i can post to why does it matter if it was founded through chinesse search engine?
I want to get as many different results as i can

s4nt0s

@eLeSlash - Because some people are trying to post to websites that are hosted in a different country or that are in a different language. If someone has a Chinese website and wants to build Chinese backlinks, selecting search engines that handle that language might yield better results for them. Especially if they're using Chinese keywords.

eLeSlash

Yes Santos, i am aware of that, great feature if you want to get ranked on a targeted country much faster, but i would want to build backlinks wherever i can, i dont really care if i get backlinks from chinesse websites for a .com website that i want to promote, a backlink is still a backlink on google's eyes.
That is why i am so proud of GSA SER, it supports creating backlinks on alot of different languages websites, that is really great, no other tool is coded that good

Ozz

You should at least filter which SEs you want to use as not all giving much results at all. Some of them give you only 10 results per keyword and that won't help you much and just wasting rescources.

Test them with the "search online for url"-tool in options --> advanced --> tools with a simple keyword to get an idea if a SE might be useful or not.

eLeSlash

Thank you Ozz this make perfect sense, but i dont have the time to test and filter the 1k SEness, i will continue checking them all and search from them all... dunno what else to do

LeeG

Something else to watch out for is the "already parsed" message

Its normally a sign of using too many clone engines

Any engine you use, check their details and see if they are using the results of the big players, ie google or bing. There are a few independent engines out there. Baidu from China and Yandex from Russia

An easy way to find the independent engines is to check your own website traffic logs and see who or what hits on a regular basis

KayKay

found some nice chart of the best 15 search engines with some nice stats

http://www.ebizmba.com/articles/search-engines

KayKay

btw @Ozz and @Sven is there a way to search without footprints ?

i wanted to add another search engine (seems a google clone so or soo but..)

now i did :

[SearchingQ : Added]
country=international
url=http://www.searchinq.com/web?page=%page%&amp;q=%search%
links_on_page=10
start_page=1
inc_page=100
enabled=1
ignore=http://www.searchinq.com
site=site:

but when i check what SER does it takes the footprints with the search and produces following url:

http://www.searchinq.com/web?q="Powered+"Attached+Image:+"Attached+Image:+"++by+Blogtronix"+test&amp;page=1

..whithout the footprints i get proper results :

http://www.searchinq.com/web?q=test&page=1


also i noticed .. after useing (Active S) when you see alot of [001/001] and you check the link aka click on it ...it often means you are blocked like on yandex or ecosia

Sven

I will add that search engine on next release. You seem to have forgotten to make &amp; to & only.

Ozz

"inc_page" seems not right to me also

Sven

and this engine did not support quotes. I have extended the configuration for it in next version with query_remove=" so it works also with that engine.

KayKay

@Ozz ups yep inc page i set to 10 .. great @Sven thx

GeeSA

All the ecosia.org search engines are not working for me anymore, would you please check on your end if this is the same for you? This is a good bing clone.