The REAL Search Engine Optimisation Thread

OzzOzz
edited August 2012 in Other / Mixed
Sorry for the title, no pun intended ;)

This thread is all about search engines:
- How to add/fix SEs to GSA
- Recommendations of SEs in general
- Which SE to use for what purposes (scraping,  project searching)
- Results of tests (search results, speed,...)

I think that there are many misconceptions regarding the use of search engines. Too many user are just hitting the "use all english language" button and oversee that not all search engines give you good results for your purpose or might even slowing down GSA. Because of that I want to share my thoughts and experience about this topic.

First of all let me say that if you are using Google only you will miss a lot of targets. Google may be the best SE to use because it is fast, give good results and supports all advanced operators like "inurl:" or "intitle:". But many SEs support the most common operators too and give different but also good results for your search terms. This means that you will find more targets to post to altogether.

To get an idea what "advanced operators" are and which effect they have in your footprints, you should take a look at the Google Guide.
To know which operators are supported by GSA and the search engines you have to take a look into the "se.dat"-file you'll find in your main directory of GSA (...\Program Files (x86)\GSA Search Engine Ranker). You can open that file with a text editor of your choice and will see which SE is supported by which operator ("site=", "inurl="...).
«1

Comments

  • OzzOzz
    edited August 2012
    How to add a new search engine?

    Adding a search engine to GSA is pretty easy if you know how to do it properly. 

    To add a SE with a GUI of GSA you just have to click "Add"-button.
    image

    All of your added SEs are stored in user-SE-file called "user_se.dat" which you find in your appdata folder (...\AppData\Roaming\GSA Search Engine Ranker).

    Because the GUI doesn't define the advanced operators by now you have to add them manually with an editor. If you have done it right the code looks something like this:
    image

    To illustrate how to add a SE by an example I've picked out a SE called mojeek which is not supported by GSA so far. Once you've opened the site you will notice that you can search "The Web" or "UK only". This means that we can add an "International" and an "UK" search engine but we have to figure out how to define them.

    Many of the search sites have a link to "Advanced Search" on their page. If you find something like this you should open those links first. In mojeek the link to advanced search is: http://www.mojeek.co.uk/advanced.html
    There are some nice feature this engine have. We can change "results per page" to 40 and display "no more than ... result per site" to 1.

    Next step to take is to do a simple search for "vbulletin" and compare the results:

    vbulletin = %search%
    1/41/81 = %page%
    40 = results per page
    si=1 <-- the operator for only one domain per search query
    &reg=uk <-- this operator is used for "UK only"

    Once you figuered all out you can transfer it to the GUI.
    image

    The variables %search% and %page% are to used within the url of the SE.
    "Links on page", "Start page" and "Increase" vary from SE to SE. Compare the urls of each search page (1,2,3,...) to know these variables.

    "Ignore:" is used to filter ad sites and other links that are not usefull.

    [mojeek]
    desc=
    country=International
    ignore=mojeek.co.uk
    must_have=
    links_on_page=40
    start_page=1
    inc_page=40
    site=site:

    [mojeek UK]
    desc=
    country=United Kingdom
    ignore=mojeek.co.uk
    must_have=
    links_on_page=40
    start_page=1
    inc_page=40
    site=site:

    mojeek is a fast SE with good results I can recommend.
  • SvenSven www.GSA-Online.de

    Wow excellent work again. And I didn't even tell you a single thing about how this all works. You figured all out yourself. Very nice! Anyway I keep this new search engines out for now to let people play with it.

    Next release will also have improved search engines for China in case you care.

  • Great Post Ozz, i havent really used the search aspect of GSA SER apart from a few default campaigns i run when i have nothing else to do.

    Now that you can save your selections for other projects, i think i will spend some time testing, and your posts are a fantastic start.

    That tiny little 'no more than x results per domain' is an amazing little operator than i wish more SEs used. Do you know of any other SEs that use something similar?

    I am also going to test some of the better ones (starting with your recommendations in the other thread) for IP ban thresholds. I bet there are plenty of decent SEs that dont have such a low tolerance as the big three.
  • s4nt0ss4nt0s Houston, Texas
    @ozz

    Fantastic post. This is a great learning experience for me as well. I have no idea how you figured all that out by yourself as well as a lot of your other tutorials.

    You're definitely on another level with this software.

    Keep up the good work. :)
  • Thanks to all of you.

    >
    I keep this new search engines out for now to let people play with it.
     
    No problem. Once we figured out which SE makes sense you can sort out the "bad apples" and add some new instead.

    >That tiny little 'no more than x results per domain' is an amazing little operator than i wish more SEs used. Do you know of any other SEs that use something similar?
    No, I've seen this the first time. This is a fantastic feature I like to see a lot more often.

    >I am also going to test some of the better ones (starting with your recommendations in the other thread) for IP ban thresholds.
    That would be fantastic. The more people do some tests, the better results we can give for recommendations.

    About the "bad apples" I've mentioned. I suppose that some SE are useless and will only block valuable rescources of GSA. I'll give you an example. SE like yandex.com (not .ru!) or the excellent duckduckgo.com (which I use in private along with ixquick) don't use "pages". So GSA only read the first page and get only 10 results. Some SEs have slow servers or some other issues.
    (As a side note: I've sent Sven some of these "bad apples" like yandex.com ;) )

    Because of that I think that if would benefit for everyone if we are doing some tests with the "Search online for URLs"-Tool to get an idea which SE are usefull and which are not.

    I did some research about the international SEs a few weeks ago with the "search online for url"-tool and keywords "powered by vbulletin" and"leave a comment" website. Not all results are accurate but thats a good start, I think.

    Here are my results without dupes (KW1|KW2):
    -google (453|394) -> very, very fast
    -msn/bing (145|17) -> wtf??. good speed though 
    -yahoo (855|825) -> very fast
    -hotbot (850|807) -> moderate speed
    -ecosia (849|792) -> very fast
    -info (677|607) -> moderate speed
    -lexxe (573|454) -> moderate speed
    -metacrawler (416|416) -> moderate speed
    -sky (406|305) -> moderate speed
    -euroseek (430|287*) -> slow. *idle mode after some time (aborted)
    -exalead (94*|599) -> moderate speed. *idle mode after some time (aborted)
    -mamma (241|209) -> moderate speed
    -ask (192*|129*) -> *idle mode (aborted)

    crappy results: search, searchhippo, jayde
    only page 1 results: blekko, dogpile, exite, gigablast, ixquick, scrirus, yandex
    not working: aol, charter, clusty, lycos, scrubtheweb, teoma, thunderstone, verizon, volunia


  • SvenSven www.GSA-Online.de
    added the mentioned search engines to upcoming version
  • OzzOzz
    edited March 2013
    [google ME]
    country=Montenegro
    links_on_page=100
    start_page=0
    inc_page=100
    enabled=1
    ignore=&oi=translate&|*&h=*&prev=*&zoom=1&q=*|*&sa=X&ei=*&usg=*|atdmt.com|google.|googleusercontent.|googleusercontent.|youtube.
    site=site:
    inurl=inurl:
    intitle=intitle:
    link=link:



  • OzzOzz
    edited October 2012
    I did another "Identifier"-test with search term "powered by vbulletin". As a result I use this selection of SEs. The results might differ from someone else when tested, because of proxies, SE query time or whatever, but I feel that this is a good selection for most.

    Download: http://mediafire.com/?ju4ub6ajjsd7rkb (once dowloaded -> right click SEs in GSA project -> load selection)

    Included are 37 SEs from these countries: AU, CA, IE, INT, NZ, UK and US.
    With this SEs: Google, MSN, Yahoo, Ecosia, Exite, Hotbot, Info, Lexxe, Metacrawler, Sky

    Possible additions to this selection: 
    mediocre results - aol, ask, euroseek, mamma, teoma
    not tested - mojeek (uk/int), zapmeta (uk/us), izito (uk/us)
  • SvenSven www.GSA-Online.de
    Nice :) Thanks for this analysis.
  • AlexRAlexR Cape Town
    @Ozz- Thanks! Will try those out. 
  • i am going thru that now, just a little heads up thou, mojeek doesnt support the advanced search anymore, i ll get 403 error with those queries, so please double check yourself, but it seems the mojeek advanced SE in SER doesnt work anymore since they dont support those params anymore
  • SvenSven www.GSA-Online.de

    correct settings for them (fixed in next release as well)

    [mojeek]

    desc=fast SE with good results
    country=International
    url=http://www.mojeek.co.uk/search?q=%search%&amp;s=%page%&amp;si=1
    ignore=mojeek.|w3.org
    links_on_page=10
    start_page=1
    inc_page=10
    site=site:

    [mojeek UK]
    desc=fast SE with good results
    country=United Kingdom
    url=http://www.mojeek.co.uk/search?q=%search%&amp;s=%page%&amp;si=1&amp;reg=UK
    ignore=mojeek.|w3.org
    links_on_page=10
    start_page=1
    inc_page=10
    site=site:
  • edited September 2012
    So which are the best SEs to use? Ozz above tested all the International SEs so I thought maybe I should use those. Then a few posts down he said Google Montenegro is also good. Do I select just all the Google ones or what? Does anyone have an updated list?

    EDIT: Oops, sorry! I missed the mediafire link as it's not a clickable link but just plain text! :o Thanks, Ozz for doing the Lord's work! Please let us know if you have updated your SE list again...
  • You could do the tests also. Why not give us an update about your results?
  • @Ozz - thanks for the SE list - Made a decent difference to speed :-)


  • But when i look in the GSA there are 1000 search engines. I allways select all of them. Shouldnt i? I also added the 37 search engines but i didnt saw them added....
  • SvenSven www.GSA-Online.de
    selecting all search engines is not very clever. A lot are dependent on special countries or languages. And if you e.g. have only English keywords in your project, what sense does it make to use e.g. German search engines. 
  • Isnt GSA searches for new target websites to post to? If it finds a new URL that i can post to why does it matter if it was founded through chinesse search engine?
    I want to get as many different results as i can
  • s4nt0ss4nt0s Houston, Texas
    edited December 2012
    @eLeSlash - Because some people are trying to post to websites that are hosted in a different country or that are in a different language. If someone has a Chinese website and wants to build Chinese backlinks, selecting search engines that handle that language might yield better results for them. Especially if they're using Chinese keywords.
  • Yes Santos, i am aware of that, great feature if you want to get ranked on a targeted country much faster, but i would want to build backlinks wherever i can, i dont really care if i get backlinks from chinesse websites for a .com website that i want to promote, a backlink is still a backlink on google's eyes.
    That is why i am so proud of GSA SER, it supports creating backlinks on alot of different languages websites, that is really great, no other tool is coded that good
  • You should at least filter which SEs you want to use as not all giving much results at all. Some of them give you only 10 results per keyword and that won't help you much and just wasting rescources.

    Test them with the "search online for url"-tool in options --> advanced --> tools with a simple keyword to get an idea if a SE might be useful or not.
  • edited December 2012
    Thank you Ozz this make perfect sense, but i dont have the time to test and filter the 1k SEness, i will continue checking them all and search from them all... dunno what else to do
  • LeeGLeeG Eating your first bourne

    Something else to watch out for is the "already parsed" message

    Its normally a sign of using too many clone engines

    Any engine you use, check their details and see if they are using the results of the big players, ie google or bing. There are a few independent engines out there. Baidu from China and Yandex from Russia

    An easy way to find the independent engines is to check your own website traffic logs and see who or what hits on a regular basis

  • found some nice chart of the best 15 search engines with some nice stats

    http://www.ebizmba.com/articles/search-engines
  • edited June 2013
    btw @Ozz and @Sven is there a way to search without footprints ?

    i wanted to add another search engine (seems a google clone so or soo but..)

    now i did :

    [SearchingQ : Added]
    country=international
    url=http://www.searchinq.com/web?page=%page%&amp;q=%search%
    links_on_page=10
    start_page=1
    inc_page=100
    enabled=1
    ignore=http://www.searchinq.com
    site=site:

    but when i check what SER does it takes the footprints with the search and produces following url:

    http://www.searchinq.com/web?q="Powered+"Attached+Image:+"Attached+Image:+"++by+Blogtronix"+test&amp;page=1

    ..whithout the footprints i get proper results :

    http://www.searchinq.com/web?q=test&page=1


    also i noticed .. after useing (Active S) when you see alot of [001/001] and you check the link aka click on it ...it often means you are blocked like on yandex or ecosia
  • SvenSven www.GSA-Online.de
    I will add that search engine on next release. You seem to have forgotten to make &amp; to & only.
  • "inc_page" seems not right to me also
  • SvenSven www.GSA-Online.de
    and this engine did not support quotes. I have extended the configuration for it in next version with query_remove=" so it works also with that engine.
  • @Ozz ups yep inc page i set to 10 .. great @Sven thx
  • All the ecosia.org search engines are not working for me anymore, would you please check on your end if this is the same for you? This is a good bing clone.
Sign In or Register to comment.