Cleaning up the Search Engines

cherub

I decided to take some time to try and gauge the effectiveness of the non Google, Yahoo and Bing search engines, as these engines can be good sources of urls, particularly if your proxies are prone to bans. So, to do this I created a simple engine file with one footprint - "powered by wordpress". Surely this has to be one of the most widely spread footprints on the web, and any worthy search engine should be able to give us some results for it.

Next, I unchecked by mask all search engines with google, yahoo or msn in the name. This left me with around 575 search engines. I then started a scrape at 50 threads, 60 seconds between searches, and using a plain residential ISP IP.

After the scrape was finished, I took note of the search engines that gave no results. These were:
Acoon
Ananzi
AOL
Atlas
Centrum
DuckDuckGo
Ecosia
EuroSeek
Expopage
FindLink
Gigablast
inout-search-ultimate
iZito
Jayde
krstarica
kvasir
List.ru
Mamma
Meta RU
Metabot.ru
Metager 2
mojeek
Nigma
pathfinder
Portelo
SearchHippo
SearchingQ
Thunderstone
vinden.nl
Volunia
webcrawler
Wirtualna Polska
zapmeta
Zoznam
cn.bing
yahoo.cn

When I have time, I'm going to try and go through each engine individually, to see if it needs fixing or just needs deleting. But so far, I see these updates need making:

Euroseek * - this now seems to be just a directory, with no search. I'd remove it.
Expopage - does not appear to be a search engine.
FindLink - appears to be dead.
Gigablast - seems to give an ascii-based captcha that I don't think is solvable via automated tools. Remove?

And some quick fixes to some engines:

[Acoon]
country=germany
url=https://www.acoon.de/cgi/search.exe?begriff=%search%&startwith=%page%
links_on_page=10
start_page=1
inc_page=10
enabled=0
ignore=acoon|overture.com|www.w3.org

[Atlas]
country=Czech Republic
url=http://searchatlas.centrum.cz/?q=%search%&l-choose=&l=cs&kibitz=0&kibitz-db=&trigger=button&offset=%page% ignore=atlas.cz|centrum.cz|najisto.cz|heureka.cz|aktualne.cz|zena.cz|bearshare.com|centrumholdings.|clicktale.net|www.isb.cz|www.w3.org|.atdmt.com|i0.cz|ippi.cz|google-analytics.com
inc_page=10
start_page=1
links_on_page=10
enabled=0
site=site:
inurl=inurl:
intitle=intitle:

[Jayde]
country=international
url=http://www.jayde.com/sch.html?q=%search%&ds=%page%
links_on_page=5
start_page=0
inc_page=5
enabled=0
ignore=jayde.com|/jayde/|600z.com|www.w3.org|ientry.com|twellow.com|ientrymail.com|www.website.com|www.yoursite.com|ientry.net|quantserve.com

Will add to this thread when I have time.

KayKay

great thx

Sven

thanks, working on this as well now.

rodol

wow nice, hope sven add this modifications on next update

gooner

@cherub - You're a star, thanks for doing all that.

2Take2

@cherub - Nice work, many thanks.

system0102

@cherub Big respect  =D>

cefege

Awesome @cherub

cherub

A couple more search engines working.

[kvasir]
country=Norway
url=http://www.kvasir.no/alle?offset=%page%&q=%search%
inc_page=10
start_page=0
links_on_page=10
ignore=kvasir.no|sol.no|kvasirbeta.no
enabled=0
site=site:
inurl=inurl:
intitle=intitle:
link=link:

[Nigma]
country=Russian Federation
desc=multi-search
url=http://www.nigma.ru/index.php?startpos=%page%&s=%search%&srt=0&gl=1&yh=1&ms=1&yn=1&rm=1&av=1&nm=1&k=loyT
ignore=nigma.ru|yandex.ru|google.ru|rambler.ru|yahoo.com|bing.com|.mail.ru|ru.wikipedia.org|.msu.ru|.stanford.edu|.w3.org/|ogp.me/
enabled=0
inc_page=10
start_page=0

Sven

thanks, next update will have the fixes.

cherub

Everything indicates that the DuckDuckGo engines should work, but am getting no results. I'm wondering if this is because there are no actual html links in the results page, as it's just a javascript array. Perhaps some sort of regex needs adding to it eg "u":"([^"]+)"

Sven

will have a look on that.