How to add a new search engine ?

minevision2

Hello. How to add a new search engine to GSA Content Generator ? http://www.liveinternet.ru/q/? and http://yandex.ru/yandsearch? and http://nova.rambler.ru/search? and http://go.mail.ru/search? Or http://www.liveinternet.ru , http://yandex.ru etc ?
I am interested  liveinternet.ru , yandex.ru , rambler.ru ,mail.ru

Sven

http://www.liveinternet.ru/q/ << added in next update
http://yandex.ru/yandsearch << already added
http://nova.rambler.ru/ << already added
http://go.mail.ru/ << added in next update

minevision2

Thank you!

Lord_Alfred

@Sven, I suggest some new search engines.

For titles parsing propose to use "suggestions" from search engines (check it manually, some engines might not working now, I get this links from my php-scripts/c#-programms):

google: 

'http://suggestqueries.google.com/complete/search?output=chrome&hl=' . $lang . '&ie=utf8&oe=utf8&q='.urlencode($keyword);

or

String.Format("https://clients1.google.com/s?hl={0}&client=opera&q={1}", language, query);

or

String.Format("https://clients1.google.com/complete/search?q={1}&hl={0}&nolabels=t&client=hp&ds=", language, query);

yandex:

'http://suggest.yandex.ru/suggest-ff.cgi?part='.urlencode($keyword);

or

String.Format("https://suggest.yandex.com/suggest-ya.cgi?uil={1}&v=1&part={0}", query, language);

nigma:

'http://autocomplete.nigma.ru/complete/query_help.php?suggest=true&q='.urlencode($keyword);

yahoo:

'http://search.yahoo.com/sugg/gossip/gossip-us-fp/?nresults=20&command='.urlencode($keyword);

ask:

'http://ss.ask.com/query?li=ff&sstype=prefix&num=20&q='.urlencode($keyword);

duckduckgo:

'https://duckduckgo.com/ac/?callback=autocompleteCallback&q='.urlencode($keyword);

or

String.Format("https://duckduckgo.com/ac/?q={0}", query);

Youtube:

'https://clients1.google.com/complete/search?client=youtube&q='.urlencode($keyword);

or

String.Format("https://clients1.google.com/complete/search?q={1}&hl={0}&nolabels=t&client=youtube&ds=yt", language, query);

google books:

String.Format("https://clients1.google.com/complete/search?q={1}&hl={0}&nolabels=t&client=books&ds=bo", language, query);

google products:

String.Format("https://clients1.google.com/complete/search?q={1}&hl={0}&nolabels=t&client=products-cc&ds=sh", language, query);

google news:

String.Format("https://clients1.google.com/complete/search?q={1}&hl={0}&nolabels=t&client=news-cc&ds=n", language, query);

bing:

switch(language.ToLower()) {
	case "de": language_code = "de-de"; break;
	case "es": language_code = "es-es"; break;
	case "it": language_code = "it-it"; break;
	case "en": language_code = "en-us"; break;
	case "fr": language_code = "fr-fr"; break;
	case "nl": language_code = "nl-nl"; break;
	case "ru": language_code = "ru-ru"; break;
	default: language_code = "en-us"; break;
}

var url = String.Format("https://www.bing.com/AS/Suggestions?mkt={0}&qry={1}&cvid={2}", language_code, query, rnd.Next(100, 999));

qwant:

String.Format("https://api.qwant.com/api/suggest?q={0}&lang={1}&count=1", query, language_code);

or

String.Format("https://lite.qwant.com/suggest/?count=4&word={0}", query);

yahoo answers:

String.Format("https://search.yahoo.com/sugg/ss/gossip-us_ss-vertical_ss/?pubid=1307&appid=yanswer&command={0}&callback=", query);

answerscom:

String.Format("http://www.answers.com/solo/header/redcarpet/fayt?search_string={0}&limit=200", query);

zapmeta:

String.Format("http://www.zapmeta.com/jx?q={0}&limit=15&hl=1", query);

hulbee:

String.Format("https://suggest.hulbee.com/suggest?query={0}&callback=callback&count=50&culture={1}&bucket=Web", query, language_code); // lang code same as bing

wotbox:

String.Format("http://www.wotbox.com/autocomplete?q={0}&la={1}", query, language);

exalead:

String.Format("http://www.exalead.com/search/web/suggest/?format=json&q={0}", query);

mailru:

String.Format("http://suggests.go.mail.ru/sg_u?q={0}", query);

rambler:

String.Format("https://nova.rambler.ru/suggest?v=3&query={0}", query);

mailcom:

String.Format("https://search.mail.com/s/?sType=web&uiLang={1}&host=mailcom&entryPoint=*&count=100&outEnc=UTF-8&q={0}", query, language);

ecosia:

String.Format("https://ac.ecosia.org/autocomplete?q={0}&mkt={1}", query, language_code);

gogoby:

String.Format("http://www.gogo.by/default/index/autosuggest/?q={0}", query);

sputnik:

String.Format("http://sgs.sputnik.ru/?format=json&type=web&query={0}", query);

aol:

String.Format("http://autosuggest.search.aol.com/autocomplete/get?q={0}&it=ws-landing&locale={1}&lang={2}&count=8&source=web&output=json", query, language, language_code);

amazon:

String.Format("http://completion.amazon.com/search/complete?search-alias=aps&client=amazon-search-ui&mkt=1&q={0}", query);

And, of course, you can use some of these search engines for their intended purpose - parsing articles/links.

Lord_Alfred

@Sven, for parsing links/articles I propose using many domains of open-source "Searx" meta-engine: https://github.com/asciimoo/searx (docs: https://asciimoo.github.io/searx/)

Public Searx instances:
https://github.com/asciimoo/searx/wiki/Searx-instances
and
http://stats.searx.oe5tpo.com/

url to parsing like:

$url = $domain.'?q='.urlencode($keyword).'&category_general=1&format=rss&pageno=1';

+

duckduckgo:

'https://duckduckgo.com/html/?q='.$keyword;

Lord_Alfred

Why for scraping authority links use only 3 services? I think, for more relevant maybe using top-10 from all search engines.
Now all my proxies banned in google and I dont get any authority links... But I need it.



UPD: when no authority links - CG set empty links (without url in href):

Sven

thanks for the links. Next version has them all in + a auth-link scraper based on SearX