How to add a new search engine ?

Comments

• Sven www.GSA-Online.de

  December 2017

  http://www.liveinternet.ru/q/ << added in next update
  http://yandex.ru/yandsearch << already added
  http://nova.rambler.ru/ << already added
  http://go.mail.ru/ << added in next update
  

  Thanked by 1Deeeeeeee
• minevision2 Russia

  December 2017

  Thank you!
• Lord_Alfred world

  January 2018 edited January 2018

  @Sven, I suggest some new search engines.
  
  For titles parsing propose to use "suggestions" from search engines (check it manually, some engines might not working now, I get this links from my php-scripts/c#-programms):
  
  google: 
  

  'http://suggestqueries.google.com/complete/search?output=chrome&hl=' . $lang . '&ie=utf8&oe=utf8&q='.urlencode($keyword);

  or

  String.Format("https://clients1.google.com/s?hl={0}&client=opera&q={1}", language, query);

  or

  String.Format("https://clients1.google.com/complete/search?q={1}&hl={0}&nolabels=t&client=hp&ds=", language, query);

  yandex:

  'http://suggest.yandex.ru/suggest-ff.cgi?part='.urlencode($keyword);

  or

  String.Format("https://suggest.yandex.com/suggest-ya.cgi?uil={1}&v=1&part={0}", query, language);

  nigma:
  

  'http://autocomplete.nigma.ru/complete/query_help.php?suggest=true&q='.urlencode($keyword);

  yahoo:
  

  'http://search.yahoo.com/sugg/gossip/gossip-us-fp/?nresults=20&command='.urlencode($keyword);

  ask:
  

  'http://ss.ask.com/query?li=ff&sstype=prefix&num=20&q='.urlencode($keyword);

  duckduckgo:
  

  'https://duckduckgo.com/ac/?callback=autocompleteCallback&q='.urlencode($keyword);

  or

  String.Format("https://duckduckgo.com/ac/?q={0}", query);

  Youtube:
  

  'https://clients1.google.com/complete/search?client=youtube&q='.urlencode($keyword);

  or

  String.Format("https://clients1.google.com/complete/search?q={1}&hl={0}&nolabels=t&client=youtube&ds=yt", language, query);

  google books:

  String.Format("https://clients1.google.com/complete/search?q={1}&hl={0}&nolabels=t&client=books&ds=bo", language, query);

  google products:

  String.Format("https://clients1.google.com/complete/search?q={1}&hl={0}&nolabels=t&client=products-cc&ds=sh", language, query);

  google news:

  String.Format("https://clients1.google.com/complete/search?q={1}&hl={0}&nolabels=t&client=news-cc&ds=n", language, query);

  bing:

  switch(language.ToLower()) {
  	case "de": language_code = "de-de"; break;
  	case "es": language_code = "es-es"; break;
  	case "it": language_code = "it-it"; break;
  	case "en": language_code = "en-us"; break;
  	case "fr": language_code = "fr-fr"; break;
  	case "nl": language_code = "nl-nl"; break;
  	case "ru": language_code = "ru-ru"; break;
  	default: language_code = "en-us"; break;
  }
  
  var url = String.Format("https://www.bing.com/AS/Suggestions?mkt={0}&qry={1}&cvid={2}", language_code, query, rnd.Next(100, 999));
  

  qwant:

  String.Format("https://api.qwant.com/api/suggest?q={0}&lang={1}&count=1", query, language_code);

  or

  String.Format("https://lite.qwant.com/suggest/?count=4&word={0}", query);

  yahoo answers:

  String.Format("https://search.yahoo.com/sugg/ss/gossip-us_ss-vertical_ss/?pubid=1307&appid=yanswer&command={0}&callback=", query);

  answerscom:

  String.Format("http://www.answers.com/solo/header/redcarpet/fayt?search_string={0}&limit=200", query);

  zapmeta:

  String.Format("http://www.zapmeta.com/jx?q={0}&limit=15&hl=1", query);

  hulbee:

  String.Format("https://suggest.hulbee.com/suggest?query={0}&callback=callback&count=50&culture={1}&bucket=Web", query, language_code); // lang code same as bing

  wotbox:

  String.Format("http://www.wotbox.com/autocomplete?q={0}&la={1}", query, language);

  exalead:

  String.Format("http://www.exalead.com/search/web/suggest/?format=json&q={0}", query);

  mailru:

  String.Format("http://suggests.go.mail.ru/sg_u?q={0}", query);

  rambler:

  String.Format("https://nova.rambler.ru/suggest?v=3&query={0}", query);

  mailcom:

  String.Format("https://search.mail.com/s/?sType=web&uiLang={1}&host=mailcom&entryPoint=*&count=100&outEnc=UTF-8&q={0}", query, language);

  ecosia:

  String.Format("https://ac.ecosia.org/autocomplete?q={0}&mkt={1}", query, language_code);

  gogoby:

  String.Format("http://www.gogo.by/default/index/autosuggest/?q={0}", query);

  sputnik:

  String.Format("http://sgs.sputnik.ru/?format=json&type=web&query={0}", query);

  aol:

  String.Format("http://autosuggest.search.aol.com/autocomplete/get?q={0}&it=ws-landing&locale={1}&lang={2}&count=8&source=web&output=json", query, language, language_code);

  amazon:

  String.Format("http://completion.amazon.com/search/complete?search-alias=aps&client=amazon-search-ui&mkt=1&q={0}", query);

  
  

  And, of course, you can use some of these search engines for their intended purpose - parsing articles/links.
• Lord_Alfred world

  January 2018 edited January 2018

  @Sven, for parsing links/articles I propose using many domains of open-source "Searx" meta-engine: https://github.com/asciimoo/searx (docs: https://asciimoo.github.io/searx/)
  
  Public Searx instances:
  https://github.com/asciimoo/searx/wiki/Searx-instances
  and
  http://stats.searx.oe5tpo.com/
  
  url to parsing like:
  

  $url = $domain.'?q='.urlencode($keyword).'&category_general=1&format=rss&pageno=1';
  

  
  +
  
  duckduckgo:
  

  'https://duckduckgo.com/html/?q='.$keyword;

  
  
• Lord_Alfred world

  January 2018 edited January 2018

  Why for scraping authority links use only 3 services? I think, for more relevant maybe using top-10 from all search engines.
  Now all my proxies banned in google and I dont get any authority links... But I need it.
  
  
  
  UPD: when no authority links - CG set empty links (without url in href):
  
  
• Sven www.GSA-Online.de

  January 2018

  thanks for the links. Next version has them all in + a auth-link scraper based on SearX
  

  Thanked by 1Lord_Alfred