.oO SearXNG Developer Documentation Oo.
Loading...
Searching...
No Matches
searx.engines.google Namespace Reference

Functions

str ui_async (int start)
 get_google_info (params, eng_traits)
 detect_google_sorry (resp)
 request (query, params)
 parse_data_images (str text)
EngineResults response (resp)
 fetch_traits (EngineTraits engine_traits, bool add_domains=True)

Variables

dict about
list categories = ['general', 'web']
bool paging = True
int max_page = 50
bool time_range_support = True
bool safesearch = True
dict time_range_dict = {'day': 'd', 'week': 'w', 'month': 'm', 'year': 'y'}
dict filter_mapping = {0: 'off', 1: 'medium', 2: 'high'}
str suggestion_xpath = '//div[contains(@class, "EIaa9b")]//a'
str _arcid_range = string.ascii_letters + string.digits + "_-"
tuple _arcid_random = None
 RE_DATA_IMAGE = re.compile(r'"(dimg_[^"]*)"[^;]*;(data:image[^;]*;[^;]*);')
 RE_DATA_IMAGE_end = re.compile(r'"(dimg_[^"]*)"[^;]*;(data:image[^;]*;[^;]*)$')
list skip_countries

Detailed Description

This is the implementation of the Google WEB engine.  Some of this
implementations (manly the :py:obj:`get_google_info`) are shared by other
engines:

- :ref:`google images engine`
- :ref:`google news engine`
- :ref:`google videos engine`
- :ref:`google scholar engine`
- :ref:`google autocomplete`

Function Documentation

◆ detect_google_sorry()

searx.engines.google.detect_google_sorry ( resp)

Definition at line 271 of file google.py.

271def detect_google_sorry(resp):
272 if resp.url.host == 'sorry.google.com' or resp.url.path.startswith('/sorry'):
273 raise SearxEngineCaptchaException()
274
275

Referenced by response().

Here is the caller graph for this function:

◆ fetch_traits()

searx.engines.google.fetch_traits ( EngineTraits engine_traits,
bool add_domains = True )
Fetch languages from Google.

Definition at line 450 of file google.py.

450def fetch_traits(engine_traits: EngineTraits, add_domains: bool = True):
451 """Fetch languages from Google."""
452 # pylint: disable=import-outside-toplevel, too-many-branches
453
454 engine_traits.custom['supported_domains'] = {}
455
456 resp = get('https://www.google.com/preferences')
457 if not resp.ok: # type: ignore
458 raise RuntimeError("Response from Google's preferences is not OK.")
459
460 dom = html.fromstring(resp.text.replace('<?xml version="1.0" encoding="UTF-8"?>', ''))
461
462 # supported language codes
463
464 lang_map = {'no': 'nb'}
465 for x in eval_xpath_list(dom, "//select[@name='hl']/option"):
466 eng_lang = x.get("value")
467 try:
468 locale = babel.Locale.parse(lang_map.get(eng_lang, eng_lang), sep='-')
469 except babel.UnknownLocaleError:
470 print("INFO: google UI language %s (%s) is unknown by babel" % (eng_lang, x.text.split("(")[0].strip()))
471 continue
472 sxng_lang = language_tag(locale)
473
474 conflict = engine_traits.languages.get(sxng_lang)
475 if conflict:
476 if conflict != eng_lang:
477 print("CONFLICT: babel %s --> %s, %s" % (sxng_lang, conflict, eng_lang))
478 continue
479 engine_traits.languages[sxng_lang] = 'lang_' + eng_lang
480
481 # alias languages
482 engine_traits.languages['zh'] = 'lang_zh-CN'
483
484 # supported region codes
485
486 for x in eval_xpath_list(dom, "//select[@name='gl']/option"):
487 eng_country = x.get("value")
488
489 if eng_country in skip_countries:
490 continue
491 if eng_country == 'ZZ':
492 engine_traits.all_locale = 'ZZ'
493 continue
494
495 sxng_locales = get_official_locales(eng_country, engine_traits.languages.keys(), regional=True)
496
497 if not sxng_locales:
498 print("ERROR: can't map from google country %s (%s) to a babel region." % (x.get('data-name'), eng_country))
499 continue
500
501 for sxng_locale in sxng_locales:
502 engine_traits.regions[region_tag(sxng_locale)] = eng_country
503
504 # alias regions
505 engine_traits.regions['zh-CN'] = 'HK'
506
507 # supported domains
508
509 if add_domains:
510 resp = get('https://www.google.com/supported_domains')
511 if not resp.ok: # type: ignore
512 raise RuntimeError("Response from https://www.google.com/supported_domains is not OK.")
513
514 for domain in resp.text.split(): # type: ignore
515 domain = domain.strip()
516 if not domain or domain in [
517 '.google.com',
518 ]:
519 continue
520 region = domain.split('.')[-1].upper()
521 engine_traits.custom['supported_domains'][region] = 'www' + domain # type: ignore
522 if region == 'HK':
523 # There is no google.cn, we use .com.hk for zh-CN
524 engine_traits.custom['supported_domains']['CN'] = 'www' + domain # type: ignore

◆ get_google_info()

searx.engines.google.get_google_info ( params,
eng_traits )
Composing various (language) properties for the google engines (:ref:`google
API`).

This function is called by the various google engines (:ref:`google web
engine`, :ref:`google images engine`, :ref:`google news engine` and
:ref:`google videos engine`).

:param dict param: Request parameters of the engine.  At least
    a ``searxng_locale`` key should be in the dictionary.

:param eng_traits: Engine's traits fetched from google preferences
    (:py:obj:`searx.enginelib.traits.EngineTraits`)

:rtype: dict
:returns:
    Py-Dictionary with the key/value pairs:

    language:
        The language code that is used by google (e.g. ``lang_en`` or
        ``lang_zh-TW``)

    country:
        The country code that is used by google (e.g. ``US`` or ``TW``)

    locale:
        A instance of :py:obj:`babel.core.Locale` build from the
        ``searxng_locale`` value.

    subdomain:
        Google subdomain :py:obj:`google_domains` that fits to the country
        code.

    params:
        Py-Dictionary with additional request arguments (can be passed to
        :py:func:`urllib.parse.urlencode`).

        - ``hl`` parameter: specifies the interface language of user interface.
        - ``lr`` parameter: restricts search results to documents written in
          a particular language.
        - ``cr`` parameter: restricts search results to documents
          originating in a particular country.
        - ``ie`` parameter: sets the character encoding scheme that should
          be used to interpret the query string ('utf8').
        - ``oe`` parameter: sets the character encoding scheme that should
          be used to decode the XML result ('utf8').

    headers:
        Py-Dictionary with additional HTTP headers (can be passed to
        request's headers)

        - ``Accept: '*/*``

Definition at line 92 of file google.py.

92def get_google_info(params, eng_traits):
93 """Composing various (language) properties for the google engines (:ref:`google
94 API`).
95
96 This function is called by the various google engines (:ref:`google web
97 engine`, :ref:`google images engine`, :ref:`google news engine` and
98 :ref:`google videos engine`).
99
100 :param dict param: Request parameters of the engine. At least
101 a ``searxng_locale`` key should be in the dictionary.
102
103 :param eng_traits: Engine's traits fetched from google preferences
104 (:py:obj:`searx.enginelib.traits.EngineTraits`)
105
106 :rtype: dict
107 :returns:
108 Py-Dictionary with the key/value pairs:
109
110 language:
111 The language code that is used by google (e.g. ``lang_en`` or
112 ``lang_zh-TW``)
113
114 country:
115 The country code that is used by google (e.g. ``US`` or ``TW``)
116
117 locale:
118 A instance of :py:obj:`babel.core.Locale` build from the
119 ``searxng_locale`` value.
120
121 subdomain:
122 Google subdomain :py:obj:`google_domains` that fits to the country
123 code.
124
125 params:
126 Py-Dictionary with additional request arguments (can be passed to
127 :py:func:`urllib.parse.urlencode`).
128
129 - ``hl`` parameter: specifies the interface language of user interface.
130 - ``lr`` parameter: restricts search results to documents written in
131 a particular language.
132 - ``cr`` parameter: restricts search results to documents
133 originating in a particular country.
134 - ``ie`` parameter: sets the character encoding scheme that should
135 be used to interpret the query string ('utf8').
136 - ``oe`` parameter: sets the character encoding scheme that should
137 be used to decode the XML result ('utf8').
138
139 headers:
140 Py-Dictionary with additional HTTP headers (can be passed to
141 request's headers)
142
143 - ``Accept: '*/*``
144
145 """
146
147 ret_val = {
148 'language': None,
149 'country': None,
150 'subdomain': None,
151 'params': {},
152 'headers': {},
153 'cookies': {},
154 'locale': None,
155 }
156
157 sxng_locale = params.get('searxng_locale', 'all')
158 try:
159 locale = babel.Locale.parse(sxng_locale, sep='-')
160 except babel.core.UnknownLocaleError:
161 locale = None
162
163 eng_lang = eng_traits.get_language(sxng_locale, 'lang_en')
164 lang_code = eng_lang.split('_')[-1] # lang_zh-TW --> zh-TW / lang_en --> en
165 country = eng_traits.get_region(sxng_locale, eng_traits.all_locale)
166
167 # Test zh_hans & zh_hant --> in the topmost links in the result list of list
168 # TW and HK you should a find wiktionary.org zh_hant link. In the result
169 # list of zh-CN should not be no hant link instead you should find
170 # zh.m.wikipedia.org/zh somewhere in the top.
171
172 # '!go 日 :zh-TW' --> https://zh.m.wiktionary.org/zh-hant/%E6%97%A5
173 # '!go 日 :zh-CN' --> https://zh.m.wikipedia.org/zh/%E6%97%A5
174
175 ret_val['language'] = eng_lang
176 ret_val['country'] = country
177 ret_val['locale'] = locale
178 ret_val['subdomain'] = eng_traits.custom['supported_domains'].get(country.upper(), 'www.google.com')
179
180 # hl parameter:
181 # The hl parameter specifies the interface language (host language) of
182 # your user interface. To improve the performance and the quality of your
183 # search results, you are strongly encouraged to set this parameter
184 # explicitly.
185 # https://developers.google.com/custom-search/docs/xml_results#hlsp
186 # The Interface Language:
187 # https://developers.google.com/custom-search/docs/xml_results_appendices#interfaceLanguages
188
189 # https://github.com/searxng/searxng/issues/2515#issuecomment-1607150817
190 ret_val['params']['hl'] = f'{lang_code}-{country}'
191
192 # lr parameter:
193 # The lr (language restrict) parameter restricts search results to
194 # documents written in a particular language.
195 # https://developers.google.com/custom-search/docs/xml_results#lrsp
196 # Language Collection Values:
197 # https://developers.google.com/custom-search/docs/xml_results_appendices#languageCollections
198 #
199 # To select 'all' languages an empty 'lr' value is used.
200 #
201 # Different to other google services, Google Scholar supports to select more
202 # than one language. The languages are separated by a pipe '|' (logical OR).
203 # By example: &lr=lang_zh-TW%7Clang_de selects articles written in
204 # traditional chinese OR german language.
205
206 ret_val['params']['lr'] = eng_lang
207 if sxng_locale == 'all':
208 ret_val['params']['lr'] = ''
209
210 # cr parameter:
211 # The cr parameter restricts search results to documents originating in a
212 # particular country.
213 # https://developers.google.com/custom-search/docs/xml_results#crsp
214
215 # specify a region (country) only if a region is given in the selected
216 # locale --> https://github.com/searxng/searxng/issues/2672
217 ret_val['params']['cr'] = ''
218 if len(sxng_locale.split('-')) > 1:
219 ret_val['params']['cr'] = 'country' + country
220
221 # gl parameter: (mandatory by Google News)
222 # The gl parameter value is a two-letter country code. For WebSearch
223 # results, the gl parameter boosts search results whose country of origin
224 # matches the parameter value. See the Country Codes section for a list of
225 # valid values.
226 # Specifying a gl parameter value in WebSearch requests should improve the
227 # relevance of results. This is particularly true for international
228 # customers and, even more specifically, for customers in English-speaking
229 # countries other than the United States.
230 # https://developers.google.com/custom-search/docs/xml_results#glsp
231
232 # https://github.com/searxng/searxng/issues/2515#issuecomment-1606294635
233 # ret_val['params']['gl'] = country
234
235 # ie parameter:
236 # The ie parameter sets the character encoding scheme that should be used
237 # to interpret the query string. The default ie value is latin1.
238 # https://developers.google.com/custom-search/docs/xml_results#iesp
239
240 ret_val['params']['ie'] = 'utf8'
241
242 # oe parameter:
243 # The oe parameter sets the character encoding scheme that should be used
244 # to decode the XML result. The default oe value is latin1.
245 # https://developers.google.com/custom-search/docs/xml_results#oesp
246
247 ret_val['params']['oe'] = 'utf8'
248
249 # num parameter:
250 # The num parameter identifies the number of search results to return.
251 # The default num value is 10, and the maximum value is 20. If you request
252 # more than 20 results, only 20 results will be returned.
253 # https://developers.google.com/custom-search/docs/xml_results#numsp
254
255 # HINT: seems to have no effect (tested in google WEB & Images)
256 # ret_val['params']['num'] = 20
257
258 # HTTP headers
259
260 ret_val['headers']['Accept'] = '*/*'
261
262 # Cookies
263
264 # - https://github.com/searxng/searxng/pull/1679#issuecomment-1235432746
265 # - https://github.com/searxng/searxng/issues/1555
266 ret_val['cookies']['CONSENT'] = "YES+"
267
268 return ret_val
269
270

Referenced by request().

Here is the caller graph for this function:

◆ parse_data_images()

searx.engines.google.parse_data_images ( str text)

Definition at line 329 of file google.py.

329def parse_data_images(text: str):
330 data_image_map = {}
331
332 for img_id, data_image in RE_DATA_IMAGE.findall(text):
333 end_pos = data_image.rfind('=')
334 if end_pos > 0:
335 data_image = data_image[: end_pos + 1]
336 data_image_map[img_id] = data_image
337 last = RE_DATA_IMAGE_end.search(text)
338 if last:
339 data_image_map[last.group(1)] = last.group(2)
340 logger.debug('data:image objects --> %s', list(data_image_map.keys()))
341 return data_image_map
342
343

Referenced by response().

Here is the caller graph for this function:

◆ request()

searx.engines.google.request ( query,
params )
Google search request

Definition at line 276 of file google.py.

276def request(query, params):
277 """Google search request"""
278 # pylint: disable=line-too-long
279 start = (params['pageno'] - 1) * 10
280 str_async = ui_async(start)
281 google_info = get_google_info(params, traits)
282 logger.debug("ARC_ID: %s", str_async)
283
284 # https://www.google.de/search?q=corona&hl=de&lr=lang_de&start=0&tbs=qdr%3Ad&safe=medium
285 query_url = (
286 'https://'
287 + google_info['subdomain']
288 + '/search'
289 + "?"
290 + urlencode(
291 {
292 'q': query,
293 **google_info['params'],
294 'filter': '0',
295 'start': start,
296 # 'vet': '12ahUKEwik3ZbIzfn7AhXMX_EDHbUDBh0QxK8CegQIARAC..i',
297 # 'ved': '2ahUKEwik3ZbIzfn7AhXMX_EDHbUDBh0Q_skCegQIARAG',
298 # 'cs' : 1,
299 # 'sa': 'N',
300 # 'yv': 3,
301 # 'prmd': 'vin',
302 # 'ei': 'GASaY6TxOcy_xc8PtYeY6AE',
303 # 'sa': 'N',
304 # 'sstk': 'AcOHfVkD7sWCSAheZi-0tx_09XDO55gTWY0JNq3_V26cNN-c8lfD45aZYPI8s_Bqp8s57AHz5pxchDtAGCA_cikAWSjy9kw3kgg'
305 # formally known as use_mobile_ui
306 'asearch': 'arc',
307 'async': str_async,
308 }
309 )
310 )
311
312 if params['time_range'] in time_range_dict:
313 query_url += '&' + urlencode({'tbs': 'qdr:' + time_range_dict[params['time_range']]})
314 if params['safesearch']:
315 query_url += '&' + urlencode({'safe': filter_mapping[params['safesearch']]})
316 params['url'] = query_url
317
318 params['cookies'] = google_info['cookies']
319 params['headers'].update(google_info['headers'])
320 return params
321
322
323# =26;[3,"dimg_ZNMiZPCqE4apxc8P3a2tuAQ_137"]a87;data:image/jpeg;base64,/9j/4AAQSkZJRgABA
324# ...6T+9Nl4cnD+gr9OK8I56/tX3l86nWYw//2Q==26;

References get_google_info(), and ui_async().

Here is the call graph for this function:

◆ response()

EngineResults searx.engines.google.response ( resp)
Get response from google's search request

Definition at line 344 of file google.py.

344def response(resp) -> EngineResults:
345 """Get response from google's search request"""
346 # pylint: disable=too-many-branches, too-many-statements
347 detect_google_sorry(resp)
348 data_image_map = parse_data_images(resp.text)
349
350 results = EngineResults()
351
352 # convert the text to dom
353 dom = html.fromstring(resp.text)
354
355 # results --> answer
356 answer_list = eval_xpath(dom, '//div[contains(@class, "LGOjhe")]')
357 for item in answer_list:
358 for bubble in eval_xpath(item, './/div[@class="nnFGuf"]'):
359 bubble.drop_tree()
360 results.add(
361 results.types.Answer(
362 answer=extract_text(item),
363 url=(eval_xpath(item, '../..//a/@href') + [None])[0],
364 )
365 )
366
367 # parse results
368
369 for result in eval_xpath_list(dom, './/div[contains(@jscontroller, "SC7lYd")]'):
370 # pylint: disable=too-many-nested-blocks
371
372 try:
373 title_tag = eval_xpath_getindex(result, './/a/h3[1]', 0, default=None)
374 if title_tag is None:
375 # this not one of the common google results *section*
376 logger.debug('ignoring item from the result_xpath list: missing title')
377 continue
378 title = extract_text(title_tag)
379
380 url = eval_xpath_getindex(result, './/a[h3]/@href', 0, None)
381 if url is None:
382 logger.debug('ignoring item from the result_xpath list: missing url of title "%s"', title)
383 continue
384
385 content_nodes = eval_xpath(result, './/div[contains(@data-sncf, "1")]')
386 for item in content_nodes:
387 for script in item.xpath(".//script"):
388 script.getparent().remove(script)
389
390 content = extract_text(content_nodes)
391
392 if not content:
393 logger.debug('ignoring item from the result_xpath list: missing content of title "%s"', title)
394 continue
395
396 thumbnail = content_nodes[0].xpath('.//img/@src')
397 if thumbnail:
398 thumbnail = thumbnail[0]
399 if thumbnail.startswith('data:image'):
400 img_id = content_nodes[0].xpath('.//img/@id')
401 if img_id:
402 thumbnail = data_image_map.get(img_id[0])
403 else:
404 thumbnail = None
405
406 results.append({'url': url, 'title': title, 'content': content, 'thumbnail': thumbnail})
407
408 except Exception as e: # pylint: disable=broad-except
409 logger.error(e, exc_info=True)
410 continue
411
412 # parse suggestion
413 for suggestion in eval_xpath_list(dom, suggestion_xpath):
414 # append suggestion
415 results.append({'suggestion': extract_text(suggestion)})
416
417 # return results
418 return results
419
420
421# get supported languages from their site
422
423

References detect_google_sorry(), and parse_data_images().

Here is the call graph for this function:

◆ ui_async()

str searx.engines.google.ui_async ( int start)
Format of the response from UI's async request.

- ``arc_id:<...>,use_ac:true,_fmt:prog``

The arc_id is random generated every hour.

Definition at line 70 of file google.py.

70def ui_async(start: int) -> str:
71 """Format of the response from UI's async request.
72
73 - ``arc_id:<...>,use_ac:true,_fmt:prog``
74
75 The arc_id is random generated every hour.
76 """
77 global _arcid_random # pylint: disable=global-statement
78
79 use_ac = "use_ac:true"
80 # _fmt:html returns a HTTP 500 when user search for celebrities like
81 # '!google natasha allegri' or '!google chris evans'
82 _fmt = "_fmt:prog"
83
84 # create a new random arc_id every hour
85 if not _arcid_random or (int(time.time()) - _arcid_random[1]) > 3600:
86 _arcid_random = (''.join(random.choices(_arcid_range, k=23)), int(time.time()))
87 arc_id = f"arc_id:srp_{_arcid_random[0]}_1{start:02}"
88
89 return ",".join([arc_id, use_ac, _fmt])
90
91

Referenced by request().

Here is the caller graph for this function:

Variable Documentation

◆ _arcid_random

tuple searx.engines.google._arcid_random = None
protected

Definition at line 67 of file google.py.

◆ _arcid_range

str searx.engines.google._arcid_range = string.ascii_letters + string.digits + "_-"
protected

Definition at line 66 of file google.py.

◆ about

dict searx.engines.google.about
Initial value:
1= {
2 "website": 'https://www.google.com',
3 "wikidata_id": 'Q9366',
4 "official_api_documentation": 'https://developers.google.com/custom-search/',
5 "use_official_api": False,
6 "require_api_key": False,
7 "results": 'HTML',
8}

Definition at line 33 of file google.py.

◆ categories

list searx.engines.google.categories = ['general', 'web']

Definition at line 43 of file google.py.

◆ filter_mapping

dict searx.engines.google.filter_mapping = {0: 'off', 1: 'medium', 2: 'high'}

Definition at line 56 of file google.py.

◆ max_page

int searx.engines.google.max_page = 50

Definition at line 45 of file google.py.

◆ paging

bool searx.engines.google.paging = True

Definition at line 44 of file google.py.

◆ RE_DATA_IMAGE

searx.engines.google.RE_DATA_IMAGE = re.compile(r'"(dimg_[^"]*)"[^;]*;(data:image[^;]*;[^;]*);')

Definition at line 325 of file google.py.

◆ RE_DATA_IMAGE_end

searx.engines.google.RE_DATA_IMAGE_end = re.compile(r'"(dimg_[^"]*)"[^;]*;(data:image[^;]*;[^;]*)$')

Definition at line 326 of file google.py.

◆ safesearch

bool searx.engines.google.safesearch = True

Definition at line 51 of file google.py.

◆ skip_countries

list searx.engines.google.skip_countries
Initial value:
1= [
2 # official language of google-country not in google-languages
3 'AL', # Albanien (sq)
4 'AZ', # Aserbaidschan (az)
5 'BD', # Bangladesch (bn)
6 'BN', # Brunei Darussalam (ms)
7 'BT', # Bhutan (dz)
8 'ET', # Äthiopien (am)
9 'GE', # Georgien (ka, os)
10 'GL', # Grönland (kl)
11 'KH', # Kambodscha (km)
12 'LA', # Laos (lo)
13 'LK', # Sri Lanka (si, ta)
14 'ME', # Montenegro (sr)
15 'MK', # Nordmazedonien (mk, sq)
16 'MM', # Myanmar (my)
17 'MN', # Mongolei (mn)
18 'MV', # Malediven (dv) // dv_MV is unknown by babel
19 'MY', # Malaysia (ms)
20 'NP', # Nepal (ne)
21 'TJ', # Tadschikistan (tg)
22 'TM', # Turkmenistan (tk)
23 'UZ', # Usbekistan (uz)
24]

Definition at line 424 of file google.py.

◆ suggestion_xpath

str searx.engines.google.suggestion_xpath = '//div[contains(@class, "EIaa9b")]//a'

Definition at line 63 of file google.py.

◆ time_range_dict

dict searx.engines.google.time_range_dict = {'day': 'd', 'week': 'w', 'month': 'm', 'year': 'y'}

Definition at line 53 of file google.py.

◆ time_range_support

bool searx.engines.google.time_range_support = True

Definition at line 50 of file google.py.