searx.engines.google Namespace Reference

Functions
	get_google_info (params, eng_traits)
	detect_google_sorry (resp)
	request (query, params)
	_parse_data_images (dom)
	response (resp)
	fetch_traits (EngineTraits engine_traits, bool add_domains=True)

Variables
logging	logger .Logger
dict	about
list	categories = ['general', 'web']
bool	paging = True
int	max_page = 50
bool	time_range_support = True
bool	safesearch = True
dict	time_range_dict = {'day': 'd', 'week': 'w', 'month': 'm', 'year': 'y'}
dict	filter_mapping = {0: 'off', 1: 'medium', 2: 'high'}
str	suggestion_xpath = '//div[contains(@class, "EIaa9b")]//a'
str	UI_ASYNC = 'use_ac:true,_fmt:prog'
	RE_DATA_IMAGE = re.compile(r'"(dimg_[^"]*)"[^;]*;(data:image[^;]*;[^;]*);')
list	skip_countries

Detailed Description

This is the implementation of the Google WEB engine.  Some of this
implementations (manly the :py:obj:`get_google_info`) are shared by other
engines:

- :ref:`google images engine`
- :ref:`google news engine`
- :ref:`google videos engine`
- :ref:`google scholar engine`
- :ref:`google autocomplete`

Function Documentation

_parse_data_images()

searx.engines.google._parse_data_images ( dom )

protected

Definition at line 307 of file google.py.

def _parse_data_images(dom):
    data_image_map = {}
    for img_id, data_image in RE_DATA_IMAGE.findall(dom.text_content()):
        end_pos = data_image.rfind('=')
        if end_pos > 0:
            data_image = data_image[: end_pos + 1]
        data_image_map[img_id] = data_image
    logger.debug('data:image objects --> %s', list(data_image_map.keys()))
    return data_image_map
 
 

Referenced by searx.engines.google.response().

Here is the caller graph for this function:

detect_google_sorry()

searx.engines.google.detect_google_sorry

(

resp

)

Definition at line 252 of file google.py.

def detect_google_sorry(resp):
    if resp.url.host == 'sorry.google.com' or resp.url.path.startswith('/sorry'):
        raise SearxEngineCaptchaException()
 
 

fetch_traits()

searx.engines.google.fetch_traits	(	EngineTraits	engine_traits,
		bool	add_domains = True )

Fetch languages from Google.

Definition at line 424 of file google.py.

def fetch_traits(engine_traits: EngineTraits, add_domains: bool = True):
    """Fetch languages from Google."""
    # pylint: disable=import-outside-toplevel, too-many-branches
 
    engine_traits.custom['supported_domains'] = {}
 
    resp = get('https://www.google.com/preferences')
    if not resp.ok:  # type: ignore
        raise RuntimeError("Response from Google's preferences is not OK.")
 
    dom = html.fromstring(resp.text.replace('<?xml version="1.0" encoding="UTF-8"?>', ''))
 
    # supported language codes
 
    lang_map = {'no': 'nb'}
    for x in eval_xpath_list(dom, "//select[@name='hl']/option"):
        eng_lang = x.get("value")
        try:
            locale = babel.Locale.parse(lang_map.get(eng_lang, eng_lang), sep='-')
        except babel.UnknownLocaleError:
            print("INFO:  google UI language %s (%s) is unknown by babel" % (eng_lang, x.text.split("(")[0].strip()))
            continue
        sxng_lang = language_tag(locale)
 
        conflict = engine_traits.languages.get(sxng_lang)
        if conflict:
            if conflict != eng_lang:
                print("CONFLICT: babel %s --> %s, %s" % (sxng_lang, conflict, eng_lang))
            continue
        engine_traits.languages[sxng_lang] = 'lang_' + eng_lang
 
    # alias languages
    engine_traits.languages['zh'] = 'lang_zh-CN'
 
    # supported region codes
 
    for x in eval_xpath_list(dom, "//select[@name='gl']/option"):
        eng_country = x.get("value")
 
        if eng_country in skip_countries:
            continue
        if eng_country == 'ZZ':
            engine_traits.all_locale = 'ZZ'
            continue
 
        sxng_locales = get_official_locales(eng_country, engine_traits.languages.keys(), regional=True)
 
        if not sxng_locales:
            print("ERROR: can't map from google country %s (%s) to a babel region." % (x.get('data-name'), eng_country))
            continue
 
        for sxng_locale in sxng_locales:
            engine_traits.regions[region_tag(sxng_locale)] = eng_country
 
    # alias regions
    engine_traits.regions['zh-CN'] = 'HK'
 
    # supported domains
 
    if add_domains:
        resp = get('https://www.google.com/supported_domains')
        if not resp.ok:  # type: ignore
            raise RuntimeError("Response from https://www.google.com/supported_domains is not OK.")
 
        for domain in resp.text.split():  # type: ignore
            domain = domain.strip()
            if not domain or domain in [
                '.google.com',
            ]:
                continue
            region = domain.split('.')[-1].upper()
            engine_traits.custom['supported_domains'][region] = 'www' + domain  # type: ignore
            if region == 'HK':
                # There is no google.cn, we use .com.hk for zh-CN
                engine_traits.custom['supported_domains']['CN'] = 'www' + domain  # type: ignore

get_google_info()

searx.engines.google.get_google_info	(		params,
			eng_traits )

Composing various (language) properties for the google engines (:ref:`google
API`).

This function is called by the various google engines (:ref:`google web
engine`, :ref:`google images engine`, :ref:`google news engine` and
:ref:`google videos engine`).

:param dict param: Request parameters of the engine.  At least
    a ``searxng_locale`` key should be in the dictionary.

:param eng_traits: Engine's traits fetched from google preferences
    (:py:obj:`searx.enginelib.traits.EngineTraits`)

:rtype: dict
:returns:
    Py-Dictionary with the key/value pairs:

    language:
        The language code that is used by google (e.g. ``lang_en`` or
        ``lang_zh-TW``)

    country:
        The country code that is used by google (e.g. ``US`` or ``TW``)

    locale:
        A instance of :py:obj:`babel.core.Locale` build from the
        ``searxng_locale`` value.

    subdomain:
        Google subdomain :py:obj:`google_domains` that fits to the country
        code.

    params:
        Py-Dictionary with additional request arguments (can be passed to
        :py:func:`urllib.parse.urlencode`).

        - ``hl`` parameter: specifies the interface language of user interface.
        - ``lr`` parameter: restricts search results to documents written in
          a particular language.
        - ``cr`` parameter: restricts search results to documents
          originating in a particular country.
        - ``ie`` parameter: sets the character encoding scheme that should
          be used to interpret the query string ('utf8').
        - ``oe`` parameter: sets the character encoding scheme that should
          be used to decode the XML result ('utf8').

    headers:
        Py-Dictionary with additional HTTP headers (can be passed to
        request's headers)

        - ``Accept: '*/*``

Definition at line 73 of file google.py.

def get_google_info(params, eng_traits):
    """Composing various (language) properties for the google engines (:ref:`google
    API`).
 
    This function is called by the various google engines (:ref:`google web
    engine`, :ref:`google images engine`, :ref:`google news engine` and
    :ref:`google videos engine`).
 
    :param dict param: Request parameters of the engine.  At least
        a ``searxng_locale`` key should be in the dictionary.
 
    :param eng_traits: Engine's traits fetched from google preferences
        (:py:obj:`searx.enginelib.traits.EngineTraits`)
 
    :rtype: dict
    :returns:
        Py-Dictionary with the key/value pairs:
 
        language:
            The language code that is used by google (e.g. ``lang_en`` or
            ``lang_zh-TW``)
 
        country:
            The country code that is used by google (e.g. ``US`` or ``TW``)
 
        locale:
            A instance of :py:obj:`babel.core.Locale` build from the
            ``searxng_locale`` value.
 
        subdomain:
            Google subdomain :py:obj:`google_domains` that fits to the country
            code.
 
        params:
            Py-Dictionary with additional request arguments (can be passed to
            :py:func:`urllib.parse.urlencode`).
 
            - ``hl`` parameter: specifies the interface language of user interface.
            - ``lr`` parameter: restricts search results to documents written in
              a particular language.
            - ``cr`` parameter: restricts search results to documents
              originating in a particular country.
            - ``ie`` parameter: sets the character encoding scheme that should
              be used to interpret the query string ('utf8').
            - ``oe`` parameter: sets the character encoding scheme that should
              be used to decode the XML result ('utf8').
 
        headers:
            Py-Dictionary with additional HTTP headers (can be passed to
            request's headers)
 
            - ``Accept: '*/*``
 
    """
 
    ret_val = {
        'language': None,
        'country': None,
        'subdomain': None,
        'params': {},
        'headers': {},
        'cookies': {},
        'locale': None,
    }
 
    sxng_locale = params.get('searxng_locale', 'all')
    try:
        locale = babel.Locale.parse(sxng_locale, sep='-')
    except babel.core.UnknownLocaleError:
        locale = None
 
    eng_lang = eng_traits.get_language(sxng_locale, 'lang_en')
    lang_code = eng_lang.split('_')[-1]  # lang_zh-TW --> zh-TW / lang_en --> en
    country = eng_traits.get_region(sxng_locale, eng_traits.all_locale)
 
    # Test zh_hans & zh_hant --> in the topmost links in the result list of list
    # TW and HK you should a find wiktionary.org zh_hant link.  In the result
    # list of zh-CN should not be no hant link instead you should find
    # zh.m.wikipedia.org/zh somewhere in the top.
 
    # '!go 日 :zh-TW' --> https://zh.m.wiktionary.org/zh-hant/%E6%97%A5
    # '!go 日 :zh-CN' --> https://zh.m.wikipedia.org/zh/%E6%97%A5
 
    ret_val['language'] = eng_lang
    ret_val['country'] = country
    ret_val['locale'] = locale
    ret_val['subdomain'] = eng_traits.custom['supported_domains'].get(country.upper(), 'www.google.com')
 
    # hl parameter:
    #   The hl parameter specifies the interface language (host language) of
    #   your user interface. To improve the performance and the quality of your
    #   search results, you are strongly encouraged to set this parameter
    #   explicitly.
    #   https://developers.google.com/custom-search/docs/xml_results#hlsp
    # The Interface Language:
    #   https://developers.google.com/custom-search/docs/xml_results_appendices#interfaceLanguages
 
    # https://github.com/searxng/searxng/issues/2515#issuecomment-1607150817
    ret_val['params']['hl'] = f'{lang_code}-{country}'
 
    # lr parameter:
    #   The lr (language restrict) parameter restricts search results to
    #   documents written in a particular language.
    #   https://developers.google.com/custom-search/docs/xml_results#lrsp
    #   Language Collection Values:
    #   https://developers.google.com/custom-search/docs/xml_results_appendices#languageCollections
    #
    # To select 'all' languages an empty 'lr' value is used.
    #
    # Different to other google services, Google Scholar supports to select more
    # than one language. The languages are separated by a pipe '|' (logical OR).
    # By example: &lr=lang_zh-TW%7Clang_de selects articles written in
    # traditional chinese OR german language.
 
    ret_val['params']['lr'] = eng_lang
    if sxng_locale == 'all':
        ret_val['params']['lr'] = ''
 
    # cr parameter:
    #   The cr parameter restricts search results to documents originating in a
    #   particular country.
    #   https://developers.google.com/custom-search/docs/xml_results#crsp
 
    # specify a region (country) only if a region is given in the selected
    # locale --> https://github.com/searxng/searxng/issues/2672
    ret_val['params']['cr'] = ''
    if len(sxng_locale.split('-')) > 1:
        ret_val['params']['cr'] = 'country' + country
 
    # gl parameter: (mandatory by Google News)
    #   The gl parameter value is a two-letter country code. For WebSearch
    #   results, the gl parameter boosts search results whose country of origin
    #   matches the parameter value. See the Country Codes section for a list of
    #   valid values.
    #   Specifying a gl parameter value in WebSearch requests should improve the
    #   relevance of results. This is particularly true for international
    #   customers and, even more specifically, for customers in English-speaking
    #   countries other than the United States.
    #   https://developers.google.com/custom-search/docs/xml_results#glsp
 
    # https://github.com/searxng/searxng/issues/2515#issuecomment-1606294635
    # ret_val['params']['gl'] = country
 
    # ie parameter:
    #   The ie parameter sets the character encoding scheme that should be used
    #   to interpret the query string. The default ie value is latin1.
    #   https://developers.google.com/custom-search/docs/xml_results#iesp
 
    ret_val['params']['ie'] = 'utf8'
 
    # oe parameter:
    #   The oe parameter sets the character encoding scheme that should be used
    #   to decode the XML result. The default oe value is latin1.
    #   https://developers.google.com/custom-search/docs/xml_results#oesp
 
    ret_val['params']['oe'] = 'utf8'
 
    # num parameter:
    #   The num parameter identifies the number of search results to return.
    #   The default num value is 10, and the maximum value is 20. If you request
    #   more than 20 results, only 20 results will be returned.
    #   https://developers.google.com/custom-search/docs/xml_results#numsp
 
    # HINT: seems to have no effect (tested in google WEB & Images)
    # ret_val['params']['num'] = 20
 
    # HTTP headers
 
    ret_val['headers']['Accept'] = '*/*'
 
    # Cookies
 
    # - https://github.com/searxng/searxng/pull/1679#issuecomment-1235432746
    # - https://github.com/searxng/searxng/issues/1555
    ret_val['cookies']['CONSENT'] = "YES+"
 
    return ret_val
 
 

request()

searx.engines.google.request	(		query,
			params )

Google search request

Definition at line 257 of file google.py.

def request(query, params):
    """Google search request"""
    # pylint: disable=line-too-long
    offset = (params['pageno'] - 1) * 10
    google_info = get_google_info(params, traits)
 
    # https://www.google.de/search?q=corona&hl=de&lr=lang_de&start=0&tbs=qdr%3Ad&safe=medium
    query_url = (
        'https://'
        + google_info['subdomain']
        + '/search'
        + "?"
        + urlencode(
            {
                'q': query,
                **google_info['params'],
                'filter': '0',
                'start': offset,
                # 'vet': '12ahUKEwik3ZbIzfn7AhXMX_EDHbUDBh0QxK8CegQIARAC..i',
                # 'ved': '2ahUKEwik3ZbIzfn7AhXMX_EDHbUDBh0Q_skCegQIARAG',
                # 'cs' : 1,
                # 'sa': 'N',
                # 'yv': 3,
                # 'prmd': 'vin',
                # 'ei': 'GASaY6TxOcy_xc8PtYeY6AE',
                # 'sa': 'N',
                # 'sstk': 'AcOHfVkD7sWCSAheZi-0tx_09XDO55gTWY0JNq3_V26cNN-c8lfD45aZYPI8s_Bqp8s57AHz5pxchDtAGCA_cikAWSjy9kw3kgg'
                # formally known as use_mobile_ui
                'asearch': 'arc',
                'async': UI_ASYNC,
            }
        )
    )
 
    if params['time_range'] in time_range_dict:
        query_url += '&' + urlencode({'tbs': 'qdr:' + time_range_dict[params['time_range']]})
    if params['safesearch']:
        query_url += '&' + urlencode({'safe': filter_mapping[params['safesearch']]})
    params['url'] = query_url
 
    params['cookies'] = google_info['cookies']
    params['headers'].update(google_info['headers'])
    return params
 
 
# =26;[3,"dimg_ZNMiZPCqE4apxc8P3a2tuAQ_137"]a87;data:image/jpeg;base64,/9j/4AAQSkZJRgABA
# ...6T+9Nl4cnD+gr9OK8I56/tX3l86nWYw//2Q==26;

response()

searx.engines.google.response

(

resp

)

Get response from google's search request

Definition at line 318 of file google.py.

def response(resp):
    """Get response from google's search request"""
    # pylint: disable=too-many-branches, too-many-statements
    detect_google_sorry(resp)
 
    results = []
 
    # convert the text to dom
    dom = html.fromstring(resp.text)
    data_image_map = _parse_data_images(dom)
 
    # results --> answer
    answer_list = eval_xpath(dom, '//div[contains(@class, "LGOjhe")]')
    for item in answer_list:
        for bubble in eval_xpath(item, './/div[@class="nnFGuf"]'):
            bubble.drop_tree()
        results.append(
            {
                'answer': extract_text(item),
                'url': (eval_xpath(item, '../..//a/@href') + [None])[0],
            }
        )
 
    # parse results
 
    for result in eval_xpath_list(dom, './/div[contains(@jscontroller, "SC7lYd")]'):
        # pylint: disable=too-many-nested-blocks
 
        try:
            title_tag = eval_xpath_getindex(result, './/a/h3[1]', 0, default=None)
            if title_tag is None:
                # this not one of the common google results *section*
                logger.debug('ignoring item from the result_xpath list: missing title')
                continue
            title = extract_text(title_tag)
 
            url = eval_xpath_getindex(result, './/a[h3]/@href', 0, None)
            if url is None:
                logger.debug('ignoring item from the result_xpath list: missing url of title "%s"', title)
                continue
 
            content_nodes = eval_xpath(result, './/div[contains(@data-sncf, "1")]')
            for item in content_nodes:
                for script in item.xpath(".//script"):
                    script.getparent().remove(script)
 
            content = extract_text(content_nodes)
 
            if not content:
                logger.debug('ignoring item from the result_xpath list: missing content of title "%s"', title)
                continue
 
            thumbnail = content_nodes[0].xpath('.//img/@src')
            if thumbnail:
                thumbnail = thumbnail[0]
                if thumbnail.startswith('data:image'):
                    img_id = content_nodes[0].xpath('.//img/@id')
                    if img_id:
                        thumbnail = data_image_map.get(img_id[0])
            else:
                thumbnail = None
 
            results.append({'url': url, 'title': title, 'content': content, 'thumbnail': thumbnail})
 
        except Exception as e:  # pylint: disable=broad-except
            logger.error(e, exc_info=True)
            continue
 
    # parse suggestion
    for suggestion in eval_xpath_list(dom, suggestion_xpath):
        # append suggestion
        results.append({'suggestion': extract_text(suggestion)})
 
    # return results
    return results
 
 
# get supported languages from their site
 
 

References searx.engines.google._parse_data_images().

Here is the call graph for this function:

Variable Documentation

about

dict searx.engines.google.about

Initial value:

=  {
    "website": 'https://www.google.com',
    "wikidata_id": 'Q9366',
    "official_api_documentation": 'https://developers.google.com/custom-search/',
    "use_official_api": False,
    "require_api_key": False,
    "results": 'HTML',
}

Definition at line 38 of file google.py.

categories

list searx.engines.google.categories = ['general', 'web']

Definition at line 48 of file google.py.

filter_mapping

dict searx.engines.google.filter_mapping = {0: 'off', 1: 'medium', 2: 'high'}

Definition at line 57 of file google.py.

logger

logging searx.engines.google.logger .Logger

Definition at line 32 of file google.py.

max_page

int searx.engines.google.max_page = 50

Definition at line 50 of file google.py.

paging

bool searx.engines.google.paging = True

Definition at line 49 of file google.py.

RE_DATA_IMAGE

searx.engines.google.RE_DATA_IMAGE = re.compile(r'"(dimg_[^"]*)"[^;]*;(data:image[^;]*;[^;]*);')

Definition at line 304 of file google.py.

safesearch

bool searx.engines.google.safesearch = True

Definition at line 52 of file google.py.

skip_countries

list searx.engines.google.skip_countries

Initial value:

=  [
    # official language of google-country not in google-languages
    'AL',  # Albanien (sq)
    'AZ',  # Aserbaidschan  (az)
    'BD',  # Bangladesch (bn)
    'BN',  # Brunei Darussalam (ms)
    'BT',  # Bhutan (dz)
    'ET',  # Äthiopien (am)
    'GE',  # Georgien (ka, os)
    'GL',  # Grönland (kl)
    'KH',  # Kambodscha (km)
    'LA',  # Laos (lo)
    'LK',  # Sri Lanka (si, ta)
    'ME',  # Montenegro (sr)
    'MK',  # Nordmazedonien (mk, sq)
    'MM',  # Myanmar (my)
    'MN',  # Mongolei (mn)
    'MV',  # Malediven (dv) // dv_MV is unknown by babel
    'MY',  # Malaysia (ms)
    'NP',  # Nepal (ne)
    'TJ',  # Tadschikistan (tg)
    'TM',  # Turkmenistan (tk)
    'UZ',  # Usbekistan (uz)
]

Definition at line 398 of file google.py.

suggestion_xpath

str searx.engines.google.suggestion_xpath = '//div[contains(@class, "EIaa9b")]//a'

Definition at line 64 of file google.py.

time_range_dict

dict searx.engines.google.time_range_dict = {'day': 'd', 'week': 'w', 'month': 'm', 'year': 'y'}

Definition at line 54 of file google.py.

time_range_support

bool searx.engines.google.time_range_support = True

Definition at line 51 of file google.py.

UI_ASYNC

str searx.engines.google.UI_ASYNC = 'use_ac:true,_fmt:prog'

Definition at line 69 of file google.py.