searx.engines.duckduckgo Namespace Reference

Functions
	get_cache ()
str	get_vqd (str query, str region, bool force_request=False)
	set_vqd (str query, str region, str value)
	get_ddg_lang (EngineTraits eng_traits, sxng_locale, default='en_US')

Variables
logging	logger .Logger
dict	about
bool	send_accept_language_header = True
list	categories = ['general', 'web']
bool	paging = True
bool	time_range_support = True
bool	safesearch = True
str	url = "https://html.duckduckgo.com/html/"
dict	time_range_dict = {'day': 'd', 'week': 'w', 'month': 'm', 'year': 'y'}
dict	form_data = {'v': 'l', 'api': 'd.js', 'o': 'json'}
EngineCache	_CACHE = None
dict	ddg_reg_map

Detailed Description

DuckDuckGo WEB
~~~~~~~~~~~~~~

Function Documentation

get_cache()

searx.engines.duckduckgo.get_cache

(

)

Definition at line 71 of file duckduckgo.py.

def get_cache():
    global _CACHE  # pylint: disable=global-statement
    if _CACHE is None:
        _CACHE = EngineCache("duckduckgo")  # type:ignore
    return _CACHE
 
 

Referenced by get_vqd(), and set_vqd().

Here is the caller graph for this function:

get_ddg_lang()

searx.engines.duckduckgo.get_ddg_lang	(	EngineTraits	eng_traits,
			sxng_locale,
			default = 'en_US' )

Get DuckDuckGo's language identifier from SearXNG's locale.

DuckDuckGo defines its languages by region codes (see
:py:obj:`fetch_traits`).

To get region and language of a DDG service use:

.. code: python

   eng_region = traits.get_region(params['searxng_locale'], traits.all_locale)
   eng_lang = get_ddg_lang(traits, params['searxng_locale'])

It might confuse, but the ``l`` value of the cookie is what SearXNG calls
the *region*:

.. code:: python

    # !ddi paris :es-AR --> {'ad': 'es_AR', 'ah': 'ar-es', 'l': 'ar-es'}
    params['cookies']['ad'] = eng_lang
    params['cookies']['ah'] = eng_region
    params['cookies']['l'] = eng_region

.. hint::

   `DDG-lite <https://lite.duckduckgo.com/lite>`__ and the *no Javascript*
   page https://html.duckduckgo.com/html do not offer a language selection
   to the user, only a region can be selected by the user (``eng_region``
   from the example above).  DDG-lite and *no Javascript* store the selected
   region in a cookie::

     params['cookies']['kl'] = eng_region  # 'ar-es'

Definition at line 142 of file duckduckgo.py.

def get_ddg_lang(eng_traits: EngineTraits, sxng_locale, default='en_US'):
    """Get DuckDuckGo's language identifier from SearXNG's locale.
 
    DuckDuckGo defines its languages by region codes (see
    :py:obj:`fetch_traits`).
 
    To get region and language of a DDG service use:
 
    .. code: python
 
       eng_region = traits.get_region(params['searxng_locale'], traits.all_locale)
       eng_lang = get_ddg_lang(traits, params['searxng_locale'])
 
    It might confuse, but the ``l`` value of the cookie is what SearXNG calls
    the *region*:
 
    .. code:: python
 
        # !ddi paris :es-AR --> {'ad': 'es_AR', 'ah': 'ar-es', 'l': 'ar-es'}
        params['cookies']['ad'] = eng_lang
        params['cookies']['ah'] = eng_region
        params['cookies']['l'] = eng_region
 
    .. hint::
 
       `DDG-lite <https://lite.duckduckgo.com/lite>`__ and the *no Javascript*
       page https://html.duckduckgo.com/html do not offer a language selection
       to the user, only a region can be selected by the user (``eng_region``
       from the example above).  DDG-lite and *no Javascript* store the selected
       region in a cookie::
 
         params['cookies']['kl'] = eng_region  # 'ar-es'
 
    """
    return eng_traits.custom['lang_region'].get(  # type: ignore
        sxng_locale, eng_traits.get_language(sxng_locale, default)
    )
 
 

get_vqd()

str searx.engines.duckduckgo.get_vqd	(	str	query,
		str	region,
		bool	force_request = False )

Returns the ``vqd`` that fits to the *query*.

:param query: The query term
:param region: DDG's region code
:param force_request: force a request to get a vqd value from DDG

TL;DR; the ``vqd`` value is needed to pass DDG's bot protection and is used
by all request to DDG:

- DuckDuckGo Lite: ``https://lite.duckduckgo.com/lite`` (POST form data)
- DuckDuckGo Web: ``https://links.duckduckgo.com/d.js?q=...&vqd=...``
- DuckDuckGo Images: ``https://duckduckgo.com/i.js??q=...&vqd=...``
- DuckDuckGo Videos: ``https://duckduckgo.com/v.js??q=...&vqd=...``
- DuckDuckGo News: ``https://duckduckgo.com/news.js??q=...&vqd=...``

DDG's bot detection is sensitive to the ``vqd`` value.  For some search terms
(such as extremely long search terms that are often sent by bots), no ``vqd``
value can be determined.

If SearXNG cannot determine a ``vqd`` value, then no request should go out
to DDG.

.. attention::

   A request with a wrong ``vqd`` value leads to DDG temporarily putting
   SearXNG's IP on a block list.

Requests from IPs in this block list run into timeouts.  Not sure, but it
seems the block list is a sliding window: to get my IP rid from the bot list
I had to cool down my IP for 1h (send no requests from that IP to DDG).

Definition at line 78 of file duckduckgo.py.

def get_vqd(query: str, region: str, force_request: bool = False) -> str:
    """Returns the ``vqd`` that fits to the *query*.
 
    :param query: The query term
    :param region: DDG's region code
    :param force_request: force a request to get a vqd value from DDG
 
    TL;DR; the ``vqd`` value is needed to pass DDG's bot protection and is used
    by all request to DDG:
 
    - DuckDuckGo Lite: ``https://lite.duckduckgo.com/lite`` (POST form data)
    - DuckDuckGo Web: ``https://links.duckduckgo.com/d.js?q=...&vqd=...``
    - DuckDuckGo Images: ``https://duckduckgo.com/i.js??q=...&vqd=...``
    - DuckDuckGo Videos: ``https://duckduckgo.com/v.js??q=...&vqd=...``
    - DuckDuckGo News: ``https://duckduckgo.com/news.js??q=...&vqd=...``
 
    DDG's bot detection is sensitive to the ``vqd`` value.  For some search terms
    (such as extremely long search terms that are often sent by bots), no ``vqd``
    value can be determined.
 
    If SearXNG cannot determine a ``vqd`` value, then no request should go out
    to DDG.
 
    .. attention::
 
       A request with a wrong ``vqd`` value leads to DDG temporarily putting
       SearXNG's IP on a block list.
 
    Requests from IPs in this block list run into timeouts.  Not sure, but it
    seems the block list is a sliding window: to get my IP rid from the bot list
    I had to cool down my IP for 1h (send no requests from that IP to DDG).
    """
    cache = get_cache()
    key = cache.secret_hash(f"{query}//{region}")
    value = cache.get(key=key)
    if value is not None and not force_request:
        logger.debug("vqd: re-use cached value: %s", value)
        return value
 
    logger.debug("vqd: request value from from duckduckgo.com")
    resp = get(f'https://duckduckgo.com/?q={quote_plus(query)}')
    if resp.status_code == 200:  # type: ignore
        value = extr(resp.text, 'vqd="', '"')  # type: ignore
        if value:
            logger.debug("vqd value from duckduckgo.com request: '%s'", value)
        else:
            logger.error("vqd: can't parse value from ddg response (return empty string)")
            return ""
    else:
        logger.error("vqd: got HTTP %s from duckduckgo.com", resp.status_code)
 
    if value:
        cache.set(key=key, value=value)
    else:
        logger.error("vqd value from duckduckgo.com ", resp.status_code)
    return value
 
 

References get_cache().

Here is the call graph for this function:

set_vqd()

searx.engines.duckduckgo.set_vqd	(	str	query,
		str	region,
		str	value )

Definition at line 136 of file duckduckgo.py.

def set_vqd(query: str, region: str, value: str):
    cache = get_cache()
    key = cache.secret_hash(f"{query}//{region}")
    cache.set(key=key, value=value, expire=3600)
 
 

References get_cache().

Here is the call graph for this function:

Variable Documentation

_CACHE

EngineCache searx.engines.duckduckgo._CACHE = None

protected

Definition at line 66 of file duckduckgo.py.

about

dict searx.engines.duckduckgo.about

Initial value:

=  {
    "website": 'https://lite.duckduckgo.com/lite/',
    "wikidata_id": 'Q12805',
    "use_official_api": False,
    "require_api_key": False,
    "results": 'HTML',
}

Definition at line 41 of file duckduckgo.py.

categories

list searx.engines.duckduckgo.categories = ['general', 'web']

Definition at line 56 of file duckduckgo.py.

ddg_reg_map

dict searx.engines.duckduckgo.ddg_reg_map

Initial value:

=  {
    'tw-tzh': 'zh_TW',
    'hk-tzh': 'zh_HK',
    'ct-ca': 'skip',  # ct-ca and es-ca both map to ca_ES
    'es-ca': 'ca_ES',
    'id-en': 'id_ID',
    'no-no': 'nb_NO',
    'jp-jp': 'ja_JP',
    'kr-kr': 'ko_KR',
    'xa-ar': 'ar_SA',
    'sl-sl': 'sl_SI',
    'th-en': 'th_TH',
    'vn-en': 'vi_VN',
}

Definition at line 181 of file duckduckgo.py.

form_data

dict searx.engines.duckduckgo.form_data = {'v': 'l', 'api': 'd.js', 'o': 'json'}

Definition at line 64 of file duckduckgo.py.

logger

logging searx.engines.duckduckgo.logger .Logger

Definition at line 37 of file duckduckgo.py.

paging

bool searx.engines.duckduckgo.paging = True

Definition at line 57 of file duckduckgo.py.

safesearch

bool searx.engines.duckduckgo.safesearch = True

Definition at line 59 of file duckduckgo.py.

send_accept_language_header

bool searx.engines.duckduckgo.send_accept_language_header = True

Definition at line 49 of file duckduckgo.py.

time_range_dict

dict searx.engines.duckduckgo.time_range_dict = {'day': 'd', 'week': 'w', 'month': 'm', 'year': 'y'}

Definition at line 63 of file duckduckgo.py.

time_range_support

bool searx.engines.duckduckgo.time_range_support = True

Definition at line 58 of file duckduckgo.py.

url

str searx.engines.duckduckgo.url = "https://html.duckduckgo.com/html/"

Definition at line 61 of file duckduckgo.py.