.oO SearXNG Developer Documentation Oo.
Loading...
Searching...
No Matches
Functions | Variables
searx.engines.duckduckgo Namespace Reference

Functions

 cache_vqd (query, value)
 
 get_vqd (query)
 
 get_ddg_lang (EngineTraits eng_traits, sxng_locale, default='en_US')
 

Variables

logging logger .Logger
 
EngineTraits traits
 
dict about
 
bool send_accept_language_header = True
 
list categories = ['general', 'web']
 
bool paging = True
 
bool time_range_support = True
 
bool safesearch = True
 
str url = 'https://lite.duckduckgo.com/lite/'
 
dict time_range_dict = {'day': 'd', 'week': 'w', 'month': 'm', 'year': 'y'}
 
dict form_data = {'v': 'l', 'api': 'd.js', 'o': 'json'}
 
dict ddg_reg_map
 

Detailed Description

DuckDuckGo Lite
~~~~~~~~~~~~~~~

Function Documentation

◆ cache_vqd()

searx.engines.duckduckgo.cache_vqd ( query,
value ) 
Caches a ``vqd`` value from a query.

Definition at line 62 of file duckduckgo.py.

62def cache_vqd(query, value):
63 """Caches a ``vqd`` value from a query."""
64 c = redisdb.client()
65 if c:
66 logger.debug("cache vqd value: %s", value)
67 key = 'SearXNG_ddg_web_vqd' + redislib.secret_hash(query)
68 c.set(key, value, ex=600)
69
70

Referenced by searx.engines.duckduckgo.get_vqd().

+ Here is the caller graph for this function:

◆ get_ddg_lang()

searx.engines.duckduckgo.get_ddg_lang ( EngineTraits eng_traits,
sxng_locale,
default = 'en_US' ) 
Get DuckDuckGo's language identifier from SearXNG's locale.

DuckDuckGo defines its languages by region codes (see
:py:obj:`fetch_traits`).

To get region and language of a DDG service use:

.. code: python

   eng_region = traits.get_region(params['searxng_locale'], traits.all_locale)
   eng_lang = get_ddg_lang(traits, params['searxng_locale'])

It might confuse, but the ``l`` value of the cookie is what SearXNG calls
the *region*:

.. code:: python

    # !ddi paris :es-AR --> {'ad': 'es_AR', 'ah': 'ar-es', 'l': 'ar-es'}
    params['cookies']['ad'] = eng_lang
    params['cookies']['ah'] = eng_region
    params['cookies']['l'] = eng_region

.. hint::

   `DDG-lite <https://lite.duckduckgo.com/lite>`__ does not offer a language
   selection to the user, only a region can be selected by the user
   (``eng_region`` from the example above).  DDG-lite stores the selected
   region in a cookie::

     params['cookies']['kl'] = eng_region  # 'ar-es'

Definition at line 132 of file duckduckgo.py.

132def get_ddg_lang(eng_traits: EngineTraits, sxng_locale, default='en_US'):
133 """Get DuckDuckGo's language identifier from SearXNG's locale.
134
135 DuckDuckGo defines its languages by region codes (see
136 :py:obj:`fetch_traits`).
137
138 To get region and language of a DDG service use:
139
140 .. code: python
141
142 eng_region = traits.get_region(params['searxng_locale'], traits.all_locale)
143 eng_lang = get_ddg_lang(traits, params['searxng_locale'])
144
145 It might confuse, but the ``l`` value of the cookie is what SearXNG calls
146 the *region*:
147
148 .. code:: python
149
150 # !ddi paris :es-AR --> {'ad': 'es_AR', 'ah': 'ar-es', 'l': 'ar-es'}
151 params['cookies']['ad'] = eng_lang
152 params['cookies']['ah'] = eng_region
153 params['cookies']['l'] = eng_region
154
155 .. hint::
156
157 `DDG-lite <https://lite.duckduckgo.com/lite>`__ does not offer a language
158 selection to the user, only a region can be selected by the user
159 (``eng_region`` from the example above). DDG-lite stores the selected
160 region in a cookie::
161
162 params['cookies']['kl'] = eng_region # 'ar-es'
163
164 """
165 return eng_traits.custom['lang_region'].get( # type: ignore
166 sxng_locale, eng_traits.get_language(sxng_locale, default)
167 )
168
169

◆ get_vqd()

searx.engines.duckduckgo.get_vqd ( query) 
Returns the ``vqd`` that fits to the *query*.  If there is no ``vqd`` cached
(:py:obj:`cache_vqd`) the query is sent to DDG to get a vqd value from the
response.

.. hint::

   If an empty string is returned there are no results for the ``query`` and
   therefore no ``vqd`` value.

DDG's bot detection is sensitive to the ``vqd`` value.  For some search terms
(such as extremely long search terms that are often sent by bots), no ``vqd``
value can be determined.

If SearXNG cannot determine a ``vqd`` value, then no request should go out
to DDG:

    A request with a wrong ``vqd`` value leads to DDG temporarily putting
    SearXNG's IP on a block list.

    Requests from IPs in this block list run into timeouts.

Not sure, but it seems the block list is a sliding window: to get my IP rid
from the bot list I had to cool down my IP for 1h (send no requests from
that IP to DDG).

TL;DR; the ``vqd`` value is needed to pass DDG's bot protection and is used
by all request to DDG:

- DuckDuckGo Lite: ``https://lite.duckduckgo.com/lite`` (POST form data)
- DuckDuckGo Web: ``https://links.duckduckgo.com/d.js?q=...&vqd=...``
- DuckDuckGo Images: ``https://duckduckgo.com/i.js??q=...&vqd=...``
- DuckDuckGo Videos: ``https://duckduckgo.com/v.js??q=...&vqd=...``
- DuckDuckGo News: ``https://duckduckgo.com/news.js??q=...&vqd=...``

Definition at line 71 of file duckduckgo.py.

71def get_vqd(query):
72 """Returns the ``vqd`` that fits to the *query*. If there is no ``vqd`` cached
73 (:py:obj:`cache_vqd`) the query is sent to DDG to get a vqd value from the
74 response.
75
76 .. hint::
77
78 If an empty string is returned there are no results for the ``query`` and
79 therefore no ``vqd`` value.
80
81 DDG's bot detection is sensitive to the ``vqd`` value. For some search terms
82 (such as extremely long search terms that are often sent by bots), no ``vqd``
83 value can be determined.
84
85 If SearXNG cannot determine a ``vqd`` value, then no request should go out
86 to DDG:
87
88 A request with a wrong ``vqd`` value leads to DDG temporarily putting
89 SearXNG's IP on a block list.
90
91 Requests from IPs in this block list run into timeouts.
92
93 Not sure, but it seems the block list is a sliding window: to get my IP rid
94 from the bot list I had to cool down my IP for 1h (send no requests from
95 that IP to DDG).
96
97 TL;DR; the ``vqd`` value is needed to pass DDG's bot protection and is used
98 by all request to DDG:
99
100 - DuckDuckGo Lite: ``https://lite.duckduckgo.com/lite`` (POST form data)
101 - DuckDuckGo Web: ``https://links.duckduckgo.com/d.js?q=...&vqd=...``
102 - DuckDuckGo Images: ``https://duckduckgo.com/i.js??q=...&vqd=...``
103 - DuckDuckGo Videos: ``https://duckduckgo.com/v.js??q=...&vqd=...``
104 - DuckDuckGo News: ``https://duckduckgo.com/news.js??q=...&vqd=...``
105
106 """
107 value = None
108 c = redisdb.client()
109 if c:
110 key = 'SearXNG_ddg_web_vqd' + redislib.secret_hash(query)
111 value = c.get(key)
112 if value or value == b'':
113 value = value.decode('utf-8')
114 logger.debug("re-use cached vqd value: %s", value)
115 return value
116
117 query_url = 'https://duckduckgo.com/?' + urlencode({'q': query})
118 res = get(query_url)
119 doc = lxml.html.fromstring(res.text)
120 for script in doc.xpath("//script[@type='text/javascript']"):
121 script = script.text
122 if 'vqd="' in script:
123 value = script[script.index('vqd="') + 5 :]
124 value = value[: value.index('"')]
125 break
126 logger.debug("new vqd value: '%s'", value)
127 if value is not None:
128 cache_vqd(query, value)
129 return value
130
131

References searx.engines.duckduckgo.cache_vqd().

+ Here is the call graph for this function:

Variable Documentation

◆ about

dict searx.engines.duckduckgo.about
Initial value:
1= {
2 "website": 'https://lite.duckduckgo.com/lite/',
3 "wikidata_id": 'Q12805',
4 "use_official_api": False,
5 "require_api_key": False,
6 "results": 'HTML',
7}

Definition at line 35 of file duckduckgo.py.

◆ categories

list searx.engines.duckduckgo.categories = ['general', 'web']

Definition at line 50 of file duckduckgo.py.

◆ ddg_reg_map

dict searx.engines.duckduckgo.ddg_reg_map
Initial value:
1= {
2 'tw-tzh': 'zh_TW',
3 'hk-tzh': 'zh_HK',
4 'ct-ca': 'skip', # ct-ca and es-ca both map to ca_ES
5 'es-ca': 'ca_ES',
6 'id-en': 'id_ID',
7 'no-no': 'nb_NO',
8 'jp-jp': 'ja_JP',
9 'kr-kr': 'ko_KR',
10 'xa-ar': 'ar_SA',
11 'sl-sl': 'sl_SI',
12 'th-en': 'th_TH',
13 'vn-en': 'vi_VN',
14}

Definition at line 170 of file duckduckgo.py.

◆ form_data

dict searx.engines.duckduckgo.form_data = {'v': 'l', 'api': 'd.js', 'o': 'json'}

Definition at line 59 of file duckduckgo.py.

◆ logger

logging searx.engines.duckduckgo.logger .Logger

Definition at line 31 of file duckduckgo.py.

◆ paging

bool searx.engines.duckduckgo.paging = True

Definition at line 51 of file duckduckgo.py.

◆ safesearch

bool searx.engines.duckduckgo.safesearch = True

Definition at line 53 of file duckduckgo.py.

◆ send_accept_language_header

bool searx.engines.duckduckgo.send_accept_language_header = True

Definition at line 43 of file duckduckgo.py.

◆ time_range_dict

dict searx.engines.duckduckgo.time_range_dict = {'day': 'd', 'week': 'w', 'month': 'm', 'year': 'y'}

Definition at line 58 of file duckduckgo.py.

◆ time_range_support

bool searx.engines.duckduckgo.time_range_support = True

Definition at line 52 of file duckduckgo.py.

◆ traits

EngineTraits searx.engines.duckduckgo.traits

Definition at line 33 of file duckduckgo.py.

◆ url

str searx.engines.duckduckgo.url = 'https://lite.duckduckgo.com/lite/'

Definition at line 55 of file duckduckgo.py.