.oO SearXNG Developer Documentation Oo.
Loading...
Searching...
No Matches
searx.engines.wikipedia Namespace Reference

Functions

 get_wiki_params (sxng_locale, eng_traits)
 request (query, params)
 response (resp)
 fetch_traits (EngineTraits engine_traits)
 fetch_wikimedia_traits (EngineTraits engine_traits)

Variables

dict about
list display_type = ["infobox"]
bool send_accept_language_header = True
str list_of_wikipedias = 'https://meta.wikimedia.org/wiki/List_of_Wikipedias'
str wikipedia_article_depth = 'https://meta.wikimedia.org/wiki/Wikipedia_article_depth'
str rest_v1_summary_url = 'https://{wiki_netloc}/api/rest_v1/page/summary/{title}'
dict wiki_lc_locale_variants
dict wikipedia_script_variants
 lang_map = locales.LOCALE_BEST_MATCH.copy()

Detailed Description

This module implements the Wikipedia engine.  Some of this implementations
are shared by other engines:

- :ref:`wikidata engine`

The list of supported languages is :py:obj:`fetched <fetch_wikimedia_traits>` from
the article linked by :py:obj:`list_of_wikipedias`.

Unlike traditional search engines, wikipedia does not support one Wikipedia for
all languages, but there is one Wikipedia for each supported language. Some of
these Wikipedias have a LanguageConverter_ enabled
(:py:obj:`rest_v1_summary_url`).

A LanguageConverter_ (LC) is a system based on language variants that
automatically converts the content of a page into a different variant. A variant
is mostly the same language in a different script.

- `Wikipedias in multiple writing systems`_
- `Automatic conversion between traditional and simplified Chinese characters`_

PR-2554_:
  The Wikipedia link returned by the API is still the same in all cases
  (`https://zh.wikipedia.org/wiki/出租車`_) but if your browser's
  ``Accept-Language`` is set to any of ``zh``, ``zh-CN``, ``zh-TW``, ``zh-HK``
  or .. Wikipedia's LC automatically returns the desired script in their
  web-page.

  - You can test the API here: https://reqbin.com/gesg2kvx

.. _https://zh.wikipedia.org/wiki/出租車:
   https://zh.wikipedia.org/wiki/%E5%87%BA%E7%A7%9F%E8%BB%8A

To support Wikipedia's LanguageConverter_, a SearXNG request to Wikipedia uses
:py:obj:`get_wiki_params` and :py:obj:`wiki_lc_locale_variants' in the
:py:obj:`fetch_wikimedia_traits` function.

To test in SearXNG, query for ``!wp 出租車`` with each of the available Chinese
options:

- ``!wp 出租車 :zh``    should show 出租車
- ``!wp 出租車 :zh-CN`` should show 出租车
- ``!wp 出租車 :zh-TW`` should show 計程車
- ``!wp 出租車 :zh-HK`` should show 的士
- ``!wp 出租車 :zh-SG`` should show 德士

.. _LanguageConverter:
   https://www.mediawiki.org/wiki/Writing_systems#LanguageConverter
.. _Wikipedias in multiple writing systems:
   https://meta.wikimedia.org/wiki/Wikipedias_in_multiple_writing_systems
.. _Automatic conversion between traditional and simplified Chinese characters:
   https://en.wikipedia.org/wiki/Chinese_Wikipedia#Automatic_conversion_between_traditional_and_simplified_Chinese_characters
.. _PR-2554: https://github.com/searx/searx/pull/2554

Function Documentation

◆ fetch_traits()

searx.engines.wikipedia.fetch_traits ( EngineTraits engine_traits)

Definition at line 238 of file wikipedia.py.

238def fetch_traits(engine_traits: EngineTraits):
239 fetch_wikimedia_traits(engine_traits)
240 print("WIKIPEDIA_LANGUAGES: %s" % len(engine_traits.custom['WIKIPEDIA_LANGUAGES']))
241
242

References fetch_wikimedia_traits().

Here is the call graph for this function:

◆ fetch_wikimedia_traits()

searx.engines.wikipedia.fetch_wikimedia_traits ( EngineTraits engine_traits)
Fetch languages from Wikipedia.  Not all languages from the
:py:obj:`list_of_wikipedias` are supported by SearXNG locales, only those
known from :py:obj:`searx.locales.LOCALE_NAMES` or those with a minimal
:py:obj:`editing depth <wikipedia_article_depth>`.

The location of the Wikipedia address of a language is mapped in a
:py:obj:`custom field <searx.enginelib.traits.EngineTraits.custom>`
(``wiki_netloc``).  Here is a reduced example:

.. code:: python

   traits.custom['wiki_netloc'] = {
       "en": "en.wikipedia.org",
       ..
       "gsw": "als.wikipedia.org",
       ..
       "zh": "zh.wikipedia.org",
       "zh-classical": "zh-classical.wikipedia.org"
   }

Definition at line 243 of file wikipedia.py.

243def fetch_wikimedia_traits(engine_traits: EngineTraits):
244 """Fetch languages from Wikipedia. Not all languages from the
245 :py:obj:`list_of_wikipedias` are supported by SearXNG locales, only those
246 known from :py:obj:`searx.locales.LOCALE_NAMES` or those with a minimal
247 :py:obj:`editing depth <wikipedia_article_depth>`.
248
249 The location of the Wikipedia address of a language is mapped in a
250 :py:obj:`custom field <searx.enginelib.traits.EngineTraits.custom>`
251 (``wiki_netloc``). Here is a reduced example:
252
253 .. code:: python
254
255 traits.custom['wiki_netloc'] = {
256 "en": "en.wikipedia.org",
257 ..
258 "gsw": "als.wikipedia.org",
259 ..
260 "zh": "zh.wikipedia.org",
261 "zh-classical": "zh-classical.wikipedia.org"
262 }
263 """
264 # pylint: disable=too-many-branches
265 engine_traits.custom['wiki_netloc'] = {}
266 engine_traits.custom['WIKIPEDIA_LANGUAGES'] = []
267
268 # insert alias to map from a script or region to a wikipedia variant
269
270 for eng_tag, sxng_tag_list in wikipedia_script_variants.items():
271 for sxng_tag in sxng_tag_list:
272 engine_traits.languages[sxng_tag] = eng_tag
273 for eng_tag, sxng_tag_list in wiki_lc_locale_variants.items():
274 for sxng_tag in sxng_tag_list:
275 engine_traits.regions[sxng_tag] = eng_tag
276
277 resp = _network.get(list_of_wikipedias)
278 if not resp.ok:
279 print("ERROR: response from Wikipedia is not OK.")
280
281 dom = html.fromstring(resp.text)
282 for row in dom.xpath('//table[contains(@class,"sortable")]//tbody/tr'):
283
284 cols = row.xpath('./td')
285 if not cols:
286 continue
287 cols = [c.text_content().strip() for c in cols]
288
289 depth = float(cols[11].replace('-', '0').replace(',', ''))
290 articles = int(cols[4].replace(',', '').replace(',', ''))
291
292 eng_tag = cols[3]
293 wiki_url = row.xpath('./td[4]/a/@href')[0]
294 wiki_url = urllib.parse.urlparse(wiki_url)
295
296 try:
297 sxng_tag = locales.language_tag(babel.Locale.parse(lang_map.get(eng_tag, eng_tag), sep='-'))
298 except babel.UnknownLocaleError:
299 # print("ERROR: %s [%s] is unknown by babel" % (cols[0], eng_tag))
300 continue
301 finally:
302 engine_traits.custom['WIKIPEDIA_LANGUAGES'].append(eng_tag)
303
304 if sxng_tag not in locales.LOCALE_NAMES:
305
306 if articles < 10000:
307 # exclude languages with too few articles
308 continue
309
310 if int(depth) < 20:
311 # Rough indicator of a Wikipedia’s quality, showing how
312 # frequently its articles are updated.
313 continue
314
315 conflict = engine_traits.languages.get(sxng_tag)
316 if conflict:
317 if conflict != eng_tag:
318 print("CONFLICT: babel %s --> %s, %s" % (sxng_tag, conflict, eng_tag))
319 continue
320
321 engine_traits.languages[sxng_tag] = eng_tag
322 engine_traits.custom['wiki_netloc'][eng_tag] = wiki_url.netloc
323
324 engine_traits.custom['WIKIPEDIA_LANGUAGES'].sort()

Referenced by fetch_traits().

Here is the caller graph for this function:

◆ get_wiki_params()

searx.engines.wikipedia.get_wiki_params ( sxng_locale,
eng_traits )
Returns the Wikipedia language tag and the netloc that fits to the
``sxng_locale``.  To support LanguageConverter_ this function rates a locale
(region) higher than a language (compare :py:obj:`wiki_lc_locale_variants`).

Definition at line 141 of file wikipedia.py.

141def get_wiki_params(sxng_locale, eng_traits):
142 """Returns the Wikipedia language tag and the netloc that fits to the
143 ``sxng_locale``. To support LanguageConverter_ this function rates a locale
144 (region) higher than a language (compare :py:obj:`wiki_lc_locale_variants`).
145
146 """
147 eng_tag = eng_traits.get_region(sxng_locale, eng_traits.get_language(sxng_locale, 'en'))
148 wiki_netloc = eng_traits.custom['wiki_netloc'].get(eng_tag, 'en.wikipedia.org')
149 return eng_tag, wiki_netloc
150
151

Referenced by request().

Here is the caller graph for this function:

◆ request()

searx.engines.wikipedia.request ( query,
params )
Assemble a request (`wikipedia rest_v1 summary API`_).

Definition at line 152 of file wikipedia.py.

152def request(query, params):
153 """Assemble a request (`wikipedia rest_v1 summary API`_)."""
154 if query.islower():
155 query = query.title()
156
157 _eng_tag, wiki_netloc = get_wiki_params(params['searxng_locale'], traits)
158 title = urllib.parse.quote(query)
159 params['url'] = rest_v1_summary_url.format(wiki_netloc=wiki_netloc, title=title)
160
161 params['raise_for_httperror'] = False
162 params['soft_max_redirects'] = 2
163
164 return params
165
166
167# get response from search-request

References get_wiki_params().

Here is the call graph for this function:

◆ response()

searx.engines.wikipedia.response ( resp)

Definition at line 168 of file wikipedia.py.

168def response(resp):
169
170 results = []
171 if resp.status_code == 404:
172 return []
173 if resp.status_code == 400:
174 try:
175 api_result = resp.json()
176 except Exception: # pylint: disable=broad-except
177 pass
178 else:
179 if (
180 api_result['type'] == 'https://mediawiki.org/wiki/HyperSwitch/errors/bad_request'
181 and api_result['detail'] == 'title-invalid-characters'
182 ):
183 return []
184
185 _network.raise_for_httperror(resp)
186
187 api_result = resp.json()
188 title = utils.html_to_text(api_result.get('titles', {}).get('display') or api_result.get('title'))
189 wikipedia_link = api_result['content_urls']['desktop']['page']
190
191 if "list" in display_type or api_result.get('type') != 'standard':
192 # show item in the result list if 'list' is in the display options or it
193 # is a item that can't be displayed in a infobox.
194 results.append({'url': wikipedia_link, 'title': title, 'content': api_result.get('description', '')})
195
196 if "infobox" in display_type:
197 if api_result.get('type') == 'standard':
198 results.append(
199 {
200 'infobox': title,
201 'id': wikipedia_link,
202 'content': api_result.get('extract', ''),
203 'img_src': api_result.get('thumbnail', {}).get('source'),
204 'urls': [{'title': 'Wikipedia', 'url': wikipedia_link}],
205 }
206 )
207
208 return results
209
210
211# Nonstandard language codes
212#
213# These Wikipedias use language codes that do not conform to the ISO 639
214# standard (which is how wiki subdomains are chosen nowadays).
215

Variable Documentation

◆ about

dict searx.engines.wikipedia.about
Initial value:
1= {
2 "website": 'https://www.wikipedia.org/',
3 "wikidata_id": 'Q52',
4 "official_api_documentation": 'https://en.wikipedia.org/api/',
5 "use_official_api": True,
6 "require_api_key": False,
7 "results": 'JSON',
8}

Definition at line 68 of file wikipedia.py.

◆ display_type

list searx.engines.wikipedia.display_type = ["infobox"]

Definition at line 77 of file wikipedia.py.

◆ lang_map

searx.engines.wikipedia.lang_map = locales.LOCALE_BEST_MATCH.copy()

Definition at line 216 of file wikipedia.py.

◆ list_of_wikipedias

str searx.engines.wikipedia.list_of_wikipedias = 'https://meta.wikimedia.org/wiki/List_of_Wikipedias'

Definition at line 86 of file wikipedia.py.

◆ rest_v1_summary_url

str searx.engines.wikipedia.rest_v1_summary_url = 'https://{wiki_netloc}/api/rest_v1/page/summary/{title}'

Definition at line 97 of file wikipedia.py.

◆ send_accept_language_header

bool searx.engines.wikipedia.send_accept_language_header = True

Definition at line 82 of file wikipedia.py.

◆ wiki_lc_locale_variants

dict searx.engines.wikipedia.wiki_lc_locale_variants
Initial value:
1= {
2 "zh": (
3 "zh-CN",
4 "zh-HK",
5 "zh-MO",
6 "zh-MY",
7 "zh-SG",
8 "zh-TW",
9 ),
10 "zh-classical": ("zh-classical",),
11}

Definition at line 114 of file wikipedia.py.

◆ wikipedia_article_depth

str searx.engines.wikipedia.wikipedia_article_depth = 'https://meta.wikimedia.org/wiki/Wikipedia_article_depth'

Definition at line 90 of file wikipedia.py.

◆ wikipedia_script_variants

dict searx.engines.wikipedia.wikipedia_script_variants
Initial value:
1= {
2 "zh": (
3 "zh_Hant",
4 "zh_Hans",
5 )
6}

Definition at line 133 of file wikipedia.py.