Classes
class	WDAmountAttribute

class	WDArticle

class	WDAttribute

class	WDDateAttribute

class	WDGeoAttribute

class	WDImageAttribute

class	WDLabelAttribute

class	WDURLAttribute

Functions
	get_headers ()

	get_label_for_entity (entity_id, language)

	send_wikidata_query (query, method='GET')

	request (query, params)

	response (resp)

	get_thumbnail (img_src)

	get_results (attribute_result, attributes, language)

	get_query (query, language)

	get_attributes (language)

	debug_explain_wikidata_query (query, method='GET')

	init (engine_settings=None)

	fetch_traits (EngineTraits engine_traits)

Variables
logging	logger .Logger

dict	about

list	display_type = ["infobox"]

str	SPARQL_ENDPOINT_URL = 'https://query.wikidata.org/sparql'

str	SPARQL_EXPLAIN_URL = 'https://query.wikidata.org/bigdata/namespace/wdq/sparql?explain'

dict	WIKIDATA_PROPERTIES

str	QUERY_TEMPLATE

str	QUERY_PROPERTY_NAMES

	DUMMY_ENTITY_URLS

	sparql_string_escape

	replace_http_by_https = get_string_replaces_function({'http:': 'https:'})

str	_IMG_SRC_DEFAULT_URL_PREFIX = "https://commons.wikimedia.org/wiki/Special:FilePath/"

str	_IMG_SRC_NEW_URL_PREFIX = "https://upload.wikimedia.org/wikipedia/commons/thumb/"

Detailed Description

This module implements the Wikidata engine.  Some implementations are shared
from :ref:`wikipedia engine`.

Function Documentation

◆ debug_explain_wikidata_query()

searx.engines.wikidata.debug_explain_wikidata_query	(		query,
			method = 'GET' )

Definition at line 754 of file wikidata.py.

def debug_explain_wikidata_query(query, method='GET'):
    if method == 'GET':
        http_response = get(SPARQL_EXPLAIN_URL + '&' + urlencode({'query': query}), headers=get_headers())
    else:
        http_response = post(SPARQL_EXPLAIN_URL, data={'query': query}, headers=get_headers())
    http_response.raise_for_status()
    return http_response.content
 
 

References searx.engines.wikidata.get_headers().

Here is the call graph for this function:

◆ fetch_traits()

searx.engines.wikidata.fetch_traits ( EngineTraits engine_traits )

Uses languages evaluated from :py:obj:`wikipedia.fetch_wikimedia_traits
<searx.engines.wikipedia.fetch_wikimedia_traits>` and removes

- ``traits.custom['wiki_netloc']``: wikidata does not have net-locations for
  the languages and the list of all

- ``traits.custom['WIKIPEDIA_LANGUAGES']``: not used in the wikipedia engine

Definition at line 783 of file wikidata.py.

def fetch_traits(engine_traits: EngineTraits):
    """Uses languages evaluated from :py:obj:`wikipedia.fetch_wikimedia_traits
    <searx.engines.wikipedia.fetch_wikimedia_traits>` and removes
 
    - ``traits.custom['wiki_netloc']``: wikidata does not have net-locations for
      the languages and the list of all
 
    - ``traits.custom['WIKIPEDIA_LANGUAGES']``: not used in the wikipedia engine
 
    """
 
    fetch_wikimedia_traits(engine_traits)
    engine_traits.custom['wiki_netloc'] = {}
    engine_traits.custom['WIKIPEDIA_LANGUAGES'] = []

◆ get_attributes()

searx.engines.wikidata.get_attributes ( language )

Definition at line 353 of file wikidata.py.

def get_attributes(language):
    # pylint: disable=too-many-statements
    attributes = []
 
    def add_value(name):
        attributes.append(WDAttribute(name))
 
    def add_amount(name):
        attributes.append(WDAmountAttribute(name))
 
    def add_label(name):
        attributes.append(WDLabelAttribute(name))
 
    def add_url(name, url_id=None, **kwargs):
        attributes.append(WDURLAttribute(name, url_id, kwargs))
 
    def add_image(name, url_id=None, priority=1):
        attributes.append(WDImageAttribute(name, url_id, priority))
 
    def add_date(name):
        attributes.append(WDDateAttribute(name))
 
    # Dates
    for p in [
        'P571',  # inception date
        'P576',  # dissolution date
        'P580',  # start date
        'P582',  # end date
        'P569',  # date of birth
        'P570',  # date of death
        'P619',  # date of spacecraft launch
        'P620',
    ]:  # date of spacecraft landing
        add_date(p)
 
    for p in [
        'P27',  # country of citizenship
        'P495',  # country of origin
        'P17',  # country
        'P159',
    ]:  # headquarters location
        add_label(p)
 
    # Places
    for p in [
        'P36',  # capital
        'P35',  # head of state
        'P6',  # head of government
        'P122',  # basic form of government
        'P37',
    ]:  # official language
        add_label(p)
 
    add_value('P1082')  # population
    add_amount('P2046')  # area
    add_amount('P281')  # postal code
    add_label('P38')  # currency
    add_amount('P2048')  # height (building)
 
    # Media
    for p in [
        'P400',  # platform (videogames, computing)
        'P50',  # author
        'P170',  # creator
        'P57',  # director
        'P175',  # performer
        'P178',  # developer
        'P162',  # producer
        'P176',  # manufacturer
        'P58',  # screenwriter
        'P272',  # production company
        'P264',  # record label
        'P123',  # publisher
        'P449',  # original network
        'P750',  # distributed by
        'P86',
    ]:  # composer
        add_label(p)
 
    add_date('P577')  # publication date
    add_label('P136')  # genre (music, film, artistic...)
    add_label('P364')  # original language
    add_value('P212')  # ISBN-13
    add_value('P957')  # ISBN-10
    add_label('P275')  # copyright license
    add_label('P277')  # programming language
    add_value('P348')  # version
    add_label('P840')  # narrative location
 
    # Languages
    add_value('P1098')  # number of speakers
    add_label('P282')  # writing system
    add_label('P1018')  # language regulatory body
    add_value('P218')  # language code (ISO 639-1)
 
    # Other
    add_label('P169')  # ceo
    add_label('P112')  # founded by
    add_label('P1454')  # legal form (company, organization)
    add_label('P137')  # operator (service, facility, ...)
    add_label('P1029')  # crew members (tripulation)
    add_label('P225')  # taxon name
    add_value('P274')  # chemical formula
    add_label('P1346')  # winner (sports, contests, ...)
    add_value('P1120')  # number of deaths
    add_value('P498')  # currency code (ISO 4217)
 
    # URL
    add_url('P856', official=True)  # official website
    attributes.append(WDArticle(language))  # wikipedia (user language)
    if not language.startswith('en'):
        attributes.append(WDArticle('en'))  # wikipedia (english)
 
    add_url('P1324')  # source code repository
    add_url('P1581')  # blog
    add_url('P434', url_id='musicbrainz_artist')
    add_url('P435', url_id='musicbrainz_work')
    add_url('P436', url_id='musicbrainz_release_group')
    add_url('P966', url_id='musicbrainz_label')
    add_url('P345', url_id='imdb_id')
    add_url('P2397', url_id='youtube_channel')
    add_url('P1651', url_id='youtube_video')
    add_url('P2002', url_id='twitter_profile')
    add_url('P2013', url_id='facebook_profile')
    add_url('P2003', url_id='instagram_profile')
 
    # Map
    attributes.append(WDGeoAttribute('P625'))
 
    # Image
    add_image('P15', priority=1, url_id='wikimedia_image')  # route map
    add_image('P242', priority=2, url_id='wikimedia_image')  # locator map
    add_image('P154', priority=3, url_id='wikimedia_image')  # logo
    add_image('P18', priority=4, url_id='wikimedia_image')  # image
    add_image('P41', priority=5, url_id='wikimedia_image')  # flag
    add_image('P2716', priority=6, url_id='wikimedia_image')  # collage
    add_image('P2910', priority=7, url_id='wikimedia_image')  # icon
 
    return attributes
 
 

Referenced by searx.engines.wikidata.get_query(), and searx.engines.wikidata.init().

Here is the caller graph for this function:

◆ get_headers()

searx.engines.wikidata.get_headers ( )

Definition at line 140 of file wikidata.py.

def get_headers():
    # user agent: https://www.mediawiki.org/wiki/Wikidata_Query_Service/User_Manual#Query_limits
    return {'Accept': 'application/sparql-results+json', 'User-Agent': searx_useragent()}
 
 

Referenced by searx.engines.wikidata.debug_explain_wikidata_query(), searx.engines.wikidata.request(), and searx.engines.wikidata.send_wikidata_query().

Here is the caller graph for this function:

◆ get_label_for_entity()

searx.engines.wikidata.get_label_for_entity	(		entity_id,
			language )

Definition at line 145 of file wikidata.py.

def get_label_for_entity(entity_id, language):
    name = WIKIDATA_PROPERTIES.get(entity_id)
    if name is None:
        name = WIKIDATA_PROPERTIES.get((entity_id, language))
    if name is None:
        name = WIKIDATA_PROPERTIES.get((entity_id, language.split('-')[0]))
    if name is None:
        name = WIKIDATA_PROPERTIES.get((entity_id, 'en'))
    if name is None:
        name = entity_id
    return name
 
 

Referenced by searx.engines.wikidata.WDAttribute.get_label(), and searx.engines.wikidata.WDAmountAttribute.get_str().

Here is the caller graph for this function:

◆ get_query()

searx.engines.wikidata.get_query	(		query,
			language )

Definition at line 336 of file wikidata.py.

def get_query(query, language):
    attributes = get_attributes(language)
    select = [a.get_select() for a in attributes]
    where = list(filter(lambda s: len(s) > 0, [a.get_where() for a in attributes]))
    wikibase_label = list(filter(lambda s: len(s) > 0, [a.get_wikibase_label() for a in attributes]))
    group_by = list(filter(lambda s: len(s) > 0, [a.get_group_by() for a in attributes]))
    query = (
        QUERY_TEMPLATE.replace('%QUERY%', sparql_string_escape(query))
        .replace('%SELECT%', ' '.join(select))
        .replace('%WHERE%', '\n  '.join(where))
        .replace('%WIKIBASE_LABELS%', '\n      '.join(wikibase_label))
        .replace('%GROUP_BY%', ' '.join(group_by))
        .replace('%LANGUAGE%', language)
    )
    return query, attributes
 
 

References searx.engines.wikidata.get_attributes().

Referenced by searx.engines.wikidata.request().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ get_results()

searx.engines.wikidata.get_results	(	attribute_result,
		attributes,
		language )

Definition at line 253 of file wikidata.py.

def get_results(attribute_result, attributes, language):
    # pylint: disable=too-many-branches
    results = []
    infobox_title = attribute_result.get('itemLabel')
    infobox_id = attribute_result['item']
    infobox_id_lang = None
    infobox_urls = []
    infobox_attributes = []
    infobox_content = attribute_result.get('itemDescription', [])
    img_src = None
    img_src_priority = 0
 
    for attribute in attributes:
        value = attribute.get_str(attribute_result, language)
        if value is not None and value != '':
            attribute_type = type(attribute)
 
            if attribute_type in (WDURLAttribute, WDArticle):
                # get_select() method : there is group_concat(distinct ...;separator=", ")
                # split the value here
                for url in value.split(', '):
                    infobox_urls.append({'title': attribute.get_label(language), 'url': url, **attribute.kwargs})
                    # "normal" results (not infobox) include official website and Wikipedia links.
                    if "list" in display_type and (attribute.kwargs.get('official') or attribute_type == WDArticle):
                        results.append({'title': infobox_title, 'url': url, "content": infobox_content})
 
                    # update the infobox_id with the wikipedia URL
                    # first the local wikipedia URL, and as fallback the english wikipedia URL
                    if attribute_type == WDArticle and (
                        (attribute.language == 'en' and infobox_id_lang is None) or attribute.language != 'en'
                    ):
                        infobox_id_lang = attribute.language
                        infobox_id = url
            elif attribute_type == WDImageAttribute:
                # this attribute is an image.
                # replace the current image only the priority is lower
                # (the infobox contain only one image).
                if attribute.priority > img_src_priority:
                    img_src = get_thumbnail(value)
                    img_src_priority = attribute.priority
            elif attribute_type == WDGeoAttribute:
                # geocoordinate link
                # use the area to get the OSM zoom
                # Note: ignore the unit (must be km² otherwise the calculation is wrong)
                # Should use normalized value p:P2046/psn:P2046/wikibase:quantityAmount
                area = attribute_result.get('P2046')
                osm_zoom = area_to_osm_zoom(area) if area else 19
                url = attribute.get_geo_url(attribute_result, osm_zoom=osm_zoom)
                if url:
                    infobox_urls.append({'title': attribute.get_label(language), 'url': url, 'entity': attribute.name})
            else:
                infobox_attributes.append(
                    {'label': attribute.get_label(language), 'value': value, 'entity': attribute.name}
                )
 
    if infobox_id:
        infobox_id = replace_http_by_https(infobox_id)
 
    # add the wikidata URL at the end
    infobox_urls.append({'title': 'Wikidata', 'url': attribute_result['item']})
 
    if (
        "list" in display_type
        and img_src is None
        and len(infobox_attributes) == 0
        and len(infobox_urls) == 1
        and len(infobox_content) == 0
    ):
        results.append({'url': infobox_urls[0]['url'], 'title': infobox_title, 'content': infobox_content})
    elif "infobox" in display_type:
        results.append(
            {
                'infobox': infobox_title,
                'id': infobox_id,
                'content': infobox_content,
                'img_src': img_src,
                'urls': infobox_urls,
                'attributes': infobox_attributes,
            }
        )
    return results
 
 

References searx.engines.wikidata.replace_http_by_https.

Referenced by searx.engines.wikidata.response().

Here is the caller graph for this function:

◆ get_thumbnail()

searx.engines.wikidata.get_thumbnail ( img_src )

Get Thumbnail image from wikimedia commons

Images from commons.wikimedia.org are (HTTP) redirected to
upload.wikimedia.org.  The redirected URL can be calculated by this
function.

- https://stackoverflow.com/a/33691240

Definition at line 214 of file wikidata.py.

def get_thumbnail(img_src):
    """Get Thumbnail image from wikimedia commons
 
    Images from commons.wikimedia.org are (HTTP) redirected to
    upload.wikimedia.org.  The redirected URL can be calculated by this
    function.
 
    - https://stackoverflow.com/a/33691240
 
    """
    logger.debug('get_thumbnail(): %s', img_src)
    if not img_src is None and _IMG_SRC_DEFAULT_URL_PREFIX in img_src.split()[0]:
        img_src_name = unquote(img_src.replace(_IMG_SRC_DEFAULT_URL_PREFIX, "").split("?", 1)[0].replace("%20", "_"))
        img_src_name_first = img_src_name
        img_src_name_second = img_src_name
 
        if ".svg" in img_src_name.split()[0]:
            img_src_name_second = img_src_name + ".png"
 
        img_src_size = img_src.replace(_IMG_SRC_DEFAULT_URL_PREFIX, "").split("?", 1)[1]
        img_src_size = img_src_size[img_src_size.index("=") + 1 : img_src_size.index("&")]
        img_src_name_md5 = md5(img_src_name.encode("utf-8")).hexdigest()
        img_src = (
            _IMG_SRC_NEW_URL_PREFIX
            + img_src_name_md5[0]
            + "/"
            + img_src_name_md5[0:2]
            + "/"
            + img_src_name_first
            + "/"
            + img_src_size
            + "px-"
            + img_src_name_second
        )
        logger.debug('get_thumbnail() redirected: %s', img_src)
 
    return img_src
 
 

◆ init()

searx.engines.wikidata.init ( engine_settings = None )

Definition at line 763 of file wikidata.py.

def init(engine_settings=None):  # pylint: disable=unused-argument
    # WIKIDATA_PROPERTIES : add unit symbols
    for k, v in WIKIDATA_UNITS.items():
        WIKIDATA_PROPERTIES[k] = v['symbol']
 
    # WIKIDATA_PROPERTIES : add property labels
    wikidata_property_names = []
    for attribute in get_attributes('en'):
        if type(attribute) in (WDAttribute, WDAmountAttribute, WDURLAttribute, WDDateAttribute, WDLabelAttribute):
            if attribute.name not in WIKIDATA_PROPERTIES:
                wikidata_property_names.append("wd:" + attribute.name)
    query = QUERY_PROPERTY_NAMES.replace('%ATTRIBUTES%', " ".join(wikidata_property_names))
    jsonresponse = send_wikidata_query(query)
    for result in jsonresponse.get('results', {}).get('bindings', {}):
        name = result['name']['value']
        lang = result['name']['xml:lang']
        entity_id = result['item']['value'].replace('http://www.wikidata.org/entity/', '')
        WIKIDATA_PROPERTIES[(entity_id, lang)] = name.capitalize()
 
 

References searx.engines.wikidata.get_attributes().

Here is the call graph for this function:

◆ request()

searx.engines.wikidata.request	(		query,
			params )

Definition at line 172 of file wikidata.py.

def request(query, params):
 
    eng_tag, _wiki_netloc = get_wiki_params(params['searxng_locale'], traits)
    query, attributes = get_query(query, eng_tag)
    logger.debug("request --> language %s // len(attributes): %s", eng_tag, len(attributes))
 
    params['method'] = 'POST'
    params['url'] = SPARQL_ENDPOINT_URL
    params['data'] = {'query': query}
    params['headers'] = get_headers()
    params['language'] = eng_tag
    params['attributes'] = attributes
 
    return params
 
 

References searx.engines.wikidata.get_headers(), and searx.engines.wikidata.get_query().

Here is the call graph for this function:

◆ response()

searx.engines.wikidata.response ( resp )

Definition at line 188 of file wikidata.py.

def response(resp):
 
    results = []
    jsonresponse = loads(resp.content.decode())
 
    language = resp.search_params['language']
    attributes = resp.search_params['attributes']
    logger.debug("request --> language %s // len(attributes): %s", language, len(attributes))
 
    seen_entities = set()
    for result in jsonresponse.get('results', {}).get('bindings', []):
        attribute_result = {key: value['value'] for key, value in result.items()}
        entity_url = attribute_result['item']
        if entity_url not in seen_entities and entity_url not in DUMMY_ENTITY_URLS:
            seen_entities.add(entity_url)
            results += get_results(attribute_result, attributes, language)
        else:
            logger.debug('The SPARQL request returns duplicate entities: %s', str(attribute_result))
 
    return results
 
 

References searx.engines.wikidata.get_results().

Here is the call graph for this function:

◆ send_wikidata_query()

searx.engines.wikidata.send_wikidata_query	(		query,
			method = 'GET' )

Definition at line 158 of file wikidata.py.

def send_wikidata_query(query, method='GET'):
    if method == 'GET':
        # query will be cached by wikidata
        http_response = get(SPARQL_ENDPOINT_URL + '?' + urlencode({'query': query}), headers=get_headers())
    else:
        # query won't be cached by wikidata
        http_response = post(SPARQL_ENDPOINT_URL, data={'query': query}, headers=get_headers())
    if http_response.status_code != 200:
        logger.debug('SPARQL endpoint error %s', http_response.content.decode())
    logger.debug('request time %s', str(http_response.elapsed))
    http_response.raise_for_status()
    return loads(http_response.content.decode())
 
 

References searx.engines.wikidata.get_headers().

Here is the call graph for this function:

Variable Documentation

◆ _IMG_SRC_DEFAULT_URL_PREFIX

str searx.engines.wikidata._IMG_SRC_DEFAULT_URL_PREFIX = "https://commons.wikimedia.org/wiki/Special:FilePath/"

protected

Definition at line 210 of file wikidata.py.

◆ _IMG_SRC_NEW_URL_PREFIX

str searx.engines.wikidata._IMG_SRC_NEW_URL_PREFIX = "https://upload.wikimedia.org/wikipedia/commons/thumb/"

protected

Definition at line 211 of file wikidata.py.

◆ about

dict searx.engines.wikidata.about

Initial value:

=  {
    "website": 'https://wikidata.org/',
    "wikidata_id": 'Q2013',
    "official_api_documentation": 'https://query.wikidata.org/',
    "use_official_api": True,
    "require_api_key": False,
    "results": 'JSON',
}

Definition at line 34 of file wikidata.py.

◆ display_type

list searx.engines.wikidata.display_type = ["infobox"]

Definition at line 43 of file wikidata.py.

◆ DUMMY_ENTITY_URLS

searx.engines.wikidata.DUMMY_ENTITY_URLS

Initial value:

=  set(
    "http://www.wikidata.org/entity/" + wid for wid in ("Q4115189", "Q13406268", "Q15397819", "Q17339402")
)

Definition at line 115 of file wikidata.py.

◆ logger

logging searx.engines.wikidata.logger .Logger

Definition at line 29 of file wikidata.py.

◆ QUERY_PROPERTY_NAMES

str searx.engines.wikidata.QUERY_PROPERTY_NAMES

Initial value:

=  """
SELECT ?item ?name
WHERE {
    {
      SELECT ?item
      WHERE { ?item wdt:P279* wd:Q12132 }
    } UNION {
      VALUES ?item { %ATTRIBUTES% }
    }
    OPTIONAL { ?item rdfs:label ?name. }
}
"""

Definition at line 100 of file wikidata.py.

◆ QUERY_TEMPLATE

str searx.engines.wikidata.QUERY_TEMPLATE

Initial value:

=  """
SELECT ?item ?itemLabel ?itemDescription ?lat ?long %SELECT%
WHERE
{
  SERVICE wikibase:mwapi {
        bd:serviceParam wikibase:endpoint "www.wikidata.org";
        wikibase:api "EntitySearch";
        wikibase:limit 1;
        mwapi:search "%QUERY%";
        mwapi:language "%LANGUAGE%".
        ?item wikibase:apiOutputItem mwapi:item.
  }
  hint:Prior hint:runFirst "true".
 
  %WHERE%
 
  SERVICE wikibase:label {
      bd:serviceParam wikibase:language "%LANGUAGE%,en".
      ?item rdfs:label ?itemLabel .
      ?item schema:description ?itemDescription .
      %WIKIBASE_LABELS%
  }
 
}
GROUP BY ?item ?itemLabel ?itemDescription ?lat ?long %GROUP_BY%
"""

Definition at line 72 of file wikidata.py.

◆ replace_http_by_https

searx.engines.wikidata.replace_http_by_https = get_string_replaces_function({'http:': 'https:'})

Definition at line 137 of file wikidata.py.

Referenced by searx.engines.wikidata.get_results().

◆ SPARQL_ENDPOINT_URL

str searx.engines.wikidata.SPARQL_ENDPOINT_URL = 'https://query.wikidata.org/sparql'

Definition at line 50 of file wikidata.py.

◆ SPARQL_EXPLAIN_URL

str searx.engines.wikidata.SPARQL_EXPLAIN_URL = 'https://query.wikidata.org/bigdata/namespace/wdq/sparql?explain'

Definition at line 51 of file wikidata.py.

◆ sparql_string_escape

searx.engines.wikidata.sparql_string_escape

Initial value:

=  get_string_replaces_function(
    # fmt: off
    {
        '\t': '\\\t',
        '\n': '\\\n',
        '\r': '\\\r',
        '\b': '\\\b',
        '\f': '\\\f',
        '\"': '\\\"',
        '\'': '\\\'',
        '\\': '\\\\'
    }
    # fmt: on
)

Definition at line 122 of file wikidata.py.

◆ WIKIDATA_PROPERTIES

dict searx.engines.wikidata.WIKIDATA_PROPERTIES

Initial value:

=  {
    'P434': 'MusicBrainz',
    'P435': 'MusicBrainz',
    'P436': 'MusicBrainz',
    'P966': 'MusicBrainz',
    'P345': 'IMDb',
    'P2397': 'YouTube',
    'P1651': 'YouTube',
    'P2002': 'Twitter',
    'P2013': 'Facebook',
    'P2003': 'Instagram',
}

Definition at line 52 of file wikidata.py.

Classes

Functions

Variables

Detailed Description

Function Documentation

◆ debug_explain_wikidata_query()

◆ fetch_traits()

◆ get_attributes()

◆ get_headers()

◆ get_label_for_entity()

◆ get_query()

◆ get_results()

◆ get_thumbnail()

◆ init()

◆ request()

◆ response()

◆ send_wikidata_query()

Variable Documentation

◆ _IMG_SRC_DEFAULT_URL_PREFIX

◆ _IMG_SRC_NEW_URL_PREFIX

◆ about

◆ display_type

◆ DUMMY_ENTITY_URLS

◆ logger

◆ QUERY_PROPERTY_NAMES

◆ QUERY_TEMPLATE

◆ replace_http_by_https

◆ SPARQL_ENDPOINT_URL

◆ SPARQL_EXPLAIN_URL

◆ sparql_string_escape

◆ WIKIDATA_PROPERTIES