searx.utils Namespace Reference

Classes
class	_HTMLTextExtractor
class	_HTMLTextExtractorException
class	_NotSetClass

Functions
str	searx_useragent ()
str	gen_useragent (Optional[str] os_string=None)
str	html_to_text (str html_str)
str	markdown_to_text (str markdown_str)
Optional[str]	extract_text (xpath_results, bool allow_none=False)
str	normalize_url (str url, str base_url)
str	extract_url (xpath_results, base_url)
Dict	dict_subset (MutableMapping dictionary, Set[str] properties)
	humanize_bytes (size, precision=2)
	humanize_number (size, precision=0)
int	convert_str_to_int (str number_str)
	extr (str txt, str begin, str end, str default="")
int	int_or_zero (Union[List[str], str] num)
Optional[Tuple[bool, str, str]]	is_valid_lang (lang)
types.ModuleType	load_module (str filename, str module_dir)
str	to_string (Any obj)
str	ecma_unescape (str string)
	remove_pua_from_str (string)
Callable[[str], str]	get_string_replaces_function (Dict[str, str] replaces)
Dict	get_engine_from_settings (str name)
XPath	get_xpath (XPathSpecType xpath_spec)
	eval_xpath (ElementBase element, XPathSpecType xpath_spec)
	eval_xpath_list (ElementBase element, XPathSpecType xpath_spec, Optional[int] min_len=None)
	eval_xpath_getindex (ElementBase elements, XPathSpecType xpath_spec, int index, default=_NOTSET)
"fasttext.FastText._FastText"	_get_fasttext_model ()
	get_embeded_stream_url (url)
Optional[str]	detect_language (str text, float threshold=0.3, bool only_search_languages=False)
	js_variable_to_python (js_variable)

Variables
	logger = logger.getChild('utils')
	XPathSpecType = Union[str, XPath]
tuple	_BLOCKED_TAGS = ('script', 'style')
	_ECMA_UNESCAPE4_RE = re.compile(r'%u([0-9a-fA-F]{4})', re.UNICODE)
	_ECMA_UNESCAPE2_RE = re.compile(r'%([0-9a-fA-F]{2})', re.UNICODE)
	_JS_QUOTE_KEYS_RE = re.compile(r'([\{\s,])(\w+)(:)')
	_JS_VOID_RE = re.compile(r'void\s+[0-9]+|void\s*\‍([0-9]+\‍)')
	_JS_DECIMAL_RE = re.compile(r":\s*\.")
dict	_XPATH_CACHE = {}
dict	_LANG_TO_LC_CACHE = {}
Optional	_FASTTEXT_MODEL = None
	SEARCH_LANGUAGE_CODES = frozenset([searxng_locale[0].split('-')[0] for searxng_locale in sxng_locales])
	_NOTSET = _NotSetClass()

Detailed Description

Utility functions for the engines

Function Documentation

_get_fasttext_model()

"fasttext.FastText._FastText" searx.utils._get_fasttext_model ( )

protected

Definition at line 622 of file utils.py.

def _get_fasttext_model() -> "fasttext.FastText._FastText":  # type: ignore
    global _FASTTEXT_MODEL  # pylint: disable=global-statement
    if _FASTTEXT_MODEL is None:
        import fasttext  # pylint: disable=import-outside-toplevel
 
        # Monkey patch: prevent fasttext from showing a (useless) warning when loading a model.
        fasttext.FastText.eprint = lambda x: None
        _FASTTEXT_MODEL = fasttext.load_model(str(data_dir / 'lid.176.ftz'))
    return _FASTTEXT_MODEL
 
 

Referenced by detect_language().

Here is the caller graph for this function:

convert_str_to_int()

int searx.utils.convert_str_to_int

(

str

number_str

)

Convert number_str to int or 0 if number_str is not a number.

Definition at line 349 of file utils.py.

def convert_str_to_int(number_str: str) -> int:
    """Convert number_str to int or 0 if number_str is not a number."""
    if number_str.isdigit():
        return int(number_str)
    return 0
 
 

Referenced by int_or_zero().

Here is the caller graph for this function:

detect_language()

Optional[str] searx.utils.detect_language	(	str	text,
		float	threshold = 0.3,
		bool	only_search_languages = False )

Detect the language of the ``text`` parameter.

:param str text: The string whose language is to be detected.

:param float threshold: Threshold filters the returned labels by a threshold
    on probability.  A choice of 0.3 will return labels with at least 0.3
    probability.

:param bool only_search_languages: If ``True``, returns only supported
    SearXNG search languages.  see :py:obj:`searx.languages`

:rtype: str, None
:returns:
    The detected language code or ``None``. See below.

:raises ValueError: If ``text`` is not a string.

The language detection is done by using `a fork`_ of the fastText_ library
(`python fasttext`_). fastText_ distributes the `language identification
model`_, for reference:

- `FastText.zip: Compressing text classification models`_
- `Bag of Tricks for Efficient Text Classification`_

The `language identification model`_ support the language codes
(ISO-639-3)::

    af als am an ar arz as ast av az azb ba bar bcl be bg bh bn bo bpy br bs
    bxr ca cbk ce ceb ckb co cs cv cy da de diq dsb dty dv el eml en eo es
    et eu fa fi fr frr fy ga gd gl gn gom gu gv he hi hif hr hsb ht hu hy ia
    id ie ilo io is it ja jbo jv ka kk km kn ko krc ku kv kw ky la lb lez li
    lmo lo lrc lt lv mai mg mhr min mk ml mn mr mrj ms mt mwl my myv mzn nah
    nap nds ne new nl nn no oc or os pa pam pfl pl pms pnb ps pt qu rm ro ru
    rue sa sah sc scn sco sd sh si sk sl so sq sr su sv sw ta te tg th tk tl
    tr tt tyv ug uk ur uz vec vep vi vls vo wa war wuu xal xmf yi yo yue zh

By using ``only_search_languages=True`` the `language identification model`_
is harmonized with the SearXNG's language (locale) model.  General
conditions of SearXNG's locale model are:

a. SearXNG's locale of a query is passed to the
   :py:obj:`searx.locales.get_engine_locale` to get a language and/or region
   code that is used by an engine.

b. Most of SearXNG's engines do not support all the languages from `language
   identification model`_ and there is also a discrepancy in the ISO-639-3
   (fasttext) and ISO-639-2 (SearXNG)handling.  Further more, in SearXNG the
   locales like ``zh-TH`` (``zh-CN``) are mapped to ``zh_Hant``
   (``zh_Hans``) while the `language identification model`_ reduce both to
   ``zh``.

.. _a fork: https://github.com/searxng/fasttext-predict
.. _fastText: https://fasttext.cc/
.. _python fasttext: https://pypi.org/project/fasttext/
.. _language identification model: https://fasttext.cc/docs/en/language-identification.html
.. _Bag of Tricks for Efficient Text Classification: https://arxiv.org/abs/1607.01759
.. _`FastText.zip: Compressing text classification models`: https://arxiv.org/abs/1612.03651

Definition at line 695 of file utils.py.

def detect_language(text: str, threshold: float = 0.3, only_search_languages: bool = False) -> Optional[str]:
    """Detect the language of the ``text`` parameter.
 
    :param str text: The string whose language is to be detected.
 
    :param float threshold: Threshold filters the returned labels by a threshold
        on probability.  A choice of 0.3 will return labels with at least 0.3
        probability.
 
    :param bool only_search_languages: If ``True``, returns only supported
        SearXNG search languages.  see :py:obj:`searx.languages`
 
    :rtype: str, None
    :returns:
        The detected language code or ``None``. See below.
 
    :raises ValueError: If ``text`` is not a string.
 
    The language detection is done by using `a fork`_ of the fastText_ library
    (`python fasttext`_). fastText_ distributes the `language identification
    model`_, for reference:
 
    - `FastText.zip: Compressing text classification models`_
    - `Bag of Tricks for Efficient Text Classification`_
 
    The `language identification model`_ support the language codes
    (ISO-639-3)::
 
        af als am an ar arz as ast av az azb ba bar bcl be bg bh bn bo bpy br bs
        bxr ca cbk ce ceb ckb co cs cv cy da de diq dsb dty dv el eml en eo es
        et eu fa fi fr frr fy ga gd gl gn gom gu gv he hi hif hr hsb ht hu hy ia
        id ie ilo io is it ja jbo jv ka kk km kn ko krc ku kv kw ky la lb lez li
        lmo lo lrc lt lv mai mg mhr min mk ml mn mr mrj ms mt mwl my myv mzn nah
        nap nds ne new nl nn no oc or os pa pam pfl pl pms pnb ps pt qu rm ro ru
        rue sa sah sc scn sco sd sh si sk sl so sq sr su sv sw ta te tg th tk tl
        tr tt tyv ug uk ur uz vec vep vi vls vo wa war wuu xal xmf yi yo yue zh
 
    By using ``only_search_languages=True`` the `language identification model`_
    is harmonized with the SearXNG's language (locale) model.  General
    conditions of SearXNG's locale model are:
 
    a. SearXNG's locale of a query is passed to the
       :py:obj:`searx.locales.get_engine_locale` to get a language and/or region
       code that is used by an engine.
 
    b. Most of SearXNG's engines do not support all the languages from `language
       identification model`_ and there is also a discrepancy in the ISO-639-3
       (fasttext) and ISO-639-2 (SearXNG)handling.  Further more, in SearXNG the
       locales like ``zh-TH`` (``zh-CN``) are mapped to ``zh_Hant``
       (``zh_Hans``) while the `language identification model`_ reduce both to
       ``zh``.
 
    .. _a fork: https://github.com/searxng/fasttext-predict
    .. _fastText: https://fasttext.cc/
    .. _python fasttext: https://pypi.org/project/fasttext/
    .. _language identification model: https://fasttext.cc/docs/en/language-identification.html
    .. _Bag of Tricks for Efficient Text Classification: https://arxiv.org/abs/1607.01759
    .. _`FastText.zip: Compressing text classification models`: https://arxiv.org/abs/1612.03651
 
    """
    if not isinstance(text, str):
        raise ValueError('text must a str')
    r = _get_fasttext_model().predict(text.replace('\n', ' '), k=1, threshold=threshold)
    if isinstance(r, tuple) and len(r) == 2 and len(r[0]) > 0 and len(r[1]) > 0:
        language = r[0][0].split('__label__')[1]
        if only_search_languages and language not in SEARCH_LANGUAGE_CODES:
            return None
        return language
    return None
 
 

References _get_fasttext_model().

Here is the call graph for this function:

dict_subset()

Dict searx.utils.dict_subset	(	MutableMapping	dictionary,
		Set[str]	properties )

Extract a subset of a dict

Examples:
    >>> dict_subset({'A': 'a', 'B': 'b', 'C': 'c'}, ['A', 'C'])
    {'A': 'a', 'C': 'c'}
    >>> >> dict_subset({'A': 'a', 'B': 'b', 'C': 'c'}, ['A', 'D'])
    {'A': 'a'}

Definition at line 313 of file utils.py.

def dict_subset(dictionary: MutableMapping, properties: Set[str]) -> Dict:
    """Extract a subset of a dict
 
    Examples:
        >>> dict_subset({'A': 'a', 'B': 'b', 'C': 'c'}, ['A', 'C'])
        {'A': 'a', 'C': 'c'}
        >>> >> dict_subset({'A': 'a', 'B': 'b', 'C': 'c'}, ['A', 'D'])
        {'A': 'a'}
    """
    return {k: dictionary[k] for k in properties if k in dictionary}
 
 

ecma_unescape()

str searx.utils.ecma_unescape

(

str

string

)

Python implementation of the unescape javascript function

https://www.ecma-international.org/ecma-262/6.0/#sec-unescape-string
https://developer.mozilla.org/fr/docs/Web/JavaScript/Reference/Objets_globaux/unescape

Examples:
    >>> ecma_unescape('%u5409')
    '吉'
    >>> ecma_unescape('%20')
    ' '
    >>> ecma_unescape('%F3')
    'ó'

Definition at line 452 of file utils.py.

def ecma_unescape(string: str) -> str:
    """Python implementation of the unescape javascript function
 
    https://www.ecma-international.org/ecma-262/6.0/#sec-unescape-string
    https://developer.mozilla.org/fr/docs/Web/JavaScript/Reference/Objets_globaux/unescape
 
    Examples:
        >>> ecma_unescape('%u5409')
        '吉'
        >>> ecma_unescape('%20')
        ' '
        >>> ecma_unescape('%F3')
        'ó'
    """
    # "%u5409" becomes "吉"
    string = _ECMA_UNESCAPE4_RE.sub(lambda e: chr(int(e.group(1), 16)), string)
    # "%20" becomes " ", "%F3" becomes "ó"
    string = _ECMA_UNESCAPE2_RE.sub(lambda e: chr(int(e.group(1), 16)), string)
    return string
 
 

eval_xpath()

searx.utils.eval_xpath	(	ElementBase	element,
		XPathSpecType	xpath_spec )

Equivalent of element.xpath(xpath_str) but compile xpath_str once for all.
See https://lxml.de/xpathxslt.html#xpath-return-values

Args:
    * element (ElementBase): [description]
    * xpath_spec (str|lxml.etree.XPath): XPath as a str or lxml.etree.XPath

Returns:
    * result (bool, float, list, str): Results.

Raises:
    * TypeError: Raise when xpath_spec is neither a str nor a lxml.etree.XPath
    * SearxXPathSyntaxException: Raise when there is a syntax error in the XPath
    * SearxEngineXPathException: Raise when the XPath can't be evaluated.

Definition at line 545 of file utils.py.

def eval_xpath(element: ElementBase, xpath_spec: XPathSpecType):
    """Equivalent of element.xpath(xpath_str) but compile xpath_str once for all.
    See https://lxml.de/xpathxslt.html#xpath-return-values
 
    Args:
        * element (ElementBase): [description]
        * xpath_spec (str|lxml.etree.XPath): XPath as a str or lxml.etree.XPath
 
    Returns:
        * result (bool, float, list, str): Results.
 
    Raises:
        * TypeError: Raise when xpath_spec is neither a str nor a lxml.etree.XPath
        * SearxXPathSyntaxException: Raise when there is a syntax error in the XPath
        * SearxEngineXPathException: Raise when the XPath can't be evaluated.
    """
    xpath = get_xpath(xpath_spec)
    try:
        return xpath(element)
    except XPathError as e:
        arg = ' '.join([str(i) for i in e.args])
        raise SearxEngineXPathException(xpath_spec, arg) from e
 
 

References get_xpath().

Referenced by eval_xpath_list().

Here is the call graph for this function:

Here is the caller graph for this function:

eval_xpath_getindex()

searx.utils.eval_xpath_getindex	(	ElementBase	elements,
		XPathSpecType	xpath_spec,
		int	index,
			default = _NOTSET )

Call eval_xpath_list then get one element using the index parameter.
If the index does not exist, either raise an exception is default is not set,
other return the default value (can be None).

Args:
    * elements (ElementBase): lxml element to apply the xpath.
    * xpath_spec (str|lxml.etree.XPath): XPath as a str or lxml.etree.XPath.
    * index (int): index to get
    * default (Object, optional): Defaults if index doesn't exist.

Raises:
    * TypeError: Raise when xpath_spec is neither a str nor a lxml.etree.XPath
    * SearxXPathSyntaxException: Raise when there is a syntax error in the XPath
    * SearxEngineXPathException: if the index is not found. Also see eval_xpath.

Returns:
    * result (bool, float, list, str): Results.

Definition at line 593 of file utils.py.

def eval_xpath_getindex(elements: ElementBase, xpath_spec: XPathSpecType, index: int, default=_NOTSET):
    """Call eval_xpath_list then get one element using the index parameter.
    If the index does not exist, either raise an exception is default is not set,
    other return the default value (can be None).
 
    Args:
        * elements (ElementBase): lxml element to apply the xpath.
        * xpath_spec (str|lxml.etree.XPath): XPath as a str or lxml.etree.XPath.
        * index (int): index to get
        * default (Object, optional): Defaults if index doesn't exist.
 
    Raises:
        * TypeError: Raise when xpath_spec is neither a str nor a lxml.etree.XPath
        * SearxXPathSyntaxException: Raise when there is a syntax error in the XPath
        * SearxEngineXPathException: if the index is not found. Also see eval_xpath.
 
    Returns:
        * result (bool, float, list, str): Results.
    """
    result = eval_xpath_list(elements, xpath_spec)
    if -len(result) <= index < len(result):
        return result[index]
    if default == _NOTSET:
        # raise an SearxEngineXPathException instead of IndexError
        # to record xpath_spec
        raise SearxEngineXPathException(xpath_spec, 'index ' + str(index) + ' not found')
    return default
 
 

References eval_xpath_list().

Here is the call graph for this function:

eval_xpath_list()

searx.utils.eval_xpath_list	(	ElementBase	element,
		XPathSpecType	xpath_spec,
		Optional[int]	min_len = None )

Same as eval_xpath, check if the result is a list

Args:
    * element (ElementBase): [description]
    * xpath_spec (str|lxml.etree.XPath): XPath as a str or lxml.etree.XPath
    * min_len (int, optional): [description]. Defaults to None.

Raises:
    * TypeError: Raise when xpath_spec is neither a str nor a lxml.etree.XPath
    * SearxXPathSyntaxException: Raise when there is a syntax error in the XPath
    * SearxEngineXPathException: raise if the result is not a list

Returns:
    * result (bool, float, list, str): Results.

Definition at line 569 of file utils.py.

def eval_xpath_list(element: ElementBase, xpath_spec: XPathSpecType, min_len: Optional[int] = None):
    """Same as eval_xpath, check if the result is a list
 
    Args:
        * element (ElementBase): [description]
        * xpath_spec (str|lxml.etree.XPath): XPath as a str or lxml.etree.XPath
        * min_len (int, optional): [description]. Defaults to None.
 
    Raises:
        * TypeError: Raise when xpath_spec is neither a str nor a lxml.etree.XPath
        * SearxXPathSyntaxException: Raise when there is a syntax error in the XPath
        * SearxEngineXPathException: raise if the result is not a list
 
    Returns:
        * result (bool, float, list, str): Results.
    """
    result = eval_xpath(element, xpath_spec)
    if not isinstance(result, list):
        raise SearxEngineXPathException(xpath_spec, 'the result is not a list')
    if min_len is not None and min_len > len(result):
        raise SearxEngineXPathException(xpath_spec, 'len(xpath_str) < ' + str(min_len))
    return result
 
 

References eval_xpath().

Referenced by eval_xpath_getindex().

Here is the call graph for this function:

Here is the caller graph for this function:

extr()

searx.utils.extr	(	str	txt,
		str	begin,
		str	end,
		str	default = "" )

Extract the string between ``begin`` and ``end`` from ``txt``

:param txt:     String to search in
:param begin:   First string to be searched for
:param end:     Second string to be searched for after ``begin``
:param default: Default value if one of ``begin`` or ``end`` is not
                found.  Defaults to an empty string.
:return: The string between the two search-strings ``begin`` and ``end``.
         If at least one of ``begin`` or ``end`` is not found, the value of
         ``default`` is returned.

Examples:
  >>> extr("abcde", "a", "e")
  "bcd"
  >>> extr("abcde", "a", "z", deafult="nothing")
  "nothing"

Definition at line 356 of file utils.py.

def extr(txt: str, begin: str, end: str, default: str = ""):
    """Extract the string between ``begin`` and ``end`` from ``txt``
 
    :param txt:     String to search in
    :param begin:   First string to be searched for
    :param end:     Second string to be searched for after ``begin``
    :param default: Default value if one of ``begin`` or ``end`` is not
                    found.  Defaults to an empty string.
    :return: The string between the two search-strings ``begin`` and ``end``.
             If at least one of ``begin`` or ``end`` is not found, the value of
             ``default`` is returned.
 
    Examples:
      >>> extr("abcde", "a", "e")
      "bcd"
      >>> extr("abcde", "a", "z", deafult="nothing")
      "nothing"
 
    """
 
    # From https://github.com/mikf/gallery-dl/blob/master/gallery_dl/text.py#L129
 
    try:
        first = txt.index(begin) + len(begin)
        return txt[first : txt.index(end, first)]
    except ValueError:
        return default
 
 

extract_text()

Optional[str] searx.utils.extract_text	(		xpath_results,
		bool	allow_none = False )

Extract text from a lxml result

* if xpath_results is list, extract the text from each result and concat the list
* if xpath_results is a xml element, extract all the text node from it
  ( text_content() method from lxml )
* if xpath_results is a string element, then it's already done

Definition at line 194 of file utils.py.

def extract_text(xpath_results, allow_none: bool = False) -> Optional[str]:
    """Extract text from a lxml result
 
    * if xpath_results is list, extract the text from each result and concat the list
    * if xpath_results is a xml element, extract all the text node from it
      ( text_content() method from lxml )
    * if xpath_results is a string element, then it's already done
    """
    if isinstance(xpath_results, list):
        # it's list of result : concat everything using recursive call
        result = ''
        for e in xpath_results:
            result = result + (extract_text(e) or '')
        return result.strip()
    if isinstance(xpath_results, ElementBase):
        # it's a element
        text: str = html.tostring(xpath_results, encoding='unicode', method='text', with_tail=False)
        text = text.strip().replace('\n', ' ')
        return ' '.join(text.split())
    if isinstance(xpath_results, (str, Number, bool)):
        return str(xpath_results)
    if xpath_results is None and allow_none:
        return None
    if xpath_results is None and not allow_none:
        raise ValueError('extract_text(None, allow_none=False)')
    raise ValueError('unsupported type')
 
 

References extract_text().

Referenced by extract_text(), and extract_url().

Here is the call graph for this function:

Here is the caller graph for this function:

extract_url()

str searx.utils.extract_url	(		xpath_results,
			base_url )

Extract and normalize URL from lxml Element

Args:
    * xpath_results (Union[List[html.HtmlElement], html.HtmlElement]): lxml Element(s)
    * base_url (str): Base URL

Example:
    >>> def f(s, search_url):
    >>>    return searx.utils.extract_url(html.fromstring(s), search_url)
    >>> f('<span id="42">https://example.com</span>', 'http://example.com/')
    'https://example.com/'
    >>> f('https://example.com', 'http://example.com/')
    'https://example.com/'
    >>> f('//example.com', 'http://example.com/')
    'http://example.com/'
    >>> f('//example.com', 'https://example.com/')
    'https://example.com/'
    >>> f('/path?a=1', 'https://example.com')
    'https://example.com/path?a=1'
    >>> f('', 'https://example.com')
    raise lxml.etree.ParserError
    >>> searx.utils.extract_url([], 'https://example.com')
    raise ValueError

Raises:
    * ValueError
    * lxml.etree.ParserError

Returns:
    * str: normalized URL

Definition at line 272 of file utils.py.

def extract_url(xpath_results, base_url) -> str:
    """Extract and normalize URL from lxml Element
 
    Args:
        * xpath_results (Union[List[html.HtmlElement], html.HtmlElement]): lxml Element(s)
        * base_url (str): Base URL
 
    Example:
        >>> def f(s, search_url):
        >>>    return searx.utils.extract_url(html.fromstring(s), search_url)
        >>> f('<span id="42">https://example.com</span>', 'http://example.com/')
        'https://example.com/'
        >>> f('https://example.com', 'http://example.com/')
        'https://example.com/'
        >>> f('//example.com', 'http://example.com/')
        'http://example.com/'
        >>> f('//example.com', 'https://example.com/')
        'https://example.com/'
        >>> f('/path?a=1', 'https://example.com')
        'https://example.com/path?a=1'
        >>> f('', 'https://example.com')
        raise lxml.etree.ParserError
        >>> searx.utils.extract_url([], 'https://example.com')
        raise ValueError
 
    Raises:
        * ValueError
        * lxml.etree.ParserError
 
    Returns:
        * str: normalized URL
    """
    if xpath_results == []:
        raise ValueError('Empty url resultset')
 
    url = extract_text(xpath_results)
    if url:
        return normalize_url(url, base_url)
    raise ValueError('URL not found')
 
 

References extract_text(), and normalize_url().

Here is the call graph for this function:

gen_useragent()

str searx.utils.gen_useragent

(

Optional[str]

os_string = None

)

Return a random browser User Agent

See searx/data/useragents.json

Definition at line 72 of file utils.py.

def gen_useragent(os_string: Optional[str] = None) -> str:
    """Return a random browser User Agent
 
    See searx/data/useragents.json
    """
    return USER_AGENTS['ua'].format(os=os_string or choice(USER_AGENTS['os']), version=choice(USER_AGENTS['versions']))
 
 

get_embeded_stream_url()

searx.utils.get_embeded_stream_url

(

url

)

Converts a standard video URL into its embed format. Supported services include Youtube,
Facebook, Instagram, TikTok, Dailymotion, and Bilibili.

Definition at line 633 of file utils.py.

def get_embeded_stream_url(url):
    """
    Converts a standard video URL into its embed format. Supported services include Youtube,
    Facebook, Instagram, TikTok, Dailymotion, and Bilibili.
    """
    parsed_url = urlparse(url)
    iframe_src = None
 
    # YouTube
    if parsed_url.netloc in ['www.youtube.com', 'youtube.com'] and parsed_url.path == '/watch' and parsed_url.query:
        video_id = parse_qs(parsed_url.query).get('v', [])
        if video_id:
            iframe_src = 'https://www.youtube-nocookie.com/embed/' + video_id[0]
 
    # Facebook
    elif parsed_url.netloc in ['www.facebook.com', 'facebook.com']:
        encoded_href = urlencode({'href': url})
        iframe_src = 'https://www.facebook.com/plugins/video.php?allowfullscreen=true&' + encoded_href
 
    # Instagram
    elif parsed_url.netloc in ['www.instagram.com', 'instagram.com'] and parsed_url.path.startswith('/p/'):
        if parsed_url.path.endswith('/'):
            iframe_src = url + 'embed'
        else:
            iframe_src = url + '/embed'
 
    # TikTok
    elif (
        parsed_url.netloc in ['www.tiktok.com', 'tiktok.com']
        and parsed_url.path.startswith('/@')
        and '/video/' in parsed_url.path
    ):
        path_parts = parsed_url.path.split('/video/')
        video_id = path_parts[1]
        iframe_src = 'https://www.tiktok.com/embed/' + video_id
 
    # Dailymotion
    elif parsed_url.netloc in ['www.dailymotion.com', 'dailymotion.com'] and parsed_url.path.startswith('/video/'):
        path_parts = parsed_url.path.split('/')
        if len(path_parts) == 3:
            video_id = path_parts[2]
            iframe_src = 'https://www.dailymotion.com/embed/video/' + video_id
 
    # Bilibili
    elif parsed_url.netloc in ['www.bilibili.com', 'bilibili.com'] and parsed_url.path.startswith('/video/'):
        path_parts = parsed_url.path.split('/')
 
        video_id = path_parts[2]
        param_key = None
        if video_id.startswith('av'):
            video_id = video_id[2:]
            param_key = 'aid'
        elif video_id.startswith('BV'):
            param_key = 'bvid'
 
        iframe_src = (
            f'https://player.bilibili.com/player.html?{param_key}={video_id}&high_quality=1&autoplay=false&danmaku=0'
        )
 
    return iframe_src
 
 

get_engine_from_settings()

Dict searx.utils.get_engine_from_settings

(

str

name

)

Return engine configuration from settings.yml of a given engine name

Definition at line 498 of file utils.py.

def get_engine_from_settings(name: str) -> Dict:
    """Return engine configuration from settings.yml of a given engine name"""
 
    if 'engines' not in settings:
        return {}
 
    for engine in settings['engines']:
        if 'name' not in engine:
            continue
        if name == engine['name']:
            return engine
 
    return {}
 
 

get_string_replaces_function()

Callable[[str], str] searx.utils.get_string_replaces_function

(

Dict[str, str]

replaces

)

Definition at line 488 of file utils.py.

def get_string_replaces_function(replaces: Dict[str, str]) -> Callable[[str], str]:
    rep = {re.escape(k): v for k, v in replaces.items()}
    pattern = re.compile("|".join(rep.keys()))
 
    def func(text):
        return pattern.sub(lambda m: rep[re.escape(m.group(0))], text)
 
    return func
 
 

get_xpath()

XPath searx.utils.get_xpath

(

XPathSpecType

xpath_spec

)

Return cached compiled XPath

There is no thread lock.
Worst case scenario, xpath_str is compiled more than one time.

Args:
    * xpath_spec (str|lxml.etree.XPath): XPath as a str or lxml.etree.XPath

Returns:
    * result (bool, float, list, str): Results.

Raises:
    * TypeError: Raise when xpath_spec is neither a str nor a lxml.etree.XPath
    * SearxXPathSyntaxException: Raise when there is a syntax error in the XPath

Definition at line 513 of file utils.py.

def get_xpath(xpath_spec: XPathSpecType) -> XPath:
    """Return cached compiled XPath
 
    There is no thread lock.
    Worst case scenario, xpath_str is compiled more than one time.
 
    Args:
        * xpath_spec (str|lxml.etree.XPath): XPath as a str or lxml.etree.XPath
 
    Returns:
        * result (bool, float, list, str): Results.
 
    Raises:
        * TypeError: Raise when xpath_spec is neither a str nor a lxml.etree.XPath
        * SearxXPathSyntaxException: Raise when there is a syntax error in the XPath
    """
    if isinstance(xpath_spec, str):
        result = _XPATH_CACHE.get(xpath_spec, None)
        if result is None:
            try:
                result = XPath(xpath_spec)
            except XPathSyntaxError as e:
                raise SearxXPathSyntaxException(xpath_spec, str(e.msg)) from e
            _XPATH_CACHE[xpath_spec] = result
        return result
 
    if isinstance(xpath_spec, XPath):
        return xpath_spec
 
    raise TypeError('xpath_spec must be either a str or a lxml.etree.XPath')
 
 

Referenced by eval_xpath().

Here is the caller graph for this function:

html_to_text()

str searx.utils.html_to_text

(

str

html_str

)

Extract text from a HTML string

Args:
    * html_str (str): string HTML

Returns:
    * str: extracted text

Examples:
    >>> html_to_text('Example <span id="42">#2</span>')
    'Example #2'

    >>> html_to_text('<style>.span { color: red; }</style><span>Example</span>')
    'Example'

    >>> html_to_text(r'regexp: (?<![a-zA-Z]')
    'regexp: (?<![a-zA-Z]'

Definition at line 139 of file utils.py.

def html_to_text(html_str: str) -> str:
    """Extract text from a HTML string
 
    Args:
        * html_str (str): string HTML
 
    Returns:
        * str: extracted text
 
    Examples:
        >>> html_to_text('Example <span id="42">#2</span>')
        'Example #2'
 
        >>> html_to_text('<style>.span { color: red; }</style><span>Example</span>')
        'Example'
 
        >>> html_to_text(r'regexp: (?<![a-zA-Z]')
        'regexp: (?<![a-zA-Z]'
    """
    html_str = html_str.replace('\n', ' ').replace('\r', ' ')
    html_str = ' '.join(html_str.split())
    s = _HTMLTextExtractor()
    try:
        s.feed(html_str)
    except AssertionError:
        s = _HTMLTextExtractor()
        s.feed(escape(html_str, quote=True))
    except _HTMLTextExtractorException:
        logger.debug("HTMLTextExtractor: invalid HTML\n%s", html_str)
    return s.get_text()
 
 

Referenced by markdown_to_text().

Here is the caller graph for this function:

humanize_bytes()

searx.utils.humanize_bytes	(		size,
			precision = 2 )

Determine the *human readable* value of bytes on 1024 base (1KB=1024B).

Definition at line 325 of file utils.py.

def humanize_bytes(size, precision=2):
    """Determine the *human readable* value of bytes on 1024 base (1KB=1024B)."""
    s = ['B ', 'KB', 'MB', 'GB', 'TB']
 
    x = len(s)
    p = 0
    while size > 1024 and p < x:
        p += 1
        size = size / 1024.0
    return "%.*f %s" % (precision, size, s[p])
 
 

humanize_number()

searx.utils.humanize_number	(		size,
			precision = 0 )

Determine the *human readable* value of a decimal number.

Definition at line 337 of file utils.py.

def humanize_number(size, precision=0):
    """Determine the *human readable* value of a decimal number."""
    s = ['', 'K', 'M', 'B', 'T']
 
    x = len(s)
    p = 0
    while size > 1000 and p < x:
        p += 1
        size = size / 1000.0
    return "%.*f%s" % (precision, size, s[p])
 
 

int_or_zero()

int searx.utils.int_or_zero

(

Union[List[str], str]

num

)

Convert num to int or 0. num can be either a str or a list.
If num is a list, the first element is converted to int (or return 0 if the list is empty).
If num is a str, see convert_str_to_int

Definition at line 385 of file utils.py.

def int_or_zero(num: Union[List[str], str]) -> int:
    """Convert num to int or 0. num can be either a str or a list.
    If num is a list, the first element is converted to int (or return 0 if the list is empty).
    If num is a str, see convert_str_to_int
    """
    if isinstance(num, list):
        if len(num) < 1:
            return 0
        num = num[0]
    return convert_str_to_int(num)
 
 

References convert_str_to_int().

Here is the call graph for this function:

is_valid_lang()

Optional[Tuple[bool, str, str]] searx.utils.is_valid_lang

(

lang

)

Return language code and name if lang describe a language.

Examples:
    >>> is_valid_lang('zz')
    None
    >>> is_valid_lang('uk')
    (True, 'uk', 'ukrainian')
    >>> is_valid_lang(b'uk')
    (True, 'uk', 'ukrainian')
    >>> is_valid_lang('en')
    (True, 'en', 'english')
    >>> searx.utils.is_valid_lang('Español')
    (True, 'es', 'spanish')
    >>> searx.utils.is_valid_lang('Spanish')
    (True, 'es', 'spanish')

Definition at line 397 of file utils.py.

def is_valid_lang(lang) -> Optional[Tuple[bool, str, str]]:
    """Return language code and name if lang describe a language.
 
    Examples:
        >>> is_valid_lang('zz')
        None
        >>> is_valid_lang('uk')
        (True, 'uk', 'ukrainian')
        >>> is_valid_lang(b'uk')
        (True, 'uk', 'ukrainian')
        >>> is_valid_lang('en')
        (True, 'en', 'english')
        >>> searx.utils.is_valid_lang('Español')
        (True, 'es', 'spanish')
        >>> searx.utils.is_valid_lang('Spanish')
        (True, 'es', 'spanish')
    """
    if isinstance(lang, bytes):
        lang = lang.decode()
    is_abbr = len(lang) == 2
    lang = lang.lower()
    if is_abbr:
        for l in sxng_locales:
            if l[0][:2] == lang:
                return (True, l[0][:2], l[3].lower())
        return None
    for l in sxng_locales:
        if l[1].lower() == lang or l[3].lower() == lang:
            return (True, l[0][:2], l[3].lower())
    return None
 
 

js_variable_to_python()

searx.utils.js_variable_to_python

(

js_variable

)

Convert a javascript variable into JSON and then load the value

It does not deal with all cases, but it is good enough for now.
chompjs has a better implementation.

Definition at line 766 of file utils.py.

def js_variable_to_python(js_variable):
    """Convert a javascript variable into JSON and then load the value
 
    It does not deal with all cases, but it is good enough for now.
    chompjs has a better implementation.
    """
    # when in_string is not None, it contains the character that has opened the string
    # either simple quote or double quote
    in_string = None
    # cut the string:
    # r"""{ a:"f\"irst", c:'sec"ond'}"""
    # becomes
    # ['{ a:', '"', 'f\\', '"', 'irst', '"', ', c:', "'", 'sec', '"', 'ond', "'", '}']
    parts = re.split(r'(["\'])', js_variable)
    # previous part (to check the escape character antislash)
    previous_p = ""
    for i, p in enumerate(parts):
        # parse characters inside a ECMA string
        if in_string:
            # we are in a JS string: replace the colon by a temporary character
            # so quote_keys_regex doesn't have to deal with colon inside the JS strings
            parts[i] = parts[i].replace(':', chr(1))
            if in_string == "'":
                # the JS string is delimited by simple quote.
                # This is not supported by JSON.
                # simple quote delimited string are converted to double quote delimited string
                # here, inside a JS string, we escape the double quote
                parts[i] = parts[i].replace('"', r'\"')
 
        # deal with delimiters and escape character
        if not in_string and p in ('"', "'"):
            # we are not in string
            # but p is double or simple quote
            # that's the start of a new string
            # replace simple quote by double quote
            # (JSON doesn't support simple quote)
            parts[i] = '"'
            in_string = p
            continue
        if p == in_string:
            # we are in a string and the current part MAY close the string
            if len(previous_p) > 0 and previous_p[-1] == '\\':
                # there is an antislash just before: the ECMA string continue
                continue
            # the current p close the string
            # replace simple quote by double quote
            parts[i] = '"'
            in_string = None
 
        if not in_string:
            # replace void 0 by null
            # https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/void
            # we are sure there is no string in p
            parts[i] = _JS_VOID_RE.sub("null", p)
        # update previous_p
        previous_p = p
    # join the string
    s = ''.join(parts)
    # add quote around the key
    # { a: 12 }
    # becomes
    # { "a": 12 }
    s = _JS_QUOTE_KEYS_RE.sub(r'\1"\2"\3', s)
    s = _JS_DECIMAL_RE.sub(":0.", s)
    # replace the surogate character by colon
    s = s.replace(chr(1), ':')
    # load the JSON and return the result
    return json.loads(s)

load_module()

types.ModuleType searx.utils.load_module	(	str	filename,
		str	module_dir )

Definition at line 429 of file utils.py.

def load_module(filename: str, module_dir: str) -> types.ModuleType:
    modname = splitext(filename)[0]
    modpath = join(module_dir, filename)
    # and https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly
    spec = importlib.util.spec_from_file_location(modname, modpath)
    if not spec:
        raise ValueError(f"Error loading '{modpath}' module")
    module = importlib.util.module_from_spec(spec)
    if not spec.loader:
        raise ValueError(f"Error loading '{modpath}' module")
    spec.loader.exec_module(module)
    return module
 
 

markdown_to_text()

str searx.utils.markdown_to_text

(

str

markdown_str

)

Extract text from a Markdown string

Args:
    * markdown_str (str): string Markdown

Returns:
    * str: extracted text

Examples:
    >>> markdown_to_text('[example](https://example.com)')
    'example'

    >>> markdown_to_text('## Headline')
    'Headline'

Definition at line 171 of file utils.py.

def markdown_to_text(markdown_str: str) -> str:
    """Extract text from a Markdown string
 
    Args:
        * markdown_str (str): string Markdown
 
    Returns:
        * str: extracted text
 
    Examples:
        >>> markdown_to_text('[example](https://example.com)')
        'example'
 
        >>> markdown_to_text('## Headline')
        'Headline'
    """
 
    html_str = (
        MarkdownIt("commonmark", {"typographer": True}).enable(["replacements", "smartquotes"]).render(markdown_str)
    )
    return html_to_text(html_str)
 
 

References html_to_text().

Here is the call graph for this function:

normalize_url()

str searx.utils.normalize_url	(	str	url,
		str	base_url )

Normalize URL: add protocol, join URL with base_url, add trailing slash if there is no path

Args:
    * url (str): Relative URL
    * base_url (str): Base URL, it must be an absolute URL.

Example:
    >>> normalize_url('https://example.com', 'http://example.com/')
    'https://example.com/'
    >>> normalize_url('//example.com', 'http://example.com/')
    'http://example.com/'
    >>> normalize_url('//example.com', 'https://example.com/')
    'https://example.com/'
    >>> normalize_url('/path?a=1', 'https://example.com')
    'https://example.com/path?a=1'
    >>> normalize_url('', 'https://example.com')
    'https://example.com/'
    >>> normalize_url('/test', '/path')
    raise ValueError

Raises:
    * lxml.etree.ParserError

Returns:
    * str: normalized URL

Definition at line 222 of file utils.py.

def normalize_url(url: str, base_url: str) -> str:
    """Normalize URL: add protocol, join URL with base_url, add trailing slash if there is no path
 
    Args:
        * url (str): Relative URL
        * base_url (str): Base URL, it must be an absolute URL.
 
    Example:
        >>> normalize_url('https://example.com', 'http://example.com/')
        'https://example.com/'
        >>> normalize_url('//example.com', 'http://example.com/')
        'http://example.com/'
        >>> normalize_url('//example.com', 'https://example.com/')
        'https://example.com/'
        >>> normalize_url('/path?a=1', 'https://example.com')
        'https://example.com/path?a=1'
        >>> normalize_url('', 'https://example.com')
        'https://example.com/'
        >>> normalize_url('/test', '/path')
        raise ValueError
 
    Raises:
        * lxml.etree.ParserError
 
    Returns:
        * str: normalized URL
    """
    if url.startswith('//'):
        # add http or https to this kind of url //example.com/
        parsed_search_url = urlparse(base_url)
        url = '{0}:{1}'.format(parsed_search_url.scheme or 'http', url)
    elif url.startswith('/'):
        # fix relative url to the search engine
        url = urljoin(base_url, url)
 
    # fix relative urls that fall through the crack
    if '://' not in url:
        url = urljoin(base_url, url)
 
    parsed_url = urlparse(url)
 
    # add a / at this end of the url if there is no path
    if not parsed_url.netloc:
        raise ValueError('Cannot parse url')
    if not parsed_url.path:
        url += '/'
 
    return url
 
 

Referenced by extract_url().

Here is the caller graph for this function:

remove_pua_from_str()

searx.utils.remove_pua_from_str

(

string

)

Removes unicode's "PRIVATE USE CHARACTER"s (PUA_) from a string.

.. _PUA: https://en.wikipedia.org/wiki/Private_Use_Areas

Definition at line 473 of file utils.py.

def remove_pua_from_str(string):
    """Removes unicode's "PRIVATE USE CHARACTER"s (PUA_) from a string.
 
    .. _PUA: https://en.wikipedia.org/wiki/Private_Use_Areas
    """
    pua_ranges = ((0xE000, 0xF8FF), (0xF0000, 0xFFFFD), (0x100000, 0x10FFFD))
    s = []
    for c in string:
        i = ord(c)
        if any(a <= i <= b for (a, b) in pua_ranges):
            continue
        s.append(c)
    return "".join(s)
 
 

searx_useragent()

str searx.utils.searx_useragent

(

)

Return the searx User Agent

Definition at line 65 of file utils.py.

def searx_useragent() -> str:
    """Return the searx User Agent"""
    return 'searx/{searx_version} {suffix}'.format(
        searx_version=VERSION_TAG, suffix=settings['outgoing']['useragent_suffix']
    ).strip()
 
 

to_string()

str searx.utils.to_string

(

Any

obj

)

Convert obj to its string representation.

Definition at line 443 of file utils.py.

def to_string(obj: Any) -> str:
    """Convert obj to its string representation."""
    if isinstance(obj, str):
        return obj
    if hasattr(obj, '__str__'):
        return str(obj)
    return repr(obj)
 
 

Variable Documentation

_BLOCKED_TAGS

tuple searx.utils._BLOCKED_TAGS = ('script', 'style')

protected

Definition at line 38 of file utils.py.

_ECMA_UNESCAPE2_RE

searx.utils._ECMA_UNESCAPE2_RE = re.compile(r'%([0-9a-fA-F]{2})', re.UNICODE)

protected

Definition at line 41 of file utils.py.

_ECMA_UNESCAPE4_RE

searx.utils._ECMA_UNESCAPE4_RE = re.compile(r'%u([0-9a-fA-F]{4})', re.UNICODE)

protected

Definition at line 40 of file utils.py.

_FASTTEXT_MODEL

Optional searx.utils._FASTTEXT_MODEL = None

protected

Definition at line 50 of file utils.py.

_JS_DECIMAL_RE

searx.utils._JS_DECIMAL_RE = re.compile(r":\s*\.")

protected

Definition at line 45 of file utils.py.

_JS_QUOTE_KEYS_RE

searx.utils._JS_QUOTE_KEYS_RE = re.compile(r'([\{\s,])(\w+)(:)')

protected

Definition at line 43 of file utils.py.

_JS_VOID_RE

searx.utils._JS_VOID_RE = re.compile(r'void\s+[0-9]+|void\s*\‍([0-9]+\‍)')

protected

Definition at line 44 of file utils.py.

_LANG_TO_LC_CACHE

dict searx.utils._LANG_TO_LC_CACHE = {}

protected

Definition at line 48 of file utils.py.

_NOTSET

searx.utils._NOTSET = _NotSetClass()

protected

Definition at line 62 of file utils.py.

_XPATH_CACHE

dict searx.utils._XPATH_CACHE = {}

protected

Definition at line 47 of file utils.py.

logger

searx.utils.logger = logger.getChild('utils')

Definition at line 34 of file utils.py.

SEARCH_LANGUAGE_CODES

searx.utils.SEARCH_LANGUAGE_CODES = frozenset([searxng_locale[0].split('-')[0] for searxng_locale in sxng_locales])

Definition at line 53 of file utils.py.

XPathSpecType

searx.utils.XPathSpecType = Union[str, XPath]

Definition at line 36 of file utils.py.