.oO SearXNG Developer Documentation Oo.
|
Classes | |
class | _NotSetClass |
class | HTMLTextExtractor |
Functions | |
str | searxng_useragent () |
str | gen_useragent (str|None os_string=None) |
str | html_to_text (str html_str) |
str | markdown_to_text (str markdown_str) |
str|None | extract_text (list[ElementBase]|ElementBase|str|Number|bool|None xpath_results, bool allow_none=False) |
str | normalize_url (str url, str base_url) |
str | extract_url (list[ElementBase]|ElementBase|str|Number|bool|None xpath_results, str base_url) |
MutableMapping[str, t.Any] | dict_subset (MutableMapping[t.Any, t.Any] dictionary, set[str] properties) |
humanize_bytes (int|float size, int precision=2) | |
humanize_number (int|float size, int precision=0) | |
int | convert_str_to_int (str number_str) |
extr (str txt, str begin, str end, str default="") | |
int | int_or_zero (list[str]|str num) |
tuple[bool, str, str]|None | is_valid_lang (str lang) |
types.ModuleType | load_module (str filename, str module_dir) |
str | to_string (t.Any obj) |
str | ecma_unescape (str string) |
remove_pua_from_str (str string) | |
Callable[[str], str] | get_string_replaces_function (dict[str, str] replaces) |
dict[str, dict[str, str]] | get_engine_from_settings (str name) |
XPath | get_xpath (XPathSpecType xpath_spec) |
t.Any | eval_xpath (ElementBase element, XPathSpecType xpath_spec) |
list[t.Any] | eval_xpath_list (ElementBase element, XPathSpecType xpath_spec, int|None min_len=None) |
t.Any | eval_xpath_getindex (ElementBase element, XPathSpecType xpath_spec, int index, t.Any default=_NOTSET) |
"fasttext.FastText._FastText" | _get_fasttext_model () |
get_embeded_stream_url (str url) | |
str|None | detect_language (str text, float threshold=0.3, bool only_search_languages=False) |
str | js_variable_to_python (str js_variable) |
timedelta|None | parse_duration_string (str duration_str) |
Variables | |
logger = logger.getChild('utils') | |
t | XPathSpecType = str | XPath |
tuple | _BLOCKED_TAGS = ('script', 'style') |
_ECMA_UNESCAPE4_RE = re.compile(r'%u([0-9a-fA-F]{4})', re.UNICODE) | |
_ECMA_UNESCAPE2_RE = re.compile(r'%([0-9a-fA-F]{2})', re.UNICODE) | |
_JS_QUOTE_KEYS_RE = re.compile(r'([\{\s,])(\w+)(:)') | |
_JS_VOID_RE = re.compile(r'void\s+[0-9]+|void\s*\([0-9]+\)') | |
_JS_DECIMAL_RE = re.compile(r":\s*\.") | |
dict | _XPATH_CACHE = {} |
dict | _LANG_TO_LC_CACHE = {} |
str | _FASTTEXT_MODEL = None |
SEARCH_LANGUAGE_CODES = frozenset([searxng_locale[0].split('-')[0] for searxng_locale in sxng_locales]) | |
_NOTSET = _NotSetClass() |
Utility functions for the engines
|
protected |
Definition at line 620 of file utils.py.
Referenced by detect_language().
int searx.utils.convert_str_to_int | ( | str | number_str | ) |
Convert number_str to int or 0 if number_str is not a number.
Definition at line 365 of file utils.py.
Referenced by int_or_zero().
str | None searx.utils.detect_language | ( | str | text, |
float | threshold = 0.3, | ||
bool | only_search_languages = False ) |
Detect the language of the ``text`` parameter. :param str text: The string whose language is to be detected. :param float threshold: Threshold filters the returned labels by a threshold on probability. A choice of 0.3 will return labels with at least 0.3 probability. :param bool only_search_languages: If ``True``, returns only supported SearXNG search languages. see :py:obj:`searx.languages` :rtype: str, None :returns: The detected language code or ``None``. See below. :raises ValueError: If ``text`` is not a string. The language detection is done by using `a fork`_ of the fastText_ library (`python fasttext`_). fastText_ distributes the `language identification model`_, for reference: - `FastText.zip: Compressing text classification models`_ - `Bag of Tricks for Efficient Text Classification`_ The `language identification model`_ support the language codes (ISO-639-3):: af als am an ar arz as ast av az azb ba bar bcl be bg bh bn bo bpy br bs bxr ca cbk ce ceb ckb co cs cv cy da de diq dsb dty dv el eml en eo es et eu fa fi fr frr fy ga gd gl gn gom gu gv he hi hif hr hsb ht hu hy ia id ie ilo io is it ja jbo jv ka kk km kn ko krc ku kv kw ky la lb lez li lmo lo lrc lt lv mai mg mhr min mk ml mn mr mrj ms mt mwl my myv mzn nah nap nds ne new nl nn no oc or os pa pam pfl pl pms pnb ps pt qu rm ro ru rue sa sah sc scn sco sd sh si sk sl so sq sr su sv sw ta te tg th tk tl tr tt tyv ug uk ur uz vec vep vi vls vo wa war wuu xal xmf yi yo yue zh By using ``only_search_languages=True`` the `language identification model`_ is harmonized with the SearXNG's language (locale) model. General conditions of SearXNG's locale model are: a. SearXNG's locale of a query is passed to the :py:obj:`searx.locales.get_engine_locale` to get a language and/or region code that is used by an engine. b. Most of SearXNG's engines do not support all the languages from `language identification model`_ and there is also a discrepancy in the ISO-639-3 (fasttext) and ISO-639-2 (SearXNG)handling. Further more, in SearXNG the locales like ``zh-TH`` (``zh-CN``) are mapped to ``zh_Hant`` (``zh_Hans``) while the `language identification model`_ reduce both to ``zh``. .. _a fork: https://github.com/searxng/fasttext-predict .. _fastText: https://fasttext.cc/ .. _python fasttext: https://pypi.org/project/fasttext/ .. _language identification model: https://fasttext.cc/docs/en/language-identification.html .. _Bag of Tricks for Efficient Text Classification: https://arxiv.org/abs/1607.01759 .. _`FastText.zip: Compressing text classification models`: https://arxiv.org/abs/1612.03651
Definition at line 693 of file utils.py.
References _get_fasttext_model().
MutableMapping[str, t.Any] searx.utils.dict_subset | ( | MutableMapping[t.Any, t.Any] | dictionary, |
set[str] | properties ) |
Extract a subset of a dict Examples: >>> dict_subset({'A': 'a', 'B': 'b', 'C': 'c'}, ['A', 'C']) {'A': 'a', 'C': 'c'} >>> >> dict_subset({'A': 'a', 'B': 'b', 'C': 'c'}, ['A', 'D']) {'A': 'a'}
Definition at line 329 of file utils.py.
str searx.utils.ecma_unescape | ( | str | string | ) |
Python implementation of the unescape javascript function https://www.ecma-international.org/ecma-262/6.0/#sec-unescape-string https://developer.mozilla.org/fr/docs/Web/JavaScript/Reference/Objets_globaux/unescape Examples: >>> ecma_unescape('%u5409') '吉' >>> ecma_unescape('%20') ' ' >>> ecma_unescape('%F3') 'ó'
Definition at line 468 of file utils.py.
t.Any searx.utils.eval_xpath | ( | ElementBase | element, |
XPathSpecType | xpath_spec ) |
Equivalent of ``element.xpath(xpath_str)`` but compile ``xpath_str`` into a :py:obj:`lxml.etree.XPath` object once for all. The return value of ``xpath(..)`` is complex, read `XPath return values`_ for more details. .. _XPath return values: https://lxml.de/xpathxslt.html#xpath-return-values ``TypeError``: Raised when ``xpath_spec`` is neither a :py:obj:`str` nor a :py:obj:`lxml.etree.XPath`. ``SearxXPathSyntaxException``: Raised when there is a syntax error in the *XPath* selector (``str``). ``SearxEngineXPathException:`` Raised when the XPath can't be evaluated (masked :py:obj:`lxml.etree..XPathError`).
Definition at line 555 of file utils.py.
References get_xpath().
Referenced by eval_xpath_list().
t.Any searx.utils.eval_xpath_getindex | ( | ElementBase | element, |
XPathSpecType | xpath_spec, | ||
int | index, | ||
t.Any | default = _NOTSET ) |
Same as :py:obj:`searx.utils.eval_xpath_list`, but returns item on position ``index`` from the list (index starts with ``0``). The exceptions known from :py:obj:`searx.utils.eval_xpath` are thrown. If a default is specified, this is returned if an element at position ``index`` could not be determined.
Definition at line 596 of file utils.py.
References eval_xpath_list().
list[t.Any] searx.utils.eval_xpath_list | ( | ElementBase | element, |
XPathSpecType | xpath_spec, | ||
int | None | min_len = None ) |
Same as :py:obj:`searx.utils.eval_xpath`, but additionally ensures the return value is a :py:obj:`list`. The minimum length of the list is also checked (if ``min_len`` is set).
Definition at line 583 of file utils.py.
References eval_xpath().
Referenced by eval_xpath_getindex().
searx.utils.extr | ( | str | txt, |
str | begin, | ||
str | end, | ||
str | default = "" ) |
Extract the string between ``begin`` and ``end`` from ``txt`` :param txt: String to search in :param begin: First string to be searched for :param end: Second string to be searched for after ``begin`` :param default: Default value if one of ``begin`` or ``end`` is not found. Defaults to an empty string. :return: The string between the two search-strings ``begin`` and ``end``. If at least one of ``begin`` or ``end`` is not found, the value of ``default`` is returned. Examples: >>> extr("abcde", "a", "e") "bcd" >>> extr("abcde", "a", "z", deafult="nothing") "nothing"
Definition at line 372 of file utils.py.
str | None searx.utils.extract_text | ( | list[ElementBase] | ElementBase | str | Number | bool | None | xpath_results, |
bool | allow_none = False ) |
Extract text from a lxml result * if xpath_results is list, extract the text from each result and concat the list * if xpath_results is a xml element, extract all the text node from it ( text_content() method from lxml ) * if xpath_results is a string element, then it's already done
Definition at line 206 of file utils.py.
References extract_text().
Referenced by extract_text(), and extract_url().
str searx.utils.extract_url | ( | list[ElementBase] | ElementBase | str | Number | bool | None | xpath_results, |
str | base_url ) |
Extract and normalize URL from lxml Element Example: >>> def f(s, search_url): >>> return searx.utils.extract_url(html.fromstring(s), search_url) >>> f('<span id="42">https://example.com</span>', 'http://example.com/') 'https://example.com/' >>> f('https://example.com', 'http://example.com/') 'https://example.com/' >>> f('//example.com', 'http://example.com/') 'http://example.com/' >>> f('//example.com', 'https://example.com/') 'https://example.com/' >>> f('/path?a=1', 'https://example.com') 'https://example.com/path?a=1' >>> f('', 'https://example.com') raise lxml.etree.ParserError >>> searx.utils.extract_url([], 'https://example.com') raise ValueError Raises: * ValueError * lxml.etree.ParserError Returns: * str: normalized URL
Definition at line 292 of file utils.py.
References extract_text(), and normalize_url().
str searx.utils.gen_useragent | ( | str | None | os_string = None | ) |
Return a random browser User Agent See searx/data/useragents.json
Definition at line 75 of file utils.py.
searx.utils.get_embeded_stream_url | ( | str | url | ) |
Converts a standard video URL into its embed format. Supported services include Youtube, Facebook, Instagram, TikTok, Dailymotion, and Bilibili.
Definition at line 631 of file utils.py.
dict[str, dict[str, str]] searx.utils.get_engine_from_settings | ( | str | name | ) |
Return engine configuration from settings.yml of a given engine name
Definition at line 514 of file utils.py.
Callable[[str], str] searx.utils.get_string_replaces_function | ( | dict[str, str] | replaces | ) |
Definition at line 504 of file utils.py.
XPath searx.utils.get_xpath | ( | XPathSpecType | xpath_spec | ) |
Return cached compiled :py:obj:`lxml.etree.XPath` object. ``TypeError``: Raised when ``xpath_spec`` is neither a :py:obj:`str` nor a :py:obj:`lxml.etree.XPath`. ``SearxXPathSyntaxException``: Raised when there is a syntax error in the *XPath* selector (``str``).
Definition at line 529 of file utils.py.
Referenced by eval_xpath().
str searx.utils.html_to_text | ( | str | html_str | ) |
Extract text from a HTML string Args: * html_str (str): string HTML Returns: * str: extracted text Examples: >>> html_to_text('Example <span id="42">#2</span>') 'Example #2' >>> html_to_text('<style>.span { color: red; }</style><span>Example</span>') 'Example' >>> html_to_text(r'regexp: (?<![a-zA-Z]') 'regexp: (?<![a-zA-Z]' >>> html_to_text(r'<p><b>Lorem ipsum </i>dolor sit amet</p>') 'Lorem ipsum </i>dolor sit amet</p>' >>> html_to_text(r'> < a') '> < a'
Definition at line 142 of file utils.py.
Referenced by markdown_to_text().
searx.utils.humanize_bytes | ( | int | float | size, |
int | precision = 2 ) |
Determine the *human readable* value of bytes on 1024 base (1KB=1024B).
Definition at line 341 of file utils.py.
searx.utils.humanize_number | ( | int | float | size, |
int | precision = 0 ) |
Determine the *human readable* value of a decimal number.
Definition at line 353 of file utils.py.
int searx.utils.int_or_zero | ( | list[str] | str | num | ) |
Convert num to int or 0. num can be either a str or a list. If num is a list, the first element is converted to int (or return 0 if the list is empty). If num is a str, see convert_str_to_int
Definition at line 401 of file utils.py.
References convert_str_to_int().
tuple[bool, str, str] | None searx.utils.is_valid_lang | ( | str | lang | ) |
Return language code and name if lang describe a language. Examples: >>> is_valid_lang('zz') None >>> is_valid_lang('uk') (True, 'uk', 'ukrainian') >>> is_valid_lang(b'uk') (True, 'uk', 'ukrainian') >>> is_valid_lang('en') (True, 'en', 'english') >>> searx.utils.is_valid_lang('Español') (True, 'es', 'spanish') >>> searx.utils.is_valid_lang('Spanish') (True, 'es', 'spanish')
Definition at line 413 of file utils.py.
str searx.utils.js_variable_to_python | ( | str | js_variable | ) |
Convert a javascript variable into JSON and then load the value It does not deal with all cases, but it is good enough for now. chompjs has a better implementation.
Definition at line 764 of file utils.py.
types.ModuleType searx.utils.load_module | ( | str | filename, |
str | module_dir ) |
Definition at line 445 of file utils.py.
str searx.utils.markdown_to_text | ( | str | markdown_str | ) |
Extract text from a Markdown string Args: * markdown_str (str): string Markdown Returns: * str: extracted text Examples: >>> markdown_to_text('[example](https://example.com)') 'example' >>> markdown_to_text('## Headline') 'Headline'
Definition at line 183 of file utils.py.
References html_to_text().
str searx.utils.normalize_url | ( | str | url, |
str | base_url ) |
Normalize URL: add protocol, join URL with base_url, add trailing slash if there is no path Args: * url (str): Relative URL * base_url (str): Base URL, it must be an absolute URL. Example: >>> normalize_url('https://example.com', 'http://example.com/') 'https://example.com/' >>> normalize_url('//example.com', 'http://example.com/') 'http://example.com/' >>> normalize_url('//example.com', 'https://example.com/') 'https://example.com/' >>> normalize_url('/path?a=1', 'https://example.com') 'https://example.com/path?a=1' >>> normalize_url('', 'https://example.com') 'https://example.com/' >>> normalize_url('/test', '/path') raise ValueError Raises: * lxml.etree.ParserError Returns: * str: normalized URL
Definition at line 242 of file utils.py.
Referenced by extract_url().
timedelta | None searx.utils.parse_duration_string | ( | str | duration_str | ) |
Parse a time string in format MM:SS or HH:MM:SS and convert it to a `timedelta` object. Returns None if the provided string doesn't match any of the formats.
Definition at line 839 of file utils.py.
searx.utils.remove_pua_from_str | ( | str | string | ) |
Removes unicode's "PRIVATE USE CHARACTER"s (PUA_) from a string. .. _PUA: https://en.wikipedia.org/wiki/Private_Use_Areas
Definition at line 489 of file utils.py.
str searx.utils.searxng_useragent | ( | ) |
str searx.utils.to_string | ( | t.Any | obj | ) |
|
protected |
|
protected |
|
protected |
|
protected |
|
protected |
|
protected |
|
protected |
searx.utils.SEARCH_LANGUAGE_CODES = frozenset([searxng_locale[0].split('-')[0] for searxng_locale in sxng_locales]) |