searx.botdetection.config Namespace Reference

Classes
class	Config
class	FALSE
class	SchemaIssue

Functions
	toml_load (file_name)
	value (str name, dict data_dict)
typing.Tuple[bool, list]	validate (typing.Dict schema_dict, typing.Dict data_dict, typing.Dict[str, str] deprecated)
typing.Tuple[bool, typing.List]	_validate (typing.List names, typing.List issue_list, typing.Dict schema_dict, typing.Dict data_dict, typing.Dict[str, str] deprecated)
	dict_deepupdate (dict base_dict, dict upd_dict, names=None)

Variables
	pytomlpp = None
bool	USE_TOMLLIB = True
	tomllib = None
list	__all__ = ['Config', 'UNSET', 'SchemaIssue']
	log = logging.getLogger(__name__)
	UNSET = FALSE('<UNSET>')

Detailed Description

Configuration class :py:class:`Config` with deep-update, schema validation
and deprecated names.

The :py:class:`Config` class implements a configuration that is based on
structured dictionaries.  The configuration schema is defined in a dictionary
structure and the configuration data is given in a dictionary structure.

Function Documentation

_validate()

typing.Tuple[bool, typing.List] searx.botdetection.config._validate ( typing.List names, typing.List issue_list, typing.Dict schema_dict, typing.Dict data_dict, typing.Dict[str, str] deprecated )

protected

Definition at line 276 of file config.py.

) -> typing.Tuple[bool, typing.List]:
 
    is_valid = True
 
    for key, data_value in data_dict.items():
 
        names.append(key)
        name = '.'.join(names)
 
        deprecated_msg = deprecated.get(name)
        # print("XXX %s: key %s //   data_value: %s" % (name, key, data_value))
        if deprecated_msg:
            issue_list.append(SchemaIssue('warn', f"data_dict '{name}': deprecated - {deprecated_msg}"))
 
        schema_value = value(name, schema_dict)
        # print("YYY %s: key %s // schema_value: %s" % (name, key, schema_value))
        if schema_value is UNSET:
            if not deprecated_msg:
                issue_list.append(SchemaIssue('invalid', f"data_dict '{name}': key unknown in schema_dict"))
                is_valid = False
 
        elif type(schema_value) != type(data_value):  # pylint: disable=unidiomatic-typecheck
            issue_list.append(
                SchemaIssue(
                    'invalid',
                    (f"data_dict: type mismatch '{name}':" f" expected {type(schema_value)}, is: {type(data_value)}"),
                )
            )
            is_valid = False
 
        elif isinstance(data_value, dict):
            _valid, _ = _validate(names, issue_list, schema_dict, data_value, deprecated)
            is_valid = is_valid and _valid
        names.pop()
 
    return is_valid, issue_list
 
 

References searx.botdetection.config._validate(), and searx.botdetection.config.value().

Referenced by searx.botdetection.config._validate(), and searx.botdetection.config.validate().

Here is the call graph for this function:

Here is the caller graph for this function:

dict_deepupdate()

searx.botdetection.config.dict_deepupdate	(	dict	base_dict,
		dict	upd_dict,
			names = None )

Deep-update of dictionary in ``base_dict`` by dictionary in ``upd_dict``.

For each ``upd_key`` & ``upd_val`` pair in ``upd_dict``:

0. If types of ``base_dict[upd_key]`` and ``upd_val`` do not match raise a
   :py:obj:`TypeError`.

1. If ``base_dict[upd_key]`` is a dict: recursively deep-update it by ``upd_val``.

2. If ``base_dict[upd_key]`` not exist: set ``base_dict[upd_key]`` from a
   (deep-) copy of ``upd_val``.

3. If ``upd_val`` is a list, extend list in ``base_dict[upd_key]`` by the
   list in ``upd_val``.

4. If ``upd_val`` is a set, update set in ``base_dict[upd_key]`` by set in
   ``upd_val``.

Definition at line 320 of file config.py.

def dict_deepupdate(base_dict: dict, upd_dict: dict, names=None):
    """Deep-update of dictionary in ``base_dict`` by dictionary in ``upd_dict``.
 
    For each ``upd_key`` & ``upd_val`` pair in ``upd_dict``:
 
    0. If types of ``base_dict[upd_key]`` and ``upd_val`` do not match raise a
       :py:obj:`TypeError`.
 
    1. If ``base_dict[upd_key]`` is a dict: recursively deep-update it by ``upd_val``.
 
    2. If ``base_dict[upd_key]`` not exist: set ``base_dict[upd_key]`` from a
       (deep-) copy of ``upd_val``.
 
    3. If ``upd_val`` is a list, extend list in ``base_dict[upd_key]`` by the
       list in ``upd_val``.
 
    4. If ``upd_val`` is a set, update set in ``base_dict[upd_key]`` by set in
       ``upd_val``.
    """
    # pylint: disable=too-many-branches
    if not isinstance(base_dict, dict):
        raise TypeError("argument 'base_dict' is not a ditionary type")
    if not isinstance(upd_dict, dict):
        raise TypeError("argument 'upd_dict' is not a ditionary type")
 
    if names is None:
        names = []
 
    for upd_key, upd_val in upd_dict.items():
        # For each upd_key & upd_val pair in upd_dict:
 
        if isinstance(upd_val, dict):
 
            if upd_key in base_dict:
                # if base_dict[upd_key] exists, recursively deep-update it
                if not isinstance(base_dict[upd_key], dict):
                    raise TypeError(f"type mismatch {'.'.join(names)}: is not a dict type in base_dict")
                dict_deepupdate(
                    base_dict[upd_key],
                    upd_val,
                    names
                    + [
                        upd_key,
                    ],
                )
 
            else:
                # if base_dict[upd_key] not exist, set base_dict[upd_key] from deepcopy of upd_val
                base_dict[upd_key] = copy.deepcopy(upd_val)
 
        elif isinstance(upd_val, list):
 
            if upd_key in base_dict:
                # if base_dict[upd_key] exists, base_dict[up_key] is extended by
                # the list from upd_val
                if not isinstance(base_dict[upd_key], list):
                    raise TypeError(f"type mismatch {'.'.join(names)}: is not a list type in base_dict")
                base_dict[upd_key].extend(upd_val)
 
            else:
                # if base_dict[upd_key] doesn't exists, set base_dict[key] from a deepcopy of the
                # list in upd_val.
                base_dict[upd_key] = copy.deepcopy(upd_val)
 
        elif isinstance(upd_val, set):
 
            if upd_key in base_dict:
                # if base_dict[upd_key] exists, base_dict[up_key] is updated by the set in upd_val
                if not isinstance(base_dict[upd_key], set):
                    raise TypeError(f"type mismatch {'.'.join(names)}: is not a set type in base_dict")
                base_dict[upd_key].update(upd_val.copy())
 
            else:
                # if base_dict[upd_key] doesn't exists, set base_dict[upd_key] from a copy of the
                # set in upd_val
                base_dict[upd_key] = upd_val.copy()
 
        else:
            # for any other type of upd_val replace or add base_dict[upd_key] by a copy
            # of upd_val
            base_dict[upd_key] = copy.copy(upd_val)

References searx.botdetection.config.dict_deepupdate().

Referenced by searx.botdetection.config.dict_deepupdate(), and searx.botdetection.config.Config.update().

Here is the call graph for this function:

Here is the caller graph for this function:

toml_load()

searx.botdetection.config.toml_load

(

file_name

)

Definition at line 185 of file config.py.

def toml_load(file_name):
    if USE_TOMLLIB:
        # Python >= 3.11
        try:
            with open(file_name, "rb") as f:
                return tomllib.load(f)
        except tomllib.TOMLDecodeError as exc:
            msg = str(exc).replace('\t', '').replace('\n', ' ')
            log.error("%s: %s", file_name, msg)
            raise
    # fallback to pytomlpp for Python < 3.11
    try:
        return pytomlpp.load(file_name)
    except pytomlpp.DecodeError as exc:
        msg = str(exc).replace('\t', '').replace('\n', ' ')
        log.error("%s: %s", file_name, msg)
        raise
 
 
# working with dictionaries
 
 

Referenced by searx.botdetection.config.Config.from_toml().

Here is the caller graph for this function:

validate()

typing.Tuple[bool, list] searx.botdetection.config.validate	(	typing.Dict	schema_dict,
		typing.Dict	data_dict,
		typing.Dict[str, str]	deprecated )

Deep validation of dictionary in ``data_dict`` against dictionary in
``schema_dict``.  Argument deprecated is a dictionary that maps deprecated
configuration names to a messages::

    deprecated = {
        "foo.bar" : "config 'foo.bar' is deprecated, use 'bar.foo'",
        "..."     : "..."
    }

The function returns a python tuple ``(is_valid, issue_list)``:

``is_valid``:
  A bool value indicating ``data_dict`` is valid or not.

``issue_list``:
  A list of messages (:py:obj:`SchemaIssue`) from the validation::

      [schema warn] data_dict: deprecated 'fontlib.foo': <DEPRECATED['foo.bar']>
      [schema invalid] data_dict: key unknown 'fontlib.foo'
      [schema invalid] data_dict: type mismatch 'fontlib.foo': expected ..., is ...

If ``schema_dict`` or ``data_dict`` is not a dictionary type a
:py:obj:`SchemaIssue` is raised.

Definition at line 235 of file config.py.

) -> typing.Tuple[bool, list]:
    """Deep validation of dictionary in ``data_dict`` against dictionary in
    ``schema_dict``.  Argument deprecated is a dictionary that maps deprecated
    configuration names to a messages::
 
        deprecated = {
            "foo.bar" : "config 'foo.bar' is deprecated, use 'bar.foo'",
            "..."     : "..."
        }
 
    The function returns a python tuple ``(is_valid, issue_list)``:
 
    ``is_valid``:
      A bool value indicating ``data_dict`` is valid or not.
 
    ``issue_list``:
      A list of messages (:py:obj:`SchemaIssue`) from the validation::
 
          [schema warn] data_dict: deprecated 'fontlib.foo': <DEPRECATED['foo.bar']>
          [schema invalid] data_dict: key unknown 'fontlib.foo'
          [schema invalid] data_dict: type mismatch 'fontlib.foo': expected ..., is ...
 
    If ``schema_dict`` or ``data_dict`` is not a dictionary type a
    :py:obj:`SchemaIssue` is raised.
 
    """
    names = []
    is_valid = True
    issue_list = []
 
    if not isinstance(schema_dict, dict):
        raise SchemaIssue('invalid', "schema_dict is not a dict type")
    if not isinstance(data_dict, dict):
        raise SchemaIssue('invalid', f"data_dict issue{'.'.join(names)} is not a dict type")
 
    is_valid, issue_list = _validate(names, issue_list, schema_dict, data_dict, deprecated)
    return is_valid, issue_list
 
 

References searx.botdetection.config._validate().

Here is the call graph for this function:

value()

searx.botdetection.config.value	(	str	name,
		dict	data_dict )

Returns the value to which ``name`` points in the ``dat_dict``.

.. code: python

    >>> data_dict = {
            "foo": {"bar": 1 },
            "bar": {"foo": 2 },
            "foobar": [1, 2, 3],
        }
    >>> value('foobar', data_dict)
    [1, 2, 3]
    >>> value('foo.bar', data_dict)
    1
    >>> value('foo.bar.xxx', data_dict)
    <UNSET>

Definition at line 207 of file config.py.

def value(name: str, data_dict: dict):
    """Returns the value to which ``name`` points in the ``dat_dict``.
 
    .. code: python
 
        >>> data_dict = {
                "foo": {"bar": 1 },
                "bar": {"foo": 2 },
                "foobar": [1, 2, 3],
            }
        >>> value('foobar', data_dict)
        [1, 2, 3]
        >>> value('foo.bar', data_dict)
        1
        >>> value('foo.bar.xxx', data_dict)
        <UNSET>
 
    """
 
    ret_val = data_dict
    for part in name.split('.'):
        if isinstance(ret_val, dict):
            ret_val = ret_val.get(part, UNSET)
        if ret_val is UNSET:
            break
    return ret_val
 
 

Referenced by searx.botdetection.config.Config._get_parent_dict(), searx.botdetection.config._validate(), and searx.botdetection.config.Config.default().

Here is the caller graph for this function:

Variable Documentation

__all__

list searx.botdetection.config.__all__ = ['Config', 'UNSET', 'SchemaIssue']

private

Definition at line 29 of file config.py.

log

searx.botdetection.config.log = logging.getLogger(__name__)

Definition at line 31 of file config.py.

pytomlpp

searx.botdetection.config.pytomlpp = None

Definition at line 20 of file config.py.

tomllib

searx.botdetection.config.tomllib = None

Definition at line 25 of file config.py.

UNSET

searx.botdetection.config.UNSET = FALSE('<UNSET>')

Definition at line 50 of file config.py.

USE_TOMLLIB

bool searx.botdetection.config.USE_TOMLLIB = True

Definition at line 21 of file config.py.