.oO SearXNG Developer Documentation Oo.
Loading...
Searching...
No Matches
searx.botdetection.config Namespace Reference

Classes

class  Config
class  FALSE
class  SchemaIssue

Functions

 set_global_cfg ("Config" cfg)
"Config" get_global_cfg ()
 toml_load (str|pathlib.Path file_name)
 value (str name, dict[str, typing.Any] data_dict)
tuple[bool, list[SchemaIssue]] validate (dict[str, typing.Any] schema_dict, dict[str, typing.Any] data_dict, dict[str, str] deprecated)
tuple[bool, list[SchemaIssue]] _validate (list[str] names, list[SchemaIssue] issue_list, dict[str, typing.Any] schema_dict, dict[str, typing.Any] data_dict, dict[str, str] deprecated)
 dict_deepupdate (dict[str, typing.Any] base_dict, dict[str, typing.Any] upd_dict, list[str]|None names=None)

Variables

list __all__ = ['Config', 'UNSET', 'SchemaIssue', 'set_global_cfg', 'get_global_cfg']
 log = logging.getLogger(__name__)
str CFG = None
 UNSET = FALSE('<UNSET>')

Detailed Description

Configuration class :py:class:`Config` with deep-update, schema validation
and deprecated names.

The :py:class:`Config` class implements a configuration that is based on
structured dictionaries.  The configuration schema is defined in a dictionary
structure and the configuration data is given in a dictionary structure.

Function Documentation

◆ _validate()

tuple[bool, list[SchemaIssue]] searx.botdetection.config._validate ( list[str] names,
list[SchemaIssue] issue_list,
dict[str, typing.Any] schema_dict,
dict[str, typing.Any] data_dict,
dict[str, str] deprecated )
protected

Definition at line 272 of file config.py.

278) -> tuple[bool, list[SchemaIssue]]:
279
280 is_valid = True
281
282 data_value: dict[str, typing.Any]
283 for key, data_value in data_dict.items():
284
285 names.append(key)
286 name = '.'.join(names)
287
288 deprecated_msg = deprecated.get(name)
289 # print("XXX %s: key %s // data_value: %s" % (name, key, data_value))
290 if deprecated_msg:
291 issue_list.append(SchemaIssue('warn', f"data_dict '{name}': deprecated - {deprecated_msg}"))
292
293 schema_value = value(name, schema_dict)
294 # print("YYY %s: key %s // schema_value: %s" % (name, key, schema_value))
295 if schema_value is UNSET:
296 if not deprecated_msg:
297 issue_list.append(SchemaIssue('invalid', f"data_dict '{name}': key unknown in schema_dict"))
298 is_valid = False
299
300 elif type(schema_value) != type(data_value): # pylint: disable=unidiomatic-typecheck
301 issue_list.append(
302 SchemaIssue(
303 'invalid',
304 (f"data_dict: type mismatch '{name}':" f" expected {type(schema_value)}, is: {type(data_value)}"),
305 )
306 )
307 is_valid = False
308
309 elif isinstance(data_value, dict):
310 _valid, _ = _validate(names, issue_list, schema_dict, data_value, deprecated)
311 is_valid = is_valid and _valid
312 names.pop()
313
314 return is_valid, issue_list
315
316

References _validate(), and value().

Referenced by _validate(), and validate().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ dict_deepupdate()

searx.botdetection.config.dict_deepupdate ( dict[str, typing.Any] base_dict,
dict[str, typing.Any] upd_dict,
list[str] | None names = None )
Deep-update of dictionary in ``base_dict`` by dictionary in ``upd_dict``.

For each ``upd_key`` & ``upd_val`` pair in ``upd_dict``:

0. If types of ``base_dict[upd_key]`` and ``upd_val`` do not match raise a
   :py:obj:`TypeError`.

1. If ``base_dict[upd_key]`` is a dict: recursively deep-update it by ``upd_val``.

2. If ``base_dict[upd_key]`` not exist: set ``base_dict[upd_key]`` from a
   (deep-) copy of ``upd_val``.

3. If ``upd_val`` is a list, extend list in ``base_dict[upd_key]`` by the
   list in ``upd_val``.

4. If ``upd_val`` is a set, update set in ``base_dict[upd_key]`` by set in
   ``upd_val``.

Definition at line 317 of file config.py.

317def dict_deepupdate(base_dict: dict[str, typing.Any], upd_dict: dict[str, typing.Any], names: list[str] | None = None):
318 """Deep-update of dictionary in ``base_dict`` by dictionary in ``upd_dict``.
319
320 For each ``upd_key`` & ``upd_val`` pair in ``upd_dict``:
321
322 0. If types of ``base_dict[upd_key]`` and ``upd_val`` do not match raise a
323 :py:obj:`TypeError`.
324
325 1. If ``base_dict[upd_key]`` is a dict: recursively deep-update it by ``upd_val``.
326
327 2. If ``base_dict[upd_key]`` not exist: set ``base_dict[upd_key]`` from a
328 (deep-) copy of ``upd_val``.
329
330 3. If ``upd_val`` is a list, extend list in ``base_dict[upd_key]`` by the
331 list in ``upd_val``.
332
333 4. If ``upd_val`` is a set, update set in ``base_dict[upd_key]`` by set in
334 ``upd_val``.
335 """
336 # pylint: disable=too-many-branches
337 if not isinstance(base_dict, dict):
338 raise TypeError("argument 'base_dict' is not a dictionary type")
339 if not isinstance(upd_dict, dict):
340 raise TypeError("argument 'upd_dict' is not a dictionary type")
341
342 if names is None:
343 names = []
344
345 for upd_key, upd_val in upd_dict.items():
346 # For each upd_key & upd_val pair in upd_dict:
347
348 if isinstance(upd_val, dict):
349
350 if upd_key in base_dict:
351 # if base_dict[upd_key] exists, recursively deep-update it
352 if not isinstance(base_dict[upd_key], dict):
353 raise TypeError(f"type mismatch {'.'.join(names)}: is not a dict type in base_dict")
354 dict_deepupdate(
355 base_dict[upd_key],
356 upd_val, # pyright: ignore[reportUnknownArgumentType]
357 names
358 + [
359 upd_key,
360 ],
361 )
362
363 else:
364 # if base_dict[upd_key] not exist, set base_dict[upd_key] from deepcopy of upd_val
365 base_dict[upd_key] = copy.deepcopy(upd_val) # pyright: ignore[reportUnknownArgumentType]
366
367 elif isinstance(upd_val, list):
368
369 if upd_key in base_dict:
370 # if base_dict[upd_key] exists, base_dict[up_key] is extended by
371 # the list from upd_val
372 if not isinstance(base_dict[upd_key], list):
373 raise TypeError(f"type mismatch {'.'.join(names)}: is not a list type in base_dict")
374 base_dict[upd_key].extend(upd_val)
375
376 else:
377 # if base_dict[upd_key] doesn't exists, set base_dict[key] from a deepcopy of the
378 # list in upd_val.
379 base_dict[upd_key] = copy.deepcopy(upd_val) # pyright: ignore[reportUnknownArgumentType]
380
381 elif isinstance(upd_val, set):
382
383 if upd_key in base_dict:
384 # if base_dict[upd_key] exists, base_dict[up_key] is updated by the set in upd_val
385 if not isinstance(base_dict[upd_key], set):
386 raise TypeError(f"type mismatch {'.'.join(names)}: is not a set type in base_dict")
387 base_dict[upd_key].update(upd_val.copy())
388
389 else:
390 # if base_dict[upd_key] doesn't exists, set base_dict[upd_key] from a copy of the
391 # set in upd_val
392 base_dict[upd_key] = upd_val.copy()
393
394 else:
395 # for any other type of upd_val replace or add base_dict[upd_key] by a copy
396 # of upd_val
397 base_dict[upd_key] = copy.copy(upd_val)

References dict_deepupdate().

Referenced by dict_deepupdate(), and searx.botdetection.config.Config.update().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ get_global_cfg()

"Config" searx.botdetection.config.get_global_cfg ( )

Definition at line 31 of file config.py.

31def get_global_cfg() -> "Config":
32 if CFG is None:
33 raise ValueError("Botdetection's config is not yet initialized.")
34 return CFG
35
36
37@typing.final

◆ set_global_cfg()

searx.botdetection.config.set_global_cfg ( "Config" cfg)

Definition at line 26 of file config.py.

26def set_global_cfg(cfg: "Config"):
27 global CFG # pylint: disable=global-statement
28 CFG = cfg
29
30

◆ toml_load()

searx.botdetection.config.toml_load ( str | pathlib.Path file_name)

Definition at line 190 of file config.py.

190def toml_load(file_name: str | pathlib.Path):
191 try:
192 with open(file_name, "rb") as f:
193 return tomllib.load(f)
194 except tomllib.TOMLDecodeError as exc:
195 msg = str(exc).replace('\t', '').replace('\n', ' ')
196 log.error("%s: %s", file_name, msg)
197 raise
198
199
200# working with dictionaries
201
202

Referenced by searx.botdetection.config.Config.from_toml().

Here is the caller graph for this function:

◆ validate()

tuple[bool, list[SchemaIssue]] searx.botdetection.config.validate ( dict[str, typing.Any] schema_dict,
dict[str, typing.Any] data_dict,
dict[str, str] deprecated )
Deep validation of dictionary in ``data_dict`` against dictionary in
``schema_dict``.  Argument deprecated is a dictionary that maps deprecated
configuration names to a messages::

    deprecated = {
        "foo.bar" : "config 'foo.bar' is deprecated, use 'bar.foo'",
        "..."     : "..."
    }

The function returns a python tuple ``(is_valid, issue_list)``:

``is_valid``:
  A bool value indicating ``data_dict`` is valid or not.

``issue_list``:
  A list of messages (:py:obj:`SchemaIssue`) from the validation::

      [schema warn] data_dict: deprecated 'fontlib.foo': <DEPRECATED['foo.bar']>
      [schema invalid] data_dict: key unknown 'fontlib.foo'
      [schema invalid] data_dict: type mismatch 'fontlib.foo': expected ..., is ...

If ``schema_dict`` or ``data_dict`` is not a dictionary type a
:py:obj:`SchemaIssue` is raised.

Definition at line 231 of file config.py.

233) -> tuple[bool, list[SchemaIssue]]:
234 """Deep validation of dictionary in ``data_dict`` against dictionary in
235 ``schema_dict``. Argument deprecated is a dictionary that maps deprecated
236 configuration names to a messages::
237
238 deprecated = {
239 "foo.bar" : "config 'foo.bar' is deprecated, use 'bar.foo'",
240 "..." : "..."
241 }
242
243 The function returns a python tuple ``(is_valid, issue_list)``:
244
245 ``is_valid``:
246 A bool value indicating ``data_dict`` is valid or not.
247
248 ``issue_list``:
249 A list of messages (:py:obj:`SchemaIssue`) from the validation::
250
251 [schema warn] data_dict: deprecated 'fontlib.foo': <DEPRECATED['foo.bar']>
252 [schema invalid] data_dict: key unknown 'fontlib.foo'
253 [schema invalid] data_dict: type mismatch 'fontlib.foo': expected ..., is ...
254
255 If ``schema_dict`` or ``data_dict`` is not a dictionary type a
256 :py:obj:`SchemaIssue` is raised.
257
258 """
259 names: list[str] = []
260 is_valid: bool = True
261 issue_list: list[SchemaIssue] = []
262
263 if not isinstance(schema_dict, dict):
264 raise SchemaIssue('invalid', "schema_dict is not a dict type")
265 if not isinstance(data_dict, dict):
266 raise SchemaIssue('invalid', f"data_dict issue{'.'.join(names)} is not a dict type")
267
268 is_valid, issue_list = _validate(names, issue_list, schema_dict, data_dict, deprecated)
269 return is_valid, issue_list
270
271

References _validate().

Here is the call graph for this function:

◆ value()

searx.botdetection.config.value ( str name,
dict[str, typing.Any] data_dict )
Returns the value to which ``name`` points in the ``dat_dict``.

.. code: python

    >>> data_dict = {
            "foo": {"bar": 1 },
            "bar": {"foo": 2 },
            "foobar": [1, 2, 3],
        }
    >>> value('foobar', data_dict)
    [1, 2, 3]
    >>> value('foo.bar', data_dict)
    1
    >>> value('foo.bar.xxx', data_dict)
    <UNSET>

Definition at line 203 of file config.py.

203def value(name: str, data_dict: dict[str, typing.Any]):
204 """Returns the value to which ``name`` points in the ``dat_dict``.
205
206 .. code: python
207
208 >>> data_dict = {
209 "foo": {"bar": 1 },
210 "bar": {"foo": 2 },
211 "foobar": [1, 2, 3],
212 }
213 >>> value('foobar', data_dict)
214 [1, 2, 3]
215 >>> value('foo.bar', data_dict)
216 1
217 >>> value('foo.bar.xxx', data_dict)
218 <UNSET>
219
220 """
221
222 ret_val = data_dict
223 for part in name.split('.'):
224 if isinstance(ret_val, dict):
225 ret_val = ret_val.get(part, UNSET)
226 if ret_val is UNSET:
227 break
228 return ret_val
229
230

Referenced by searx.botdetection.config.Config._get_parent_dict(), _validate(), and searx.botdetection.config.Config.default().

Here is the caller graph for this function:

Variable Documentation

◆ __all__

list searx.botdetection.config.__all__ = ['Config', 'UNSET', 'SchemaIssue', 'set_global_cfg', 'get_global_cfg']
private

Definition at line 18 of file config.py.

◆ CFG

str searx.botdetection.config.CFG = None

Definition at line 22 of file config.py.

◆ log

searx.botdetection.config.log = logging.getLogger(__name__)

Definition at line 20 of file config.py.

◆ UNSET

searx.botdetection.config.UNSET = FALSE('<UNSET>')

Definition at line 54 of file config.py.