.oO SearXNG Developer Documentation Oo.
Loading...
Searching...
No Matches
Functions | Variables
searx.botdetection._helpers Namespace Reference

Functions

 dump_request (flask.Request request)
 
werkzeug.Response|None too_many_requests (IPv4Network|IPv6Network network, str log_msg)
 
IPv4Network|IPv6Network get_network (IPv4Address|IPv6Address real_ip, config.Config cfg)
 
 _log_error_only_once (err_msg)
 
str get_real_ip (flask.Request request)
 

Variables

 logger = logger.getChild('botdetection')
 
list _logged_errors = []
 

Function Documentation

◆ _log_error_only_once()

searx.botdetection._helpers._log_error_only_once ( err_msg)
protected

Definition at line 63 of file _helpers.py.

63def _log_error_only_once(err_msg):
64 if err_msg not in _logged_errors:
65 logger.error(err_msg)
66 _logged_errors.append(err_msg)
67
68

Referenced by searx.botdetection._helpers.get_real_ip().

+ Here is the caller graph for this function:

◆ dump_request()

searx.botdetection._helpers.dump_request ( flask.Request request)

Definition at line 21 of file _helpers.py.

21def dump_request(request: flask.Request):
22 return (
23 request.path
24 + " || X-Forwarded-For: %s" % request.headers.get('X-Forwarded-For')
25 + " || X-Real-IP: %s" % request.headers.get('X-Real-IP')
26 + " || form: %s" % request.form
27 + " || Accept: %s" % request.headers.get('Accept')
28 + " || Accept-Language: %s" % request.headers.get('Accept-Language')
29 + " || Accept-Encoding: %s" % request.headers.get('Accept-Encoding')
30 + " || Content-Type: %s" % request.headers.get('Content-Type')
31 + " || Content-Length: %s" % request.headers.get('Content-Length')
32 + " || Connection: %s" % request.headers.get('Connection')
33 + " || User-Agent: %s" % request.headers.get('User-Agent')
34 )
35
36

◆ get_network()

IPv4Network | IPv6Network searx.botdetection._helpers.get_network ( IPv4Address | IPv6Address real_ip,
config.Config cfg ) 
Returns the (client) network of whether the real_ip is part of.

Definition at line 48 of file _helpers.py.

48def get_network(real_ip: IPv4Address | IPv6Address, cfg: config.Config) -> IPv4Network | IPv6Network:
49 """Returns the (client) network of whether the real_ip is part of."""
50
51 if real_ip.version == 6:
52 prefix = cfg['real_ip.ipv6_prefix']
53 else:
54 prefix = cfg['real_ip.ipv4_prefix']
55 network = ip_network(f"{real_ip}/{prefix}", strict=False)
56 # logger.debug("get_network(): %s", network.compressed)
57 return network
58
59

◆ get_real_ip()

str searx.botdetection._helpers.get_real_ip ( flask.Request request) 
Returns real IP of the request.  Since not all proxies set all the HTTP
headers and incoming headers can be faked it may happen that the IP cannot
be determined correctly.

.. sidebar:: :py:obj:`flask.Request.remote_addr`

   SearXNG uses Werkzeug's ProxyFix_ (with it default ``x_for=1``).

This function tries to get the remote IP in the order listed below,
additional some tests are done and if inconsistencies or errors are
detected, they are logged.

The remote IP of the request is taken from (first match):

- X-Forwarded-For_ header
- `X-real-IP header <https://github.com/searxng/searxng/issues/1237#issuecomment-1147564516>`__
- :py:obj:`flask.Request.remote_addr`

.. _ProxyFix:
   https://werkzeug.palletsprojects.com/middleware/proxy_fix/

.. _X-Forwarded-For:
  https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For

Definition at line 69 of file _helpers.py.

69def get_real_ip(request: flask.Request) -> str:
70 """Returns real IP of the request. Since not all proxies set all the HTTP
71 headers and incoming headers can be faked it may happen that the IP cannot
72 be determined correctly.
73
74 .. sidebar:: :py:obj:`flask.Request.remote_addr`
75
76 SearXNG uses Werkzeug's ProxyFix_ (with it default ``x_for=1``).
77
78 This function tries to get the remote IP in the order listed below,
79 additional some tests are done and if inconsistencies or errors are
80 detected, they are logged.
81
82 The remote IP of the request is taken from (first match):
83
84 - X-Forwarded-For_ header
85 - `X-real-IP header <https://github.com/searxng/searxng/issues/1237#issuecomment-1147564516>`__
86 - :py:obj:`flask.Request.remote_addr`
87
88 .. _ProxyFix:
89 https://werkzeug.palletsprojects.com/middleware/proxy_fix/
90
91 .. _X-Forwarded-For:
92 https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For
93
94 """
95
96 forwarded_for = request.headers.get("X-Forwarded-For")
97 real_ip = request.headers.get('X-Real-IP')
98 remote_addr = request.remote_addr
99 # logger.debug(
100 # "X-Forwarded-For: %s || X-Real-IP: %s || request.remote_addr: %s", forwarded_for, real_ip, remote_addr
101 # )
102
103 if not forwarded_for:
104 _log_error_only_once("X-Forwarded-For header is not set!")
105 else:
106 from . import cfg # pylint: disable=import-outside-toplevel, cyclic-import
107
108 forwarded_for = [x.strip() for x in forwarded_for.split(',')]
109 x_for: int = cfg['real_ip.x_for'] # type: ignore
110 forwarded_for = forwarded_for[-min(len(forwarded_for), x_for)]
111
112 if not real_ip:
113 _log_error_only_once("X-Real-IP header is not set!")
114
115 if forwarded_for and real_ip and forwarded_for != real_ip:
116 logger.warning("IP from X-Real-IP (%s) is not equal to IP from X-Forwarded-For (%s)", real_ip, forwarded_for)
117
118 if forwarded_for and remote_addr and forwarded_for != remote_addr:
119 logger.warning(
120 "IP from WSGI environment (%s) is not equal to IP from X-Forwarded-For (%s)", remote_addr, forwarded_for
121 )
122
123 if real_ip and remote_addr and real_ip != remote_addr:
124 logger.warning("IP from WSGI environment (%s) is not equal to IP from X-Real-IP (%s)", remote_addr, real_ip)
125
126 request_ip = forwarded_for or real_ip or remote_addr or '0.0.0.0'
127 # logger.debug("get_real_ip() -> %s", request_ip)
128 return request_ip

References searx.botdetection._helpers._log_error_only_once().

+ Here is the call graph for this function:

◆ too_many_requests()

werkzeug.Response | None searx.botdetection._helpers.too_many_requests ( IPv4Network | IPv6Network network,
str log_msg ) 
Returns a HTTP 429 response object and writes a ERROR message to the
'botdetection' logger.  This function is used in part by the filter methods
to return the default ``Too Many Requests`` response.

Definition at line 37 of file _helpers.py.

37def too_many_requests(network: IPv4Network | IPv6Network, log_msg: str) -> werkzeug.Response | None:
38 """Returns a HTTP 429 response object and writes a ERROR message to the
39 'botdetection' logger. This function is used in part by the filter methods
40 to return the default ``Too Many Requests`` response.
41
42 """
43
44 logger.debug("BLOCK %s: %s", network.compressed, log_msg)
45 return flask.make_response(('Too Many Requests', 429))
46
47

Variable Documentation

◆ _logged_errors

list searx.botdetection._helpers._logged_errors = []
protected

Definition at line 60 of file _helpers.py.

◆ logger

searx.botdetection._helpers.logger = logger.getChild('botdetection')

Definition at line 18 of file _helpers.py.