.oO SearXNG Developer Documentation Oo.
Loading...
Searching...
No Matches
Functions | Variables
searx.botdetection._helpers Namespace Reference

Functions

 dump_request (SXNG_Request request)
 
werkzeug.Response|None too_many_requests (IPv4Network|IPv6Network network, str log_msg)
 
IPv4Network|IPv6Network get_network (IPv4Address|IPv6Address real_ip, config.Config cfg)
 
 _log_error_only_once (err_msg)
 
str get_real_ip (SXNG_Request request)
 

Variables

 logger = logger.getChild('botdetection')
 
list _logged_errors = []
 

Function Documentation

◆ _log_error_only_once()

searx.botdetection._helpers._log_error_only_once ( err_msg)
protected

Definition at line 65 of file _helpers.py.

65def _log_error_only_once(err_msg):
66 if err_msg not in _logged_errors:
67 logger.error(err_msg)
68 _logged_errors.append(err_msg)
69
70

Referenced by get_real_ip().

+ Here is the caller graph for this function:

◆ dump_request()

searx.botdetection._helpers.dump_request ( SXNG_Request request)

Definition at line 23 of file _helpers.py.

23def dump_request(request: SXNG_Request):
24 return (
25 request.path
26 + " || X-Forwarded-For: %s" % request.headers.get('X-Forwarded-For')
27 + " || X-Real-IP: %s" % request.headers.get('X-Real-IP')
28 + " || form: %s" % request.form
29 + " || Accept: %s" % request.headers.get('Accept')
30 + " || Accept-Language: %s" % request.headers.get('Accept-Language')
31 + " || Accept-Encoding: %s" % request.headers.get('Accept-Encoding')
32 + " || Content-Type: %s" % request.headers.get('Content-Type')
33 + " || Content-Length: %s" % request.headers.get('Content-Length')
34 + " || Connection: %s" % request.headers.get('Connection')
35 + " || User-Agent: %s" % request.headers.get('User-Agent')
36 )
37
38

◆ get_network()

IPv4Network | IPv6Network searx.botdetection._helpers.get_network ( IPv4Address | IPv6Address real_ip,
config.Config cfg ) 
Returns the (client) network of whether the real_ip is part of.

Definition at line 50 of file _helpers.py.

50def get_network(real_ip: IPv4Address | IPv6Address, cfg: config.Config) -> IPv4Network | IPv6Network:
51 """Returns the (client) network of whether the real_ip is part of."""
52
53 if real_ip.version == 6:
54 prefix = cfg['real_ip.ipv6_prefix']
55 else:
56 prefix = cfg['real_ip.ipv4_prefix']
57 network = ip_network(f"{real_ip}/{prefix}", strict=False)
58 # logger.debug("get_network(): %s", network.compressed)
59 return network
60
61

◆ get_real_ip()

str searx.botdetection._helpers.get_real_ip ( SXNG_Request request) 
Returns real IP of the request.  Since not all proxies set all the HTTP
headers and incoming headers can be faked it may happen that the IP cannot
be determined correctly.

.. sidebar:: :py:obj:`flask.Request.remote_addr`

   SearXNG uses Werkzeug's ProxyFix_ (with it default ``x_for=1``).

This function tries to get the remote IP in the order listed below,
additional some tests are done and if inconsistencies or errors are
detected, they are logged.

The remote IP of the request is taken from (first match):

- X-Forwarded-For_ header
- `X-real-IP header <https://github.com/searxng/searxng/issues/1237#issuecomment-1147564516>`__
- :py:obj:`flask.Request.remote_addr`

.. _ProxyFix:
   https://werkzeug.palletsprojects.com/middleware/proxy_fix/

.. _X-Forwarded-For:
  https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For

Definition at line 71 of file _helpers.py.

71def get_real_ip(request: SXNG_Request) -> str:
72 """Returns real IP of the request. Since not all proxies set all the HTTP
73 headers and incoming headers can be faked it may happen that the IP cannot
74 be determined correctly.
75
76 .. sidebar:: :py:obj:`flask.Request.remote_addr`
77
78 SearXNG uses Werkzeug's ProxyFix_ (with it default ``x_for=1``).
79
80 This function tries to get the remote IP in the order listed below,
81 additional some tests are done and if inconsistencies or errors are
82 detected, they are logged.
83
84 The remote IP of the request is taken from (first match):
85
86 - X-Forwarded-For_ header
87 - `X-real-IP header <https://github.com/searxng/searxng/issues/1237#issuecomment-1147564516>`__
88 - :py:obj:`flask.Request.remote_addr`
89
90 .. _ProxyFix:
91 https://werkzeug.palletsprojects.com/middleware/proxy_fix/
92
93 .. _X-Forwarded-For:
94 https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For
95
96 """
97
98 forwarded_for = request.headers.get("X-Forwarded-For")
99 real_ip = request.headers.get('X-Real-IP')
100 remote_addr = request.remote_addr
101 # logger.debug(
102 # "X-Forwarded-For: %s || X-Real-IP: %s || request.remote_addr: %s", forwarded_for, real_ip, remote_addr
103 # )
104
105 if not forwarded_for:
106 _log_error_only_once("X-Forwarded-For header is not set!")
107 else:
108 from . import cfg # pylint: disable=import-outside-toplevel, cyclic-import
109
110 forwarded_for = [x.strip() for x in forwarded_for.split(',')]
111 x_for: int = cfg['real_ip.x_for'] # type: ignore
112 forwarded_for = forwarded_for[-min(len(forwarded_for), x_for)]
113
114 if not real_ip:
115 _log_error_only_once("X-Real-IP header is not set!")
116
117 if forwarded_for and real_ip and forwarded_for != real_ip:
118 logger.warning("IP from X-Real-IP (%s) is not equal to IP from X-Forwarded-For (%s)", real_ip, forwarded_for)
119
120 if forwarded_for and remote_addr and forwarded_for != remote_addr:
121 logger.warning(
122 "IP from WSGI environment (%s) is not equal to IP from X-Forwarded-For (%s)", remote_addr, forwarded_for
123 )
124
125 if real_ip and remote_addr and real_ip != remote_addr:
126 logger.warning("IP from WSGI environment (%s) is not equal to IP from X-Real-IP (%s)", remote_addr, real_ip)
127
128 request_ip = forwarded_for or real_ip or remote_addr or '0.0.0.0'
129 # logger.debug("get_real_ip() -> %s", request_ip)
130 return request_ip

References _log_error_only_once().

+ Here is the call graph for this function:

◆ too_many_requests()

werkzeug.Response | None searx.botdetection._helpers.too_many_requests ( IPv4Network | IPv6Network network,
str log_msg ) 
Returns a HTTP 429 response object and writes a ERROR message to the
'botdetection' logger.  This function is used in part by the filter methods
to return the default ``Too Many Requests`` response.

Definition at line 39 of file _helpers.py.

39def too_many_requests(network: IPv4Network | IPv6Network, log_msg: str) -> werkzeug.Response | None:
40 """Returns a HTTP 429 response object and writes a ERROR message to the
41 'botdetection' logger. This function is used in part by the filter methods
42 to return the default ``Too Many Requests`` response.
43
44 """
45
46 logger.debug("BLOCK %s: %s", network.compressed, log_msg)
47 return flask.make_response(('Too Many Requests', 429))
48
49

Variable Documentation

◆ _logged_errors

list searx.botdetection._helpers._logged_errors = []
protected

Definition at line 62 of file _helpers.py.

◆ logger

searx.botdetection._helpers.logger = logger.getChild('botdetection')

Definition at line 20 of file _helpers.py.