.oO SearXNG Developer Documentation Oo.
Loading...
Searching...
No Matches
searx.limiter Namespace Reference

Functions

config.Config get_cfg ()
werkzeug.Response|None filter_request (SXNG_Request request)
 pre_request ()
 is_installed ()
 initialize (flask.Flask app, settings)

Variables

 logger = logger.getChild('limiter')
config CFG = None
bool _INSTALLED = False
str LIMITER_CFG_SCHEMA = Path(__file__).parent / "limiter.toml"

Detailed Description

Bot protection / IP rate limitation.  The intention of rate limitation is to
limit suspicious requests from an IP.  The motivation behind this is the fact
that SearXNG passes through requests from bots and is thus classified as a bot
itself.  As a result, the SearXNG engine then receives a CAPTCHA or is blocked
by the search engine (the origin) in some other way.

To avoid blocking, the requests from bots to SearXNG must also be blocked, this
is the task of the limiter.  To perform this task, the limiter uses the methods
from the :ref:`botdetection`:

- Analysis of the HTTP header in the request / :ref:`botdetection probe headers`
  can be easily bypassed.

- Block and pass lists in which IPs are listed / :ref:`botdetection ip_lists`
  are hard to maintain, since the IPs of bots are not all known and change over
  the time.

- Detection & dynamically :ref:`botdetection rate limit` of bots based on the
  behavior of the requests.  For dynamically changeable IP lists a Valkey
  database is needed.

The prerequisite for IP based methods is the correct determination of the IP of
the client. The IP of the client is determined via the X-Forwarded-For_ HTTP
header.

.. attention::

   A correct setup of the HTTP request headers ``X-Forwarded-For`` and
   ``X-Real-IP`` is essential to be able to assign a request to an IP correctly:

   - `NGINX RequestHeader`_
   - `Apache RequestHeader`_

.. _X-Forwarded-For:
    https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For
.. _NGINX RequestHeader:
    https://docs.searxng.org/admin/installation-nginx.html#nginx-s-searxng-site
.. _Apache RequestHeader:
    https://docs.searxng.org/admin/installation-apache.html#apache-s-searxng-site

Enable Limiter
==============

To enable the limiter activate:

.. code:: yaml

   server:
     ...
     limiter: true  # rate limit the number of request on the instance, block some bots

and set the valkey-url connection. Check the value, it depends on your valkey DB
(see :ref:`settings valkey`), by example:

.. code:: yaml

   valkey:
     url: valkey://localhost:6379/0


Configure Limiter
=================

The methods of :ref:`botdetection` the limiter uses are configured in a local
file ``/etc/searxng/limiter.toml``.  The defaults are shown in limiter.toml_ /
Don't copy all values to your local configuration, just enable what you need by
overwriting the defaults.  For instance to activate the ``link_token`` method in
the :ref:`botdetection.ip_limit` you only need to set this option to ``true``:

.. code:: toml

   [botdetection.ip_limit]
   link_token = true

.. _limiter.toml:

``limiter.toml``
================

In this file the limiter finds the configuration of the :ref:`botdetection`:

- :ref:`botdetection ip_lists`
- :ref:`botdetection rate limit`
- :ref:`botdetection probe headers`

.. kernel-include:: $SOURCEDIR/limiter.toml
   :code: toml

Implementation
==============

Function Documentation

◆ filter_request()

werkzeug.Response | None searx.limiter.filter_request ( SXNG_Request request)

Definition at line 147 of file limiter.py.

147def filter_request(request: SXNG_Request) -> werkzeug.Response | None:
148 # pylint: disable=too-many-return-statements
149
150 cfg = get_cfg()
151 real_ip = ip_address(request.remote_addr)
152 network = get_network(real_ip, cfg)
153
154 if request.path == '/healthz':
155 return None
156
157 # link-local
158
159 if network.is_link_local:
160 return None
161
162 # block- & pass- lists
163 #
164 # 1. The IP of the request is first checked against the pass-list; if the IP
165 # matches an entry in the list, the request is not blocked.
166 # 2. If no matching entry is found in the pass-list, then a check is made against
167 # the block list; if the IP matches an entry in the list, the request is
168 # blocked.
169 # 3. If the IP is not in either list, the request is not blocked.
170
171 match, msg = ip_lists.pass_ip(real_ip, cfg)
172 if match:
173 logger.warning("PASS %s: matched PASSLIST - %s", network.compressed, msg)
174 return None
175
176 match, msg = ip_lists.block_ip(real_ip, cfg)
177 if match:
178 logger.error("BLOCK %s: matched BLOCKLIST - %s", network.compressed, msg)
179 return flask.make_response(('IP is on BLOCKLIST - %s' % msg, 429))
180
181 # methods applied on all requests
182
183 for func in [
184 http_user_agent,
185 ]:
186 val = func.filter_request(network, request, cfg)
187 if val is not None:
188 logger.debug(f"NOT OK ({func.__name__}): {network}: %s", dump_request(sxng_request))
189 return val
190
191 # methods applied on /search requests
192
193 if request.path == '/search':
194
195 for func in [
196 http_accept,
197 http_accept_encoding,
198 http_accept_language,
199 http_user_agent,
200 http_sec_fetch,
201 ip_limit,
202 ]:
203 val = func.filter_request(network, request, cfg)
204 if val is not None:
205 logger.debug(f"NOT OK ({func.__name__}): {network}: %s", dump_request(sxng_request))
206 return val
207
208 logger.debug(f"OK {network}: %s", dump_request(sxng_request))
209 return None
210
211

References get_cfg().

Referenced by pre_request().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ get_cfg()

config.Config searx.limiter.get_cfg ( )
Returns SearXNG's global limiter configuration.

Definition at line 133 of file limiter.py.

133def get_cfg() -> config.Config:
134 """Returns SearXNG's global limiter configuration."""
135 global CFG # pylint: disable=global-statement
136
137 if CFG is None:
138 from . import settings_loader # pylint: disable=import-outside-toplevel
139
140 cfg_file = (settings_loader.get_user_cfg_folder() or Path("/etc/searxng")) / "limiter.toml"
141 CFG = config.Config.from_toml(LIMITER_CFG_SCHEMA, cfg_file, searx.compat.LIMITER_CFG_DEPRECATED)
142 searx.compat.limiter_fix_cfg(CFG, cfg_file)
143
144 return CFG
145
146
limiter_fix_cfg(cfg, cfg_file)
Definition compat.py:34

References searx.compat.limiter_fix_cfg().

Referenced by filter_request().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ initialize()

searx.limiter.initialize ( flask.Flask app,
settings )
Install the limiter

Definition at line 222 of file limiter.py.

222def initialize(app: flask.Flask, settings):
223 """Install the limiter"""
224 global _INSTALLED # pylint: disable=global-statement
225
226 # even if the limiter is not activated, the botdetection must be activated
227 # (e.g. the self_info plugin uses the botdetection to get client IP)
228
229 cfg = get_cfg()
230 valkey_client = valkeydb.client()
231 botdetection.init(cfg, valkey_client)
232
233 if not (settings['server']['limiter'] or settings['server']['public_instance']):
234 return
235
236 if not valkey_client:
237 logger.error(
238 "The limiter requires Valkey, please consult the documentation: "
239 "https://docs.searxng.org/admin/searx.limiter.html"
240 )
241 if settings['server']['public_instance']:
242 sys.exit(1)
243 return
244
245 _INSTALLED = True
246
247 if settings['server']['public_instance']:
248 # overwrite limiter.toml setting
249 cfg.set('botdetection.ip_limit.link_token', True)
250
251 app.before_request(pre_request)

◆ is_installed()

searx.limiter.is_installed ( )
Returns ``True`` if limiter is active and a valkey DB is available.

Definition at line 217 of file limiter.py.

217def is_installed():
218 """Returns ``True`` if limiter is active and a valkey DB is available."""
219 return _INSTALLED
220
221

◆ pre_request()

searx.limiter.pre_request ( )
See :py:obj:`flask.Flask.before_request`

Definition at line 212 of file limiter.py.

212def pre_request():
213 """See :py:obj:`flask.Flask.before_request`"""
214 return filter_request(sxng_request)
215
216

References filter_request().

Here is the call graph for this function:

Variable Documentation

◆ _INSTALLED

bool searx.limiter._INSTALLED = False
protected

Definition at line 127 of file limiter.py.

◆ CFG

config searx.limiter.CFG = None

Definition at line 126 of file limiter.py.

◆ LIMITER_CFG_SCHEMA

str searx.limiter.LIMITER_CFG_SCHEMA = Path(__file__).parent / "limiter.toml"

Definition at line 129 of file limiter.py.

◆ logger

searx.limiter.logger = logger.getChild('limiter')

Definition at line 124 of file limiter.py.