Functions
werkzeug.Response\|None	filter_request (IPv4Network\|IPv6Network network, SXNG_Request request, config.Config cfg)

Variables
	logger = logger.getChild('ip_limit')

int	BURST_WINDOW = 20

int	BURST_MAX = 15

int	BURST_MAX_SUSPICIOUS = 2

int	LONG_WINDOW = 600

int	LONG_MAX = 150

int	LONG_MAX_SUSPICIOUS = 10

int	API_WINDOW = 3600

int	API_MAX = 4

int	SUSPICIOUS_IP_WINDOW = 3600 * 24 * 30

int	SUSPICIOUS_IP_MAX = 3

Detailed Description

.. _botdetection.ip_limit:

Method ``ip_limit``
-------------------

The ``ip_limit`` method counts request from an IP in *sliding windows*.  If
there are to many requests in a sliding window, the request is evaluated as a
bot request.  This method requires a redis DB and needs a HTTP X-Forwarded-For_
header.  To take privacy only the hash value of an IP is stored in the redis DB
and at least for a maximum of 10 minutes.

The :py:obj:`.link_token` method can be used to investigate whether a request is
*suspicious*.  To activate the :py:obj:`.link_token` method in the
:py:obj:`.ip_limit` method add the following configuration:

.. code:: toml

   [botdetection.ip_limit]
   link_token = true

If the :py:obj:`.link_token` method is activated and a request is *suspicious*
the request rates are reduced:

- :py:obj:`BURST_MAX` -> :py:obj:`BURST_MAX_SUSPICIOUS`
- :py:obj:`LONG_MAX` -> :py:obj:`LONG_MAX_SUSPICIOUS`

To intercept bots that get their IPs from a range of IPs, there is a
:py:obj:`SUSPICIOUS_IP_WINDOW`.  In this window the suspicious IPs are stored
for a longer time.  IPs stored in this sliding window have a maximum of
:py:obj:`SUSPICIOUS_IP_MAX` accesses before they are blocked.  As soon as the IP
makes a request that is not suspicious, the sliding window for this IP is
dropped.

.. _X-Forwarded-For:
   https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For

Function Documentation

◆ filter_request()

werkzeug.Response \| None searx.botdetection.ip_limit.filter_request	(	IPv4Network \| IPv6Network	network,
		SXNG_Request	request,
		config.Config	cfg )

Definition at line 93 of file ip_limit.py.

) -> werkzeug.Response | None:
 
    # pylint: disable=too-many-return-statements
    redis_client = redisdb.client()
 
    if network.is_link_local and not cfg['botdetection.ip_limit.filter_link_local']:
        logger.debug("network %s is link-local -> not monitored by ip_limit method", network.compressed)
        return None
 
    if request.args.get('format', 'html') != 'html':
        c = incr_sliding_window(redis_client, 'ip_limit.API_WINDOW:' + network.compressed, API_WINDOW)
        if c > API_MAX:
            return too_many_requests(network, "too many request in API_WINDOW")
 
    if cfg['botdetection.ip_limit.link_token']:
 
        suspicious = link_token.is_suspicious(network, request, True)
 
        if not suspicious:
            # this IP is no longer suspicious: release ip again / delete the counter of this IP
            drop_counter(redis_client, 'ip_limit.SUSPICIOUS_IP_WINDOW' + network.compressed)
            return None
 
        # this IP is suspicious: count requests from this IP
        c = incr_sliding_window(
            redis_client, 'ip_limit.SUSPICIOUS_IP_WINDOW' + network.compressed, SUSPICIOUS_IP_WINDOW
        )
        if c > SUSPICIOUS_IP_MAX:
            logger.error("BLOCK: too many request from %s in SUSPICIOUS_IP_WINDOW (redirect to /)", network)
            response = flask.redirect(flask.url_for('index'), code=302)
            response.headers["Cache-Control"] = "no-store, max-age=0"
            return response
 
        c = incr_sliding_window(redis_client, 'ip_limit.BURST_WINDOW' + network.compressed, BURST_WINDOW)
        if c > BURST_MAX_SUSPICIOUS:
            return too_many_requests(network, "too many request in BURST_WINDOW (BURST_MAX_SUSPICIOUS)")
 
        c = incr_sliding_window(redis_client, 'ip_limit.LONG_WINDOW' + network.compressed, LONG_WINDOW)
        if c > LONG_MAX_SUSPICIOUS:
            return too_many_requests(network, "too many request in LONG_WINDOW (LONG_MAX_SUSPICIOUS)")
 
        return None
 
    # vanilla limiter without extensions counts BURST_MAX and LONG_MAX
    c = incr_sliding_window(redis_client, 'ip_limit.BURST_WINDOW' + network.compressed, BURST_WINDOW)
    if c > BURST_MAX:
        return too_many_requests(network, "too many request in BURST_WINDOW (BURST_MAX)")
 
    c = incr_sliding_window(redis_client, 'ip_limit.LONG_WINDOW' + network.compressed, LONG_WINDOW)
    if c > LONG_MAX:
        return too_many_requests(network, "too many request in LONG_WINDOW (LONG_MAX)")
 
    return None

Variable Documentation

◆ API_MAX

int searx.botdetection.ip_limit.API_MAX = 4

Definition at line 83 of file ip_limit.py.

◆ API_WINDOW

int searx.botdetection.ip_limit.API_WINDOW = 3600

Definition at line 80 of file ip_limit.py.

◆ BURST_MAX

int searx.botdetection.ip_limit.BURST_MAX = 15

Definition at line 65 of file ip_limit.py.

◆ BURST_MAX_SUSPICIOUS

int searx.botdetection.ip_limit.BURST_MAX_SUSPICIOUS = 2

Definition at line 68 of file ip_limit.py.

◆ BURST_WINDOW

int searx.botdetection.ip_limit.BURST_WINDOW = 20

Definition at line 62 of file ip_limit.py.

◆ logger

searx.botdetection.ip_limit.logger = logger.getChild('ip_limit')

Definition at line 60 of file ip_limit.py.

◆ LONG_MAX

int searx.botdetection.ip_limit.LONG_MAX = 150

Definition at line 74 of file ip_limit.py.

◆ LONG_MAX_SUSPICIOUS

int searx.botdetection.ip_limit.LONG_MAX_SUSPICIOUS = 10

Definition at line 77 of file ip_limit.py.

◆ LONG_WINDOW

int searx.botdetection.ip_limit.LONG_WINDOW = 600

Definition at line 71 of file ip_limit.py.

◆ SUSPICIOUS_IP_MAX

int searx.botdetection.ip_limit.SUSPICIOUS_IP_MAX = 3

Definition at line 89 of file ip_limit.py.

◆ SUSPICIOUS_IP_WINDOW

int searx.botdetection.ip_limit.SUSPICIOUS_IP_WINDOW = 3600 * 24 * 30

Definition at line 86 of file ip_limit.py.

Functions

Variables