.oO SearXNG Developer Documentation Oo.
Loading...
Searching...
No Matches
searx.botdetection.link_token Namespace Reference

Functions

 is_suspicious (IPv4Network|IPv6Network network, flask.Request request, bool renew=False)
 ping (flask.Request request, str token)
str get_ping_key (IPv4Network|IPv6Network network, flask.Request request)
bool token_is_valid (token)
str get_token ()

Variables

int TOKEN_LIVE_TIME = 600
int PING_LIVE_TIME = 3600
str PING_KEY = 'SearXNG_limiter.ping'
str TOKEN_KEY = 'SearXNG_limiter.token'
 logger = logger.getChild('botdetection.link_token')

Detailed Description

Method ``link_token``
---------------------

The ``link_token`` method evaluates a request as :py:obj:`suspicious
<is_suspicious>` if the URL ``/client<token>.css`` is not requested by the
client.  By adding a random component (the token) in the URL, a bot can not send
a ping by request a static URL.

.. note::

   This method requires a valkey DB and needs a HTTP X-Forwarded-For_ header.

To get in use of this method a flask URL route needs to be added:

.. code:: python

   @app.route('/client<token>.css', methods=['GET', 'POST'])
   def client_token(token=None):
       link_token.ping(request, token)
       return Response('', mimetype='text/css')

And in the HTML template from flask a stylesheet link is needed (the value of
``link_token`` comes from :py:obj:`get_token`):

.. code:: html

   <link rel="stylesheet"
         href="{{ url_for('client_token', token=link_token) }}"
         type="text/css" >

.. _X-Forwarded-For:
   https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For

Function Documentation

◆ get_ping_key()

str searx.botdetection.link_token.get_ping_key ( IPv4Network | IPv6Network network,
flask.Request request ) 
Generates a hashed key that fits (more or less) to a *WEB-browser
session* in a network.

Definition at line 115 of file link_token.py.

115def get_ping_key(network: IPv4Network | IPv6Network, request: flask.Request) -> str:
116 """Generates a hashed key that fits (more or less) to a *WEB-browser
117 session* in a network."""
118 return (
119 PING_KEY
120 + "["
121 + secret_hash(
122 network.compressed + request.headers.get('Accept-Language', '') + request.headers.get('User-Agent', '')
123 )
124 + "]"
125 )
126
127

Referenced by is_suspicious(), and ping().

Here is the caller graph for this function:

◆ get_token()

str searx.botdetection.link_token.get_token ( ) 
Returns current token.  If there is no currently active token a new token
is generated randomly and stored in the Valkey DB.  Without without a
database connection, string "12345678" is returned.

- :py:obj:`TOKEN_LIVE_TIME`
- :py:obj:`TOKEN_KEY`

Definition at line 134 of file link_token.py.

134def get_token() -> str:
135 """Returns current token. If there is no currently active token a new token
136 is generated randomly and stored in the Valkey DB. Without without a
137 database connection, string "12345678" is returned.
138
139 - :py:obj:`TOKEN_LIVE_TIME`
140 - :py:obj:`TOKEN_KEY`
141
142 """
143 try:
144 valkey_client = valkeydb.get_valkey_client()
145 except ValueError:
146 # This function is also called when limiter is inactive / no valkey DB
147 # (see render function in webapp.py)
148 return '12345678'
149
150 token = valkey_client.get(TOKEN_KEY)
151 if token:
152 token = token.decode('UTF-8') # type: ignore
153 else:
154 token = ''.join(random.choice(string.ascii_lowercase + string.digits) for _ in range(16))
155 valkey_client.set(TOKEN_KEY, token, ex=TOKEN_LIVE_TIME)
156 return token

Referenced by token_is_valid().

Here is the caller graph for this function:

◆ is_suspicious()

searx.botdetection.link_token.is_suspicious ( IPv4Network | IPv6Network network,
flask.Request request,
bool renew = False ) 
Checks whether a valid ping is exists for this (client) network, if not
this request is rated as *suspicious*.  If a valid ping exists and argument
``renew`` is ``True`` the expire time of this ping is reset to
:py:obj:`PING_LIVE_TIME`.

Definition at line 73 of file link_token.py.

73def is_suspicious(network: IPv4Network | IPv6Network, request: flask.Request, renew: bool = False):
74 """Checks whether a valid ping is exists for this (client) network, if not
75 this request is rated as *suspicious*. If a valid ping exists and argument
76 ``renew`` is ``True`` the expire time of this ping is reset to
77 :py:obj:`PING_LIVE_TIME`.
78
79 """
80 valkey_client = valkeydb.get_valkey_client()
81 ping_key = get_ping_key(network, request)
82 if not valkey_client.get(ping_key):
83 logger.info("missing ping (IP: %s) / request: %s", network.compressed, ping_key)
84 return True
85
86 if renew:
87 valkey_client.set(ping_key, 1, ex=PING_LIVE_TIME)
88
89 logger.debug("found ping for (client) network %s -> %s", network.compressed, ping_key)
90 return False
91
92

References get_ping_key().

Here is the call graph for this function:

◆ ping()

searx.botdetection.link_token.ping ( flask.Request request,
str token ) 
This function is called by a request to URL ``/client<token>.css``.  If
``token`` is valid a :py:obj:`PING_KEY` for the client is stored in the DB.
The expire time of this ping-key is :py:obj:`PING_LIVE_TIME`.

Definition at line 93 of file link_token.py.

93def ping(request: flask.Request, token: str):
94 """This function is called by a request to URL ``/client<token>.css``. If
95 ``token`` is valid a :py:obj:`PING_KEY` for the client is stored in the DB.
96 The expire time of this ping-key is :py:obj:`PING_LIVE_TIME`.
97
98 """
99 valkey_client = valkeydb.get_valkey_client()
100 cfg = config.get_global_cfg()
101
102 if not token_is_valid(token):
103 return
104
105 real_ip = ip_address(request.remote_addr) # type: ignore
106 network = get_network(real_ip, cfg)
107
108 ping_key = get_ping_key(network, request)
109 logger.debug(
110 "store ping_key for (client) network %s (IP %s) -> %s", network.compressed, real_ip.compressed, ping_key
111 )
112 valkey_client.set(ping_key, 1, ex=PING_LIVE_TIME)
113
114

References get_ping_key(), and token_is_valid().

Here is the call graph for this function:

◆ token_is_valid()

bool searx.botdetection.link_token.token_is_valid ( token)

Definition at line 128 of file link_token.py.

128def token_is_valid(token) -> bool:
129 valid = token == get_token()
130 logger.debug("token is valid --> %s", valid)
131 return valid
132
133

References get_token().

Referenced by ping().

Here is the call graph for this function:
Here is the caller graph for this function:

Variable Documentation

◆ logger

searx.botdetection.link_token.logger = logger.getChild('botdetection.link_token')

Definition at line 70 of file link_token.py.

◆ PING_KEY

str searx.botdetection.link_token.PING_KEY = 'SearXNG_limiter.ping'

Definition at line 64 of file link_token.py.

◆ PING_LIVE_TIME

int searx.botdetection.link_token.PING_LIVE_TIME = 3600

Definition at line 61 of file link_token.py.

◆ TOKEN_KEY

str searx.botdetection.link_token.TOKEN_KEY = 'SearXNG_limiter.token'

Definition at line 67 of file link_token.py.

◆ TOKEN_LIVE_TIME

int searx.botdetection.link_token.TOKEN_LIVE_TIME = 600

Definition at line 58 of file link_token.py.