Functions
	regexp_user_agent ()
werkzeug.Response\|None	filter_request (IPv4Network\|IPv6Network network, flask.Request request, config.Config cfg)

Variables
tuple	USER_AGENT
	_regexp = None

Detailed Description

Method ``http_user_agent``
--------------------------

The ``http_user_agent`` method evaluates a request as the request of a bot if
the User-Agent_ header is unset or matches the regular expression
:py:obj:`USER_AGENT`.

.. _User-Agent:
   https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent

Function Documentation

◆ filter_request()

werkzeug.Response \| None searx.botdetection.http_user_agent.filter_request	(	IPv4Network \| IPv6Network	network,
		flask.Request	request,
		config.Config	cfg )

Definition at line 56 of file http_user_agent.py.

) -> werkzeug.Response | None:
 
    user_agent = request.headers.get('User-Agent', 'unknown')
    if regexp_user_agent().match(user_agent):
        return too_many_requests(network, f"bot detected, HTTP header User-Agent: {user_agent}")
    return None

References regexp_user_agent().

Here is the call graph for this function:

◆ regexp_user_agent()

searx.botdetection.http_user_agent.regexp_user_agent ( )

Definition at line 49 of file http_user_agent.py.

def regexp_user_agent():
    global _regexp  # pylint: disable=global-statement
    if not _regexp:
        _regexp = re.compile(USER_AGENT)
    return _regexp
 
 

Referenced by filter_request().

Here is the caller graph for this function:

Variable Documentation

◆ _regexp

searx.botdetection.http_user_agent._regexp = None

protected

Definition at line 46 of file http_user_agent.py.

◆ USER_AGENT

tuple searx.botdetection.http_user_agent.USER_AGENT

Initial value:

=  (
    r'('
    + r'unknown'
    + r'|[Cc][Uu][Rr][Ll]|[wW]get|Scrapy|splash|JavaFX|FeedFetcher|python-requests|Go-http-client|Java|Jakarta|okhttp'
    + r'|HttpClient|Jersey|Python|libwww-perl|Ruby|SynHttpClient|UniversalFeedParser|Googlebot|GoogleImageProxy'
    + r'|bingbot|Baiduspider|yacybot|YandexMobileBot|YandexBot|Yahoo! Slurp|MJ12bot|AhrefsBot|archive.org_bot|msnbot'
    + r'|MJ12bot|SeznamBot|linkdexbot|Netvibes|SMTBot|zgrab|James BOT|Sogou|Abonti|Pixray|Spinn3r|SemrushBot|Exabot'
    + r'|ZmEu|BLEXBot|bitlybot|HeadlessChrome'
    # unmaintained Farside instances
    + r'|'
    + re.escape(r'Mozilla/5.0 (compatible; Farside/0.1.0; +https://farside.link)')
    # other bots and client to block
    + '|.*PetalBot.*'
    + r')'
)

Definition at line 29 of file http_user_agent.py.

Functions