Inheritance diagram for searx.search.processors.online.OnlineProcessor:

Collaboration diagram for searx.search.processors.online.OnlineProcessor:

Public Member Functions
bool	init_engine (self)
	init_network_in_thread (self, float start_time, float timeout_limit)
OnlineParams\|None	get_params (self, "SearchQuery" search_query, str engine_category)
	search (self, str query, OnlineParams params, "ResultContainer" result_container, float start_time, float timeout_limit)
Public Member Functions inherited from searx.search.processors.abstract.EngineProcessor
	__init__ (self, "Engine\|types.ModuleType" engine)
	initialize (self, t.Callable[["EngineProcessor", bool], bool] callback)
	handle_exception (self, "ResultContainer" result_container, BaseException\|str exception_or_message, bool suspend=False)
	extend_container (self, "ResultContainer" result_container, float start_time, "list[Result \| LegacyResult]\|None" search_results)
bool	extend_container_if_suspended (self, "ResultContainer" result_container)
	get_tests (self)
	get_default_tests (self)

Static Public Attributes
str	engine_type = "online"

Protected Member Functions
	_send_http_request (self, OnlineParams params)
"EngineResults\|None"	_search_basic (self, str query, OnlineParams params)
Protected Member Functions inherited from searx.search.processors.abstract.EngineProcessor
	_extend_container_basic (self, "ResultContainer" result_container, float start_time, "list[Result \| LegacyResult]" search_results)

Additional Inherited Members
Public Attributes inherited from searx.search.processors.abstract.EngineProcessor
	engine = engine
logging.Logger	logger = engines[engine.name].logger
SuspendedStatus	suspended_status = SUSPENDED_STATUS.setdefault(key, SuspendedStatus())

Detailed Description

Processor class for ``online`` engines.

Definition at line 113 of file online.py.

Member Function Documentation

◆ _search_basic()

"EngineResults\|None" searx.search.processors.online.OnlineProcessor._search_basic	(		self,
		str	query,
		OnlineParams	params )

protected

Definition at line 221 of file online.py.

    def _search_basic(self, query: str, params: OnlineParams) -> "EngineResults|None":
        # update request parameters dependent on
        # search-engine (contained in engines folder)
        self.engine.request(query, params)
 
        # ignoring empty urls
        if not params["url"]:
            return None
 
        # send request
        response = self._send_http_request(params)
 
        # parse the response
        response.search_params = params
        return self.engine.response(response)
 

References _send_http_request(), searx.result_types._base.LegacyResult.engine, searx.result_types._base.MainResult.engine, searx.result_types._base.Result.engine, and searx.search.processors.abstract.EngineProcessor.engine.

Referenced by search().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ _send_http_request()

searx.search.processors.online.OnlineProcessor._send_http_request	(		self,
		OnlineParams	params )

protected

Definition at line 162 of file online.py.

    def _send_http_request(self, params: OnlineParams):
 
        # create dictionary which contain all information about the request
        request_args: dict[str, t.Any] = {
            "headers": params["headers"],
            "cookies": params["cookies"],
            "auth": params["auth"],
        }
 
        verify = params.get("verify")
        if verify is not None:
            request_args["verify"] = verify
 
        # max_redirects
        max_redirects = params.get("max_redirects")
        if max_redirects:
            request_args["max_redirects"] = max_redirects
 
        # allow_redirects
        if "allow_redirects" in params:
            request_args["allow_redirects"] = params["allow_redirects"]
 
        # soft_max_redirects
        soft_max_redirects: int = params.get("soft_max_redirects", max_redirects or 0)
 
        # raise_for_status
        request_args["raise_for_httperror"] = params.get("raise_for_httperror", True)
 
        # specific type of request (GET or POST)
        if params["method"] == "GET":
            req = searx.network.get
        else:
            req = searx.network.post
            if params["data"]:
                request_args["data"] = params["data"]
            if params["json"]:
                request_args["json"] = params["json"]
            if params["content"]:
                request_args["content"] = params["content"]
 
        # send the request
        response = req(params["url"], **request_args)
 
        # check soft limit of the redirect count
        if len(response.history) > soft_max_redirects:
            # unexpected redirect : record an error
            # but the engine might still return valid results.
            status_code = str(response.status_code or "")
            reason = response.reason_phrase or ""
            hostname = response.url.host
            count_error(
                self.engine.name,
                "{} redirects, maximum: {}".format(len(response.history), soft_max_redirects),
                (status_code, reason, hostname),
                secondary=True,
            )
 
        return response
 

References searx.result_types._base.LegacyResult.engine, searx.result_types._base.MainResult.engine, searx.result_types._base.Result.engine, and searx.search.processors.abstract.EngineProcessor.engine.

Referenced by _search_basic().

Here is the caller graph for this function:

◆ get_params()

OnlineParams \| None searx.search.processors.online.OnlineProcessor.get_params	(		self,
		"SearchQuery"	search_query,
		str	engine_category )

Returns a dictionary with the :ref:`request params <engine request
online>` (:py:obj:`OnlineParams`), if the search condition is not
supported by the engine, ``None`` is returned.

Reimplemented from searx.search.processors.abstract.EngineProcessor.

Reimplemented in searx.search.processors.online_currency.OnlineCurrencyProcessor, searx.search.processors.online_dictionary.OnlineDictionaryProcessor, and searx.search.processors.online_url_search.OnlineUrlSearchProcessor.

Definition at line 132 of file online.py.

    def get_params(self, search_query: "SearchQuery", engine_category: str) -> OnlineParams | None:
        """Returns a dictionary with the :ref:`request params <engine request
        online>` (:py:obj:`OnlineParams`), if the search condition is not
        supported by the engine, ``None`` is returned."""
 
        base_params: RequestParams | None = super().get_params(search_query, engine_category)
        if base_params is None:
            return base_params
 
        params: OnlineParams = {**default_request_params(), **base_params}
 
        headers = params["headers"]
 
        # add an user agent
        headers["User-Agent"] = gen_useragent()
 
        # add Accept-Language header
        if self.engine.send_accept_language_header and search_query.locale:
            ac_lang = search_query.locale.language
            if search_query.locale.territory:
                ac_lang = "%s-%s,%s;q=0.9,*;q=0.5" % (
                    search_query.locale.language,
                    search_query.locale.territory,
                    search_query.locale.language,
                )
            headers["Accept-Language"] = ac_lang
 
        self.logger.debug("HTTP Accept-Language: %s", headers.get("Accept-Language", ""))
        return params
 

References searx.search.processors.online.default_request_params(), searx.result_types._base.LegacyResult.engine, searx.result_types._base.MainResult.engine, searx.result_types._base.Result.engine, searx.search.processors.abstract.EngineProcessor.engine, get_params(), searx.enginelib.Engine.logger, and searx.search.processors.abstract.EngineProcessor.logger.

Referenced by get_params().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ init_engine()

bool searx.search.processors.online.OnlineProcessor.init_engine ( self )

This method is called in a thread, and before the base method is
called, the network must be set up for the ``online`` engines.

Reimplemented from searx.search.processors.abstract.EngineProcessor.

Definition at line 118 of file online.py.

    def init_engine(self) -> bool:
        """This method is called in a thread, and before the base method is
        called, the network must be set up for the ``online`` engines."""
        self.init_network_in_thread(start_time=default_timer(), timeout_limit=self.engine.timeout)
        return super().init_engine()
 

References searx.result_types._base.LegacyResult.engine, searx.result_types._base.MainResult.engine, searx.result_types._base.Result.engine, searx.search.processors.abstract.EngineProcessor.engine, init_engine(), and init_network_in_thread().

Referenced by init_engine().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ init_network_in_thread()

searx.search.processors.online.OnlineProcessor.init_network_in_thread	(		self,
		float	start_time,
		float	timeout_limit )

Definition at line 124 of file online.py.

    def init_network_in_thread(self, start_time: float, timeout_limit: float):
        # set timeout for all HTTP requests
        searx.network.set_timeout_for_thread(timeout_limit, start_time=start_time)
        # reset the HTTP total time
        searx.network.reset_time_for_thread()
        # set the network
        searx.network.set_context_network_name(self.engine.name)
 

References searx.result_types._base.LegacyResult.engine, searx.result_types._base.MainResult.engine, searx.result_types._base.Result.engine, searx.search.processors.abstract.EngineProcessor.engine, searx.network.reset_time_for_thread(), searx.network.set_context_network_name(), and searx.network.set_timeout_for_thread().

Referenced by init_engine(), and search().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ search()

searx.search.processors.online.OnlineProcessor.search	(		self,
		str	query,
		OnlineParams	params,
		"ResultContainer"	result_container,
		float	start_time,
		float	timeout_limit )

Reimplemented from searx.search.processors.abstract.EngineProcessor.

Definition at line 237 of file online.py.

    ):
        self.init_network_in_thread(start_time, timeout_limit)
 
        try:
            # send requests and parse the results
            search_results = self._search_basic(query, params)
            self.extend_container(result_container, start_time, search_results)
        except ssl.SSLError as e:
            # requests timeout (connect or read)
            self.handle_exception(result_container, e, suspend=True)
            self.logger.error("SSLError {}, verify={}".format(e, searx.network.get_network(self.engine.name).verify))
        except (httpx.TimeoutException, asyncio.TimeoutError) as e:
            # requests timeout (connect or read)
            self.handle_exception(result_container, e, suspend=True)
            self.logger.error(
                "HTTP requests timeout (search duration : {0} s, timeout: {1} s) : {2}".format(
                    default_timer() - start_time, timeout_limit, e.__class__.__name__
                )
            )
        except (httpx.HTTPError, httpx.StreamError) as e:
            # other requests exception
            self.handle_exception(result_container, e, suspend=True)
            self.logger.exception(
                "requests exception (search duration : {0} s, timeout: {1} s) : {2}".format(
                    default_timer() - start_time, timeout_limit, e
                )
            )
        except (
            SearxEngineCaptchaException,
            SearxEngineTooManyRequestsException,
            SearxEngineAccessDeniedException,
        ) as e:
            self.handle_exception(result_container, e, suspend=True)
            self.logger.exception(e.message)
        except Exception as e:  # pylint: disable=broad-except
            self.handle_exception(result_container, e)
            self.logger.exception("exception : {0}".format(e))

References _search_basic(), searx.result_types._base.LegacyResult.engine, searx.result_types._base.MainResult.engine, searx.result_types._base.Result.engine, searx.search.processors.abstract.EngineProcessor.engine, searx.search.processors.abstract.EngineProcessor.extend_container(), searx.search.processors.abstract.EngineProcessor.handle_exception(), init_network_in_thread(), searx.enginelib.Engine.logger, and searx.search.processors.abstract.EngineProcessor.logger.

Here is the call graph for this function:

Member Data Documentation

◆ engine_type

str searx.search.processors.online.OnlineProcessor.engine_type = "online"

static

Definition at line 116 of file online.py.

The documentation for this class was generated from the following file:

/home/andrew/Documents/code/public/searxng/searx/search/processors/online.py

Public Member Functions

Static Public Attributes

Protected Member Functions

Additional Inherited Members

Detailed Description

Member Function Documentation

◆ _search_basic()

◆ _send_http_request()

◆ get_params()

◆ init_engine()

◆ init_network_in_thread()

◆ search()

Member Data Documentation

◆ engine_type