Inheritance diagram for searx.search.processors.abstract.EngineProcessor:

Collaboration diagram for searx.search.processors.abstract.EngineProcessor:

Public Member Functions
	__init__ (self, "Engine\|types.ModuleType" engine)
	initialize (self, t.Callable[["EngineProcessor", bool], bool] callback)
bool	init_engine (self)
	handle_exception (self, "ResultContainer" result_container, BaseException\|str exception_or_message, bool suspend=False)
	extend_container (self, "ResultContainer" result_container, float start_time, "list[Result \| LegacyResult]\|None" search_results)
bool	extend_container_if_suspended (self, "ResultContainer" result_container)
RequestParams\|None	get_params (self, "SearchQuery" search_query, str engine_category)
	search (self, str query, RequestParams params, "ResultContainer" result_container, float start_time, float timeout_limit)
	get_tests (self)
	get_default_tests (self)

Public Attributes
	engine = engine
logging.Logger	logger = engines[engine.name].logger
SuspendedStatus	suspended_status = SUSPENDED_STATUS.setdefault(key, SuspendedStatus())

Protected Member Functions
	_extend_container_basic (self, "ResultContainer" result_container, float start_time, "list[Result \| LegacyResult]" search_results)

Detailed Description

Base classes used for all types of request processors.

Definition at line 111 of file abstract.py.

Constructor & Destructor Documentation

◆ init()

searx.search.processors.abstract.EngineProcessor.__init__	(		self,
		"Engine\|types.ModuleType"	engine )

Definition at line 116 of file abstract.py.

    def __init__(self, engine: "Engine|types.ModuleType"):
        self.engine: "Engine" = engine  # pyright: ignore[reportAttributeAccessIssue]
        self.logger: logging.Logger = engines[engine.name].logger
        key = get_network(self.engine.name)
        key = id(key) if key else self.engine.name
        self.suspended_status: SuspendedStatus = SUSPENDED_STATUS.setdefault(key, SuspendedStatus())
 

Member Function Documentation

◆ _extend_container_basic()

searx.search.processors.abstract.EngineProcessor._extend_container_basic	(		self,
		"ResultContainer"	result_container,
		float	start_time,
		"list[Result \| LegacyResult]"	search_results )

protected

Definition at line 193 of file abstract.py.

    ):
        # update result_container
        result_container.extend(self.engine.name, search_results)
        engine_time = default_timer() - start_time
        page_load_time = get_time_for_thread()
        result_container.add_timing(self.engine.name, engine_time, page_load_time)
        # metrics
        counter_inc('engine', self.engine.name, 'search', 'count', 'successful')
        histogram_observe(engine_time, 'engine', self.engine.name, 'time', 'total')
        if page_load_time is not None:
            histogram_observe(page_load_time, 'engine', self.engine.name, 'time', 'http')
 

References searx.result_types._base.LegacyResult.engine, searx.result_types._base.MainResult.engine, searx.result_types._base.Result.engine, and engine.

Referenced by extend_container().

Here is the caller graph for this function:

◆ extend_container()

searx.search.processors.abstract.EngineProcessor.extend_container	(		self,
		"ResultContainer"	result_container,
		float	start_time,
		"list[Result \| LegacyResult]\|None"	search_results )

Definition at line 210 of file abstract.py.

    ):
        if getattr(threading.current_thread(), '_timeout', False):
            # the main thread is not waiting anymore
            self.handle_exception(result_container, 'timeout', False)
        else:
            # check if the engine accepted the request
            if search_results is not None:
                self._extend_container_basic(result_container, start_time, search_results)
            self.suspended_status.resume()
 

References _extend_container_basic(), handle_exception(), and suspended_status.

Referenced by searx.search.processors.offline.OfflineProcessor.search(), and searx.search.processors.online.OnlineProcessor.search().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ extend_container_if_suspended()

bool searx.search.processors.abstract.EngineProcessor.extend_container_if_suspended	(		self,
		"ResultContainer"	result_container )

Definition at line 225 of file abstract.py.

    def extend_container_if_suspended(self, result_container: "ResultContainer") -> bool:
        if self.suspended_status.is_suspended:
            result_container.add_unresponsive_engine(
                self.engine.name, self.suspended_status.suspend_reason, suspended=True
            )
            return True
        return False
 

References searx.result_types._base.LegacyResult.engine, searx.result_types._base.MainResult.engine, searx.result_types._base.Result.engine, engine, and suspended_status.

◆ get_default_tests()

searx.search.processors.abstract.EngineProcessor.get_default_tests ( self )

Definition at line 296 of file abstract.py.

    def get_default_tests(self):
        # deprecated!
        return {}

◆ get_params()

RequestParams \| None searx.search.processors.abstract.EngineProcessor.get_params	(		self,
		"SearchQuery"	search_query,
		str	engine_category )

Returns a dictionary with the :ref:`request parameters <engine
request arguments>` (:py:obj:`RequestParams`), if the search condition
is not supported by the engine, ``None`` is returned:

- *time range* filter in search conditions, but the engine does not have
   a corresponding filter
- page number > 1 when engine does not support paging
- page number > ``max_page``

Reimplemented in searx.search.processors.online.OnlineProcessor, searx.search.processors.online_currency.OnlineCurrencyProcessor, searx.search.processors.online_dictionary.OnlineDictionaryProcessor, and searx.search.processors.online_url_search.OnlineUrlSearchProcessor.

Definition at line 233 of file abstract.py.

    def get_params(self, search_query: "SearchQuery", engine_category: str) -> RequestParams | None:
        """Returns a dictionary with the :ref:`request parameters <engine
        request arguments>` (:py:obj:`RequestParams`), if the search condition
        is not supported by the engine, ``None`` is returned:
 
        - *time range* filter in search conditions, but the engine does not have
           a corresponding filter
        - page number > 1 when engine does not support paging
        - page number > ``max_page``
 
        """
        # if paging is not supported, skip
        if search_query.pageno > 1 and not self.engine.paging:
            return None
 
        # if max page is reached, skip
        max_page = self.engine.max_page or get_setting("search.max_page")
        if max_page and max_page < search_query.pageno:
            return None
 
        # if time_range is not supported, skip
        if search_query.time_range and not self.engine.time_range_support:
            return None
 
        params: RequestParams = {
            "query": search_query.query,
            "category": engine_category,
            "pageno": search_query.pageno,
            "safesearch": search_query.safesearch,
            "time_range": search_query.time_range,
            "engine_data": search_query.engine_data.get(self.engine.name, {}),
            "searxng_locale": search_query.lang,
        }
 
        # deprecated / vintage --> use params["searxng_locale"]
        #
        # Conditions related to engine's traits are implemented in engine.traits
        # module. Don't do "locale" decisions here in the abstract layer of the
        # search processor, just pass the value from user's choice unchanged to
        # the engine request.
 
        if hasattr(self.engine, "language") and self.engine.language:
            params["language"] = self.engine.language  # pyright: ignore[reportGeneralTypeIssues]
        else:
            params["language"] = search_query.lang  # pyright: ignore[reportGeneralTypeIssues]
 
        return params
 

References searx.result_types._base.LegacyResult.engine, searx.result_types._base.MainResult.engine, searx.result_types._base.Result.engine, engine, and searx.get_setting().

Here is the call graph for this function:

◆ get_tests()

searx.search.processors.abstract.EngineProcessor.get_tests ( self )

Definition at line 292 of file abstract.py.

    def get_tests(self):
        # deprecated!
        return {}
 

◆ handle_exception()

searx.search.processors.abstract.EngineProcessor.handle_exception	(		self,
		"ResultContainer"	result_container,
		BaseException \| str	exception_or_message,
		bool	suspend = False )

Definition at line 165 of file abstract.py.

    ):
        # update result_container
        if isinstance(exception_or_message, BaseException):
            exception_class = exception_or_message.__class__
            module_name = getattr(exception_class, '__module__', 'builtins')
            module_name = '' if module_name == 'builtins' else module_name + '.'
            error_message = module_name + exception_class.__qualname__
        else:
            error_message = exception_or_message
        result_container.add_unresponsive_engine(self.engine.name, error_message)
        # metrics
        counter_inc('engine', self.engine.name, 'search', 'count', 'error')
        if isinstance(exception_or_message, BaseException):
            count_exception(self.engine.name, exception_or_message)
        else:
            count_error(self.engine.name, exception_or_message)
        # suspend the engine ?
        if suspend:
            suspended_time = None
            if isinstance(exception_or_message, SearxEngineAccessDeniedException):
                suspended_time = exception_or_message.suspended_time
            self.suspended_status.suspend(suspended_time, error_message)  # pylint: disable=no-member
 

References searx.result_types._base.LegacyResult.engine, searx.result_types._base.MainResult.engine, searx.result_types._base.Result.engine, engine, and suspended_status.

Referenced by extend_container(), searx.search.processors.offline.OfflineProcessor.search(), and searx.search.processors.online.OnlineProcessor.search().

Here is the caller graph for this function:

◆ init_engine()

bool searx.search.processors.abstract.EngineProcessor.init_engine ( self )

Reimplemented in searx.search.processors.online.OnlineProcessor.

Definition at line 151 of file abstract.py.

    def init_engine(self) -> bool:
        eng_setting = get_engine_from_settings(self.engine.name)
        init_ok: bool | None = False
        try:
            init_ok = self.engine.init(eng_setting)
        except Exception:  # pylint: disable=broad-except
            logger.exception("Init method of engine %s failed due to an exception.", self.engine.name)
            init_ok = False
        # In older engines, None is returned from the init method, which is
        # equivalent to indicating that the initialization was successful.
        if init_ok is None:
            init_ok = True
        return init_ok
 

References searx.result_types._base.LegacyResult.engine, searx.result_types._base.MainResult.engine, searx.result_types._base.Result.engine, and engine.

Referenced by initialize().

Here is the caller graph for this function:

◆ initialize()

searx.search.processors.abstract.EngineProcessor.initialize	(		self,
		t.Callable[["EngineProcessor", bool], bool]	callback )

Initialization of *this* :py:obj:`EngineProcessor`.

If processor's engine has an ``init`` method, it is called first.
Engine's ``init`` method is executed in a thread, meaning that the
*registration* (the ``callback``) may occur later and is not already
established by the return from this registration method.

Registration only takes place if the ``init`` method is not available or
is successfully run through.

Reimplemented in searx.search.processors.online_currency.OnlineCurrencyProcessor.

Definition at line 123 of file abstract.py.

    def initialize(self, callback: t.Callable[["EngineProcessor", bool], bool]):
        """Initialization of *this* :py:obj:`EngineProcessor`.
 
        If processor's engine has an ``init`` method, it is called first.
        Engine's ``init`` method is executed in a thread, meaning that the
        *registration* (the ``callback``) may occur later and is not already
        established by the return from this registration method.
 
        Registration only takes place if the ``init`` method is not available or
        is successfully run through.
        """
 
        if not hasattr(self.engine, "init"):
            callback(self, True)
            return
 
        if not callable(self.engine.init):
            logger.error("Engine's init method isn't a callable (is of type: %s).", type(self.engine.init))
            callback(self, False)
            return
 
        def __init_processor_thread():
            eng_ok = self.init_engine()
            callback(self, eng_ok)
 
        # set up and start a thread
        threading.Thread(target=__init_processor_thread, daemon=True).start()
 

References searx.result_types._base.LegacyResult.engine, searx.result_types._base.MainResult.engine, searx.result_types._base.Result.engine, engine, and init_engine().

Here is the call graph for this function:

◆ search()

searx.search.processors.abstract.EngineProcessor.search	(		self,
		str	query,
		RequestParams	params,
		"ResultContainer"	result_container,
		float	start_time,
		float	timeout_limit )

Reimplemented in searx.search.processors.offline.OfflineProcessor, and searx.search.processors.online.OnlineProcessor.

Definition at line 282 of file abstract.py.

    ):
        pass
 

Member Data Documentation

◆ engine

searx.search.processors.abstract.EngineProcessor.engine = engine

Definition at line 117 of file abstract.py.

Referenced by _extend_container_basic(), searx.search.processors.online.OnlineProcessor._search_basic(), searx.search.processors.online.OnlineProcessor._send_http_request(), extend_container_if_suspended(), get_params(), searx.search.processors.online.OnlineProcessor.get_params(), handle_exception(), init_engine(), searx.search.processors.online.OnlineProcessor.init_engine(), searx.search.processors.online.OnlineProcessor.init_network_in_thread(), initialize(), searx.search.processors.offline.OfflineProcessor.search(), and searx.search.processors.online.OnlineProcessor.search().

◆ logger

logging.Logger searx.search.processors.abstract.EngineProcessor.logger = engines[engine.name].logger

Definition at line 118 of file abstract.py.

Referenced by searx.search.processors.online.OnlineProcessor.get_params(), searx.search.processors.offline.OfflineProcessor.search(), and searx.search.processors.online.OnlineProcessor.search().

◆ suspended_status

SuspendedStatus searx.search.processors.abstract.EngineProcessor.suspended_status = SUSPENDED_STATUS.setdefault(key, SuspendedStatus())

Definition at line 121 of file abstract.py.

Referenced by extend_container(), extend_container_if_suspended(), and handle_exception().

The documentation for this class was generated from the following file:

/home/andrew/Documents/code/public/searxng/searx/search/processors/abstract.py

Public Member Functions

Public Attributes

Protected Member Functions

Detailed Description

Constructor & Destructor Documentation

◆ __init__()

Member Function Documentation

◆ _extend_container_basic()

◆ extend_container()

◆ extend_container_if_suspended()

◆ get_default_tests()

◆ get_params()

◆ get_tests()

◆ handle_exception()

◆ init_engine()

◆ initialize()

◆ search()

Member Data Documentation

◆ engine

◆ logger

◆ suspended_status

◆ init()