Inheritance diagram for searx.search.processors.abstract.EngineProcessor:

Collaboration diagram for searx.search.processors.abstract.EngineProcessor:

Public Member Functions
	__init__ (self, engine, str engine_name)

	initialize (self)

	has_initialize_function (self)

	handle_exception (self, result_container, exception_or_message, suspend=False)

	extend_container (self, result_container, start_time, search_results)

	extend_container_if_suspended (self, result_container)

	get_params (self, search_query, engine_category)

	search (self, query, params, result_container, start_time, timeout_limit)

	get_tests (self)

	get_default_tests (self)

Public Attributes
	engine

	engine_name

	logger

	suspended_status

Protected Member Functions
	_extend_container_basic (self, result_container, start_time, search_results)

Static Private Attributes
str	__slots__ = 'engine', 'engine_name', 'lock', 'suspended_status', 'logger'

Detailed Description

Base classes used for all types of request processors.

Definition at line 58 of file abstract.py.

Constructor & Destructor Documentation

◆ init()

searx.search.processors.abstract.EngineProcessor.__init__	(		self,
			engine,
		str	engine_name )

Definition at line 63 of file abstract.py.

    def __init__(self, engine, engine_name: str):
        self.engine = engine
        self.engine_name = engine_name
        self.logger = engines[engine_name].logger
        key = get_network(self.engine_name)
        key = id(key) if key else self.engine_name
        self.suspended_status = SUSPENDED_STATUS.setdefault(key, SuspendedStatus())
 

Member Function Documentation

◆ _extend_container_basic()

searx.search.processors.abstract.EngineProcessor._extend_container_basic	(	self,
		result_container,
		start_time,
		search_results )

protected

Definition at line 108 of file abstract.py.

    def _extend_container_basic(self, result_container, start_time, search_results):
        # update result_container
        result_container.extend(self.engine_name, search_results)
        engine_time = default_timer() - start_time
        page_load_time = get_time_for_thread()
        result_container.add_timing(self.engine_name, engine_time, page_load_time)
        # metrics
        counter_inc('engine', self.engine_name, 'search', 'count', 'successful')
        histogram_observe(engine_time, 'engine', self.engine_name, 'time', 'total')
        if page_load_time is not None:
            histogram_observe(page_load_time, 'engine', self.engine_name, 'time', 'http')
 

References searx.search.processors.abstract.EngineProcessor.engine_name, searx.search.processors.offline.OfflineProcessor.engine_name, and searx.search.processors.online.OnlineProcessor.engine_name.

Referenced by searx.search.processors.abstract.EngineProcessor.extend_container().

Here is the caller graph for this function:

◆ extend_container()

searx.search.processors.abstract.EngineProcessor.extend_container	(	self,
		result_container,
		start_time,
		search_results )

Definition at line 120 of file abstract.py.

    def extend_container(self, result_container, start_time, search_results):
        if getattr(threading.current_thread(), '_timeout', False):
            # the main thread is not waiting anymore
            self.handle_exception(result_container, 'timeout', None)
        else:
            # check if the engine accepted the request
            if search_results is not None:
                self._extend_container_basic(result_container, start_time, search_results)
            self.suspended_status.resume()
 

References searx.search.processors.abstract.EngineProcessor._extend_container_basic(), searx.search.processors.abstract.EngineProcessor.handle_exception(), and searx.search.processors.abstract.EngineProcessor.suspended_status.

Referenced by searx.search.processors.offline.OfflineProcessor.search(), and searx.search.processors.online.OnlineProcessor.search().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ extend_container_if_suspended()

searx.search.processors.abstract.EngineProcessor.extend_container_if_suspended	(		self,
			result_container )

Definition at line 130 of file abstract.py.

    def extend_container_if_suspended(self, result_container):
        if self.suspended_status.is_suspended:
            result_container.add_unresponsive_engine(
                self.engine_name, self.suspended_status.suspend_reason, suspended=True
            )
            return True
        return False
 

References searx.search.processors.abstract.EngineProcessor.engine_name, searx.search.processors.offline.OfflineProcessor.engine_name, searx.search.processors.online.OnlineProcessor.engine_name, and searx.search.processors.abstract.EngineProcessor.suspended_status.

◆ get_default_tests()

searx.search.processors.abstract.EngineProcessor.get_default_tests ( self )

Reimplemented in searx.search.processors.online.OnlineProcessor, searx.search.processors.online_currency.OnlineCurrencyProcessor, and searx.search.processors.online_dictionary.OnlineDictionaryProcessor.

Definition at line 193 of file abstract.py.

193 def get_default_tests(self):

194 return {}

◆ get_params()

searx.search.processors.abstract.EngineProcessor.get_params	(	self,
		search_query,
		engine_category )

Returns a set of (see :ref:`request params <engine request arguments>`) or
``None`` if request is not supported.

Not supported conditions (``None`` is returned):

- A page-number > 1 when engine does not support paging.
- A time range when the engine does not support time range.

Reimplemented in searx.search.processors.online.OnlineProcessor, searx.search.processors.online_currency.OnlineCurrencyProcessor, searx.search.processors.online_dictionary.OnlineDictionaryProcessor, and searx.search.processors.online_url_search.OnlineUrlSearchProcessor.

Definition at line 138 of file abstract.py.

    def get_params(self, search_query, engine_category):
        """Returns a set of (see :ref:`request params <engine request arguments>`) or
        ``None`` if request is not supported.
 
        Not supported conditions (``None`` is returned):
 
        - A page-number > 1 when engine does not support paging.
        - A time range when the engine does not support time range.
        """
        # if paging is not supported, skip
        if search_query.pageno > 1 and not self.engine.paging:
            return None
 
        # if max page is reached, skip
        max_page = self.engine.max_page or settings['search']['max_page']
        if max_page and max_page < search_query.pageno:
            return None
 
        # if time_range is not supported, skip
        if search_query.time_range and not self.engine.time_range_support:
            return None
 
        params = {}
        params['category'] = engine_category
        params['pageno'] = search_query.pageno
        params['safesearch'] = search_query.safesearch
        params['time_range'] = search_query.time_range
        params['engine_data'] = search_query.engine_data.get(self.engine_name, {})
        params['searxng_locale'] = search_query.lang
 
        # deprecated / vintage --> use params['searxng_locale']
        #
        # Conditions related to engine's traits are implemented in engine.traits
        # module. Don't do 'locale' decisions here in the abstract layer of the
        # search processor, just pass the value from user's choice unchanged to
        # the engine request.
 
        if hasattr(self.engine, 'language') and self.engine.language:
            params['language'] = self.engine.language
        else:
            params['language'] = search_query.lang
 
        return params
 

References searx.enginelib.Engine.engine, searx.results.Timing.engine, searx.results.UnresponsiveEngine.engine, searx.search.processors.abstract.EngineProcessor.engine, searx.search.processors.online.OnlineProcessor.engine, searx.search.processors.online_dictionary.OnlineDictionaryProcessor.engine, searx.search.processors.abstract.EngineProcessor.engine_name, searx.search.processors.offline.OfflineProcessor.engine_name, and searx.search.processors.online.OnlineProcessor.engine_name.

◆ get_tests()

searx.search.processors.abstract.EngineProcessor.get_tests ( self )

Definition at line 186 of file abstract.py.

    def get_tests(self):
        tests = getattr(self.engine, 'tests', None)
        if tests is None:
            tests = getattr(self.engine, 'additional_tests', {})
            tests.update(self.get_default_tests())
        return tests
 

◆ handle_exception()

searx.search.processors.abstract.EngineProcessor.handle_exception	(	self,
		result_container,
		exception_or_message,
		suspend = False )

Definition at line 85 of file abstract.py.

    def handle_exception(self, result_container, exception_or_message, suspend=False):
        # update result_container
        if isinstance(exception_or_message, BaseException):
            exception_class = exception_or_message.__class__
            module_name = getattr(exception_class, '__module__', 'builtins')
            module_name = '' if module_name == 'builtins' else module_name + '.'
            error_message = module_name + exception_class.__qualname__
        else:
            error_message = exception_or_message
        result_container.add_unresponsive_engine(self.engine_name, error_message)
        # metrics
        counter_inc('engine', self.engine_name, 'search', 'count', 'error')
        if isinstance(exception_or_message, BaseException):
            count_exception(self.engine_name, exception_or_message)
        else:
            count_error(self.engine_name, exception_or_message)
        # suspend the engine ?
        if suspend:
            suspended_time = None
            if isinstance(exception_or_message, SearxEngineAccessDeniedException):
                suspended_time = exception_or_message.suspended_time
            self.suspended_status.suspend(suspended_time, error_message)  # pylint: disable=no-member
 

References searx.search.processors.abstract.EngineProcessor.engine_name, searx.search.processors.offline.OfflineProcessor.engine_name, searx.search.processors.online.OnlineProcessor.engine_name, and searx.search.processors.abstract.EngineProcessor.suspended_status.

Referenced by searx.search.processors.abstract.EngineProcessor.extend_container(), and searx.search.processors.online.OnlineProcessor.search().

Here is the caller graph for this function:

◆ has_initialize_function()

searx.search.processors.abstract.EngineProcessor.has_initialize_function ( self )

Definition at line 82 of file abstract.py.

    def has_initialize_function(self):
        return hasattr(self.engine, 'init')
 

References searx.enginelib.Engine.engine, searx.results.Timing.engine, searx.results.UnresponsiveEngine.engine, searx.search.processors.abstract.EngineProcessor.engine, searx.search.processors.online.OnlineProcessor.engine, and searx.search.processors.online_dictionary.OnlineDictionaryProcessor.engine.

◆ initialize()

searx.search.processors.abstract.EngineProcessor.initialize ( self )

Reimplemented in searx.search.processors.online.OnlineProcessor.

Definition at line 71 of file abstract.py.

    def initialize(self):
        try:
            self.engine.init(get_engine_from_settings(self.engine_name))
        except SearxEngineResponseException as exc:
            self.logger.warning('Fail to initialize // %s', exc)
        except Exception:  # pylint: disable=broad-except
            self.logger.exception('Fail to initialize')
        else:
            self.logger.debug('Initialized')
 

References searx.enginelib.Engine.engine, searx.results.Timing.engine, searx.results.UnresponsiveEngine.engine, searx.search.processors.abstract.EngineProcessor.engine, searx.search.processors.online.OnlineProcessor.engine, searx.search.processors.online_dictionary.OnlineDictionaryProcessor.engine, searx.search.processors.abstract.EngineProcessor.engine_name, searx.search.processors.offline.OfflineProcessor.engine_name, searx.search.processors.online.OnlineProcessor.engine_name, and searx.search.processors.abstract.EngineProcessor.logger.

◆ search()

searx.search.processors.abstract.EngineProcessor.search	(	self,
		query,
		params,
		result_container,
		start_time,
		timeout_limit )

Reimplemented in searx.search.processors.offline.OfflineProcessor, and searx.search.processors.online.OnlineProcessor.

Definition at line 183 of file abstract.py.

    def search(self, query, params, result_container, start_time, timeout_limit):
        pass
 

Member Data Documentation

◆ slots

str searx.search.processors.abstract.EngineProcessor.__slots__ = 'engine', 'engine_name', 'lock', 'suspended_status', 'logger'

staticprivate

Definition at line 61 of file abstract.py.

◆ engine

searx.search.processors.abstract.EngineProcessor.engine

Definition at line 64 of file abstract.py.

Referenced by searx.search.processors.offline.OfflineProcessor._search_basic(), searx.search.processors.online.OnlineProcessor._search_basic(), searx.search.processors.abstract.EngineProcessor.get_params(), searx.search.processors.online.OnlineProcessor.get_params(), searx.search.processors.abstract.EngineProcessor.has_initialize_function(), searx.search.processors.abstract.EngineProcessor.initialize(), and searx.search.processors.online.OnlineProcessor.initialize().

◆ engine_name

searx.search.processors.abstract.EngineProcessor.engine_name

Definition at line 65 of file abstract.py.

Referenced by searx.search.processors.abstract.EngineProcessor._extend_container_basic(), searx.search.processors.online.OnlineProcessor._send_http_request(), searx.search.processors.abstract.EngineProcessor.extend_container_if_suspended(), searx.search.processors.abstract.EngineProcessor.get_params(), searx.search.processors.abstract.EngineProcessor.handle_exception(), searx.search.processors.abstract.EngineProcessor.initialize(), and searx.search.processors.online.OnlineProcessor.search().

◆ logger

searx.search.processors.abstract.EngineProcessor.logger

Definition at line 66 of file abstract.py.

Referenced by searx.search.processors.online.OnlineProcessor.get_params(), searx.search.processors.abstract.EngineProcessor.initialize(), and searx.search.processors.online.OnlineProcessor.search().

◆ suspended_status

searx.search.processors.abstract.EngineProcessor.suspended_status

Definition at line 69 of file abstract.py.

Referenced by searx.search.processors.abstract.EngineProcessor.extend_container(), searx.search.processors.abstract.EngineProcessor.extend_container_if_suspended(), and searx.search.processors.abstract.EngineProcessor.handle_exception().

The documentation for this class was generated from the following file:

/home/andrew/Documents/code/public/searxng/searx/search/processors/abstract.py

Public Member Functions

Public Attributes

Protected Member Functions

Static Private Attributes

Detailed Description

Constructor & Destructor Documentation

◆ __init__()

Member Function Documentation

◆ _extend_container_basic()

◆ extend_container()

◆ extend_container_if_suspended()

◆ get_default_tests()

◆ get_params()

◆ get_tests()

◆ handle_exception()

◆ has_initialize_function()

◆ initialize()

◆ search()

Member Data Documentation

◆ __slots__

◆ engine

◆ engine_name

◆ logger

◆ suspended_status

◆ init()

◆ slots