searx.search.processors.online.OnlineProcessor Class Reference

Inheritance diagram for searx.search.processors.online.OnlineProcessor:

Collaboration diagram for searx.search.processors.online.OnlineProcessor:

Public Member Functions
	initialize (self)
	get_params (self, search_query, engine_category)
	search (self, query, params, result_container, start_time, timeout_limit)
	get_default_tests (self)
Public Member Functions inherited from searx.search.processors.abstract.EngineProcessor
	__init__ (self, engine, str engine_name)
	has_initialize_function (self)
	handle_exception (self, result_container, exception_or_message, suspend=False)
	extend_container (self, result_container, start_time, search_results)
	extend_container_if_suspended (self, result_container)
	get_tests (self)

Public Attributes
	engine_name
	engine
Public Attributes inherited from searx.search.processors.abstract.EngineProcessor
	engine = engine
	engine_name = engine_name
	logger = engines[engine_name].logger
	suspended_status = SUSPENDED_STATUS.setdefault(key, SuspendedStatus())

Static Public Attributes
str	engine_type = 'online'

Protected Member Functions
	_send_http_request (self, params)
	_search_basic (self, query, params)
Protected Member Functions inherited from searx.search.processors.abstract.EngineProcessor
	_extend_container_basic (self, result_container, start_time, search_results)

Detailed Description

Processor class for ``online`` engines.

Definition at line 37 of file online.py.

Member Function Documentation

_search_basic()

searx.search.processors.online.OnlineProcessor._search_basic ( self, query, params )

protected

Definition at line 134 of file online.py.

    def _search_basic(self, query, params):
        # update request parameters dependent on
        # search-engine (contained in engines folder)
        self.engine.request(query, params)
 
        # ignoring empty urls
        if params['url'] is None:
            return None
 
        if not params['url']:
            return None
 
        # send request
        response = self._send_http_request(params)
 
        # parse the response
        response.search_params = params
        return self.engine.response(response)
 

References searx.search.processors.online.OnlineProcessor._send_http_request(), searx.search.processors.abstract.EngineProcessor.engine, searx.search.processors.online.OnlineProcessor.engine, and searx.search.processors.online_dictionary.OnlineDictionaryProcessor.engine.

Referenced by searx.search.processors.offline.OfflineProcessor.search(), and searx.search.processors.online.OnlineProcessor.search().

Here is the call graph for this function:

Here is the caller graph for this function:

_send_http_request()

searx.search.processors.online.OnlineProcessor._send_http_request ( self, params )

protected

Definition at line 79 of file online.py.

    def _send_http_request(self, params):
        # create dictionary which contain all
        # information about the request
        request_args = dict(headers=params['headers'], cookies=params['cookies'], auth=params['auth'])
 
        # verify
        # if not None, it overrides the verify value defined in the network.
        # use False to accept any server certificate
        # use a path to file to specify a server certificate
        verify = params.get('verify')
        if verify is not None:
            request_args['verify'] = params['verify']
 
        # max_redirects
        max_redirects = params.get('max_redirects')
        if max_redirects:
            request_args['max_redirects'] = max_redirects
 
        # allow_redirects
        if 'allow_redirects' in params:
            request_args['allow_redirects'] = params['allow_redirects']
 
        # soft_max_redirects
        soft_max_redirects = params.get('soft_max_redirects', max_redirects or 0)
 
        # raise_for_status
        request_args['raise_for_httperror'] = params.get('raise_for_httperror', True)
 
        # specific type of request (GET or POST)
        if params['method'] == 'GET':
            req = searx.network.get
        else:
            req = searx.network.post
 
        request_args['data'] = params['data']
 
        # send the request
        response = req(params['url'], **request_args)
 
        # check soft limit of the redirect count
        if len(response.history) > soft_max_redirects:
            # unexpected redirect : record an error
            # but the engine might still return valid results.
            status_code = str(response.status_code or '')
            reason = response.reason_phrase or ''
            hostname = response.url.host
            count_error(
                self.engine_name,
                '{} redirects, maximum: {}'.format(len(response.history), soft_max_redirects),
                (status_code, reason, hostname),
                secondary=True,
            )
 
        return response
 

References searx.search.processors.abstract.EngineProcessor.engine_name, searx.search.processors.offline.OfflineProcessor.engine_name, searx.search.processors.online.OnlineProcessor.engine_name, and searx.format.

Referenced by searx.search.processors.online.OnlineProcessor._search_basic().

Here is the caller graph for this function:

get_default_tests()

searx.search.processors.online.OnlineProcessor.get_default_tests

(

self

)

Reimplemented from searx.search.processors.abstract.EngineProcessor.

Reimplemented in searx.search.processors.online_currency.OnlineCurrencyProcessor, and searx.search.processors.online_dictionary.OnlineDictionaryProcessor.

Definition at line 198 of file online.py.

    def get_default_tests(self):
        tests = {}
 
        tests['simple'] = {
            'matrix': {'query': ('life', 'computer')},
            'result_container': ['not_empty'],
        }
 
        if getattr(self.engine, 'paging', False):
            tests['paging'] = {
                'matrix': {'query': 'time', 'pageno': (1, 2, 3)},
                'result_container': ['not_empty'],
                'test': ['unique_results'],
            }
            if 'general' in self.engine.categories:
                # avoid documentation about HTML tags (<time> and <input type="time">)
                tests['paging']['matrix']['query'] = 'news'
 
        if getattr(self.engine, 'time_range', False):
            tests['time_range'] = {
                'matrix': {'query': 'news', 'time_range': (None, 'day')},
                'result_container': ['not_empty'],
                'test': ['unique_results'],
            }
 
        if getattr(self.engine, 'traits', False):
            tests['lang_fr'] = {
                'matrix': {'query': 'paris', 'lang': 'fr'},
                'result_container': ['not_empty', ('has_language', 'fr')],
            }
            tests['lang_en'] = {
                'matrix': {'query': 'paris', 'lang': 'en'},
                'result_container': ['not_empty', ('has_language', 'en')],
            }
 
        if getattr(self.engine, 'safesearch', False):
            tests['safesearch'] = {'matrix': {'query': 'porn', 'safesearch': (0, 2)}, 'test': ['unique_results']}
 
        return tests

get_params()

searx.search.processors.online.OnlineProcessor.get_params	(		self,
			search_query,
			engine_category )

Returns a set of :ref:`request params <engine request online>` or ``None``
if request is not supported.

Reimplemented from searx.search.processors.abstract.EngineProcessor.

Reimplemented in searx.search.processors.online_currency.OnlineCurrencyProcessor, searx.search.processors.online_dictionary.OnlineDictionaryProcessor, and searx.search.processors.online_url_search.OnlineUrlSearchProcessor.

Definition at line 51 of file online.py.

    def get_params(self, search_query, engine_category):
        """Returns a set of :ref:`request params <engine request online>` or ``None``
        if request is not supported.
        """
        params = super().get_params(search_query, engine_category)
        if params is None:
            return None
 
        # add default params
        params.update(default_request_params())
 
        # add an user agent
        params['headers']['User-Agent'] = gen_useragent()
 
        # add Accept-Language header
        if self.engine.send_accept_language_header and search_query.locale:
            ac_lang = search_query.locale.language
            if search_query.locale.territory:
                ac_lang = "%s-%s,%s;q=0.9,*;q=0.5" % (
                    search_query.locale.language,
                    search_query.locale.territory,
                    search_query.locale.language,
                )
            params['headers']['Accept-Language'] = ac_lang
 
        self.logger.debug('HTTP Accept-Language: %s', params['headers'].get('Accept-Language', ''))
        return params
 

References searx.search.processors.online.default_request_params(), searx.search.processors.abstract.EngineProcessor.engine, searx.search.processors.online.OnlineProcessor.engine, searx.search.processors.online_dictionary.OnlineDictionaryProcessor.engine, searx.search.processors.online.OnlineProcessor.get_params(), and searx.search.processors.abstract.EngineProcessor.logger.

Referenced by searx.search.processors.online.OnlineProcessor.get_params().

Here is the call graph for this function:

Here is the caller graph for this function:

initialize()

searx.search.processors.online.OnlineProcessor.initialize

(

self

)

Reimplemented from searx.search.processors.abstract.EngineProcessor.

Definition at line 42 of file online.py.

    def initialize(self):
        # set timeout for all HTTP requests
        searx.network.set_timeout_for_thread(self.engine.timeout, start_time=default_timer())
        # reset the HTTP total time
        searx.network.reset_time_for_thread()
        # set the network
        searx.network.set_context_network_name(self.engine_name)
        super().initialize()
 

References searx.search.processors.abstract.EngineProcessor.engine, searx.search.processors.online.OnlineProcessor.engine, searx.search.processors.online_dictionary.OnlineDictionaryProcessor.engine, searx.network.reset_time_for_thread(), and searx.network.set_timeout_for_thread().

Here is the call graph for this function:

search()

searx.search.processors.online.OnlineProcessor.search	(		self,
			query,
			params,
			result_container,
			start_time,
			timeout_limit )

Reimplemented from searx.search.processors.abstract.EngineProcessor.

Definition at line 153 of file online.py.

    def search(self, query, params, result_container, start_time, timeout_limit):
        # set timeout for all HTTP requests
        searx.network.set_timeout_for_thread(timeout_limit, start_time=start_time)
        # reset the HTTP total time
        searx.network.reset_time_for_thread()
        # set the network
        searx.network.set_context_network_name(self.engine_name)
 
        try:
            # send requests and parse the results
            search_results = self._search_basic(query, params)
            self.extend_container(result_container, start_time, search_results)
        except ssl.SSLError as e:
            # requests timeout (connect or read)
            self.handle_exception(result_container, e, suspend=True)
            self.logger.error("SSLError {}, verify={}".format(e, searx.network.get_network(self.engine_name).verify))
        except (httpx.TimeoutException, asyncio.TimeoutError) as e:
            # requests timeout (connect or read)
            self.handle_exception(result_container, e, suspend=True)
            self.logger.error(
                "HTTP requests timeout (search duration : {0} s, timeout: {1} s) : {2}".format(
                    default_timer() - start_time, timeout_limit, e.__class__.__name__
                )
            )
        except (httpx.HTTPError, httpx.StreamError) as e:
            # other requests exception
            self.handle_exception(result_container, e, suspend=True)
            self.logger.exception(
                "requests exception (search duration : {0} s, timeout: {1} s) : {2}".format(
                    default_timer() - start_time, timeout_limit, e
                )
            )
        except SearxEngineCaptchaException as e:
            self.handle_exception(result_container, e, suspend=True)
            self.logger.exception('CAPTCHA')
        except SearxEngineTooManyRequestsException as e:
            self.handle_exception(result_container, e, suspend=True)
            self.logger.exception('Too many requests')
        except SearxEngineAccessDeniedException as e:
            self.handle_exception(result_container, e, suspend=True)
            self.logger.exception('SearXNG is blocked')
        except Exception as e:  # pylint: disable=broad-except
            self.handle_exception(result_container, e)
            self.logger.exception('exception : {0}'.format(e))
 

References searx.search.processors.offline.OfflineProcessor._search_basic(), searx.search.processors.online.OnlineProcessor._search_basic(), searx.search.processors.abstract.EngineProcessor.engine_name, searx.search.processors.offline.OfflineProcessor.engine_name, searx.search.processors.online.OnlineProcessor.engine_name, searx.search.processors.abstract.EngineProcessor.extend_container(), searx.format, searx.search.processors.abstract.EngineProcessor.handle_exception(), searx.search.processors.abstract.EngineProcessor.logger, searx.network.reset_time_for_thread(), searx.network.set_context_network_name(), and searx.network.set_timeout_for_thread().

Here is the call graph for this function:

Member Data Documentation

engine

searx.search.processors.online.OnlineProcessor.engine

Initial value:

=  {
                'matrix': {'query': 'news', 'time_range': (None, 'day')},
                'result_container': ['not_empty'],
                'test': ['unique_results'],
            }

Definition at line 206 of file online.py.

Referenced by searx.search.processors.offline.OfflineProcessor._search_basic(), searx.search.processors.online.OnlineProcessor._search_basic(), searx.search.processors.abstract.EngineProcessor.get_params(), searx.search.processors.online.OnlineProcessor.get_params(), searx.search.processors.abstract.EngineProcessor.has_initialize_function(), searx.search.processors.abstract.EngineProcessor.initialize(), and searx.search.processors.online.OnlineProcessor.initialize().

engine_name

searx.search.processors.online.OnlineProcessor.engine_name

Definition at line 48 of file online.py.

Referenced by searx.search.processors.abstract.EngineProcessor._extend_container_basic(), searx.search.processors.online.OnlineProcessor._send_http_request(), searx.search.processors.abstract.EngineProcessor.extend_container_if_suspended(), searx.search.processors.abstract.EngineProcessor.get_params(), searx.search.processors.abstract.EngineProcessor.handle_exception(), searx.search.processors.abstract.EngineProcessor.initialize(), and searx.search.processors.online.OnlineProcessor.search().

engine_type

str searx.search.processors.online.OnlineProcessor.engine_type = 'online'

static

Definition at line 40 of file online.py.

---

The documentation for this class was generated from the following file:

• /home/andrew/Documents/code/public/searxng/searx/search/processors/online.py