.oO SearXNG Developer Documentation Oo.
Loading...
Searching...
No Matches
searx.search.processors.online.OnlineProcessor Class Reference
Inheritance diagram for searx.search.processors.online.OnlineProcessor:
Collaboration diagram for searx.search.processors.online.OnlineProcessor:

Public Member Functions

bool init_engine (self)
 init_network_in_thread (self, float start_time, float timeout_limit)
OnlineParams|None get_params (self, "SearchQuery" search_query, str engine_category)
 search (self, str query, OnlineParams params, "ResultContainer" result_container, float start_time, float timeout_limit)
Public Member Functions inherited from searx.search.processors.abstract.EngineProcessor
 __init__ (self, "Engine|types.ModuleType" engine)
 initialize (self, t.Callable[["EngineProcessor", bool], bool] callback)
 handle_exception (self, "ResultContainer" result_container, BaseException|str exception_or_message, bool suspend=False)
 extend_container (self, "ResultContainer" result_container, float start_time, "list[Result | LegacyResult]|None" search_results)
bool extend_container_if_suspended (self, "ResultContainer" result_container)
 get_tests (self)
 get_default_tests (self)

Static Public Attributes

str engine_type = "online"

Protected Member Functions

 _send_http_request (self, OnlineParams params)
"EngineResults|None" _search_basic (self, str query, OnlineParams params)
Protected Member Functions inherited from searx.search.processors.abstract.EngineProcessor
 _extend_container_basic (self, "ResultContainer" result_container, float start_time, "list[Result | LegacyResult]" search_results)

Additional Inherited Members

Public Attributes inherited from searx.search.processors.abstract.EngineProcessor
 engine = engine
logging.Logger logger = engines[engine.name].logger
SuspendedStatus suspended_status = SUSPENDED_STATUS.setdefault(key, SuspendedStatus())

Detailed Description

Processor class for ``online`` engines.

Definition at line 113 of file online.py.

Member Function Documentation

◆ _search_basic()

"EngineResults|None" searx.search.processors.online.OnlineProcessor._search_basic ( self,
str query,
OnlineParams params )
protected

Definition at line 221 of file online.py.

221 def _search_basic(self, query: str, params: OnlineParams) -> "EngineResults|None":
222 # update request parameters dependent on
223 # search-engine (contained in engines folder)
224 self.engine.request(query, params)
225
226 # ignoring empty urls
227 if not params["url"]:
228 return None
229
230 # send request
231 response = self._send_http_request(params)
232
233 # parse the response
234 response.search_params = params
235 return self.engine.response(response)
236

References _send_http_request(), searx.result_types._base.LegacyResult.engine, searx.result_types._base.MainResult.engine, searx.result_types._base.Result.engine, and searx.search.processors.abstract.EngineProcessor.engine.

Referenced by search().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ _send_http_request()

searx.search.processors.online.OnlineProcessor._send_http_request ( self,
OnlineParams params )
protected

Definition at line 162 of file online.py.

162 def _send_http_request(self, params: OnlineParams):
163
164 # create dictionary which contain all information about the request
165 request_args: dict[str, t.Any] = {
166 "headers": params["headers"],
167 "cookies": params["cookies"],
168 "auth": params["auth"],
169 }
170
171 verify = params.get("verify")
172 if verify is not None:
173 request_args["verify"] = verify
174
175 # max_redirects
176 max_redirects = params.get("max_redirects")
177 if max_redirects:
178 request_args["max_redirects"] = max_redirects
179
180 # allow_redirects
181 if "allow_redirects" in params:
182 request_args["allow_redirects"] = params["allow_redirects"]
183
184 # soft_max_redirects
185 soft_max_redirects: int = params.get("soft_max_redirects", max_redirects or 0)
186
187 # raise_for_status
188 request_args["raise_for_httperror"] = params.get("raise_for_httperror", True)
189
190 # specific type of request (GET or POST)
191 if params["method"] == "GET":
192 req = searx.network.get
193 else:
194 req = searx.network.post
195 if params["data"]:
196 request_args["data"] = params["data"]
197 if params["json"]:
198 request_args["json"] = params["json"]
199 if params["content"]:
200 request_args["content"] = params["content"]
201
202 # send the request
203 response = req(params["url"], **request_args)
204
205 # check soft limit of the redirect count
206 if len(response.history) > soft_max_redirects:
207 # unexpected redirect : record an error
208 # but the engine might still return valid results.
209 status_code = str(response.status_code or "")
210 reason = response.reason_phrase or ""
211 hostname = response.url.host
212 count_error(
213 self.engine.name,
214 "{} redirects, maximum: {}".format(len(response.history), soft_max_redirects),
215 (status_code, reason, hostname),
216 secondary=True,
217 )
218
219 return response
220

References searx.result_types._base.LegacyResult.engine, searx.result_types._base.MainResult.engine, searx.result_types._base.Result.engine, and searx.search.processors.abstract.EngineProcessor.engine.

Referenced by _search_basic().

Here is the caller graph for this function:

◆ get_params()

OnlineParams | None searx.search.processors.online.OnlineProcessor.get_params ( self,
"SearchQuery" search_query,
str engine_category )
Returns a dictionary with the :ref:`request params <engine request
online>` (:py:obj:`OnlineParams`), if the search condition is not
supported by the engine, ``None`` is returned.

Reimplemented from searx.search.processors.abstract.EngineProcessor.

Reimplemented in searx.search.processors.online_currency.OnlineCurrencyProcessor, searx.search.processors.online_dictionary.OnlineDictionaryProcessor, and searx.search.processors.online_url_search.OnlineUrlSearchProcessor.

Definition at line 132 of file online.py.

132 def get_params(self, search_query: "SearchQuery", engine_category: str) -> OnlineParams | None:
133 """Returns a dictionary with the :ref:`request params <engine request
134 online>` (:py:obj:`OnlineParams`), if the search condition is not
135 supported by the engine, ``None`` is returned."""
136
137 base_params: RequestParams | None = super().get_params(search_query, engine_category)
138 if base_params is None:
139 return base_params
140
141 params: OnlineParams = {**default_request_params(), **base_params}
142
143 headers = params["headers"]
144
145 # add an user agent
146 headers["User-Agent"] = gen_useragent()
147
148 # add Accept-Language header
149 if self.engine.send_accept_language_header and search_query.locale:
150 ac_lang = search_query.locale.language
151 if search_query.locale.territory:
152 ac_lang = "%s-%s,%s;q=0.9,*;q=0.5" % (
153 search_query.locale.language,
154 search_query.locale.territory,
155 search_query.locale.language,
156 )
157 headers["Accept-Language"] = ac_lang
158
159 self.logger.debug("HTTP Accept-Language: %s", headers.get("Accept-Language", ""))
160 return params
161

References searx.search.processors.online.default_request_params(), searx.result_types._base.LegacyResult.engine, searx.result_types._base.MainResult.engine, searx.result_types._base.Result.engine, searx.search.processors.abstract.EngineProcessor.engine, get_params(), searx.enginelib.Engine.logger, and searx.search.processors.abstract.EngineProcessor.logger.

Referenced by get_params().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ init_engine()

bool searx.search.processors.online.OnlineProcessor.init_engine ( self)
This method is called in a thread, and before the base method is
called, the network must be set up for the ``online`` engines.

Reimplemented from searx.search.processors.abstract.EngineProcessor.

Definition at line 118 of file online.py.

118 def init_engine(self) -> bool:
119 """This method is called in a thread, and before the base method is
120 called, the network must be set up for the ``online`` engines."""
121 self.init_network_in_thread(start_time=default_timer(), timeout_limit=self.engine.timeout)
122 return super().init_engine()
123

References searx.result_types._base.LegacyResult.engine, searx.result_types._base.MainResult.engine, searx.result_types._base.Result.engine, searx.search.processors.abstract.EngineProcessor.engine, init_engine(), and init_network_in_thread().

Referenced by init_engine().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ init_network_in_thread()

searx.search.processors.online.OnlineProcessor.init_network_in_thread ( self,
float start_time,
float timeout_limit )

Definition at line 124 of file online.py.

124 def init_network_in_thread(self, start_time: float, timeout_limit: float):
125 # set timeout for all HTTP requests
126 searx.network.set_timeout_for_thread(timeout_limit, start_time=start_time)
127 # reset the HTTP total time
129 # set the network
131
set_context_network_name(str network_name)
Definition __init__.py:46
set_timeout_for_thread(float timeout, float|None start_time=None)
Definition __init__.py:41
reset_time_for_thread()
Definition __init__.py:32

References searx.result_types._base.LegacyResult.engine, searx.result_types._base.MainResult.engine, searx.result_types._base.Result.engine, searx.search.processors.abstract.EngineProcessor.engine, searx.network.reset_time_for_thread(), searx.network.set_context_network_name(), and searx.network.set_timeout_for_thread().

Referenced by init_engine(), and search().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ search()

searx.search.processors.online.OnlineProcessor.search ( self,
str query,
OnlineParams params,
"ResultContainer" result_container,
float start_time,
float timeout_limit )

Reimplemented from searx.search.processors.abstract.EngineProcessor.

Definition at line 237 of file online.py.

244 ):
245 self.init_network_in_thread(start_time, timeout_limit)
246
247 try:
248 # send requests and parse the results
249 search_results = self._search_basic(query, params)
250 self.extend_container(result_container, start_time, search_results)
251 except ssl.SSLError as e:
252 # requests timeout (connect or read)
253 self.handle_exception(result_container, e, suspend=True)
254 self.logger.error("SSLError {}, verify={}".format(e, searx.network.get_network(self.engine.name).verify))
255 except (httpx.TimeoutException, asyncio.TimeoutError) as e:
256 # requests timeout (connect or read)
257 self.handle_exception(result_container, e, suspend=True)
258 self.logger.error(
259 "HTTP requests timeout (search duration : {0} s, timeout: {1} s) : {2}".format(
260 default_timer() - start_time, timeout_limit, e.__class__.__name__
261 )
262 )
263 except (httpx.HTTPError, httpx.StreamError) as e:
264 # other requests exception
265 self.handle_exception(result_container, e, suspend=True)
266 self.logger.exception(
267 "requests exception (search duration : {0} s, timeout: {1} s) : {2}".format(
268 default_timer() - start_time, timeout_limit, e
269 )
270 )
271 except (
272 SearxEngineCaptchaException,
273 SearxEngineTooManyRequestsException,
274 SearxEngineAccessDeniedException,
275 ) as e:
276 self.handle_exception(result_container, e, suspend=True)
277 self.logger.exception(e.message)
278 except Exception as e: # pylint: disable=broad-except
279 self.handle_exception(result_container, e)
280 self.logger.exception("exception : {0}".format(e))

References _search_basic(), searx.result_types._base.LegacyResult.engine, searx.result_types._base.MainResult.engine, searx.result_types._base.Result.engine, searx.search.processors.abstract.EngineProcessor.engine, searx.search.processors.abstract.EngineProcessor.extend_container(), searx.search.processors.abstract.EngineProcessor.handle_exception(), init_network_in_thread(), searx.enginelib.Engine.logger, and searx.search.processors.abstract.EngineProcessor.logger.

Here is the call graph for this function:

Member Data Documentation

◆ engine_type

str searx.search.processors.online.OnlineProcessor.engine_type = "online"
static

Definition at line 116 of file online.py.


The documentation for this class was generated from the following file:
  • /home/andrew/Documents/code/public/searxng/searx/search/processors/online.py