.oO SearXNG Developer Documentation Oo.
Loading...
Searching...
No Matches
searx.network Namespace Reference

Namespaces

namespace  client
namespace  network
namespace  raise_for_httperror

Classes

class  Request

Functions

 reset_time_for_thread ()
float|None get_time_for_thread ()
 set_timeout_for_thread (float timeout, float|None start_time=None)
 set_context_network_name (str network_name)
"Network" get_context_network ()
 _record_http_time ()
float _get_timeout (float start_time, t.Any kwargs)
SXNG_Response request (str method, str url, **t.Any kwargs)
list[httpx.Response|Exception] multi_requests (list["Request"] request_list)
SXNG_Response get (str url, **t.Any kwargs)
SXNG_Response options (str url, **t.Any kwargs)
SXNG_Response head (str url, **t.Any kwargs)
SXNG_Response post (str url, dict[str, t.Any]|None data=None, **t.Any kwargs)
SXNG_Response put (str url, dict[str, t.Any]|None data=None, **t.Any kwargs)
SXNG_Response patch (str url, dict[str, t.Any]|None data=None, **t.Any kwargs)
SXNG_Response delete (str url, **t.Any kwargs)
 stream_chunk_to_queue (network, queue, str method, str url, **t.Any kwargs)
 _stream_generator (str method, str url, **t.Any kwargs)
 _close_response_method (self)
tuple[SXNG_Response, Iterable[bytes]] stream (str method, str url, **t.Any kwargs)

Variables

list __all__ = ["get_network", "initialize", "check_network_configuration", "raise_for_httperror"]
 THREADLOCAL = threading.local()
# pylint disable _generator = protected-access

Function Documentation

◆ _close_response_method()

searx.network._close_response_method ( self)
protected

Definition at line 245 of file __init__.py.

245def _close_response_method(self):
246 asyncio.run_coroutine_threadsafe(self.aclose(), get_loop())
247 # reach the end of _self.generator ( _stream_generator ) to an avoid memory leak.
248 # it makes sure that :
249 # * the httpx response is closed (see the stream_chunk_to_queue function)
250 # * to call future.result() in _stream_generator
251 for _ in self._generator: # pylint: disable=protected-access
252 continue
253
254

◆ _get_timeout()

float searx.network._get_timeout ( float start_time,
t.Any kwargs )
protected

Definition at line 73 of file __init__.py.

73def _get_timeout(start_time: float, kwargs: t.Any) -> float:
74 # pylint: disable=too-many-branches
75
76 timeout: float | None
77 # timeout (httpx)
78 if 'timeout' in kwargs:
79 timeout = kwargs['timeout']
80 else:
81 timeout = getattr(THREADLOCAL, 'timeout', None)
82 if timeout is not None:
83 kwargs['timeout'] = timeout
84
85 # 2 minutes timeout for the requests without timeout
86 timeout = timeout or 120
87
88 # adjust actual timeout
89 timeout += 0.2 # overhead
90 if start_time:
91 timeout -= default_timer() - start_time
92
93 return timeout
94
95

Referenced by multi_requests(), and request().

Here is the caller graph for this function:

◆ _record_http_time()

searx.network._record_http_time ( )
protected

Definition at line 59 of file __init__.py.

59def _record_http_time():
60 # pylint: disable=too-many-branches
61 time_before_request = default_timer()
62 start_time = getattr(THREADLOCAL, 'start_time', time_before_request)
63 try:
64 yield start_time
65 finally:
66 # update total_time.
67 # See get_time_for_thread() and reset_time_for_thread()
68 if hasattr(THREADLOCAL, 'total_time'):
69 time_after_request = default_timer()
70 THREADLOCAL.total_time += time_after_request - time_before_request
71
72

Referenced by multi_requests(), and request().

Here is the caller graph for this function:

◆ _stream_generator()

searx.network._stream_generator ( str method,
str url,
**t.Any kwargs )
protected

Definition at line 230 of file __init__.py.

230def _stream_generator(method: str, url: str, **kwargs: t.Any):
231 queue = SimpleQueue()
232 network = get_context_network()
233 future = asyncio.run_coroutine_threadsafe(stream_chunk_to_queue(network, queue, method, url, **kwargs), get_loop())
234
235 # yield chunks
236 obj_or_exception = queue.get()
237 while obj_or_exception is not None:
238 if isinstance(obj_or_exception, Exception):
239 raise obj_or_exception
240 yield obj_or_exception
241 obj_or_exception = queue.get()
242 future.result()
243
244

References get_context_network(), and stream_chunk_to_queue().

Referenced by stream().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ delete()

SXNG_Response searx.network.delete ( str url,
**t.Any kwargs )

Definition at line 200 of file __init__.py.

200def delete(url: str, **kwargs: t.Any) -> SXNG_Response:
201 return request('delete', url, **kwargs)
202
203

References request().

Here is the call graph for this function:

◆ get()

SXNG_Response searx.network.get ( str url,
**t.Any kwargs )

Definition at line 173 of file __init__.py.

173def get(url: str, **kwargs: t.Any) -> SXNG_Response:
174 kwargs.setdefault('allow_redirects', True)
175 return request('get', url, **kwargs)
176
177

References request().

Here is the call graph for this function:

◆ get_context_network()

"Network" searx.network.get_context_network ( )
If set return thread's network.

If unset, return value from :py:obj:`get_network`.

Definition at line 50 of file __init__.py.

50def get_context_network() -> "Network":
51 """If set return thread's network.
52
53 If unset, return value from :py:obj:`get_network`.
54 """
55 return THREADLOCAL.__dict__.get('network') or get_network()
56
57
58@contextmanager

Referenced by _stream_generator(), multi_requests(), and request().

Here is the caller graph for this function:

◆ get_time_for_thread()

float | None searx.network.get_time_for_thread ( )
returns thread's total time or None

Definition at line 36 of file __init__.py.

36def get_time_for_thread() -> float | None:
37 """returns thread's total time or None"""
38 return THREADLOCAL.__dict__.get('total_time')
39
40

◆ head()

SXNG_Response searx.network.head ( str url,
**t.Any kwargs )

Definition at line 183 of file __init__.py.

183def head(url: str, **kwargs: t.Any) -> SXNG_Response:
184 kwargs.setdefault('allow_redirects', False)
185 return request('head', url, **kwargs)
186
187

References request().

Here is the call graph for this function:

◆ multi_requests()

list[httpx.Response | Exception] searx.network.multi_requests ( list["Request"] request_list)
send multiple HTTP requests in parallel. Wait for all requests to finish.

Definition at line 111 of file __init__.py.

111def multi_requests(request_list: list["Request"]) -> list[httpx.Response | Exception]:
112 """send multiple HTTP requests in parallel. Wait for all requests to finish."""
113 with _record_http_time() as start_time:
114 # send the requests
115 network = get_context_network()
116 loop = get_loop()
117 future_list = []
118 for request_desc in request_list:
119 timeout = _get_timeout(start_time, request_desc.kwargs)
120 future = asyncio.run_coroutine_threadsafe(
121 network.request(request_desc.method, request_desc.url, **request_desc.kwargs), loop
122 )
123 future_list.append((future, timeout))
124
125 # read the responses
126 responses = []
127 for future, timeout in future_list:
128 try:
129 responses.append(future.result(timeout))
130 except concurrent.futures.TimeoutError:
131 responses.append(httpx.TimeoutException('Timeout', request=None))
132 except Exception as e: # pylint: disable=broad-except
133 responses.append(e)
134 return responses
135
136

References _get_timeout(), _record_http_time(), and get_context_network().

Here is the call graph for this function:

◆ options()

SXNG_Response searx.network.options ( str url,
**t.Any kwargs )

Definition at line 178 of file __init__.py.

178def options(url: str, **kwargs: t.Any) -> SXNG_Response:
179 kwargs.setdefault('allow_redirects', True)
180 return request('options', url, **kwargs)
181
182

References request().

Here is the call graph for this function:

◆ patch()

SXNG_Response searx.network.patch ( str url,
dict[str, t.Any] | None data = None,
**t.Any kwargs )

Definition at line 196 of file __init__.py.

196def patch(url: str, data: dict[str, t.Any] | None = None, **kwargs: t.Any) -> SXNG_Response:
197 return request('patch', url, data=data, **kwargs)
198
199

References request().

Here is the call graph for this function:

◆ post()

SXNG_Response searx.network.post ( str url,
dict[str, t.Any] | None data = None,
**t.Any kwargs )

Definition at line 188 of file __init__.py.

188def post(url: str, data: dict[str, t.Any] | None = None, **kwargs: t.Any) -> SXNG_Response:
189 return request('post', url, data=data, **kwargs)
190
191

References request().

Here is the call graph for this function:

◆ put()

SXNG_Response searx.network.put ( str url,
dict[str, t.Any] | None data = None,
**t.Any kwargs )

Definition at line 192 of file __init__.py.

192def put(url: str, data: dict[str, t.Any] | None = None, **kwargs: t.Any) -> SXNG_Response:
193 return request('put', url, data=data, **kwargs)
194
195

References request().

Here is the call graph for this function:

◆ request()

SXNG_Response searx.network.request ( str method,
str url,
**t.Any kwargs )
same as requests/requests/api.py request(...)

Definition at line 96 of file __init__.py.

96def request(method: str, url: str, **kwargs: t.Any) -> SXNG_Response:
97 """same as requests/requests/api.py request(...)"""
98 with _record_http_time() as start_time:
99 network = get_context_network()
100 timeout = _get_timeout(start_time, kwargs)
101 future = asyncio.run_coroutine_threadsafe(
102 network.request(method, url, **kwargs),
103 get_loop(),
104 )
105 try:
106 return future.result(timeout)
107 except concurrent.futures.TimeoutError as e:
108 raise httpx.TimeoutException('Timeout', request=None) from e
109
110

References _get_timeout(), _record_http_time(), and get_context_network().

Referenced by delete(), get(), head(), options(), patch(), post(), and put().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ reset_time_for_thread()

searx.network.reset_time_for_thread ( )

Definition at line 32 of file __init__.py.

32def reset_time_for_thread():
33 THREADLOCAL.total_time = 0
34
35

Referenced by searx.search.processors.online.OnlineProcessor.init_network_in_thread().

Here is the caller graph for this function:

◆ set_context_network_name()

searx.network.set_context_network_name ( str network_name)

Definition at line 46 of file __init__.py.

46def set_context_network_name(network_name: str):
47 THREADLOCAL.network = get_network(network_name)
48
49

Referenced by searx.search.processors.online.OnlineProcessor.init_network_in_thread().

Here is the caller graph for this function:

◆ set_timeout_for_thread()

searx.network.set_timeout_for_thread ( float timeout,
float | None start_time = None )

Definition at line 41 of file __init__.py.

41def set_timeout_for_thread(timeout: float, start_time: float | None = None):
42 THREADLOCAL.timeout = timeout
43 THREADLOCAL.start_time = start_time
44
45

Referenced by searx.search.processors.online.OnlineProcessor.init_network_in_thread().

Here is the caller graph for this function:

◆ stream()

tuple[SXNG_Response, Iterable[bytes]] searx.network.stream ( str method,
str url,
**t.Any kwargs )
Replace httpx.stream.

Usage:
response, stream = poolrequests.stream(...)
for chunk in stream:
    ...

httpx.Client.stream requires to write the httpx.HTTPTransport version of the
the httpx.AsyncHTTPTransport declared above.

Definition at line 255 of file __init__.py.

255def stream(method: str, url: str, **kwargs: t.Any) -> tuple[SXNG_Response, Iterable[bytes]]:
256 """Replace httpx.stream.
257
258 Usage:
259 response, stream = poolrequests.stream(...)
260 for chunk in stream:
261 ...
262
263 httpx.Client.stream requires to write the httpx.HTTPTransport version of the
264 the httpx.AsyncHTTPTransport declared above.
265 """
266 generator = _stream_generator(method, url, **kwargs)
267
268 # yield response
269 response = next(generator) # pylint: disable=stop-iteration-return
270 if isinstance(response, Exception):
271 raise response
272
273 response._generator = generator # pylint: disable=protected-access
274 response.close = MethodType(_close_response_method, response)
275
276 return response, generator

References _stream_generator().

Here is the call graph for this function:

◆ stream_chunk_to_queue()

searx.network.stream_chunk_to_queue ( network,
queue,
str method,
str url,
**t.Any kwargs )

Definition at line 204 of file __init__.py.

204async def stream_chunk_to_queue(network, queue, method: str, url: str, **kwargs: t.Any):
205 try:
206 async with await network.stream(method, url, **kwargs) as response:
207 queue.put(response)
208 # aiter_raw: access the raw bytes on the response without applying any HTTP content decoding
209 # https://www.python-httpx.org/quickstart/#streaming-responses
210 async for chunk in response.aiter_raw(65536):
211 if len(chunk) > 0:
212 queue.put(chunk)
213 except (httpx.StreamClosed, anyio.ClosedResourceError):
214 # the response was queued before the exception.
215 # the exception was raised on aiter_raw.
216 # we do nothing here: in the finally block, None will be queued
217 # so stream(method, url, **kwargs) generator can stop
218 pass
219 except Exception as e: # pylint: disable=broad-except
220 # broad except to avoid this scenario:
221 # exception in network.stream(method, url, **kwargs)
222 # -> the exception is not catch here
223 # -> queue None (in finally)
224 # -> the function below steam(method, url, **kwargs) has nothing to return
225 queue.put(e)
226 finally:
227 queue.put(None)
228
229

Referenced by _stream_generator().

Here is the caller graph for this function:

Variable Documentation

◆ __all__

list searx.network.__all__ = ["get_network", "initialize", "check_network_configuration", "raise_for_httperror"]
private

Definition at line 4 of file __init__.py.

◆ _generator

# pylint disable searx.network._generator = protected-access
protected

Definition at line 251 of file __init__.py.

◆ THREADLOCAL

searx.network.THREADLOCAL = threading.local()

Definition at line 28 of file __init__.py.