.oO SearXNG Developer Documentation Oo.
Loading...
Searching...
No Matches
Functions | Variables
searx.redislib Namespace Reference

Functions

 lua_script_storage (client, script)
 
 purge_by_prefix (client, str prefix="SearXNG_")
 
 secret_hash (str name)
 
 incr_counter (client, str name, int limit=0, int expire=0)
 
 drop_counter (client, name)
 
 incr_sliding_window (client, str name, int duration)
 

Variables

dict LUA_SCRIPT_STORAGE = {}
 
str PURGE_BY_PREFIX
 
str INCR_COUNTER
 
str INCR_SLIDING_WINDOW
 

Detailed Description

A collection of convenient functions and redis/lua scripts.

This code was partial inspired by the `Bullet-Proofing Lua Scripts in RedisPy`_
article.

.. _Bullet-Proofing Lua Scripts in RedisPy:
   https://redis.com/blog/bullet-proofing-lua-scripts-in-redispy/

Function Documentation

◆ drop_counter()

searx.redislib.drop_counter ( client,
name ) 
Drop counter with redis key ``SearXNG_counter_<name>``

The replacement ``<name>`` is a *secret hash* of the value from argument
``name`` (see :py:func:`incr_counter` and :py:func:`incr_sliding_window`).

Definition at line 159 of file redislib.py.

159def drop_counter(client, name):
160 """Drop counter with redis key ``SearXNG_counter_<name>``
161
162 The replacement ``<name>`` is a *secret hash* of the value from argument
163 ``name`` (see :py:func:`incr_counter` and :py:func:`incr_sliding_window`).
164 """
165 name = "SearXNG_counter_" + secret_hash(name)
166 client.delete(name)
167
168

◆ incr_counter()

searx.redislib.incr_counter ( client,
str name,
int limit = 0,
int expire = 0 ) 
Increment a counter and return the new value.

If counter with redis key ``SearXNG_counter_<name>`` does not exists it is
created with initial value 1 returned.  The replacement ``<name>`` is a
*secret hash* of the value from argument ``name`` (see
:py:func:`secret_hash`).

The implementation of the redis counter is the lua script from string
:py:obj:`INCR_COUNTER`.

:param name: name of the counter
:type name: str

:param expire: live-time of the counter in seconds (default ``None`` means
  infinite).
:type expire: int / see EXPIRE_

:param limit: limit where the counter stops to increment (default ``None``)
:type limit: int / limit is 2^64 see INCR_

:return: value of the incremented counter
:type return: int

.. _EXPIRE: https://redis.io/commands/expire/
.. _INCR: https://redis.io/commands/incr/

A simple demo of a counter with expire time and limit::

  >>> for i in range(6):
  ...   i, incr_counter(client, "foo", 3, 5) # max 3, duration 5 sec
  ...   time.sleep(1) # from the third call on max has been reached
  ...
  (0, 1)
  (1, 2)
  (2, 3)
  (3, 3)
  (4, 3)
  (5, 1)

Definition at line 112 of file redislib.py.

112def incr_counter(client, name: str, limit: int = 0, expire: int = 0):
113 """Increment a counter and return the new value.
114
115 If counter with redis key ``SearXNG_counter_<name>`` does not exists it is
116 created with initial value 1 returned. The replacement ``<name>`` is a
117 *secret hash* of the value from argument ``name`` (see
118 :py:func:`secret_hash`).
119
120 The implementation of the redis counter is the lua script from string
121 :py:obj:`INCR_COUNTER`.
122
123 :param name: name of the counter
124 :type name: str
125
126 :param expire: live-time of the counter in seconds (default ``None`` means
127 infinite).
128 :type expire: int / see EXPIRE_
129
130 :param limit: limit where the counter stops to increment (default ``None``)
131 :type limit: int / limit is 2^64 see INCR_
132
133 :return: value of the incremented counter
134 :type return: int
135
136 .. _EXPIRE: https://redis.io/commands/expire/
137 .. _INCR: https://redis.io/commands/incr/
138
139 A simple demo of a counter with expire time and limit::
140
141 >>> for i in range(6):
142 ... i, incr_counter(client, "foo", 3, 5) # max 3, duration 5 sec
143 ... time.sleep(1) # from the third call on max has been reached
144 ...
145 (0, 1)
146 (1, 2)
147 (2, 3)
148 (3, 3)
149 (4, 3)
150 (5, 1)
151
152 """
153 script = lua_script_storage(client, INCR_COUNTER)
154 name = "SearXNG_counter_" + secret_hash(name)
155 c = script(args=[limit, expire], keys=[name])
156 return c
157
158

◆ incr_sliding_window()

searx.redislib.incr_sliding_window ( client,
str name,
int duration ) 
Increment a sliding-window counter and return the new value.

If counter with redis key ``SearXNG_counter_<name>`` does not exists it is
created with initial value 1 returned.  The replacement ``<name>`` is a
*secret hash* of the value from argument ``name`` (see
:py:func:`secret_hash`).

:param name: name of the counter
:type name: str

:param duration: live-time of the sliding window in seconds
:typeduration: int

:return: value of the incremented counter
:type return: int

The implementation of the redis counter is the lua script from string
:py:obj:`INCR_SLIDING_WINDOW`.  The lua script uses `sorted sets in Redis`_
to implement a sliding window for the redis key ``SearXNG_counter_<name>``
(ZADD_).  The current TIME_ is used to score the items in the sorted set and
the time window is moved by removing items with a score lower current time
minus *duration* time (ZREMRANGEBYSCORE_).

The EXPIRE_ time (the duration of the sliding window) is refreshed on each
call (increment) and if there is no call in this duration, the sorted
set expires from the redis DB.

The return value is the amount of items in the sorted set (ZCOUNT_), what
means the number of calls in the sliding window.

.. _Sorted sets in Redis:
   https://redis.com/ebook/part-1-getting-started/chapter-1-getting-to-know-redis/1-2-what-redis-data-structures-look-like/1-2-5-sorted-sets-in-redis/
.. _TIME: https://redis.io/commands/time/
.. _ZADD: https://redis.io/commands/zadd/
.. _EXPIRE: https://redis.io/commands/expire/
.. _ZREMRANGEBYSCORE: https://redis.io/commands/zremrangebyscore/
.. _ZCOUNT: https://redis.io/commands/zcount/

A simple demo of the sliding window::

  >>> for i in range(5):
  ...   incr_sliding_window(client, "foo", 3) # duration 3 sec
  ...   time.sleep(1) # from the third call (second) on the window is moved
  ...
  1
  2
  3
  3
  3
  >>> time.sleep(3)  # wait until expire
  >>> incr_sliding_window(client, "foo", 3)
  1

Definition at line 182 of file redislib.py.

182def incr_sliding_window(client, name: str, duration: int):
183 """Increment a sliding-window counter and return the new value.
184
185 If counter with redis key ``SearXNG_counter_<name>`` does not exists it is
186 created with initial value 1 returned. The replacement ``<name>`` is a
187 *secret hash* of the value from argument ``name`` (see
188 :py:func:`secret_hash`).
189
190 :param name: name of the counter
191 :type name: str
192
193 :param duration: live-time of the sliding window in seconds
194 :typeduration: int
195
196 :return: value of the incremented counter
197 :type return: int
198
199 The implementation of the redis counter is the lua script from string
200 :py:obj:`INCR_SLIDING_WINDOW`. The lua script uses `sorted sets in Redis`_
201 to implement a sliding window for the redis key ``SearXNG_counter_<name>``
202 (ZADD_). The current TIME_ is used to score the items in the sorted set and
203 the time window is moved by removing items with a score lower current time
204 minus *duration* time (ZREMRANGEBYSCORE_).
205
206 The EXPIRE_ time (the duration of the sliding window) is refreshed on each
207 call (increment) and if there is no call in this duration, the sorted
208 set expires from the redis DB.
209
210 The return value is the amount of items in the sorted set (ZCOUNT_), what
211 means the number of calls in the sliding window.
212
213 .. _Sorted sets in Redis:
214 https://redis.com/ebook/part-1-getting-started/chapter-1-getting-to-know-redis/1-2-what-redis-data-structures-look-like/1-2-5-sorted-sets-in-redis/
215 .. _TIME: https://redis.io/commands/time/
216 .. _ZADD: https://redis.io/commands/zadd/
217 .. _EXPIRE: https://redis.io/commands/expire/
218 .. _ZREMRANGEBYSCORE: https://redis.io/commands/zremrangebyscore/
219 .. _ZCOUNT: https://redis.io/commands/zcount/
220
221 A simple demo of the sliding window::
222
223 >>> for i in range(5):
224 ... incr_sliding_window(client, "foo", 3) # duration 3 sec
225 ... time.sleep(1) # from the third call (second) on the window is moved
226 ...
227 1
228 2
229 3
230 3
231 3
232 >>> time.sleep(3) # wait until expire
233 >>> incr_sliding_window(client, "foo", 3)
234 1
235
236 """
237 script = lua_script_storage(client, INCR_SLIDING_WINDOW)
238 name = "SearXNG_counter_" + secret_hash(name)
239 c = script(args=[duration], keys=[name])
240 return c

◆ lua_script_storage()

searx.redislib.lua_script_storage ( client,
script ) 
Returns a redis :py:obj:`Script
<redis.commands.core.CoreCommands.register_script>` instance.

Due to performance reason the ``Script`` object is instantiated only once
for a client (``client.register_script(..)``) and is cached in
:py:obj:`LUA_SCRIPT_STORAGE`.

Definition at line 21 of file redislib.py.

21def lua_script_storage(client, script):
22 """Returns a redis :py:obj:`Script
23 <redis.commands.core.CoreCommands.register_script>` instance.
24
25 Due to performance reason the ``Script`` object is instantiated only once
26 for a client (``client.register_script(..)``) and is cached in
27 :py:obj:`LUA_SCRIPT_STORAGE`.
28
29 """
30
31 # redis connection can be closed, lets use the id() of the redis connector
32 # as key in the script-storage:
33 client_id = id(client)
34
35 if LUA_SCRIPT_STORAGE.get(client_id) is None:
36 LUA_SCRIPT_STORAGE[client_id] = {}
37
38 if LUA_SCRIPT_STORAGE[client_id].get(script) is None:
39 LUA_SCRIPT_STORAGE[client_id][script] = client.register_script(script)
40
41 return LUA_SCRIPT_STORAGE[client_id][script]
42
43

◆ purge_by_prefix()

searx.redislib.purge_by_prefix ( client,
str prefix = "SearXNG_" ) 
Purge all keys with ``prefix`` from database.

Queries all keys in the database by the given prefix and set expire time to
zero.  The default prefix will drop all keys which has been set by SearXNG
(drops SearXNG schema entirely from database).

The implementation is the lua script from string :py:obj:`PURGE_BY_PREFIX`.
The lua script uses EXPIRE_ instead of DEL_: if there are a lot keys to
delete and/or their values are big, `DEL` could take more time and blocks
the command loop while `EXPIRE` turns back immediate.

:param prefix: prefix of the key to delete (default: ``SearXNG_``)
:type name: str

.. _EXPIRE: https://redis.io/commands/expire/
.. _DEL: https://redis.io/commands/del/

Definition at line 52 of file redislib.py.

52def purge_by_prefix(client, prefix: str = "SearXNG_"):
53 """Purge all keys with ``prefix`` from database.
54
55 Queries all keys in the database by the given prefix and set expire time to
56 zero. The default prefix will drop all keys which has been set by SearXNG
57 (drops SearXNG schema entirely from database).
58
59 The implementation is the lua script from string :py:obj:`PURGE_BY_PREFIX`.
60 The lua script uses EXPIRE_ instead of DEL_: if there are a lot keys to
61 delete and/or their values are big, `DEL` could take more time and blocks
62 the command loop while `EXPIRE` turns back immediate.
63
64 :param prefix: prefix of the key to delete (default: ``SearXNG_``)
65 :type name: str
66
67 .. _EXPIRE: https://redis.io/commands/expire/
68 .. _DEL: https://redis.io/commands/del/
69
70 """
71 script = lua_script_storage(client, PURGE_BY_PREFIX)
72 script(args=[prefix])
73
74

◆ secret_hash()

searx.redislib.secret_hash ( str name) 
Creates a hash of the ``name``.

Combines argument ``name`` with the ``secret_key`` from :ref:`settings
server`.  This function can be used to get a more anonymized name of a Redis
KEY.

:param name: the name to create a secret hash for
:type name: str

Definition at line 75 of file redislib.py.

75def secret_hash(name: str):
76 """Creates a hash of the ``name``.
77
78 Combines argument ``name`` with the ``secret_key`` from :ref:`settings
79 server`. This function can be used to get a more anonymized name of a Redis
80 KEY.
81
82 :param name: the name to create a secret hash for
83 :type name: str
84 """
85 m = hmac.new(bytes(name, encoding='utf-8'), digestmod='sha256')
86 m.update(bytes(get_setting('server.secret_key'), encoding='utf-8'))
87 return m.hexdigest()
88
89

Variable Documentation

◆ INCR_COUNTER

str searx.redislib.INCR_COUNTER
Initial value:
1= """
2local limit = tonumber(ARGV[1])
3local expire = tonumber(ARGV[2])
4local c_name = KEYS[1]
5
6local c = redis.call('GET', c_name)
7
8if not c then
9 c = redis.call('INCR', c_name)
10 if expire > 0 then
11 redis.call('EXPIRE', c_name, expire)
12 end
13else
14 c = tonumber(c)
15 if limit == 0 or c < limit then
16 c = redis.call('INCR', c_name)
17 end
18end
19return c
20"""

Definition at line 90 of file redislib.py.

◆ INCR_SLIDING_WINDOW

str searx.redislib.INCR_SLIDING_WINDOW
Initial value:
1= """
2local expire = tonumber(ARGV[1])
3local name = KEYS[1]
4local current_time = redis.call('TIME')
5
6redis.call('ZREMRANGEBYSCORE', name, 0, current_time[1] - expire)
7redis.call('ZADD', name, current_time[1], current_time[1] .. current_time[2])
8local result = redis.call('ZCOUNT', name, 0, current_time[1] + 1)
9redis.call('EXPIRE', name, expire)
10return result
11"""

Definition at line 169 of file redislib.py.

◆ LUA_SCRIPT_STORAGE

dict searx.redislib.LUA_SCRIPT_STORAGE = {}

Definition at line 16 of file redislib.py.

◆ PURGE_BY_PREFIX

str searx.redislib.PURGE_BY_PREFIX
Initial value:
1= """
2local prefix = tostring(ARGV[1])
3for i, name in ipairs(redis.call('KEYS', prefix .. '*')) do
4 redis.call('EXPIRE', name, 0)
5end
6"""

Definition at line 44 of file redislib.py.