.oO SearXNG Developer Documentation Oo.
Loading...
Searching...
No Matches
searx.cache.ExpireCacheSQLite Class Reference
Inheritance diagram for searx.cache.ExpireCacheSQLite:
Collaboration diagram for searx.cache.ExpireCacheSQLite:

Public Member Functions

 __init__ (self, ExpireCacheCfg cfg)
bool init (self, sqlite3.Connection conn)
bool maintenance (self, bool force=False, bool truncate=False)
bool create_table (self, str table)
list[str] table_names (self)
 truncate_tables (self, list[str] table_names)
int next_maintenance_time (self)
bool set (self, str key, typing.Any value, int|None expire, str|None ctx=None)
typing.Any get (self, str key, typing.Any default=None, str|None ctx=None)
Iterator[tuple[str, typing.Any]] pairs (self, str ctx)
ExpireCacheStats state (self)
Public Member Functions inherited from searx.sqlitedb.SQLiteAppl
 __init__ (self, str db_url)
sqlite3.Connection connect (self)
 register_functions (self, sqlite3.Connection conn)
sqlite3.Connection DB (self)
 create_schema (self, sqlite3.Connection conn)
Public Member Functions inherited from searx.cache.ExpireCache
bytes serialize (self, typing.Any value)
typing.Any deserialize (self, bytes value)
str secret_hash (self, str|bytes name)

Public Attributes

ExpireCacheCfg cfg = cfg
 next_maintenance_time
 table_names
Public Attributes inherited from searx.sqlitedb.SQLiteAppl
str db_url = db_url
SQLiteProperties properties = SQLiteProperties(db_url)

Static Public Attributes

str CACHE_TABLE_PREFIX = "CACHE-TABLE"
Static Public Attributes inherited from searx.sqlitedb.SQLiteAppl
dict DDL_CREATE_TABLES = {}
int DB_SCHEMA = 1
dict SQLITE_THREADING_MODE
str SQLITE_JOURNAL_MODE = "WAL"
dict SQLITE_CONNECT_ARGS
Static Public Attributes inherited from searx.cache.ExpireCache
str hash_token = "hash_token"

Additional Inherited Members

Static Public Member Functions inherited from searx.cache.ExpireCache
"ExpireCacheSQLite" build_cache (ExpireCacheCfg cfg)
str normalize_name (str name)
Protected Member Functions inherited from searx.sqlitedb.SQLiteAppl
 _compatibility (self)
sqlite3.Connection _connect (self)
Protected Attributes inherited from searx.sqlitedb.SQLiteAppl
bool _init_done = False
sqlite3.Connection|None _DB = None

Detailed Description

Cache that manages key/value pairs in a SQLite DB.  The DB model in the
SQLite DB is implemented in abstract class :py:obj:`SQLiteAppl
<searx.sqlitedb.SQLiteAppl>`.

The following configurations are required / supported:

- :py:obj:`ExpireCacheCfg.db_url`
- :py:obj:`ExpireCacheCfg.MAXHOLD_TIME`
- :py:obj:`ExpireCacheCfg.MAINTENANCE_PERIOD`
- :py:obj:`ExpireCacheCfg.MAINTENANCE_MODE`

Definition at line 210 of file cache.py.

Constructor & Destructor Documentation

◆ __init__()

searx.cache.ExpireCacheSQLite.__init__ ( self,
ExpireCacheCfg cfg )
An instance of the SQLite expire cache is build up from a
:py:obj:`config <ExpireCacheCfg>`.

Definition at line 230 of file cache.py.

230 def __init__(self, cfg: ExpireCacheCfg):
231 """An instance of the SQLite expire cache is build up from a
232 :py:obj:`config <ExpireCacheCfg>`."""
233
234 self.cfg: ExpireCacheCfg = cfg
235 if cfg.db_url == ":memory:":
236 log.critical("don't use SQLite DB in :memory: in production!!")
237 super().__init__(cfg.db_url)
238

Member Function Documentation

◆ create_table()

bool searx.cache.ExpireCacheSQLite.create_table ( self,
str table )
Create DB ``table`` if it has not yet been created, no recreates are
initiated if the table already exists.

Definition at line 284 of file cache.py.

284 def create_table(self, table: str) -> bool:
285 """Create DB ``table`` if it has not yet been created, no recreates are
286 initiated if the table already exists.
287 """
288 if table in self.table_names:
289 # log.debug("key/value table %s exists in DB (no need to recreate)", table)
290 return False
291
292 log.info("key/value table '%s' NOT exists in DB -> create DB table ..", table)
293 sql_table = "\n".join(
294 [
295 f"CREATE TABLE IF NOT EXISTS {table} (",
296 " key TEXT,",
297 " value BLOB,",
298 f" expire INTEGER DEFAULT (strftime('%s', 'now') + {self.cfg.MAXHOLD_TIME}),",
299 "PRIMARY KEY (key))",
300 ]
301 )
302 sql_index = f"CREATE INDEX IF NOT EXISTS index_expire_{table} ON {table}(expire);"
303 with self.connect() as conn:
304 conn.execute(sql_table)
305 conn.execute(sql_index)
306 conn.close()
307
308 self.properties.set(f"{self.CACHE_TABLE_PREFIX}-{table}", table)
309 return True
310

References searx.sqlitedb.SQLiteAppl.connect(), searx.sqlitedb.SQLiteAppl.properties, set(), and table_names.

Referenced by set().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ get()

typing.Any searx.cache.ExpireCacheSQLite.get ( self,
str key,
typing.Any default = None,
str | None ctx = None )
Get value of ``key`` from table given by argument ``ctx``.  If
``ctx`` argument is ``None`` (the default), a table name is generated
from the :py:obj:`ExpireCacheCfg.name`.  If ``key`` not exists (in
table), the ``default`` value is returned.

Reimplemented from searx.cache.ExpireCache.

Definition at line 375 of file cache.py.

375 def get(self, key: str, default: typing.Any = None, ctx: str | None = None) -> typing.Any:
376 """Get value of ``key`` from table given by argument ``ctx``. If
377 ``ctx`` argument is ``None`` (the default), a table name is generated
378 from the :py:obj:`ExpireCacheCfg.name`. If ``key`` not exists (in
379 table), the ``default`` value is returned.
380
381 """
382 table = ctx
383 self.maintenance()
384
385 if not table:
386 table = self.normalize_name(self.cfg.name)
387
388 if table not in self.table_names:
389 return default
390
391 sql = f"SELECT value FROM {table} WHERE key = ?"
392 row = self.DB.execute(sql, (key,)).fetchone()
393 if row is None:
394 return default
395
396 return self.deserialize(row[0])
397

References searx.botdetection.config.Config.cfg, cfg, DB, searx.cache.ExpireCache.deserialize(), searx.cache.ExpireCache.maintenance(), searx.cache.ExpireCache.normalize_name(), and table_names.

Referenced by searx.result_types._base.LegacyResult.__init__(), searx.result_types._base.LegacyResult.defaults_from(), and searx.answerers._core.AnswerStorage.register().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ init()

bool searx.cache.ExpireCacheSQLite.init ( self,
sqlite3.Connection conn )
Initializes the DB schema and properties, is only executed once even
if called several times.

If the initialization has not yet taken place, it is carried out and a
`True` is returned to the caller at the end.  If the initialization has
already been carried out in the past, `False` is returned.

Reimplemented from searx.sqlitedb.SQLiteAppl.

Definition at line 239 of file cache.py.

239 def init(self, conn: sqlite3.Connection) -> bool:
240 ret_val = super().init(conn)
241 if not ret_val:
242 return False
243
244 new = hashlib.sha256(self.cfg.password).hexdigest()
245 old = self.properties(self.hash_token)
246 if old != new:
247 if old is not None:
248 log.warning("[%s] hash token changed: truncate all cache tables", self.cfg.name)
249 self.maintenance(force=True, truncate=True)
250 self.properties.set(self.hash_token, new)
251
252 return True
253

References searx.botdetection.config.Config.cfg, cfg, searx.cache.ExpireCache.hash_token, init(), searx.cache.ExpireCache.maintenance(), and searx.sqlitedb.SQLiteAppl.properties.

Referenced by searx.sqlitedb.SQLiteAppl.connect(), searx.sqlitedb.SQLiteAppl.DB(), init(), searx.data.currencies.CurrenciesDB.iso4217_to_name(), searx.data.currencies.CurrenciesDB.name_to_iso4217(), and searx.data.tracker_patterns.TrackerPatternsDB.rules().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ maintenance()

bool searx.cache.ExpireCacheSQLite.maintenance ( self,
bool force = False,
bool truncate = False )
Performs maintenance on the cache.

``force``:
  Maintenance should be carried out even if the maintenance interval has
  not yet been reached.

``truncate``:
  Truncate the entire cache, which is necessary, for example, if the
  password has changed.

Reimplemented from searx.cache.ExpireCache.

Definition at line 254 of file cache.py.

254 def maintenance(self, force: bool = False, truncate: bool = False) -> bool:
255
256 if not force and int(time.time()) < self.next_maintenance_time:
257 # log.debug("no maintenance required yet, next maintenance interval is in the future")
258 return False
259
260 # Prevent parallel DB maintenance cycles from other DB connections
261 # (e.g. in multi thread or process environments).
262 self.properties.set("LAST_MAINTENANCE", "") # hint: this (also) sets the m_time of the property!
263
264 if truncate:
265 self.truncate_tables(self.table_names)
266 return True
267
268 # drop items by expire time stamp ..
269 expire = int(time.time())
270
271 with self.connect() as conn:
272 for table in self.table_names:
273 res = conn.execute(f"DELETE FROM {table} WHERE expire < ?", (expire,))
274 log.debug("deleted %s keys from table %s (expire date reached)", res.rowcount, table)
275
276 # Vacuuming the WALs
277 # https://www.theunterminatedstring.com/sqlite-vacuuming/
278
279 conn.execute("PRAGMA wal_checkpoint(TRUNCATE)")
280 conn.close()
281
282 return True
283

Referenced by searx.favicons.cache.FaviconCacheSQLite.set().

Here is the caller graph for this function:

◆ next_maintenance_time()

int searx.cache.ExpireCacheSQLite.next_maintenance_time ( self)
Returns (unix epoch) time of the next maintenance.

Definition at line 327 of file cache.py.

327 def next_maintenance_time(self) -> int:
328 """Returns (unix epoch) time of the next maintenance."""
329
330 return self.cfg.MAINTENANCE_PERIOD + self.properties.m_time("LAST_MAINTENANCE", int(time.time()))
331

References searx.botdetection.config.Config.cfg, cfg, and searx.sqlitedb.SQLiteAppl.properties.

◆ pairs()

Iterator[tuple[str, typing.Any]] searx.cache.ExpireCacheSQLite.pairs ( self,
str ctx )
Iterate over key/value pairs from table given by argument ``ctx``.
If ``ctx`` argument is ``None`` (the default), a table name is
generated from the :py:obj:`ExpireCacheCfg.name`.

Definition at line 398 of file cache.py.

398 def pairs(self, ctx: str) -> Iterator[tuple[str, typing.Any]]:
399 """Iterate over key/value pairs from table given by argument ``ctx``.
400 If ``ctx`` argument is ``None`` (the default), a table name is
401 generated from the :py:obj:`ExpireCacheCfg.name`."""
402 table = ctx
403 self.maintenance()
404
405 if not table:
406 table = self.normalize_name(self.cfg.name)
407
408 if table in self.table_names:
409 for row in self.DB.execute(f"SELECT key, value FROM {table}"):
410 yield row[0], self.deserialize(row[1])
411

References searx.botdetection.config.Config.cfg, cfg, DB, searx.cache.ExpireCache.deserialize(), searx.cache.ExpireCache.maintenance(), searx.cache.ExpireCache.normalize_name(), and table_names.

Here is the call graph for this function:

◆ set()

bool searx.cache.ExpireCacheSQLite.set ( self,
str key,
typing.Any value,
int | None expire,
str | None ctx = None )
Set key/value in DB table given by argument ``ctx``.  If expire is
unset the default is taken from :py:obj:`ExpireCacheCfg.MAXHOLD_TIME`.
If ``ctx`` argument is ``None`` (the default), a table name is
generated from the :py:obj:`ExpireCacheCfg.name`.  If DB table does not
exists, it will be created (on demand) by :py:obj:`self.create_table
<ExpireCacheSQLite.create_table>`.

Reimplemented from searx.cache.ExpireCache.

Definition at line 334 of file cache.py.

334 def set(self, key: str, value: typing.Any, expire: int | None, ctx: str | None = None) -> bool:
335 """Set key/value in DB table given by argument ``ctx``. If expire is
336 unset the default is taken from :py:obj:`ExpireCacheCfg.MAXHOLD_TIME`.
337 If ``ctx`` argument is ``None`` (the default), a table name is
338 generated from the :py:obj:`ExpireCacheCfg.name`. If DB table does not
339 exists, it will be created (on demand) by :py:obj:`self.create_table
340 <ExpireCacheSQLite.create_table>`.
341 """
342 table = ctx
343 self.maintenance()
344
345 value = self.serialize(value=value)
346 if len(value) > self.cfg.MAX_VALUE_LEN:
347 log.warning("ExpireCache.set(): %s.key='%s' - value too big to cache (len: %s) ", table, value, len(value))
348 return False
349
350 if not expire:
351 expire = self.cfg.MAXHOLD_TIME
352 expire = int(time.time()) + expire
353
354 table_name = table
355 if not table_name:
356 table_name = self.normalize_name(self.cfg.name)
357 self.create_table(table_name)
358
359 sql = (
360 f"INSERT INTO {table_name} (key, value, expire) VALUES (?, ?, ?)"
361 f" ON CONFLICT DO "
362 f"UPDATE SET value=?, expire=?"
363 )
364
365 if table:
366 with self.DB:
367 self.DB.execute(sql, (key, value, expire, value, expire))
368 else:
369 with self.connect() as conn:
370 conn.execute(sql, (key, value, expire, value, expire))
371 conn.close()
372
373 return True
374

References searx.botdetection.config.Config.cfg, cfg, create_table(), searx.cache.ExpireCache.maintenance(), searx.cache.ExpireCache.normalize_name(), and searx.cache.ExpireCache.serialize().

Referenced by create_table().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ state()

ExpireCacheStats searx.cache.ExpireCacheSQLite.state ( self)
Returns a :py:obj:`ExpireCacheStats`, which provides information
about the status of the cache.

Reimplemented from searx.cache.ExpireCache.

Definition at line 412 of file cache.py.

412 def state(self) -> ExpireCacheStats:
413 cached_items: dict[str, list[tuple[str, typing.Any, int]]] = {}
414 for table in self.table_names:
415 cached_items[table] = []
416 for row in self.DB.execute(f"SELECT key, value, expire FROM {table}"):
417 cached_items[table].append((row[0], self.deserialize(row[1]), row[2]))
418 return ExpireCacheStats(cached_items=cached_items)

References DB, searx.cache.ExpireCache.deserialize(), and table_names.

Here is the call graph for this function:

◆ table_names()

list[str] searx.cache.ExpireCacheSQLite.table_names ( self)
List of key/value tables already created in the DB.

Definition at line 312 of file cache.py.

312 def table_names(self) -> list[str]:
313 """List of key/value tables already created in the DB."""
314 sql = f"SELECT value FROM properties WHERE name LIKE '{self.CACHE_TABLE_PREFIX}%%'"
315 rows = self.DB.execute(sql).fetchall() or []
316 return [r[0] for r in rows]
317

References DB.

◆ truncate_tables()

searx.cache.ExpireCacheSQLite.truncate_tables ( self,
list[str] table_names )

Definition at line 318 of file cache.py.

318 def truncate_tables(self, table_names: list[str]):
319 log.debug("truncate table: %s", ",".join(table_names))
320 with self.connect() as conn:
321 for table in table_names:
322 conn.execute(f"DELETE FROM {table}")
323 conn.close()
324 return True
325

References searx.sqlitedb.SQLiteAppl.connect().

Here is the call graph for this function:

Member Data Documentation

◆ CACHE_TABLE_PREFIX

str searx.cache.ExpireCacheSQLite.CACHE_TABLE_PREFIX = "CACHE-TABLE"
static

Definition at line 228 of file cache.py.

◆ cfg

◆ next_maintenance_time

searx.cache.ExpireCacheSQLite.next_maintenance_time

Definition at line 256 of file cache.py.

Referenced by searx.favicons.cache.FaviconCacheSQLite.set().

◆ table_names

searx.cache.ExpireCacheSQLite.table_names

Definition at line 265 of file cache.py.

Referenced by create_table(), get(), pairs(), and state().


The documentation for this class was generated from the following file:
  • /home/andrew/Documents/code/public/searxng/searx/cache.py