.oO SearXNG Developer Documentation Oo.
Loading...
Searching...
No Matches
searx.cache.ExpireCacheSQLite Class Reference
Inheritance diagram for searx.cache.ExpireCacheSQLite:
Collaboration diagram for searx.cache.ExpireCacheSQLite:

Public Member Functions

 __init__ (self, ExpireCacheCfg cfg)
bool init (self, sqlite3.Connection conn)
bool maintenance (self, bool force=False, bool truncate=False)
bool create_table (self, str table)
list[str] table_names (self)
 truncate_tables (self, list[str] table_names)
int next_maintenance_time (self)
bool set (self, str key, typing.Any value, int|None expire, str|None ctx=None)
typing.Any get (self, str key, default=None, str|None ctx=None)
Iterator[tuple[str, typing.Any]] pairs (self, str ctx)
ExpireCacheStats state (self)
Public Member Functions inherited from searx.sqlitedb.SQLiteAppl
 __init__ (self, db_url)
sqlite3.Connection connect (self)
 register_functions (self, conn)
sqlite3.Connection DB (self)
 create_schema (self, sqlite3.Connection conn)
Public Member Functions inherited from searx.cache.ExpireCache
bytes serialize (self, typing.Any value)
typing.Any deserialize (self, bytes value)
str secret_hash (self, str|bytes name)

Public Attributes

 cfg = cfg
 next_maintenance_time
 table_names
Public Attributes inherited from searx.sqlitedb.SQLiteAppl
 db_url = db_url
 properties = SQLiteProperties(db_url)

Static Public Attributes

str CACHE_TABLE_PREFIX = "CACHE-TABLE"
Static Public Attributes inherited from searx.sqlitedb.SQLiteAppl
dict DDL_CREATE_TABLES = {}
int DB_SCHEMA = 1
dict SQLITE_THREADING_MODE
str SQLITE_JOURNAL_MODE = "WAL"
dict SQLITE_CONNECT_ARGS
Static Public Attributes inherited from searx.cache.ExpireCache
ExpireCacheCfg hash_token = "hash_token"

Additional Inherited Members

Static Public Member Functions inherited from searx.cache.ExpireCache
ExpireCache build_cache (ExpireCacheCfg cfg)
str normalize_name (str name)
Protected Member Functions inherited from searx.sqlitedb.SQLiteAppl
 _compatibility (self)
sqlite3.Connection _connect (self)
Protected Attributes inherited from searx.sqlitedb.SQLiteAppl
bool _init_done = False
 _DB = None

Detailed Description

Cache that manages key/value pairs in a SQLite DB.  The DB model in the
SQLite DB is implemented in abstract class :py:obj:`SQLiteAppl
<searx.sqlitedb.SQLiteAppl>`.

The following configurations are required / supported:

- :py:obj:`ExpireCacheCfg.db_url`
- :py:obj:`ExpireCacheCfg.MAXHOLD_TIME`
- :py:obj:`ExpireCacheCfg.MAINTENANCE_PERIOD`
- :py:obj:`ExpireCacheCfg.MAINTENANCE_MODE`

Definition at line 212 of file cache.py.

Constructor & Destructor Documentation

◆ __init__()

searx.cache.ExpireCacheSQLite.__init__ ( self,
ExpireCacheCfg cfg )
An instance of the SQLite expire cache is build up from a
:py:obj:`config <ExpireCacheCfg>`.

Definition at line 232 of file cache.py.

232 def __init__(self, cfg: ExpireCacheCfg):
233 """An instance of the SQLite expire cache is build up from a
234 :py:obj:`config <ExpireCacheCfg>`."""
235
236 self.cfg = cfg
237 if cfg.db_url == ":memory:":
238 log.critical("don't use SQLite DB in :memory: in production!!")
239 super().__init__(cfg.db_url)
240

Member Function Documentation

◆ create_table()

bool searx.cache.ExpireCacheSQLite.create_table ( self,
str table )
Create DB ``table`` if it has not yet been created, no recreates are
initiated if the table already exists.

Definition at line 286 of file cache.py.

286 def create_table(self, table: str) -> bool:
287 """Create DB ``table`` if it has not yet been created, no recreates are
288 initiated if the table already exists.
289 """
290 if table in self.table_names:
291 # log.debug("key/value table %s exists in DB (no need to recreate)", table)
292 return False
293
294 log.info("key/value table '%s' NOT exists in DB -> create DB table ..", table)
295 sql_table = "\n".join(
296 [
297 f"CREATE TABLE IF NOT EXISTS {table} (",
298 " key TEXT,",
299 " value BLOB,",
300 f" expire INTEGER DEFAULT (strftime('%s', 'now') + {self.cfg.MAXHOLD_TIME}),",
301 "PRIMARY KEY (key))",
302 ]
303 )
304 sql_index = f"CREATE INDEX IF NOT EXISTS index_expire_{table} ON {table}(expire);"
305 with self.connect() as conn:
306 conn.execute(sql_table)
307 conn.execute(sql_index)
308 conn.close()
309
310 self.properties.set(f"{self.CACHE_TABLE_PREFIX}-{table}", table)
311 return True
312

References searx.sqlitedb.SQLiteAppl.connect(), searx.sqlitedb.SQLiteAppl.properties, set(), and table_names.

Referenced by set().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ get()

typing.Any searx.cache.ExpireCacheSQLite.get ( self,
str key,
default = None,
str | None ctx = None )
Get value of ``key`` from table given by argument ``ctx``.  If
``ctx`` argument is ``None`` (the default), a table name is generated
from the :py:obj:`ExpireCacheCfg.name`.  If ``key`` not exists (in
table), the ``default`` value is returned.

Reimplemented from searx.cache.ExpireCache.

Definition at line 377 of file cache.py.

377 def get(self, key: str, default=None, ctx: str | None = None) -> typing.Any:
378 """Get value of ``key`` from table given by argument ``ctx``. If
379 ``ctx`` argument is ``None`` (the default), a table name is generated
380 from the :py:obj:`ExpireCacheCfg.name`. If ``key`` not exists (in
381 table), the ``default`` value is returned.
382
383 """
384 table = ctx
385 self.maintenance()
386
387 if not table:
388 table = self.normalize_name(self.cfg.name)
389
390 if table not in self.table_names:
391 return default
392
393 sql = f"SELECT value FROM {table} WHERE key = ?"
394 row = self.DB.execute(sql, (key,)).fetchone()
395 if row is None:
396 return default
397
398 return self.deserialize(row[0])
399

References searx.botdetection.config.Config.cfg, cfg, DB, searx.cache.ExpireCache.deserialize(), searx.cache.ExpireCache.maintenance(), searx.cache.ExpireCache.normalize_name(), and table_names.

Referenced by searx.result_types._base.LegacyResult.__init__(), searx.result_types._base.LegacyResult.defaults_from(), and searx.answerers._core.AnswerStorage.register().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ init()

bool searx.cache.ExpireCacheSQLite.init ( self,
sqlite3.Connection conn )
Initializes the DB schema and properties, is only executed once even
if called several times.

If the initialization has not yet taken place, it is carried out and a
`True` is returned to the caller at the end.  If the initialization has
already been carried out in the past, `False` is returned.

Reimplemented from searx.sqlitedb.SQLiteAppl.

Definition at line 241 of file cache.py.

241 def init(self, conn: sqlite3.Connection) -> bool:
242 ret_val = super().init(conn)
243 if not ret_val:
244 return False
245
246 new = hashlib.sha256(self.cfg.password).hexdigest()
247 old = self.properties(self.hash_token)
248 if old != new:
249 if old is not None:
250 log.warning("[%s] hash token changed: truncate all cache tables", self.cfg.name)
251 self.maintenance(force=True, truncate=True)
252 self.properties.set(self.hash_token, new)
253
254 return True
255

References searx.botdetection.config.Config.cfg, cfg, searx.cache.ExpireCache.hash_token, init(), searx.cache.ExpireCache.maintenance(), and searx.sqlitedb.SQLiteAppl.properties.

Referenced by searx.sqlitedb.SQLiteAppl.connect(), searx.sqlitedb.SQLiteAppl.DB(), init(), searx.data.currencies.CurrenciesDB.iso4217_to_name(), searx.data.currencies.CurrenciesDB.name_to_iso4217(), and searx.data.tracker_patterns.TrackerPatternsDB.rules().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ maintenance()

bool searx.cache.ExpireCacheSQLite.maintenance ( self,
bool force = False,
bool truncate = False )
Performs maintenance on the cache.

``force``:
  Maintenance should be carried out even if the maintenance interval has
  not yet been reached.

``truncate``:
  Truncate the entire cache, which is necessary, for example, if the
  password has changed.

Reimplemented from searx.cache.ExpireCache.

Definition at line 256 of file cache.py.

256 def maintenance(self, force: bool = False, truncate: bool = False) -> bool:
257
258 if not force and int(time.time()) < self.next_maintenance_time:
259 # log.debug("no maintenance required yet, next maintenance interval is in the future")
260 return False
261
262 # Prevent parallel DB maintenance cycles from other DB connections
263 # (e.g. in multi thread or process environments).
264 self.properties.set("LAST_MAINTENANCE", "") # hint: this (also) sets the m_time of the property!
265
266 if truncate:
267 self.truncate_tables(self.table_names)
268 return True
269
270 # drop items by expire time stamp ..
271 expire = int(time.time())
272
273 with self.connect() as conn:
274 for table in self.table_names:
275 res = conn.execute(f"DELETE FROM {table} WHERE expire < ?", (expire,))
276 log.debug("deleted %s keys from table %s (expire date reached)", res.rowcount, table)
277
278 # Vacuuming the WALs
279 # https://www.theunterminatedstring.com/sqlite-vacuuming/
280
281 conn.execute("PRAGMA wal_checkpoint(TRUNCATE)")
282 conn.close()
283
284 return True
285

Referenced by searx.favicons.cache.FaviconCacheSQLite.set().

Here is the caller graph for this function:

◆ next_maintenance_time()

int searx.cache.ExpireCacheSQLite.next_maintenance_time ( self)
Returns (unix epoch) time of the next maintenance.

Definition at line 329 of file cache.py.

329 def next_maintenance_time(self) -> int:
330 """Returns (unix epoch) time of the next maintenance."""
331
332 return self.cfg.MAINTENANCE_PERIOD + self.properties.m_time("LAST_MAINTENANCE", int(time.time()))
333

References searx.botdetection.config.Config.cfg, cfg, and searx.sqlitedb.SQLiteAppl.properties.

◆ pairs()

Iterator[tuple[str, typing.Any]] searx.cache.ExpireCacheSQLite.pairs ( self,
str ctx )
Iterate over key/value pairs from table given by argument ``ctx``.
If ``ctx`` argument is ``None`` (the default), a table name is
generated from the :py:obj:`ExpireCacheCfg.name`.

Definition at line 400 of file cache.py.

400 def pairs(self, ctx: str) -> Iterator[tuple[str, typing.Any]]:
401 """Iterate over key/value pairs from table given by argument ``ctx``.
402 If ``ctx`` argument is ``None`` (the default), a table name is
403 generated from the :py:obj:`ExpireCacheCfg.name`."""
404 table = ctx
405 self.maintenance()
406
407 if not table:
408 table = self.normalize_name(self.cfg.name)
409
410 if table in self.table_names:
411 for row in self.DB.execute(f"SELECT key, value FROM {table}"):
412 yield row[0], self.deserialize(row[1])
413

References searx.botdetection.config.Config.cfg, cfg, DB, searx.cache.ExpireCache.deserialize(), searx.cache.ExpireCache.maintenance(), searx.cache.ExpireCache.normalize_name(), and table_names.

Here is the call graph for this function:

◆ set()

bool searx.cache.ExpireCacheSQLite.set ( self,
str key,
typing.Any value,
int | None expire,
str | None ctx = None )
Set key/value in DB table given by argument ``ctx``.  If expire is
unset the default is taken from :py:obj:`ExpireCacheCfg.MAXHOLD_TIME`.
If ``ctx`` argument is ``None`` (the default), a table name is
generated from the :py:obj:`ExpireCacheCfg.name`.  If DB table does not
exists, it will be created (on demand) by :py:obj:`self.create_table
<ExpireCacheSQLite.create_table>`.

Reimplemented from searx.cache.ExpireCache.

Definition at line 336 of file cache.py.

336 def set(self, key: str, value: typing.Any, expire: int | None, ctx: str | None = None) -> bool:
337 """Set key/value in DB table given by argument ``ctx``. If expire is
338 unset the default is taken from :py:obj:`ExpireCacheCfg.MAXHOLD_TIME`.
339 If ``ctx`` argument is ``None`` (the default), a table name is
340 generated from the :py:obj:`ExpireCacheCfg.name`. If DB table does not
341 exists, it will be created (on demand) by :py:obj:`self.create_table
342 <ExpireCacheSQLite.create_table>`.
343 """
344 table = ctx
345 self.maintenance()
346
347 value = self.serialize(value=value)
348 if len(value) > self.cfg.MAX_VALUE_LEN:
349 log.warning("ExpireCache.set(): %s.key='%s' - value too big to cache (len: %s) ", table, value, len(value))
350 return False
351
352 if not expire:
353 expire = self.cfg.MAXHOLD_TIME
354 expire = int(time.time()) + expire
355
356 table_name = table
357 if not table_name:
358 table_name = self.normalize_name(self.cfg.name)
359 self.create_table(table_name)
360
361 sql = (
362 f"INSERT INTO {table_name} (key, value, expire) VALUES (?, ?, ?)"
363 f" ON CONFLICT DO "
364 f"UPDATE SET value=?, expire=?"
365 )
366
367 if table:
368 with self.DB:
369 self.DB.execute(sql, (key, value, expire, value, expire))
370 else:
371 with self.connect() as conn:
372 conn.execute(sql, (key, value, expire, value, expire))
373 conn.close()
374
375 return True
376

References searx.botdetection.config.Config.cfg, cfg, create_table(), searx.cache.ExpireCache.maintenance(), searx.cache.ExpireCache.normalize_name(), and searx.cache.ExpireCache.serialize().

Referenced by create_table().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ state()

ExpireCacheStats searx.cache.ExpireCacheSQLite.state ( self)
Returns a :py:obj:`ExpireCacheStats`, which provides information
about the status of the cache.

Reimplemented from searx.cache.ExpireCache.

Definition at line 414 of file cache.py.

414 def state(self) -> ExpireCacheStats:
415 cached_items = {}
416 for table in self.table_names:
417 cached_items[table] = []
418 for row in self.DB.execute(f"SELECT key, value, expire FROM {table}"):
419 cached_items[table].append((row[0], self.deserialize(row[1]), row[2]))
420 return ExpireCacheStats(cached_items=cached_items)

References DB, searx.cache.ExpireCache.deserialize(), and table_names.

Here is the call graph for this function:

◆ table_names()

list[str] searx.cache.ExpireCacheSQLite.table_names ( self)
List of key/value tables already created in the DB.

Definition at line 314 of file cache.py.

314 def table_names(self) -> list[str]:
315 """List of key/value tables already created in the DB."""
316 sql = f"SELECT value FROM properties WHERE name LIKE '{self.CACHE_TABLE_PREFIX}%%'"
317 rows = self.DB.execute(sql).fetchall() or []
318 return [r[0] for r in rows]
319

References DB.

◆ truncate_tables()

searx.cache.ExpireCacheSQLite.truncate_tables ( self,
list[str] table_names )

Definition at line 320 of file cache.py.

320 def truncate_tables(self, table_names: list[str]):
321 log.debug("truncate table: %s", ",".join(table_names))
322 with self.connect() as conn:
323 for table in table_names:
324 conn.execute(f"DELETE FROM {table}")
325 conn.close()
326 return True
327

References searx.sqlitedb.SQLiteAppl.connect().

Here is the call graph for this function:

Member Data Documentation

◆ CACHE_TABLE_PREFIX

str searx.cache.ExpireCacheSQLite.CACHE_TABLE_PREFIX = "CACHE-TABLE"
static

Definition at line 230 of file cache.py.

◆ cfg

◆ next_maintenance_time

searx.cache.ExpireCacheSQLite.next_maintenance_time

Definition at line 258 of file cache.py.

Referenced by searx.favicons.cache.FaviconCacheSQLite.set().

◆ table_names

searx.cache.ExpireCacheSQLite.table_names

Definition at line 267 of file cache.py.

Referenced by create_table(), get(), pairs(), and state().


The documentation for this class was generated from the following file:
  • /home/andrew/Documents/code/public/searxng/searx/cache.py