searx.cache.ExpireCacheSQLite Class Reference

Inheritance diagram for searx.cache.ExpireCacheSQLite:

Collaboration diagram for searx.cache.ExpireCacheSQLite:

Public Member Functions
	__init__ (self, ExpireCacheCfg cfg)
bool	init (self, sqlite3.Connection conn)
bool	maintenance (self, bool force=False, bool truncate=False)
bool	create_table (self, str table)
list[str]	table_names (self)
	truncate_tables (self, list[str] table_names)
int	next_maintenance_time (self)
bool	set (self, str key, typing.Any value, int|None expire, str|None ctx=None)
typing.Any	get (self, str key, default=None, str|None ctx=None)
ExpireCacheStats	state (self)
Public Member Functions inherited from searx.sqlitedb.SQLiteAppl
	__init__ (self, db_url)
sqlite3.Connection	connect (self)
	register_functions (self, conn)
sqlite3.Connection	DB (self)
	create_schema (self, sqlite3.Connection conn)
Public Member Functions inherited from searx.cache.ExpireCache
bytes	serialize (self, typing.Any value)
typing.Any	deserialize (self, bytes value)
str	secret_hash (self, str|bytes name)

Public Attributes
	cfg = cfg
	next_maintenance_time
	table_names
Public Attributes inherited from searx.sqlitedb.SQLiteAppl
	db_url = db_url
	properties = SQLiteProperties(db_url)

Static Public Attributes
str	CACHE_TABLE_PREFIX = "CACHE-TABLE-"
Static Public Attributes inherited from searx.sqlitedb.SQLiteAppl
dict	DDL_CREATE_TABLES = {}
int	DB_SCHEMA = 1
dict	SQLITE_THREADING_MODE
str	SQLITE_JOURNAL_MODE = "WAL"
dict	SQLITE_CONNECT_ARGS
Static Public Attributes inherited from searx.cache.ExpireCache
ExpireCacheCfg	hash_token = "hash_token"

Additional Inherited Members
Static Public Member Functions inherited from searx.cache.ExpireCache
ExpireCache	build_cache (ExpireCacheCfg cfg)
str	normalize_name (str name)
Protected Member Functions inherited from searx.sqlitedb.SQLiteAppl
	_compatibility (self)
sqlite3.Connection	_connect (self)
Protected Attributes inherited from searx.sqlitedb.SQLiteAppl
bool	_init_done = False
	_DB = None

Detailed Description

Cache that manages key/value pairs in a SQLite DB.  The DB model in the
SQLite DB is implemented in abstract class :py:obj:`SQLiteAppl
<searx.sqlitedb.SQLiteAppl>`.

The following configurations are required / supported:

- :py:obj:`ExpireCacheCfg.db_url`
- :py:obj:`ExpireCacheCfg.MAXHOLD_TIME`
- :py:obj:`ExpireCacheCfg.MAINTENANCE_PERIOD`
- :py:obj:`ExpireCacheCfg.MAINTENANCE_MODE`

Definition at line 211 of file cache.py.

Constructor & Destructor Documentation

__init__()

searx.cache.ExpireCacheSQLite.__init__	(		self,
		ExpireCacheCfg	cfg )

An instance of the SQLite expire cache is build up from a
:py:obj:`config <ExpireCacheCfg>`.

Definition at line 231 of file cache.py.

    def __init__(self, cfg: ExpireCacheCfg):
        """An instance of the SQLite expire cache is build up from a
        :py:obj:`config <ExpireCacheCfg>`."""
 
        self.cfg = cfg
        if cfg.db_url == ":memory:":
            log.critical("don't use SQLite DB in :memory: in production!!")
        super().__init__(cfg.db_url)
 

Member Function Documentation

create_table()

bool searx.cache.ExpireCacheSQLite.create_table	(		self,
		str	table )

Create DB ``table`` if it has not yet been created, no recreates are
initiated if the table already exists.

Definition at line 285 of file cache.py.

    def create_table(self, table: str) -> bool:
        """Create DB ``table`` if it has not yet been created, no recreates are
        initiated if the table already exists.
        """
        if table in self.table_names:
            # log.debug("key/value table %s exists in DB (no need to recreate)", table)
            return False
 
        log.info("key/value table '%s' NOT exists in DB -> create DB table ..", table)
        sql_table = "\n".join(
            [
                f"CREATE TABLE IF NOT EXISTS {table} (",
                "  key        TEXT,",
                "  value      BLOB,",
                f"  expire     INTEGER DEFAULT (strftime('%s', 'now') + {self.cfg.MAXHOLD_TIME}),",
                "PRIMARY KEY (key))",
            ]
        )
        sql_index = f"CREATE INDEX IF NOT EXISTS index_expire_{table} ON {table}(expire);"
        with self.connect() as conn:
            conn.execute(sql_table)
            conn.execute(sql_index)
        conn.close()
 
        self.properties.set(f"{self.CACHE_TABLE_PREFIX}-{table}", table)
        return True
 

References searx.sqlitedb.SQLiteAppl.connect(), searx.sqlitedb.SQLiteAppl.properties, set(), and table_names.

Referenced by set().

Here is the call graph for this function:

Here is the caller graph for this function:

get()

typing.Any searx.cache.ExpireCacheSQLite.get	(		self,
		str	key,
			default = None,
		str | None	ctx = None )

Get value of ``key`` from table given by argument ``ctx``.  If
``ctx`` argument is ``None`` (the default), a table name is generated
from the :py:obj:`ExpireCacheCfg.name`.  If ``key`` not exists (in
table), the ``default`` value is returned.

Reimplemented from searx.cache.ExpireCache.

Definition at line 376 of file cache.py.

    def get(self, key: str, default=None, ctx: str | None = None) -> typing.Any:
        """Get value of ``key`` from table given by argument ``ctx``.  If
        ``ctx`` argument is ``None`` (the default), a table name is generated
        from the :py:obj:`ExpireCacheCfg.name`.  If ``key`` not exists (in
        table), the ``default`` value is returned.
 
        """
        table = ctx
        self.maintenance()
 
        if not table:
            table = self.normalize_name(self.cfg.name)
 
        if table not in self.table_names:
            return default
 
        sql = f"SELECT value FROM {table} WHERE key = ?"
        row = self.DB.execute(sql, (key,)).fetchone()
        if row is None:
            return default
 
        return self.deserialize(row[0])
 

References searx.botdetection.config.Config.cfg, cfg, DB, searx.cache.ExpireCache.deserialize(), searx.cache.ExpireCache.maintenance(), searx.cache.ExpireCache.normalize_name(), and table_names.

Referenced by searx.result_types._base.LegacyResult.__init__(), searx.result_types._base.LegacyResult.defaults_from(), and searx.answerers._core.AnswerStorage.register().

Here is the call graph for this function:

Here is the caller graph for this function:

init()

bool searx.cache.ExpireCacheSQLite.init	(		self,
		sqlite3.Connection	conn )

Initializes the DB schema and properties, is only executed once even
if called several times.

If the initialization has not yet taken place, it is carried out and a
`True` is returned to the caller at the end.  If the initialization has
already been carried out in the past, `False` is returned.

Reimplemented from searx.sqlitedb.SQLiteAppl.

Definition at line 240 of file cache.py.

    def init(self, conn: sqlite3.Connection) -> bool:
        ret_val = super().init(conn)
        if not ret_val:
            return False
 
        new = hashlib.sha256(self.cfg.password).hexdigest()
        old = self.properties(self.hash_token)
        if old != new:
            if old is not None:
                log.warning("[%s] hash token changed: truncate all cache tables", self.cfg.name)
            self.maintenance(force=True, truncate=True)
            self.properties.set(self.hash_token, new)
 
        return True
 

References searx.botdetection.config.Config.cfg, cfg, searx.cache.ExpireCache.hash_token, init(), searx.cache.ExpireCache.maintenance(), and searx.sqlitedb.SQLiteAppl.properties.

Referenced by searx.sqlitedb.SQLiteAppl.connect(), searx.sqlitedb.SQLiteAppl.DB(), and init().

Here is the call graph for this function:

Here is the caller graph for this function:

maintenance()

bool searx.cache.ExpireCacheSQLite.maintenance	(		self,
		bool	force = False,
		bool	truncate = False )

Performs maintenance on the cache.

``force``:
  Maintenance should be carried out even if the maintenance interval has
  not yet been reached.

``truncate``:
  Truncate the entire cache, which is necessary, for example, if the
  password has changed.

Reimplemented from searx.cache.ExpireCache.

Definition at line 255 of file cache.py.

    def maintenance(self, force: bool = False, truncate: bool = False) -> bool:
 
        if not force and int(time.time()) < self.next_maintenance_time:
            # log.debug("no maintenance required yet, next maintenance interval is in the future")
            return False
 
        # Prevent parallel DB maintenance cycles from other DB connections
        # (e.g. in multi thread or process environments).
        self.properties.set("LAST_MAINTENANCE", "")  # hint: this (also) sets the m_time of the property!
 
        if truncate:
            self.truncate_tables(self.table_names)
            return True
 
        # drop items by expire time stamp ..
        expire = int(time.time())
 
        with self.connect() as conn:
            for table in self.table_names:
                res = conn.execute(f"DELETE FROM {table} WHERE expire < ?", (expire,))
                log.debug("deleted %s keys from table %s (expire date reached)", res.rowcount, table)
 
        # Vacuuming the WALs
        # https://www.theunterminatedstring.com/sqlite-vacuuming/
 
        conn.execute("PRAGMA wal_checkpoint(TRUNCATE)")
        conn.close()
 
        return True
 

Referenced by searx.favicons.cache.FaviconCacheSQLite.set().

Here is the caller graph for this function:

next_maintenance_time()

int searx.cache.ExpireCacheSQLite.next_maintenance_time

(

self

)

Returns (unix epoch) time of the next maintenance.

Definition at line 328 of file cache.py.

    def next_maintenance_time(self) -> int:
        """Returns (unix epoch) time of the next maintenance."""
 
        return self.cfg.MAINTENANCE_PERIOD + self.properties.m_time("LAST_MAINTENANCE", int(time.time()))
 

References searx.botdetection.config.Config.cfg, cfg, and searx.sqlitedb.SQLiteAppl.properties.

set()

bool searx.cache.ExpireCacheSQLite.set	(		self,
		str	key,
		typing.Any	value,
		int | None	expire,
		str | None	ctx = None )

Set key/value in DB table given by argument ``ctx``.  If expire is
unset the default is taken from :py:obj:`ExpireCacheCfg.MAXHOLD_TIME`.
If ``ctx`` argument is ``None`` (the default), a table name is
generated from the :py:obj:`ExpireCacheCfg.name`.  If DB table does not
exists, it will be created (on demand) by :py:obj:`self.create_table
<ExpireCacheSQLite.create_table>`.

Reimplemented from searx.cache.ExpireCache.

Definition at line 335 of file cache.py.

    def set(self, key: str, value: typing.Any, expire: int | None, ctx: str | None = None) -> bool:
        """Set key/value in DB table given by argument ``ctx``.  If expire is
        unset the default is taken from :py:obj:`ExpireCacheCfg.MAXHOLD_TIME`.
        If ``ctx`` argument is ``None`` (the default), a table name is
        generated from the :py:obj:`ExpireCacheCfg.name`.  If DB table does not
        exists, it will be created (on demand) by :py:obj:`self.create_table
        <ExpireCacheSQLite.create_table>`.
        """
        table = ctx
        self.maintenance()
 
        value = self.serialize(value=value)
        if len(value) > self.cfg.MAX_VALUE_LEN:
            log.warning("ExpireCache.set(): %s.key='%s' - value too big to cache (len: %s)  ", table, value, len(value))
            return False
 
        if not expire:
            expire = self.cfg.MAXHOLD_TIME
        expire = int(time.time()) + expire
 
        table_name = table
        if not table_name:
            table_name = self.normalize_name(self.cfg.name)
        self.create_table(table_name)
 
        sql = (
            f"INSERT INTO {table_name} (key, value, expire) VALUES (?, ?, ?)"
            f"    ON CONFLICT DO "
            f"UPDATE SET value=?, expire=?"
        )
 
        if table:
            with self.DB:
                self.DB.execute(sql, (key, value, expire, value, expire))
        else:
            with self.connect() as conn:
                conn.execute(sql, (key, value, expire, value, expire))
            conn.close()
 
        return True
 

References searx.botdetection.config.Config.cfg, cfg, create_table(), searx.cache.ExpireCache.maintenance(), searx.cache.ExpireCache.normalize_name(), and searx.cache.ExpireCache.serialize().

Referenced by create_table().

Here is the call graph for this function:

Here is the caller graph for this function:

state()

ExpireCacheStats searx.cache.ExpireCacheSQLite.state

(

self

)

Returns a :py:obj:`ExpireCacheStats`, which provides information
about the status of the cache.

Reimplemented from searx.cache.ExpireCache.

Definition at line 399 of file cache.py.

    def state(self) -> ExpireCacheStats:
        cached_items = {}
        for table in self.table_names:
            cached_items[table] = []
            for row in self.DB.execute(f"SELECT key, value, expire FROM {table}"):
                cached_items[table].append((row[0], self.deserialize(row[1]), row[2]))
        return ExpireCacheStats(cached_items=cached_items)

References DB, searx.cache.ExpireCache.deserialize(), and table_names.

Here is the call graph for this function:

table_names()

list[str] searx.cache.ExpireCacheSQLite.table_names

(

self

)

List of key/value tables already created in the DB.

Definition at line 313 of file cache.py.

    def table_names(self) -> list[str]:
        """List of key/value tables already created in the DB."""
        sql = f"SELECT value FROM properties WHERE name LIKE '{self.CACHE_TABLE_PREFIX}%%'"
        rows = self.DB.execute(sql).fetchall() or []
        return [r[0] for r in rows]
 

References DB.

truncate_tables()

searx.cache.ExpireCacheSQLite.truncate_tables	(		self,
		list[str]	table_names )

Definition at line 319 of file cache.py.

    def truncate_tables(self, table_names: list[str]):
        log.debug("truncate table: %s", ",".join(table_names))
        with self.connect() as conn:
            for table in table_names:
                conn.execute(f"DELETE FROM {table}")
        conn.close()
        return True
 

References searx.sqlitedb.SQLiteAppl.connect().

Here is the call graph for this function:

Member Data Documentation

CACHE_TABLE_PREFIX

str searx.cache.ExpireCacheSQLite.CACHE_TABLE_PREFIX = "CACHE-TABLE-"

static

Definition at line 229 of file cache.py.

cfg

searx.cache.ExpireCacheSQLite.cfg = cfg

Definition at line 235 of file cache.py.

Referenced by get(), init(), next_maintenance_time(), searx.favicons.cache.FaviconCacheSQLite.next_maintenance_time(), searx.cache.ExpireCache.secret_hash(), set(), and searx.favicons.cache.FaviconCacheSQLite.set().

next_maintenance_time

searx.cache.ExpireCacheSQLite.next_maintenance_time

Definition at line 257 of file cache.py.

Referenced by searx.favicons.cache.FaviconCacheSQLite.set().

table_names

searx.cache.ExpireCacheSQLite.table_names

Definition at line 266 of file cache.py.

Referenced by create_table(), get(), and state().

---

The documentation for this class was generated from the following file:

• /home/andrew/Documents/code/public/searxng/searx/cache.py