Inheritance diagram for searx.favicons.cache.FaviconCacheSQLite:

Collaboration diagram for searx.favicons.cache.FaviconCacheSQLite:

Public Member Functions
	__init__ (self, FaviconCacheConfig cfg)
None\|tuple[None\|bytes, None\|str]	__call__ (self, str resolver, str authority)
bool	set (self, str resolver, str authority, str\|None mime, bytes\|None data)
int	next_maintenance_time (self)
	maintenance (self, bool force=False)
FaviconCacheStats	state (self)
Public Member Functions inherited from searx.sqlitedb.SQLiteAppl
	__init__ (self, str db_url)
sqlite3.Connection	connect (self)
	register_functions (self, sqlite3.Connection conn)
sqlite3.Connection	DB (self)
bool	init (self, sqlite3.Connection conn)
	create_schema (self, sqlite3.Connection conn)
Public Member Functions inherited from searx.favicons.cache.FaviconCache
	__init__ (self, FaviconCacheConfig cfg)
None\|tuple[None\|bytes, None\|str]	__call__ (self, str resolver, str authority)

Public Attributes
	cfg = cfg
	next_maintenance_time
Public Attributes inherited from searx.sqlitedb.SQLiteAppl
str	db_url = db_url
SQLiteProperties	properties = SQLiteProperties(db_url)

Static Public Attributes
str	DDL_BLOBS
str	DDL_BLOB_MAP
tuple	SQL_DROP_LEFTOVER_BLOBS
tuple	SQL_ITER_BLOBS_SHA256_BYTES_C
tuple	SQL_INSERT_BLOBS
tuple	SQL_INSERT_BLOB_MAP
Static Public Attributes inherited from searx.sqlitedb.SQLiteAppl
dict	DDL_CREATE_TABLES = {}
int	DB_SCHEMA = 1
dict	SQLITE_THREADING_MODE
str	SQLITE_JOURNAL_MODE = "WAL"
dict	SQLITE_CONNECT_ARGS

Protected Member Functions
	_query_val (self, str sql, t.Any default=None)
Protected Member Functions inherited from searx.sqlitedb.SQLiteAppl
	_compatibility (self)
sqlite3.Connection	_connect (self)

Additional Inherited Members
Protected Attributes inherited from searx.sqlitedb.SQLiteAppl
bool	_init_done = False
sqlite3.Connection\|None	_DB = None

Detailed Description

Favicon cache that manages the favicon BLOBs in a SQLite DB.  The DB
model in the SQLite DB is implemented using the abstract class
:py:obj:`sqlitedb.SQLiteAppl`.

For introspection of the DB, jump into developer environment and run command
to show cache state::

    $ ./manage pyenv.cmd bash --norc --noprofile
    (py3) python -m searx.favicons cache state

The following configurations are required / supported:

- :py:obj:`FaviconCacheConfig.db_url`
- :py:obj:`FaviconCacheConfig.HOLD_TIME`
- :py:obj:`FaviconCacheConfig.LIMIT_TOTAL_BYTES`
- :py:obj:`FaviconCacheConfig.BLOB_MAX_BYTES`
- :py:obj:`MAINTENANCE_PERIOD`
- :py:obj:`MAINTENANCE_MODE`

Definition at line 236 of file cache.py.

Constructor & Destructor Documentation

◆ init()

searx.favicons.cache.FaviconCacheSQLite.__init__	(		self,
		FaviconCacheConfig	cfg )

An instance of the favicon cache is build up from the configuration.

Definition at line 312 of file cache.py.

    def __init__(self, cfg: FaviconCacheConfig):
        """An instance of the favicon cache is build up from the configuration."""  #
 
        if cfg.db_url == ":memory:":
            logger.critical("don't use SQLite DB in :memory: in production!!")
        super().__init__(cfg.db_url)
        self.cfg = cfg
 

References __init__().

Referenced by __init__().

Here is the call graph for this function:

Here is the caller graph for this function:

Member Function Documentation

◆ call()

None \| tuple[None \| bytes, None \| str] searx.favicons.cache.FaviconCacheSQLite.__call__	(		self,
		str	resolver,
		str	authority )

Definition at line 320 of file cache.py.

    def __call__(self, resolver: str, authority: str) -> None | tuple[None | bytes, None | str]:
 
        sql = "SELECT sha256 FROM blob_map WHERE resolver = ? AND authority = ?"
        res = self.DB.execute(sql, (resolver, authority)).fetchone()
        if res is None:
            return None
 
        data, mime = (None, None)
        sha256 = res[0]
        if sha256 == FALLBACK_ICON:
            return data, mime
 
        sql = "SELECT data, mime FROM blobs WHERE sha256 = ?"
        res = self.DB.execute(sql, (sha256,)).fetchone()
        if res is not None:
            data, mime = res
        return data, mime
 

References searx.cache.ExpireCacheSQLite.DB, searx.sqlitedb.SQLiteAppl.DB(), and searx.sqlitedb.SQLiteProperties.DB.

Here is the call graph for this function:

◆ _query_val()

searx.favicons.cache.FaviconCacheSQLite._query_val	(		self,
		str	sql,
		t.Any	default = None )

protected

Definition at line 429 of file cache.py.

    def _query_val(self, sql: str, default: t.Any = None):
        val = self.DB.execute(sql).fetchone()
        if val is not None:
            val = val[0]
        if val is None:
            val = default
        return val
 

References searx.cache.ExpireCacheSQLite.DB, searx.sqlitedb.SQLiteAppl.DB(), and searx.sqlitedb.SQLiteProperties.DB.

Referenced by state().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ maintenance()

searx.favicons.cache.FaviconCacheSQLite.maintenance	(		self,
		bool	force = False )

Performs maintenance on the cache

Reimplemented from searx.favicons.cache.FaviconCache.

Definition at line 381 of file cache.py.

    def maintenance(self, force: bool = False):
 
        # Prevent parallel DB maintenance cycles from other DB connections
        # (e.g. in multi thread or process environments).
 
        if not force and int(time.time()) < self.next_maintenance_time:
            logger.debug("no maintenance required yet, next maintenance interval is in the future")
            return
        self.properties.set("LAST_MAINTENANCE", "")  # hint: this (also) sets the m_time of the property!
 
        # Do maintenance tasks.  This can be take a little more time, to avoid
        # DB locks, establish a new DB connection.
 
        with self.connect() as conn:
 
            # drop items not in HOLD time
            res = conn.execute(
                f"DELETE FROM blob_map"
                f" WHERE cast(m_time as integer) < cast(strftime('%s', 'now') as integer) - {self.cfg.HOLD_TIME}"
            )
            logger.debug("dropped %s obsolete blob_map items from db", res.rowcount)
            res = conn.execute(self.SQL_DROP_LEFTOVER_BLOBS)
            logger.debug("dropped %s obsolete BLOBS from db", res.rowcount)
 
            # drop old items to be in LIMIT_TOTAL_BYTES
            total_bytes = conn.execute("SELECT SUM(bytes_c) FROM blobs").fetchone()[0] or 0
            if total_bytes > self.cfg.LIMIT_TOTAL_BYTES:
 
                x = total_bytes - self.cfg.LIMIT_TOTAL_BYTES
                c = 0
                sha_list: list[str] = []
                for row in conn.execute(self.SQL_ITER_BLOBS_SHA256_BYTES_C):
                    sha256, bytes_c = row
                    sha_list.append(sha256)
                    c += bytes_c
                    if c > x:
                        break
                if sha_list:
                    conn.execute("DELETE FROM blobs WHERE sha256 IN ('%s')" % "','".join(sha_list))
                    conn.execute("DELETE FROM blob_map WHERE sha256 IN ('%s')" % "','".join(sha_list))
                    logger.debug("dropped %s blobs with total size of %s bytes", len(sha_list), c)
 
        # Vacuuming the WALs
        # https://www.theunterminatedstring.com/sqlite-vacuuming/
 
        conn.execute("PRAGMA wal_checkpoint(TRUNCATE)")
        conn.close()
 

◆ next_maintenance_time()

int searx.favicons.cache.FaviconCacheSQLite.next_maintenance_time ( self )

Returns (unix epoch) time of the next maintenance.

Definition at line 376 of file cache.py.

    def next_maintenance_time(self) -> int:
        """Returns (unix epoch) time of the next maintenance."""
 
        return self.cfg.MAINTENANCE_PERIOD + self.properties.m_time("LAST_MAINTENANCE")
 

References searx.botdetection.config.Config.cfg, searx.cache.ExpireCacheSQLite.cfg, cfg, and searx.sqlitedb.SQLiteAppl.properties.

◆ set()

bool searx.favicons.cache.FaviconCacheSQLite.set	(		self,
		str	resolver,
		str	authority,
		str \| None	mime,
		bytes \| None	data )

Set data and mime-type in the cache.  If data is None, the
:py:obj:`FALLBACK_ICON` is registered. in the cache.

Reimplemented from searx.favicons.cache.FaviconCache.

Definition at line 338 of file cache.py.

    def set(self, resolver: str, authority: str, mime: str | None, data: bytes | None) -> bool:
 
        if self.cfg.MAINTENANCE_MODE == "auto" and int(time.time()) > self.next_maintenance_time:
            # Should automatic maintenance be moved to a new thread?
            self.maintenance()
 
        if data is not None and mime is None:
            logger.error(
                "favicon resolver %s tries to cache mime-type None for authority %s",
                resolver,
                authority,
            )
            return False
 
        bytes_c = len(data or b"")
        if bytes_c > self.cfg.BLOB_MAX_BYTES:
            logger.info(
                "favicon of resolver: %s / authority: %s to big to cache (bytes: %s) " % (resolver, authority, bytes_c)
            )
            return False
 
        if data is None:
            sha256 = FALLBACK_ICON
        else:
            sha256 = hashlib.sha256(data).hexdigest()
 
        with self.connect() as conn:
            if sha256 != FALLBACK_ICON:
                conn.execute(self.SQL_INSERT_BLOBS, (sha256, bytes_c, mime, data))
            conn.execute(self.SQL_INSERT_BLOB_MAP, (sha256, resolver, authority))
        # hint: the with context of the connection object closes the transaction
        # but not the DB connection.  The connection has to be closed by the
        # caller of self.connect()!
        conn.close()
 
        return True
 

References searx.botdetection.config.Config.cfg, searx.cache.ExpireCacheSQLite.cfg, cfg, searx.sqlitedb.SQLiteAppl.connect(), searx.cache.ExpireCache.maintenance(), searx.cache.ExpireCacheSQLite.maintenance(), searx.favicons.cache.FaviconCache.maintenance(), searx.cache.ExpireCacheSQLite.next_maintenance_time, next_maintenance_time, SQL_INSERT_BLOB_MAP, and SQL_INSERT_BLOBS.

Here is the call graph for this function:

◆ state()

FaviconCacheStats searx.favicons.cache.FaviconCacheSQLite.state ( self )

Returns a :py:obj:`FaviconCacheStats` (key/values) with information
on the state of the cache.

Reimplemented from searx.favicons.cache.FaviconCache.

Definition at line 437 of file cache.py.

    def state(self) -> FaviconCacheStats:
        return FaviconCacheStats(
            favicons=self._query_val("SELECT count(*) FROM blobs", 0),
            bytes=self._query_val("SELECT SUM(bytes_c) FROM blobs", 0),
            domains=self._query_val("SELECT count(*) FROM (SELECT authority FROM blob_map GROUP BY authority)", 0),
            resolvers=self._query_val("SELECT count(*) FROM (SELECT resolver FROM blob_map GROUP BY resolver)", 0),
        )
 
 
@t.final

References _query_val().

Here is the call graph for this function:

Member Data Documentation

◆ cfg

searx.favicons.cache.FaviconCacheSQLite.cfg = cfg

Definition at line 318 of file cache.py.

Referenced by next_maintenance_time(), searx.cache.ExpireCache.secret_hash(), and set().

◆ DDL_BLOB_MAP

str searx.favicons.cache.FaviconCacheSQLite.DDL_BLOB_MAP

static

Initial value:

=  """\
CREATE TABLE IF NOT EXISTS blob_map (
    m_time     INTEGER DEFAULT (strftime('%s', 'now')),  -- last modified (unix epoch) time in sec.
    sha256     TEXT,
    resolver   TEXT,
    authority  TEXT,
    PRIMARY KEY (resolver, authority))"""

Definition at line 269 of file cache.py.

◆ DDL_BLOBS

str searx.favicons.cache.FaviconCacheSQLite.DDL_BLOBS

static

Initial value:

=  """\
CREATE TABLE IF NOT EXISTS blobs (
  sha256     TEXT,
  bytes_c    INTEGER,
  mime       TEXT NOT NULL,
  data       BLOB NOT NULL,
  PRIMARY KEY (sha256))"""

Definition at line 259 of file cache.py.

◆ next_maintenance_time

searx.favicons.cache.FaviconCacheSQLite.next_maintenance_time

Definition at line 386 of file cache.py.

Referenced by set().

◆ SQL_DROP_LEFTOVER_BLOBS

tuple searx.favicons.cache.FaviconCacheSQLite.SQL_DROP_LEFTOVER_BLOBS

static

Initial value:

=  (
        "DELETE FROM blobs WHERE sha256 IN ("
        " SELECT b.sha256"
        "   FROM blobs b"
        "   LEFT JOIN blob_map bm"
        "     ON b.sha256 = bm.sha256"
        "  WHERE bm.sha256 IS NULL)"
    )

Definition at line 284 of file cache.py.

◆ SQL_INSERT_BLOB_MAP

searx.favicons.cache.FaviconCacheSQLite.SQL_INSERT_BLOB_MAP

static

Initial value:

=  (
        "INSERT INTO blob_map (sha256, resolver, authority) VALUES (?, ?, ?)"
        "    ON CONFLICT DO UPDATE "
        "   SET sha256=excluded.sha256, m_time=strftime('%s', 'now')"
    )

Definition at line 306 of file cache.py.

Referenced by set().

◆ SQL_INSERT_BLOBS

searx.favicons.cache.FaviconCacheSQLite.SQL_INSERT_BLOBS

static

Initial value:

=  (
        "INSERT INTO blobs (sha256, bytes_c, mime, data) VALUES (?, ?, ?, ?)"
        "    ON CONFLICT (sha256) DO NOTHING"
    )

Definition at line 301 of file cache.py.

Referenced by set().

◆ SQL_ITER_BLOBS_SHA256_BYTES_C

searx.favicons.cache.FaviconCacheSQLite.SQL_ITER_BLOBS_SHA256_BYTES_C

static

Initial value:

=  (
        "SELECT b.sha256, b.bytes_c FROM blobs b"
        "  JOIN blob_map bm "
        "    ON b.sha256 = bm.sha256"
        " ORDER BY bm.m_time ASC"
    )

Definition at line 294 of file cache.py.

The documentation for this class was generated from the following file:

/home/andrew/Documents/code/public/searxng/searx/favicons/cache.py

Public Member Functions

Public Attributes

Static Public Attributes

Protected Member Functions

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

◆ __init__()

Member Function Documentation

◆ __call__()

◆ _query_val()

◆ maintenance()

◆ next_maintenance_time()

◆ set()

◆ state()

Member Data Documentation

◆ cfg

◆ DDL_BLOB_MAP

◆ DDL_BLOBS

◆ next_maintenance_time

◆ SQL_DROP_LEFTOVER_BLOBS

◆ SQL_INSERT_BLOB_MAP

◆ SQL_INSERT_BLOBS

◆ SQL_ITER_BLOBS_SHA256_BYTES_C

◆ init()

◆ call()