Inheritance diagram for searx.favicons.cache.FaviconCacheSQLite:

Collaboration diagram for searx.favicons.cache.FaviconCacheSQLite:

Public Member Functions
	__init__ (self, FaviconCacheConfig cfg)

None\|tuple[None\|bytes, None\|str]	__call__ (self, str resolver, str authority)

bool	set (self, str resolver, str authority, str\|None mime, bytes\|None data)

int	next_maintenance_time (self)

	maintenance (self, force=False)

FaviconCacheStats	state (self)

Public Member Functions inherited from searx.sqlitedb.SQLiteAppl
	__init__ (self, db_url)

sqlite3.Connection	connect (self)

	register_functions (self, conn)

sqlite3.Connection	DB (self)

bool	init (self, sqlite3.Connection conn)

	create_schema (self, sqlite3.Connection conn)

Public Member Functions inherited from searx.favicons.cache.FaviconCache
	__init__ (self, FaviconCacheConfig cfg)

None\|tuple[None\|bytes, None\|str]	__call__ (self, str resolver, str authority)

Public Attributes
	cfg = cfg

	next_maintenance_time

Public Attributes inherited from searx.sqlitedb.SQLiteAppl
	db_url = db_url

	properties = SQLiteProperties(db_url)

Static Public Attributes
str	DDL_BLOBS

str	DDL_BLOB_MAP

tuple	SQL_DROP_LEFTOVER_BLOBS

tuple	SQL_ITER_BLOBS_SHA256_BYTES_C

tuple	SQL_INSERT_BLOBS

tuple	SQL_INSERT_BLOB_MAP

Static Public Attributes inherited from searx.sqlitedb.SQLiteAppl
dict	DDL_CREATE_TABLES = {}

int	DB_SCHEMA = 1

dict	SQLITE_THREADING_MODE

str	SQLITE_JOURNAL_MODE = "WAL"

dict	SQLITE_CONNECT_ARGS

Protected Member Functions
	_query_val (self, sql, default=None)

Protected Member Functions inherited from searx.sqlitedb.SQLiteAppl
	_compatibility (self)

sqlite3.Connection	_connect (self)

Additional Inherited Members
Protected Attributes inherited from searx.sqlitedb.SQLiteAppl
bool	_init_done = False

	_DB = None

Detailed Description

Favicon cache that manages the favicon BLOBs in a SQLite DB.  The DB
model in the SQLite DB is implemented using the abstract class
:py:obj:`sqlitedb.SQLiteAppl`.

For introspection of the DB, jump into developer environment and run command
to show cache state::

    $ ./manage pyenv.cmd bash --norc --noprofile
    (py3) python -m searx.favicons cache state

The following configurations are required / supported:

- :py:obj:`FaviconCacheConfig.db_url`
- :py:obj:`FaviconCacheConfig.HOLD_TIME`
- :py:obj:`FaviconCacheConfig.LIMIT_TOTAL_BYTES`
- :py:obj:`FaviconCacheConfig.BLOB_MAX_BYTES`
- :py:obj:`MAINTENANCE_PERIOD`
- :py:obj:`MAINTENANCE_MODE`

Definition at line 234 of file cache.py.

Constructor & Destructor Documentation

◆ init()

searx.favicons.cache.FaviconCacheSQLite.__init__	(		self,
		FaviconCacheConfig	cfg )

An instance of the favicon cache is build up from the configuration.

Definition at line 310 of file cache.py.

    def __init__(self, cfg: FaviconCacheConfig):
        """An instance of the favicon cache is build up from the configuration."""  #
 
        if cfg.db_url == ":memory:":
            logger.critical("don't use SQLite DB in :memory: in production!!")
        super().__init__(cfg.db_url)
        self.cfg = cfg
 

References __init__().

Referenced by __init__().

Here is the call graph for this function:

Here is the caller graph for this function:

Member Function Documentation

◆ call()

None \| tuple[None \| bytes, None \| str] searx.favicons.cache.FaviconCacheSQLite.__call__	(		self,
		str	resolver,
		str	authority )

Definition at line 318 of file cache.py.

    def __call__(self, resolver: str, authority: str) -> None | tuple[None | bytes, None | str]:
 
        sql = "SELECT sha256 FROM blob_map WHERE resolver = ? AND authority = ?"
        res = self.DB.execute(sql, (resolver, authority)).fetchone()
        if res is None:
            return None
 
        data, mime = (None, None)
        sha256 = res[0]
        if sha256 == FALLBACK_ICON:
            return data, mime
 
        sql = "SELECT data, mime FROM blobs WHERE sha256 = ?"
        res = self.DB.execute(sql, (sha256,)).fetchone()
        if res is not None:
            data, mime = res
        return data, mime
 

References searx.cache.ExpireCacheSQLite.DB, searx.sqlitedb.SQLiteAppl.DB(), and searx.sqlitedb.SQLiteProperties.DB.

Here is the call graph for this function:

◆ _query_val()

searx.favicons.cache.FaviconCacheSQLite._query_val	(	self,
		sql,
		default = None )

protected

Definition at line 427 of file cache.py.

    def _query_val(self, sql, default=None):
        val = self.DB.execute(sql).fetchone()
        if val is not None:
            val = val[0]
        if val is None:
            val = default
        return val
 

References searx.cache.ExpireCacheSQLite.DB, searx.sqlitedb.SQLiteAppl.DB(), and searx.sqlitedb.SQLiteProperties.DB.

Referenced by state().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ maintenance()

searx.favicons.cache.FaviconCacheSQLite.maintenance	(		self,
			force = False )

Performs maintenance on the cache

Reimplemented from searx.favicons.cache.FaviconCache.

Definition at line 379 of file cache.py.

    def maintenance(self, force=False):
 
        # Prevent parallel DB maintenance cycles from other DB connections
        # (e.g. in multi thread or process environments).
 
        if not force and int(time.time()) < self.next_maintenance_time:
            logger.debug("no maintenance required yet, next maintenance interval is in the future")
            return
        self.properties.set("LAST_MAINTENANCE", "")  # hint: this (also) sets the m_time of the property!
 
        # Do maintenance tasks.  This can be take a little more time, to avoid
        # DB locks, etablish a new DB connecton.
 
        with self.connect() as conn:
 
            # drop items not in HOLD time
            res = conn.execute(
                f"DELETE FROM blob_map"
                f" WHERE cast(m_time as integer) < cast(strftime('%s', 'now') as integer) - {self.cfg.HOLD_TIME}"
            )
            logger.debug("dropped %s obsolete blob_map items from db", res.rowcount)
            res = conn.execute(self.SQL_DROP_LEFTOVER_BLOBS)
            logger.debug("dropped %s obsolete BLOBS from db", res.rowcount)
 
            # drop old items to be in LIMIT_TOTAL_BYTES
            total_bytes = conn.execute("SELECT SUM(bytes_c) FROM blobs").fetchone()[0] or 0
            if total_bytes > self.cfg.LIMIT_TOTAL_BYTES:
 
                x = total_bytes - self.cfg.LIMIT_TOTAL_BYTES
                c = 0
                sha_list = []
                for row in conn.execute(self.SQL_ITER_BLOBS_SHA256_BYTES_C):
                    sha256, bytes_c = row
                    sha_list.append(sha256)
                    c += bytes_c
                    if c > x:
                        break
                if sha_list:
                    conn.execute("DELETE FROM blobs WHERE sha256 IN ('%s')" % "','".join(sha_list))
                    conn.execute("DELETE FROM blob_map WHERE sha256 IN ('%s')" % "','".join(sha_list))
                    logger.debug("dropped %s blobs with total size of %s bytes", len(sha_list), c)
 
        # Vacuuming the WALs
        # https://www.theunterminatedstring.com/sqlite-vacuuming/
 
        conn.execute("PRAGMA wal_checkpoint(TRUNCATE)")
        conn.close()
 

◆ next_maintenance_time()

int searx.favicons.cache.FaviconCacheSQLite.next_maintenance_time ( self )

Returns (unix epoch) time of the next maintenance.

Definition at line 374 of file cache.py.

    def next_maintenance_time(self) -> int:
        """Returns (unix epoch) time of the next maintenance."""
 
        return self.cfg.MAINTENANCE_PERIOD + self.properties.m_time("LAST_MAINTENANCE")
 

References searx.botdetection.config.Config.cfg, searx.cache.ExpireCacheSQLite.cfg, cfg, and searx.sqlitedb.SQLiteAppl.properties.

◆ set()

bool searx.favicons.cache.FaviconCacheSQLite.set	(		self,
		str	resolver,
		str	authority,
		str \| None	mime,
		bytes \| None	data )

Set data and mime-type in the cache.  If data is None, the
:py:obj:`FALLBACK_ICON` is registered. in the cache.

Reimplemented from searx.favicons.cache.FaviconCache.

Definition at line 336 of file cache.py.

    def set(self, resolver: str, authority: str, mime: str | None, data: bytes | None) -> bool:
 
        if self.cfg.MAINTENANCE_MODE == "auto" and int(time.time()) > self.next_maintenance_time:
            # Should automatic maintenance be moved to a new thread?
            self.maintenance()
 
        if data is not None and mime is None:
            logger.error(
                "favicon resolver %s tries to cache mime-type None for authority %s",
                resolver,
                authority,
            )
            return False
 
        bytes_c = len(data or b"")
        if bytes_c > self.cfg.BLOB_MAX_BYTES:
            logger.info(
                "favicon of resolver: %s / authority: %s to big to cache (bytes: %s) " % (resolver, authority, bytes_c)
            )
            return False
 
        if data is None:
            sha256 = FALLBACK_ICON
        else:
            sha256 = hashlib.sha256(data).hexdigest()
 
        with self.connect() as conn:
            if sha256 != FALLBACK_ICON:
                conn.execute(self.SQL_INSERT_BLOBS, (sha256, bytes_c, mime, data))
            conn.execute(self.SQL_INSERT_BLOB_MAP, (sha256, resolver, authority))
        # hint: the with context of the connection object closes the transaction
        # but not the DB connection.  The connection has to be closed by the
        # caller of self.connect()!
        conn.close()
 
        return True
 

References searx.botdetection.config.Config.cfg, searx.cache.ExpireCacheSQLite.cfg, cfg, searx.sqlitedb.SQLiteAppl.connect(), searx.cache.ExpireCache.maintenance(), searx.cache.ExpireCacheSQLite.maintenance(), searx.favicons.cache.FaviconCache.maintenance(), searx.cache.ExpireCacheSQLite.next_maintenance_time, next_maintenance_time, SQL_INSERT_BLOB_MAP, and SQL_INSERT_BLOBS.

Here is the call graph for this function:

◆ state()

FaviconCacheStats searx.favicons.cache.FaviconCacheSQLite.state ( self )

Returns a :py:obj:`FaviconCacheStats` (key/values) with information
on the state of the cache.

Reimplemented from searx.favicons.cache.FaviconCache.

Definition at line 435 of file cache.py.

    def state(self) -> FaviconCacheStats:
        return FaviconCacheStats(
            favicons=self._query_val("SELECT count(*) FROM blobs", 0),
            bytes=self._query_val("SELECT SUM(bytes_c) FROM blobs", 0),
            domains=self._query_val("SELECT count(*) FROM (SELECT authority FROM blob_map GROUP BY authority)", 0),
            resolvers=self._query_val("SELECT count(*) FROM (SELECT resolver FROM blob_map GROUP BY resolver)", 0),
        )
 
 

References _query_val().

Here is the call graph for this function:

Member Data Documentation

◆ cfg

searx.favicons.cache.FaviconCacheSQLite.cfg = cfg

Definition at line 316 of file cache.py.

Referenced by next_maintenance_time(), searx.cache.ExpireCache.secret_hash(), and set().

◆ DDL_BLOB_MAP

str searx.favicons.cache.FaviconCacheSQLite.DDL_BLOB_MAP

static

Initial value:

=  """\
CREATE TABLE IF NOT EXISTS blob_map (
    m_time     INTEGER DEFAULT (strftime('%s', 'now')),  -- last modified (unix epoch) time in sec.
    sha256     TEXT,
    resolver   TEXT,
    authority  TEXT,
    PRIMARY KEY (resolver, authority))"""

Definition at line 267 of file cache.py.

◆ DDL_BLOBS

str searx.favicons.cache.FaviconCacheSQLite.DDL_BLOBS

static

Initial value:

=  """\
CREATE TABLE IF NOT EXISTS blobs (
  sha256     TEXT,
  bytes_c    INTEGER,
  mime       TEXT NOT NULL,
  data       BLOB NOT NULL,
  PRIMARY KEY (sha256))"""

Definition at line 257 of file cache.py.

◆ next_maintenance_time

searx.favicons.cache.FaviconCacheSQLite.next_maintenance_time

Definition at line 384 of file cache.py.

Referenced by set().

◆ SQL_DROP_LEFTOVER_BLOBS

tuple searx.favicons.cache.FaviconCacheSQLite.SQL_DROP_LEFTOVER_BLOBS

static

Initial value:

=  (
        "DELETE FROM blobs WHERE sha256 IN ("
        " SELECT b.sha256"
        "   FROM blobs b"
        "   LEFT JOIN blob_map bm"
        "     ON b.sha256 = bm.sha256"
        "  WHERE bm.sha256 IS NULL)"
    )

Definition at line 282 of file cache.py.

◆ SQL_INSERT_BLOB_MAP

searx.favicons.cache.FaviconCacheSQLite.SQL_INSERT_BLOB_MAP

static

Initial value:

=  (
        "INSERT INTO blob_map (sha256, resolver, authority) VALUES (?, ?, ?)"
        "    ON CONFLICT DO UPDATE "
        "   SET sha256=excluded.sha256, m_time=strftime('%s', 'now')"
    )

Definition at line 304 of file cache.py.

Referenced by set().

◆ SQL_INSERT_BLOBS

searx.favicons.cache.FaviconCacheSQLite.SQL_INSERT_BLOBS

static

Initial value:

=  (
        "INSERT INTO blobs (sha256, bytes_c, mime, data) VALUES (?, ?, ?, ?)"
        "    ON CONFLICT (sha256) DO NOTHING"
    )

Definition at line 299 of file cache.py.

Referenced by set().

◆ SQL_ITER_BLOBS_SHA256_BYTES_C

searx.favicons.cache.FaviconCacheSQLite.SQL_ITER_BLOBS_SHA256_BYTES_C

static

Initial value:

=  (
        "SELECT b.sha256, b.bytes_c FROM blobs b"
        "  JOIN blob_map bm "
        "    ON b.sha256 = bm.sha256"
        " ORDER BY bm.m_time ASC"
    )

Definition at line 292 of file cache.py.

The documentation for this class was generated from the following file:

/home/andrew/Documents/code/public/searxng/searx/favicons/cache.py

Public Member Functions

Public Attributes

Static Public Attributes

Protected Member Functions

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

◆ __init__()

Member Function Documentation

◆ __call__()

◆ _query_val()

◆ maintenance()

◆ next_maintenance_time()

◆ set()

◆ state()

Member Data Documentation

◆ cfg

◆ DDL_BLOB_MAP

◆ DDL_BLOBS

◆ next_maintenance_time

◆ SQL_DROP_LEFTOVER_BLOBS

◆ SQL_INSERT_BLOB_MAP

◆ SQL_INSERT_BLOBS

◆ SQL_ITER_BLOBS_SHA256_BYTES_C

◆ init()

◆ call()