.oO SearXNG Developer Documentation Oo.
Loading...
Searching...
No Matches
searx.sqlitedb.SQLiteAppl Class Reference
Inheritance diagram for searx.sqlitedb.SQLiteAppl:
Collaboration diagram for searx.sqlitedb.SQLiteAppl:

Public Member Functions

 __init__ (self, str db_url)
sqlite3.Connection connect (self)
 register_functions (self, sqlite3.Connection conn)
sqlite3.Connection DB (self)
bool init (self, sqlite3.Connection conn)
 create_schema (self, sqlite3.Connection conn)

Public Attributes

str db_url = db_url
SQLiteProperties properties = SQLiteProperties(db_url)

Static Public Attributes

dict DDL_CREATE_TABLES = {}
int DB_SCHEMA = 1
dict SQLITE_THREADING_MODE
str SQLITE_JOURNAL_MODE = "WAL"
dict SQLITE_CONNECT_ARGS

Protected Member Functions

 _compatibility (self)
sqlite3.Connection _connect (self)

Protected Attributes

bool _init_done = False
sqlite3.Connection|None _DB = None

Detailed Description

Abstract base class for implementing convenient DB access in SQLite
applications.  In the constructor, a :py:obj:`SQLiteProperties` instance is
already aggregated under ``self.properties``.

Definition at line 91 of file sqlitedb.py.

Constructor & Destructor Documentation

◆ __init__()

searx.sqlitedb.SQLiteAppl.__init__ ( self,
str db_url )

Definition at line 154 of file sqlitedb.py.

154 def __init__(self, db_url: str):
155
156 self.db_url: str = db_url
157 self.properties: SQLiteProperties = SQLiteProperties(db_url)
158 self._init_done: bool = False
159 self._compatibility()
160 # atexit.register(self.tear_down)
161

Member Function Documentation

◆ _compatibility()

searx.sqlitedb.SQLiteAppl._compatibility ( self)
protected

Definition at line 170 of file sqlitedb.py.

170 def _compatibility(self):
171
172 if self.SQLITE_THREADING_MODE == "serialized":
173 self._DB: sqlite3.Connection | None = None
174 else:
175 msg = (
176 f"SQLite library is compiled with {self.SQLITE_THREADING_MODE} mode,"
177 " read https://docs.python.org/3/library/sqlite3.html#sqlite3.threadsafety"
178 )
179 if threading.active_count() > 1:
180 logger.error(msg)
181 else:
182 logger.warning(msg)
183
184 if sqlite3.sqlite_version_info <= (3, 35):
185 # See "Generalize UPSERT:" in https://sqlite.org/releaselog/3_35_0.html
186 logger.critical(
187 "SQLite runtime library version %s is not supported (require >= 3.35)", sqlite3.sqlite_version
188 )
189

References SQLITE_THREADING_MODE.

◆ _connect()

sqlite3.Connection searx.sqlitedb.SQLiteAppl._connect ( self)
protected

Definition at line 190 of file sqlitedb.py.

190 def _connect(self) -> sqlite3.Connection:
191 conn = sqlite3.Connection(self.db_url, **self.SQLITE_CONNECT_ARGS) # type: ignore
192 conn.execute(f"PRAGMA journal_mode={self.SQLITE_JOURNAL_MODE}")
193 self.register_functions(conn)
194 return conn
195

References searx.cache.ExpireCacheCfg.db_url, searx.favicons.cache.FaviconCacheConfig.db_url, db_url, register_functions(), and SQLITE_CONNECT_ARGS.

Referenced by connect().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ connect()

sqlite3.Connection searx.sqlitedb.SQLiteAppl.connect ( self)
Creates a new DB connection (:py:obj:`SQLITE_CONNECT_ARGS`).  If not
already done, the DB schema is set up.  The caller must take care of
closing the resource.  Alternatively, :py:obj:`SQLiteAppl.DB` can also
be used (the resource behind `self.DB` is automatically closed when the
process or thread is terminated).

Definition at line 196 of file sqlitedb.py.

196 def connect(self) -> sqlite3.Connection:
197 """Creates a new DB connection (:py:obj:`SQLITE_CONNECT_ARGS`). If not
198 already done, the DB schema is set up. The caller must take care of
199 closing the resource. Alternatively, :py:obj:`SQLiteAppl.DB` can also
200 be used (the resource behind `self.DB` is automatically closed when the
201 process or thread is terminated).
202 """
203 if sys.version_info < (3, 12):
204 # Prior Python 3.12 there is no "autocommit" option
205 self.SQLITE_CONNECT_ARGS.pop("autocommit", None) # pyright: ignore[reportUnreachable]
206
207 msg = (
208 f"[{threading.current_thread().ident}] {self.__class__.__name__}({self.db_url})"
209 f" {self.SQLITE_CONNECT_ARGS} // {self.SQLITE_JOURNAL_MODE}"
210 )
211 logger.debug(msg)
212
213 with self._connect() as conn:
214 self.init(conn)
215 return conn
216

References _connect(), searx.cache.ExpireCacheSQLite.init(), searx.data.currencies.CurrenciesDB.init(), searx.data.tracker_patterns.TrackerPatternsDB.init(), searx.enginelib.Engine.init(), searx.plugins._core.Plugin.init(), searx.plugins._core.PluginStorage.init(), searx.plugins.ahmia_filter.SXNGPlugin.init(), searx.plugins.hostnames.SXNGPlugin.init(), searx.plugins.tracker_url_remover.SXNGPlugin.init(), init(), and SQLITE_CONNECT_ARGS.

Referenced by searx.cache.ExpireCacheSQLite.create_table(), searx.favicons.cache.FaviconCacheSQLite.set(), and searx.cache.ExpireCacheSQLite.truncate_tables().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ create_schema()

searx.sqlitedb.SQLiteAppl.create_schema ( self,
sqlite3.Connection conn )

Reimplemented in searx.sqlitedb.SQLiteProperties.

Definition at line 308 of file sqlitedb.py.

308 def create_schema(self, conn: sqlite3.Connection):
309
310 logger.debug("create schema ..")
311 self.properties.set("DB_SCHEMA", self.DB_SCHEMA)
312 self.properties.set("LAST_MAINTENANCE", "")
313 with conn:
314 for table_name, sql in self.DDL_CREATE_TABLES.items():
315 conn.execute(sql)
316 self.properties.set(f"Table {table_name} created", table_name)
317
318

References searx.cache.ExpireCacheSQLite.DB_SCHEMA, searx.favicons.cache.FaviconCacheSQLite.DB_SCHEMA, DB_SCHEMA, searx.cache.ExpireCacheSQLite.DDL_CREATE_TABLES, searx.favicons.cache.FaviconCacheSQLite.DDL_CREATE_TABLES, DDL_CREATE_TABLES, and properties.

Referenced by init(), and searx.sqlitedb.SQLiteProperties.init().

Here is the caller graph for this function:

◆ DB()

sqlite3.Connection searx.sqlitedb.SQLiteAppl.DB ( self)
Provides a DB connection.  The connection is a *singleton* and
therefore well suited for read access.  If
:py:obj:`SQLITE_THREADING_MODE` is ``serialized`` only one DB connection
is created for all threads.

.. note::

   For dedicated `transaction control`_, it is recommended to create a
   new connection (:py:obj:`SQLiteAppl.connect`).

.. _transaction control:
    https://docs.python.org/3/library/sqlite3.html#sqlite3-controlling-transactions

Definition at line 242 of file sqlitedb.py.

242 def DB(self) -> sqlite3.Connection:
243 """Provides a DB connection. The connection is a *singleton* and
244 therefore well suited for read access. If
245 :py:obj:`SQLITE_THREADING_MODE` is ``serialized`` only one DB connection
246 is created for all threads.
247
248 .. note::
249
250 For dedicated `transaction control`_, it is recommended to create a
251 new connection (:py:obj:`SQLiteAppl.connect`).
252
253 .. _transaction control:
254 https://docs.python.org/3/library/sqlite3.html#sqlite3-controlling-transactions
255 """
256
257 conn: sqlite3.Connection
258
259 if self.SQLITE_THREADING_MODE == "serialized":
260 # Theoretically it is possible to reuse the DB cursor across threads
261 # as of Python 3.12, in practice the threading of the cursor seems
262 # to me a little faulty that I prefer to establish one connection
263 # per thread.
264 #
265 # may we can activate this code one day ..
266 # if self._DB is None:
267 # self._DB = self.connect()
268 # conn = self._DB
269 conn = DBSession.get_connect(self)
270 else:
271 conn = DBSession.get_connect(self)
272
273 # Since more than one instance of SQLiteAppl share the same DB
274 # connection, we need to make sure that each SQLiteAppl instance has run
275 # its init method at least once.
276 self.init(conn)
277
278 return conn
279

References searx.cache.ExpireCacheSQLite.init(), searx.data.currencies.CurrenciesDB.init(), searx.data.tracker_patterns.TrackerPatternsDB.init(), searx.enginelib.Engine.init(), searx.plugins._core.Plugin.init(), searx.plugins._core.PluginStorage.init(), searx.plugins.ahmia_filter.SXNGPlugin.init(), searx.plugins.hostnames.SXNGPlugin.init(), searx.plugins.tracker_url_remover.SXNGPlugin.init(), init(), and SQLITE_THREADING_MODE.

Referenced by searx.favicons.cache.FaviconCacheSQLite.__call__(), and searx.favicons.cache.FaviconCacheSQLite._query_val().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ init()

bool searx.sqlitedb.SQLiteAppl.init ( self,
sqlite3.Connection conn )
Initializes the DB schema and properties, is only executed once even
if called several times.

If the initialization has not yet taken place, it is carried out and a
`True` is returned to the caller at the end.  If the initialization has
already been carried out in the past, `False` is returned.

Reimplemented in searx.cache.ExpireCacheSQLite, and searx.sqlitedb.SQLiteProperties.

Definition at line 280 of file sqlitedb.py.

280 def init(self, conn: sqlite3.Connection) -> bool:
281 """Initializes the DB schema and properties, is only executed once even
282 if called several times.
283
284 If the initialization has not yet taken place, it is carried out and a
285 `True` is returned to the caller at the end. If the initialization has
286 already been carried out in the past, `False` is returned.
287 """
288
289 if self._init_done:
290 return False
291 self._init_done = True
292
293 logger.debug("init DB: %s", self.db_url)
294 self.properties.init(conn)
295
296 ver = self.properties("DB_SCHEMA")
297 if ver is None:
298 with conn:
299 self.create_schema(conn)
300 else:
301 ver = int(ver)
302 if ver != self.DB_SCHEMA:
303 raise sqlite3.DatabaseError("Expected DB schema v%s, DB schema is v%s" % (self.DB_SCHEMA, ver))
304 logger.debug("DB_SCHEMA = %s", ver)
305
306 return True
307

References _init_done, create_schema(), searx.cache.ExpireCacheSQLite.DB_SCHEMA, searx.favicons.cache.FaviconCacheSQLite.DB_SCHEMA, DB_SCHEMA, searx.cache.ExpireCacheCfg.db_url, searx.favicons.cache.FaviconCacheConfig.db_url, db_url, init(), and properties.

Referenced by connect(), DB(), and init().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ register_functions()

searx.sqlitedb.SQLiteAppl.register_functions ( self,
sqlite3.Connection conn )
Create user-defined_ SQL functions.

``REGEXP(<pattern>, <field>)`` : 0 | 1
   `re.search`_ returns (int) 1 for a match and 0 for none match of
   ``<pattern>`` in ``<field>``.

   .. code:: sql

      SELECT '12' AS field WHERE REGEXP('^[0-9][0-9]$', field)
      -- 12

      SELECT REGEXP('[0-9][0-9]', 'X12Y')
      -- 1
      SELECT REGEXP('[0-9][0-9]', 'X1Y')
      -- 0

.. _user-defined: https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.create_function
.. _deterministic: https://sqlite.org/deterministic.html
.. _re.search: https://docs.python.org/3/library/re.html#re.search

Definition at line 217 of file sqlitedb.py.

217 def register_functions(self, conn: sqlite3.Connection):
218 """Create user-defined_ SQL functions.
219
220 ``REGEXP(<pattern>, <field>)`` : 0 | 1
221 `re.search`_ returns (int) 1 for a match and 0 for none match of
222 ``<pattern>`` in ``<field>``.
223
224 .. code:: sql
225
226 SELECT '12' AS field WHERE REGEXP('^[0-9][0-9]$', field)
227 -- 12
228
229 SELECT REGEXP('[0-9][0-9]', 'X12Y')
230 -- 1
231 SELECT REGEXP('[0-9][0-9]', 'X1Y')
232 -- 0
233
234 .. _user-defined: https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.create_function
235 .. _deterministic: https://sqlite.org/deterministic.html
236 .. _re.search: https://docs.python.org/3/library/re.html#re.search
237 """
238
239 conn.create_function("regexp", 2, lambda x, y: 1 if re.search(x, y) else 0, deterministic=True) # type: ignore
240

Referenced by _connect().

Here is the caller graph for this function:

Member Data Documentation

◆ _DB

sqlite3.Connection | None searx.sqlitedb.SQLiteAppl._DB = None
protected

Definition at line 173 of file sqlitedb.py.

◆ _init_done

bool searx.sqlitedb.SQLiteAppl._init_done = False
protected

Definition at line 158 of file sqlitedb.py.

Referenced by init(), and searx.sqlitedb.SQLiteProperties.init().

◆ DB_SCHEMA

searx.sqlitedb.SQLiteAppl.DB_SCHEMA = 1
static

Definition at line 98 of file sqlitedb.py.

Referenced by create_schema(), and init().

◆ db_url

searx.sqlitedb.SQLiteAppl.db_url = db_url

Definition at line 156 of file sqlitedb.py.

Referenced by _connect(), init(), and searx.sqlitedb.SQLiteProperties.init().

◆ DDL_CREATE_TABLES

dict searx.sqlitedb.SQLiteAppl.DDL_CREATE_TABLES = {}
static

Definition at line 96 of file sqlitedb.py.

Referenced by create_schema().

◆ properties

◆ SQLITE_CONNECT_ARGS

dict searx.sqlitedb.SQLiteAppl.SQLITE_CONNECT_ARGS
static
Initial value:
= {
# "timeout": 5.0,
# "detect_types": 0,
"check_same_thread": bool(SQLITE_THREADING_MODE != "serialized"),
"cached_statements": 0, # https://github.com/python/cpython/issues/118172
# "uri": False,
"isolation_level": None,
}

Definition at line 124 of file sqlitedb.py.

Referenced by _connect(), and connect().

◆ SQLITE_JOURNAL_MODE

str searx.sqlitedb.SQLiteAppl.SQLITE_JOURNAL_MODE = "WAL"
static

Definition at line 118 of file sqlitedb.py.

◆ SQLITE_THREADING_MODE

dict searx.sqlitedb.SQLiteAppl.SQLITE_THREADING_MODE
static
Initial value:
= {
0: "single-thread",
1: "multi-thread",
3: "serialized"}[sqlite3.threadsafety]

Definition at line 103 of file sqlitedb.py.

Referenced by _compatibility(), and DB().


The documentation for this class was generated from the following file:
  • /home/andrew/Documents/code/public/searxng/searx/sqlitedb.py