searx.data.tracker_patterns.TrackerPatternsDB Class Reference

Classes
class	Fields

Public Member Functions
	__init__ (self)
	init (self)
	load (self)
	add (self, RuleType rule)
Iterator[RuleType]	rules (self)
Iterator[RuleType]	iter_clear_list (self)
bool|str	clean_url (self, str url)

Public Attributes
	cache = get_cache()

Static Public Attributes
str	ctx_name = "data_tracker_patterns"
list	CLEAR_LIST_URL

Detailed Description

Definition at line 20 of file tracker_patterns.py.

Constructor & Destructor Documentation

__init__()

searx.data.tracker_patterns.TrackerPatternsDB.__init__

(

self

)

Definition at line 38 of file tracker_patterns.py.

    def __init__(self):
        self.cache = get_cache()
 

Member Function Documentation

add()

searx.data.tracker_patterns.TrackerPatternsDB.add	(		self,
		RuleType	rule )

Definition at line 55 of file tracker_patterns.py.

    def add(self, rule: RuleType):
        self.cache.set(
            key=rule[self.Fields.url_regexp],
            value=(
                rule[self.Fields.url_ignore],
                rule[self.Fields.del_args],
            ),
            ctx=self.ctx_name,
            expire=None,
        )
 

References searx.data.currencies.CurrenciesDB.cache, cache, and ctx_name.

Referenced by load().

Here is the caller graph for this function:

clean_url()

bool | str searx.data.tracker_patterns.TrackerPatternsDB.clean_url	(		self,
		str	url )

The URL arguments are normalized and cleaned of tracker parameters.

Returns bool ``True`` to use URL unchanged (``False`` to ignore URL).
If URL should be modified, the returned string is the new URL to use.

Definition at line 99 of file tracker_patterns.py.

    def clean_url(self, url: str) -> bool | str:
        """The URL arguments are normalized and cleaned of tracker parameters.
 
        Returns bool ``True`` to use URL unchanged (``False`` to ignore URL).
        If URL should be modified, the returned string is the new URL to use.
        """
 
        new_url = url
        parsed_new_url = urlparse(url=new_url)
 
        for rule in self.rules():
 
            if not re.match(rule[self.Fields.url_regexp], new_url):
                # no match / ignore pattern
                continue
 
            do_ignore = False
            for pattern in rule[self.Fields.url_ignore]:
                if re.match(pattern, new_url):
                    do_ignore = True
                    break
 
            if do_ignore:
                # pattern is in the list of exceptions / ignore pattern
                # HINT:
                #    we can't break the outer pattern loop since we have
                #    overlapping urlPattern like ".*"
                continue
 
            # remove tracker arguments from the url-query part
            query_args: list[tuple[str, str]] = list(parse_qsl(parsed_new_url.query))
 
            for name, val in query_args.copy():
                # remove URL arguments
                for pattern in rule[self.Fields.del_args]:
                    if re.match(pattern, name):
                        log.debug("TRACKER_PATTERNS: %s remove tracker arg: %s='%s'", parsed_new_url.netloc, name, val)
                        query_args.remove((name, val))
 
            parsed_new_url = parsed_new_url._replace(query=urlencode(query_args))
            new_url = urlunparse(parsed_new_url)
 
        if new_url != url:
            return new_url
 
        return True
 
 

References rules().

Here is the call graph for this function:

init()

searx.data.tracker_patterns.TrackerPatternsDB.init

(

self

)

Definition at line 41 of file tracker_patterns.py.

    def init(self):
        if self.cache.properties("tracker_patterns loaded") != "OK":
            # To avoid parallel initializations, the property is set first
            self.cache.properties.set("tracker_patterns loaded", "OK")
            self.load()
        # F I X M E:
        #     do we need a maintenance .. remember: database is stored
        #     in /tmp and will be rebuild during the reboot anyway
 

References searx.data.currencies.CurrenciesDB.cache, cache, searx.data.currencies.CurrenciesDB.load(), and load().

Referenced by searx.sqlitedb.SQLiteAppl.connect(), searx.sqlitedb.SQLiteAppl.DB(), and rules().

Here is the call graph for this function:

Here is the caller graph for this function:

iter_clear_list()

Iterator[RuleType] searx.data.tracker_patterns.TrackerPatternsDB.iter_clear_list

(

self

)

Definition at line 71 of file tracker_patterns.py.

    def iter_clear_list(self) -> Iterator[RuleType]:
        resp = None
        for url in self.CLEAR_LIST_URL:
            log.debug("TRACKER_PATTERNS: Trying to fetch %s...", url)
            try:
                resp = http_get(url, timeout=3)
 
            except HTTPError as exc:
                log.warning("TRACKER_PATTERNS: HTTPError (%s) occured while fetching %s", url, exc)
                continue
 
            if resp.status_code != 200:
                log.warning(f"TRACKER_PATTERNS: ClearURL ignore HTTP {resp.status_code} {url}")
                continue
 
            break
 
        if resp is None:
            log.error("TRACKER_PATTERNS: failed fetching ClearURL rule lists")
            return
 
        for rule in resp.json()["providers"].values():
            yield (
                rule["urlPattern"].replace("\\\\", "\\"),  # fix javascript regex syntax
                [exc.replace("\\\\", "\\") for exc in rule.get("exceptions", [])],
                rule.get("rules", []),
            )
 

References CLEAR_LIST_URL.

Referenced by load().

Here is the caller graph for this function:

load()

searx.data.tracker_patterns.TrackerPatternsDB.load

(

self

)

Definition at line 50 of file tracker_patterns.py.

    def load(self):
        log.debug("init searx.data.TRACKER_PATTERNS")
        for rule in self.iter_clear_list():
            self.add(rule)
 

References add(), and iter_clear_list().

Referenced by init().

Here is the call graph for this function:

Here is the caller graph for this function:

rules()

Iterator[RuleType] searx.data.tracker_patterns.TrackerPatternsDB.rules

(

self

)

Definition at line 66 of file tracker_patterns.py.

    def rules(self) -> Iterator[RuleType]:
        self.init()
        for key, value in self.cache.pairs(ctx=self.ctx_name):
            yield key, value[0], value[1]
 

References searx.data.currencies.CurrenciesDB.cache, cache, ctx_name, searx.cache.ExpireCacheSQLite.init(), searx.data.currencies.CurrenciesDB.init(), and init().

Referenced by clean_url().

Here is the call graph for this function:

Here is the caller graph for this function:

Member Data Documentation

cache

searx.data.tracker_patterns.TrackerPatternsDB.cache = get_cache()

Definition at line 39 of file tracker_patterns.py.

Referenced by add(), init(), and rules().

CLEAR_LIST_URL

list searx.data.tracker_patterns.TrackerPatternsDB.CLEAR_LIST_URL

static

Initial value:

=  [
        # ClearURL rule lists, the first one that responds HTTP 200 is used
        "https://rules1.clearurls.xyz/data.minify.json",
        "https://rules2.clearurls.xyz/data.minify.json",
        "https://raw.githubusercontent.com/ClearURLs/Rules/refs/heads/master/data.min.json",
    ]

Definition at line 25 of file tracker_patterns.py.

Referenced by iter_clear_list().

ctx_name

str searx.data.tracker_patterns.TrackerPatternsDB.ctx_name = "data_tracker_patterns"

static

Definition at line 23 of file tracker_patterns.py.

Referenced by add(), and rules().

---

The documentation for this class was generated from the following file:

• /home/andrew/Documents/code/public/searxng/searx/data/tracker_patterns.py