.oO SearXNG Developer Documentation Oo.
Loading...
Searching...
No Matches
Functions | Variables
searx.plugins.hostnames Namespace Reference

Functions

dict|set|None _load_regular_expressions (settings_key)
 
 _matches_parsed_url (result, pattern)
 
bool on_result (_request, _search, result)
 

Variables

 name = gettext('Hostnames plugin')
 
 description = gettext('Rewrite hostnames, remove results or prioritize them based on the hostname')
 
bool default_on = False
 
str preference_section = 'general'
 
str plugin_id = 'hostnames'
 
str parsed = 'parsed_url'
 
list _url_fields = ['iframe_src', 'audio_src']
 
dict replacements = _load_regular_expressions('replace') or {}
 
set removables = _load_regular_expressions('remove') or set()
 
set high_priority = _load_regular_expressions('high_priority') or set()
 
set low_priority = _load_regular_expressions('low_priority') or set()
 

Detailed Description

.. attention::

    The **"Hostname replace"** plugin has been replace by **"Hostnames
    plugin"**, see :pull:`3463` & :pull:`3552`.

The **Hostnames plugin** can be enabled by adding it to the
``enabled_plugins`` **list** in the ``setting.yml`` like so.

  .. code:: yaml

     enabled_plugins:
       - 'Hostnames plugin'
       ...

- ``hostnames.replace``: A **mapping** of regular expressions to hostnames to be
  replaced by other hostnames.

  .. code:: yaml

     hostnames:
       replace:
         '(.*\\.)?youtube\\.com$': 'invidious.example.com'
         '(.*\\.)?youtu\\.be$': 'invidious.example.com'
         ...

- ``hostnames.remove``: A **list** of regular expressions of the hostnames whose
  results should be taken from the results list.

  .. code:: yaml

     hostnames:
       remove:
         - '(.*\\.)?facebook.com$'
         - ...

- ``hostnames.high_priority``: A **list** of regular expressions for hostnames
  whose result should be given higher priority. The results from these hosts are
  arranged higher in the results list.

  .. code:: yaml

     hostnames:
       high_priority:
         - '(.*\\.)?wikipedia.org$'
         - ...

- ``hostnames.lower_priority``: A **list** of regular expressions for hostnames
  whose result should be given lower priority. The results from these hosts are
  arranged lower in the results list.

  .. code:: yaml

     hostnames:
       low_priority:
         - '(.*\\.)?google(\\..*)?$'
         - ...

If the URL matches the pattern of ``high_priority`` AND ``low_priority``, the
higher priority wins over the lower priority.

Alternatively, you can also specify a file name for the **mappings** or
**lists** to load these from an external file:

.. code:: yaml

   hostnames:
     replace: 'rewrite-hosts.yml'
     remove:
       - '(.*\\.)?facebook.com$'
       - ...
     low_priority:
       - '(.*\\.)?google(\\..*)?$'
       - ...
     high_priority:
       - '(.*\\.)?wikipedia.org$'
       - ...

The ``rewrite-hosts.yml`` from the example above must be in the folder in which
the ``settings.yml`` file is already located (``/etc/searxng``). The file then
only contains the lists or the mapping tables without further information on the
namespaces.  In the example above, this would be a mapping table that looks
something like this:

.. code:: yaml

   '(.*\\.)?youtube\\.com$': 'invidious.example.com'
   '(.*\\.)?youtu\\.be$': 'invidious.example.com'

Function Documentation

◆ _load_regular_expressions()

dict | set | None searx.plugins.hostnames._load_regular_expressions ( settings_key)
protected

Definition at line 116 of file hostnames.py.

116def _load_regular_expressions(settings_key) -> dict | set | None:
117 setting_value = settings.get(plugin_id, {}).get(settings_key)
118
119 if not setting_value:
120 return None
121
122 # load external file with configuration
123 if isinstance(setting_value, str):
124 setting_value = get_yaml_cfg(setting_value)
125
126 if isinstance(setting_value, list):
127 return {re.compile(r) for r in setting_value}
128
129 if isinstance(setting_value, dict):
130 return {re.compile(p): r for (p, r) in setting_value.items()}
131
132 return None
133
134

◆ _matches_parsed_url()

searx.plugins.hostnames._matches_parsed_url ( result,
pattern )
protected

Definition at line 141 of file hostnames.py.

141def _matches_parsed_url(result, pattern):
142 return result[parsed] and (parsed in result and pattern.search(result[parsed].netloc))
143
144

Referenced by on_result().

+ Here is the caller graph for this function:

◆ on_result()

bool searx.plugins.hostnames.on_result ( _request,
_search,
result )

Definition at line 145 of file hostnames.py.

145def on_result(_request, _search, result) -> bool:
146 for pattern, replacement in replacements.items():
147 if _matches_parsed_url(result, pattern):
148 # logger.debug(result['url'])
149 result[parsed] = result[parsed]._replace(netloc=pattern.sub(replacement, result[parsed].netloc))
150 result['url'] = urlunparse(result[parsed])
151 # logger.debug(result['url'])
152
153 for url_field in _url_fields:
154 if not getattr(result, url_field, None):
155 continue
156
157 url_src = urlparse(result[url_field])
158 if pattern.search(url_src.netloc):
159 url_src = url_src._replace(netloc=pattern.sub(replacement, url_src.netloc))
160 result[url_field] = urlunparse(url_src)
161
162 for pattern in removables:
163 if _matches_parsed_url(result, pattern):
164 return False
165
166 for url_field in _url_fields:
167 if not getattr(result, url_field, None):
168 continue
169
170 url_src = urlparse(result[url_field])
171 if pattern.search(url_src.netloc):
172 del result[url_field]
173
174 for pattern in low_priority:
175 if _matches_parsed_url(result, pattern):
176 result['priority'] = 'low'
177
178 for pattern in high_priority:
179 if _matches_parsed_url(result, pattern):
180 result['priority'] = 'high'
181
182 return True

References _matches_parsed_url().

+ Here is the call graph for this function:

Variable Documentation

◆ _url_fields

list searx.plugins.hostnames._url_fields = ['iframe_src', 'audio_src']
protected

Definition at line 113 of file hostnames.py.

◆ default_on

bool searx.plugins.hostnames.default_on = False

Definition at line 107 of file hostnames.py.

◆ description

searx.plugins.hostnames.description = gettext('Rewrite hostnames, remove results or prioritize them based on the hostname')

Definition at line 106 of file hostnames.py.

◆ high_priority

set searx.plugins.hostnames.high_priority = _load_regular_expressions('high_priority') or set()

Definition at line 137 of file hostnames.py.

◆ low_priority

set searx.plugins.hostnames.low_priority = _load_regular_expressions('low_priority') or set()

Definition at line 138 of file hostnames.py.

◆ name

searx.plugins.hostnames.name = gettext('Hostnames plugin')

Definition at line 105 of file hostnames.py.

◆ parsed

str searx.plugins.hostnames.parsed = 'parsed_url'

Definition at line 112 of file hostnames.py.

◆ plugin_id

str searx.plugins.hostnames.plugin_id = 'hostnames'

Definition at line 110 of file hostnames.py.

◆ preference_section

str searx.plugins.hostnames.preference_section = 'general'

Definition at line 108 of file hostnames.py.

◆ removables

set searx.plugins.hostnames.removables = _load_regular_expressions('remove') or set()

Definition at line 136 of file hostnames.py.

◆ replacements

dict searx.plugins.hostnames.replacements = _load_regular_expressions('replace') or {}

Definition at line 135 of file hostnames.py.