Classes
class	Checker

class	CheckerTests

class	ResultContainerTests

class	TestResults

Functions
	get_check_no_html ()

	_is_url (url)

bool	_download_and_check_if_image (str image_url)

bool	_is_url_image (image_url)

typing.Dict[str, typing.Any]	_search_query_to_dict (SearchQuery search_query)

typing.Tuple[typing.Dict[str, typing.Any], typing.Dict[str, typing.Any]]	_search_query_diff (SearchQuery sq1, SearchQuery sq2)

Variables
	logger = logger.getChild('searx.search.checker')

list	HTML_TAGS

	_check_no_html = get_check_no_html()

Function Documentation

◆ _download_and_check_if_image()

bool searx.search.checker.impl._download_and_check_if_image ( str image_url )

protected

Download an URL and check if the Content-Type starts with "image/"
This function should not be called directly: use _is_url_image
otherwise the cache of functools.lru_cache contains data: URL which might be huge.

Definition at line 64 of file impl.py.

def _download_and_check_if_image(image_url: str) -> bool:
    """Download an URL and check if the Content-Type starts with "image/"
    This function should not be called directly: use _is_url_image
    otherwise the cache of functools.lru_cache contains data: URL which might be huge.
    """
    retry = 2
 
    while retry > 0:
        a = time()
        try:
            # use "image_proxy" (avoid HTTP/2)
            network.set_context_network_name('image_proxy')
            r, stream = network.stream(
                'GET',
                image_url,
                timeout=10.0,
                allow_redirects=True,
                headers={
                    'User-Agent': gen_useragent(),
                    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
                    'Accept-Language': 'en-US;q=0.5,en;q=0.3',
                    'Accept-Encoding': 'gzip, deflate, br',
                    'DNT': '1',
                    'Connection': 'keep-alive',
                    'Upgrade-Insecure-Requests': '1',
                    'Sec-GPC': '1',
                    'Cache-Control': 'max-age=0',
                },
            )
            r.close()
            if r.status_code == 200:
                is_image = r.headers.get('content-type', '').startswith('image/')
            else:
                is_image = False
            del r
            del stream
            return is_image
        except httpx.TimeoutException:
            logger.error('Timeout for %s: %i', image_url, int(time() - a))
            retry -= 1
        except httpx.HTTPError:
            logger.exception('Exception for %s', image_url)
            return False
    return False
 
 

Referenced by searx.search.checker.impl._is_url_image().

Here is the caller graph for this function:

◆ _is_url()

searx.search.checker.impl._is_url ( url )

protected

Definition at line 53 of file impl.py.

def _is_url(url):
    try:
        result = urlparse(url)
    except ValueError:
        return False
    if result.scheme not in ('http', 'https'):
        return False
    return True
 
 
@functools.lru_cache(maxsize=8192)

Referenced by searx.search.checker.impl._is_url_image().

Here is the caller graph for this function:

◆ _is_url_image()

bool searx.search.checker.impl._is_url_image ( image_url )

protected

Normalize image_url

Definition at line 110 of file impl.py.

def _is_url_image(image_url) -> bool:
    """Normalize image_url"""
    if not isinstance(image_url, str):
        return False
 
    if image_url.startswith('//'):
        image_url = 'https:' + image_url
 
    if image_url.startswith('data:'):
        return image_url.startswith('data:image/')
 
    if not _is_url(image_url):
        return False
 
    return _download_and_check_if_image(image_url)
 
 

References searx.search.checker.impl._download_and_check_if_image(), and searx.search.checker.impl._is_url().

Referenced by searx.search.checker.impl.ResultContainerTests._check_result().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ _search_query_diff()

typing.Tuple[typing.Dict[str, typing.Any], typing.Dict[str, typing.Any]] searx.search.checker.impl._search_query_diff	(	SearchQuery	sq1,
		SearchQuery	sq2 )

protected

Definition at line 137 of file impl.py.

) -> typing.Tuple[typing.Dict[str, typing.Any], typing.Dict[str, typing.Any]]:
    param1 = _search_query_to_dict(sq1)
    param2 = _search_query_to_dict(sq2)
    common = {}
    diff = {}
    for k, value1 in param1.items():
        value2 = param2[k]
        if value1 == value2:
            common[k] = value1
        else:
            diff[k] = (value1, value2)
    return (common, diff)
 
 

References searx.search.checker.impl._search_query_to_dict().

Here is the call graph for this function:

◆ _search_query_to_dict()

typing.Dict[str, typing.Any] searx.search.checker.impl._search_query_to_dict ( SearchQuery search_query )

protected

Definition at line 127 of file impl.py.

def _search_query_to_dict(search_query: SearchQuery) -> typing.Dict[str, typing.Any]:
    return {
        'query': search_query.query,
        'lang': search_query.lang,
        'pageno': search_query.pageno,
        'safesearch': search_query.safesearch,
        'time_range': search_query.time_range,
    }
 
 

Referenced by searx.search.checker.impl.ResultContainerTests._record_error(), and searx.search.checker.impl._search_query_diff().

Here is the caller graph for this function:

◆ get_check_no_html()

searx.search.checker.impl.get_check_no_html ( )

Definition at line 39 of file impl.py.

def get_check_no_html():
    rep = ['<' + tag + r'[^>]*>' for tag in HTML_TAGS]
    rep += ['</' + tag + '>' for tag in HTML_TAGS]
    pattern = re.compile('|'.join(rep))
 
    def f(text):
        return pattern.search(text.lower()) is None
 
    return f
 
 

Variable Documentation

◆ _check_no_html

searx.search.checker.impl._check_no_html = get_check_no_html()

protected

Definition at line 50 of file impl.py.

Referenced by searx.search.checker.impl.ResultContainerTests._check_answers(), searx.search.checker.impl.ResultContainerTests._check_infoboxes(), and searx.search.checker.impl.ResultContainerTests._check_result().

◆ HTML_TAGS

list searx.search.checker.impl.HTML_TAGS

Initial value:

=  [
    # fmt: off
    'embed', 'iframe', 'object', 'param', 'picture', 'source', 'svg', 'math', 'canvas', 'noscript', 'script',
    'del', 'ins', 'area', 'audio', 'img', 'map', 'track', 'video', 'a', 'abbr', 'b', 'bdi', 'bdo', 'br', 'cite',
    'code', 'data', 'dfn', 'em', 'i', 'kdb', 'mark', 'q', 'rb', 'rp', 'rt', 'rtc', 'ruby', 's', 'samp', 'small',
    'span', 'strong', 'sub', 'sup', 'time', 'u', 'var', 'wbr', 'style', 'blockquote', 'dd', 'div', 'dl', 'dt',
    'figcaption', 'figure', 'hr', 'li', 'ol', 'p', 'pre', 'ul', 'button', 'datalist', 'fieldset', 'form', 'input',
    'label', 'legend', 'meter', 'optgroup', 'option', 'output', 'progress', 'select', 'textarea', 'applet',
    'frame', 'frameset'
    # fmt: on
]

Definition at line 26 of file impl.py.

◆ logger

searx.search.checker.impl.logger = logger.getChild('searx.search.checker')

Definition at line 24 of file impl.py.

Classes

Functions

Variables

Function Documentation

◆ _download_and_check_if_image()

◆ _is_url()

◆ _is_url_image()

◆ _search_query_diff()

◆ _search_query_to_dict()

◆ get_check_no_html()

Variable Documentation

◆ _check_no_html

◆ HTML_TAGS

◆ logger