searx.utils.HTMLTextExtractor Class Reference

Inheritance diagram for searx.utils.HTMLTextExtractor:

Collaboration diagram for searx.utils.HTMLTextExtractor:

Public Member Functions
	__init__ (self)
None	handle_starttag (self, str tag, list[tuple[str, str|None]] attrs)
None	handle_endtag (self, str tag)
	is_valid_tag (self)
None	handle_data (self, str data)
None	handle_charref (self, str name)
None	handle_entityref (self, str name)
	get_text (self)
None	error (self, str message)

Public Attributes
list	result = []
list	tags = []

Detailed Description

Internal class to extract text from HTML

Definition at line 90 of file utils.py.

Constructor & Destructor Documentation

__init__()

searx.utils.HTMLTextExtractor.__init__

(

self

)

Definition at line 93 of file utils.py.

    def __init__(self):
        HTMLParser.__init__(self)
        self.result: list[str] = []
        self.tags: list[str] = []
 

Member Function Documentation

error()

None searx.utils.HTMLTextExtractor.error	(		self,
		str	message )

Definition at line 140 of file utils.py.

    def error(self, message: str) -> None:
        # error handle is needed in <py3.10
        # https://github.com/python/cpython/pull/8562/files
        raise AssertionError(message)
 
 

get_text()

searx.utils.HTMLTextExtractor.get_text

(

self

)

Definition at line 137 of file utils.py.

    def get_text(self):
        return ''.join(self.result).strip()
 

References result.

handle_charref()

None searx.utils.HTMLTextExtractor.handle_charref	(		self,
		str	name )

Definition at line 121 of file utils.py.

    def handle_charref(self, name: str) -> None:
        if not self.is_valid_tag():
            return
        if name[0] in ('x', 'X'):
            codepoint = int(name[1:], 16)
        else:
            codepoint = int(name)
        self.result.append(chr(codepoint))
 

References is_valid_tag(), and result.

Here is the call graph for this function:

handle_data()

None searx.utils.HTMLTextExtractor.handle_data	(		self,
		str	data )

Definition at line 116 of file utils.py.

    def handle_data(self, data: str) -> None:
        if not self.is_valid_tag():
            return
        self.result.append(data)
 

References is_valid_tag(), and result.

Here is the call graph for this function:

handle_endtag()

None searx.utils.HTMLTextExtractor.handle_endtag	(		self,
		str	tag )

Definition at line 103 of file utils.py.

    def handle_endtag(self, tag: str) -> None:
        if not self.tags:
            return
 
        if tag != self.tags[-1]:
            self.result.append(f"</{tag}>")
            return
 
        self.tags.pop()
 

References result, searx.result_types.paper.Paper.tags, and tags.

handle_entityref()

None searx.utils.HTMLTextExtractor.handle_entityref	(		self,
		str	name )

Definition at line 130 of file utils.py.

    def handle_entityref(self, name: str) -> None:
        if not self.is_valid_tag():
            return
        # codepoint = htmlentitydefs.name2codepoint[name]
        # self.result.append(chr(codepoint))
        self.result.append(name)
 

References is_valid_tag(), and result.

Here is the call graph for this function:

handle_starttag()

None searx.utils.HTMLTextExtractor.handle_starttag	(		self,
		str	tag,
		list[tuple[str, str | None]]	attrs )

Definition at line 98 of file utils.py.

    def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
        self.tags.append(tag)
        if tag == 'br':
            self.result.append(' ')
 

References result, searx.result_types.paper.Paper.tags, and tags.

is_valid_tag()

searx.utils.HTMLTextExtractor.is_valid_tag

(

self

)

Definition at line 113 of file utils.py.

    def is_valid_tag(self):
        return not self.tags or self.tags[-1] not in _BLOCKED_TAGS
 

References searx.result_types.paper.Paper.tags, and tags.

Referenced by handle_charref(), handle_data(), and handle_entityref().

Here is the caller graph for this function:

Member Data Documentation

result

searx.utils.HTMLTextExtractor.result = []

Definition at line 95 of file utils.py.

Referenced by get_text(), handle_charref(), handle_data(), handle_endtag(), handle_entityref(), and handle_starttag().

tags

list searx.utils.HTMLTextExtractor.tags = []

Definition at line 96 of file utils.py.

Referenced by handle_endtag(), handle_starttag(), and is_valid_tag().

---

The documentation for this class was generated from the following file:

• /home/andrew/Documents/code/public/searxng/searx/utils.py