searx.engines.github_code Namespace Reference

Functions
None	request (str query, dict[str, t.Any] params)
tuple[list[str], set[int]]	extract_code (list[dict[str, t.Any]] code_matches)
EngineResults	response (SXNG_Response resp)

Variables
dict	about
list	categories = ['code']
str	search_url = 'https://api.github.com/search/code?sort=indexed&{query}&{page}'
str	accept_header = 'application/vnd.github.text-match+json'
bool	paging = True
dict	ghc_auth
bool	ghc_highlight_matching_lines = True
bool	ghc_strip_new_lines = True
bool	ghc_strip_whitespace = False
str	ghc_api_version = "2022-11-28"
bool	ghc_insert_block_separator = False

Detailed Description

GitHub code search with `search syntax`_ as described in `Constructing a
search query`_ in the documentation of GitHub's REST API.

.. _search syntax:
    https://docs.github.com/en/search-github/getting-started-with-searching-on-github/understanding-the-search-syntax
.. _Constructing a search query:
    https://docs.github.com/en/rest/search/search?apiVersion=2022-11-28#constructing-a-search-query
.. _Github REST API for code search:
    https://docs.github.com/en/rest/search/search?apiVersion=2022-11-28#search-code
.. _Github REST API auth for code search:
    https://docs.github.com/en/rest/search/search?apiVersion=2022-11-28#search-code--fine-grained-access-tokens

Configuration
=============

The engine has the following mandatory setting:

- :py:obj:`ghc_auth`
  Change the authentication method used when using the API, defaults to none.

Optional settings are:

- :py:obj:`ghc_highlight_matching_lines`
   Control the highlighting of the matched text (turns off/on).
- :py:obj:`ghc_strip_new_lines`
   Strip new lines at the start or end of each code fragment.
- :py:obj:`ghc_strip_whitespace`
   Strip any whitespace at the start or end of each code fragment.
- :py:obj:`ghc_insert_block_separator`
   Add a `...` between each code fragment before merging them.

.. code:: yaml

  - name: github code
    engine: github_code
    shortcut: ghc
    ghc_auth:
      type: "none"

  - name: github code
    engine: github_code
    shortcut: ghc
    ghc_auth:
      type: "personal_access_token"
      token: "<token>"
    ghc_highlight_matching_lines: true
    ghc_strip_whitespace: true
    ghc_strip_new_lines: true


  - name: github code
    engine: github_code
    shortcut: ghc
    ghc_auth:
      type: "bearer"
      token: "<token>"

Implementation
===============

GitHub does not return the code line indices alongside the code fragment in the
search API. Since these are not super important for the user experience all the
code lines are just relabeled (starting from 1) and appended (a disjoint set of
code blocks in a single file might be returned from the API).

Function Documentation

extract_code()

tuple[list[str], set[int]] searx.engines.github_code.extract_code

(

list[dict[str, t.Any]]

code_matches

)

Iterate over multiple possible matches, for each extract a code fragment.
Github additionally sends context for _word_ highlights; pygments supports
highlighting lines, as such we calculate which lines to highlight while
traversing the text.

Definition at line 162 of file github_code.py.

def extract_code(code_matches: list[dict[str, t.Any]]) -> tuple[list[str], set[int]]:
    """
    Iterate over multiple possible matches, for each extract a code fragment.
    Github additionally sends context for _word_ highlights; pygments supports
    highlighting lines, as such we calculate which lines to highlight while
    traversing the text.
    """
    lines: list[str] = []
    highlighted_lines_index: set[int] = set()
 
    for i, match in enumerate(code_matches):
        if i > 0 and ghc_insert_block_separator:
            lines.append("...")
        buffer: list[str] = []
        highlight_groups = [highlight_group['indices'] for highlight_group in match['matches']]
 
        code: str = match['fragment']
        original_code_lenght = len(code)
 
        if ghc_strip_whitespace:
            code = code.lstrip()
        if ghc_strip_new_lines:
            code = code.lstrip("\n")
 
        offset = original_code_lenght - len(code)
 
        if ghc_strip_whitespace:
            code = code.rstrip()
        if ghc_strip_new_lines:
            code = code.rstrip("\n")
 
        for i, letter in enumerate(code):
            if len(highlight_groups) > 0:
                # the API ensures these are sorted already, and we have a
                # guaranteed match in the code (all indices are in the range 0
                # and len(fragment)), so only check the first highlight group
                [after, before] = highlight_groups[0]
                if after <= (i + offset) < before:
                    # pygments enumerates lines from 1, highlight the next line
                    highlighted_lines_index.add(len(lines) + 1)
                    highlight_groups.pop(0)
 
            if letter == "\n":
                lines.append("".join(buffer))
                buffer = []
                continue
 
            buffer.append(letter)
        lines.append("".join(buffer))
    return lines, highlighted_lines_index
 
 

Referenced by response().

Here is the caller graph for this function:

request()

None searx.engines.github_code.request	(	str	query,
		dict[str, t.Any]	params )

Definition at line 144 of file github_code.py.

def request(query: str, params: dict[str, t.Any]) -> None:
 
    params['url'] = search_url.format(query=urlencode({'q': query}), page=urlencode({'page': params['pageno']}))
    params['headers']['Accept'] = accept_header
    params['headers']['X-GitHub-Api-Version'] = ghc_api_version
 
    if ghc_auth['type'] == "none":
        # Without the auth header the query fails, so add a dummy instead.
        # Queries without auth are heavily rate limited.
        params['headers']['Authorization'] = "placeholder"
    if ghc_auth['type'] == "personal_access_token":
        params['headers']['Authorization'] = f"token {ghc_auth['token']}"
    if ghc_auth['type'] == "bearer":
        params['headers']['Authorization'] = f"Bearer {ghc_auth['token']}"
 
    params['raise_for_httperror'] = False
 
 

response()

EngineResults searx.engines.github_code.response

(

SXNG_Response

resp

)

Definition at line 214 of file github_code.py.

def response(resp: SXNG_Response) -> EngineResults:
    res = EngineResults()
 
    if resp.status_code == 422:
        # on a invalid search term the status code 422 "Unprocessable Content"
        # is returned / e.g. search term is "user: foo" instead "user:foo"
        return res
    # raise for other errors
    raise_for_httperror(resp)
 
    for item in resp.json().get('items', []):
        repo: dict[str, str] = item['repository']  # pyright: ignore[reportAny]
        text_matches: list[dict[str, str]] = item['text_matches']  # pyright: ignore[reportAny]
        # ensure picking only the code contents in the blob
        code_matches = [
            match for match in text_matches if match["object_type"] == "FileContent" and match["property"] == "content"
        ]
        lines, highlighted_lines_index = extract_code(code_matches)
        if not ghc_highlight_matching_lines:
            highlighted_lines_index: set[int] = set()
 
        res.add(
            res.types.Code(
                url=item["html_url"],  # pyright: ignore[reportAny]
                title=f"{repo['full_name']} · {item['name']}",
                filename=f"{item['path']}",
                content=repo['description'],
                repository=repo['html_url'],
                codelines=[(i + 1, line) for (i, line) in enumerate(lines)],
                hl_lines=highlighted_lines_index,
                strip_whitespace=ghc_strip_whitespace,
                strip_new_lines=ghc_strip_new_lines,
            )
        )
 
    return res

References extract_code().

Here is the call graph for this function:

Variable Documentation

about

dict searx.engines.github_code.about

Initial value:

=  {
    "website": 'https://github.com/',
    "wikidata_id": 'Q364',
    "official_api_documentation": 'https://docs.github.com/en/rest/search/search?apiVersion=2022-11-28#search-code',
    "use_official_api": True,
    "require_api_key": False,
    "results": 'JSON',
}

Definition at line 77 of file github_code.py.

accept_header

str searx.engines.github_code.accept_header = 'application/vnd.github.text-match+json'

Definition at line 92 of file github_code.py.

categories

list searx.engines.github_code.categories = ['code']

Definition at line 87 of file github_code.py.

ghc_api_version

str searx.engines.github_code.ghc_api_version = "2022-11-28"

Definition at line 133 of file github_code.py.

ghc_auth

dict searx.engines.github_code.ghc_auth

Initial value:

=  {
    "type": "none",
    "token": "",
}

Definition at line 95 of file github_code.py.

ghc_highlight_matching_lines

bool searx.engines.github_code.ghc_highlight_matching_lines = True

Definition at line 119 of file github_code.py.

ghc_insert_block_separator

bool searx.engines.github_code.ghc_insert_block_separator = False

Definition at line 137 of file github_code.py.

ghc_strip_new_lines

bool searx.engines.github_code.ghc_strip_new_lines = True

Definition at line 122 of file github_code.py.

ghc_strip_whitespace

bool searx.engines.github_code.ghc_strip_whitespace = False

Definition at line 127 of file github_code.py.

paging

bool searx.engines.github_code.paging = True

Definition at line 93 of file github_code.py.

search_url

str searx.engines.github_code.search_url = 'https://api.github.com/search/code?sort=indexed&{query}&{page}'

Definition at line 90 of file github_code.py.