Functions
None	request (str query, "OnlineParams" params)
EngineResults	response ("SXNG_Response" resp)
dict[str, int]	time_range_args ("OnlineParams" params)
	detect_google_captcha (ElementType dom)
tuple[list[str], str, str, datetime\|None]	parse_gs_a (str\|None text)

Variables
dict	about
list	categories = ["science", "scientific publications"]
bool	paging = True
int	max_page = 50
bool	language_support = True
bool	time_range_support = True
bool	safesearch = False
bool	send_accept_language_header = True

Detailed Description

Google Scholar is a freely accessible web search engine that indexes the full
text or metadata of scholarly literature across an array of publishing formats
and disciplines.

Compared to other Google services the Scholar engine has a simple GET REST-API
and there does not exists ``async`` API.  Even though the API slightly vintage
we can make use of the :ref:`google API` to assemble the arguments of the GET
request.

Configuration
=============

.. code:: yaml

  - name: google scholar
    engine: google_scholar
    shortcut: gos

Implementations
===============

Function Documentation

◆ detect_google_captcha()

searx.engines.google_scholar.detect_google_captcha ( ElementType dom )

In case of CAPTCHA Google Scholar open its own *not a Robot* dialog and is
not redirected to ``sorry.google.com``.

Definition at line 190 of file google_scholar.py.

def detect_google_captcha(dom: ElementType):
    """In case of CAPTCHA Google Scholar open its own *not a Robot* dialog and is
    not redirected to ``sorry.google.com``.
    """
    if eval_xpath(dom, "//form[@id='gs_captcha_f']"):
        raise SearxEngineCaptchaException()
 
 

◆ parse_gs_a()

tuple[list[str], str, str, datetime | None] searx.engines.google_scholar.parse_gs_a ( str | None text )

Parse the text written in green.

Possible formats:
* "{authors} - {journal}, {year} - {publisher}"
* "{authors} - {year} - {publisher}"
* "{authors} - {publisher}"

Definition at line 198 of file google_scholar.py.

def parse_gs_a(text: str | None) -> tuple[list[str], str, str, datetime | None]:
    """Parse the text written in green.
 
    Possible formats:
    * "{authors} - {journal}, {year} - {publisher}"
    * "{authors} - {year} - {publisher}"
    * "{authors} - {publisher}"
    """
    if text is None or text == "":
        return [], "", "", None
 
    s_text = text.split(" - ")
    authors: list[str] = s_text[0].split(", ")
    publisher: str = s_text[-1]
    if len(s_text) != 3:
        return authors, "", publisher, None
 
    # the format is "{authors} - {journal}, {year} - {publisher}" or "{authors} - {year} - {publisher}"
    # get journal and year
    journal_year = s_text[1].split(", ")
    # journal is optional and may contains some coma
    if len(journal_year) > 1:
        journal: str = ", ".join(journal_year[0:-1])
        if journal == "…":
            journal = ""
    else:
        journal = ""
    # year
    year = journal_year[-1]
    try:
        publishedDate = datetime.strptime(year.strip(), "%Y")
    except ValueError:
        publishedDate = None
    return authors, journal, publisher, publishedDate

◆ request()

None searx.engines.google_scholar.request	(	str	query,
		"OnlineParams"	params )

Google-Scholar search request

Definition at line 76 of file google_scholar.py.

def request(query: str, params: "OnlineParams") -> None:
    """Google-Scholar search request"""
 
    google_info = get_google_info(params, traits)
    # subdomain is: scholar.google.xy
    google_info["subdomain"] = google_info["subdomain"].replace("www.", "scholar.")
 
    args = {
        "q": query,
        **google_info["params"],
        "start": (params["pageno"] - 1) * 10,
        "as_sdt": "2007",  # include patents / to disable set "0,5"
        "as_vis": "0",  # include citations / to disable set "1"
    }
    args.update(time_range_args(params))
 
    params["url"] = "https://" + google_info["subdomain"] + "/scholar?" + urlencode(args)
    params["cookies"] = google_info["cookies"]
    params["headers"].update(google_info["headers"])
 
 

References time_range_args().

Here is the call graph for this function:

◆ response()

EngineResults searx.engines.google_scholar.response ( "SXNG_Response" resp )

Parse response from Google Scholar

Definition at line 97 of file google_scholar.py.

def response(resp: "SXNG_Response") -> EngineResults:  # pylint: disable=too-many-locals
    """Parse response from Google Scholar"""
 
    res = EngineResults()
    dom = html.fromstring(resp.text)
    detect_google_captcha(dom)
 
    # parse results
    for result in eval_xpath_list(dom, "//div[@data-rp]"):
 
        title = extract_text(eval_xpath(result, ".//h3[1]//a"))
        if not title:
            # this is a [ZITATION] block
            continue
 
        pub_type: str = extract_text(eval_xpath(result, ".//span[@class='gs_ctg2']")) or ""
        if pub_type:
            pub_type = pub_type[1:-1].lower()
 
        url: str = eval_xpath_getindex(result, ".//h3[1]//a/@href", 0)
        content: str = extract_text(eval_xpath(result, ".//div[@class='gs_rs']")) or ""
        authors, journal, publisher, publishedDate = parse_gs_a(
            extract_text(eval_xpath(result, ".//div[@class='gs_a']"))
        )
        if publisher in url:
            publisher = ""
 
        # cited by
        comments: str = (
            extract_text(eval_xpath(result, ".//div[@class='gs_fl']/a[starts-with(@href,'/scholar?cites=')]")) or ""
        )
 
        # link to the html or pdf document
        html_url: str = ""
        pdf_url: str = ""
        doc_url = eval_xpath_getindex(result, ".//div[@class='gs_or_ggsm']/a/@href", 0, default=None)
        doc_type = extract_text(eval_xpath(result, ".//span[@class='gs_ctg2']"))
        if doc_type == "[PDF]":
            pdf_url = doc_url
        else:
            html_url = doc_url
 
        res.add(
            res.types.Paper(
                type=pub_type,
                url=url,
                title=title,
                authors=authors,
                publisher=publisher,
                journal=journal,
                publishedDate=publishedDate,
                content=content,
                comments=comments,
                html_url=html_url,
                pdf_url=pdf_url,
            )
        )
 
    # parse suggestion
    for suggestion in eval_xpath(dom, "//div[contains(@class, 'gs_qsuggest_wrap')]//li//a"):
        res.add(res.types.LegacyResult(suggestion=extract_text(suggestion)))
 
    for correction in eval_xpath(dom, "//div[@class='gs_r gs_pda']/a"):
        res.add(res.types.LegacyResult(correction=extract_text(correction)))
    return res
 
 

◆ time_range_args()

dict[str, int] searx.engines.google_scholar.time_range_args ( "OnlineParams" params )

Returns a dictionary with a time range arguments based on
``params["time_range"]``.

Google Scholar supports a detailed search by year.  Searching by *last
month* or *last week* (as offered by SearXNG) is uncommon for scientific
publications and is not supported by Google Scholar.

To limit the result list when the users selects a range, all the SearXNG
ranges (*day*, *week*, *month*, *year*) are mapped to *year*.  If no range
is set an empty dictionary of arguments is returned.

Example; when user selects a time range and we find ourselves in the year
2025 (current year minus one):

.. code:: python

    { "as_ylo" : 2024 }

Definition at line 164 of file google_scholar.py.

def time_range_args(params: "OnlineParams") -> dict[str, int]:
    """Returns a dictionary with a time range arguments based on
    ``params["time_range"]``.
 
    Google Scholar supports a detailed search by year.  Searching by *last
    month* or *last week* (as offered by SearXNG) is uncommon for scientific
    publications and is not supported by Google Scholar.
 
    To limit the result list when the users selects a range, all the SearXNG
    ranges (*day*, *week*, *month*, *year*) are mapped to *year*.  If no range
    is set an empty dictionary of arguments is returned.
 
    Example; when user selects a time range and we find ourselves in the year
    2025 (current year minus one):
 
    .. code:: python
 
        { "as_ylo" : 2024 }
 
    """
    ret_val: dict[str, int] = {}
    if params["time_range"] in time_range_dict:
        ret_val["as_ylo"] = datetime.now().year - 1
    return ret_val
 
 

Referenced by request().

Here is the caller graph for this function:

Variable Documentation

◆ about

dict searx.engines.google_scholar.about

Initial value:

=  {
    "website": "https://scholar.google.com",
    "wikidata_id": "Q494817",
    "official_api_documentation": "https://developers.google.com/custom-search",
    "use_official_api": False,
    "require_api_key": False,
    "results": "HTML",
}

Definition at line 53 of file google_scholar.py.

◆ categories

list searx.engines.google_scholar.categories = ["science", "scientific publications"]

Definition at line 63 of file google_scholar.py.

◆ language_support

bool searx.engines.google_scholar.language_support = True

Definition at line 70 of file google_scholar.py.

◆ max_page

int searx.engines.google_scholar.max_page = 50

Definition at line 65 of file google_scholar.py.

◆ paging

bool searx.engines.google_scholar.paging = True

Definition at line 64 of file google_scholar.py.

◆ safesearch

bool searx.engines.google_scholar.safesearch = False

Definition at line 72 of file google_scholar.py.

◆ send_accept_language_header

bool searx.engines.google_scholar.send_accept_language_header = True

Definition at line 73 of file google_scholar.py.

◆ time_range_support

bool searx.engines.google_scholar.time_range_support = True

Definition at line 71 of file google_scholar.py.

Functions

Variables