searx.engines.semantic_scholar Namespace Reference

Functions
	_get_ui_version ()
	request (query, params)
	response (resp)

Variables
dict	about
list	categories = ['science', 'scientific publications']
bool	paging = True
str	search_url = 'https://www.semanticscholar.org/api/1/search'
str	base_url = 'https://www.semanticscholar.org'

Detailed Description

Semantic Scholar (Science)

Function Documentation

_get_ui_version()

searx.engines.semantic_scholar._get_ui_version ( )

protected

Definition at line 28 of file semantic_scholar.py.

def _get_ui_version():
    resp = get(base_url)
    if not resp.ok:
        raise RuntimeError("Can't determine Semantic Scholar UI version")
 
    doc = html.fromstring(resp.text)
    ui_version = eval_xpath_getindex(doc, "//meta[@name='s2-ui-version']/@content", 0)
    if not ui_version:
        raise RuntimeError("Can't determine Semantic Scholar UI version")
 
    return ui_version
 
 

Referenced by request().

Here is the caller graph for this function:

request()

searx.engines.semantic_scholar.request	(		query,
			params )

Definition at line 41 of file semantic_scholar.py.

def request(query, params):
    params['url'] = search_url
    params['method'] = 'POST'
    params['headers'] = {
        'Content-Type': 'application/json',
        'X-S2-UI-Version': _get_ui_version(),
        'X-S2-Client': "webapp-browser",
        'User-Agent': gen_useragent(),
    }
    params['data'] = dumps(
        {
            "queryString": query,
            "page": params['pageno'],
            "pageSize": 10,
            "sort": "relevance",
            "getQuerySuggestions": False,
            "authors": [],
            "coAuthors": [],
            "venues": [],
            "performTitleMatch": True,
        }
    )
    return params
 
 

References _get_ui_version().

Here is the call graph for this function:

response()

searx.engines.semantic_scholar.response

(

resp

)

Definition at line 66 of file semantic_scholar.py.

def response(resp):
    res = resp.json()
 
    results = []
    for result in res['results']:
        url = result.get('primaryPaperLink', {}).get('url')
        if not url and result.get('links'):
            url = result.get('links')[0]
        if not url:
            alternatePaperLinks = result.get('alternatePaperLinks')
            if alternatePaperLinks:
                url = alternatePaperLinks[0].get('url')
        if not url:
            url = base_url + '/paper/%s' % result['id']
 
        # publishedDate
        if 'pubDate' in result:
            publishedDate = datetime.strptime(result['pubDate'], "%Y-%m-%d")
        else:
            publishedDate = None
 
        # authors
        authors = [author[0]['name'] for author in result.get('authors', [])]
 
        # pick for the first alternate link, but not from the crawler
        pdf_url = None
        for doc in result.get('alternatePaperLinks', []):
            if doc['linkType'] not in ('crawler', 'doi'):
                pdf_url = doc['url']
                break
 
        # comments
        comments = None
        if 'citationStats' in result:
            comments = gettext(
                '{numCitations} citations from the year {firstCitationVelocityYear} to {lastCitationVelocityYear}'
            ).format(
                numCitations=result['citationStats']['numCitations'],
                firstCitationVelocityYear=result['citationStats']['firstCitationVelocityYear'],
                lastCitationVelocityYear=result['citationStats']['lastCitationVelocityYear'],
            )
 
        results.append(
            {
                'template': 'paper.html',
                'url': url,
                'title': result['title']['text'],
                'content': html_to_text(result['paperAbstract']['text']),
                'journal': result.get('venue', {}).get('text') or result.get('journal', {}).get('name'),
                'doi': result.get('doiInfo', {}).get('doi'),
                'tags': result.get('fieldsOfStudy'),
                'authors': authors,
                'pdf_url': pdf_url,
                'publishedDate': publishedDate,
                'comments': comments,
            }
        )
 
    return results

Variable Documentation

about

dict searx.engines.semantic_scholar.about

Initial value:

=  {
    "website": 'https://www.semanticscholar.org/',
    "wikidata_id": 'Q22908627',
    "official_api_documentation": 'https://api.semanticscholar.org/',
    "use_official_api": True,
    "require_api_key": False,
    "results": 'JSON',
}

Definition at line 13 of file semantic_scholar.py.

base_url

str searx.engines.semantic_scholar.base_url = 'https://www.semanticscholar.org'

Definition at line 25 of file semantic_scholar.py.

categories

list searx.engines.semantic_scholar.categories = ['science', 'scientific publications']

Definition at line 22 of file semantic_scholar.py.

paging

bool searx.engines.semantic_scholar.paging = True

Definition at line 23 of file semantic_scholar.py.

search_url

str searx.engines.semantic_scholar.search_url = 'https://www.semanticscholar.org/api/1/search'

Definition at line 24 of file semantic_scholar.py.