Functions
	init (_)
	get_sc_code (searxng_locale, params)
	request (query, params)
tuple[str, datetime\|None]	_parse_published_date (str content)
	_get_web_result (result)
	_get_news_result (result)
dict[str, Any]\|None	_get_image_result (result)
	response (resp)
	fetch_traits (EngineTraits engine_traits)

Variables
logging	logger .Logger
dict	about
str	startpage_categ = 'web'
bool	send_accept_language_header = True
list	categories = ['general', 'web']
bool	paging = True
int	max_page = 18
bool	time_range_support = True
bool	safesearch = True
dict	time_range_dict = {'day': 'd', 'week': 'w', 'month': 'm', 'year': 'y'}
dict	safesearch_dict = {0: '0', 1: '1', 2: '1'}
str	base_url = 'https://www.startpage.com'
str	search_url = base_url + '/sp/search'
str	search_form_xpath = '//form[@id="search"]'
int	sc_code_cache_sec = 3600

Detailed Description

Startpage's language & region selectors are a mess ..

.. _startpage regions:

Startpage regions
=================

In the list of regions there are tags we need to map to common region tags::

  pt-BR_BR --> pt_BR
  zh-CN_CN --> zh_Hans_CN
  zh-TW_TW --> zh_Hant_TW
  zh-TW_HK --> zh_Hant_HK
  en-GB_GB --> en_GB

and there is at least one tag with a three letter language tag (ISO 639-2)::

  fil_PH --> fil_PH

The locale code ``no_NO`` from Startpage does not exists and is mapped to
``nb-NO``::

    babel.core.UnknownLocaleError: unknown locale 'no_NO'

For reference see languages-subtag at iana; ``no`` is the macrolanguage [1]_ and
W3C recommends subtag over macrolanguage [2]_.

.. [1] `iana: language-subtag-registry
   <https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry>`_ ::

      type: language
      Subtag: nb
      Description: Norwegian Bokmål
      Added: 2005-10-16
      Suppress-Script: Latn
      Macrolanguage: no

.. [2]
   Use macrolanguages with care.  Some language subtags have a Scope field set to
   macrolanguage, i.e. this primary language subtag encompasses a number of more
   specific primary language subtags in the registry.  ...  As we recommended for
   the collection subtags mentioned above, in most cases you should try to use
   the more specific subtags ... `W3: The primary language subtag
   <https://www.w3.org/International/questions/qa-choosing-language-tags#langsubtag>`_

.. _startpage languages:

Startpage languages
===================

:py:obj:`send_accept_language_header`:
  The displayed name in Startpage's settings page depend on the location of the
  IP when ``Accept-Language`` HTTP header is unset.  In :py:obj:`fetch_traits`
  we use::

    'Accept-Language': "en-US,en;q=0.5",
    ..

  to get uniform names independent from the IP).

.. _startpage categories:

Startpage categories
====================

Startpage's category (for Web-search, News, Videos, ..) is set by
:py:obj:`startpage_categ` in  settings.yml::

  - name: startpage
    engine: startpage
    startpage_categ: web
    ...

.. hint::

  Supported categories are ``web``, ``news`` and ``images``.

Function Documentation

◆ _get_image_result()

dict[str, Any] | None searx.engines.startpage._get_image_result ( result )

protected

Definition at line 380 of file startpage.py.

def _get_image_result(result) -> dict[str, Any] | None:
    url = result.get('altClickUrl')
    if not url:
        return None
 
    thumbnailUrl = None
    if result.get('thumbnailUrl'):
        thumbnailUrl = base_url + result['thumbnailUrl']
 
    resolution = None
    if result.get('width') and result.get('height'):
        resolution = f"{result['width']}x{result['height']}"
 
    filesize = None
    if result.get('filesize'):
        size_str = ''.join(filter(str.isdigit, result['filesize']))
        filesize = humanize_bytes(int(size_str))
 
    return {
        'template': 'images.html',
        'url': url,
        'title': html_to_text(result['title']),
        'content': '',
        'img_src': result.get('rawImageUrl'),
        'thumbnail_src': thumbnailUrl,
        'resolution': resolution,
        'img_format': result.get('format'),
        'filesize': filesize,
    }
 
 

Referenced by response().

Here is the caller graph for this function:

◆ _get_news_result()

searx.engines.startpage._get_news_result ( result )

protected

Definition at line 358 of file startpage.py.

def _get_news_result(result):
 
    title = remove_pua_from_str(html_to_text(result['title']))
    content = remove_pua_from_str(html_to_text(result.get('description')))
 
    publishedDate = None
    if result.get('date'):
        publishedDate = datetime.fromtimestamp(result['date'] / 1000)
 
    thumbnailUrl = None
    if result.get('thumbnailUrl'):
        thumbnailUrl = base_url + result['thumbnailUrl']
 
    return {
        'url': result['clickUrl'],
        'title': title,
        'content': content,
        'publishedDate': publishedDate,
        'thumbnail': thumbnailUrl,
    }
 
 

Referenced by response().

Here is the caller graph for this function:

◆ _get_web_result()

searx.engines.startpage._get_web_result ( result )

protected

Definition at line 346 of file startpage.py.

def _get_web_result(result):
    content = html_to_text(result.get('description'))
    content, publishedDate = _parse_published_date(content)
 
    return {
        'url': result['clickUrl'],
        'title': html_to_text(result['title']),
        'content': content,
        'publishedDate': publishedDate,
    }
 
 

References _parse_published_date().

Referenced by response().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ _parse_published_date()

tuple[str, datetime | None] searx.engines.startpage._parse_published_date ( str content )

protected

Definition at line 317 of file startpage.py.

def _parse_published_date(content: str) -> tuple[str, datetime | None]:
    published_date = None
 
    # check if search result starts with something like: "2 Sep 2014 ... "
    if re.match(r"^([1-9]|[1-2][0-9]|3[0-1]) [A-Z][a-z]{2} [0-9]{4} \.\.\. ", content):
        date_pos = content.find('...') + 4
        date_string = content[0 : date_pos - 5]
        # fix content string
        content = content[date_pos:]
 
        try:
            published_date = dateutil.parser.parse(date_string, dayfirst=True)
        except ValueError:
            pass
 
    # check if search result starts with something like: "5 days ago ... "
    elif re.match(r"^[0-9]+ days? ago \.\.\. ", content):
        date_pos = content.find('...') + 4
        date_string = content[0 : date_pos - 5]
 
        # calculate datetime
        published_date = datetime.now() - timedelta(days=int(re.match(r'\d+', date_string).group()))  # type: ignore
 
        # fix content string
        content = content[date_pos:]
 
    return content, published_date
 
 

Referenced by _get_web_result().

Here is the caller graph for this function:

◆ fetch_traits()

searx.engines.startpage.fetch_traits ( EngineTraits engine_traits )

Fetch :ref:`languages <startpage languages>` and :ref:`regions <startpage
regions>` from Startpage.

Definition at line 432 of file startpage.py.

def fetch_traits(engine_traits: EngineTraits):
    """Fetch :ref:`languages <startpage languages>` and :ref:`regions <startpage
    regions>` from Startpage."""
    # pylint: disable=too-many-branches
 
    headers = {
        'User-Agent': gen_useragent(),
        'Accept-Language': "en-US,en;q=0.5",  # bing needs to set the English language
    }
    resp = get('https://www.startpage.com/do/settings', headers=headers)
 
    if not resp.ok:  # type: ignore
        print("ERROR: response from Startpage is not OK.")
 
    dom = lxml.html.fromstring(resp.text)  # type: ignore
 
    # regions
 
    sp_region_names = []
    for option in dom.xpath('//form[@name="settings"]//select[@name="search_results_region"]/option'):
        sp_region_names.append(option.get('value'))
 
    for eng_tag in sp_region_names:
        if eng_tag == 'all':
            continue
        babel_region_tag = {'no_NO': 'nb_NO'}.get(eng_tag, eng_tag)  # norway
 
        if '-' in babel_region_tag:
            l, r = babel_region_tag.split('-')
            r = r.split('_')[-1]
            sxng_tag = region_tag(babel.Locale.parse(l + '_' + r, sep='_'))
 
        else:
            try:
                sxng_tag = region_tag(babel.Locale.parse(babel_region_tag, sep='_'))
 
            except babel.UnknownLocaleError:
                print("ERROR: can't determine babel locale of startpage's locale %s" % eng_tag)
                continue
 
        conflict = engine_traits.regions.get(sxng_tag)
        if conflict:
            if conflict != eng_tag:
                print("CONFLICT: babel %s --> %s, %s" % (sxng_tag, conflict, eng_tag))
            continue
        engine_traits.regions[sxng_tag] = eng_tag
 
    # languages
 
    catalog_engine2code = {name.lower(): lang_code for lang_code, name in babel.Locale('en').languages.items()}
 
    # get the native name of every language known by babel
 
    for lang_code in filter(lambda lang_code: lang_code.find('_') == -1, babel.localedata.locale_identifiers()):
        native_name = babel.Locale(lang_code).get_language_name()
        if not native_name:
            print(f"ERROR: language name of startpage's language {lang_code} is unknown by babel")
            continue
        native_name = native_name.lower()
        # add native name exactly as it is
        catalog_engine2code[native_name] = lang_code
 
        # add "normalized" language name (i.e. français becomes francais and español becomes espanol)
        unaccented_name = ''.join(filter(lambda c: not combining(c), normalize('NFKD', native_name)))
        if len(unaccented_name) == len(unaccented_name.encode()):
            # add only if result is ascii (otherwise "normalization" didn't work)
            catalog_engine2code[unaccented_name] = lang_code
 
    # values that can't be determined by babel's languages names
 
    catalog_engine2code.update(
        {
            # traditional chinese used in ..
            'fantizhengwen': 'zh_Hant',
            # Korean alphabet
            'hangul': 'ko',
            # Malayalam is one of 22 scheduled languages of India.
            'malayam': 'ml',
            'norsk': 'nb',
            'sinhalese': 'si',
        }
    )
 
    skip_eng_tags = {
        'english_uk',  # SearXNG lang 'en' already maps to 'english'
    }
 
    for option in dom.xpath('//form[@name="settings"]//select[@name="language"]/option'):
 
        eng_tag = option.get('value')
        if eng_tag in skip_eng_tags:
            continue
        name = extract_text(option).lower()  # type: ignore
 
        sxng_tag = catalog_engine2code.get(eng_tag)
        if sxng_tag is None:
            sxng_tag = catalog_engine2code[name]
 
        conflict = engine_traits.languages.get(sxng_tag)
        if conflict:
            if conflict != eng_tag:
                print("CONFLICT: babel %s --> %s, %s" % (sxng_tag, conflict, eng_tag))
            continue
        engine_traits.languages[sxng_tag] = eng_tag

◆ get_sc_code()

searx.engines.startpage.get_sc_code	(		searxng_locale,
			params )

Get an actual ``sc`` argument from Startpage's search form (HTML page).

Startpage puts a ``sc`` argument on every HTML :py:obj:`search form
<search_form_xpath>`.  Without this argument Startpage considers the request
is from a bot.  We do not know what is encoded in the value of the ``sc``
argument, but it seems to be a kind of a *timestamp*.

Startpage's search form generates a new sc-code on each request.  This
function scrapes a new sc-code from Startpage's home page every
:py:obj:`sc_code_cache_sec` seconds.

Definition at line 180 of file startpage.py.

def get_sc_code(searxng_locale, params):
    """Get an actual ``sc`` argument from Startpage's search form (HTML page).
 
    Startpage puts a ``sc`` argument on every HTML :py:obj:`search form
    <search_form_xpath>`.  Without this argument Startpage considers the request
    is from a bot.  We do not know what is encoded in the value of the ``sc``
    argument, but it seems to be a kind of a *timestamp*.
 
    Startpage's search form generates a new sc-code on each request.  This
    function scrapes a new sc-code from Startpage's home page every
    :py:obj:`sc_code_cache_sec` seconds."""
 
    sc_code = CACHE.get("SC_CODE")
 
    if sc_code:
        logger.debug("get_sc_code: using cached value: %s", sc_code)
        return sc_code
 
    headers = {**params['headers']}
 
    # add Accept-Language header
    if searxng_locale == 'all':
        searxng_locale = 'en-US'
    locale = babel.Locale.parse(searxng_locale, sep='-')
 
    if send_accept_language_header:
        ac_lang = locale.language
        if locale.territory:
            ac_lang = "%s-%s,%s;q=0.9,*;q=0.5" % (
                locale.language,
                locale.territory,
                locale.language,
            )
        headers['Accept-Language'] = ac_lang
 
    get_sc_url = base_url + '/'
    logger.debug("get_sc_code: querying new sc timestamp @ %s", get_sc_url)
    logger.debug("get_sc_code: request headers: %s", headers)
    resp = get(get_sc_url, headers=headers)
 
    # ?? x = network.get('https://www.startpage.com/sp/cdn/images/filter-chevron.svg', headers=headers)
    # ?? https://www.startpage.com/sp/cdn/images/filter-chevron.svg
    # ?? ping-back URL: https://www.startpage.com/sp/pb?sc=TLsB0oITjZ8F21
 
    if str(resp.url).startswith('https://www.startpage.com/sp/captcha'):  # type: ignore
        raise SearxEngineCaptchaException(
            message="get_sc_code: got redirected to https://www.startpage.com/sp/captcha",
        )
 
    dom = lxml.html.fromstring(resp.text)  # type: ignore
 
    try:
        sc_code = eval_xpath(dom, search_form_xpath + '//input[@name="sc"]/@value')[0]
    except IndexError as exc:
        logger.debug("suspend startpage API --> https://github.com/searxng/searxng/pull/695")
        raise SearxEngineCaptchaException(
            message="get_sc_code: [PR-695] querying new sc timestamp failed! (%s)" % resp.url,  # type: ignore
        ) from exc
 
    sc_code = str(sc_code)
    logger.debug("get_sc_code: new value is: %s", sc_code)
    CACHE.set(key="SC_CODE", value=sc_code, expire=sc_code_cache_sec)
    return sc_code
 
 

Referenced by request().

Here is the caller graph for this function:

◆ init()

searx.engines.startpage.init ( _ )

Definition at line 168 of file startpage.py.

def init(_):
    global CACHE  # pylint: disable=global-statement
 
    # hint: all three startpage engines (WEB, Images & News) can/should use the
    # same sc_code ..
    CACHE = EngineCache("startpage")  # type:ignore
 
 

◆ request()

searx.engines.startpage.request	(		query,
			params )

Assemble a Startpage request.

To avoid CAPTCHAs we need to send a well formed HTTP POST request with a
cookie. We need to form a request that is identical to the request built by
Startpage's search form:

- in the cookie the **region** is selected
- in the HTTP POST data the **language** is selected

Additionally the arguments form Startpage's search form needs to be set in
HTML POST data / compare ``<input>`` elements: :py:obj:`search_form_xpath`.

Definition at line 245 of file startpage.py.

def request(query, params):
    """Assemble a Startpage request.
 
    To avoid CAPTCHAs we need to send a well formed HTTP POST request with a
    cookie. We need to form a request that is identical to the request built by
    Startpage's search form:
 
    - in the cookie the **region** is selected
    - in the HTTP POST data the **language** is selected
 
    Additionally the arguments form Startpage's search form needs to be set in
    HTML POST data / compare ``<input>`` elements: :py:obj:`search_form_xpath`.
    """
    engine_region = traits.get_region(params['searxng_locale'], 'en-US')
    engine_language = traits.get_language(params['searxng_locale'], 'en')
 
    params['headers']['Origin'] = base_url
    params['headers']['Referer'] = base_url + '/'
 
    # Build form data
    args = {
        'query': query,
        'cat': startpage_categ,
        't': 'device',
        'sc': get_sc_code(params['searxng_locale'], params),  # hint: this func needs HTTP headers
        'with_date': time_range_dict.get(params['time_range'], ''),
        'abp': '1',
        'abd': '1',
        'abe': '1',
    }
 
    if engine_language:
        args['language'] = engine_language
        args['lui'] = engine_language
 
    if params['pageno'] > 1:
        args['page'] = params['pageno']
        args['segment'] = 'startpage.udog'
 
    # Build cookie
    lang_homepage = 'en'
    cookie = OrderedDict()
    cookie['date_time'] = 'world'
    cookie['disable_family_filter'] = safesearch_dict[params['safesearch']]
    cookie['disable_open_in_new_window'] = '0'
    cookie['enable_post_method'] = '1'  # hint: POST
    cookie['enable_proxy_safety_suggest'] = '1'
    cookie['enable_stay_control'] = '1'
    cookie['instant_answers'] = '1'
    cookie['lang_homepage'] = 's/device/%s/' % lang_homepage
    cookie['num_of_results'] = '10'
    cookie['suggestions'] = '1'
    cookie['wt_unit'] = 'celsius'
 
    if engine_language:
        cookie['language'] = engine_language
        cookie['language_ui'] = engine_language
 
    if engine_region:
        cookie['search_results_region'] = engine_region
 
    params['cookies']['preferences'] = 'N1N'.join(["%sEEE%s" % x for x in cookie.items()])
    logger.debug('cookie preferences: %s', params['cookies']['preferences'])
 
    logger.debug("data: %s", args)
    params['data'] = args
    params['method'] = 'POST'
    params['url'] = search_url
 
    return params
 
 

References get_sc_code().

Here is the call graph for this function:

◆ response()

searx.engines.startpage.response ( resp )

Definition at line 411 of file startpage.py.

def response(resp):
    categ = startpage_categ.capitalize()
    results_raw = '{' + extr(resp.text, f"React.createElement(UIStartpage.AppSerp{categ}, {{", '}})') + '}}'
    results_json = loads(results_raw)
    results_obj = results_json.get('render', {}).get('presenter', {}).get('regions', {})
 
    results = []
    for results_categ in results_obj.get('mainline', []):
        for item in results_categ.get('results', []):
            if results_categ['display_type'] == 'web-google':
                results.append(_get_web_result(item))
            elif results_categ['display_type'] == 'news-bing':
                results.append(_get_news_result(item))
            elif 'images' in results_categ['display_type']:
                item = _get_image_result(item)
                if item:
                    results.append(item)
 
    return results
 
 

References _get_image_result(), _get_news_result(), and _get_web_result().

Here is the call graph for this function:

Variable Documentation

◆ about

dict searx.engines.startpage.about

Initial value:

=  {
    "website": 'https://startpage.com',
    "wikidata_id": 'Q2333295',
    "official_api_documentation": None,
    "use_official_api": False,
    "require_api_key": False,
    "results": 'HTML',
}

Definition at line 109 of file startpage.py.

◆ base_url

str searx.engines.startpage.base_url = 'https://www.startpage.com'

Definition at line 141 of file startpage.py.

◆ categories

list searx.engines.startpage.categories = ['general', 'web']

Definition at line 129 of file startpage.py.

◆ logger

logging searx.engines.startpage.logger .Logger

Definition at line 104 of file startpage.py.

◆ max_page

int searx.engines.startpage.max_page = 18

Definition at line 131 of file startpage.py.

◆ paging

bool searx.engines.startpage.paging = True

Definition at line 130 of file startpage.py.

◆ safesearch

bool searx.engines.startpage.safesearch = True

Definition at line 135 of file startpage.py.

◆ safesearch_dict

dict searx.engines.startpage.safesearch_dict = {0: '0', 1: '1', 2: '1'}

Definition at line 138 of file startpage.py.

◆ sc_code_cache_sec

int searx.engines.startpage.sc_code_cache_sec = 3600

Definition at line 176 of file startpage.py.

◆ search_form_xpath

str searx.engines.startpage.search_form_xpath = '//form[@id="search"]'

Definition at line 147 of file startpage.py.

◆ search_url

str searx.engines.startpage.search_url = base_url + '/sp/search'

Definition at line 142 of file startpage.py.

◆ send_accept_language_header

bool searx.engines.startpage.send_accept_language_header = True

Definition at line 122 of file startpage.py.

◆ startpage_categ

str searx.engines.startpage.startpage_categ = 'web'

Definition at line 118 of file startpage.py.

◆ time_range_dict

dict searx.engines.startpage.time_range_dict = {'day': 'd', 'week': 'w', 'month': 'm', 'year': 'y'}

Definition at line 137 of file startpage.py.

◆ time_range_support

bool searx.engines.startpage.time_range_support = True

Definition at line 134 of file startpage.py.

Functions

Variables

Detailed Description

Function Documentation

◆ _get_image_result()

◆ _get_news_result()

◆ _get_web_result()

◆ _parse_published_date()

◆ fetch_traits()

◆ get_sc_code()

◆ init()

◆ request()

◆ response()

Variable Documentation

◆ about

◆ base_url

◆ categories

◆ logger

◆ max_page

◆ paging

◆ safesearch

◆ safesearch_dict

◆ sc_code_cache_sec

◆ search_form_xpath

◆ search_url

◆ send_accept_language_header

◆ startpage_categ

◆ time_range_dict

◆ time_range_support