Setting

Description

Example/Default setting

Crawling Root*

Specifies one or more starting points (= roots) for the crawl. Each crawling root defines where the crawler begins. It is also possible to define multiple crawling roots to seed the crawl from different locations.

The activation or deactivation of the setting “Delta Crawling” has the following effect on the defined crawling roots:

Example:

https://inspire.mindbreeze.com/blog/

URL Regex

Defines a pattern with a regular expression, whereby links that match the pattern are crawled. The pattern must match the entire URL, including parameters.

If left empty, all pages that have the same host and domain components as the “Crawling Root” are indexed (except rules specified in the settings “Include URL by Metadata” or “Exclude URL by Metadata”).

Hint: Please note the following hierarchy when using certain settings simultaneously:

1. URL Regex – crawls pages that match the pattern.
2. URL Exclude Pattern – excludes pages that match the pattern.
3. Include/Exclude URL by Metadata – filters pages that match the patterns.

Example:

^https?://example\.com/.*$

URL Exclude Pattern

Specifies a pattern for URLs that should be excluded. URLs matching this pattern will not be crawled or used for further link extraction. The pattern must match the entire URL, including parameters.

Hint: Please note the following hierarchy when using certain settings simultaneously:

1. URL Regex – crawls pages that match the pattern.
2. URL Exclude Pattern – excludes pages that match the pattern.
3. Include/Exclude URL by Metadata – filters pages that match the patterns.

Example:

^https?://(?:www\.)?example\.com/(login|checkout).*$

Include URL by Metadata

Can be used for the crawling of sitemaps to filter web pages based on metadata. If a page’s metadata matches the defined pattern, the page will be crawled.

The following settings must be configured to define the pattern:

For more information on how to specify metadata in sitemaps, see the chapter “Mindbreeze-Extension of the Sitemaps.org Protocol”.

Hint: Please note the following hierarchy when using certain settings simultaneously:

1. URL Regex – crawls pages that match the pattern.
2. URL Exclude Pattern – excludes pages that match the pattern.
3. Include/Exclude URL by Metadata – filters pages that match the patterns.

Exclude URL by Metadata

Can be used for the crawling of sitemaps to filter web pages based on metadata. If a page’s metadata matches the defined pattern, the page will not be crawled.

The following settings must be configured to define the pattern:

For more information on how to specify metadata in sitemaps, see the chapter “Mindbreeze-Extension of the Sitemaps.org Protocol”.

Hint: Please note the following hierarchy when using certain settings simultaneously:

1. URL Regex – crawls pages that match the pattern.
2. URL Exclude Pattern – excludes pages that match the pattern.
3. Include/Exclude URL by Metadata – filters pages that match the patterns.

Convert Document Keys to Lower Case

Converts document keys (header/mes:key metadatum) of the indexed documents/websites to lower case.

Default Value:

Deactivated

Thumbnailer URL Exclude Pattern

Defines, via a pattern, a blacklist of URLs that the thumbnail generator should ignore.

When a document is indexed and a thumbnail is created, any network request whose target URL matches the specified pattern is skipped, thereby preventing unwanted external calls.

Attention: This setting only works with HTML documents. Also, make sure that the filter “FilterPlugin.JerichoWithThumbnails” is active for HTML files.

Example:

^https?://unwanted\.host\.com\/.*$

Maximum Link Depth

Sets the maximum number of hops allowed from the crawling root to descendant pages. URLs discovered that would require more hops than this limit are ignored.

A value of “1”, for example, would be appropriate for a sitemap.

Example:

5

User Agent

Specifies the string “User-Agent” sent with every HTTP request which identifies the crawler to the web servers.

Default setting:

Mozilla/5.0 (compatible; heritrix/@VERSION@ +http://www.mindbreeze.com)

Additional Hosts File

Specifies an additional hosts file that is used for DNS resolution while crawling. This setting can help when DNS resolution for certain web servers fails for network‑related reasons.

The file must follow the same format as /etc/hosts.

Example:

/data/hosts/custom-host-resolution

Ignore Proxy

With the activation of this setting, the crawler bypasses any configured proxy settings for requests.

Default setting:

Disabled

Ignore SSL/Certificate Errors

(Advanced)

With the activation of this setting, the crawler will accept invalid or self-signed SSL certificates.

Default setting:

Disabled

Accept-Headers

(Advanced)

Defines additional Accept-* Headers (e.g., Accept-Language, Accept-Encoding) that the crawler should send with HTTP requests.

Use this setting specifically for content negotiation headers that tell the server which response formats are acceptable.

The Accept Header only takes effect in the internal Heritrix and thus are ineffective for situations like Authentication Requests performed by the Crawler.

Example Value:

Accept-Language:en-US,en;q=0.9  

Incomplete Delta Crawl Runs

(Advanced)

With the activation of this setting, pages that are not reachable from the current Crawling Root are not deleted from the index at the end of the crawl run.

To minimize the load of subsequent crawl runs on your site, you can provide a crawling root with links to updated pages only.

Attention: Do not use this setting for sitemap delta crawling. For more information, see the chapter. “Sitemap Crawling Strategy”.

Default setting:

Disabled

Delta Crawling

• With the activation of this setting, the crawler retrieves the web pages that are listed in the sitemap exclusively. Here, both the attributes “lastmod” and “changefreq” of the sitemap pages are compared with the modification date of the indexed pages (if available).
• See the chapter “Sitemap Crawling Strategy” for an explanation in detail.

Attention: This setting should not be enabled, if the setting “Allowed Delete Ratio” is enabled.

Default setting:

Disabled

Pass Sitemap ACL and Metadata to Redirect Target URLs

(Advanced)

With the activation of this setting, any sitemap URL that redirects to another URL will pass along its metadata and access control (ACL) settings to the redirect target URLs.

Attention: This setting only works, if the setting “Support Redirects in Crawling Roots” is enabled as well.

Default setting:

Enabled

Enable Default ACLs

(Advanced)

When this setting is enabled, the crawler automatically assigns ACLs to web documents that do not already have explicitly defined ACLs (for example, via sitemaps that contain <mes:acl>).

The ACLs that are applied in these cases are defined by the setting “Default ACL Principals”.

If this setting is deactivated, documents without explicit ACLs are left without any access control information.

If this setting is disabled while the setting “Default ACL Principals” is enabled, the ACL principal defined in “Default ACL Principals” (default: everyone) will be used.

Default setting:

Enabled

Default ACL Principals

(Advanced)

Specifies the principals that will be used for the default ACLs. Enter one principal per line. Multiple default ACL principals can be specified by separating them with line breaks.

Please be aware of the following points:

• If the list is empty while the setting “Enable Default ACLs” is activated, the crawler uses the builtin principal “everyone”.
• The values provided are applied to every document that receives a default ACL.

Default setting:

everyone

Example:

• max.mustermann@mindbreeze.com
• sales-department (resolvable by a cache)

Inherit Crawling Root Query Parameter Pattern

(Advanced)

Enables the inheritance of URL query parameters from the crawling root to its child URLs (= discovered URLs).

The value can be any regular expression that matches the names of the query parameters. If a child URL already has query parameters with the same name, they will be overwritten by the query parameter defined on the crawling root.

This setting can, for example, be used on web pages that deliver different content depending on the query parameters.

For example, the following crawling roots provide different content with the query parameter “location”:

• https://mysite.com/?location=us
• https://mysite.com/?location=en

Likewise, the following child URLs provide different content with the same query parameter “location”.

• https://mysite.com/events/sponsored?location=us
• https://mysite.com/events/sponsored?location=de

To apply the “location” query parameter from the crawling root to the child URLs, set this setting to the value “location.”

Example:

• location
• ^(lang|locale)$

Use extension defined in metadata for sitemap URLs

If the sitemap URL contains the metadata field mes:extension, the crawler will use the value of this metadata to enforce the file extension when processing the sitemap.

Default setting:

Disabled

Enforce Extension From URL if Matches

Defines a pattern that is matched against crawled URLs. If a URL matches the pattern, the “Extension” metadata is derived from the URL itself rather than from the HTTP header “Content‑Type”.

Example:

.*\\.(pdf|docx|xlsx)$

Cleanup non matching URL-s from Index

When this setting is enabled, all URLs are removed from the index that do not match the patterns configured in the setting “Include URL by Metadata” or that match the patterns configured in the setting “Exclude URL by Metadata”.

Default setting:

Disabled

Invalid document deletion Schedule

Defines a cron-like schedule to delete invalid or outdated documents from the index.

To define a schedule, an extended cron format must be used:

second, minute, hour, day, month, day-of-week.

Example Value:

0 30 2 * * ?

Skip Up-to-date Documents

(Advanced)

When this setting is enabled, a document will only be fully processed (= for example, content extracting, checking for content changes) if either mes:date (see here) or the access information has changed since the last crawl run.

Default setting:

Enabled

Retrieve Modification Date from Index

(Advanced)

When this setting is enabled, the index retrieves the existing modification date of an already indexed document before it is reindexed. This value is stored in the metadata field mes:existingmodificationdate.

This enables the comparison between the previously indexed modification date and the currently indexed modification date of the document to gain insights into changes over time.

Default setting:

Disabled

Use Stream Sitemap Parser

(Advanced)

When this setting is enabled, a streaming parser for sitemaps is used, which is more memory-efficient for very large sitemaps, but less tolerant of XML errors.

Default setting:

Disabled

Enable GZIP Compressed Sitemaps

(Advanced)

When this setting is enabled, the crawler can automatically read and parse GZIP compressed sitemap files.

Default setting:

Disabled

Sitemap Metadata Prefix

(Advanced)

This setting is useful for preventing key collisions or namespace metadata.

It defines a prefix that will be included in the keys used by the crawler when storing sitemap metadata.

Example:

• sitemap_
• sitemap_meta

Use Hashing Queue Assignment Policy

(Advanced)

When this setting is enabled, a hash-based algorithm is used to distribute the URIs over the queues. The hash-based distribution allows for faster crawls, but puts more load on the target server.

If this setting is disabled, the URIs are distributed based on their host.

Default setting:

Enabled

Parallel queue count

(Advanced)

Defines the number of queues used for parallel crawling.

Attention: This setting is only effective, when the setting „Use Hashing Queue Assignment Policy“ is enabled.

Default setting:

5

Max Document Size (MB)

(Advanced)

Defines the maximum allowed file size of documents that are downloaded.

If a document is larger than the defined value, the document is truncated. The truncation can make documents invalid and should be treated with care (for example, invalid truncated PDF).

A value of 0 means that no maximum file size is defined.

Default setting:

50 MB

Robots Honoring Policy

Determines how the crawler handles robots.txt rules when accessing websites.

The following options are available:

Default setting:

Obey all robots.txt rules for configured user

Website Cache Directory

Defines the path, where the crawler should create the cache directory.

If the path is defined, the caching is enabled and HTTP communication is going to be stored/cached locally for later reuse.

Example:

/data/web/cache

Maximum Mirror Database Size (MB)

Defines the maximum size of the local cache database in megabytes.

Default setting:

512 MB

Use Cache Only

When this setting is enabled, the crawler will only use cached content and will not perform any network requests.

Default setting:

Disabled

Maximum Number of Extracted Links

Defines a maximum limit for the number of links that the crawler will extract from a single page.

When this limit is reached, no additional links on that page are followed or queued for crawling.

Default setting:

6000

HTTP Request Header

Defines any custom HTTP headers to include with each request (e.g. custom client headers).

Example:

• Cache-Control: no-cache  
• Cookie: theme=light

Content Signature Type

(Advanced)

For this setting to work, a plugin is required. The plugin is available at https://www.mindbreeze.com/support/updates-pi.

Can be used in combination with the filter “PostFilterTransformerPlugin.SignatureToKeyRewriter” to deduplicate documents based on their content rather than just their ID (mes:key). Here, the filter “PostFilterTransformerPlugin.SignatureToKeyRewriter“ must be configured as a Post-Filter-Transformer.

The following options are available:

Default setting:

Text Profile Signature

Disable Diffie-Hellman Ciphers

(Advanced)

When this setting is enabled, the crawler prevents the use of Diffie-Hellman key exchange ciphers for HTTPS connections.

Enabling this setting improves security by avoiding weak implementations of Diffie-Hellman key exchange ciphers. However, it should be noted that connections to older servers that only support Diffie-Hellman key exchange ciphers may be prevented.

Default setting:

Disabled

Support Redirects in Crawling Roots

(Advanced)

When this setting is enabled, the crawler follows HTTP redirects when accessing the specified Crawling Roots.

Default setting:

Enabled

HTTP Protocol Version

(Advanced)

Defines the HTTP protocol version the crawler should use when fetching websites. Only changes the protocol version in the HTTP request. Features or performance are not affected.

This setting should only be changed, when the target server rejects the currently selected version.

Default setting:

HTTP/1.1

Option

Description

Sitemap-based Incomplete

With this option, the URL entries from the configured sitemaps will be crawled and the already indexed URLs, which are not found in the sitemaps, are left in the index.

Sitemap-based Complete

With this option, the URL entries from the configured sitemaps will be crawled and the already indexed URLs, which are not found in the sitemaps, are deleted from the index.

XML Tag

Description

<mes:meta>

(Optional)

Can be defined multiple times within an <url> tag to define metadata.

The following attributes are available:

Example:

  <url>

    <loc>https://www.mindbreeze.com</loc>

    <mes:meta key="breadcrumb">

      <mes:value>Mindbreeze</mes:value>

      <mes:value>Home</mes:value>

    </mes:meta>

<mes:value>

(Required)

Can be defined multiple times within a <mes:meta> tag to define one or more values for a metadata.

Example:

    <mes:meta key="breadcrumb">

      <mes:value>Mindbreeze</mes:value>

      <mes:value>Home</mes:value>

    </mes:meta>

Notes:

• Numbers must be formatted in JSON Number format. (e.g.: 1234 or -12.42)
• If the value is not a valid number, a warning is logged and the value is interpreted as a string.
• Currently, the only allowed value of the format attribute is “number“. If the value is to be interpreted as a string, the format attribute must be omitted, since this is the default behavior.

<mes:annotated>

(Optional)

Can be defined multiple times within a <mes:meta> tag to define one or more annotated values for a metadata.

Example:

    <mes:meta key="parent">

<mes:annotated>

         <mes:value>Main parent</mes:value>

         <mes:ctx-annotation metakey="related" aggregatable="false">Fabasoft</mes:ctx-annotation>

         <mes:ref-annotation metakey="parentkey">abcd.1.2.3.4</mes:ref-annotation>

       </mes:annotated>

    </mes:meta>

<mes:value>

(Required)

Must be defined once within a <mes:annotated> tag to define the value that is to be annotated.

Only string values are supported for annotated values.

Example:

    <mes:meta key="parent">

<mes:annotated>

         <mes:value>Main parent</mes:value>

         <mes:ctx-annotation metakey="related" aggregatable="false">Fabasoft</mes:ctx-annotation>

         <mes:ref-annotation metakey="parentkey">abcd.1.2.3.4</mes:ref-annotation>

       </mes:annotated>

    </mes:meta>

<mes:ctx-annotation>

(Optional)

Can be defined multiple times within a <mes:annotated> tag to add annotations to the value.

The following attributes are available:

Example:

    <mes:meta key="parent">

<mes:annotated>

         <mes:value>Main parent</mes:value>

         <mes:ctx-annotation metakey="related" aggregatable="false">Fabasoft</mes:ctx-annotation>

         <mes:ref-annotation metakey="parentkey">abcd.1.2.3.4</mes:ref-annotation>

       </mes:annotated>

    </mes:meta>

<mes:ref-annotation>

(Optional)

Can be defined multiple times within a <mes:annotated> tag to add reference annotations to the metadata.

The following attribute is available:

Example:

    <mes:meta key="parent">

<mes:annotated>

         <mes:value>Main parent</mes:value>

         <mes:ctx-annotation metakey="related" aggregatable="false">Fabasoft</mes:ctx-annotation>

         <mes:ref-annotation metakey="parentkey">abcd.1.2.3.4</mes:ref-annotation>

       </mes:annotated>

    </mes:meta>

<mes:acl>

(Optional)

Can be used within a <mes:meta> tag to define ACLs.

Example:

    <mes:acl>

      <mes:require>cn=domain users,cn=users,dc=myorganization,dc=com</mes:require>

      <mes:deny>unauthorized</mes:deny>

<mes:grant>cn=marketing,cn=users,dc=myorganization,dc=com</mes:grant>

    </mes:acl>

Attention: This tag is only considered, when the setting "Enable Default ACLs" is activated. Please also note, that ACLs from sitemaps are not compatible with “Access Check Rules”.

<mes:grant>

(Optional)

Can be defined multiple times within a <mes:acl> tag to grant access for a principal.

Example:

    <mes:acl>

      <mes:require>cn=domain users,cn=users,dc=myorganization,dc=com</mes:require>

      <mes:deny>unauthorized</mes:deny>

<mes:grant>cn=marketing,cn=users,dc=myorganization,dc=com</mes:grant>

    </mes:acl>

<mes:deny>

(Optional)

Can be defined multiple times within a <mes:acl> tag to deny access for a principal.

Example:

    <mes:acl>

      <mes:require>cn=domain users,cn=users,dc=myorganization,dc=com</mes:require>

      <mes:deny>unauthorized</mes:deny>  <mes:grant>cn=marketing,cn=users,dc=myorganization,dc=com</mes:grant>

    </mes:acl>

<mes:require>

(Optional)

Can be defined multiple times within a <mes:acl> tag to define in which groups a user must be.

The tag <mes:require> does not grant access in general, but denies the access, if the user is not in the group. Thus, after the last <mes:require> tag an additional <mes:grant> tag is necessary to grant access.

Example:

    <mes:acl>

      <mes:require>cn=domain users,cn=users,dc=myorganization,dc=com</mes:require>

      <mes:deny>unauthorized</mes:deny>   <mes:grant>cn=marketing,cn=users,dc=myorganization,dc=com</mes:grant>

    </mes:acl>

Setting

Description

Example/Default setting

Title

XPath expression matching the tag “title” (first matching element is chosen).

Attention: This setting only works with HTML documents and the following filters:

• FilterPlugin.HTMLCompleteJericho
• FilterPlugin.Jericho
• FilterPlugin.JerichoWithThumbnails

Example:

//h1

Title Element

(Advanced)

Element name of title tags.

Attention: This setting only works with HTML documents and the following filters:

• FilterPlugin.HTMLCompleteJericho
• FilterPlugin.Jericho
• FilterPlugin.JerichoWithThumbnails

Example:

h1

Use Link Text for Title

(Advanced)

This setting allows you to override the default document title detected by the web crawler and instead use the text of the hyperlink pointing to the page as the document’s title.

To use this setting, a regex pattern must be defined. The crawler will only apply this behavior to pages whose URLs match the pattern. This allows selective application of link-text titles for specific sites or URL structures.

Example:

^https://example\.com/articles/.*

Process Canonical Link

By defining a regular expression, you can specify which URLs should have their URL taken from the canonical tag. That URL is then used as the index key and stored as a URL metadata field.

Example:

^https?://example\.com/.*$

Encode Canonical Links

When this setting is enabled, the URLs extracted from the “canonical” tag are URL‑encoded before they are stored in the index. This is useful because the URLs are then saved in the same format as when you copy them from a browser. Browsers typically encode URLs automatically.

Hint: To ensure proper operation, it is recommended to clean the index and perform a re‑indexing.

Default setting:

Disabled

Content

XPath expression matching the content.

Attention: This setting only works with HTML documents and the following filters:

• FilterPlugin.HTMLCompleteJericho
• FilterPlugin.Jericho
• FilterPlugin.JerichoWithThumbnails

Example:

//div[@class='content']

Content Metadata Selector

(Advanced)

This setting is deprecated and the use of XPath is recommended.

-

Exclude Tags from Content

Exclude tags from content. The defined XPath expression is matching with HTML tags.

Attention: This setting only works with HTML documents and the following filters:

• FilterPlugin.HTMLCompleteJericho
• FilterPlugin.Jericho
• FilterPlugin.JerichoWithThumbnails

Example:

//div[@class='index']

Apply googleon/googleoff Tags

When this setting is enabled, content marked by googleon/googleoff tags will not be indexed.

Attention: This setting only works with HTML documents and the following filters:

• FilterPlugin.HTMLCompleteJericho
• FilterPlugin.Jericho
• FilterPlugin.JerichoWithThumbnails

Default setting:

Disabled

Metadata Selector

(Advanced)

This setting is deprecated.

-

Metadata Value Pattern

(Advanced)

• This setting is deprecated.

-

URLs Excluded from Filtering

(Advanced)

Defines the URLs that the crawler should fetch and follow for link extraction, without filtering, processing, or indexing their content.

Example:

^https://example\.com/archives/.*

Display Date Timezone

Defines the time zone for “Display Date”.

Attention: This setting only works with HTML documents and the following filters:

• FilterPlugin.HTMLCompleteJericho
• FilterPlugin.Jericho
• FilterPlugin.JerichoWithThumbnails

Example:

CET

Default Encoding

Defines the default character encoding for HTTP content, when no encoding is defined in the header of the HTTP content.

Default setting:

ISO_8859_1

HTTP Elements Encoding

Defines the character encoding for HTTP protocol elements like request headers, query parameters and possibly cookies.

Default setting:

ISO_8859_1

Default Content Type

Defines a fallback MIME type when the HTTP response does not include a Content-Type header.

Example:

text/html

Extract Metadata

Extracts metadata from HTML documents.

To extract metadata, the following settings must be configured in the property:

Example:

Exclude Documents With Matching Elements

Defines an XPath expression, where the document is not indexed, when the expression matches with the document.

To exclude documents, the following setting must be configured in the property:

Attention: This setting only works with HTML documents and the following filters:

• FilterPlugin.HTMLCompleteJericho
• FilterPlugin.Jericho
• FilterPlugin.JerichoWithThumbnails

Example:

/div[@class=’noindex’]

Assign Metadata

(Advanced)

This setting is deprecated.

-

Setting

Influence on the crawling direction

Crawling Root

Defines which URL is used to start crawling.

URL Regex

Defines which URLs are followed up.

URL Exclude Regex

Defines which URLs are excluded and not followed up.

URLs Excluded from Filtering

(Advanced)

Defines which crawled documents are to finally be indexed.

Setting

Description

Example/Default setting

Enable JavaScript

Enables JavaScript processing.

Hint: When this setting is enabled, the connector consumes significantly more resources. This results in slower crawling and higher memory usage. Therefore, it is recommended to enable the setting only on a single crawler and to restrict JavaScript processing with the subsequent settings „Include JavaScript URL (regex)“ and „Exclude JavaScript URL (regex)“ to only those URLs that actually require JavaScript.

Default setting:

Disabled

Enable Verbose Logging

(Advanced)

Enables advanced logging for diagnostic purposes.

Default setting:

Disabled

Skip Head Request

(Advanced)

When this setting is enabled, the HEAD request is skipped.

The HEAD request is used to determine whether the JavaScript on a web page should be processed. By default, JavaScript is evaluated only for text/HTML content.

Default setting:

Disabled

Include JavaScript URL (regex)

Defines a list of regular expressions that determine which URLs should have JavaScript processed.

If the setting “Exclude JavaScript URL (regex)” is also configured, the settings are applied in the following order:

• Exclude JavaScript URL (regex)
• Include JavaScript URL (regex)

Hint: It is recommended to configure this setting when the setting “Enable JavaScript” is enabled, to minimize resource consumption.

Default setting:

.*

Example:

^https://example\.com/.*

Exclude JavaScript URL (regex)

Defines a list of regular expressions that determine which URLs must not have JavaScript processed.

If the setting “Include JavaScript URL (regex)” is also configured, the settings are applied in the following order:

• Exclude JavaScript URL (regex)
• Include JavaScript URL (regex)

Hint: It is recommended to configure this setting when the setting “Enable JavaScript” is enabled, to minimize resource consumption.

Example:

^https://example\.com/login.*

Thumbnail Width

(Advanced)

Defines the width of the generated thumbnails in pixels.

Default setting:

100

Thumbnail Height

(Advanced)

Defines the height of the generated thumbnails in pixels.

Default setting:

75

Browser Width

(Advanced)

Defines the width of the browser window that is used for crawling.

Default setting:

1024

Browser Height

(Advanced)

Defines the height of the browser window that is used for crawling.

Default setting:

768

Page Load Strategy

(Advanced)

Defines the strategy used to load web pages.

Attention: This setting is intended for internal use only and should not be changed.

Default setting:

Eager

Page Ready State

(Advanced)

Defines from which point a webpage is considered fully loaded.

Attention: This setting is intended for internal use only and should not be changed.

Default setting:

Complete Or Inactive

Setting

Description

Example/Default setting

Content Selector Type (deprecated)

Same behavior as the setting “Content Selector Type” in the Content Presence Selector.

-

Content Presence Selector (deprecated)

Same behavior as the setting “Content Presence Selector” in the Content Presence Selector.

-

Content Presence Selector

The following settings must be configured to add a Content Presence Selector.

URL Patterns (regex)

Defines the regex pattern used to match website URLs.

If a URL matches the pattern, the Content Presence Selector is applied.

Example:

https://site-where-the-selector-should-trigger.com/site-with-content.*

Content Selector Type

Defines the type of the Content Presence Selector.

Available options:

• CSS
• XPath

Default setting:

CSS

Content Presence Selector

Defines a CSS or XPath expression that determines whether the targeted content is present.

Hint: When defining the CSS or XPath expression, single quotation marks must be used.

Attention: This setting must not be used if the setting “Assume Content if no Script triggered” is enabled.

Example:

//*[@id='content']

Assume Content if no Script triggered

When this setting is enabled, the content is assumed to be loaded even though no “Script Trigger Selector” is activated.

Attention: This setting must not be used if the setting “Content Presence Selector” is configured.

Default setting:

Disabled

Scripts

The following settings must be configured to add a script

Script Name

Defines a name for the script that is used for identification in the logs.

Example:

CookieBanner

URL Patterns (regex)

Defines the regex pattern used to match website URLs.

If a URL matches the pattern, the script will be executed.

If the setting is empty, it will be interpreted as “.*“.

Example:

https://site-with-cookie-banner.com/cookies.*

Script Selector Type

Defines the type of the Script Trigger Selector.

Available options:

• CSS
• XPath

Default setting:

CSS

Script Trigger Selector

Defines a CSS or XPath expression that determines when to trigger the script.

Hint: When defining the CSS or XPath expression, single quotation marks must be used.

Example:

//*[@id='cookieBannersId']

Script

Defines the script to be executed when the Script Trigger Selector is active.

Example:

document.getElementById('okBtn').click()

Credential Scripts

The following settings must be configured to add a credential script

Script Name

Defines a name for the credential script that is used for identification in the logs.

Example:

LoginScript

Allowed Hostnames

Defines the hostnames on which the script should be executed.

Pattern matching is not supported for this setting.

Example:

some.trusted.sites.com

Script Selector Type

Defines the type of the Script Trigger Selector.

Available options:

• CSS
• XPath

Default setting:

CSS

Script Trigger Selector

Defines a CSS or XPath expression that determines when to trigger the script.

Hint: When defining the CSS or XPath expression, single quotation marks must be used.

Example:

//*[@id='username'] and //*[@id='password']

Script

Defines the Script, which is to be executed when the Script Trigger Selector is active.

In contrast to typical scripts, you can use the provided object “mesCredential”.

This object contains the variables domain, username, and password, which are populated based on the selected credential:

• let password = mesCredential.password;
• let username = mesCredential.username;
• let domain = mesCredential.domain;

Example:

document.getElementById('username').value =

mesCredential.username;

document.getElementById('username').value =

mesCredential.password;

document.getElementById('submit').click()

Credential

Defines the credential that should be used in the script and that was created in the “Network” tab.

Currently, the following credential types are supported:

• Password
• Username/Password

Example:

WEB

On New Document Script

Defines a script that is executed once, directly before a page loads.

This setting is useful for use cases where standard scripts are executed too late.

Example:

// Override window.open to block popups

window.open = function(...args) {

    console.log("Popup blocked: window.open was called, but no action was taken.", args);

    // Optionally, you could store the args for analysis or logging

    return null; // Prevent any popup from opening

};

HTTP Authentication Dialog

The following settings must be configured to add credentials for browser HTTP authentication dialogs.

Allowed Hostnames

Defines a list of hostnames on which the credentials are allowed to be used.

To specify multiple hostnames, separate each hostname with a line break.

Example:

some.trusted.sites.com

Credential

Defines the credential that should be sent to the respective hostnames and that was created in the “Network” tab.

Example:

WEB

Setting

Description

Example/Default setting

Additional Network Resources Hosts

Defines a list of hostnames to which network requests are allowed.

Using patterns for this setting is only possible, when the setting “Match "Additional Network Resources Hosts" as regex” is enabled.

Example:

additional.trusted.host.com

trustful.host.com

allowed.to.load.here.at

mindbreeze.com

Match "Additional Network Resources Hosts" as regex

When this setting is enabled, hostnames in the setting “Additional Network Resources Hosts” are interpreted as regex patterns.

Each script must have at least one URL pattern defined.

Default setting:

Disabled

Allowed Network Resources

Defines a list of network resources to which requests are allowed.

Default setting:

DOCUMENT

STYLESHEET

IMAGE

MEDIA

FONT

SCRIPT

XHR

FETCH

PING

CSPVIOLATIONREPORT

OTHER

Additional Arguments

This setting is intended for the internal use and should not be changed.

-

Setting

Description

Example/Default setting

Page Load Timeout

Defines in seconds, how much time is allowed for the loading and executing of JavaScript on a web page.

If this time is exceeded, the processing is aborted.

The maximum value for this setting is 20 seconds.

Hint: The value of this setting should always be lower than the value of the setting “Network Timeout”.

Default setting:

10

Network Timeout

Defines in seconds, how long the connector may wait for a response from the website.

Hint: The value of this setting should always be bigger than the value of the setting “Page Load Timeout”.

Default setting:

10

Browser Recycle Threshold

Defines the number of web pages that are processed with an internal browser instance before the instance is automatically terminated and restarted to conserve system resources.

Default setting:

1000

Browser Control Process XmX

Defines the maximum heap memory of the browser control process.

Default setting:

512m

Setting

Description

Example/Default setting

Memory Profile

Defines the amount of memory allocated to the crawler.

The profile “InSite” uses the same amount of memory as other crawlers.

The profile “InSpire” is recommended when the crawler needs more memory or when it must crawl data with very large file sizes.

Default setting:

InSite

Number of Crawler Threads

Defines the number of threads crawling the specified web pages and subpages in parallel.

Example:

5

Minimum Request Interval

Defines the minimum value in seconds for the delay between consecutive crawler requests.

Example:

100 ms

Maximum Request Interval

Defines the maximum value in seconds for the delay between consecutive crawler requests.

Example:

200 ms

Crawler Queue Size

Defines the maximum number of documents in the queue that will be sent to the index.

Example:

10

Mindbreeze Dispatcher Thread Count

Defines the number of threads that send data to the index in parallel.

Example:

10

Allowed Delete Ratio

Defines the maximum allowable proportion of documents to be deleted relative to the number of documents in the index.

This setting is intended to prevent data loss in the event of network failures or accidental configuration changes.

For example, if more than 0.8 or 80 % of the documents would be deleted in a single crawl run, the crawl run is automatically aborted to avoid data loss.

Attention: This setting is only effective when the setting “Delta Crawling” is disabled.

Default setting:

0.8 (= 80%)

Setting

Description

Example/Default setting

NTLM Credential

The NTLM credential to be used.

To create the credential, go to the “Network” tab.

Example:

NTLM Credential

Mindbreeze InSpire Fully Qualified Domain Name

The Hostname which is used for NTLM authenticaton.

Example:

my-inspire.mindbreeze.com

OAuth Credential

The OAuth credential to be used.

To create the credential, go to the “Network” tab.

Example:

OAuth Credential

OAuth Scope

(Advanced)

Configure this setting to request specific permissions for OAuth authentication.

Example:

api://1123-4566-77789/.default

Use Kerberos Authentication

Enables the Kerberos authentication.

Default setting:

Disabled

Login URL

If a static form-based login with session management is to be configured, the URL to be used must be entered in this setting.

For instance, this URL can be determined using the debugging functions of the web browser.

Example:

http://webserver.myorganization.com/login

Session Initialization URL

In some cases it is necessary to retrieve a dynamically generated cookie from a specific URL and send it along already with the form-based login.

An HTTP GET request is executed on the URL entered here and the cookies generated from this are sent along with the actual login.

For complex form-based authentication, this URL is opened at the beginning so that it can then be dynamically redirected. The cookies received in the process are retained for the session.

Example:

http://webserver.myorganization.com/protected/documents

Include Matching Cookies (Regular Expression)

Defines the restriction that determines which cookies are to be stored for session management.

For the definition of the restriction, a regular expression must be put in that contains the names of the cookies that are to be enabled and used for the session.

Example:

.*JSESSIONID.*

Login Form Parameters

If hidden fields are set in the login form, they can be listed here. They are extracted and sent along with the login request.

A typical example of this is the dynamically generated FormID, which is returned as a hidden parameter from the Web server.

Example:

formid

Login URL Patterns

All redirects that correspond to the regular expressions specified here are tracked during the login process.

Example:

http:\/\/webserver\.myorganization\.com\/protected\/login.*

Login Post URL Patterns

When tracking the redirects that correspond to the regular expressions specified here, all collected form parameters are sent using an HTTP POST request.

Example:

http:\/\/webserver\.myorganization\.com\/protected\/login\/loginform.*

Logged in URL Patterns

If you are forwarded to an URL that matches the regular expressions specified here, the login process was successful.

Example:

http:\/\/webserver\.myorganization\.com\/protected\/documents.*

Maximum Allowed Count of Redirects

This can be used to set the maximum depth of the tracked redirects.

Example:

5

Maximum Session Age in Seconds

Maximum session age in seconds.

Default setting:

20

Post to Configured Login URL

If this setting is enabled, redirects to a "Login Post URL" are replaced by an HTTP POST request to the URL configured under "Session Initialization URL".

Default setting:

Disabled

Follow Redirects for Login Post

If this setting is enabled, all redirects are tracked after the HTTP POST request to the login URL, and all cookies are collected until no further redirect is requested or the authentication is successful.

Default setting:

Disabled

Reset Session Before Login

(Advanced)

If this setting is set, old session cookies are not used to create a new session when a session expires.

Default setting:

Disabled

Form Elements

The names and values of the elements that are used in the HTTP POST request on the login URL have to be specified in this setting. In doing this, the name of the HTML form field should be entered.

All password fields have to be entered under Password Elements.

Configure the following settings to add a “Form Element”:

Example:

Form Password Elements

The names and values of the elements that are used in the HTTP POST request on the login URL have to be specified in this setting. In doing this, the name of the HTML form field should be entered.

All password fields have to be entered under Password Elements.

Configure the following settings to add a “Form Password Element”:

Example:

Setting

Entry

Name

Example:

NTLM Credential

Type

Username/Password

Username

Example:

developer

Domain

Example:

myorganization.com

Password

Example:

my-secure-password-123!

Setting

Entry

Name

Example:

OAuth 2 Credential

Type

OAuth 2

Access Token URL

Example:

https://myorganization.com/auth/realms/master/product

Realm

Example:

Master

Client ID

Example:

developer

Username

-

Password

-

Client Secret

The Client Secret that is provided by the Identity Provider (Microsoft Azure, Keycloak etc.).

fish <!--googleoff: index-->shark <!--googleon: index-->dog

“fish“ and “dog“ are indexed; “shark“ is not

fish <!--googleoff: snippet-->shark <!--googleon: snippet-->dog

fish <!--googleoff: all-->shark <!--googleon: all-->dog

<!--googleoff: anchor--><A href=subsite.html>shark </A>dog <!--googleon: anchor-->

“dog“ is indexed; “shark“ is not

Setting

Description

Entry

Access Check Principal

The username can be defined in the following formats:

• username@domain
• domain\username
• distinguished name

Group names can only be in the “distinguished name” format.

Additionally a capture group from the selection pattern can be referenced here (See Access Rules[3]).

Example:

cn=domain users.cn=users.dc=mydomain.dc=com

Access Check Action

Controls the access type, where the following options are available:

• Grant
• Deny

Example:

Grant

Metadata Key for Selection (e.g. url)

A metadata name used by access rule.

Can be empty, which selects all documents.

Example:

url

Selection Pattern (e.g. .*html)

A regular expression.

Can be empty, which selects all documents.

Example:

.*

Fehlercode

Beschreibung

1

Successful DNS lookup.

0

Fetch never tried (perhaps protocol unsupported or illegal URI).

-1

DNS lookup failed.

-2

HTTP connect failed.

-3

HTTP connect broken.

-4

HTTP timeout.

-5

Unexpected runtime exception.  See runtime-errors.log.

-6

Prerequisite domain-lookup failed, precluding fetch attempt.

-7

URI recognized as unsupported or illegal.

-8

Multiple retries failed, retry limit reached.

-50

Temporary status assigned to URIs awaiting preconditions.  Appearance in logs may be a bug.

-60

URIs assigned a failure status.  They could not be queued by the Frontier and may be unfetchable.

-61

Prerequisite robots.txt fetch failed, precluding a fetch attempt.

-62

Some other prerequisite failed, precluding a fetch attempt.

-63

A prerequisite (of any type) could not be scheduled, precluding a fetch attempt.

-404

Empty HTTP response interpreted as a 404.

-3000

Severe Java Error condition occured such as OutOfMemoryError or StackOverflowError during URI processing.

-4000

"Chaff" detection of traps/content with negligible value applied.

-4001

The URI is too many link hops away from the seed.

-4002

The URI is too many embed/transitive hops away from the last URI in scope.

-5000

The URI is out of scope upon reexamination.  This only happens if the scope changes during the crawl.

-5001

Blocked from fetch by user setting.

-5002

Blocked by a custom processor.

-5003

Blocked due to exceeding an established quota.

-5004

Blocked due to exceeding an established runtime

-6000

Deleted from Frontier by user.

-7000

Processing thread was killed by the operator.  This could happen if a thread is a non-responsive condition.

-9998

Robots.txt rules precluded fetch.