White Paper

Installation and Configuration of Mindbreeze InSpire

Introduction

This document describes the installation and configuration of Mindbreeze InSpire in a Microsoft Windows environment.

Mindbreeze InSpire consists of:

• The Mindbreeze InSpire Node, which is used to create indexes which can be queried by the users.
• The Mindbreeze InSpire Management Node, which is used to configure and administer the Mindbreeze InSpire Nodes which are distributed over the network via a graphical user interface.

These components can be installed as required.

Chapter 9 describes the Mindbreeze InSpire query language.

Software Requirements

All information contained in this document implicitly assumes a Mindbreeze InSpire environment and Mindbreeze InSpire 2016 Spring Release.

Requirements:

• All information about our current software requirements you can find in our Software Product Information

The required setup packages are provided on the Mindbreeze InSpire ZIP/ ISO File in the prerequisites directory.

Advanced Configuration of Mindbreeze InSpire

After installing Mindbreeze InSpire, the user interface for configuring Mindbreeze InSpire will be displayed in a web browser.

It is recommended to select „Apply changes and restart on save“ option before saving any changes in the configuration. The services will restart after saving configuration changes. Therefore, it is recommended that these changes should be performed only during maintenance times.  

The “Overview“ tab

This screen gives an overview of all Services, Nodes, and Category Plugins configured on the server.

“Indices“ Tab

Index Services can be managed using the “Indices“ tab. All configured index services are listed here. They can be edited and deleted from here. Additionally, new index services can be created.

Index service settings can be imported directly from or to an existing index. For detailed instructions visit chapter “Import/Export of Settings”.

To create an Index Service on the “Indexes” tab, perform the following steps:

Click the “+ Add Index” button located on the top right-hand side.

In the following window, select an ‘Index Node’, a client service and a data source. Confirm your selection with ‘Apply.

You can now also make the ‘Display name’ and other settings. These settings are explained in detail below. Save with „Save“ to save the changes.

Hint: By clicking on ‘Enable’ or ‘Disable’, you can temporarily switch an index that has already been created on or off.

Index Service Settings

In the “Setup” box, the fields “Display Name”, “Index Node”, “Index Path” and “Filter Service” are available by default. The following values can be specified in those fields:

To obtain additional setting options for an index, click on the ‘Indices’ tab again and click on the ‘Advanced Settings’ field at the top right.

You will receive the following additional setting options:

Query performance settings.

The settings in this section are used to improve query performance:

Aggregation settings

Index Synchronization

These settings are relevant if you use Mindbreeze InSpire in Distributed Operation (G7) mode.

Index Compactification

Index-performance may degrade over time if many documents are added and deleted. The compactification feature removes buckets that contain deleted documents from the index.
Remaining documents are moved to a new bucket.

Automatic Compactification

Manual Compactification

Additionally, a command line interface using the “mescontrol”-tool is available.

• „bucketsinfo”: Prints the current state of the buckets including the deleted documents percentage.
• „listtasks“: Lists all running tasks.
• „taskcancel <taskid>“: Cancels the specified task
• „taskwait <taskid>“:  Waits until the specified task is finished
• „taskstatus <taskid>“: Prints the status of the specified task
• „deletebuckets  [--sync] [--min-percent-deleted-docs=<0..1>] [<bucketid_1>… <bucketed_n>] [--log-unreferenced-unfiltered-documents] [--cleanup-unreferenced-unfiltered-documents]: Deletes the specified buckets.
  • „--sync”: the command returns after the task has finished
  • „—min-percent-deleted-docs“: deletes buckets only if the percentage of deleted document exceeds this value
  • „bucketid.“: the IDs of the buckets to be deleted
  • „--log-unreferenced-unfiltered-documents“: If there are generated metafiles for documents e.g. Thumbnails for PDFs, but the original files were deleted without the metafiles, then logs will be written.
  • „--cleanup-unreferenced-unfiltered-documents“: If there are generated metafiles for documents e.g. Thumbnails for PDFs, but the original files were deleted without the metafiles, then these files will be deleted. If „--log-unreferenced-unfiltered-documents“ is also set, logs will be written and the files will be deleted.

Network Settings

You can choose under Network Properties if you want to use the HTTP keep-alive for item transformations. This will reduce the number of open connections to static resources or reuse the connections. This feature is disabled by default.

Item Transformation Service Plugin Timeout: Item transformation requests are aborted after this timeout and the document is inverted without this transformation.

Entity Recognition Parameter

These settings enable the index service to extract metadata from document contents. For more information, see Configuration - Entity Recognition - Entity Recognition Parameter.

If Query Transformation plugins are installed the following section is added to the Index Service configuration panel in “Advanced Settings” mode.

In the drop-down list the available Query Transformation plugins can be selected. The selected plugins can be activated for the current Filter Service using the “Add” button besides.

The activated plugins are listed above. By clicking on the “expand” button () of an active plugin, a “Plugin Properties” section will be visible. Here you can define properties for the current plugin instance in form of key-value pairs.  With the corresponding “delete” button you can remove () these custom properties.

By clicking on the delete button besides the active plugin name, the plugin will be removed from the list with all defined instance properties and will not be active for the current Index service.

Global configuration of query and item transformation plugins

Under "Global Index Settings", you can configure the preinstalled query and item transformation plugins globally for all indices. These plugins are also automatically applied to any newly added index. The global configuration is only applied to indices for which no plugins are directly configured or default plugins removed.

Repair References

Using "repairreferences" it is possible to correct references that refer to incorrect uniformitemids. If corrections are made, the DocumentInfo is automatically re-inverted..

The repair is performed using the following steps:

1. Scan all documents in the index to ensure that DocIDs are unique
2. Repair of document entries in the reference index
3. Repair of references in all documents

Usage:

The repair is started with the mescontrol command line tool "repairreferences". It is necessary that the option "Disable Unrestricted Privileged Servlets" is deactivated.

mescontrol http://<INDEXHOST>:<INDEXPORT> repairreferences [--bulk-update-size=0] [--skip-uniformitemid-check] [--dry-run] [<docid> ... <docid>]

Optional parameters:

• --bulk-update-size: The number of updates performed within a transaction. Overrides the References Repair Bulk Update Size index option. Default value: 100
• --skip-uniformitemid-check: Step 1 is skipped
• --dry-run: At the end of step 3 documents are not updated
• docid ... docid: List of documents to be repaired

Enabling Index Backups

Index Backups can be enabled in the “Global Index Settings” section beneath the service configuration sections on the „Indices“ tab.

In the field „Allowed Backup Path Pattern (Regex)“ a regular expression is used to restrict the pattern of allowed backup directory paths.

Note: When separating paths, backslashes must be escaped: \\

To start a backup, the command line tool mescontrol can be used:
    mescontrol http://<INDEXHOST>:<INDEXPORT> backup <BACKUPZIELPFAD>

To stop a currently running backup, the following command line can be used:
    mescontrol http://<INDEXHOST>:<INDEXPORT> stopbackup

Enabling Support Mode

Activating “Advanced Settings” checkbox also shows the “Support Mode” section beneath the service configuration sections on the “Indices”, “Filter”, as well as “Client Services” tabs. Support mode traces detail information about the individual services into log files defaulting to the Mindbreeze service user’s TEMP directory on Windows and to /var/opt/mindbreeze/log directory on Linux platforms. Custom log directories can be configured in the “Log Location” section below.

Note: Do not keep support mode activated in “normal” operation to avoid decreasing the performance when not needed anymore. If in doubt, keep it disabled.

To persist changes click on the “Save” button on the top right corner.

Sub Query Expression

Sub Query Expression enables reference evaluation inside of metadata. This allows, for example, searching all the files inside a folder, or vice versa, the folder in which a file is located. This can be applied for all Metadata Keys, which point to another document. The reverse direction can also be enabled by adding the metadata keys to the field Inverted Reference Metadata Keys. In addition to forward and reverse references, string and reverse string references (via property expression lookup and rev_lookup) can be used.

Tokenizer Configuration

The options in „Tokenizer Configuration“are used to change the behavior of substring matches for search results

Transaction System Settings

The options in this group can be used for index transactions subsystem fine-tuning.

Memory Analysis Settings

Optional Terms

Using the “Optional Terms”, finding documents can be simplified by providing results in which not all search terms necessarily have to occur. To make this transparent for the user, a note is displayed with the search result. This feature is active by default, but can be parameterised or completely deactivated.

It is important to note here that some options in the Global Index, Local Index and Client Service/UI of the Relevance section in the MMC configuration areas have the same name or functionality, but some options override/dominate the others.

In principle, the following mightiness applies (first is least dominant, last is most dominant):
Global Index < Local Index < Client Service/UI of the Relevance section in the MMC

Example:
Global Index: Optional Terms = activated
Local Index: Optional Terms = activated
MMC: Optional Terms = deactivated
In total: Optional Terms = deactivated

Further Example:
Global Index: Optional Terms = deactivated
Local Index: Optional Terms = deactivated
Client Service: Optional Terms = activated
In total: Optional Terms = activated

Compound Splitting

With the help of compound splitting, individual words composed of several words can be recognized and separated accordingly, so that partial words are also sufficient for the search query to find more complex words.

Example: In order to also find documents that e.g. also contain "recognition", "forbestechcouncil" etc. in the results, the following can be entered in the search input:

AI cognition techcouncil

Notes:

• The prerequisite for the compound splitting is the activation of the "Enable Language Detection" option, which activates the automatic language detection of the documents. Currently, the languages DE and EN are supported for the Compound Splitting functionality. More supportable languages will follow soon.
• The Compound Splitting functionality applies only to newly added documents. To apply the functionality to already existing documents, a full re-inversion of the index is necessary. A description of how to do this can be found here.
• The Compound Splitting function is enabled by default and the following options are available for the respective local indices as well as Global. For a more detailed description, please refer to the "Compound Splitting Strategy" option below.

Note: For Windows users, you need to install additionally: MESExtensionsSetup.exe

Named Entity Recognition (NER)

Named Entity Recognition can be used to identify and classify named entities in both the content and metadata of a document based on AI-based language detection and subsequent sentence segmentation.

Currently, the following named entities are supported, which are already pre-trained and can be adapted and extended in the further course (e.g. by tools).

• Persons (entity:person)
• Locations (entity:location)
• Organizations (entity:organization)
• Numeric values (entity:number)

Example: To find all documents by people that occur near the words "head", "academy" and "mindbreeze", the following can be entered in the search input.

entity:person:ALL NEAR head NEAR academy NEAR mindbreeze

Notes:

• The prerequisite for NER is the activation of the "Enable Language Detection" option, which activates the automatic language detection of the documents. Currently, the languages DE and EN are supported for the NER functionality. In the future, more languages will be supported.
• The composite decomposition functionality applies only to newly added documents. To apply the functionality to already existing documents, a full re-inversion of the index is necessary. A description of how to do this can be found here.
• The NER functionality is disabled by default and the following options are available for the respective local indices as well as Global. For a more detailed description, please refer to the "Compound Splitting Strategy" option below.

A description of how to customize Insight Apps (e.g. for different entity colours) ca be found here and here.

Note: For Windows users, you need to additionally install: MESExtensionsSetup.exe .

Named Entity Recognition (Client Service)

Sentence Transformation

This section describes all the Sentence Transformation configuration options. These settings relate to „Natural Language Question Answering“ (in short: NLQA). Please read the Whitepaper – Natural Language Question Answering (NLQA) first.

Storage Settings

Stop Word Catalogs Settings

Stop word catalogs can be used to skip stop words in some usecases, such as highlighting.

Text Cleaning

The “Text Cleaning” Feature enables the removal characters belonging to special Unicode categories from the Sample Text and HTML preview. This option is disabled by default.

Data Sources

To create data sources for a particular index, click one of the icons at the top right of the “Data Sources” section. These icons represent the different data sources integrated into the Mindbreeze InSpire software.

Custom data source

A custom data source makes it possible to use the Mindbreeze InSpire Client to search data sources integrated by a third party.

These connectors can be installed from the Mindbreeze Management Center (also see Configuration – Plugin Installation).

Look for detailed installation instructions in the documentation provided with the data source.

To create a custom data source proceed as follows:

1. Click the symbol. A configuration form for custom data sources will be displayed.
2. In the “Source name” field, assign an appropriate name for this data source.
3. In the “Category” field, choose the registered data category corresponding to the data source being set up.

Click the ”Save“ button in the top-right corner to save your settings.

You will find further information on how to register a new custom data source with Mindbreeze InSpire and how to configure its indexing in the documentation of the Mindbreeze connector delivered by the third party.

Crawler Scheduling

It is possible to apply one or more user defined time spans for the crawlers to run at. To set up this feature, go to the “Index” tab of your Mindbreeze InSpire Management Web Interface and enable the advanced mode by clicking on the check box at the top right corner of the page. After clicking on this check box some more user controls should appear on the screen, including one called “Crawler Schedule”. This section provides an overview over the already configured time spans that define when the current crawler should run. To add a new entry, simply click the “Add” button and enter the desired time span. To change an already existing time span, select it in the list and then click the “Edit” button. To remove one of the entries, simply select it in the list and then click the “Remove” button.

After clicking the “Add” button, the following screen will appear:

Here you can enter the time interval when the crawler should run. Please use a 24 h time format for your input. After entering your time span you can either click “Apply” to save your changes or “Cancel” to discard them.

Click the “Save” button at the top right of the screen to save your changes and make them take effect.

Filter- and Index performance optimization

Under "Advanced Settings" in the "Performance Settings" section, the option "Concurrent Filter and Index Dispatch Threads" allows you to define the number of threads that download documents in parallel and send them to the filter and index service. With a higher value (e.g. 20) you can optimize the performance, but this also increases the load on the filter and index service.
Default value: 10.

Extension Point Properties and Environment Variables

For every data source, "Extension Point Properties" and "Extension Point Environment Variables" can be defined. These settings are not relevant for you and are for internal use only.

Customizing the category descriptor

The category descriptor specifies the display options and the filter information of a data source and is an XML document stored in the plugin (typically categoryDescriptor.xml; the name is referenced via plugins.xml). The root element is the “category” element.

<?xml version="1.0" encoding="UTF-8"?>
<category id="Category" supportsPublic="false" keep-docinfo-metadata="false">
    <name>Category</name>
</category>

Attributes in the “category” element are:

• supportsPublic: specifies whether the data source may be configured in a public index. The default value is false.
• keep-docinfo-metadata: defines whether metadata with aggregatable or regexmatchable attributes will be retained or overwritten by an updated descriptor. The default value is false.

Adding custom metadata columns

A metadata definition could look like this:

<metadata>
    <metadatum aggregatable="true" id="current_state" visible="true">
       <name xml:lang="en">Ticket State</name>
        <name xml:lang="de">Ticket Status</name>
    </metadatum>
</metadata>

The following attributes can be defined in the metadatum element and are used to control the metadata in the index:

• aggregatable: If this option is set to true, the column will be available as a filter (should only be defined for properties for which the values allow a grouping of the results – the aggregatable option doesn’t make sense for unique values, which can only occur once in the search result.
• regexmatchable: specifies whether the search for these metadata can be performed with a regular expression.
• visible: specifies whether the column is displayed in the default result presentation.

Replacing the hit icons

The small icon in the data source list of the search client is defined in the file “categoryIcon.png” in the ZIP archive of the data source plugin. You can replace the icon with a 16x16 icon of your choice.

You can also define an icon with the icon tag directly in categoryDescriptor.xml. This requires a unique ID, size attributes (height and width) and the picture itself (value), encoded as Base64 value.

<context>
    <Icon alt="Ticket" height="16" width="16"
        id="tag:mindbreeze.com,2007/contextitems/contexticon;ticket"
            mimetype="image/png"
            type="tag:mindbreeze.com,2007/contextitems/contexticon"
            value="
iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAAsTA    AALEwEAmpwYAAAAB3RJTUUH3wIXECgw/xAFagAAAS1JREFUOMvFkz1uwkAUhL/nNV4Zesw9wgUILYq4gymCgCiRIG3gCkAOEnMFTgMNFT8W65cCpOAIAqmY8knzaTQ7C/eWvA8Gpd1uN3HOPQFyo0+NMYm1tkcrjmuzJNH/apYk2orjmu+c88tRxHq91tVqJSJ/hBBhu9kQRZGWo0icc74PoFmGqspkPMZae9G/3W55brdJ01Q0ywA4AIAgCPgYDi+agyAAYL/fIyLo8e4fKlEKhQKf0+nZBM45ur1eDoTqCeCo17e36/WrctpTDtDtdAjD8KzROUej0eCxXs/dc4DhaJSjh2FIsVi8nkA8D0Dn87kU/B9mpVLhoVr99ZICoOJ5AuAbY/bLxQJAms3mzQteLhYYY5wM+v1SmqbT45S5ecqe9xVY+3L3z8g3o1Sele9r3SQAAAAASUVORK5CYII= " />
</context>

Adding user-defined hit actions

You can add user-defined actions based on specific metadata, for instance, in order to open a ticket search result in a custom ticket web application.

<context>
<Menu>
<Action name="Open" pattern="http://intranet.mycompany.com/ticketing/show.html?ticketid={{mes:key}}">
  <name xml:lang="en">Open Ticket</name>
  <name xml:lang="de">Ticket Öffnen</name>
</Action>
</Menu>
</context>

For the changes to take effect, you can upload the ZIP archive plugin with the modified categoryDescriptor.xml via the Mindbreeze configuration interface.

Note: We recommend renaming modified plugins with a separate name to better detect changes in product updates.

Limitations

The usage of the same category descriptor in two custom plugins simultaneously is not supported. In that case the deployment order during a snapshot is not defined.

app.telmetry configuration for Crawl Runs

The crawl run information can be stored in a separate LogPool.

To do this, the Fabasoft app.telemetry log definitions for Mindbreeze Services must be downloaded from the Mindbreeze configuration interface. Select the "Overview" tab and click on the link "Fabasoft app.telemetry log definitions and Dashboards".

Then create a new log pool with the following settings:

Application: Mindbreeze

Application ID: *

Application Tier: Crawler Service Run

Tier ID: *

In the tab "Log Definition Columns" the xml file: apptelemetrylogdefinitions_crawlerservicerun.xml must be uploaded from the log definition archive.

“Filters“ tab

On the “Filters“ tab, all Filter Services can be managed. In the “Filter Services” box, the available Filter Services are shown. Using the “Add new filter“ button (the plus icon, located toward the top right-hand side) additional Filter Services can be defined.

To create a new Filter Service, perform the following steps:

1. Click the plus icon () located at the top right-hand side.
2. Edit the properties of the Filter Service.
3. In the “Setup” box, the “Display Name” field is available by default. A name for the Filter Service can be specified in this field.
4. In the “Node” field, the servers on which the Filter Service will be run can be selected.

To configure a Filter Service in more detail, perform the following steps:

1. Select “Advanced settings” to display the file extensions that the Filter Service will filter. Available Filters are displayed as well as the filters provided by Mindbreeze InSpire
2. If Java filters need more memory increase “Embedded Java VM Maximum Heap Size”.
3. To analyze problems with Filters activate “Dump Requests/Responses” and select a directory to store the dumps with the option “Dump Directory”.
4. If you want to filter Documents that are bigger than 50 MB increase „Maximal Input Size (MB)”.
5. To accelerate filtering of ZIPs and PST-Files increase “Maximum Threads per Filter Request”.
6. Configure a custom ”Filter Recursion Depth” to control the number of extracted objects from nested containers like ZIPs .
7. Specify a Regular Expression for “Filter Pass Through Extensions Regular Expression” to specify extensions which are sent directly to the Index instead of filtering without contents.

If “Index Empty Content Regardless of Extension” is enabled, documents with empty content are always indexed regardless of extension and availability of matching Content Filter.

1. To try all matching Plugins in case of errors in their relative order for a set of extensions, specify the extension as regular expression in the “Probe Filters Matching Extension Regex” field.
2. In order to avoid overwriting data source metadata during content filtering select “Keep Datasource Metadata” form “Metadata Merge Strategy” dropdown box.
3. Edit the “Filter Plugins” as required: files and documents will be filtered differently depending on their extension, you can adjust which extensions should be treated by which filter plugin from the filter service.
4. Select “Ignore Heartbeat Error” to prevent Filter Service restart if it fails to send hearbeat to Node. This may happen in occasions when the system is very busy.
5. Select “Generate CRC64 of Metadata“ to generate a CRC64 of the filtered metadata. If the CRC differs, the document will be replaced in the index if needed. Additional Regular Expression rules can be defined to include or exclude Metadata fields sent by the crawler.

Metadata CRC:

• If the option „Generate CRC64 of Metadata“ is activated, the metadatacrc is additionally used to check whether an item should be replaced in the index.
• In the further options a "Regular Expression" can be specified to include or exclude metadata in the CRC.

Extracting additional PDF Meta Keys

The PDF-Filter extracts the following Meta Keys from PDF documents if they are available:

• document:title
• document:Author
• document:Subject
• document:Keywords
• document:Creator
• document:Producer
• document:CreationDate
• document:LastModified
• document:Trapped
• document:PageCount

To extract additional Meta Keys add the plugin „FilterPlugin.PDFPreviewFPDFFilter“ to the „Global Filter Plugin Properties“ and configure the property „PDF Meta Keys“. Multiple values are separated via semicolon („;“):

Saving HTML meta tags as metadata

The HTML filter plugins Jericho and JerichoWithThumbnails extract the HTML meta tags as metadata values. If these HTML meta tags occur multiple times (with the same name and value), it is possible to save them only once as a metadata value.

Add the plugin “FilterPlugin.JerichoWithThumbnails” or “FilterPlugin.Jericho” (if no HTML thumbnail generation is used) to the filter under “Global Filter Plugin Properties” and enable the property “Store only distinct HTML meta tag values as metadata”.

If a regular expression is defined here as "Parsable HTML meta tag pattern," only meta tags with matching "name" or "http-equiv" attributes are stored as metadata.

If Post Filter Transformation plugins are installed (f. ex. SignatureToKeyRewriter), the following section is additionally shown in the Filter Service configuration:

In the drop-down list the available Post Filter Transformation plugins can be selected. The selected plugins can be activated for the current Filter Service using the “Add” button besides.

The activated plugins are listed above. By clicking on the “expand” button () of an active plugin, a “Plugin Properties” section will be visible. Here you can define properties for the current plugin instance in form of key-value pairs.  With the corresponding “delete” button you can remove () these custom properties.

By clicking on the delete button besides the active plugin name, the plugin will be removed from the list with all defined instance properties and will not be active for the current Filter service.

1. To save the modifications, click “Save“. Modifications in the configuration are propagated to the appropriate server nodes.
2. Switch to the “Overview“ tab-to get an overview of the mapping of services and server nodes.

Similar to the ”Indexes“ tab, the ”Filters“ tab offers the ability to enable the support mode.

Destination rewrite

"Client Services" Tab

The ”Client Services“ tab is used to manage all client services. A client service provides the server-side support for the Mindbreeze InSpire Web Client. The field “Web Client Services” shows all existing web client services.

In order to create a new web client service, proceed as follows:

Click the plus symbol () located at the top right-hand side.

Modify the settings of the new client service.

In the “Setup” field you can manage the settings “Display Name”, “Node”, “Port (HTTPS)”, “Requires Authentication” and URL of Help-Website. The following values can be defined using those fields:

• „Additional Client Service Nodes“: Mindbreeze nodes on which the Client Service is also started.
• In the section "Data Sources" the sorting of the data sources is defined. “Group by Category" groups the data sources by data source type (category) and then sorts them alphabetically. If you want to change the order of the data source types, enter the desired order, separated by a comma, in the field "Order of Categories". To make a customised sort by data source name, use the "Manual Order of Data Sources" field. If "Group by Category" is activated, the data sources are sorted to the front in the group. If you do not specify all data source types or names used, the remaining data sources will be displayed according to the default sorting. If a value contains a comma, you can use a JSON array instead of the comma-separated list.
• In the part “Filters” filters that are displayed in the Client Service can be chosen. Filters that can be selected depend on the selected Query Services and the defined “Aggregation Metadata Keys”.
• In the “Query Engines” field you can choose the data sources which will be made available over the client service.
• „Federated Query Engines“: Enter URLs of query services that are not managed by this Manager Node here. For authentication a special multi master configuration is necessary, see the whitepaper "SAML-based Authentication (eng)" for details.
• „Federated Client Services“: URLs of Client Services which should be included for all users.
• „Display federated results immediately, ignoring global relevancy ranking “: if activated all requested results from a source are displayed. This may include less relevant results on the first page.
• "Federated Client Services Use Legacy Messageframe Channel": If this option is activated, the communication to the federated Client Services runs via the deprecated Messageframe Channel. This option can be used if you want to include Client Services that run on an InSpire appliance version 20.2 or older.
• „Use Legacy Messageframe Channel“: To also use the Legacy Messageframe Channel for Client Services that were federated directly at the Search Client via the settings, set the option "Use Legacy Messageframe Channel" to “Enable”. The following values can be selected:
  • "Auto": Mindbreeze InSpire decides itself whether to use the Legacy Messageframe Channel (recommended)
  • "Enable": explicitly enables the Legacy Messageframe Channel
  • "Disable": explicitly disables the Legacy Messageframe Channel
• „Metrics Query Engines“: If user requests are to be used for auto-completion, the URLs of query engines with recorded metrics must be entered here.
• „Federated Sources“: Activate “Enable Fabasoft Mindbreeze Cloud Sources” to federate several Fabasoft Mindbreeze Cloud services. To add own services, reference additional service lists.
• „API V2 Concurrent Request Limits”: Set the maximum number of concurrent requests for each API service. If the limit is reached, the API calls return the status message „Maximum number of concurrent requests exceeded, please try again later!“ until the number of concurrent requests is below the limit. 0 means no limit. Default: 0
• „API V2 Named Concurrent Request Limits“: Here you can configure the maximum number of simultaneously processed requests per request name. The request name can be set with the request header “x-mes-api-request-name.” If the “Request Name Pattern” regex matches the request name, the number of requests processed simultaneously will be set to the value in the “Maximum Concurrent Request Count” field. 0 also means no limit here.
• In the section "Query Persistence Settings" you can enable searches to be saved for the user. Activate the functionality with "Enable" and configure a database connection to save the searches using the fields "JDBC URL", "User", "Password" and "Database Table Prefix". The "Database Table Prefix" can be used to store different client services in the same database. The "User" and "Password" parameters can also be configured as username/password credential. To do this, you have to create an endpoint entry for the JDBC URL.
• If the option “Count Filtered Values” in section “Query Settings” is enabled filter counts are also displayed for not selected values.
• If the option “Enable Character NGRAMs” in section “Query Settings” is enabled, Character NGRAMs can be activated or deactivated. Default: true.
• If the option „Ignore Global Uniform Properties“ is enabled, metadata defined in the Uniform Property Descriptor can be overwritten by the Category Descriptor.

Memory Analysis Settings (Advanced Settings)

Configuring permitted forwarding URL for user login

If a user logs in to the client service or if the Insight App Editor is used, HTTP forwarding is performed by the browser (via the address /mashup-login) depending on the configured login type. For security reasons, no forwarding to arbitrary URLs is allowed. These settings can be used to configure which URLs are permitted.

By default, "Allow login redirect URLs to" is set to "Client Service External URL." This means that only URLs that correspond to the "External URL" of the client services are allowed. For example, if the client service external URL is https://search.mycompany.com, then the URL https://search.mycompany.com/login is permitted, but the URL https://crm.mycompany.com/login is not.
Note: If the external URL is not set in the client service, then only the URL model is checked, either HTTP or HTTPS, depending on the "Use SSL" setting in the client service.
Relative URLs are always permitted. If the URL is not permitted, the browser gets an "HTTP 403 Forbidden" error message.

The default settings are usually sufficient for simple applications. In special cases, such as load balancers with differing client service "External URLs" or reverse proxies that terminate SSL, the default settings are not suitable and cause HTTP 403 errors. For these special applications, the setting "Allow login redirect URLs to" must be set to "Custom Pattern", and a regular expression (Java) must be specified for "Custom Pattern". The regular expression is matched directly against the forwarding URL. If there is a match, the URL is permitted, otherwise an HTTP 403 error is output. An example of a regular expression would be https://search.mycompany.com.* which allows the URL https://search.mycompany.com/login, but not https://crm.mycompany.com/login.
Note: If the regular expression is missing or incorrect, then no forwarding URL is permitted.

Settings for impersonating search queries

The impersonation of search queries is used, for example, in the "InSpire AI Chat and Insight Services for Retrieval Augmented Generation" in the "Retrieval" step.

Impersonation Zone ID

For security reasons, an impersonation token issued by a client service is by default only accepted by the same client service or by synchronized client services (multi-node scenario). A possible use case is, for example, an InSpire AI Chat that is operated behind a load balancer and performs a failover to another client service.

If required, this behavior can be influenced in the section "Impersonation Settings" with the setting "Zone ID". The Zone ID can be configured to an arbitrary value, which is then used to validate the impersonation token. A possible use case is, for example, to configure the same Zone ID on another client service (on the same or on another node) so that the "InSpire AI Chat" retrieval search process is possible on a different client service than the one that is used by the user to access the service.

Security Notes:

Ensure that all Client Services where you set a Zone ID have a correctly configured authentication. A valid impersonation token enables a search query to be sent without further end user authentication. The overall system is only as secure as the least secure client service of the client services involved.

Settings for non-interactive Impersonation

Configuring settings for validating requests

The client service supports the validation of the HTTP host header of requests. This can improve security. The setting "Validate HTTP Request Host Header Pattern" can be used to specify a regular expression that matches the host. The request is only processed if there is a match. Otherwise, an error is noted in the log and the request is rejected with the status HTTP 403.

Configuring settings for the delivery of images

By default, non-static images, such as thumbnails or icons, are delivered as data URLs. If, for instance, a custom client causes problems with data URLs, the setting “Enable Get Image Resources In Separate Requests” can be used to switch the delivery to HTTP(S) URLs. This setting may be removed in the next version of Mindbreeze. Therefore, the custom client must be adapted and data URLs have to be used in order to work even after the next update. Security notice: If this setting is active, the host name in the URL will not be validated.

Configuring settings for Query Service

In a producer-consumer scenario, search queries on the consumer node are directed to the index in the consumer node by default. If, for example, the consumer index cannot be reached due to maintenance, the consumer cannot be used for a search. The Client Service setting "Enable Fallback to Query Services on other Nodes" can be activated to send the search query directly to the Producer Index in such situations. This improves the availability of the search in producer-consumer scenarios.

The setting "Use Credentials from Endpoint Mapping Fallback" is intended for internal use and does not need to be changed.

Enable Saved Searches

• In the Section “Query Persistence Settings” you can set up server-side searches. If enabled is checked, the user can store queries in a database and access them via the Client-Service. Enable this functionality by checking „Enable” and setting up your database connection. The „Database Table Prefix” can be used to use different Client-Services in the same database.

Settings:

• Enable: To save the searches on the server enable this checkbox
• JDBC URL: URL to database
• Max Number Of Database Connections: Maximum number of database connections. Default value: 10, minimum value: 2.
• User: Username of the database
• Password: Password for the user of the database
• Database Table Prefix: Set a table prefix if is needed

Optional Authentication

With the “optional authentication” setting the Mindbreeze InSpire Client Service allows anonymous search in the documents that have no access restrictions. The user can log in for accessing the contents that are restricted and can optionally return to anonymous search by logging off.

For configuring optional authentication, on the Mindbreeze InSpire configuration interface navigate to the “Client Services” tab and set the “Requires Authentication” option to “Optional”.

The optional authentication setting requires that the Client Service has the “Authentication Generates Trusted Peer Credentials” is checked and a “Trusted Peer Credential Certificate” is selected for the Client Service.

If the trusted peer certificate is not available the client service does not allow anonymous access and login is mandatory.

If optional authentication is successfully set the user can switch between authenticated and anonymous modes by clicking on the “Login” respective “Logout” links on the Client Service user interface:

Show/hide user name

If a user is logged in and a user name is available, it can be shown or hidden using the “Display Username” setting.

When this setting is enabled, the full user name is output. If the setting is not enabled, "Login" or "Logout" appears, depending on the login status.

CORS header

If you use federated search, or use a Insight App running on a different server, these settings may be relevant to you. In such scenarios, Web browsers usually prohibit communication with other servers, except for non-authenticated public servers, since no critical data is transferred there.

The option "Allowed Origins" controls which "Origins" are allowed. Origins are absolute URLs from which requests are allowed.

If you want to use a client service with authentication from other origins with different private domains, you must explicitly list the URLs of these origins in the “Allowed Origins” option. For example, http://search.mycompany.com,https://search.mycorporation.com

Alternatively, you can use the “Allowed Origins Pattern” option to control which origins are allowed using regular expressions. You can specify multiple lines here. For example, the value

https://.*\.mycompany\.com

https://.*\.mycorporation\.com

allows access from, for example

https://search.mycompany.com

https://myapp.mycompany.com

https://find.mycorporation.com

but not from

https://search.example.com

Note: as soon as you use the “Allowed Origins Pattern” option, the “Allowed Origins” option has no effect.

Default values

For non-authenticated ClientServices ("Requires Authentication": "No"), the value ".*" is assumed for "Allowed Origins Pattern" by default if "Allowed Origins (Pattern)" is not explicitly configured.

For (optional) authenticated ClientServices ("Requires Authentication": "Optional" or "Yes"), all hosts with any port that are within your top private domain are allowed by default if "Allowed Origins (Pattern)" is not explicitly configured. The "top private domain" is the domain that is one level below the public suffix (as defined in the Mozilla Foundation's Public Suffix List (PSL)). For example, such a domain would be "mindbreeze.com", from which the pattern "(.*\.)?\Qmindbreeze.com\E(:[0-9]+)? " is generated. The domain name is extracted from the "External URL". If this is not configured, it is extracted from the configured "Hostname" from the nodes configuration. If this is also not possible, the domain name is extracted from the system FQDN.

CORS with SAML

If your SAML IDP is not within the domain of your Mindbreeze InSpire appliance, configurations must be made under certain conditions. Mostly, this is of interest for SAML IDPs in the cloud. Please ensure that Mindbreeze InSpire trusts your IDP. To do this, first activate the Advanced Settings.

Attention: When redirecting through the SAML IDP back to the log-in screen in the client service, a 403 error may appear. The reason for this is that the "Origin” HTTP header has the value "null". To prevent this, set in the setting "Allowed Origins Pattern" the value "null".

Preview Settings

Whether to automatically scroll to the most relevant match result when previewing PDF files. Default: false.

Healthcheck

The health check of the Client Service is configured in the “Healthcheck Settings“ section.

The address <Adresse des Client Services>/ping indicates whether the client service is operational. This allows the service to be monitored in order to inform the operation or to enable/disable it on a load balancer, for example.

By default, the "Workload Check" checks the load of the Web server and the number of parallel requests (see also API V2 Concurrent Request Limits). The "Disabled" setting disables this check.

In addition, you can add your own checks in "JSON Healthcheck Files Directory". This allows search queries to be executed and their results can be checked and processed with Javascript.

Responses

Voting

In order to offer the possibility to give feedback on a result (positive or negative feedback), the option "Enable Voting" can be set in the client service.

If this option is active, all results in the client are equipped two additional buttons.

You can analyze the feedback in app.telemetry (Application -> Query Service Query Log -> View Telemetry Data).

Cookie Settings

The Client Service automatically sets a session cookie (JSESSIONID) by default. If desired, this can be prevented by activating the setting "Disable Session Cookie" (If, for example, the Client Service is operated publicly and cookies are not permitted for legal reasons). Hints: This setting can impair performance. This setting must not be used in conjunction with the authorization form SAML.

The "Same Site Cookie Behavior" setting determines whether cookies are used across domains. If you use the federated search and the servers are on different domains (e.g. search.mycompany.de federated to search.mycorporation.com) and you use cookies to log in to the browser (for example, with the authorization form SAML), then "Same Site Cookie Behavior" must be set to the value "Auto" (default value). This ensures that the cookies required for authorization can be transferred. Note: the "Auto" value determines that when "Use SSL" is active, effectively cookies with Same Site Cookie Behavior "None" are set. If "Use SSL" is not active (e.g. when using a load balancer), no Same Site Cookie Behavior is set.

If you are not using federated search with cookie-based authorization, you can set the “Same Site Cookie Behavior” setting to "Strict" for increased security. This will prevent cookies from being passed on. The other possible values of this setting are for internal use and should not be used.

Content Security Policy Settings

It is possible to configure a Content-Security-Policy Header (CSP Header) in the Client Service.

This is sent for every request from an Insight app unless it is a URL that contains the pattern: “https://<<your-domain.com>>/api/...”.

Basic Configuration

The minimum configuration consists of selecting the setting: “Enable CSP” and entering a valid “External URL”.

The following CSP header is generated using these two options:

Content-Security-Policy: frame-ancestor 'self' <<externalurl>>; object-src 'none';

This header can prevent potential clickjacking attacks and security scanners used will no longer list this potential vulnerability.

Advanced Configuration

In the advanced configuration, custom policy directives can be defined that overwrite the basic configuration.

To do this, the “Enable CSP” checkbox must also be selected and one or more “Custom Policy Directive” must be created.

This enables an extension of the basic configuration, whereby no automatic directives are created in the extended configuration (frame-ancestor & object-src).

It is possible to create a part of a new directive without a value, such as is the case with the following header:

Content-Security-Policy: 'unsafe-inline'; frame-ancestors 'self'; ...

Operating custom Insight Apps using the Client Service

To operate your own Insight Apps using the Client Service please use the section “Web Application Contexts Settings” as described in the document “Development of Insight Apps”

Using port 80 as the client service port on G7 appliances

To run a client service on port 80, the following steps are required on G7 appliances:
Disable the option: “Use SSL (HTTPS)” and set the “Port (HTTP)”  to 23350. Port 80 is automatically forwarded to this port.

For security reasons, access to port 80 is restricted. To allow access for specific IP addresses or subnets, edit the file: “/var/data/iptables.sh”.

In the line “iptables -t nat -A PREROUTING -m addrtype --dst-type LOCAL -s 127.0.0.1 -p tcp -m tcp --dport 80 -j DOCKER,” enter the allowed addresses instead of 127.0.0.1 or remove “-s 127.0.0.1”  to enable access from anywhere. Then restart the appliance to apply the firewall rules.

During an update, the file “/var/data/iptables.sh” is overwritten and a backup in the format "/var/data/iptables.sh.bak.YYYYY-MM-DD" is automatically created. If necessary, restore your customized rules after the update.

“License” tab

The “License“ tab is used to manage the Mindbreeze InSpire license.

To reinstall or upgrade a license, perform the following steps:

1. In the field next to "License": select your desired license file. You can use the "Choose File" button for this purpose.
2. Then click on "Upload" and save by clicking on "Save".

After saving the license file, the name of the company licensed for the current installation of Mindbreeze InSpire and the license expiration date is displayed in the “Current License Information” box.

The “Licensed Products” section displays your licensed products and their restrictions.
The restrictions include:

• Maximum User Count: Displays the maximum number of users for which the current license is issued.
• Maximum Document Count: Displays the maximum number of indexable documents for which the current license is issued.

“Certificates“ tab

General information about Trusted Peers authentication

Mindbreeze InSpire offers the possibility for third-party applications to issue queries to the Query Service without providing complete user credentials. Such applications are called ”trusted peers“ and must authenticate themselves using a SSL certificate. In order to ensure the confidentiality of the data stored in the index, it is required that such certificates are signed by a Certificate Authority (CA) which has been registered within Mindbreeze InSpire.

In order to define the trusted CA, use the ”Certificates“ tab to upload the “.CER” file containing the certificate of the CA in PEM format. If you don’t upload any CA certificate, the functionality of trusted peers will not be available. The option “Trusted Peer” enables if an available certificate is used for this purpose. The box “Current Trusted CA Information” shows the currently registered certificates.

Authentication with client-certificates

All CA-certificates („Available CAs“) can also be used for authentication via client-certificates. This type of authentication can be used from the Windows Client. Every user has to present a certificate signed by a specified CA. The CA has to be defined in the index settings by selecting a certificate for the preference “Authentication Certificate” in “Advanced Settings”.

In order to operate the Web Client Service with a different SSL certificate than the supplied one, for example to use load-balancing, upload certificates in PKCS #12 format.

Prerequisites for certificates in PKCS #12 format

SSL/TLS certificates are available in various formats. For Mindbreeze InSpire, a specific format is necessary:

• Unencrypted PKCS #12 archive format (includes in most cases .p12 or .pfx file extension) with the following contents:
  • Unencrypted private key
  • Subject public key
  • Root public key
  • Certificate chain
  • No import password set for PKCS #12

Attention: Uploading a certificate in a different format will result in the failing of the installation.

Upload and activation of SSL certificates

In the following chapters, the upload of an SSL certificate is explained as well as enabling the SSL certificate for the Client Service and the Mindbreeze Management Center. Be aware, that enabling the SSL certificate for the Client Service and the Mindbreeze Management Center has to be done separately. This is because the use of the SSL certificate differentiates. For the Mindbreeze InSpire Management Center, the SSL certificate is used for the administration interface. For the Mindbreeze Client Services, a per service configured SSL server certificate is used to correspond with the external server URL accessible to the end user.

Tutorial Video “Install SSL certificate”

Information on how to upload and activate an SSL certificate for the client service and for the Mindbreeze Management Center is available in the following video: https://www.youtube.com/watch?v=oThC_VNcc5s

The following chapters provide the information mentioned in the video and additional information.

Upload

To upload an SSL certificate, go to “Configuration” and then to the tab “Certificates”. Here you can switch the type of the certificate between “Auto”, “CA” and “SSL”. Switch the type to “SSL” and then select the SSL certificate with “Choose File”. Finally, click “Upload” to upload the chosen SSL certificate.

All the uploaded certificates are listed below in the section “Available SSL Certificates”. These certificates are available to be chosen for each Web Client Service.

Enabling SSL certificates for the Client Service

Go to „Configuration“ and then to „Client Services“. Activate “Advanced Settings” and open your Web Client Service. In the first section “Setup”, go to the setting “Use SSL (HTTPS)”. If this setting is not active, please activate it. Then, go to the setting “SSL Certificate”. The default setting “Use SSL Certificate supplied with your license” uses the certificate supplied with your license. Open the drop-down-menu and select the SSL certificate you want to use. Finally, make sure that “Apply changes and restart on save” is activated in the top right corner and click “Save”.

Attention: The Client Service only accepts SSL certificates in PKCS #12 format with empty import password. The SSL certificate file must contain a private key and the corresponding server certificate.

Enabling SSL certificates for the Mindbreeze Management Center

Go to „Setup“ and then to „SSL Certificate“. Click „Choose File“ to select the SSL certificate you want to use. Then upload the certificate with „Upload File“. After the upload is done, refresh the Mindbreeze Management Center to enable the SSL certificate.

Attention: The Management Center only accepts SSL certificates in the PKCS #12 format. If the certificate has an import password, it can be specified in the field “Password”. The SSL certificate file must contain a private key and the corresponding server certificate.

Upload and activation of SSL certificates for multiple Mindbreeze InSpire appliances

In the case of multiple Mindbreeze InSpire appliances that are interconnected with each other, the handling of the SSL certificate must be done in a different way. Such a case can be present, for example, in a Producer-Consumer infrastructure.

For the SSL certificate to work properly, one of the following two points must be provided:

• The SSL certificate is valid for all domains involved.
• The SSL certificate is a wildcard certificate.

If one of the two points is provided, the SSL certificate must be installed on the Master appliance. After that, the Task Manager will synchronize the certificate to the connected appliances when carrying out the task “Synchronize config and data”. The configured tasks of the Task Master can be found in the Management Center, in the main menu item “Setup” under “Tasks”.

Verification of a successful SSL certificate activation

After the activation of a SSL certificate for the Client Service and/or Management Center, it can happen that the old certificate is still displayed in the browser. This is because browsers often include the certificate into the cache and the new certificate won’t be displayed immediately, although the background services were restarted.

To resolve this issue, please try the URLs of the Client Service or Management Center in a different browser or restart the current browser. After a couple of minutes, the new certificate should be visible.

“Network“ Tab

The “Network“ tab enables common network configurations for all services.

Proxy Settings

These proxy settings are used by all Mindbreeze Enterprise Search services in order to access web resources through a proxy server. Host address and port of proxy server and a valid username and password is to be provided if necessary.

LDAP Settings

This information is important for the connection with the LDAP servers necessary for authorisations:

The LDAP queries are logged in the "Network Requests" log pool of AppTelemetry. Scheme "ldap" and port "389" can be used as filters. All queries that are present in the cache have the status "Persisted Cache".

“About“ Tab

The “About“ tab shows common information about the current installation of Mindbreeze InSpire, such as the version number and the copyright.

Recovering Mindbreeze Configuration from a backup

When a configuration change is saved, backups of the Mindbreeze configuration files (mesconfig.xml and pluginsite.xml) are automatically created. The backup files can be found in the same folder as the original configuration files:

%userprofile%\AppData\Roaming\Mindbreeze\Enterprise Search\Server\,

The %userprofile% folder is the profile folder of the Mindbreeze Manager Service user. If the service is started with the system user, the configuration files are located in

C:\Windows\System32\config\systemprofile\ AppData\Roaming\Mindbreeze\Enterprise Search\Server\

The backups have the following naming schema: mesconfig.xml.backup_<timestamp> und pluginsite.xml.backup_<timestamp>.

For recovering the last state of the Mindbreeze configuration the following steps are necessary:

• Stop the Mindbreeze Manager and Node services
• Replace the files mesconfig.xml and pluginsite.xml with the corresponding backups:  mesconfig.xml.backup_<timestamp>  
• Start the Mindbreeze Manager and Node services.

Import/Export of Settings

Settings of various services can be imported and exported using this component:

Format

The following format is used for the import and export of settings:
<settings>
    <attributes>
        <attribute name="name" value="value"></attribute>
    </attributes>
    <properties>
        <property name="name" value="value"></property>
    </properties>
</settings>

Export

The export window (on the left) reads all available options from the service. These options can then be uses for importing into another service.

Import

The import window (in the middle) displays the updated configuration. Notice: The services have to be of the same type for this to work.

Changes

The changes window (on the right) displays a visual diff of the changes.

Remove exisiting settings

If this option is active the configuration of the target service is overwritten.

The following options are never overwritten:

• Service Name
• Index Path
• Index Port (HTTP)
• Data Port (TCP/IP)
• Query Port (HTTPS)
• Filter Service
• Caching Principal Resolution Service
• Authorization Service

If you only wanyt to extend or update the configuration you can disable this option.

Parameterization

Introduction

Through configuration parameters and so-called “Development Snapshots”, changes to

• the Mindbreeze Service configuration (add/remove and customize connectors, indices, filters, client services, ...)
• the semantics pipeline
• the Query Transformation Pipeline
• InSpire Insight Apps
• Any resource files like boosts, relevance parameters

can be exported as a development snapshot and then automatically transferred to production. Any settings (e.g. the data source URL to be indexed) can be overwritten locally as parameters on the respective environment. This ensures that the production data sources are indexed productively and the developer data sources in the development system. Credentials, certificates are not stored and are preserved.

Enable/Disable Parameterization

This feature is available only for G7 appliances.

From the ‘Indices’ tab, check the advanced settings, you can enable or disable the parameterization feature from the following table.

Note: To disable parametrization, click on the ‘Disable Parameterized Configuration’ button. If there are any active parameterized configuration options, the button is disabled. To disable parameterization in that case, you have to remove all parameters first.

Add a Parameter

Once parametrization is enabled, you can parameterize a configuration option from the following ‘Add/Update Parameter’ (…) button.

You can select one of the existing parameters in your node environment or add a new parameter from the following table.

Note: The parameter’s name must not include any white spaces or special characters.

To apply changes, you have to choose a parameter from the table list.

Note: Choosing or selecting a parameter is done by clicking on the table row. If selected, it is yellow highlighted.

After clicking apply, the value of the configuration option ‘Crawling Root [1]’ will be the value of the parameter chosen from the previous table. In addition, the configuration option’s value is now read-only.

Update a Parameter

To edit the parameterized configuration option (e.g. change the parameter value or choose another parameter), click on the following ‘Add/Update parameter’ button.

The dialog is opened, where the referenced parameter for this configuration option is automatically chosen (highlighted) from the available list.

Change the value of the parameter (e.g. ‘https://another_site.com/events’) and click apply:

The configuration option’s value will also change accordingly.

Note: Another possible change is to select another parameter (e.g. ‘param2’). In this case, the configuration ‘Crawling Root [1]’ will now refer to the newly chosen parameter.

And the value of ‘Crawling Root [1]’ will accordingly refer to the value of ‘param2’.

Remove a Parameter

To remove a parameter from the configuration option ‘Crawling Root [1]’, you can click on the following ‘Remove parameter’ (x) button.

Note: The value of the configuration option ‘Crawling Root [1]’ is now read/write and it takes the value of the last referenced parameter.

Now clicking on the above ‘Add/Update Parameter’ button will re-open the dialog to parameterize this configuration option from the beginning.

Mindbreeze InSpire Query language

The Mindbreeze InSpire Query language is used to specify queries.

4.1 Querying single terms

To search for a word or the first letters of a word no wildcard characters (%, *, etc...) are required.

Entering “act” initiates a search for objects that start with the term “act” or contain the word “act”. During a search capitalization is ignored, i.e. a search for the term “act” returns the same results as a search for “Act” or “ACT”, since the query language does not distinguish between upper and lower case letters.

Search for multiple terms in one document

In addition to querying single terms, you can query for multiple terms within one document. A search for multiple terms covers documents containing the terms themselves as well as documents containing words starting with these terms. To be part of the search result, all terms entered have to be contained in a document.

These three alternatives return the same search result: documents containing words beginning with ”car“ and “test” or containing “car” and “test” as independent terms. The query language does not distinguish between upper and lower case letters.

Search for phrases/definite search

A search for phrases searches for definite words or phrases. This kind of search is initiated via quotation marks (“) at the beginning and at the end of a phrase.

The exact phrase is searched for. Searching for phrases does not make sense if the exact spelling of the words or the phrase is unknown.

Restriction to file extensions

Mindbreeze InSpire is able to restrict the search to files with particular file extensions.

This query searches all files with the file extensions “.doc” (Microsoft Word), ”.xls” (Microsoft Excel) and ”.msg” (Microsoft Outlook) for the word “mind” or words starting with "mind" in upper case or lower case letters.

Logic operations

AND

Phrases, words and word beginnings in a search query are implicitly combined with the logical operator AND. The search delivers documents containing all phrases, words and word beginnings listed in the search query. The keyword AND can also be included explicitly in a (for example nested) search query.

OR

The logical operator OR delivers all documents containing at least one of the search criteria: at least one of the phrases, words or word beginnings entered. The search result also contains documents, containing only one entered term or one word beginning with an entered term or containing one of the entered phrases. The key word OR has to be explicitly defined within a search query and can also be used in a nested query.

These two queries deliver all documents containing the word “Mindbreeze” and/or the word ”Search” together with the word “Software“. They deliver documents containing the combinations ”Mindbreeze” and ”Software“, “Search” and ”Software” or ”Mindbreeze”, ”Search” and ”Software”.

Key words

NEAR

A search with the NEAR operator delivers documents, in which one word is found near another word.

NOT

A search with the NOT operator returns results within a source set where the word does not occur. NOT cannot be specified without any other word that yields results.

Metadata search

A metadata search is primarily used to refine a search result via additional restrictions. Mindbreeze InSpire provides some default metadata. In addition, manufacturer dependent metadata (defined by Mindbreeze partners) can be used.

Syntax of a metadata search: <metadatum>:<value>

A search for a file extension can be defined via the metadatum ”extension”.

In this example both alternatives produce the same search result: Microsoft Word files containing the word “mind” or words starting with “mind”.

The following table shows the metadata available for the data sources provided by Mindbreeze InSpire by default:

The Microsoft Exchange Connector defines the metadata terms from and to.

This search query delivers all objects sent by an address with the term ”bauernf”.

Interval Search

A Query containing the „TO“ operator returns search terms between the left and the rigth side of the operator. This is particularly useful when combined with numerical strings. Mindbreeze recognizes numerical values in various formats, for example:

Interval Search Syntax: <from> TO <to>

Extended Metadata Interval Search

Extended metadata interval search syntax:

label:[from> TO <to>]

label:[<from>]

label:[TO <to>]

Combination of language elements

It is possible to combine the described language elements of the Mindbreeze InSpire query language.

This example delivers Microsoft Word documents sent by an address with the term ”bauernf“ in it and with a title containing the word “Integration" or a word beginning with ”Integration”.

Useful search results even if not all search terms match

Using the Optional Terms, finding documents can be simplified by providing results in which not all search terms necessarily have to occur. To make this transparent for the user, a note is displayed with the search result. This feature is active by default, but can be parameterised or completely deactivated (see section Optional Terms).

By default, documents are found in which at least two thirds (67%) of the search terms occur. Since the search query in this example contains 5 search terms, one term can be missing in the result. The screenshot below shows an example where a document is found for this search query that does not contain the term "Article".

Operation and Maintenance

Changing the Index Service-Mode

A Mindbreeze InSpire Index Service supports the following modes:

• Mode: running ()
  This is the default mode of a Mindbreeze InSpire Index Service. This mode represents normal operation
• Mode: readonly ()
  This mode is used to provide index consistency during backup of the index files for Mindbreeze InSpire
  
  Hint: To allow resumption of the indexing process or start indexing of new documents (delta-indexing), the state must be changed back to “Mode: running”.
• Mode: offline ()
  This mode cannot be set explicitly by the user. An Index Service is in this mode when it has been stopped completely
  

Manually changing the Index Service-Mode

In addition to automatic state changes, the mode can also be changed via the Mindbreeze InSpire configuration user interface. In the “Services” field click the icon in the “Associated Index” column to change the mode. Clicking it again changes the mode back to the original state.

Hint: If the Index Service is not running, there will also be an indication why. The mode of the index service cannot be changed when in this state via the user interface, you must start the Index Service manually.

Changing the Index Service-Mode using a Script

• To change the index service mode to “readonly”, please run the following from the Command Prompt:
  mescontrol http://indexserver.yourcompany.com:23100 readonly

• To change the index service mode to “running”, please run the following from the Command Prompt:
  mescontrol http://indexserver.yourcompany.com:23100 readwrite

Backing up the index data

Mindbreeze InSpire uses a file-based index. These index files can be backed up completely in a consistent state.

To save index data, perform the following steps:

1. Change the mode of the index service to “Mode: readonly “().
2. Check the consistency of the index with:
   mescontrol http://index.yourcompany.com:23100 checkconsistency
3. Verify the exit code (ERRORLEVEL) of mescontrol
4. Navigate to the directory where the index is stored, and backup the files within this directory, only when no errors occurred up to now
5. Change the mode of the index service to “Mode: running” ()

Restoring index data

To restore a previously saved index, perform the following steps:

• Stop the Index Service.
• Delete any existing files in the index directory (if not required any more) or change the path to the index files of the Index Service in the configuration to a new.
  Hint: If you define another directory, make sure that the service user has write access to the defined path.
• Copy the restored index files into the directory.
• Restart the Index Service.

Index Status Information

Index Statistics

Every Index service provides detailed status information about status of the indexing process and the number of documents indexed by using the “/statistics” URL path:

An example endpoint of an index service on host “indexserver.mycompany.com” listening on 23100 would result in the following URL: http://indexserver.mycompany.com:23100/statistics

Indexed Documents

In addition to getting statistics on the indexing status one can use the “/documents” URL Path to browse indexed documents either by document key (depending on the connector in use) or by document id (docid). Please note that the docid is an internal sequence number and varies between indexing runs.

An example endpoint of an index service on host “indexserver.mycompany.com” listening on 23100 would result in the following URL: http://indexserver.mycompany.com:23100/documents

Retrieving Index Status Information

For health checking purposes one can use the raw index status handler available via /index_mode on the index service’s bind port. For instance on an Index Service running on indexserver.mycompany.com that is listening on port 23100 it would be: http://indexserver.mycompany.com:23100/index_mode

If the Index Service receives a request on this end point, the index responds with status information in form of an XML document that has the following schema:

<status mode=”<status-information>” />

<status-information> indicates the mode of the index which can be:

• normal (read write)
• readonly
• offline (closed)

Receiving an HTTP status code other than 200 also indicates that the index is not fully operational.

Backup of log files

The log files are archived regularly. This is done using a cron job, which is run every Sunday at 2:30 by default.

All log files from the paths /data/logs and /var/opt/mindbreeze/log are archived and stored in /data/backups/log-backups.

Binary files are not backed up but removed during backup.

Only the first 50GB of files larger than that are backed up.

Query Service Reconfiguration

The following query service options can temporarily be changed without index restart, and these changes will not persistent in the index configuration:

• “Query Threads per Index”
• “Number of ACL Precomputation Threads“

This reconfiguration is possible only with disabled “Disable Unrestricted Privileged Servlets” option.

To change the number of query threads, please run the following from the Command Prompt:

mescontrol http://localhost:23100 reconfigure --query-threads=<n>

To change the number of ACL precomputation threads, please run the following from the Command Prompt:

mescontrol http://localhost:23100 reconfigure --precompute-acl-threads=<n>

Appendix A

Service Restart Behavior after Configuration Changes

Appendix B

Manual Configuration of Kerberos-based Authentication

The following steps are needed for manual configuration:

• Set the HTTP/<host_fqdn> service principal name for the service user, using the setspn tool or add manually using adsiedit.msc. Here the <host_fqdn> is the fully qualified domain name of the Mindbreeze InSpire Node:
  
  e.g setspn –a HTTP/myserver.mydomain.com DOMAIN\serviceuser.
  
  Make sure that the SPN is not already set for a different user or host object from the current Windows Active Directory, e.g. using setspn -x (available on Windows Server 2008 servers) or try third party tools like dumpspn.
• Set the trusted for delegation flag for the service user in the "Users and Computers" management console plugin (dsa.msc).

Appendix C

This appendix lists useful administration details for Mindbreeze InSpire components.

Web browser configuration

The security restrictions of web-browsers limit the use of file resources (e.g. file://myserver.mycompany.com/share/letter.doc) and they are not accessible in most browsers without modification of security settings. Kerberos support for authentication is also not configured by default but Mindbreeze InSpire uses Kerberos Single Sign On for secure network wide search. This section shows how to manually configure the required browser settings based on Mozilla Firefox and Microsoft Internet Explorer.

Mozilla Firefox – Manually setting the settings

To manually configure Mozilla Firefox, type about:config in the address bar. This will show a list of configuration options.

The following configuration entries have to be set.

After restarting the Firefox browser, the changes should have been applied. Authentication with the Mindbreeze InSpire Web Client Service should now work as expected.

In addition to Kerberos authentication, another important configuration option is to be able to open file URLs. Mozilla Firefox up to version 1.4, and Mozilla Suite up to version 1.7.x provide only one global value to configure the settings of file URLs. This configuration is called ”security.checkloaduri“ and should be set to ”false“.

Warning: Globally setting this value might open a security risk when visiting malicious internet sites.

Newer versions of Mozilla Firefox (starting with version 1.5) and the Mozilla SeaMonkey Suite starting with version 1.0 are able to set the security settings for a set of web sites which are described by a policy.

To create a policy you have to manually edit the user.js configuration file that resides in your local Mozilla proflie folder. (e.g.: C:\Docments and Settings\User\Application Data\Mozilla\Firefox\Profiles\xxxxx.xxx\user.js).

Note: Please refer to the %USERPROFILE%\Application_Data\Mozilla\Firefox\profiles.ini configuration file, to find your active Firefox profile directory.

The following snippet shows the configuration of a new policy called “messecurity settings” for the Mindbreeze InSpire Web Client Service Node running on myserver.mycompany.com, on port 23350

user_pref("capability.policy.policynames", "messecuritysettings");
user_pref("capability.policy.messecuritysettings.sites", "https://myserver.mycompany.com:23350");
user_pref("capability.policy. messecuritysettings.checkloaduri.enabled", "allAccess");

Note: Several Web Client Services can be added to the policy by separating them with spaces.

Microsoft Internet Explorer

Setting the configuration manually

Microsoft Internet Explorer uses security zones to implement its security model. By default, a web site is located in the “Internet” zone. To grant the necessary rights to the Mindbreeze InSpire Web Client, add the URL of the Mindbreeze InSpire Web Client Services to the “Local intranet” zone.

Double-click the globe symbol toward the right of the status bar. Then in the “Internet Security” tab which is displayed, select “Local Intranet” and “Sites”. Add the Mindbreeze InSpire Client Service URL to the local intranet sites using the following the dialogs.

Add both the http as well as the https URL of the Mindbreeze Enterprise Search Web Client Services to the list of local intranet sites.

Example: Your Mindbreeze InSpire Web Client Service is available from myserver.mycompany.com add the following two entries to the list:

http://myserver.mycompany.com

https://myserver.mycompany.com

Also make sure that the option "Display Mixed Content" is enabled for your local intranet. You can check this setting with "Custom level".

Additionally, you should disable the “Do not save encrypted pages” option in the “Advanced” Tab.

To apply the changes, reload the Web Client page after the Web Client address has been added to the Intranet Zone.

Configuration of Microsoft Internet Explorer via Group Policies

Follow the steps below to automatically set the configuration described above for a specific organizational unit in your Active Directory domain. The following section guides you through the steps needed to create a group policy which adds the Mindbreeze InSpire Web Client Service to the Trusted Sites and the Mindbreeze certificate to the Root Certificate Authorities.

First, log on as a member of the “Domain Admins” group. Then open the “Active Directory Users and Computers Management” console. Right-click the domain or Organizational Unit where you want your Internet Explorer to be configured and click “Properties”.

Then select the Group Policy Tab and click the “New” button. Type a name for the new Group Policy Object. (e.g. MES IE Config). Then click the “Edit” button.