Release Notes for Mindbreeze InSpire

Version 22.1

Innovations and new features

Searchable video content

We are always striving to further optimize your search experience and make new and innovative file content searchable for you on an ongoing basis. This includes audio and visual based content such as movies or songs.

In the course of our 22.1 release, we will offer you a connection to video content, and the possibility to index it with your metadata and subtitles. Thus, you will now be able to query video content in addition to a normal title query.

Microsoft Stream, for example, is one such video service platform that we are making available to you as a connector as part of our new release.

Of course, we are happy to respond to your requests to index additional multimedia files or to develop a connector for you for any kind of video service platforms. For this, please contact your contact person or for new customers visit here on our homepage.

Semantic search

To make content browsing as efficient as possible, this release offers you numerous innovations in the area of semantics. As a result, Mindbreeze InSpire is capable of understanding your input and providing you with the right results even better.

Both the current and future semantic functions are based on automatic speech recognition, which is integrated into the index.

The concept of composite decomposition, newly introduced in the release, recognizes individual words that have been composed from several natural language words. Single word segments are therefore sufficient to find more complex compound words for the search query.

Link to documentation: Compound Splitting

Another semantic feature is the automatic recognition of named entities, also called Named Entity Recognition (NER). Building on an AI-based speech recognition, sentence segmentation and a POS tagging, entities are recognized in texts on the basis of learned models.
Currently, the entities listed below are supported. Pre-trained models for recognition for German and English are already shipped.

• Persons (entity:person)
• Locations (entity:location)
• Organizations (entity:organization)
• Numeric values (entity:number)

Link to documentation: Named Entity Recognition (NER)

App.telemetry dashboards for usage analysis

As part of the new release, you will now have the ability to get a visual overview of all your applications. This includes both an overall view of all applications as well as a detailed view of the individual applications. The different variants of diagrams and tables available to you enable you to always maintain an overview of the use of Mindbreeze InSpire within your company.

Among the options that are now newly available to you in the dashboards are:

• Monitoring of search queries
• Monitoring the relative usage of the applications
• Interactive diagrams
• Different time spans of the diagrams
• ...

This feature is enabled by default so that the dashboards are automatically generated and updated on a daily basis. However, there is an option to customize the configuration to your needs or disable the feature if needed.

Link to the documentation

New features in Insight App Designer

To expand your design options and at the same time make your search even more customized, there are now some new features available in the Insight App Designer.

One of these new features gives you the ability to go beyond a normal text query and additionally restrict search queries as a structural query in a JSON format.
For example, you can wrap more complex search constraints in a concise JSON format so that they remain hidden in the background and end users don't have to worry about the search constraint.

In addition, the system automatically detects whether the search restriction is a general text or structural query.

Link to documentation: Structural Query

Furthermore, for the already existing "Group results" feature, you can now use the corresponding widget in Insight App Designer and design it as you like. In addition, you have the option to group results by aggregated properties. For example, statistics of a group can be calculated and displayed.
This provides you with many setting options for grouping as well as flexible design options.

Link to documentation: Summarize By

Using the Consumer Index as a Producer

With the new Mindbreeze InSpire version, you are now able to use your standby appliance in failover operation directly for indexing and use the consumer index as producer index. Thus, for example, a Producer Index that is no longer available can be replaced, which eliminates the need to re-index the data.

Link to the documentation

Advanced connectors

Microsoft Stream Connector (New)

With the Microsoft Stream connector, you now have the ability to index videos from Microsoft Stream. In addition, both the metadata and the subtitles of the video content are also searchable and indexable.

Link to the documentation

Microsoft Azure Principal Resolution Service (New)

With the new Microsoft Azure Principal Resolution Service, you can resolve groups in the Microsoft Azure instance. With the added feature of resolving only Microsoft Teams groups, the cache can be limited to them, which also improves cache performance.
This service also replaces the Microsoft AAD Graph Principal Resolution Service and the Microsoft Teams Resolution Service.

Link to the documentation

COYO connector (new)

The new COYO connector allows you to index all possible content from your COYO instance. For example, articles, threads, blogs, wikis, forms, surveys, etc. can be found with Mindbreeze Search. In addition, the new COYO Principal Resolution Service offers you the possibility to resolve groups in your COYO instance.

Link to the documentation

Microsoft Graph Connector (New)

Another connector that is newly available to you is the Microsoft Graph connector. With this connector, all users in your Microsoft Graph instance can be indexed so that they can be found in Mindbreeze search. User data that is additionally indexed includes address, location, phone number, etc.

Link to the documentation

Microsoft Project Connector (New)

Using the new Microsoft Project connector, you can index projects from Microsoft Project in addition to your SharePoint on-premise instance. The use of the connector is recommended in combination with the Microsoft SharePoint connector.

Link to the documentation

Web connector: Reliable defense against data loss

With the new Allowed Delete Ratio option of the Web Connector, it is now possible to prevent data loss in case of network failures or unintentional configuration changes. This option is set to 80% percent by default and is also applied to the delete behaviour of delta sitemaps. This means, for example, if more than 80% of the document set would be deleted in a crawl run, the crawl run is automatically aborted. This gives the administrator the ability to decide if it is an intentional or unintentional deletion and act accordingly.

Link to the documentation

Technical extensions

Extension of the app.telemetry instrumentation

As part of the technical enhancements to our 22.1 release, the instrumentation of app.telemetry has been expanded.
This allows an even more detailed overview of your log information.

The newly added log information includes "Query Trigger Action" and "Query Trigger Type". This enables you to determine whether the query was automatic, filtered, normal, or how it was triggered.
Other new useful log information includes the URL or title of a search result.

Furthermore, we have created the ability for you to instrument any log information generically for your custom Insight Apps.

If you do not want to receive any log information, you can also disable the app.telemetry instrumentation.

Link to the documentation

Advanced automatic index compactification

For an improved resource efficiency   an automatic compactification of the update buckets is available from this release. If buckets have already been completely removed by the compactification, this feature recognizes that those documents no longer require the update information. These are then periodically removed automatically. Especially for indexes with frequent changes of documents in data sources, this leads in particular to a faster and more efficient synchronization.

Link to the documentation

Security relevant changes

22.1.2.1317

• Fix for: SQL-Injection in QueryLog Dashboards

22.1.0.1309

• Apache Tika updated to > 1.27 (CVE-2021-31811, CVE-2021-31812)
• CoreOS updated to 35.20220313.3.1
• Java updated: jdk8u322
• Libreoffice updated to >=7.0.6 (CVE-2021-25632, CVE-2021-25633, CVE-2021-25634, CVE-2021-25631, CVE-2021-25635, CVE-2021-25636)
• Moment.js update: CVE-2022-24785
• Remove Log4j2
• ShareZipLib updated: CVE-2021-32840
• TLSv1.1 disabled on nginx

Additional changes

22.1.7.1393

• Fix for: StringTable _cache log is held while trying to aquire the ExclusiveLock

22.1.6.1385

• Fix for: Segmentation Fault in TermsPositionReductionBoostTransformer

22.1.5.1377

• Fix for: JWT is broken for new realms (not supported realm key RSA-OAEP key is automatically used)
• Fix for: Performance Issue Crawlerstatus uses tryGetDocIDFromUniformItemID
• Expose createIndex Endpoint in /api/v1/indexquery/
• Fix for: Did you mean for persons may give wrong results
• Support diacritic expansion in QueryService
• Fix for: Documentum Crawler Fails to start: NoSuchMethodError

22.1.4.1342

• Fix for: Stemmer transforms labeled values but should not
• Fix for: Searchability issues when using phrase search with compound splitting
• Fix for: Did you mean? does not work for multi word searches

22.1.3.1327

• Fix for: Search using references and use_precomputed_acls_for_docinfo_access may be incomplete
• Fix for: Missing overflow check when calculating similar terms scores
• Fix for: Stemmer transliteration rules are not applied to queries with term series and subqueries
• Fix for: ZIP export does not contain PDF files

22.1.2.1317

• Improved error handling for update bucket compactification

22.1.0.1311

• Fix for: IllegalStateException in Yammer Principal Cache
• Updated code singing certificate

22.1.0.1309

• Authorized DocIDs used for access to all DocInfos
• BuildIndexTool can load items from a directory for index building
• CJK Tokenizer: Normalize strings based on fullwidth characters
• Confluence connector: Jericho filter CDATA of pages is indexed with Storage Content Representation
• Copyright in MMC changed to 2005-2022 during loading process
• Configurability of normalization of spaces in string labels perin SpanLabelTypeValueAsKeyProcessor.
• Filter Plugin for VTT File Format Support for Video Transcripts (MS Streams, YouTube)
• One-time processUpdatelist at the start of final buckets to ensure full inversion of all updates
• Basic script to create query log dashboards in app.telemetry via JSON config
• GSA Feed Adapter: do not save feeduuid as metadata
• Insight services for query expansion/question answering
• Cleanup of Java code from wildcard imports (type-import-on-demand declarations)
• Java SDK: cleanup of unneeded JARs in plugins generated by the SDK: protobuf-java*.jar, mesapi.jar, log4j-over-slf4j*.jar
• Jive Sitemap Generator: Support for Jive Version 9.9
• JWT filter: JWK file is configurable and can be updated
• Consistency of labels in the designer
• Support local index + remote connector E2E
• ParsedConfiguration: added correct .toString() for credential (and credential2)
• Principal resolution cache with nested containers supports pseudocontainers, shared containers across partitions, and container deletion
• Product: DefaultValueParser supports HTML metadata
• Refinement of logging in NLP plugin
• Python logging refactoring
• Remote QueryService plugin
• Improvement of reverse reference functionality after update to Consumer
• Salesforce: Crawl multiple HTML fields
• SDK Example Preparation for Task Management (Enable/Disable)
• Service UI - Feature Flag in MMC
• SharePoint Online Cache can resolve AAD groups on cache creation/update
• Sitecore connector: date values are (also) stored as date objects
• Sitecore connector: Correct indexing of metadata-only objects
• Sitemap Generator: Jive Sitemap Generator supports URL remapping
• Script for query log dashboards via Insight App
• SPO Crawler can exclude configured metadata (e.g. sp_expires) from indexing to reduce the number of metadata updates.
• Test 22.1: ABBYY OCR (ABBYFineReaderForPDFOCR, pdf2ocrfilterplugin)
• Improved logging of PrincipalCache synchronization in case of missing credentials
• Use log4j-over-slf4j instead of log4J in product and connectors.
• WCAG 2.1 Code Changes for "AA" Certification
• Windows Installer for packaging Models Compound Splitter on ngram frequencies via Wikipedia
• Fix for: Query expressions of type EXPR_TERMS in contextualization requests fail with protobuf parsing error in Fabasoft contextualization service
• Fix for: Queries with AND/OR conjunctions are not parsed correctly by the queryinfo widget
• Fix for: Aggregation result values of type ALTERNATIVES_LIST can lead to NPE in SearchResponseParserAndTransformer
• Fix for: Backend authentication via service accounts (JWT) does not correctly fall back to OAuth in SaaS environment
• Fix for: Editing from Category Descriptor (re-ZIP plugin with Linux Archive Manager) does not work with 21.2
• Fix for: Best Bets PhraseMatch returns unexpected results in this use case.
• Fix for: client-base.js does not work in IE11
• Fix for: Client Service Query Log Instrumentation: In a multi-node scenario with (temporarily) different versions, entries are logged in the wrong columns
• Fix for: Config UI Index Compactification Setting
• Fix for: Coyo Principal Cache: retry and robustification needed if object permissions are not reachable
• Fix for: CSS color of text selection in config is overridden by Ant design library in service UI
• Fix for: Service shutdown on expired license does not work if "Apply changes and restart on save" is not selected
• Fix for: DocInfo reversion does not continue after a termination
• Fix for: DocumentInfoVectorMerger_current defect
• Fix for: Error searching servlet for property of requested referenced element: message that required field is missing: property.value.kind
• Fix for: Global Index Settings Default values are not set
• Fix for: Handsontable in custom Insight apps: Template tree initialization could not be reported: TypeError: Handsontable is not a constructor
• Fix for: Hierarchical facets: Suggestion field does not work
• Fix for: Index DefaultTimezone parameter explicitly set creates wrong date filter values
• Fix for: InSpire AdminAPI triggers configuration change on crash/reboot if hostname_host is different from configured hostname
• Fix for: Jive Crawler/Princ Cache: User Agent header applied inconsistently
• Fix for: Jive Sitemap Generator cannot create a group list because an illegal character was found in the group name
• Fix for: JWT Backend Auth: JWTFetchTask retry doesn't handle keycloak restarts and stacktraces are missing
• Fix for: Components are not loaded in Insight App Designer
• Fix for: LDAP resolver unit test failure
• Fix for: Multiple searches and first searches when there are multiple requests.
• Fix for: mesutils FileUtils.deleteDirectory() does not delete symbolic links
• Fix for: Microsoft Azure Cache does not support nested groups
• Fix for: Possible division by zero in QueryServiceBase::documentHitIteratorForQuery
• Fix for: After installing a new MMC certificate, the nginx process hangs on shutdown and the MMC/search is no longer accessible
• Fix for: Unrecoverable consumer index crash
• Fix for: OptionalTerms does not work with people view
• Fix for: Outlook Addin cannot be downloaded from Client Service
• Fix for: Installing plugin with custom categorydescriptor causes ZipException: The version required to extract this entry is not supported (788)
• Fix for: Search relevance issues with 21.3 (active view constraints break boosting rules).
• Fix for: Problems with requests after language change
• Fix for: Salesforce NumberFormatException when no network proxy is used.
• Fix for: Disable Service Registration (old: Analysis Service Registration under Windows)
• Fix for: SharePoint Online Graph AAD Principal Cache fails with 404 when resolving members of groups
• Fix for: Using client-base.js with webpack may fail due to missing exports