Entity Recognition

Configuration

Mindbreeze GmbH, A-4020 Linz, 2024.

These documents are strictly confidential. The submission and presentation of these documents does not confer any rights to our software, our services and service outcomes, or other protected rights. The dissemination, publication, or reproduction hereof is prohibited.

For ease of readability, gender differentiation has been waived. Corresponding terms and definitions apply within the meaning and intent of the equal treatment principle for both sexes.

Introduction

This document deals with the concept, setup, and troubleshooting methods for configuring entity recognition.

Entity recognition configuration

In this chapter, the concept of entity recognition is explained using a simple example.
Follow these steps to set up:

Connect to the Management Center.
Navigate to the index that you want to configure with entity recognition.
Activate and then open the advanced settings.
Search for the section “Entity Recognition Parameter“ in the Management Center.
In the pattern-rules field, define your entity recognition rules, which should match your metadatum.
The following rule formats are supported: https://github.com/google/re2/wiki/Syntax
In our concrete example:

rule=/\// digits /\//.
digits=/\d+/.

Explanation

The first rule defines that all numbers between two slashes should match (regex):
Example: test/1234test1234/test/543/test (543 is extracted)
Now add a new metadata definition to apply the rules for metadata
In this example, Mindbreeze searches for numbers between two slashes in the string of the existing metadatum, in the “full string”. If there are numbers between two slashes, Mindbreeze takes the part of the matches configured in the sub-rule “digits” and writes it as a string in the new metadataum “myextractedVal”.

Example:
Full string: xyz/1234/herbert543/345test
Match of the rule “rule”: /1234/
Value of the rule “digits”: 1234
Value of the metadatum myextractedVal==1234

Entity Recognition Parameter

Configuring the Entity Recognition Parameters enables the index service to extract metadata from document contents. The following settings are available:

Setting	Description
Pattern Rules	Defines a set of rules that are applied during the metadata extraction. The rules are defined with a regex pattern. Please note that the regular expressions must also be enclosed with a "/".
Pattern Add Region Annotations	Adds the value of the setting "Use Link HREF pattern" to the annotation.
Process HTML Attributes	Enables the Entity Recognition to also search inside link references. Like <a href=link to be searchable.> link </a> in HTML source text.
HTML Attribute Name Pattern	Defines what attribute names should be searchable. It is defined with a regex pattern. In most cases a “href” is enough. It can also be simply extended with an ‘\|’ (“OR”) like this: “href\|link\|…” .

In the section “Add Metadata Definition” rules can be defined for each metadata. The following settings are available:

Setting	Description
If Rule Matches	A rule that defines the range in content where to extract metadata from. Should have the name of the rule that is defined in the setting “Pattern Rules”.
Name	The name of the metadata to be added to a document when the defined rule matches.
Value	A rule that defines the value of metadata. The value of the rule {{month}} can be normal text or composite.
Scope	A rule that defines an area or several areas with one entity recognition rule, in which the rules for extraction are to be applied. For this purpose, the name of the rule for selecting the area(s) is to be entered. In contrast to value extraction, you have to enter the name without “{{}}”.
Format	Enables the extraction of typed metadata like date from string. The known Types are "String", “Date”, “Number”. Only “Date” needs extra parameters “Format Options” and “Locale”.
Format Options	Mandatory for Format “Date”. Option to set Formatting of output. Exact definition can be found here: https://github.com/unicode-org/icu/blob/main/docs/userguide/format_parse/datetime/index.md#datetime-format-syntax. Define Order and what to output, like: "yyyyy.MMMM.dd HH:mm" to Print 2024.July.05 11:33
Locale	Only used for Format “Date”. Set “Locale” locale, if machine and user locale differ. Like ja_JP to display in Japanese default date format. Check https://github.com/unicode-org/icu/blob/main/docs/userguide/format_parse/datetime/index.md#datetimepatterngenerator.
In Existing Metadata	Defines to which metadata these rules should apply. For example: content, title, datasource/mes:key, <ownmetadatum>, etc.
Aggregatable	If checked, the generated metadatum will be static aggregatable.
Use Value for Sentence Embeddings	If this setting is activated, the recognized entities can be found with a Sentence Similarity Search (NLQA).
Annotate As	Defines how the entity is added to the metadata. The following options are available: Entity - The recognized entity will be added as an annotation. Recognized Entities are highlighted in the document preview. Link - The entity will be added as a link annotation. In the preview links are highlighted by a blue underline. Entity And Link - Combines the two options “Entity” and “Link”. Disabled - Entity and Link are not added.
Add Link With URL Pattern	Defines a pattern for the annotation link, if the setting “Annotate As” is set to “Link” or “Entity And Link”. It can use the regex definitions from the setting "Pattern Rules", which can be configured the same way as the setting “Value”. Used for something like: www.mindbreeze.com/link_to_item?item={{RuleName}}
Entity Label	Name of the created entity, if the setting “Annotate As” is set to “Entity” or “Entity And Link”.

Entity recognition (example: file system)

This chapter uses a simple example to explain entity recognition and its setup with Mindbreeze.

Configuration of entity recognition for a file system:

First the rules for the extraction have to be created:

host=/[^\\]+/.

share=/[^\\]+/.

directory=/[^\\]+/.

UNCPath="\\\\" host "\\" share "\\" directory "\\".

If rule matches: UNCPath

Name: Laufwerk

Value: {{share}}

In existing metadata: datasource/mes:key

If rule matches: UNCPath

Name: Projektpfad

Value: {{directory}}

In existing metadata: datasource/mes:key

Aggregated metadata keys (; separated)

Laufwerk;Projektpfad

Date formats for entity recognition are based on the ICU patterns (e.g. locale … de_AT)

Configuration for entity recognition for file system paths (variant 2) – with exceptions:

Even a complex case in which the rules are ambiguous can be achieved using alternative rules and sequencing by name, as well as the correct sequencing of the multiple metadata extraction. The path, a metadatum, is lower-case and thus better for CSV mapping.

An OR (|) operator of sub-rules does not work!

> Simple solution without exception:

Pattern rules:

LWPath=/\\\\[^\\]+\\[^\\]+\\[^\\]+\\[^\\]+/.

FilePath=/[^\\]+/.

FullPath=LWPath "\\" FilePath.

> Solution with an exception (data\it):

Pattern rules:

ASpecialPath="data\\it".

OtherPath=/[^\\]+/.

BaseShare=/\\\\[^\\]+\\[^\\]+\\[^\\]+/.

LWPathA= BaseShare "\\" ASpecialPath.

LWPathOther= BaseShare "\\" OtherPath.

FilePathA=/[^\\].*/.

FilePathOther=/[^\\].*/.

FullPathA=LWPathA "\\" FilePathA.

FullPathOther=LWPathOther "\\" FilePathOther.

The following screenshot demonstrates the configuration of the rules.

CSV-transform: the extracted value (file share) is case sensitive so the cases must match − that way the path can be used as source metadatum.

fileshare;letter

\\fileserver.mycompany.com\qa\fstest\projekte;U:

\\fileserver.mycompany.com\qa\fstest\vorlagen;T:

\\fileserver.mycompany.com\qa\fstest\allgemein;G:

\\fileserver.mycompany.com\qa\fstest\spezial;M:

\\fileserver.mycompany.com\qa\fstest\data\it;H:

\\fileserver.mycompany.com\qa\fstest\data;H:

\\fileserver.mycompany.com\qa\fstest\data-services;H:

\\fileserver.mycompany.com\qa\fstest\allgemein-retail;G:

Matching with mes:key is only possible in CSV transformation (as well as in ER rules) with: In Property = datasource/mes:key.

Please note: /documents-Servlet does not provide values that only arise via index re-invert!

Troubleshooting entity recognition

This chapter deals with troubleshooting the entity recognition rules.

Important information

In Mindbreeze InSpire, regular expressions are surrounded by a “/”.
Each rule entry must be separated by a period.
Rule names may not contain “_”
Rules are “greedy”, meaning they match as much as possible be careful with “.*” or “.+” configurations).
Rules are processed alphabetically (case-sensitive!). First in line are uppercase letters from A to Z, then lowercase letters from a to z.
If a rule matches an entity, no second rule can match. Assumption: If the words “managing board” are used both for the committee and in the keyword, only the metadatum with the rule "committee" will include the words “managing board”.
Entity recognition rules can only be created per index, that is, across all data sources within the index.

Index

Check the index status at http://localhost:8443/index/<Indexport>/statistics

Privileged servlets:

Connect to the Management Center
Navigate to the index
Activate the advanced settings
Open the index for which you want to test entity recognition
Deactivate the “Disable Unrestricted Privileged Servlets” checkbox
Then save the settings and restart the services
After the services are restarted:
- Open https://yourappliance:8443/index/Indexport (in our example: 23101/processitems) https://yourappliance:8443/index/23101/processitems
- On this page, you can test the rules (pattern rules) with a specific query (e.g. ALL)
- After filling in, click on process. If the syntax of the rules is correct, you’ll have more options to test after pressing the button.
- Select the rule that you want to match and configure the values of the rule(s).
- Then click on process to start testing the rule(s):

Deactivating the greedy strategy of the entity recognition rules

Entity Recognition rules are usually greedy In the following example, the selected rows are matched:

Rule

R1=/ (?s)(test)(?P<line>.+)\s+(.*Page) /.

Match:

If greedy is deactivated, however, not everything is matched, but instead, only those blocks that start with test and end with Page:

Rule:

(?U)(?s)(test)(?P<line>.+)\s+(.*Page)(?U)

Match:

Common error sources

An error with the following error message occurred while parsing the ER rules:

“MesQuery::Text::RE2Tokenizer ERROR: Matched empty (epsilon) token, pattern is”

… for instance, a „\“ at the end of a regex is not supported (LWPath=/\\\\[^\\]+\\/. … an error occurs better: LWPath=/\\\\[^\\]+/ “\\“.).

There can also be possible problems with “.*“ in rules.

Entity recognition rules are analyzed in alphabetical order and the first complete match wins.

Regex rules for German words do not match all characters (umlauts, etc.) with \w. Instead, you can use \pL to match all unicode characters.

No Entity Recognition on Content

If Entity Recognition should be applied on Content, either by setting Name to “Content” or “.*”, then Content has to be manually added as aggregatable. This can be done by one of the two methods:

Add “@content” to Aggregated Metadata Keys
Or add a Precomputed Synthesized Metadata with Name “@content”, Property Expression “""”, Transformation Pipeline Slot “Before Entity Recognition”, Merge Strategy “Replace Existing”, Aggregatable “Static Aggregatable (Included in Document Info and Schema)”.

Typical use cases

Personal information

Social security number

RegEx

\d{4}(\s|\.|\-)\d{6}

Example

1237 010180

1237.010180

1237-010180

Telephone number

RegEx

(\+)([\s.]*\d{1}){8,13}(-)?(\d{1,5})

Example

+43 732 606162-0

+43 732 606162-609

+49(732)606162-609

Number (with delimiters)

RegEx

z1=/\d/.z2=/\d/. (…)Dlmtr=/[\s\-_.:]?/.

z1 Dlmtr z2 Dlmtr z3 Dlmtr z4 Dlmtr z5 Dlmtr z6.

Example

12-34567

12 34 56-7

1-2 3456.7

Amount

RegEx

((\d{1,3}(\.(\d){3})*)|\d*)(,\d{1,2})

Example

0.84

100,000.49

100,000.00

1,000,000,000,000.00

Date

Handbook for date formats: http://userguide.icu-project.org/formatparse/datetime

dd(.|-|/)MM(.|-|/)yyyy
- RegEx
  ((0[1-9])|[1-9]|([1-3][0-9]))(\.|\/|-)((0[1-9])|[1-9]|10|11|12)(\.|\/|-)((19|20)\d{2})
- Example
  11.03.2014
  11.3.2014
  3.3.2014
  03.2.2010
  11/03/2014
  11/3/2014
  3/3/2014
  03/2/2010
  11-03-2014
  11-3-2014
  3-3-2014
  03-2-2010
dd. MMM yyyy
- RegEx
  ((0[1-9])|[1-9]|([1-3][0-9]))\..(|January|February|March|April|May|June|July|August|September|October|November|December).((19|20)\d{2})
- Example
  3 January 2014
  4 February 2012
  30 November 2013
MMM yyyy
- RegEx
  (January|February|March|April|May|June|July|August|September|October|November|December).((19|20)\d{2})
- Example
  February 2014
  September 2014
MM(.|-|/)yyyy
- RegEx
  (January|February|March|April|May|June|July|August|September|October|November|December).((19|20)\d{2})|((0[1-9])|[1-9]|10|11|12)(\.|\/|-)((19|20)\d{2})
- Example
  03-2014
  03.2014
  03/2014
yyyy(.|-|/)mm(.|-|/)dd
- RegEx
  ((19|20)\d{2})(\.|\/|-)((0[1-9])|[1-9]|10|11|12)(\.|\/|-)((([1-3][0-9]|0[1-9])|[1-9]))
- Example
  2014-03-21

Date-Regex total
((0[1-9])|[1-9]|([1-3][0-9]))(\.|\/|-)((0[1-9])|[1-9]|10|11|12)(\.|\/|-)((19|20)\d{2})|((0[1-9])|[1-9]|([1-3][0-9]))\..(January|February|March|April|May|June|July|August|September|October|November|December).((19|20)\d{2})|(January|February|March|April|May|June|July|August|September|October|November|December).((19|20)\d{2})|((0[1-9])|[1-9]|10|11|12)(\.|\/|-)((19|20)\d{2})|((19|20)\d{2})(\.|\/|-)((0[1-9])|[1-9]|10|11|12)(\.|\/|-)((([1-3][0-9]|0[1-9])|[1-9]))
Date-Regex total II
((((0?[1-9]|[12]\d|3[01])[\.\-\/](0?[13578]|1[02])[\.\-\/]((1[6-9]|[2-9]\d)?\d{2}))|((0?[1-9]|[12]\d|30)[\.\-\/](0?[13456789]|1[012])[\.\-\/]((1[6-9]|[2-9]\d)?\d{2}))|((0?[1-9]|1\d|2[0-8])[\.\-\/]0?2[\.\-\/]((1[6-9]|[2-9]\d)?\d{2}))|(29[\.\-\/]0?2[\.\-\/]((1[6-9]|[2-9]\d)?(0[48]|[2468][048]|[13579][26])|((16|[2468][048]|[3579][26])00)|00)))|(((0[1-9]|[12]\d|3[01])(0[13578]|1[02])((1[6-9]|[2-9]\d)?\d{2}))|((0[1-9]|[12]\d|30)(0[13456789]|1[012])((1[6-9]|[2-9]\d)?\d{2}))|((0[1-9]|1\d|2[0-8])02((1[6-9]|[2-9]\d)?\d{2}))|(2902((1[6-9]|[2-9]\d)?(0[48]|[2468][048]|[13579][26])|((16|[2468][048]|[3579][26])00)|00))))
Example
31.12.2005
12.12.12
1.2.2003
1.3.98
04-05-2004

Time

RegEx

(([0-1]?[0-9])|([2][0-3])):([0-5]?[0-9])(:([0-5]?[0-9]))?

Example

11:00:23

12:30

E-mail

RegEx

([a-zA-Z0-9_\-\.]+)@([a-zA-Z0-9_\-\.]+)\.([a-zA-Z]{2,5})

Example

david.porter@inspire.mindbreeze.com

egov@mindbreeze.com

IBAN

RegEx

AT\d{18}

Example

AT002105017000123456

Split List by “,” or other symbols

In this example a list of entries separated by semicolon will be interpreted as well as List in Mindbreeze InSpire.

Input: List of word, word,…

value=/[^\s,][^,]*[^,\s]?/.

rule=/\s*/value/\s*(,\s*|$)/.

Entity Recognition

Configuration

Introduction

Entity recognition configuration

Entity Recognition Parameter

Entity recognition (example: file system)

Configuration of entity recognition for a file system:

Configuration for entity recognition for file system paths (variant 2) – with exceptions:

Troubleshooting entity recognition

Important information

Index

Privileged servlets:

Deactivating the greedy strategy of the entity recognition rules

Common error sources

No Entity Recognition on Content

Typical use cases

Personal information

Social security number

Telephone number

Number (with delimiters)

Amount

Date

Time

E-mail

IBAN

Split List by “,” or other symbols

Download PDF

Download PDF

{{{i18n.refineSearch}}}

Entity Recognition

Configuration

Introduction

Entity recognition configuration

Entity Recognition Parameter

Entity recognition (example: file system)

Configuration of entity recognition for a file system:

Configuration for entity recognition for file system paths (variant 2) – with exceptions:

Troubleshooting entity recognition

Important information

Index

Privileged servlets:

Deactivating the greedy strategy of the entity recognition rules

Common error sources

No Entity Recognition on Content

Typical use cases

Personal information

Social security number

Telephone number

Number (with delimiters)

Amount

Date

Time

E-mail

IBAN

Split List by “,” or other symbols

Download PDF

Download PDF