Whitepaper
Text Classification Insight Service

Motivation and Overview

Text classification with Mindbreeze InSpire has never been easier. Tag a portion of your documents with predefined labels. With the help of Mindbreeze Insight Services and Machine Learning, Mindbreeze InSpire is able to expand your knowledge and store it for future use cases. Based on this knowledge, all other documents can subsequently be classified fully automatically.

Labeling can be done easily and directly via the Insight app - even without a predefined data set.

The main steps to perform this use case are:

1. Preparing the training dataset
   1. Define the possible labels for the documents.
   2. Manual labeling via the Insight App for documents and creation of the dataset to train the classification model.
2. Training a classification model
3. Labeling of documents from an index using the classification model in Semantic Pipeline (Item Transformation).

Preparation

Overview of the required Services

In order to use text classification, certain configuration steps are necessary. Configure the following services:

• Prediction Service
• Text Classification Insight Service

In addition, you still need to make configuration adjustments in the Client Service and Index Services.

Details can be found in the next sections.

Configuring the Prediction Service

In Mindbreeze Management Center, navigate to the "Configuration" menu and switch to the "Indices" tab, then add a new service.

For the additional minimal configuration, fill in the following fields in the following configuration sections:

If you have more specific use cases, see the Detailed Configuration of Prediction Service section for more information.

Configuring the Text Classification Insight Service

Now add the "Text Classification Insight Service". Assign a "Display Name" again and select the "TextClassificationInsightService" under "Service".

For a minimal configuration, fill in the following fields in the following configuration sections:

• Prediction Service
  • URL: the URL of the prediction service. E.g. http://localhost:23910 if you have selected 23910 as "Bind Port" for the Prediction Service.
  • Project ID: the name of the classification project
  • Tenant ID: e.g. the company name or the organizational unit
• Dataset Index Ports: The ports of the indexes in which the documents to be classified are located.
• Persisted Resources Feedback Processing
  • JDBC URL, Database Credentials, Database Table Prefix: configure the same values as in "Resource Persistence Settings" in the Client Service.

If you have more specific use cases, see the Detailed Configuration of Text Classification Insight Service section for more information.

Other required Configuration Changes

In addition to the Prediction Service and the Text Classification Insight Service, you still need changes in the configuration of the ClientService and the Index Services.

Client Service

To enable users to label documents in the standard Insight app, you still need to make configuration changes in the client service.

Activate the Advanced Settings and configure the Resource Persistence Settings. Then enable Document Labeling by enabling the following option:

You only need to change the other options in the "Document Labeling" configuration section if you have changed certain default values in the Text Classification Insight service:

Index Service(s)

Documents are classified as they pass through the Semantic Pipeline - more specifically, in the "Item Transformation" step.

Add the previously created "Text Classification Insight Service" to the index at the "Item Transformation Services". If you are using multiple indexes, repeat this step on all index services.

Definition of Labels and Manual Labeling

When the configuration is complete, you can define labels. These labels can be used by users to identify documents in the Insight app.

Defining the Labels

In the Mindbreeze Management Center, navigate to the "Insight Services" "Text Classification" menu. Then click on "Edit" at "Label Definitions".

Now define your labels according to which you want to classify your documents.

Define translations for the languages you want to support in your Insight app. If there is no translation for languages, the ID will be used for display in the Insight app. With the "Save" button you can confirm your entries.

More details:

• If "Ignored" is checked, documents with this label will be ignored when training the model.
• You can delete labels with the trash icon.
• Please note that you do not change the ID of labels if they are already assigned to certain documents. (The assigned label IDs on the document remain unchanged, which leads to negative effects).

Manual Labeling

Now users have the option to label documents with the labels they have just defined. After searching in the Insight app, the found documents can be labeled by selecting the desired label from the drop-down menu.

Logged in users can read and assign labels. Anonymous users who are not logged in to the Insight app can read (automatically assigned) labels. (Manually assigned labels are not visible to anonymous users).

If multiple users assign labels for the same document, all assignments are saved, but effectively only the label of the last assignment is used.

Users also have the option to remove their own feedback again (trash icon). If the document was previously labeled by another user, the previous label is now effective.

Creating / Updating the Training Dataset

If the required documents have now been marked with labels, you can create the training data set that will later be used as the data basis for creating the model. To do this, navigate to the "Insight Services" "Text Classification" menu in the Mindbreeze Management Center. Then click on "Edit" at "Labeled Data".

You can now check whether the users have manually labeled the documents correctly. If labels were assigned incorrectly, these assignments can be changed here or even ignored. Then click on "Create or Update Dataset" to save your changes and create the training dataset.

Preparing Models for Text Classification

In the next steps, a model can now be created and tested from the training data set.

Train the model

In the Mindbreeze Management Center, navigate to the "Insight Services" "Text Classification" menu. Then click on "Train" under "Models".

Now click on "Train Model" to train a model. The default parameters are sufficient for most use cases. However, you can also fine-tune them if your use case requires it. The following parameters can be adjusted:

Test and use the model

In the next step, you can now test the model to get information about the quality of the model you just trained. Scroll to "Test Model". The model you just trained should already be selected. If you now click on "Test Model", the model will be tested with the test data and you will receive key figures that give you information about the quality of the model, such as "Accuracy".

Then click on "Set Default" so that this model is used for classification.

Automated Document Labeling

As already mentioned, the documents are automatically classified when they pass through the SemanticPipeline - more precisely in the Item Transformation step. Unless explicitly configured otherwise in the service configuration, the default model that you set in the previous step with "Set Default" is used for classification.

Since the Semantic Pipeline is only run through completely for new or changed documents, only new or changed documents are classified. However, to ensure that documents that have already been indexed are also classified, you have two options, which are described in more detail in the next sections:

1. Reindex: recommended for small indexes where complete indexing is very fast (e.g. on test systems)
2. Reinvert: recommended for large indexes, where a complete indexing takes a long time

Reindex

If the index is small and a full indexing can be performed very quickly, a re-indexing is recommended to trigger a classification of all documents. To do this, navigate to "Services" in the Mindbreeze Management Center. Then click on the gear icon for the index you want to re-index and then click on "Reindex". As soon as the re-indexing is successfully completed, your documents are classified.

Reinvert

If the index is large and a complete indexing takes a long time, a re-inversion is recommended to trigger a classification of all documents. To do this, navigate to "Configuration" in the Management Center and switch to the "Indices" tab. Activate the "Advanced Settings" and change the "Aggregated Metadata Keys". Changing this option will automatically re-invert the index. For example, you can specify "label" which will result in filtering by label in the Insight app. However, you can also specify a non-existent metadatum key, such as "V1". Save the configuration afterwards.

Once the re-inversion is successfully completed, your documents are classified.

Appendix

Iterative Improvement of the Model

Once your documents are classified, users can also provide feedback widely in the Insight app and change the labeling of the documents if, for example, the automatic classification was inaccurate and in some cases incorrect (see also ManualLabeling).

This feedback can then be used to update the training dataset (see Create / Update Training Dataset).

Afterwards, a new model can be trained (see Train Model), tested and used (see Test and Use Model).

If a document now changes or a new document is indexed, the new, just trained model is already used for the classification. If you want to classify all documents, including the already indexed documents, with the new, improved model, you must trigger a reindex or reinvert.

You can perform these steps to iteratively improve the model as many times as you like until you are satisfied with the quality of your classification model.

Detailed Configuration of the Text Classification Insight Service (Advanced Use Cases)

This section describes all the options available in the Text Classification Insight service. This section is relevant to you only if you have special use cases that require special configuration.

Base Configuration

Prediction Service

For more details on the Prediction Service, see the Mindbreeze Prediction Service documentation with example text classification.

Text Classification Sources

Rule Based Labels (optional)

Here you can define rules to label certain documents without calling the Prediction Service. For example, you can use it to classify all documents as "Documentation" that contain "Doc" or "Documentation" in the title.

The first rule that matches a document is always applied. If no rule matches, then the prediction service is used to set the label.

Dataset Index Ports

Persisted Resources Feedback Processing

Configure the same values for the following options as for "Resource Persistence Settings" in the Client Service: "JDBC URL", "Database Credentials", "Database Table Prefix".

See also Resource Persistence Settings.

CSV Feedback Processing (optional)

In addition to user feedback (via the Insight app), a CSV file can be used to set labels for documents. These labels are not displayed in the Insight app, but can be used to train the classification model.

Example:

Fqcategory;Key;LabelValue;IgnoreFeedback

Web:helpmindbreeze;http://help.mindbreeze.com/de/index.php?topic=doc/Konfiguration---Microsoft-File-Connector/index.htm;performancetest;false

Web:helpmindbreeze;http://help.mindbreeze.com/de/index.php?topic=doc/Installation--Konfiguration---Caching-Principal-Resolution-Service/index.htm;performancetest;false

Detailed Configuration of the Prediction Service (Advanced Use Cases)

This section describes all other special options that are available in the Prediction Service besides the mandatory fields. This section is only relevant for you if you have special use cases that require special configuration. Also, in this section, those options that are not marked by "(Mandatory)" or "(Advanced)" are automatically considered as Advanced.

Prediction Service Parameter

Dataset Settings