Data Integration Connector
Installation and Configuration
Copyright ©
Mindbreeze GmbH, A-4020 Linz, 2021.
All rights reserved. All hardware and software names used are brand names and/or trademarks of their respective manufacturers.
These documents are strictly confidential. The submission and presentation of these documents does not confer any rights to our software, our services and service outcomes, or any other protected rights. The dissemination, publication, or reproduction hereof is prohibited.
For ease of readability, gender differentiation has been waived. Corresponding terms and definitions apply within the meaning and intent of the equal treatment principle for both sexes.
Creating a „Data Integration Process“
The Data Integration Connector can be used together with Talend Open Studio to connect your own data sources. Talend Open Studio is available for download under https://www.talend.com/products/data-integration/data-integration-open-studio/.
Older versions can be downloaded from https://www.talend.com/products/data-integration-manuals-release-notes/. Please note that you do not use Milestone versions (with M1, M2, etc. at the end of the version number). These are beta versions which are often unstable.
Note: The Mindbreeze Java SDK supports Java version 8. It is possible that Talend Open Studio automatically installs a different Java version on your system, which can then lead to problems with the Mindbreeze Java SDK later on. Make sure that Java version 8 is installed and that the JAVA_HOME environment variable is set to the Java 8 JDK installation directory.
The Data Integration Connector contains components for Talend Open Studio, which will need to be installed separately. Unpack the file components.zip from the Data Integration Connector installation package into any folder (e.g. C:\custom-talend-components).
Create a new project after installing Talend Open Studio.
Open Window -> Preferences in the Talend Open Studio menu
Select Talend -> Components and enter the name of the folder into which you unpacked the components in the field “User component folder”.
Select Import/Export Settings an activate the Option „Add classpath.jar in exported jobs“
You can now create a new job. Add data sources according to your requirements. More information about working with Talend Open Studio is available in the Talend Open Studio documentation.
The target of this kind of processing chain must always be the component named "MindbreezeIndexOutput".
In recent versions of Talend Open Studio depedencies may no be detectes automatically:
In this case the JAR file dependencies have to be resolved manually:
- Click button “Install…”
- Click on each
-Symbol - Assign JARs with matching name in the folder MindbreezeIndexOutput
Furthermore, please note that for the component to function correctly, the following fields (string type) must be defined in the data set schema:
- key
- title
- extension
- categoryClass
The following fields are optional and can be used additionally for further processing in the Mindbreeze Index:
- acl (list of string values) in the format: "TestUser1||GRANT"
- date (type "Date")
- modificationDate (type "Date")
- content (type "String")
Should further fields be defined in the schema, these are imported as metadata. It is also possible to define annotations in the following format:
val1|||mes:annotated|||categoryclass=cc1|||value=v1
In this example, "val1" becomes an annotation with the categoryClass "cc1" and the value "v1".
All "list" type fields become lists of metadata; all other fields are automatically converted into "string" types.
Testing and exporting the “Data Integration Job”
Test with logging
When your job configuration is complete, you can run it to test its functionality. The data are not sent to an index but exported to Talend Open Studio.
Test as pusher
You can test the Job by sending data to the Mindbreeze InSpire Appliance without exporting it first. The corresponding Index has to be created first:
Select the „MindbreezeIndexOutput“ component and select „Use as Pusher“.
Configure the following settings:
Category | The category |
Category Instance | The category instance |
InSpire Base URL | URL pointing to your Mindbreeze InSpire appliance |
Filter Pipeline ID | Port of the Filter-Service (23400 by default) |
Index ID | Port of the Index-Service |
Node ID | Node ID of the appliance |
Inspire Generation |
|
Username | For G6 use „inspireapi“. For G7-Appliance any backend user with the Rolsse „InSpire Index Writer“ can be used.Further configuration: https://help.mindbreeze.com/de/index.php?topic=doc/Konfiguration---Backend-Credentials/index.htm. |
Password | the password of the user, WITHOUT quotation marks Tip: If your password contains characters that must not occur in a Java String literal, they must be escaped: https://docs.oracle.com/javase/tutorial/java/data/characters.html |
Additional Settings for G7::
Client ID | Client ID of the OAuth2 Client in Keycloak (e.g mindbreeze-inspire-public) |
Client Secret | Not neccesary for „mindbreeze-inspire-public“ |
Use external Auth URL | Select this option if you use an externam Keycloak installation |
External Auth URL | the URL of the external Keycloak installation: {base-url}/realms/{realm-name}/protocol/openid-connect/token |
Export
If the functionality test runs smoothly, the job still needs to be exported. That can be done by clicking in the context menu of the job:
It is important that the generated ZIP-file is also unpacked.
The "Main-Class" required for the configuration of Fabasoft Mindbreeze Enterprise can be found in the generated batch file.
Configuration of Mindbreeze
Select the “Advanced” installation method:
Click on the “Indices” tab and then on the “Add new index” symbol to create a new index.
Enter the index path, e.g. “C:\Index”. Adapt the Display Name of the Index Service and the related Filter Service if necessary
Add a new data source with the symbol “Add new custom source” at the bottom right.
To configure the crawler, you need to enter the Job ZIP archive or the extracted job directory in "Path to Job" and the Java job class in "Main Class". Please note that Talend Job Zip archives are only supported starting with Mindbreeze InSpire G7.
If the option “Delete Unprocessed Documents” is enabled, then all unprocessed documents in the index are delete if the crawlrun was successful (exit code of the Talend-Job is 0).