Whitepaper
Preparation for Using InSpire Snapshots in a CI/CD Environment

Introduction

Mindbreeze InSpire Snapshots allows you to make changes to your appliance without having to make these changes directly on the production system:

1. The implementation of the changes takes place exclusively on the development system
2. Once a development snapshot has been created, it can be automatically applied to the test environment and tested using your CI/CD pipeline
3. After successful testing, the snapshot is deployed on the production system.

You have the following advantages:

• No more manual intervention on the test and production system
• Easy integration into an external CI/CD pipeline using the Mindbreeze InSpire SnapshotService API
• Changes can be stored in a repository as a file (snapshot archive) and can be versioned. These changes are therefore
  • reproducible
  • comprehensible
  • audit-proof

In this documentation you will learn about the snapshot in general and about which preconditions must be met to use the API. Afterwards our Java SDK example is briefly introduced. Then you will learn about the difference of “Migration Snapshots” and “Development Snapshots” and how you use them corre

Scope and Limitations

Authentication

SAML authentication is currently supported. For more details see: Client Service and SAML Authentication Settings

Clustered Environments: Master + Standby Appliance

Imported snapshots on Master Appliance are synchronized with the Standby Appliance according to the synchronization schedule. The External URL of the Client Service should point to the Standby Appliance. For more details see: Client Service and SAML Authentication Settings and Changing the Client Service and SAML Authentication Settings.

General information, Requirements and Prerequisites

Included Data in Snapshots

Custom Connector Plugin

Added or updated Custom Connector Plugins on Development Appliance are included in exported migration snapshots and development snapshots and can be imported on Development Appliance. For more details see: Add “Upload” Plugin

Insight App

Added or updated Insight Apps on Development Appliance are included in exported migration snapshots and development snapshots and can be imported on Development Appliance. For more details see: Adding a custom 3.1.2  Insight App

Credentials and Endpoints

Credentials and Endpoints on Development Appliance are included only in exported migration snapshots and can be imported on Development Appliance. Development snapshots do not include Credentials and Endpoints.

InSpire Admin API OAuth Authentication

Accessing the Mindbreeze InSpire admin API on /api/v3/admin and the Mindbreeze InSpire Blob Store for downloading exported snapshot archives (/blobs/v3/snapshotarchives) requires OAuth bearer token authentication using a service user with role InSpire Administrator.  

The OAuth token can be obtained from the token endpoint /auth/realms/master/protocol/openid-connect/token using the provided OAuth client ID and secret together with the service user credentials.

Obtaining Client ID/ Client Secret

We are recommending to create a dedicated OAuth client for the service. This can be configured using the Mindbreeze InSpire Management Center. Note: the existing mindbreeze-inspire clients should not be used, because they are overwritten on every update.

Navigate to Setup/Credentials and select the “Clients” section. Add a new OAuth client with the following parameters:

• Client Protocol: openid-connect
• Access Type: confidential
• Valid Redirect URIs: /openidc/*

After saving the client add the following Mapper configuration for supplying the user roles in the authentication token:

• Name: role
• Mapper Type: User Realm Role
• Token Claim Name: roles

Required Service User Roles

The service user for snapshot operations requires InSpire Administrator role. You can create a dedicated service user in the “Users” section of the Credentials configuration in the Mindbreeze InSpire Management Center:

In the “Role Mappings” section of the user configuration, make sure the “InSpire Administrator” role is mapped to the created service user.

Using the Java Example

The Java SDK Sample project “SnapshotClient” provides a simple tool for the following use case:

• Create a snapshot on a source Mindbreeze InSpire Appliance
• Export the created snapshot as a snapshot archive
• Download the snapshot archive
• Upload the snapshot archive on a target appliance
• Import and apply the uploaded archive

The latest SDK can be downloaded from www.mindbreeze.com/developer#quick-start. This contains among other things also the Java SDK sample project "SnapshotClient".

Project Structure

The sample SDK project has the following structure:

src

Folder containing the Java Sources of a command line tool for performing the required API calls.

• InSpireHTTPChannel.java: network communication channel implementation;
• Main.java: Main class (Command Parsing and executing the commands);
• OAuthManager.java: OAuth Login Manager;
• SnapshotClient.java: API call wrapper;

lib

Folder for the referenced libraries:

• messdk-generated.jar: a thin wrapper that uses generated classes for interoperating with the Snapshot API.
• protobuf-java-3.0.jar: Google Protobuf library
• protobuf-java-util-3.12.3.jar: Google Protobuf library
• json-20200518.jar: JSON parsing tools for OAuth authentication (https://mvnrepository.com/artifact/org.json/json)

Building the project

Building the project requires Java JDK 8+ and Apache ANT 1.9.7+.

The sample project can be build using Apache Ant with the provided build.xml. In the project folder (where the build.xml is located), execute the following command:

ant install

This builds the project and packages the JAR and the referenced JARs into a self-contained ZIP file located in the install folder.

This ZIP file can be extracted, then the tool can be executed and by default displays the usage information:

java -jar snapshot-client.jar

Initial Deployment (Migration Snapshot)

Goal

For setting up the infrastructure (e.g. for web server certificates, credentials, frontend authentication) a migration snapshot created on the development appliance is required. With this documentation, we provide examples using the Confluence Connector (Crawler and Principal Cache) and a Client Service with SAML.

Setup of the Production Environment

On the production appliance make sure the Mindbreeze InSpire license is installed.

Configuration of Services for the Development Environment

Configuration of the Confluence Data Source

Verify that the following properties are correctly configured according to development environment:

• Crawling Root
• Login URL
• Form Elements

Remember that these are the connector properties which should be changed according to the production environment before creating an initial snapshot for the production environment. (See Adapting the Development Appliance Version for Production).

Configuration of the Confluence Principal Resolution Cache

Make sure that the “Confluence Server URL” is pointing to the confluence server of the development environment and is configured as an “Endpoint” in the “Network” tab. The selected credential should be from the development environment. We will change these cache properties before we create a snapshot for the production environment. (See Adapting the Development Appliance Version for Production).

Client Service and SAML Authentication Settings

To create an authenticator: upload the SAML IdP metadata file in the “Authentication” tab from the development environment together with the already uploaded SSL certificate in the “Certificates” tab.

After uploading select “Use SAML Authentication” for the configured Client Service.

As described in the section Adapting the Development Appliance Version for Production, we will remove this authenticator and the SSL certificate and upload the IdP metadata file and the SSL certificate from the production environment later.

In the “Client Services” tab configure the “External URL” of the development environment, which will also be changed later.

Now a SAML entity descriptor can be downloaded from https://inspire-dev.myorganization.com:8443/saml20/sp/entitiesdescriptor and used for configuring the relying party on the SAML IdP server (for more details about SAML authentication, please refer to the following documentation: https://help.mindbreeze.com/en/index.php?topic=doc/SAML-based-Authentication/index.htm).

The above step will be repeated after changing the SSL certificate, IdP metadata and the External URL to create the SAML entity descriptor for the IdP server of the production environment.

Testing the Development Environment Services

After configuring the development environment, please make sure that the files from the Confluence server of the development environment are indexed and can be searched.

Creating the “Migration” Snapshot of the Development Appliance

Before we change the configuration to create a migration snapshot for the production appliance, we should create a migration snapshot of the working configuration for the development appliance as a backup with the provided snapshot tool. The development appliance should be restored by this snapshot later.

java -jar snapshot-client.jar \

         --inspire-url="https://inspire-dev.myorganization.com:8443" \

         --client-id="" \

         --client-secret="" \

         --service-username="" \

         --service-user-password="" \

         --request-type=EXPORT \

         --json-request-file=create.json \

         --output-file=inspire-dev-appliance.snapshot

The file create.json should have a full migration profile (APPLIANCE_MIGRATION)

{

  "name": "Dev Appliance Snapshot",

  "description": "Description",

  "options": {

    "profile": "APPLIANCE_MIGRATION"

  }

}

Adapting the Development Appliance Version for Production

Changing the Configuration of the Confluence Data Source

The following properties should be changed according to the production environment:

• Crawling Root
• Login URL
• Form Elements

Changing the Configuration of the Confluence Principal Resolution Cache

The field “Confluence Server URL” is pointing now to the confluence server of the production environment and it is configured as an “Endpoint” in the “Network” tab. The endpoint and the credential from the development environment should be replaced by the endpoint and credential from the production environment.

Changing the Client Service and SAML Authentication Settings

Under “Authentication” uncheck “Use SAML Authentication” and remove the SAML Authenticator. In the “Certificates” tab remove the SSL certificate.  

In the “Certificates” tab upload the SSL certificate of the production environment; then in the “Authentication” tab, upload the SAML IdP metadata file of the production environment together with this SSL certificate.

After uploading, select “Use SAML Authentication” with the new authenticator for the Client Service.

In the “Client Services” tab, configure the “External URL” of the production environment (on clustered environment it should be the Standby Appliance URL).

The SAML entity descriptor for the production environment can be downloaded from (https://inspire-dev.myorganization.com:8443/saml20/sp/entitiesdescriptor)  and used for configuring the relying party of the SAML IdP server of the production environment (for more details about SAML Authentication refer to:  https://help.mindbreeze.com/en/index.php?topic=doc/SAML-based-Authentication/index.htm.

Creating the “Migration” Snapshot for the Production Appliance

With the provided snapshot tool, we take a snapshot of the development appliance for the production appliance.

java -jar snapshot-client.jar \

         --inspire-url="https://inspire-dev.myorganization.com:8443" \

         --client-id="" \

         --client-secret="" \

         --service-username="" \

         --service-user-password="" \

         --request-type=EXPORT \

         --json-request-file=create.json \

         --output-file=inspire-prod-appliance.snapshot

The file create.json should have the full migration profile (APPLIANCE_MIGRATION)

{

  "name": "Prod Appliance Snapshot",

  "description": "Description",

  "options": {

    "profile": "APPLIANCE_MIGRATION"

  }

}

Restoring the Development Appliance

After taking the migration snapshot for the production appliance, we restore the development appliance by importing the migration snapshot taken before the changes were done.

java -jar snapshot-client.jar \

         --inspire-url="https://inspire-dev.myorganization.com:8443" \

         --client-id="" \

         --client-secret="" \

         --service-username="" \

         --service-user-password="" \

         --request-type=IMPORT \

         --input-file=inspire-dev-appliance.snapshot

Importing the Snapshot in the Production Appliance

Applying the migration snapshot on the production appliance should be performed by calling the snapshot tool as follows:

java -jar snapshot-client.jar \

         --inspire-url="https://inspire-prod.myorganization.com:8443" \

         --client-id="" \

         --client-secret="" \

         --service-username="" \

         --service-user-password="" \

         --request-type=IMPORT \

         --input-file=inspire-prod-appliance.snapshot

Optionally using the disable-sync-tasks parameter the configuration and index synchronization tasks on the target system can be disabled before importing the snapshot. These can be then re-enabled manually after successful tests.

         --disable-sync-tasks

Parameterizing the Configuration for the Development Snapshot

Before we apply the development snapshots on the production appliance, it is necessary to make sure that the configuration settings that are related to the production environment should not be overwritten by the development snapshots. This means that both appliances should have the same set of environment parameters with different values. Newly introduced environment parameters in the development appliance should be explicitly defined during the import of development snapshots on the production appliance.

For example, the following options should be configured as environment parameters:

• Confluence Data Source
  • Crawling Root
  • Login URL
  • Form Elements

• Confluence Principal Resolution Cache
  • Credentials Endpoint

• Client Service
  • External URL

To use the environment parameters, the feature has to be activated. You can find the documentation here: Documentation Mindbreeze InSpire - Enable/Disable Parameterization.

Parameterizing the crawling root can be done as follows:

The Login URL, Form Elements (Confluence Datasource), Credential Endpoint (Network tab) and External URL (Client Service) should be parameterized using the above parameterization dialog. After configuring the environment parameters, a migration snapshot from the development appliance should be exported as follows.

java -jar snapshot-client.jar \

         --inspire-url="https://inspire-dev.myorganization.com:8443" \

         --client-id="" \

         --client-secret="" \

         --service-username="" \

         --service-user-password="" \

         --request-type=EXPORT \

         --json-request-file=create.json \

         --output-file=inspire-prod-appliance.snapshot

The file create.json should have the migration profile (APPLIANCE_MIGRATION)

{

  "name": "Prod Appliance Snapshot",

  "description": "Description",

  "options": {

    "profile": "APPLIANCE_MIGRATION"

  }

}

The created migration snapshot should be imported on the production appliance. When importing, the snapshot parameters should be provided to the snapshot tool in a separate import.json file.

java -jar snapshot-client.jar \

         --inspire-url="https://inspire-prod.myorganization.com:8443" \

         --client-id="" \

         --client-secret="" \

         --service-username="" \

         --service-user-password="" \

         --request-type=IMPORT \

         --json-request-file=import.json \

         --output-file=inspire-prod-appliance.snapshot

The file import.json should contain a value for each parameter (for example: confluence_crawling_root) according to the production environment.

{

   "importOptions": {

     "configurationParameters": {

        "confluence_crawling_root": "<prod_url>",

        "<other param name>": "<other param value>"     

      }

   }

}

After initial deployment, newly added parameters can be added only through the import.json to the production appliance. See: Adding Features to Production (Development Snapshot)

Adding credentials, endpoints and certificates on the production appliance after the initial deployment should be performed through the following steps:

1. A migration snapshot of the development appliance should be created to back up the development appliance settings.
2. On the production appliance a migration snapshot should be created and imported on development appliance
3. Configuration changes on credentials, endpoints and certificates should be done and tested on development appliance.
4. A migration snapshot should be created on the development appliance.
5. The created migration snapshot should be imported back on the production appliance.
6. The development appliance should be restored from the initial backup snapshot.

Adding Features to Production (Development Snapshot)

Adding a metadata, making it aggregated, and adding a filter to the Insight App

In the development appliance, configure the following:

• In the Index Service, configure the Precomputed Synthesized Metadata as follows:
  

• In the Atlassian Confluence connector, Configure the Extract Metadata as shown below:
  

• Make it aggregated
  

• Make sure to disable (un-check) the following crawler configuration option:
  

• In the client service, you should now see the following filters:
  

• Add the filteredfacet to the Insight App index.html:
  

• In the client service, you should see the following filter (Breadcrumbs 2):
  

Create a new snapshot (Profile: DEVELOPMENT).

Import the development snapshot into the production appliance, and re-index the Index Service.

Then open the search client and you should see the same filter (Breadcrumb2) when searching for ALL.

Adding a custom Insight App

In the development appliance, add a new Insight App:

Then, configure it as an ‘Additional Context’ in the client service:

Create a new snapshot (Profile: DEVELOPMENT) and apply it to production. The new Insight App is applied to production as well.

Resetting Local Files

First, add a file to an existing Insight App on the development appliance:

Also, add another file to an existing Insight App on the production appliance:

Create a new snapshot (with Profile: DEVELOPMENT) on the development appliance.

Import the development snapshot. If the -reset option is NOT specified in the command, both files created in development and production should exist in the Insight App on production:

However, if the -reset option is specified in the command, then only the file created in the development appliance is applied to production.

Add “Upload” Plugin

On the development appliance, upload a custom or a built-in plugin, e.g., MyCustomPlugin-20.4.1.243.zip):

Create a new snapshot (with Profile: DEVELOPMENT) of the development appliance.

Import the development snapshot to the production appliance and you should see the same plugin listed:

Production Credentials are preserved

Applying a development snapshot that contains credential changes (e.g. username or password) to the production environment does not change the production’s credentials.

Production Certificates are preserved

Applying a development snapshot that contains certificate changes to the production environment does not change the production’s certificates.

Production resources are preserved except search query relevance boosting and search relevance options.

Development snapshot changes in resources (e.g. adding or deleting files under /data/resources) are not applied to the production resources.

However, only changes for search relevance boosting and search relevance options (data/resources/relevance) are applied to the production environment.