Handbook

Updates and Downgrades

Introduction

To ensure a pleasant and uncomplicated update of the version, every newly released update should normally be installed and skipped as little as possible, if at all.
Downgrades and mixed operation of versions should also be avoided in distributed operation.

If these points are not adhered to, errors may occur in special situations for certain versions, which you have to fix manually.

This document describes which errors can occur in these special situations and how to correct them.

Updates

Note:
A full backup must be performed before an update or downgrade.
Instructions can be found here.

If there are manually set up cron jobs (not recommended) on the host, customers should contact the Mindbreeze support before updating to 21.3 or later.

Update from Versions before 21.3 to Version 22.3 or newer

A direct update is not supported. Please update to 21.3 before updating to Version 22.3 or newer.

If you already updated a Version before 21.3 to 22.3 and the inspire container is not working, please issue the following commands:

docker exec -it inspire-system-loader inspire-systemloader update

docker exec -it inspire-system-manager inspire-systemmanager image update CoreOS

reboot

Update from Version 18.0 or older to Version 21.3 or newer

If Mindbreeze InSpire is still running on version 18.0 or older and you want to update to version 21.3 or newer, you cannot update directly to 21.3, but must first update to 18.1 and then to 21.3 or newer.

Update to Version 20.4 or newer

If you receive the following error message during the update:

• verify.sh: ERROR: keycloak update script failed

A possible cause of this error could be missing or incorrect configuration files.

Check the file /var/data/keycloak/config/keycloak_h2_db_name. With InSpire version 20.3 or older the file should have the content keycloak_4_6_0. With InSpire version 20.4 or later, the file should have the content keycloak_10_0_2.

If the file is missing or the content does not match, correct this and start the update again. The error should now be corrected.

First Update to Version 21.2 HF2 or newer

If you have an older version than 21.2 HF2 and the following two points also apply to your environment:

• Distributed Operation (Producer/Consumer)
• In the case of sequential updates of the nodes (instead of simultaneous updates), older and newer versions are operated simultaneously for a short time

Please update to version 21.3 or newer.

If notifications are also activated in at least one client service, please contact the support to ensure a smooth update.

Note: When updating to a 21.2 HF version, a sequential update is not possible. If you require this please contact the support as well.

Update from 22.2 HF2

When updating from version 22.2 HF2 to any version, the following message can be displayed in the update log:

Could not execute update with exception:  out of pty devices

In that case, please take one of the following actions to be able to start the update:

• Restart the appliance or
• Run docker restart update-center on the command line

Docker exec with 22.2 HF2

The same issue might cause the following error when trying to run docker exec:

OCI runtime exec failed: exec failed: unable to start container process: open /dev/pts/1: operation not permitted: unknown

To get around this you have the following possibilities:

• Update to at least 22.2 HF3
• docker restart CONTAINERNAME
• in case of the inspire container you can also access using ssh: ssh -T -i /var/data/default/config/sshkeys/mes/id_rsa mes@$(docker inspect inspire | jq .[].NetworkSettings.Networks.mindbreeze.IPAMConfig.IPv4Address -r)

Update with custom Keycloak system changes

Note: To avoid complications during the update, it is not recommended to make non-persisted changes (e.g.: individually added trusted SSL certificates to the JVM keystore) in the keycloak container.

In the keycloak container only the directories /config, /opt/keycloak/data (before Mindbreeze Inspire Version 23.3 /opt/jboss/keycloak/data) and /opt/keycloak/log (before 23.3 /opt/jboss/keycloak/log) are persisted, any changes in other directories and files will be lost during an update and have to be updated manually!

If custom trusted SSL certificates have been added to the JVM keystore, to enable e.g. User Federation with a system which has a self-signed certificate, an update will not be possible.

To be able to update first (before starting the update) manually disable all User Federation Providers and Identity Providers that rely on the added SSL certificates (Before disabling, make sure you also have an admin user which is stored in keycloak and not imported via user federation – if this is not the case, create one). This can be done in the MMC in the tab Setup > Credentials. To do this, simply go to the tabs "User Federation" and "Identity Providers", find all the necessary providers and deactivate them in their settings.

After that you can update. Once the update is complete, add the certificate in the Keycloak container to the JVM keystore again. To do this, the following commands must be executed inside the Keycloak container:

# <path_to_certificate> ist der Pfad zum Zertifikat – der Pfad muss also
# vom Container aus erreichbar sein

cp <path_to_certificate> /etc/pki/ca-trust/source/anchors

update-ca-trust extract

After that, the Docker container must be restarted (docker restart keycloak) to perceive the changes in the keystore. After the restart you can activate all providers again.

Downgrades

Note:
A full backup must be performed before an update or downgrade.
Instructions can be found here.

Basically, a distinction is made between two different cases of downgrades:

1. Downgrade after Update:
   In the past, an update was performed. When downgrading, it is intended to go back to the old version. This is the normal case.
2. Downgrade without previous version:
   Downgrade to an old version, that was never been run on the appliance before.

Limitations for Keycloak data when downgrading

When downgrading to some older versions it is not possible to retain all Keycloak data. This applies to data such as additionally created clients, synchronized users, customized user role mappings.

This means, for example, that when downgrading from 20.4 to 20.3 not all Keycloak data is preserved, when downgrading from 20.3 to 20.2 the Keycloak data is preserved.
If a downgrade to those certain versions has to be performed, the Keycloak data must be re-entered manually after the downgrade.

The Keycloak data is stored in database files in the /var/data/keycloak/data/data/ directory. The database files contain the Keycloak version in the file name.

• Mindbreeze InSpire 19.0 up to including version 20.3 uses Keycloak 4.6
• Mindbreeze InSpire 20.4 uses Keycloak 10.0
• Mindbreeze InSpire 20.5 uses Keycloak 11.0
• Mindbreeze InSpire 21.2 uses Keycloak 14.0
• Mindbreeze InSpire 21.3 uses Keycloak 15.1.1
• Mindbreeze InSpire 22.2 uses Keycloak 18.0
• Mindbreeze InSpire 23.3 uses Keycloak 21.1.0
• Mindbreeze InSpire 23.6 uses Keycloak 22.0.1
• Mindbreeze InSpire 24.1 uses Keycloak 23.0.4
• Mindbreeze InSpire 24.5 uses Keycloak 25.0.1
• Mindbreeze InSpire 24.7 uses Keycloak 25.0.6

The following sections describe the steps needed for each case:

Keycloak: Downgrade after Update

1. Before downgrading, make sure that there are old database files in the /var/data/keycloak/data/data/ directory.
   
   e.g. on an InSpire 20.4:
   # ll  /var/data/keycloak/data/data/
   total 2076
   drwxr-xr-x 2 1000 1000       6 Sep  8 13:43 content
   drwxr-xr-x 2 1000 1000      26 Sep  8 13:43 kernel
   -rw-r--r-- 1 1000 1000     146 Sep 16 16:57 keycloak_10_0_2.lock.db
   -rw-r--r-- 1 1000 1000 1302528 Sep 17 10:16 keycloak_10_0_2.mv.db
   -rw-r--r-- 1 1000 1000    7924 Sep 16 16:57 keycloak_10_0_2.trace.db
   -rw-r--r-- 1 1000 1000     146 Sep 14 10:50 keycloak_4_6_0.lock.db
   -rw-r--r-- 1 1000 1000  798720 Sep 14 11:19 keycloak_4_6_0.mv.db
   -rw-r--r-- 1 1000 1000    4507 Sep 14 10:46 keycloak_4_6_0.trace.db
   drwxr-xr-x 2 1000 1000       6 Sep 14 10:44 timer-service-data
   drwxr-xr-x 3 1000 1000      35 Sep 14 11:19 tx-object-store
   
   In this list you can see that a keycloak_4_6_0.mv.db database file exists, which is from the previous, old InSpire version. If you cannot find an older database file here, please follow the section Keycloak: Downgrade without previous version instead
2. Install the old InSpire version in the MMC under "Update". The old Keycloak database file will be used again. All Keycloak data is then back to the state before the last InSpire update.
   
   Note: if you have changed your password since the last InSpire update, you must use the old password again after the downgrade. You will also need to manually redo any changes you have made in Keycloak since the last InSpire update, if necessary.
3. You are now using the old version of InSpire again with the old version of Keycloak. In order to be able to update in the future without problems, you have to do the following steps:
   
   Since the old database files will now be used, you will have to delete the database files with the new version to prevent the data from diverging.
   # ll /var/data/keycloak/data/data/
   total 2080
   drwxr-xr-x 2 1000 1000       6 Sep  8 13:43 content
   drwxr-xr-x 2 1000 1000      26 Sep  8 13:43 kernel
   -rw-r--r-- 1 1000 1000     146 Sep 16 16:57 keycloak_10_0_2.lock.db
   -rw-r--r-- 1 1000 1000 1302528 Sep 17 10:16 keycloak_10_0_2.mv.db
   -rw-r--r-- 1 1000 1000    7924 Sep 16 16:57 keycloak_10_0_2.trace.db
   -rw-r--r-- 1 1000 1000     146 Sep 17 11:15 keycloak_4_6_0.lock.db
   -rw-r--r-- 1 1000 1000  798720 Sep 17 11:15 keycloak_4_6_0.mv.db
   -rw-r--r-- 1 1000 1000    8469 Sep 17 11:15 keycloak_4_6_0.trace.db
   drwxr-xr-x 2 1000 1000       6 Sep 14 10:44 timer-service-data
   drwxr-xr-x 3 1000 1000      35 Sep 14 11:19 tx-object-store
   
   In this case you must delete the files keycloak_10_0_2.lock.db, keycloak_10_0_2.mv.db and keycloak_10_0_2.trace.db.
   
   Then check the contents of the following files:
   # cat /var/data/keycloak/config/keycloak_version
   4_6_0
   # cat /var/data/keycloak/config/keycloak_h2_db_name
   keycloak_10_0_2
   
   The contents of the file /var/data/keycloak/config/keycloak_h2_db_name needs to be changed. Change the version number according to the file /var/data/keycloak/config/keycloak_version.
   
   Then check the contents of the following files, the versions should now match (with the old version):
   4_6_0
   # cat  /var/data/keycloak/config/keycloak_h2_db_name
   keycloak_4_6_0

This completes the downgrade from Keycloak's perspective.

Keycloak: Downgrade without previous version

1. In this case there is no database file that can be used after the downgrade. Check the directory /var/data/keycloak/data/data/ and make sure that no older database files exist.
   # ll  -a /var/data/keycloak/data/data/
   total 796
   drwxr-xr-x 6 1000 root    173 Sep 17 11:32 .
   drwxr-xr-x 4 1000 root     32 Sep 14 11:00 ..
   drwxr-xr-x 2 1000 1000      6 Sep  8 13:43 content
   drwxr-xr-x 2 1000 1000     26 Sep  8 13:43 kernel
   -rw-r--r-- 1 1000 1000    146 Sep 17 11:15 keycloak_10_0_2.lock.db
   -rw-r--r-- 1 1000 1000 798720 Sep 17 11:15 keycloak_10_0_2.mv.db
   -rw-r--r-- 1 1000 1000   8469 Sep 17 11:15 keycloak_10_0_2.trace.db
   drwxr-xr-x 2 1000 1000      6 Sep 14 10:44 timer-service-data
   drwxr-xr-x 3 1000 1000     35 Sep 14 11:19 tx-object-store
   
   If database files of older versions are listed here, continue with the section Keycloak: Downgrade after update instead.
2. Install the old InSpire version in the MMC under "Update" and active the option “Force Update on Recoverable Verification Errors”. After the downgrade no login to the MMC is possible, because the Keycloak (which was not yet downgraded) is in a faulty state. Execute the following commands to reset Keycloak to InSpire default settings and downgrade the Keycloak container:
   
3. inspire-system-manager reset keycloak -f
   
   You can monitor the progress with the command journalctl -f. With the log entry "Initial Config Successful" the operation is completed. Restart all containers with the following command: systemctl restart docker
   
   After that, you have to manually redo all the changes you made in Keycloak in the new InSpire version, if necessary

This also completes the downgrade in this case from Keycloak's perspective.

Downgrade from version 21.3 to 21.2 or older

After a downgrade from version 21.3 to 21.2, credentials can no longer be loaded, corresponding error messages are displayed in the log and affected services no longer start. Example of this error in the log:

/opt/mindbreeze/bin/mesmaster.exe[245]: 2021-12-13 17:53:19,109 [Threadpool worker] ERROR: Failed to load build-in credentials store file /etc/mindbreeze/builtincredentials.proto

To fix this problem the following steps must be performed on the command line:

1. Switch to the inspire container (in a distributed setup, this is the master node):
   docker exec -it inspire bash
2. delete the affected file:
   rm /etc/mindbreeze/builtincredentials.proto
3. restart the affected services:
   systemctl restart mesinspireadminapi
   
   This will recreate the file in the correct format and then automatically restart the affected services. (In a distributed setup, it may take a few minutes for the changes to be synchronized on all nodes)

Downgrade from version 24.7 or younger to version 24.6 or older

Versions that are older than 24.7 do not yet support cgroups in version 2. Therefore, the following steps are necessary before a downgrade:

1. rpm-ostree kargs --replace=systemd.unified_cgroup_hierarchy=1=0
2. reboot