MMC cleanup may delete referenced content on database failure

Last update: 6 July 2020

Summary

Fabasoft found an issue in the MMC cleanup functionality in a specific error condition that may lead to deletion of content in an MMC-Service area, that is still referenced in the database and therefore shouldn't be deleted.

The issue is unlikely to appear as it is dependent to several specific conditions that must apply together. These conditions are explained in the Details part of the article.

Nevertheless of the unlikeliness of this conditions, Fabasoft recommends to disable the MMC cleanup functionality temporarily and install our hotfix for this issue.

The issue is independent of the used database system.

Details

The MMC-Service cleanup functionality may delete files in the MMC-Service area if all of the following conditions occur:

  • A connection issue with the database occurred, e.g. a cluster failover or a short outage of the network connection from the COO-Service to the database.
  • The COO-Services stayed online throughout this connection issue to the database.
  • Later, after the database is online again, an MMC cleanup is started.
  • The Kernel processing the cleanup functionality gets a COO-Service thread assigned that already had an open connection to the database since before of the database outage.
  • The open connection to the database was never used since of the outage of the connection to the database.
  • A single database request to query data about an object to query the MMC file location uses the orphaned open database connection.

If all conditions are met, the database query may not return correct object data for the MMC file locations. In result of the missing data of the database, the cleanup functionality may assume a file or subset of files are not needed anymore in the system and will delete them mistakenly.

This situation can only appear once as the COO-Service automatically reconnects to the database. But for the affected query of the failure, the file or files in the MMC-Service area may have been deleted.

Solution

Fabasoft provides a hotfix to correct this issue for all Fabasoft Folio / Fabasoft eGov-Suite versions.

For Fabasoft Support customers, please open a ticket including your exact Fabasoft Folio / Fabasoft eGov-Suite version (including build number).

For Fabasoft customers without active Fabasoft Support contract, please contact Fabasoft Support via e-mail including the mentioned information.

With transmission of the hotfix, Fabasoft Support will add further information about how to install the hotfix.

About the hotfix

The hotfix fixes the coostd.exe (Windows environments) respectively coostd (Linux environments). In a ready installed environment, this single file can be swapped in a short maintenance downtime. It is not necessary to run the setup or to install a Software Component. 

The hotfix package is a ZIP file and will contain:

  • Single coostd.exe (Windows environments) and coostd (Linux environments): The appropriate file can be exchanged on all your backend servers during a planned downtime.
  • Installation Package: This can be copied into your installation kit to integrate the hotfix
    • \Setup\ComponentsBase\Services\WINDOWS_X64\FscServer.msi
    • \Setup\ComponentsBase\Services\LINUX_RHEL_X64\fsc-services-<buildnumber>.rpm

Build numbers including the hotfix:

  • 2013 Update Rollup 6: From installation kit 13.0.13.43 (Binary version: 13.0.13.43)
  • 2014 Update Rollup 6: From installation kit 14.0.13.58 (Binary version: 14.0.13.58)
  • 2016 Update Rollup 5: From installation kit 16.0.9.49 (Binary version: 16.0.9.49)
  • 2016 Update Rollup 6: From installation kit 16.0.10.98 (Binary version: 16.0.10.98)
  • 2016 Update Rollup 7: From installation kit 16.0.11.64 (Binary version: 16.0.11.64)
  • 2017 R1: From installation kit 17.4.0.58 (Binary version: 17.4.0.16)
  • 2017 R1 Update Rollup 6: From 17.4.6.8 (Binary version: 17.4.6.0)
  • 2019 Update Rollup 1: From installation kit 19.2.1.185 (Binary version: 19.2.1.7)
  • 2020: From installation kit 20.1.0.345 (Binary version: 20.1.0.9)

If your version is not listed, please contact Fabasoft Support. We will create the appropriate hotfix for your Fabasoft Folio/eGov-Suite version.

Before you have installed the hotfix

Fabasoft recommends to temporary disable the MMC area cleanup functionality until the hotfix is installed in your environment.

    How to check if the issue occurred

    The concrete occurrence of the issue is not logged as the database failure is not recognized by the cleanup feature. An indication of happened database connection problems are several Windows Event Log / COO-Service Error Log messages before and during the cleanup run. When such messages appear and the hotfix isn't installed yet, you should stop the cleanup processing to avoid the occurrence of the failure.

    The fscadmin tool provides a consistency check for files in the MMC-Service areas (-action checkconsistency). This check cannot explicitly allocate missing files from this issue, but will output all potentially missing files without any information of the type and time of occurrence.