Skip to content

README.md

Latest commit

 

History

History
179 lines (135 loc) · 10.2 KB

README.md

File metadata and controls

179 lines (135 loc) · 10.2 KB
Raw

Provisioning Service Documentation for UCS Systems

Overview

The Nubus Provisioning service was originally designed for running in Kubernetes. It was then migrated to UCS. UCS' system role concept (Primary, Backup, Replica, Memberserver) requires the Provisioning system to work slightly different in UCS and Kubernetes, with the main differences being in the "NATS cluster management" (see section Architecture) and the Backup to Primary failover mechanism (see section of the same name).

Architecture

The provisioning service on UCS systems consists of two apps:

  • provisioning-service-backend: This app installs a listener module that captures LDAP changes and writes them to the LDAP-queue. This app is only installed on the Primary Directory Node.
  • provisioning-service: This app contains the main provisioning service components: udm-transformer, prefill, dispatcher, and provisioning API and is installed on the Primary and on all Backups.

See architecture.md for an in-depth explanation of the architecture of the provisioning service on UCS.

Backup to Primary failover (univention-backup2master)

The provisioning service supports backup to primary failover.

The backup-to-primary scenario is automatically handled via appcenter/provisioning-service/scripts/50provisioning-service-backup2master, which is executed during univention-backup2master. It installs the provisioning-service-backend app on the new Primary Directory Node and reinitializes the provisioning-service app.

Location of the script on a UCS system:

/usr/lib/univention-backup2master/post/50provisioning-service-backup2master

Manual steps after univention-backup2master

  • On all remaining Backup Directoy Nodes:
    • follow this article (specifically ldap/master needs to be correct): https://help.univention.com/t/how-to-backup2master/19514 -execute univention-app configure provisioning-service --set provisioning-service/udm-rest-api-host=<REPLACE_ME> --set provisioning-service/primary/url=<REPLACE_ME>"
  • Restart all provisioning consumers (they usually just terminate and in k8s that's enough to trigger the automatic restart, but not in UCS).

Manual maintenance and update steps

The maintenance and update process consists of two primary manual workflows:

  1. Third-party dependency and image maintenance: Reviewing security scan results for third-party dependencies and updating image versions as needed to address security findings.
  2. Internal code change management: Executing all required follow-up steps after changes to internally developed code.

Third-party dependency and image maintenance

The first essential part of this workflow is our security monitoring. Security monitoring is performed using DependencyTrack, which periodically analyzes the Software Bill of Materials (SBOMs) for all container images used by us to identify known security vulnerabilities (CVEs).

When DependencyTrack reports that a package used in one of our container images is associated with a CVE, we first assess whether the vulnerability is applicable to our specific usage of the affected package ("triage"). If the vulnerability does not impact our use case, we document this assessment by creating a corresponding entry in a Rego chart, detailing the rationale for why the CVE is not relevant to the shipped software.

If the vulnerability is determined to be relevant, the affected container images must be updated to reference a package or library version that includes the corresponding fix. After updating the affected container images, new versions of N4K and the Provisioning Service app need to be released. The necessary manual steps for that are documented below.

CVE fix for NATS container image

If the NATS image version needs to be inscreased, changes must be made to two repositories: provisioning, nubus-helm.

The nats helm chart lives in this repository as a local file dependency of the provisioning helm chart and is only published as part of it.

The following steps must be performed:

  1. provisioning/helm/nats/values.yaml: update nats container image version -> bump version in provisioning/helm/nats/Chart.yaml and the matching nats dependency version in provisioning/helm/provisioning/Chart.yaml -> push -> merge -> new provisioning helm chart version generated by pipeline
  2. nubus-helm/helm/nubus/Chart.yaml: update the provisioning helm chart version in Chart.yaml -> execute helm dependeny update helm/nubus -> push -> merge -> automatic new nubus release
  3. perform Provisioning Service app release for UCS

CVE fix for Python libraries

Security fixes for internally developed components of the Provisioning Service, i.e. Dispatcher, are limited to the Python libraries they use. Library versions are managed via Python package manager UV, so remediation involves updating the version number of the affected python library in the pyproject.toml file of all affected components. After modifying a pyproject.toml, run: uv sync once before pushing.

The following steps must be performed:

  1. update pyproject.toml of affected components -> uv sync -> push -> merge
  2. perform Provisioning Service app release for UCS

Internal code change management

All Python dependencies in this project are managed using the Python package manager UV. Any changes to dependencies must be reflected in the relevant pyproject.toml files and accompanied by updated lock files (uv sync) to ensure reproducible builds across environments.

Provisioning Service (Backend) app release for UCS

Both apps release via the same app-release CI/CD component, each behind its own protected tag pattern so releasing one never rebuilds or republishes the other's artifacts:

  • provisioning-service (docker-app): tag vX.Y.Z, starting at v2.2.0. Also builds/signs/publishes container images and helm charts. App files: appcenter/provisioning-service/. Jobs are prefixed provisioning-service- (provisioning-service-create_app_version, etc.).
  • provisioning-service-backend: tag provisioning-service-backend-vX.Y.Z, starting at provisioning-service-backend-v2.0.3. Also builds the univention-provisioning-service .deb and uploads it into the app version. App files: appcenter/provisioning-service-backend/. Jobs are prefixed provisioning-service-backend- (provisioning-service-backend-create_app_version, etc.).

Testing an App Center version from a merge request:

  1. On the MR pipeline, manually run provisioning-service-create_app_version (or provisioning-service-backend-create_app_version). It creates a temporary 0.0.0-<branch-slug> version in the Test App
  2. Install that version on a test UCS system from the Test App Center.
  3. The test version is deleted automatically when the MR is closed/merged or after 3 days (GitLab environment auto-stop).

Pipelines on main keep 999.0.0-staging up to date in the Test App Center for both apps.

Performing a release:

  1. Make sure main is at the commit you want to release.
  2. Create and push a protected tag on that commit, e.g. git tag -a v2.2.0 -m "Provisioning Service 2.2.0" && git push origin v2.2.0 (provisioning-service-backend-vX.Y.Z for the backend).
  3. The tag pipeline creates the App Center version and uploads the app files (docker-app: with digest-pinned compose images, backend: the built .deb package).
  4. Verify the version in the Provider Portal / Test App Center: provisioning-service, provisioning-service-backend
  5. Trigger the manual provisioning-service-do_release (or provisioning-service-backend-do_release) job to copy the version from the Test App Center to the production App Center.
  6. Confirm the manual provisioning-service-check_release (or provisioning-service-backend-check_release) job.
  7. The pipeline creates the GitLab release and sends the announcement mail to [email protected].

Execute automated tests for Kubernetes and UCS deployment

Before releasing a new version of the Provisioning Service, all automated tests must be executed and their results reviewed.

Automated tests for Kubernetes deployment

The automated Kubernetes tests consist of multiple jobs within the nubus-helm pipeline and can be triggered manually. These jobs include smoke tests and end-to-end (E2E) tests.

Automated tests for UCS deployment

Execute this Jenkins test for UCS.

Additional information about automations of the update and maintenance processes

All Provisioning Service containers, except the NATS container, are based on the ucs-base-image. A new ucs-base-image version with pinned apt cache is created every Thursday by a scheduled GitLab pipeline and includes errata updates from the regular UCS errata release process. The ucs-base-image version is automatically updated by Renovate Bot in this repository.