PRD: Refactor Jenkins Pipeline Library — Shared Abstractions and Pipeline Consolidation

[floatingman]

Problem Statement

The rancher/tests repository contains 37 Jenkinsfiles totaling ~5,700 lines of pipeline code with zero local shared abstractions. The same boilerplate — checkout logic, credential loading, Docker cleanup, tofu lifecycle calls, S3 artifact management — is copy-pasted across files with minor variations. This creates three compounding problems:

• Maintenance burden: A change to repo URLs, credential handling, or the tofu module path requires editing every affected file independently. During the airgap pipeline development, the same checkout block was manually duplicated across 3 files.
• Onboarding difficulty: New team members face 37 differently-named Jenkinsfiles with inconsistent patterns. Some use the qa-jenkins-library shared library; most use raw scripted pipelines. Parameter names for the same concept vary between files (DESTROY_ON_FAILURE vs DESTROY_AFTER_TESTS).
• Reliability gaps: Error handling is inconsistent — some pipelines silently swallow errors in try/catch blocks, teardown logic is duplicated with subtle differences, and the same Docker cleanup sequence is written slightly differently in each file.

The airgap pipelines (developed most recently) are the best candidates for initial refactoring because they are the most modern and already use qa-jenkins-library, but the same patterns apply to the broader pipeline ecosystem.

Solution

Create a single-tier shared library in qa-jenkins-library containing all shared abstractions — infrastructure primitives, orchestration helpers, and configuration management. All shared functions live in qa-jenkins-library/vars/ and are consumed by Jenkinsfiles via the Jenkins shared library mechanism.

Originally planned as a two-tier architecture (qa-jenkins-library for primitives + local vars/ for orchestration), this was simplified during implementation. The local vars/ directory approach was abandoned because:

• All functions are equally reusable across any pipeline consuming qa-infra-automation
• Maintaining two contribution workflows (external PR + local commit) adds overhead without benefit
• The qa-jenkins-library already has a Gradle test framework (JenkinsPipelineUnit) for validation

Then consolidate pipelines by similarity group:

• Split airgap infrastructure into separate setup and destroy pipelines (NOT a single ACTION-parameterized pipeline)
• Keep airgap go-tests as a separate pipeline that shares the infra abstractions
• Collapse 4 nearly-identical simple test runners into a single parameterized pipeline
• Delete the deprecated tfp/Jenkinsfile.airgap.tests
• Convert all refactored pipelines to Declarative Pipeline syntax
• Use the qa-infra-automation Makefile as the control point for cluster/registry/rancher operations via make.runTarget()

The refactored system should enable creating a new pipeline in under 30 minutes instead of days.

User Stories

Airgap Infrastructure Pipeline

1. As a QA engineer, I want setup and destroy to be a single pipeline with an ACTION parameter, so that I don't need to maintain two nearly-identical files Changed: Split into separate setup (Jenkinsfile.setup.airgap-rke2-infra) and destroy (Jenkinsfile.destroy.airgap-rke2-infra) pipelines. Separate jobs are cleaner for Jenkins job management and allow different timeout/parameter defaults.
2. As a QA engineer, I want the infra pipeline to support ACTION=setup, ACTION=destroy, and ACTION=setup-and-test Superseded: The setup and destroy jobs are separate, with a shared JJB view grouping them. A future "setup-and-test" job can trigger both in sequence.
3. As a QA engineer, I want the DEPLOY_RANCHER parameter to remain optional on setup, so that I can choose to deploy only the Kubernetes infrastructure without Rancher
4. As a QA engineer, I want terraform.tfvars to be uploaded to S3 during setup and downloaded during destroy via a shared function Deferred: S3 artifact management was stripped from the initial PR scope. The destroy pipeline receives tfvars via the TERRAFORM_CONFIG parameter instead. S3 artifact helpers may be added in a future PR.
5. As a QA engineer, I want the tofu lifecycle (init, workspace create/select, apply, destroy, delete workspace) to be orchestrated by a shared function, so that I don't need to remember the correct sequence in every pipeline
6. As a QA engineer, I want Ansible variable configuration to use a shared function that handles SSH key paths, inventory rendering, and variable substitution, so that the same pattern is used consistently
7. As a QA engineer, I want the checkout block (clone tests + qa-infra-automation) to be a single shared function call, so that repo URLs and branch handling are defined in one place
8. As a QA engineer, I want infrastructure details (bastion DNS, load balancer hostnames) to be output consistently after setup, so that I can access the deployed environment easily

Airgap Test Pipeline

9. As a QA engineer, I want the go-tests pipeline to share the same checkout and infra setup functions as the infra pipeline, so that changes propagate to both
10. As a QA engineer, I want gotestsum invocation to be standardized with consistent flags, output format, and artifact archiving, so that test results are always captured the same way
11. As a QA engineer, I want Qase reporting to be an optional stage that uses the same pattern across all test pipelines, so that I can enable/disable it without code changes
12. As a QA engineer, I want the teardown logic (destroy on failure, destroy after tests) to use the same shared function as the destroy pipeline, so that cleanup is always consistent
13. As a QA engineer, I want cattle-config generation to be a shared function, so that the token extraction and yq patching pattern is not duplicated

Simple Test Runners

14. As a QA engineer, I want the 4 nearly-identical test runners (validation, e2e, harvester, vsphere) to be a single parameterized pipeline, so that bug fixes apply everywhere
15. As a QA engineer, I want to select the target environment via a NODE_LABEL parameter, so that I can run tests against any environment without a dedicated Jenkinsfile
16. As a QA engineer, I want credential sets to be loaded based on the target environment, so that the right cloud provider credentials are always available

Shared Library Foundation

17. As a pipeline developer, I want a resolvePipelineParams() function that parses the job name and resolves branch, repo, and timeout with standard defaults, so that I don't need to write the same 10-line block in every file
18. As a pipeline developer, I want a standardDockerCleanup() function that handles container stop, image removal, and volume cleanup, so that the 15-line cleanup sequence is defined once
19. As a pipeline developer, I want a standardCheckout() function in qa-jenkins-library that handles dual-repo cloning with parameterized branches, so that new pipelines get the correct checkout behavior automatically
20. As a pipeline developer, I want S3 artifact upload/download to be shared functions with a consistent path pattern Deferred: S3 artifact functions (s3.groovy) were removed from the initial PR scope. May be added in a future PR if needed.
21. As a pipeline developer, I want all refactored pipelines to use Declarative Pipeline syntax, so that parameter definitions, post conditions, and stage structure are more readable and get better Jenkins UI integration

Makefile Integration

22. As a QA engineer, I want cluster/registry/rancher operations to use the qa-infra-automation Makefile targets via make.runTarget(), so that the Makefile serves as the single control point for infrastructure operations
23. As a pipeline developer, I want make.runTarget() to handle Docker container execution (mounts, env vars, SSH keys) consistently, so that all Make targets run in the same environment
24. As a QA engineer, I want the destroy pipeline to continue using the Jenkins library directly (NOT the Makefile), because the Makefile's infra-down has an interactive prompt and lacks workspace management

Migration and Coexistence

25. As a QA engineer, I want old and new pipelines to coexist during migration, so that production Jenkins jobs are not disrupted
26. As a QA engineer, I want new Jenkinsfiles to use simplified naming (e.g., Jenkinsfile.setup.airgap-rke2-infra, Jenkinsfile.destroy.airgap-rke2-infra), so that the pipeline purpose is immediately clear from the filename
27. As a pipeline developer, I want parameters to be harmonized across the refactored pipelines (e.g., consistent DESTROY_ON_FAILURE semantics), so that the same parameter name always means the same thing

Developer Experience

28. As a new team member, I want a README in the pipeline directory explaining the shared library structure and how to create a new pipeline, so that I can onboard quickly
29. As a new team member, I want each shared function to have clear parameter documentation, so that I know what to pass without reading the implementation
30. As a pipeline developer, I want to create a new airgap variant (e.g., airgap K3s) in under 30 minutes by using the shared abstractions, so that we can quickly add coverage for new scenarios

Implementation Decisions

Architecture: Single-Tier Library (Revised)

Originally planned as a two-tier library (qa-jenkins-library + local vars/). Simplified during implementation to a single-tier approach.

All shared functions live in qa-jenkins-library (vars/*.groovy):

Module	Functions	Status
airgap.groovy	standardCheckout, configureAnsible, teardownInfrastructure	New in PR #18
make.groovy	runTarget (Docker-based Make target execution)	New in PR #18
config.groovy	getDefaultConfig, getConfig, getConfigValue, mergeConfig, getRepositoryConfig	Extended in PR #18
infrastructure.groovy	parseAndSubstituteVars, writeConfig, generateWorkspaceName, archiveWorkspaceName, writeSshKey, cleanupArtifacts	Extended in PR #18
tofu.groovy	initBackend, createWorkspace, apply, getOutputs	Existing, unchanged
property.groovy	useWithProperties	Existing, unchanged
ansible.groovy	runPlaybook	Existing, unchanged

No local vars/ directory in rancher-tests. All shared code is contributed to qa-jenkins-library via PR, with Gradle-based unit tests (134 tests, JenkinsPipelineUnit framework).

Pipeline Structure: Separate Setup/Destroy (Revised)

Originally planned as a single Jenkinsfile.airgap-rke2-infra with ACTION parameter. Split into separate files during implementation.

• Setup: Jenkinsfile.setup.airgap-rke2-infra — provisions infrastructure, deploys RKE2 cluster, optionally deploys Rancher. Archives Rancher admin token as build artifact.
• Destroy: Jenkinsfile.destroy.airgap-rke2-infra — tears down infrastructure using Jenkins library directly (NOT the Makefile). Takes a TARGET_WORKSPACE parameter.
• JJB: Both jobs are defined in jenkins-job-builder/qa-airgap-rke2-infra.yml with a shared view.

Rationale for splitting:

• Different timeout defaults (setup: 180min, destroy: 120min)
• Different parameter sets (setup has many more params)
• Cleaner Jenkins job management and access control
• The destroy pipeline uses the Jenkins library directly because the Makefile's infra-down has an interactive read -p confirmation prompt and no workspace select/delete management

Makefile Integration (New)

The setup pipeline uses the qa-infra-automation Makefile as the control point for cluster operations via make.runTarget():

Stage	Before (direct calls)	After (Makefile)
Deploy RKE2	Direct ansible-playbook invocation	make.runTarget(target: 'cluster', dir: env.INFRA_DIR, makeArgs: 'ENV=airgap')
Configure Registry	Direct ansible-playbook invocation	make.runTarget(target: 'registry', dir: env.INFRA_DIR, makeArgs: 'ENV=airgap')
Deploy Rancher	Direct helm commands	make.runTarget(target: 'rancher', dir: env.INFRA_DIR, makeArgs: 'ENV=airgap')
Destroy Infra	N/A (was direct tofu)	Uses airgap.teardownInfrastructure() — NOT the Makefile

The make.runTarget() function handles Docker container execution (workspace mounts, SSH key mounts, env var forwarding, AWS credential injection) consistently across all targets.

Syntax Migration

• All refactored pipelines use Declarative Pipeline syntax (pipeline { ... }).
• Complex teardown logic uses post { failure { ... } } blocks for cleanup-on-failure.
• The library directive uses env.QA_JENKINS_LIBRARY_BRANCH (not params.*) because params aren't resolved at library load time.

Parameter Harmonization

• DESTROY_ON_FAILURE is the unified parameter name (used in both infra and test pipelines).
• Sensitive parameters use password type: PRIVATE_REGISTRY_PASSWORD, RANCHER_BOOTSTRAP_PASSWORD, RANCHER_ADMIN_PASSWORD.
• All pipelines use the same parameter defaults for QA_JENKINS_LIBRARY_BRANCH, TESTS_BRANCH, QA_INFRA_BRANCH.

Dockerfile Strategy

• Dockerfile.infra includes make for Makefile target execution.
• Docker image building uses consistent platform (linux/amd64) and tagging (rancher-infra-tools:latest).

Shared Function Interfaces

• make.runTarget(target, dir, makeArgs, envVars, returnStdout, mountSsh, passAwsCreds): Runs a Make target inside the infra-tools Docker container with proper mounts and env vars.
• airgap.standardCheckout(testsRepo, infraRepo): Clones both repos and returns directory paths.
• airgap.configureAnsible(sshKey, inventoryVars, ansibleDir, inventoryFile, validate): Handles SSH key setup, variable substitution, and inventory file writing.
• airgap.teardownInfrastructure(dir, name, varFile): Full teardown sequence — select workspace, destroy, delete workspace.
• infrastructure.parseAndSubstituteVars(content, envVars): Replaces ${VAR} placeholders in config strings.
• infrastructure.writeConfig(path, content): Writes processed config to a file.
• infrastructure.generateWorkspaceName(prefix, suffix, includeTimestamp): Creates unique workspace names.
• infrastructure.archiveWorkspaceName(workspaceName): Persists workspace name to file for post-failure cleanup.
• infrastructure.writeSshKey(keyContent, keyName, dir): Writes SSH key material to disk.
• infrastructure.cleanupArtifacts(paths, force): Removes local artifacts after pipeline completion.

Testing Decisions

Testing Strategy: Gradle Unit Tests + Live Pipeline Testing

Unit testing (qa-jenkins-library):

• 134 Gradle tests using JenkinsPipelineUnit framework
• Tests cover parameter validation, Docker command construction, config merging, workspace name generation, variable substitution
• ./gradlew test runs the full suite

Live pipeline testing (rancher/tests):

• Each refactored pipeline is created alongside the original (parallel coexistence) and triggered manually to verify functional equivalence.
• The old pipelines remain active until the new ones have been verified across at least 2 successful runs.

Modules to Test via Live Execution

• Jenkinsfile.setup.airgap-rke2-infra — verify infra creation, RKE2 deployment, optional Rancher deployment
• Jenkinsfile.destroy.airgap-rke2-infra — verify teardown cleans up all resources and deletes workspace
• Post-failure cleanup in setup pipeline — verify DESTROY_ON_FAILURE triggers teardown on failure

Migration Verification Checklist

• Setup pipeline provisions identical AWS resources
• Setup pipeline deploys RKE2 cluster via Makefile target
• Setup pipeline deploys Rancher (optional) via Makefile target
• Setup pipeline archives Rancher admin token artifact
• Destroy pipeline cleanly removes all resources
• Post-failure cleanup works in setup pipeline
• Old pipelines still work during coexistence period

Out of Scope

• Phase 2+ pipeline groups: Elemental pipelines (Group D), upgrade pipelines (Group C), TFP pipelines (Group E), Neuvector consolidation (Group G), and Harvester bare-metal pipeline are not included in Phase 1. The shared library foundation will support their future migration.
• qa-jenkins-library structural changes: We are adding new functions, not refactoring existing ones. The existing tofu.*, property.*, config.* interfaces remain unchanged.
• S3 artifact management: s3.uploadArtifact and s3.downloadArtifact were removed from the initial PR scope. Can be added in a future PR if needed.
• Local vars/ directory: The two-tier library approach was abandoned. All shared functions live in qa-jenkins-library.
• Jenkinsfile.rc: The RC pipeline is complex enough (378 lines, corral packages, multi-SCM) to warrant its own dedicated refactor.
• Shell scripts in validation/pipeline/scripts/: The existing shell scripts continue to work as-is.
• CI/CD job configuration in Jenkins: This PRD covers Jenkinsfile content only, not Jenkins job configuration, credentials setup, or plugin management.
• Non-pipeline files: Go test code, Dockerfiles (except potentially minor adjustments), and other repository content are not in scope.

Further Notes

Implementation Order (Updated)

1. Create local vars/ directory with shared functions — Abandoned. All functions go to qa-jenkins-library.
2. ✅ Submit PR to qa-jenkins-library adding airgap.*, make.*, infrastructure.*, config.* additions (PR fix spelling errors #18)
3. ✅ Create Jenkinsfile.setup.airgap-rke2-infra and Jenkinsfile.destroy.airgap-rke2-infra (PR feat: split airgap RKE2 infra pipeline into setup/destroy jobs #595)
4. ✅ Create JJB definitions in jenkins-job-builder/qa-airgap-rke2-infra.yml
5. Build simpleTestPipeline in qa-jenkins-library
6. Create Jenkinsfile.validation
7. Verify all new pipelines via live execution
8. Delete tfp/Jenkinsfile.airgap.tests
9. Add README documentation for the shared library structure

Parallel Coexistence Plan

Old and new Jenkinsfiles coexist during migration. New files use the split naming convention (Jenkinsfile.setup.*, Jenkinsfile.destroy.*). Once the new pipelines are verified:

1. Update Jenkins job configurations to point to new Jenkinsfiles
2. Monitor for 1-2 weeks
3. Archive old Jenkinsfiles (move to deprecated/ directory or delete)

Risks

• qa-jenkins-library PR timeline: Since it requires external PR review, the airgap pipeline refactor may be blocked if the library PR is delayed. Mitigation: Branch references in the JJB defaults point to feature branches, so both repos can be tested before merge.
• Declarative pipeline limitations: Some complex scripted patterns (like the destroyInfrastructure closure in go-tests) may require script { } blocks within Declarative syntax. This is acceptable and well-documented.
• Build time regression: Consolidating Dockerfiles was rejected specifically to avoid increasing the infra pipeline's build time. This should be monitored.
• Makefile coupling: Using make.runTarget() couples the Jenkins pipeline to the qa-infra-automation Makefile structure. If Makefile targets change, pipeline stages may break silently. Mitigation: The Makefile targets (cluster, registry, rancher) are stable, well-documented interfaces.