| layout | post |
|---|---|
| title | How to run the Vox Pupuli test suite |
| summary | A description of how to run the Vox Pupuli test suite for Puppet modules. |
The testing and development tools have a bunch of dependencies, all managed by bundler. By default the tests use the latest version of Puppet. If you want a specific version of Puppet, you must set an environment variable such as:
export PUPPET_VERSION="~> 8.8.1"Dependencies for running tests are installed as gems via bundler and will run in Ruby 3.2 and newer (as of time of writing). It should be trivial to install via your package manager or gem.
# apt install ruby-bundlerAs of el9, dnf replaces yum. Use the appropriate package manager to get it installed.
# yum install rubygem-bundler
# dnf install rubygem-bundler# pacman -Syu extra/ruby-bundler# gem install bundlerYou can install all needed gems for spec tests into the modules directory by running:
bundle config set --local path '.vendor/bundle'
bundle config set --local without 'development system_tests release'
bundle installIf you also want to run acceptance tests, don't exclude the system_tests group.
If you don't know if you need to install or update gems, you can just add bundle update && bundle clean as an additional command.
Check out the following page if you want to add a test suite to your module or want to learn more about the Vox Pupuli test helpers:
- voxpupuli-test for unit testing
- voxpupuli-acceptance for acceptance testing
- voxpupuli-release for creating a release
To get a rough understanding, we created a dependency graph for each of our meta gem. Keep in mind:
- We exclude std- and default gems
- This graph is created by hand and not updated automatically
- All gems maintained by Vox Pupuli or the OpenVoxProject are red
- All gems maintained by Perforce are green
- The idea is to provide a rough overview of the ecosystem and ownership
classDef voxpupuli fill:#ffcccc,stroke:#aa0000,color:#000; classDef puppetlabs fill:#ccffcc,stroke:#009900,color:#000; class puppetlabs puppetlabs; class voxpupuli voxpupuli; class voxpupuli-test voxpupuli; class facterdb voxpupuli; class metadata-json-lint voxpupuli; class json-schema voxpupuli; class semantic_puppet puppetlabs; class openvox-strings voxpupuli; class openvox voxpupuli; class openfact voxpupuli; class hocon puppetlabs; class puppet-resource_api puppetlabs; class puppet_fixtures voxpupuli; class puppet-syntax voxpupuli; class rspec-puppet-facts voxpupuli; class voxpupuli-puppet-lint-plugins voxpupuli; class puppet-lint puppetlabs; class puppet-lint-absolute_classname-check voxpupuli; class puppet-lint-anchor-check voxpupuli; class puppet-lint-exec_idempotency-check voxpupuli; class puppet-lint-file_ensure-check voxpupuli; class puppet-lint-leading_zero-check voxpupuli; class puppet-lint-lookup_in_parameter-check voxpupuli; class puppet-lint-manifest_whitespace-check voxpupuli; class puppet-lint-optional_default-check voxpupuli; class puppet-lint-param-docs voxpupuli; class puppet-lint-params_empty_string-check voxpupuli; class puppet-lint-params_not_optional_with_undef-check voxpupuli; class puppet-lint-param-types voxpupuli; class puppet-lint-resource_reference_syntax voxpupuli; class puppet-lint-strict_indent-check voxpupuli; class puppet-lint-topscope-variable-check voxpupuli; class puppet-lint-trailing_comma-check voxpupuli; class puppet-lint-unquoted_string-check voxpupuli; class puppet-lint-variable_contains_upcase voxpupuli; class puppet-lint-version_comparison-check voxpupuli;
classDef voxpupuli fill:#ffcccc,stroke:#aa0000,color:#000; classDef puppetlabs fill:#ccffcc,stroke:#009900,color:#000; class puppetlabs puppetlabs; class voxpupuli voxpupuli; class voxpupuli-release voxpupuli; class openvox-strings voxpupuli; class openvox voxpupuli; class openfact voxpupuli; class hocon puppetlabs; class puppet-resource_api puppetlabs; class semantic_puppet puppetlabs; class puppet-blacksmith voxpupuli;
Vox Pupuli uses puppet-lint for better code quality. To run it:
bundle exec rake lintTo automatically fix puppet-lint issues you can use:
bundle exec rake lint_fixAso run some RuboCop tests against it:
bundle exec rake rubocopRuboCop also supports save autofixing:
bundle exec rake rubocop:autocorrectAnd another autofix option that can fix more, but might break your code:
bundle exec rake rubocop:autocorrect_allIf REFERENCE.md is now out of date you can fix it with:
bundle exec rake strings:generate:referenceThe unit test suite covers most of the code, as mentioned above please add tests if you're adding new functionality. If you've not used rspec-puppet before then feel free to ask about how best to test your new feature.
To run the linter, the syntax checker and the unit tests:
bundle exec rake testbundle exec rake specIf you have multiple cpu cores available we suggest using the following command:
bundle exec rake parallel_specSingle test file could be run by:
bundle exec rspec spec/classes/myclass_spec.rb
bundle exec rake spec SPEC=spec/classes/myclass_spec.rbTo limit test execution to a certain os or os release you can set the environment variable SPEC_FACTS_OS.
export SPEC_FACTS_OS=centos
export SPEC_FACTS_OS=centos-10The unit tests just check the code runs, not that it does exactly what we want on a real machine. For that we're using Beaker.
This fires up a new virtual machine or container and runs a series of simple tests against it after applying the module.
Follow the steps in Installing Dependencies making sure to install the system_tests group of dependencies.
bundle config set --local without 'development release'Beaker uses a local hypervisor or container stack to build required OS/Distributions to test. By default this is Docker, but it supports different hypervisors via additional gems.
Note: it also works with Podman but you need to point it to the appropriate socket, for example with systemctl start --user podman.socket && export DOCKER_HOST=unix:///run/user/$(id -u)/podman/podman.sock
Follow the instructions from Docker or your preferred/known working method to install.
For libvirt / vagrant, use the distribution's packages to configure.
Remember to add your user to the group that allows access to docker/libvirt.
For Vox Pupuli's acceptance testing suite, Beaker is managed by a set of environment variables.
-
BEAKER_HYPERVISORSets the Hypervisor,vagrantorvagrant_libvirtfor VM based testing It is a good idea to export theBEAKER_HYPERVISORvariable in your shell configuration. -
BEAKER_DESTROYShould the test environment be removed at the end of a testBEAKER_DESTROY=onpassOnly removes the environment if everything passes, allowing review of the system in the state it was in when the test suite failed.BEAKER_DESTROY=noAlways leave an artifact, which likely will break future runs without cleanup.BEAKER_DESTROY=yes(DEFAULT) always clean up.
-
BEAKER_PROVISIONShould we ensure a clean system is built to run the suite.- If
BEAKER_DESTROYis set to no,BEAKER_PROVISION=yeswill fail the run (because of the existing box/container) - If
BEAKER_DESTROYis set to no,BEAKER_PROVISION=nowill run the test suite against the system anyways, which may cause environmental issues if the test suite doesn't perfectly put things to a default state.
- If
-
BEAKER_SETFILEWhat should we be testing. Beaker will call beaker-hostgenerator to create a defaultconfiguration based on this, using the known configurations for Vox Pupuli. -
BEAKER_PUPPET_COLLECTIONWhat implementation and version of Puppet are we testing against
These are all defined in voxpupuli-acceptance to review how they are used and more specialized features.
Installed as part of the voxpupuli test suite is metadata2gha, which as its name suggests, is a helper to generate a json blob containing the variables for all supported oses.
# From the base directory of the module
$ bundle exec metadata2gha | grep puppet_beaker_test_matrix | cut -d= -f2- | jq
[
# ... truncated for length
{
"name": "Puppet 8 - Debian 12",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet8",
"BEAKER_SETFILE": "debian12-64{hostname=debian12-64-puppet8}"
}
},
{
"name": "Puppet 7 - Debian 12",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet7",
"BEAKER_SETFILE": "debian12-64{hostname=debian12-64-puppet7}"
}
},
# ... truncated for length
{
"name": "OpenVox 8 - Rocky 9",
"env": {
"BEAKER_PUPPET_COLLECTION": "openvox8",
"BEAKER_SETFILE": "rocky9-64{hostname=rocky9-64-openvox8}"
}
},
{
"name": "OpenVox 7 - Rocky 9",
"env": {
"BEAKER_PUPPET_COLLECTION": "openvox7",
"BEAKER_SETFILE": "rocky9-64{hostname=rocky9-64-openvox7}"
}
},
# ... truncated for length
]This returns names that show Puppet implementation (OpenVox or Puppet) / language version, and operating system / version. It also provides the environment variables needed to be applied to run Beaker for this combination.
Running Beaker is handled via a rake task bundle exec rake beaker, and either exporting variables or setting them for a single run are both supported:
-
export variables, then run
bundle exec rake beakerexport BEAKER_SETFILE="ubuntu2404-64" bundle exec rake beaker
-
run
bundle exec rake beakerwith inline shell variablesBEAKER_SETFILE="ubuntu2404-64" bundle exec rake beaker
Some softwares might also need to have an hostname set on the VM, to be used as FQDN, it is possible to set is through BEAKER_SETFILE:
BEAKER_SETFILE="almalinux9-64{hostname=almalinux9-64-puppet8.example.com}" bundle exec rake beakerIf you need to run tests against a different version or implementation of OpenVox (or Puppet) you can either export BEAKER_PUPPET_COLLECTION="openvox7" or add BEAKER_PUPPET_COLLECTION="openvox7" to your command line.
BEAKER_PUPPET_COLLECTION="openvox7" BEAKER_SETFILE="ubuntu2404-64" bundle exec rake beakerProjects might also have custom facts that can be set through environment variables, for example puppet-zabbix tests against a specific Zabbix version setting BEAKER_FACTER_zabbix_version which will pass a custom fact to the system of zabbix_version:
BEAKER_SETFILE="almalinux9-64" BEAKER_FACTER_zabbix_version="6.0" bundle exec rake beakerFor these cases refer to the module's documentation and/or on GitHub CIs check the env for bundle exec rake beaker in tests.
For additional information, you can review voxpupuli-acceptance.
As with unit tests, it is possible to run only a specific acceptance test, for example:
bundle exec rspec spec/acceptance/foo.rbStruggling with Ruby installations or dependency issues? VoxBox simplifies your workflow by providing a ready-to-use container with all the Ruby tools you need. To guarantee a consistent rake environment, use -f /Rakefile to explicitly specify your Rakefile, rather than relying on potentially outdated versions in a repository. Learn more in the project's README.
podman pull ghcr.io/voxpupuli/voxbox:8cd my/module
podman run -it --rm -v $PWD:/repo:Z ghcr.io/voxpupuli/voxbox:8 -f /Rakefile lint
podman run -it --rm -v $PWD:/repo:Z ghcr.io/voxpupuli/voxbox:8 -f /Rakefile lint_fix
# lint with all lint plugins enabled, puppetlabs-spec_helper deactivates some of them
podman run -it --rm -v $PWD:/repo:Z ghcr.io/voxpupuli/voxbox:8 -f /Rakefile voxpupuli:custom:lint_allpodman run -it --rm -v $PWD:/repo:Z ghcr.io/voxpupuli/voxbox:8 -f /Rakefile rubocop
podman run -it --rm -v $PWD:/repo:Z ghcr.io/voxpupuli/voxbox:8 -f /Rakefile rubocop:autocorrectpodman run -it --rm -v $PWD:/repo:Z ghcr.io/voxpupuli/voxbox:8 -f /Rakefile spec
podman run -it --rm -e "SPEC_FACTS_OS=centos" -v $PWD:/repo:Z ghcr.io/voxpupuli/voxbox:8 -f /Rakefile spec
podman run -it --rm -e "SPEC=spec/classes/myclass_spec.rb" -v $PWD:/repo:Z ghcr.io/voxpupuli/voxbox:8 -f /Rakefile specpodman run -it --rm -v $PWD:/repo:Z ghcr.io/voxpupuli/voxbox:8 -f /Rakefile strings:generate:referencepodman run -it --rm -v $PWD:/repo:Z ghcr.io/voxpupuli/voxbox:8 -f /Rakefile r10k:syntax
podman run -it --rm -v $PWD:/repo:Z ghcr.io/voxpupuli/voxbox:8 -f /Rakefile r10k:dependencies