This blog takes a look at matrix testing in GitHub Actions (GHA). They are powerful, but the naive implementation requires some maintenance. But GHA also provides tools to avoid that. A basic familiarity with GHA is assumed. The examples use Ruby, but the techniques apply to most ecosystems.
A quick introduction
Matrix testing in GitHub a typically defined like this:
jobs:
test:
name: 'Ruby ${{ matrix.ruby }}'
runs-on: ubuntu-latest
strategy:
matrix:
ruby:
- '3.2'
- '3.1'
- '3.0'
- '2.7'
steps:
- uses: actions/checkout@v3
- uses: ruby/setup-ruby@v1
with:
ruby-version: ${{ matrix.ruby }}
bundler-cache: true
- run: bundle exec rake test
This gives you a testing matrix where it runs 4 times, once for each Ruby version. Perfect for most places.
Dynamically generating the matrix
In GitHub you can define multiple steps and pass data from one step to the next. It can look something like this:
jobs:
setup_matrix:
name: "Determine Ruby versions"
runs-on: ubuntu-latest
outputs:
ruby: ${{ steps.ruby.outputs.versions }}
steps:
- id: ruby
run: echo 'ruby=["3.2", "3.1", "3.0", "2.7"]' >> $GITHUB_OUTPUT
test:
# ...
needs: setup_matrix
strategy:
matrix:
ruby: ${{ fromJSON(needs.setup_matrix.outputs.ruby }}
# ...
This looks way more complex and you can question why anyone would do this. The trick is that the trivial echo can be replaced with a script that is smarter.
In Ruby it's possible to specify the required_ruby_version in your gemspec. A simplified script to do so follows. In the real world you want to do error checking.
#!/usr/bin/env ruby
require 'json'
require 'rubygems'
VERSIONS = ['3.2', '3.1', '3.0', '2.7', '2.6', '2.5', '2.4']
filename = ARGV.first
spec = Gem::Specification::load(filename)
requirement = spec.required_ruby_version
versions = VERSIONS.select { |version| requirement =~ Gem::Version.new(version) }
File.open(ENV['GITHUB_OUTPUT'], 'a') do |f|
f.puts("#{ruby=#{versions.to_json}")
end
Now it will try a list of known versions and match it against the compatible versions in your gemspec.
Adding and removing support for a Ruby version now comes down to changing required_ruby_version in your gemspec.
Even better: Bundler will also respect this.
You may expect me to tell you where to place this script. Instead of storing it inside our gem repository we're going to make it reusable.
Composite workflows
Repeating yourself is a bad pattern. I strongly believe that templates with boilerplate are a bad practice. You end up with massive workflow definitions and if you maintain multiple repositories they are bound to get out of sync. Luckily GitHub has also seen this and came up with composite (and reusable workflows, but more on that later).
A composite workflow is a small single repository with some files in it and a definition describing the action.
First store the script we wrote earlier as some filename.
This example uses extract_compatible_ruby_versions.
Then define the action.
name: 'Derive Ruby versions'
description: "Determine Ruby version testing matrix based on the gemspec's required_ruby_version field"
inputs:
filename:
description: 'Gemspec filename'
required: false
default: '*.gemspec'
outputs:
versions:
description: "Ruby versions"
value: ${{ steps.script.outputs.ruby }}
runs:
using: "composite"
steps:
- uses: actions/checkout@v3
- id: script
run: ${{ github.action_path }}/extract_compatible_ruby_versions ${{ inputs.filename }}
shell: bash
This creates a checkout of the repository and runs the script we created on the input filename.
By default, it uses *.gemspec and relies on bash to expand it.
Keen readers will recognize this doesn't deal with spaces, subdirectories and multiple gemspecs.
Those problems are left as an exercise for the reader.
Commit this to a repository.
I have done that on ekohl/ruby-version and use the v0 branch for it.
With this knowledge we can update our setup job:
jobs:
setup_matrix:
# ...
steps:
- id: ruby
uses: ekohl/ruby-version@v0
When a new version of Ruby comes out all that needs to be updated is the single repository and anything that uses it will automatically get their test matrix expanded with Ruby it.
This may be seen as disruptive, but users running that latest version will install your gem.
If you don't want this, you should pin your required_ruby_version to avoid that new version.
Reusing workflows
This can be taken a step further with reusable workflows. Where composite workflows aim to abstract away one step (or a few), reusable workflows aim to abstract away whole jobs. Taking the previous code we'll define the whole gem testing workflow.
jobs:
setup_matrix:
name: "Determine Ruby versions"
runs-on: ubuntu-latest
outputs:
ruby: ${{ steps.ruby.outputs.versions }}
steps:
- id: ruby
uses: ekohl/ruby-version@v0
test:
name: "Ruby ${{ matrix.ruby }}"
runs-on: ubuntu-latest
needs: setup_matrix
strategy:
fail-fast: false
matrix:
ruby: ${{ fromJSON(needs.setup_matrix.outputs.ruby) }}
steps:
- uses: actions/checkout@v3
- name: Setup ruby
uses: ruby/setup-ruby@v1
with:
ruby-version: ${{ matrix.ruby }}
bundler-cache: true
- name: Run tests
run: bundle exec rake test
Now store this in a repository somewhere.
I did that in theforeman/actions as .github/workflows/test-gem.yml in the v0 branch.
Now in your gem repository you can define your CI job:
jobs:
test:
name: Tests
uses: theforeman/actions/.github/workflows/test-gem.yml@v0
Now this is something we can maintain easily. With just a few lines there's a minimal maintenance burden.
Bonus: Enabling auto-merge
GitHub has a feature to automatically merge a pull request. The idea is that as a repository maintainer you can approve a PR and enable auto-merge when certain required checks pass. Very often you have one last comment on a PR which is then addressed by the author. You review that small change, approve it and have to wait on CI to complete. Auto-merge helps people like me who forget to check back, which is frustrating for the author. Worst case a merge conflict shows up and you have to go through the whole cycle again.
For our example it would mean an admin would mark the tests Ruby 3.2, Ruby 3.1, Ruby 3.0 and Ruby 2.7 as required.
This does mean that when a PR wants to drop Ruby 2.7 support some admin needs to remove Ruby 2.7 as a required check.
That would apply to all PRs, meaning there is a period where it could break but still allow auto-merging to continue.
Similarly, when adding new versions the repository admin needs to add Ruby 3.3 as a required check.
However, the whole point of making things dynamic was to avoid manually maintaining lists.
I'm not aware of a way to make a test matrix a required check. There is a fairly simple way to achieve this though: add a trivial job that needs the matrix job.
jobs:
setup_matrix:
# ....
test:
# ....
dummy:
needs: test
runs-on: ubuntu-latest
name: Test suite
steps:
- run: echo Test suite completed
Now we can mark Test suite as a required check to be sure that the job test also passed.
There is the minor downside that it requires an additional job to run, which does cost a tiny bit of GitHub Action minutes, but it should be minimal and I think the convenience is worth it.
Of course, this can (and should) be put in a reusable workflow.
Conclusion
In this blog I've shown various techniques aimed at avoiding duplication of efforts. Rather than using large blobs of templated workflows the reusable workflows have many advantages. They allow you to quickly act on CI issues, without having to submit it to potentially 100s of repositories. In Vox Pupuli we have about 150 Puppet modules and have used gha-puppet to work around issues. Not only is your git log much cleaner, it also saves submitting pull requests. Fewer pull requests means a lower burden on your maintainers and your CI, which is great for your wallet and the planet.