Matrix Builds with GitHub Actions

Introduction

Matrix builds are a feature of GitHub Actions that allow you to run multiple jobs in parallel across different configurations, such as operating systems, language versions, and architectures. This allows you to test various configurations easily without redundant workflows.

In this blog, we aim to provide a comprehensive guide on implementing matrix builds in GitHub Actions. You’ll be able to optimize your workflows and improve testing coverage, leading to faster and more reliable software.

Understanding Matrix Builds

What are Matrix Builds?

Matrix builds in GitHub Actions allow developers to run jobs multiple times with different configurations, streamlining the testing process across various environments. This is particularly useful for testing applications across various environments, ensuring that code functions correctly regardless of the operating system, programming language version, or other variables. By defining a matrix of parameters—like OS types and versions; GitHub Actions automatically creates and executes a job for each combination of these parameters. This eliminates the need for duplicate workflows, enhancing efficiency and coverage.

In many ways, matrix builds can be compared to a "for loop" in programming. Just as a for loop iterates over a set of values to execute a block of code multiple times, matrix builds iterate over defined configurations to run jobs concurrently.

For example, if you have a web application that needs to be tested on both Windows and macOS with different versions of Node.js, matrix builds will handle this efficiently. Instead of writing separate workflows for each configuration, you can define a single matrix that includes all necessary combinations. This not only saves time but also simplifies the workflow management process, allowing developers to focus on writing code rather than managing multiple test setups.

Setting Up Matrix Builds in GitHub Actions

Let’s consider a real-world scenario where we want to test a web application across multiple operating systems and Node.js versions.

Imagine you have a web application that must be compatible with various environments. You want to ensure it works on:

Operating Systems: Ubuntu, Windows, and macOS
Node.js Versions: 12.x, 14.x, and 16.x

Here’s how you would configure your workflow file to achieve this:

1. Create a Workflow File:
Start by creating a YAML file in your repository under the .github/workflows directory. This file will define your CI/CD pipeline.

├── .github
│   └── workflows
│       └── ci.yaml // Define your workflow here
├── src
│   └── app
├── tests
│   └── app.test.js
└── package.json

2. Define the Jobs Section: In your workflow file, define the jobs section. Each job represents a task that GitHub Actions will execute.

3. Set Up the Strategy: You need to specify the strategy block within each job. This is where you will define how the jobs will run in parallel based on different configurations.

4. Define the Matrix: Under the strategy block, use the matrix key to specify the parameters you want to test across different configurations. Each parameter can have multiple values, and GitHub Actions will create a job for each combination.

Here’s an example of what this looks like in a workflow file:

name: CI

on:
  push:
    branches:
      - main

jobs:
  build:
    runs-on: ${{ matrix.runner }}
    strategy:
      matrix:
        runner: [blacksmith-4vcpu-ubuntu-2204, windows-latest, macos-latest]
        node-version: [12.x, 14.x, 16.x]

    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Set up Node.js
        uses: actions/setup-node@v2
        with:
          node-version: ${{ matrix.node-version }}

      - name: Install dependencies
        run: npm install

      - name: Run tests
        run: npm test

In this example:

jobs: This section defines a job named build.
strategy: This block specifies how to run the job with different configurations.
matrix: Here, we define two parameters: os (operating systems) and node-version. Each value specified in these arrays will create parallel jobs for every combination.

This configuration will create 9 jobs:

Running on blacksmith-4vcpu-ubuntu-2204 with Node.js 12.x
Running on blacksmith-4vcpu-ubuntu-2204 with Node.js 14.x
Running on blacksmith-4vcpu-ubuntu-2204 with Node.js 16.x
Running on windows-latest with Node.js 12.x
Running on windows-latest with Node.js 14.x
Running on windows-latest with Node.js 16.x
Running on macos-latest with Node.js 12.x
Running on macos-latest with Node.js 14.x
Running on macos-latest with Node.js 16.x

5. Push Your Changes:

After committing your changes and pushing them to GitHub, navigate to the "Actions" tab of your repository. You’ll see that GitHub Actions has automatically created jobs for each combination of operating system and Node version specified in your matrix.

6. Monitor Job Execution:

Each job will run in parallel, allowing you to quickly identify any compatibility issues across different environments. For instance, if there are problems with Node.js 14 on Windows but not on Ubuntu, you'll be able to identify this quickly.

Other Use Cases for Matrix Builds

In addition to testing across various language version (Node.js 12.x, Node.js 14.x, Node.js 16.x) and running builds on different operating systems (testing on windows, macOS, Linux.), matrix builds are can be applied to a range of scenarios related to testing. Below are some common use cases, each accompanied by practical examples: This revision emphasizes the versatility of matrix builds while transitioning into the section discussing specific use cases.

1. Sharding Tests

Sharding tests is a technique used to split a large test suite into smaller chunks that can be executed in parallel. This reduces the overall testing time and allows for a faster feedback loop when you create a change.

Example:‍

name: CI

on:
  push:
    branches:
      - main

jobs:
  test:
    runs-on: blacksmith-4vcpu-ubuntu-2204
    strategy:
      matrix:
        shard: [1, 2, 3]

    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Run tests
        run: npm test -- --shard=${{ matrix.shard }}

Here, the tests are divided into three shards (1, 2, and 3). Each job runs a different shard of the test suite, allowing for parallel execution and faster completion.

2. Multi-Architecture Docker Builds

When building Docker images, it’s often necessary to ensure compatibility across different architectures (e.g., x86 and ARM). Matrix builds allow you to define jobs that target multiple architectures efficiently.

Example:

name: Build Docker Images

on:
  push:
    branches:
      - main
jobs:
  build:
    runs-on: ${{ matrix.runner }}
    strategy:
      matrix:
        platform: [amd64, arm64]
        include:
          - platform: amd64
            runner: blacksmith-2vcpu-ubuntu-2204
          - platform: arm64
            runner: blacksmith-2vcpu-ubuntu-2204-arm

    steps:
      - name: Checkout repository
        uses: actions/checkout@v3

      - name: Login to Docker Hub
        uses: docker/login-action@v2
        with:
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_PASSWORD }}

      - name: Build and push Docker image
        uses: docker/build-push-action@v2
        with:
          context: .
          file: Dockerfile
          push: true
          tags: my-app:${{ matrix.platform }}
          platforms: ${{ matrix.platform }}

The use of the exclude keyword allows you to skip unnecessary builds, while the include keyword can be used to add specific configurations without duplicating your workflow. This approach not only simplifies your setup but also enhances performance and ensures compatibility across systems. In the example above, the workflow builds Docker images for both linux/amd64 and linux/arm64 architectures. By using matrix builds, you can run the ARM build on blacksmith-2vcpu-ubuntu-2204-arm ARM runner and the AMD64 build on a blacksmith-2vcpu-ubuntu-2204 runner, eliminating the need for QEMU emulation.

By using matrix builds, you can ensure that your images are compatible with multiple systems without duplicating the workflow.

3. Testing Different Database Versions

When developing applications that interact with databases, ensuring compatibility with various database versions is essential. Matrix builds allow you to test your application against multiple database setups, ensuring that queries and functionalities work as expected across different environments.

Example:

name: CI

on:
  push:
    branches:
      - main

jobs:
  test:
    runs-on: blacksmith-4vcpu-ubuntu-2204
    strategy:
      matrix:
        db: [mysql:5.7, mysql:8.0, postgres:12, postgres:14]

    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Set up Database
        run: |
          docker run --name db -e MYSQL_ROOT_PASSWORD=root -d ${{ matrix.db }}

      - name: Run tests
        run: npm test

In this example, the workflow tests the application against multiple versions of MySQL and PostgreSQL databases to ensure compatibility.

4. Running Performance Tests

Matrix builds can be utilized to run performance tests under different configurations or load conditions. This is particularly useful for identifying how your application behaves under varying loads or resource limits.

Example:

name: Performance Tests

on:
  push:
    branches:
      - main

jobs:
  performance:
    runs-on: blacksmith-4vcpu-ubuntu-2204
    strategy:
      matrix:
        load: [100, 500, 1000] # Different load levels

    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Run performance tests
        run: |
          echo "Running performance tests with load level ${{ matrix.load }}"
          # Command to run performance tests goes here

‍This setup allows you to evaluate how well your application performs under different user loads.

5. Testing API Endpoints Across Versions

For applications that expose APIs, it's crucial to ensure that all endpoints function correctly across different versions of the API. Matrix builds can facilitate testing against multiple API versions simultaneously.

name: API Tests

on:
  push:
    branches:
      - main

jobs:
  api-test:
    runs-on: blacksmith-4vcpu-ubuntu-2204
    strategy:
      matrix:
        api-version: [v1, v2, v3]

    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Run API tests
        run: |
          echo "Testing API version ${{ matrix.api-version }}"
          # Command to run API tests goes here

This example ensures that all specified API versions are tested for functionality and compatibility.

6. Cross-Browser Testing for Web Applications

For web applications, it’s essential to verify that the application behaves consistently across different web browsers. Matrix builds can help automate this process by running tests in various browser environments.

Example:

name: Cross-Browser Testing

on:
  push:
    branches:
      - main

jobs:
  browser-test:
    runs-on: blacksmith-4vcpu-ubuntu-2204
    strategy:
      matrix:
        browser: [chrome, firefox, safari]

    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Run browser tests
        run: |
          echo "Running tests on ${{ matrix.browser }}"
          # Command to run browser-specific tests goes here

This setup allows you to ensure that your web application functions correctly across popular browsers.

Benefits of Using Matrix Builds

Efficiency Gains

One of the most compelling reasons to use matrix builds is the substantial efficiency gains they provide. By running multiple jobs in parallel, you can significantly reduce the total time required for testing. Instead of executing tests sequentially, which can be time-consuming, matrix builds allow you to run jobs for different configurations simultaneously.

In the example we saw in the section above when you need to test your application across three operating systems and three versions of a programming language, a sequential approach could take a considerable amount of time. However, with matrix builds, all combinations can be executed at once. This parallel execution not only speeds up the testing process but also enables faster feedback on code changes, allowing developers to address issues more quickly.

Increased Test Coverage

Matrix builds also enhance test coverage by ensuring that your application is validated across various environments and configurations and can catch regressions. This is crucial for identifying compatibility issues that may arise in different setups.

By covering a wider range of scenarios, you increase confidence that your application will function correctly in real-world conditions. This thorough testing helps catch bugs early in the development cycle, ultimately leading to a more stable and reliable product.

Simplified Workflow Management

Another significant benefit of using matrix builds is the simplification they bring to workflow management. By defining a matrix strategy, you can eliminate redundancy in your workflow files. Instead of creating separate jobs for each configuration, you can define them all in one place using a matrix.

This not only makes your workflow files cleaner and easier to read but also simplifies maintenance. If you need to add or remove configurations, you can do so by updating just one section of your workflow file rather than modifying multiple job definitions. This streamlined approach reduces the likelihood of errors and makes it easier for teams to collaborate on CI/CD processes.

Advanced Features of Matrix Builds

Matrix builds in GitHub Actions come with several advanced features that enhance their functionality and flexibility. Here’s an in-depth look at these features and how they can be utilized effectively.

Dynamic Matrix Generation

Dynamic matrix generation allows you to create matrices based on the outputs of previous jobs or specific conditions. This feature is particularly useful for complex workflows where the configurations needed for testing may change based on earlier job results.

To implement dynamic matrix generation, you can use the set-output command to define a matrix dynamically. For example, you might have a build job that determines which versions of a dependency are compatible, and then pass those versions to a subsequent test job.

Here’s a simplified example:

jobs:
  build:
    runs-on: blacksmith-4vcpu-ubuntu-2204
    steps:
      - name: Determine compatible versions
        id: version-check
        run: |
          echo "::set-output name=matrix::$(echo '[{"version": "1.0"}, {"version": "2.0"}]')"

  test:
    needs: build
    runs-on: ubuntu-latest
    strategy:
      matrix:
        version: ${{ fromJson(needs.build.outputs.matrix) }}

    steps:
      - name: Run tests
        run: npm test --version ${{ matrix.version }}

In this example, the build job determines which versions are compatible and sets them as output. The test job then uses that output to create a matrix of tests to run, allowing for more flexible and responsive workflows.

Excluding and Including Configurations

When working with matrix builds, you may encounter situations where certain combinations of configurations are not necessary or even incompatible. GitHub Actions provides the

exclude and include keywords to refine your job executions.

Excluding Configurations: This feature allows you to specify combinations that should not be executed. For instance, if you want to exclude testing Node.js version 14.x on Windows due to known issues, you can do so as follows:
```
strategy:
  matrix:
    os: [ubuntu-latest, windows-latest]
    node-version: [12.x, 14.x, 16.x]
    exclude:
      - os: windows-latest
        node-version: 14.x
```

Including Additional Configurations: Conversely, you can add specific configurations that are not part of the standard matrix combinations using the include keyword. This is useful when you want to test an experimental feature or a special case without duplicating your entire workflow:
```
strategy:
  matrix:
    os: [ubuntu-latest, windows-latest]
    node-version: [12.x, 14.x]
    include:
      - os: macos-latest
        node-version: 16.x
```

This approach enhances your control over which jobs are executed and ensures that your testing is both comprehensive and relevant.

Fail-Fast Behavior and Control

The fail-fast strategy is a built-in behavior in GitHub Actions that automatically cancels all running jobs in a matrix if one job fails. This can save time and resources by preventing unnecessary executions once a failure has been identified, ultimately helping to reduce CI costs. However, there are scenarios where you might want to disable this behavior to gather more data from all jobs. By halting further job executions when a failure occurs, teams can avoid wasting compute resources on jobs that are likely to fail due to an identified issue, leading to more efficient resource management.

To disable fail-fast behavior, you can set it explicitly in your workflow:

strategy:
  fail-fast: false
  matrix:
    os: [ubuntu-latest, windows-latest]
    node-version: [12.x, 14.x]

In this configuration, even if one job fails (e.g., Node.js 12.x on Windows), all other jobs will continue to run. This is particularly useful for debugging or when you need results from all configurations for analysis.

Managing Concurrency with Max Parallel Jobs

Managing resource consumption during job execution is crucial, especially when using self-hosted runners or when your CI/CD environment has limited resources. The max-parallel key allows you to control how many jobs run concurrently within a matrix.

For example, if you want to limit the number of concurrent jobs to two at any given time, you can configure it like this:

strategy:
  max-parallel: 2
  matrix:
    os: [ubuntu-latest, windows-latest]
    node-version: [12.x, 14.x]

With this setup, even if there are multiple combinations in the matrix (e.g., four total jobs), only two will run simultaneously. This helps manage the load on your CI/CD infrastructure and can lead to more stable builds by avoiding resource contention. However, if you're using Blacksmith's runners (blacksmith-2vcpu-ubuntu-2204 ), you won’t need to worry about these concurrency limits. Blacksmith has no concurrency limits and can scale with your demand, whether it’s hundreds of jobs or thousands of vCPUs.

Best Practices for Optimizing Matrix Builds

Optimizing your matrix builds in GitHub Actions can lead to significant improvements in build times, resource utilization, and overall workflow efficiency. Here are some best practices to consider:

Efficient Resource Utilization

One of the key advantages of using matrix builds is the ability to run multiple jobs in parallel. However, to maximize this benefit, it's essential to utilize resources efficiently. Here are some tips for caching dependencies and speeding up build times:

Use Caching Actions: Leverage GitHub’s caching capabilities to store dependencies between workflow runs. For example, if you’re using Node.js, you can cache the node_modules directory:

- name: Cache node_modules
  uses: useblacksmith/cache@v5
  with:
    path: node_modules
    key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}

This action checks if the cache exists based on the specified key. If it does, it restores the cached dependencies, significantly reducing installation time.

Cache Build Artifacts: If your build process generates artifacts (like compiled binaries), consider caching these as well. This can save time in subsequent builds where those artifacts don’t need to be rebuilt.
Optimize Dependency Installation: Use tools like npm ci instead of npm install for faster installations when working with Node.js projects. The ci command installs dependencies based on the package-lock.json, which is generally faster and more reliable in CI environments.

Conditional Steps for Custom Workflows

Implementing conditional logic within your matrix builds can help streamline your workflows by skipping unnecessary steps based on specific conditions or matrix values. This not only saves time but also reduces resource consumption.

Using if Conditions: You can use the if keyword to control whether a step runs based on the current matrix configuration. For example, if you want to skip deployment steps for certain Node.js versions, you can do so like this‍‍
```
- name: Deploy to Production
  if: matrix.node-version == '14.x'
  run: |
    echo "Deploying..."
```
In this case, the deployment step will only execute if the current job is using Node.js version 14.x.

Skip Steps Based on OS: Similarly, you can skip steps based on the operating system being used in the matrix:‍‍
```
- name: Run Windows-Specific Tests
  if: matrix.os == 'windows-latest'
  run: npm run test:windows
```

By implementing these conditional steps, you can tailor your workflows to be more efficient and relevant to the specific configurations being tested.

Artifact Management Strategies

Handling artifacts efficiently within matrix jobs is crucial for maintaining a clean workflow and ensuring that necessary files are available for subsequent jobs or deployments. Here are some best practices:

Use Artifacts Wisely: Only upload artifacts that are necessary for later jobs or deployments. This minimizes storage usage and keeps your workflow organized.‍‍
```
- name: Upload Artifacts
  uses: actions/upload-artifact@v2
  with:
    name: build-artifacts
    path: ./dist/
```

In this example, only the contents of the ./dist/ directory are uploaded as artifacts, keeping things streamlined.

Download Artifacts Efficiently: If your workflow involves multiple jobs that depend on artifacts from previous jobs, ensure that you download them efficiently using the actions/download-artifact action:‍- name: Download Artifacts uses: actions/download-artifact@v2 with: name: build-artifactsThis allows subsequent jobs to access necessary files without duplicating work.

Clean Up Unused Artifacts: Regularly review and clean up old artifacts that are no longer needed. This helps manage storage limits and keeps your repository tidy.
- To maintain an efficient CI/CD pipeline, consider implementing a strategy for artifact retention:
  - Set Retention Policies: GitHub Actions allows you to specify a retention period for artifacts. By default, artifacts are retained for 90 days, but you can adjust this setting based on your project needs. For example, if you only need to keep artifacts for a week, you can set a shorter retention period to automatically delete them after that time.
```
- name: Upload Artifacts
  uses: actions/upload-artifact@v2
  with:
    name: build-artifacts
    path: ./dist/
    retention-days: 7  # Set retention to 7 days
```
  - Automate Cleanup: If your project generates a large number of artifacts over time, consider setting up a scheduled job that runs periodically to delete old artifacts. This can be done using GitHub's API to list and delete artifacts based on their creation date or other criteria.
  - Review Artifact Usage: Periodically assess which artifacts are being used in your workflows. If certain artifacts are rarely accessed or no longer relevant, consider removing them from your storage to free up space.

Conclusion

In conclusion, leveraging matrix builds in GitHub Actions significantly enhances testing efficiency and overall workflow management. Being able to run multiple jobs in parallel across various configurations not only reduces the time required for testing but also simplifies workflow files by eliminating redundancy. In addition, with features like dynamic matrix generation and the ability to include or exclude specific configurations, teams can adapt their CI/CD processes to meet evolving needs.

Thank you! We will notify you when we publish new posts!

Oops! Something went wrong while submitting the form.

Matrix Builds with GitHub Actions

Recent Posts

How The Economics of Multitenancy Work

Blacksmith Raises $3.5M led by GV and Y Combinator to Build a High-Performance CI Cloud.

You have 5 days before the new DockerHub limits f*ck you over.

Recent Guides

Best Practices for Managing Secrets in GitHub Actions

How to reduce spend in GitHub Actions

Docker Architecture: The components and processes - Part 2

Matrix Builds with GitHub Actions

Get notified about new posts

Recent Posts

How The Economics of Multitenancy Work

Blacksmith Raises $3.5M led by GV and Y Combinator to Build a High-Performance CI Cloud.

You have 5 days before the new DockerHub limits f*ck you over.

Recent Guides

Best Practices for Managing Secrets in GitHub Actions

How to reduce spend in GitHub Actions

Docker Architecture: The components and processes - Part 2