Matrix Transforms#

The taskgraph.transforms.matrix transforms can be used to split a base task into many subtasks based on a defined matrix.

These transforms are useful if you need to have many tasks that are very similar except for some small configuration differences.

Usage#

Add the transform to the transforms key in your kind.yml file:

transforms:
  - taskgraph.transforms.matrix
  # ...

Then create a matrix section in your task definition, e.g:

tasks:
  test:
    matrix:
      os: ["win", "mac", "linux"]

    # rest of task definition

This will split the test task into three; test-win, test-mac and test-linux.

Matrix with Multiple Rows#

You can add as many rows as you like to the matrix, and every combination of tasks will be generated. For example, the following matrix:

tasks:
  test:
    matrix:
      os: ["win", "mac", "linux"]
      python: ["py312", "py311"]

    # rest of task definition

Will generate these tasks:

test-win-py312
test-win-py311
test-mac-py312
test-mac-py311
test-linux-py312
test-linux-py311

Note that the name of the tasks will be built based on the order of rows in the matrix.

Substituting Matrix Context into the Task Definition#

Of course these tasks will be identical, so you’ll want to change other parts of the task definition based on the matrix values.

Substituting Values in Yaml#

The simplest way to change a matrix task’s definition, is to use the built-in yaml substitution:

tasks:
  test:
    matrix:
      os: ["win", "mac", "linux"]
    description: Run {matrix[os]} tests
    worker-type: {matrix[os]}-worker

Limiting Substitution#

By default, all keys and values in the task definition will be checked for substitution parameters. But in some cases, it might be desirable to limit which keys get substituted, such as when using the matrix transforms alongside other transforms that perform substitution, such as the task_context or chunking transforms.

To limit the fields that will be evaluated for substitution, you can pass in the substitution-fields config:

tasks:
  test:
    matrix:
      substitution-fields: ["worker-type"]
      os: ["win"]
    description: Run {matrix[os]} tests
    worker-type: {matrix[os]}-worker

In the example above, worker-type will evaluate to win-worker, whereas the description will be the literal string Run {matrix[os]} tests. Dot notation can be used in substitution-fields to limit substitution to some sub configuration of the task definition.

Substituting Values in a Later Transform#

For more advanced cases, you may wish to use a later transform to act on the result of the matrix evaluation. To accomplish this, the matrix transforms will set a matrix attribute that contains all matrix values applicable to the task.

For example, let’s say you have a kind.yml like:

transforms:
  - taskgraph.transforms.matrix
  - custom_taskgraph.transforms.custom
  # ...

tasks:
  test:
    matrix:
      os: ["win", "mac", "linux"]

Then in your custom.py transform file, you could add:

@transforms.add
def set_worker_type_and_description(config, tasks):
    for task in tasks:
        matrix = task["attributes"]["matrix"]
        task["description"] = f"Run {matrix['os']} tests"
        task["worker-type"] = f"{matrix['os']}-worker"
        yield task

This example will yield the exact same result as the yaml example above, but it allows for more complex logic.

Excluding Matrix Combinations#

Sometimes you might not want to generate every possible combination of tasks, and there may be some you wish to exclude. This can be accomplished using the exclude config:

tasks:
  test:
    matrix:
      os: ["win", "mac"]
      arch: ["x86", "arm64"]
      python: ["py312", "py311"]
      exclude:
        - os: mac
          arch: x86
        - os: win
          arch: arm64
          python: py311

This will cause all combinations where os == mac and arch == x86 to be skipped, as well as the specific combination where os == win and arch == arm64 and python == py311. This means the following tasks will be generated:

test-win-x86-py311

test-win-x86-py312

test-win-arm64-py312

test-mac-arm64-py311

test-mac-arm64-py312

Customizing Task Names#

By default, the matrix transforms will append each matrix value to the task’s name, separated by a dash. If some other format is desired, you can specify the set-name config:

tasks:
  test:
    matrix:
      set-name: "test-{matrix[os]}/{matrix[python]}"
      os: ["win"]
      python: ["py312"]

Instead of creating a task with the name test-win-py312, the name will be test-win/py312.

Matrix Transforms

Contents