Bootstrap Taskgraph in Decision Task

This how-to guide outlines the recommended way to bootstrap Taskgraph (and other task generation dependencies) in a Decision Task.

If you’d like to follow along, you can download the example .taskcluster.yml file.

Define Requirements

We’ll use a tool called pip-compile to help generate the requirements.txt that will contain our Decision task dependencies, including Taskgraph itself.

Note

You may use other tools to create the requirements.txt, such as hashin. As long as package hashes are included.

First make sure you have pip-compile installed:

pip install pip-tools

Next, create the requirements.in file:

cd taskcluster
echo "taskcluster-taskgraph" > requirements.in
# This works best if you use the same Python as the one used in the Decision
# image (currently 3.8).
pip-compile --generate-hashes --output-file requirements.txt requirements.in

Note

You may add version specifiers to packages (e.g, taskcluster-taskgraph==1.1.4), but either way the actual version that gets used will be pinned once we generate the requirements.txt.

If you intend to create transforms that require additional packages, add them to requirements.in and re-generate the lock file.

Instruct run-task to Install Requirements

The decision docker images provided by Taskgraph contain a script called run-task. This script acts as a wrapper around the command you’d like to run and handles some initial setup such as cloning repositories needed by the task.

When run-task clones a repository, it can optionally install Python dependencies that are defined within it. We accomplish this with a set of environment variables and a command line argument. Let’s assume the repo is called myrepo.

In the Decision task definition (in .taskcluster.yml), we first define an env called REPOSITORIES:

payload:
    env:
        REPOSITORIES: {$json: {myrepo: "My Repository"}}

This tells run-task which repositories will need to be cloned. For this guide, we’ll only be cloning our single repository.

Next we define some environment variables that will tell run-task how to clone the repo. Each environment variable is uppercase prefixed by the key you specified in the REPOSITORIES env:

payload:
    env:
        REPOSITORIES: {$json: {myrepo: "My Repository"}}
        MYREPO_HEAD_REPOSITORY: <repository url>
        MYREPO_HEAD_REF: <branch or ref to fetch>
        MYREPO_HEAD_REV: <revision to checkout>
        MYREPO_REPOSITORY_TYPE: git
        MYREPO_PIP_REQUIREMENTS: taskcluster/requirements.txt

The last environment variable should point at the requirements.txt file we just created.

Finally, we pass in the --myrepo-checkout flag to run-task. The name of the argument again depends on the key you defined in the REPOSITORIES env:

payload:
    command:
        - /usr/local/bin/run-task
        - '--myrepo-checkout=/builds/worker/checkouts/myrepo'
        - ...

Now run-task will both clone your repo, as well as install any packages defined in taskcluster/requirements.txt, ensuring Taskgraph is bootstrapped and ready to go.

Upgrading Taskgraph

To upgrade Taskgraph to a newer version, re-generate the requirements.txt file:

cd taskcluster
pip-compile --generate-hashes --output-file requirements.txt requirements.in

If you pinned the package to a specific version don’t forget to update requirements.in first.

Note

Taskgraph follows semantic versioning, so minor and patch versions should be fully backwards compatible.

Testing Pre-Release Versions of Taskgraph

Sometimes you may wish to test against pre-release revisions of Taskgraph, especially if you are working on changes to Taskgraph itself. This can be accomplished using pip’s version control support:

cd taskcluster
echo "taskcluster-taskgraph@git+https://github.com/taskcluster/taskgraph@refs/pull/123/head" > requirements.in
pip-compile --output-file requirements.txt requirements.in

Next edit your .taskcluster.yml to disable hashing since pip does not support hashes with url requirements:

payload:
   env:
       - PIP_DISABLE_REQUIRE_HASHES: 1

Note

Be sure to omit the --generate-hashes argument to pip-compile otherwise pip will implicitly turn hashing back on.

This way you can push an experimental change to a PR and then install it in your repo’s decision task.