Bootstrap Taskgraph in Decision Task#
This how-to guide outlines the recommended way to bootstrap Taskgraph (and other task generation dependencies) in a Decision Task.
Using the Pre-Built Docker Images#
The simplest way to make Taskgraph available to your Decision task is to use Taskgraph’s pre-built Docker images under the mozillareleases/taskgraph Docker Hub repository.
In your Decision task definition in .taskcluster.yml
, add the following to the task’s
payload
section:
payload:
image: mozillareleases/taskgraph:decision-latest
This will ensure your Decision task always run with the latest version of Taskgraph. Of course, it’s best practice to pin your CI dependencies, so it’s recommended to pin to a specific version:
payload:
image: mozillareleases/taskgraph:decision-v7.3.0
While not required, it’s also best practice to specify the image hash:
payload:
image: mozillareleases/taskgraph:decision-v7.3.0@sha256:f954e54d427d08b18954a4ff4541d43c76c476d0738f341e4fa82a959939d3fa
The hash can be found on the Docker Hub page for the specific tag you are pinning to.
Using a Python Requirements File#
Sometimes you might have more complex generation logic that requires other
packages in addition to taskgraph
. In this case you’ll want to use a
requirements file.
Like before, we’ll use a pre-built Docker image provided by Taskgraph:
payload:
image: mozillareleases/taskgraph:run-task-v7.3.0
The only difference with this run-task
image, is that it only contains
the run-task
script necessary to bootstrap the repository and not the full
Taskgraph package. It leaves how to get Taskgraph bootstrapped into the task up
to you.
This section outlines the recommended way to do that.
Note
If you’d like to follow along, you can download the example
.taskcluster.yml
file.
Define Requirements#
We’ll use a tool called pip-compile to help generate the
requirements.txt
that will contain our Decision task dependencies,
including Taskgraph itself.
Note
You may use other tools to create the requirements.txt
, such as
hashin. As long as package hashes are included.
First make sure you have pip-compile
installed:
pip install pip-tools
Next, create the requirements.in
file. For this example, let’s pretend that
in addition to taskcluster-taskgraph
we also want to depend on
requests
:
cd taskcluster
cat << EOF > requirements.in
taskcluster-taskgraph
requests
EOF
# This works best if you use the same Python as the one used in the Decision
# image (currently 3.11).
pip-compile --generate-hashes --output-file requirements.txt requirements.in
Note
You may add version specifiers to packages (e.g,
taskcluster-taskgraph==7.3.0
), but either way the actual version that
gets used will be pinned once we generate the requirements.txt
.
If you end up creating transforms that require additional packages, add them to
requirements.in
and re-generate the lock file.
Instruct run-task
to Install Requirements#
The docker images provided by Taskgraph contain a script called run-task. This script acts as a wrapper around the command you’d like to run and handles some initial setup such as cloning repositories needed by the task.
When run-task
clones a repository, it can optionally install Python
dependencies that are defined within it. We accomplish this with a set of
environment variables and a command line argument. Let’s assume the repo is
called myrepo
.
In the Decision task definition (in .taskcluster.yml
), we first define an
env called REPOSITORIES
:
payload:
env:
REPOSITORIES: {$json: {myrepo: "My Repository"}}
This tells run-task
which repositories will need to be cloned. For this guide,
we’ll only be cloning our single repository.
Next we define some environment variables that will tell run-task
how to clone the
repo. Each environment variable is uppercase prefixed by the key you specified in the
REPOSITORIES
env:
payload:
env:
REPOSITORIES: {$json: {myrepo: "My Repository"}}
MYREPO_HEAD_REPOSITORY: <repository url>
MYREPO_HEAD_REF: <branch or ref to fetch>
MYREPO_HEAD_REV: <revision to checkout>
MYREPO_REPOSITORY_TYPE: git
MYREPO_PIP_REQUIREMENTS: taskcluster/requirements.txt
The last environment variable should point at the requirements.txt
file we
just created.
Finally, we pass in the --myrepo-checkout
flag to run-task
. The name of
the argument again depends on the key you defined in the REPOSITORIES
env:
payload:
command:
- /usr/local/bin/run-task
- '--myrepo-checkout=/builds/worker/checkouts/myrepo'
- ...
Now run-task
will both clone your repo, as well as install any packages
defined in taskcluster/requirements.txt
, ensuring Taskgraph is bootstrapped
and ready to go.
Upgrading Taskgraph#
To upgrade Taskgraph to a newer version, re-generate the requirements.txt
file:
cd taskcluster
pip-compile --generate-hashes --output-file requirements.txt requirements.in
If you pinned the package to a specific version don’t forget to update
requirements.in
first.
Note
Taskgraph follows semantic versioning, so minor and patch versions should be fully backwards compatible.
Testing Pre-Release Versions of Taskgraph#
Sometimes you may wish to test against pre-release revisions of Taskgraph, especially if you are working on changes to Taskgraph itself. This can be accomplished using pip’s version control support:
cd taskcluster
echo "taskcluster-taskgraph@git+https://github.com/taskcluster/taskgraph@refs/pull/123/head" > requirements.in
pip-compile --output-file requirements.txt requirements.in
Next edit your .taskcluster.yml
to disable hashing since pip does not
support hashes with url requirements:
payload:
env:
PIP_DISABLE_REQUIRE_HASHES: 1
Note
Be sure to omit the --generate-hashes
argument to pip-compile
otherwise pip
will implicitly turn hashing back on.
This way you can push an experimental change to a PR and then install it in your repo’s decision task.