Bootstrap Taskgraph in Decision Task¶
This how-to guide outlines the recommended way to bootstrap Taskgraph (and other task generation dependencies) in a Decision Task.
If you’d like to follow along, you can download the example
.taskcluster.yml
file.
Define Requirements¶
We’ll use a tool called pip-compile to help generate the
requirements.txt
that will contain our Decision task dependencies,
including Taskgraph itself.
Note
You may use other tools to create the requirements.txt
, such as
hashin. As long as package hashes are included.
First make sure you have pip-compile
installed:
pip install pip-tools
Next, create the requirements.in
file:
cd taskcluster
echo "taskcluster-taskgraph" > requirements.in
# This works best if you use the same Python as the one used in the Decision
# image (currently 3.6).
pip-compile --generate-hashes --output-file requirements.txt requirements.in
Note
You may add version specifiers to packages (e.g,
taskcluster-taskgraph==1.1.4
), but either way the actual version that
gets used will be pinned once we generate the requirements.txt
.
If you intend to create transforms that require additional packages, add them to
requirements.in
and re-generate the lock file.
Instruct run-task
to Install Requirements¶
The decision docker images provided by Taskgraph contain a script called run-task. This script acts as a wrapper around the command you’d like to run and handles some initial setup such as cloning repositories needed by the task.
When run-task
clones a repository, it can optionally install Python
dependencies that are defined within it. We accomplish this with a set of
environment variables and a command line argument. Let’s assume the repo is
called myrepo
.
In the Decision task definition (in .taskcluster.yml
), we first define an
env called REPOSITORIES
:
payload:
env:
REPOSITORIES: {$json: {myrepo: "My Repository"}}
This tells run-task
which repositories will need to be cloned. For this guide,
we’ll only be cloning our single repository.
Next we define some environment variables that will tell run-task
how to clone the
repo. Each environment variable is uppercase prefixed by the key you specified in the
REPOSITORIES
env:
payload:
env:
REPOSITORIES: {$json: {myrepo: "My Repository"}}
MYREPO_HEAD_REPOSITORY: <repository url>
MYREPO_HEAD_REF: <branch or ref to fetch>
MYREPO_HEAD_REV: <revision to checkout>
MYREPO_REPOSITORY_TYPE: git
MYREPO_PIP_REQUIREMENTS: taskcluster/requirements.txt
The last environment variable should point at the requirements.txt
file we
just created.
Finally, we pass in the --myrepo-checkout
flag to run-task
. The name of
the argument again depends on the key you defined in the REPOSITORIES
env:
payload:
command:
- /usr/local/bin/run-task
- '--myrepo-checkout=/builds/worker/checkouts/myrepo'
- ...
Now run-task
will both clone your repo, as well as install any packages
defined in taskcluster/requirements.txt
, ensuring Taskgraph is bootstrapped
and ready to go.
Upgrading Taskgraph¶
To upgrade Taskgraph to a newer version, re-generate the requirements.txt
file:
cd taskcluster
pip-compile --generate-hashes --output-file requirements.txt requirements.in
If you pinned the package to a specific version don’t forget to update
requirements.in
first.
Note
Taskgraph follows semantic versioning, so minor and patch versions should be fully backwards compatible.
Testing Pre-Release Versions of Taskgraph¶
Sometimes you may wish to test against pre-release revisions of Taskgraph, especially if you are working on changes to Taskgraph itself. This can be accomplished using pip’s version control support:
cd taskcluster
echo "taskcluster-taskgraph@git+https://github.com/taskcluster/taskgraph@refs/pull/123/head" > requirements.in
pip-compile --output-file requirements.txt requirements.in
Next edit your .taskcluster.yml
to disable hashing since pip does not
support hashes with url requirements:
payload:
env:
- PIP_DISABLE_REQUIRE_HASHES: 1
Note
Be sure to omit the --generate-hashes
argument to pip-compile
otherwise pip
will implicitly turn hashing back on.
This way you can push an experimental change to a PR and then install it in your repo’s decision task.