Taskgraph is a Python library to generate DAGs of tasks for Taskcluster, the task execution framework which underpins Mozilla’s CI. The nodes of the DAG represent tasks, while the edges represent the dependencies between them.

Taskgraph is designed to scale to any level of complexity. From a handful of tasks, to the over 30,000 tasks and counting that make up Firefox’s CI.


Taskgraph is on Pypi and can be installed via:

pip install taskcluster-taskgraph

This provides the taskgraph binary, see taskgraph --help for available commands. To integrate Taskgraph in your project, see Creating a Simple Task Graph.

Alternatively you can install it by cloning the repo. This is useful if you need to test against a specific revision:

git clone https://github.com/taskcluster/taskgraph
cd taskgraph
python setup.py develop

How It Works#

Taskcluster tasks are defined in the tasks section of a .taskcluster.yml file at the repository root. This is adequate for simple CI systems which only have a few tasks. But as the complexity of a CI system grows, trying to fit all task definitions into a single file becomes difficult to maintain.

Instead of defining everything in the .taskcluster.yml, Taskgraph consumers define only a single task called the Decision Task. This task runs the command taskgraph decision which at a very high level:

  1. Reads a set of input YAML files defining tasks.

  2. Runs each task definition through a set of transforms.

  3. Submits the resulting DAG of tasks (with Decision task as the root) to Taskcluster via its REST API.

Taskgraph’s combination of static configuration with logic layered on top, allows a project’s CI to grow to arbitrary complexity.

Table of Contents#

Indices and tables#