Creating a Simple Task Graph#
This tutorial will walk you through the process of creating a simple task graph.
We’ll create the scaffolding for Taskgraph and set up a single “Hello World” task. Once this is working, you will have the tools and knowledge to start adding your own tasks, and modifying configuration to meet your repository’s needs.
For the purposes of this tutorial, we’ll assume your repository is called
myrepo
.
Create the File System Layout#
Create a file system layout similar to:
taskcluster
├── config.yml
├── kinds
│ └── hello
| └── kind.yml
└── myrepo_taskgraph
└── transforms
└── hello.py
The taskcluster
directory should appear at the root of the repository. It
is possible to rename / put it somewhere else, but you’ll need to jump through
a couple extra steps in that case. The location of the transforms file is
similarly only a convention (but it must be importable).
Populate the Config File#
The config file lives at taskcluster/config.yml
and must conform to the
graph_config_schema
.
Create the config.yml
file to look like:
---
trust-domain: myrepo
task-priority: low
workers:
# Values here depend on your Taskcluster instance's configuration.
aliases:
linux:
provisioner: '{trust-domain}-provisioner'
implementation: docker-worker
os: linux
worker-type: '{trust-domain}-linux'
taskgraph:
repositories:
myrepo:
name: myrepo
Here’s an explanation of the required top-level keys:
trust-domain
- Trust domains help prevent permissions and worker pools from one project from being re-used in another. The value you should use here will be configured by your Taskcluster administrator. But we’ll usemyrepo
for the purposes of this tutorial.task-priority
- The priority of tasks for this repo.workers
- The set of worker pools available to the project. This will also be configured by your Taskcluster administrator.taskgraph
- A dict containing some miscellaneous configuration needed by Taskgraph.
Define a Task#
Kinds are groupings of tasks that share certain characteristics
with one another. Each subdirectory in taskcluster/kinds
corresponds to a
different kind and contains a kind.yml
file. These files define some
properties that control how Taskgraph will generate the tasks, as well as the
starting definitions of the tasks themselves. If you followed the layout above,
you have a hello
kind. For this next section we’ll be editing
taskcluster/kinds/hello/kind.yml
.
Declare the set of transforms that will be applied to tasks. By default, taskgraph will include the
-taskgraph.transforms.run
and-taskgraph.transforms.task
transforms, which are used by the vast majority of simple tasks. It is also quite common for kind-specific transforms to be used, which we will do here for the purpose of demonstration. In our example:transforms: - myrepo_taskgraph.transforms.hello:transforms
Finally we define the task under the
tasks
key. The format for the initial definition here can vary wildly from one kind to another, it all depends on the transforms that are used. It’s conventional for transforms to define a schema (but not required). So often you can look at the first transform file to see what schema is expected of your task. But since we haven’t created the first transforms yet, let’s define our task like this for now:tasks: taskcluster: description: "Says hello to Taskcluster" text: "Taskcluster!"
Here is the combined kind.yml
file:
transforms:
- myrepo_taskgraph.transforms.hello:transforms
tasks:
taskcluster:
description: "Says hello to Taskcluster"
text: "Taskcluster!"
Create the Transform#
Transforms are Python generators that take a
-taskgraph.transforms.base.TransformConfig
instance and a generator
that yields task definitions (in dictionary form) as input. It yields task
definitions (which may or may not be modified) from the original inputs.
Typically transform files contain a schema, followed by one or more transform
functions. Rather than break it down step by step, here’s what our
taskcluster/myrepo_taskgraph/transforms/hello.py
file will look like (see
comments for explanations):
from voluptuous import Optional, Required
from taskgraph.transforms.base import TransformSequence
from taskgraph.util.schema import Schema
# Define the schema. We use the `voluptuous` package to handle validation.
hello_description_schema = Schema({
Required("text"): str,
Optional("description"): str,
})
# Create a 'TransformSequence' instance. This class collects transform
# functions to run later.
transforms = TransformSequence()
# First let's validate tasks against the schema.
transforms.add_validate(hello_description_schema)
# Register our first transform functions via decorator.
@transforms.add
def set_command(config, tasks):
"""Builds the command the task will run."""
for task in tasks:
task["command"] = f"bash -cx 'echo Hello {task.pop('text')}'"
yield task
@transforms.add
def build_task_description(config, tasks):
"""Sets the attributes required by transforms in
`taskgraph.transforms.task`"""
for task in tasks:
if "description" not in task:
task["description"] = f"Says Hello {task['text']}"
task["label"] = f"{config.kind}-{task.pop('name')}"
# This is what was defined in `taskcluster/config.yml`.
task["worker-type"] = "linux"
task["worker"] = {
"command": task.pop["command"],
"docker-image": "ubuntu:latest",
"max-run-time": 300, # seconds
}
yield task
Generate the Taskgraph#
Now it’s time to see if everything works! If you haven’t done so already, follow the Installation docs to install Taskgraph.
Next run the following command at the root of your repo:
taskgraph full
If all goes well, you should see some log output followed by a single task
called hello-taskcluster
. Try adding a second task to your tasks
key
in the kind.yml
file and re-generating the graph. You should see both
task labels!
Now run:
taskgraph morphed -J
The -J/--json
flag will display the full JSON definition of your task.
Morphed is the final phase of graph generation, so
represents your task’s final form before it would get submitted to Taskcluster.
In fact, if we hadn’t made up the trust domain and worker pool in
config.yml
, you could even copy / paste this definition into Taskcluster’s
task creator!
Next you can check out the Submitting your Graph to Taskcluster tutorial or learn more about generating the taskgraph locally.