From Dependencies#
The taskgraph.transforms.from_deps
transforms can be used to create
tasks based on the kind dependencies, filtering on common attributes like the
build-type
.
These transforms are useful when you want to create follow-up tasks for some indeterminate subset of existing tasks. For example, maybe you want to run a signing task after each build task.
Usage#
Add the transform to the transforms
key in your kind.yml
file:
transforms:
- taskgraph.transforms.from_deps
# ...
Then create a from-deps
section in your task definition, e.g:
kind-dependencies:
- build
- toolchain
tasks:
signing:
from-deps:
kinds: [build]
This example will split the signing
task into many, one for each build. If
kinds
is unspecified, then it defaults to all kinds listed in the
kind-dependencies
key. So the following is valid:
kind-dependencies:
- build
- toolchain
tasks:
signing:
from-deps: {}
In this example, a task will be created for each build and each toolchain.
Limiting Dependencies by Attribute#
It may not be desirable to create a new task for all tasks in the given kind. It’s possible to limit the tasks given some attribute:
kind-dependencies:
- build
tasks:
signing:
from-deps:
with-attributes:
platform: linux
In the above example, follow-up tasks will only be created for builds whose
platform
attribute equals “linux”. Multiple attributes can be specified,
these are resolved using the attrmatch()
utility function.
Grouping Dependencies#
Sometimes, it may be desirable to run a task after a group of related tasks rather than an individual task. One example of this, is a task to notify when a release is finished for each platform we’re shipping. We want it to depend on every task from that particular release phase.
To accomplish this, we specify the group-by
key:
kind-dependencies:
- build
- signing
- publish
tasks:
notify:
from-deps:
group-by:
attribute: platform
In this example, tasks across the build
, signing
and publish
kinds will
be scanned for an attribute called “platform” and sorted into corresponding groups.
Assuming we’re shipping on Windows, Mac and Linux, it might create the following
groups:
- build-windows, signing-windows, publish-windows
- build-mac, signing-mac, publish-mac
- build-linux, signing-linux, publish-linux
Then the notify
task will be duplicated into three, one for each group. The
notify tasks will depend on each task in its associated group.
Custom Grouping#
Only the default single
and the attribute
group-by functions are
built-in. But if more complex grouping is needed, custom functions can be
implemented as well:
from typing import List
from taskgraph.task import Task
from taskgraph.transforms.from_deps import group_by
from taskgraph.transforms.base import TransformConfig
@group_by("custom-name")
def group_by(config: TransformConfig, tasks: List[Task]) -> List[List[Task]]:
pass
This can then be used in a task like so:
from-deps:
group-by: custom-name
It’s also possible to specify a schema for your custom group-by function, which
allows tasks to pass down additional context (such as with the built-in
attribute
function):
from typing import List
from taskgraph.task import Task
from taskgraph.transforms.from_deps import group_by
from taskgraph.transforms.base import TransformConfig
from taskgraph.util.schema import Schema
@group_by("custom-name", schema=Schema(str))
def group_by(config: TransformConfig, tasks: List[Task], ctx: str) -> List[List[Task]]:
pass
The extra context can be passed by turning group-by
into an object
instead of a string:
from-deps:
group-by:
custom-name: foobar
In the above example, the value foobar
is what must conform to the schema defined
by the group_by
function.
Unique Kinds#
By default, each group can contain only a single task from a given kind. I.e, a group can contain a build task and a signing task, but not two build tasks. This is enforced at task generation time. This is typically the desired behaviour and the check is in place to prevent mistakes.
However, in some cases it may be desirable to depend on multiple tasks of the same
kind (e.g, if implementing a notify
task). In this case it’s possible to specify
unique-kinds
:
tasks:
notify:
from-deps:
unique-kinds: false
group-by: custom
This will disable the uniqueness check and switch dependency edge names to the
dependency’s label rather than its kind. Now the notify
task can be used
with a custom group-by function that returns more than one kind per group.
Primary Kind#
Each task has a primary-kind
. This is the kind dependency in each grouping
that comes first in the list of supported kinds (either via the
kind-dependencies
in the kind.yml
file, or via the from-deps.kinds
key). Note that depending how the dependencies get grouped, a given group may
not contain a dependency for each kind. Therefore the list of kind dependencies
are ordered by preference. E.g, kinds earlier in the list will be chosen as the
primary kind before kinds later in the list.
The primary kind is used to derive the task’s label, as well as copy attributes
if the copy-attributes
key is set to True
(see next section).
Each task created by the from_deps
transforms, will have a
primary-kind-dependency
attribute set.
Copying Attributes#
It’s often useful to copy attributes from a dependency. When this key is set to True
,
all attributes from the primary-kind
(see above) will be copied over to the task. If
the task contains pre-existing attributes, they will not be overwritten.