From Dependencies Transforms#

The taskgraph.transforms.from_deps transforms can be used to create tasks based on the kind dependencies, filtering on common attributes like the build-type.

These transforms are useful when you want to create follow-up tasks for some indeterminate subset of existing tasks. For example, maybe you want to run a signing task after each build task.

Schema#

All tasks must conform to the from_deps schema.

Usage#

Add the transform to the transforms key in your kind.yml file:

transforms:
  - taskgraph.transforms.from_deps
  # ...

Then create a from-deps section in your task definition, e.g:

kind-dependencies:
  - build
  - toolchain

tasks:
  signing:
    from-deps:
      kinds: [build]

This example will split the signing task into many, one for each build. If kinds is unspecified, then it defaults to all kinds listed in the kind-dependencies key. So the following is valid:

kind-dependencies:
  - build
  - toolchain

tasks:
  signing:
    from-deps: {}

In this example, a task will be created for each build and each toolchain.

Limiting Dependencies by Attribute#

It may not be desirable to create a new task for all tasks in the given kind. It’s possible to limit the tasks given some attribute:

kind-dependencies:
  - build

tasks:
  signing:
    from-deps:
      with-attributes:
        platform: linux

In the above example, follow-up tasks will only be created for builds whose platform attribute equals “linux”. Multiple attributes can be specified, these are resolved using the attrmatch() utility function.

Grouping Dependencies#

Sometimes, it may be desirable to run a task after a group of related tasks rather than an individual task. One example of this, is a task to notify when a release is finished for each platform we’re shipping. We want it to depend on every task from that particular release phase.

To accomplish this, we specify the group-by key:

kind-dependencies:
  - build
  - signing
  - publish

tasks:
  notify:
    from-deps:
      group-by:
        attribute: platform

In this example, tasks across the build, signing and publish kinds will be scanned for an attribute called “platform” and sorted into corresponding groups. Assuming we’re shipping on Windows, Mac and Linux, it might create the following groups:

- build-windows, signing-windows, publish-windows
- build-mac, signing-mac, publish-mac
- build-linux, signing-linux, publish-linux

Then the notify task will be duplicated into three, one for each group. The notify tasks will depend on each task in its associated group.

You may also provide a special value of all to the group-by function. Using all will always result in one task being generated, with all tasks from the included kinds to be set as dependencies. It is usually useful to also set unique-kinds to False when using all.

If we alter the kind definition from above as follows:

kind-dependencies:
  - build
  - signing
  - publish

tasks:
  notify:
    from-deps:
      group-by: all

We would end up with a single notify task that depends on all tasks from the build, signing, and publish kinds.

Custom Grouping#

Only the default single, all and the attribute group-by functions are built-in. But if more complex grouping is needed, custom functions can be implemented as well:

from typing import List

from taskgraph.task import Task
from taskgraph.transforms.base import TransformConfig
from taskgraph.util.dependencies import group_by

@group_by("custom-name")
def group_by(config: TransformConfig, tasks: List[Task]) -> List[List[Task]]:
   pass

This can then be used in a task like so:

from-deps:
  group-by: custom-name

It’s also possible to specify a schema for your custom group-by function, which allows tasks to pass down additional context (such as with the built-in attribute function):

from typing import List

from taskgraph.task import Task
from taskgraph.transforms.base import TransformConfig
from taskgraph.util.dependencies import group_by
from taskgraph.util.schema import LegacySchema

@group_by("custom-name", schema=LegacySchema(str))
def group_by(config: TransformConfig, tasks: List[Task], ctx: str) -> List[List[Task]]:
   pass

The extra context can be passed by turning group-by into an object instead of a string:

from-deps:
  group-by:
    custom-name: foobar

In the above example, the value foobar is what must conform to the schema defined by the group_by function.

Unique Kinds#

By default, each group can contain only a single task from a given kind. I.e, a group can contain a build task and a signing task, but not two build tasks. This is enforced at task generation time. This is typically the desired behaviour and the check is in place to prevent mistakes.

However, in some cases it may be desirable to depend on multiple tasks of the same kind (e.g, if implementing a notify task). In this case it’s possible to specify unique-kinds:

tasks:
  notify:
    from-deps:
      unique-kinds: false
      group-by: custom

This will disable the uniqueness check and switch dependency edge names to the dependency’s label rather than its kind. Now the notify task can be used with a custom group-by function that returns more than one kind per group.

Primary Kind#

Each task has a primary-kind. This is the kind dependency in each grouping that comes first in the list of supported kinds (either via the kind-dependencies in the kind.yml file, or via the from-deps.kinds key). Note that depending how the dependencies get grouped, a given group may not contain a dependency for each kind. Therefore the list of kind dependencies are ordered by preference. E.g, kinds earlier in the list will be chosen as the primary kind before kinds later in the list.

The primary kind is used to derive the task’s label, as well as copy attributes if the copy-attributes key is set to True (see next section).

Each task created by the from_deps transforms, will have a primary-kind-dependency attribute set.

Copying Attributes#

It’s often useful to copy attributes from a dependency. When this key is set to True, all attributes from the primary-kind (see above) will be copied over to the task. If the task contains pre-existing attributes, they will not be overwritten.

Generated Task Names#

By default from_deps will derive a name for generated tasks from the name on the primary-kind. This will override any name set in the kind. In some cases you may need or want precise control over the generated task name. You can use set-name: false to disable this behaviour:

tasks:
  notify:
    from-deps:
      set-name: false

Adding Fetches#

In many cases it is necessary to fetch artifacts from tasks that from_deps has added as dependencies. This can be accomplished by adding a fetches block to your from-deps entry:

kind-dependencies:
  - build

tasks:
  test:
    from-deps:
      fetches:
        build:
          - artifact: target.tar.gz

The above block will add a fetches entry to the task that is compatible with taskgraph ‘s run transforms.

In some cases, artifact names may be different for different upstream tasks within the same kind. You can often handle this by setting an attribute in the upstream tasks, which from_deps can substitute in. For example, suppose we have 3 build tasks with different suffixes for their target artifact (tar.gz, zip, and apk). You can use an entry like this to ensure each generated test task looks for the appropriate artifact:

tasks:
  test:
    from-deps:
      fetches:
        build:
          - artifact: target.{target_suffix}

This can also be useful when combined with other transforms. For example, the chunking transform sets the this_chunk attribute on the tasks it generates. In cases where the chunk number is used in artifacts produced by the chunked tasks, you can make use of this to easily collect them all in a downstream task:

tasks:
  summary:
    from-deps:
      group-by: all
      fetches:
        expensive-test:
          - artifact: test-summary-{this_chunk}.json

From Dependencies Transforms

Contents