Use Artifacts from Dependencies#
Tasks typically depend on other tasks in order to consume their output, or artifacts. Taskgraph provides a declarative mechanism to use these artifacts called “fetches”.
At a high level, a task declares which artifacts it wishes to use in a
fetches
key in its task definition. Then this information is passed down to
the fetch-content
script which runs immediately preceding the task’s
command. The fetch-content
script handles downloading (and possibly
extracting) the artifact to a known location. The task can then use the artifact
in its payload.
Using Fetches#
Fetches are implemented as part of the run
transforms. So first make sure your task’s kind.yml
file is using those
transforms:
loader: taskgraph.loader.transform:loader
transforms:
- taskgraph.transforms.run:transforms
Now let’s say we’re creating a test task. We’ll need to download and extract the build artifact from a build task. With fetches, this can be accomplished like so:
test:
dependencies:
build: build-opt
fetches:
build:
- build.zip
# <remainder of task definition>
In this example, we first add a dependency edge (named build
) on the task
labelled build-opt
. Then we declare that we want to fetch the build.zip
artifact from this dependency. Note that the key used in the fetches
dict should
correspond to the name of a dependency edge.
Now before your task’s payload is executed, the fetch-content
script will
automatically download and extract the build.zip
artifact to the
$TASK_WORKDIR/fetches
directory. Easy!
Configuring Fetches#
It’s possible to configure the fetch by specifying a dict instead of a string. For example:
test:
dependencies:
build: build-opt
fetches:
build:
- artifact: build.zip
extract: false
dest: some/other/place
This will download the artifact to some/other/place
and avoid extracting
it. If a string is used (like in the first example), the default values will be
used, {extract: true, dest: "fetches"}
.
Using fetch
and toolchain
Tasks#
There are two special kinds of task that exist for the sole purpose of exposing an artifact to downstream consumers:
fetch
- Tasks that fetch a resource from the internet and adds it to Taskcluster’s artifact storage.toolchain
- Tasks that build and/or package a resource.
Since these types of tasks expose a single well defined artifact, using them in fetches is much simpler:
test:
fetches:
fetch:
- grcov
toolchain:
- clang
There are a few differences from the earlier build
examples here:
Instead of specifying the dependency edge name, the
kind
name is used.Instead of specifying the name of the artifact, the label of the task is used (this example assumes that tasks called
grcov
andclang
exist).The dependency on the
fetch
ortoolchain
is implicitly added, so no need to specify them in thedependencies
key.
Note
It is not possible to configure the dest
or extract
values when using
fetch
or toolchain
kinds.