taskgraph.transforms.run package

taskgraph.transforms.run package#

Submodules#

taskgraph.transforms.run.common module#

Common support for various task types. These functions are all named after the worker implementation they operate on, and take the same three parameters, for consistency.

taskgraph.transforms.run.common.add_artifacts(config, task, taskdesc, path)#
taskgraph.transforms.run.common.add_cache(task, taskdesc, name, mount_point, skip_untrusted=False)#

Adds a cache based on the worker’s implementation.

Parameters:

  • task (dict) – Tasks object.

  • taskdesc (dict) – Target task description to modify.

  • name (str) – Name of the cache.

  • mount_point (path) – Path on the host to mount the cache.

  • skip_untrusted (bool) – Whether cache is used in untrusted environments (default: False). Only applies to docker-worker.

taskgraph.transforms.run.common.docker_worker_add_artifacts(config, task, taskdesc)#

Adds an artifact directory to the task

taskgraph.transforms.run.common.generic_worker_add_artifacts(config, task, taskdesc)#

Adds an artifact directory to the task

taskgraph.transforms.run.common.get_vcsdir_name(os)#
taskgraph.transforms.run.common.should_use_cache(name: str, use_caches: bool | List[str], has_checkout: bool) → bool#
taskgraph.transforms.run.common.support_caches(config: TransformConfig, task: Dict[str, Any], taskdesc: Dict[str, Any])#

Add caches for common tools.

taskgraph.transforms.run.common.support_vcs_checkout(config, task, taskdesc, repo_configs, sparse=False)#

Update a task with parameters to enable a VCS checkout.

This can only be used with run-task tasks, as the cache name is reserved for run-task tasks.

taskgraph.transforms.run.run_task module#

Support for running tasks that are invoked via the run-task script.

taskgraph.transforms.run.run_task.common_setup(config, task, taskdesc, command)#
taskgraph.transforms.run.run_task.docker_worker_run_task(config, task, taskdesc)#
taskgraph.transforms.run.run_task.generic_worker_run_task(config, task, taskdesc)#
taskgraph.transforms.run.run_task.run_task_schema#

extra keys: prevented

using (required)

type: run-task

Specifies the task type. Must be ‘run-task’.

use-caches (optional)

type: Any

Specifies which caches to use. May take a boolean in which case either all (True) or no (False) caches will be used. Alternatively, it can accept a list of caches to enable. Defaults to only the checkout cache enabled.

bool

or

list[cargo | checkout | npm | pip | uv]

checkout (required)

type: Any

If true (the default), perform a checkout on the worker. Can also be a dictionary specifying explicit checkouts.

bool

or

dict[str, dict]

cwd (optional)

type: str

Path to run command in. If a checkout is present, the path to the checkout will be interpolated with the key checkout.

sparse-profile (required)

type: Any[str | None]

The sparse checkout profile to use. Value is the filename relative to the directory where sparse profiles are defined (build/sparse-profiles/).

command (required)

type: Any

The command arguments to pass to the run-task script, after the checkout arguments. If a list, it will be passed directly; otherwise it will be included in a single argument to the command specified by exec-with.

list

Any

str

or

dict

task-reference (required)

type: str

or

dict

artifact-reference (required)

type: str

or

Any

str

or

dict

task-reference (required)

type: str

or

dict

artifact-reference (required)

type: str

exec-with (optional)

type: Any[bash | powershell]

Specifies what to execute the command with in the event the command is a string.

run-task-command (optional)

type: list

Command used to invoke the run-task script. Can be used if the script or Python installation is in a non-standard location on the workers.

workdir (required)

type: str

Base work directory used to set up the task.

run-as-root (optional)

type: bool

Whether to run as root. Defaults to False.

taskgraph.transforms.run.run_task.script_url(config, script)#

taskgraph.transforms.run.toolchain module#

Support for running toolchain-building tasks via dedicated scripts

taskgraph.transforms.run.toolchain.common_toolchain(config, task, taskdesc, is_docker)#
taskgraph.transforms.run.toolchain.docker_worker_toolchain(config, task, taskdesc)#
taskgraph.transforms.run.toolchain.generic_worker_toolchain(config, task, taskdesc)#
taskgraph.transforms.run.toolchain.get_digest_data(config, run, taskdesc)#
taskgraph.transforms.run.toolchain.toolchain_run_schema#

extra keys: allowed

using (required)

type: toolchain-script

Specifies the run type. Must be “toolchain-script”.

script (required)

type: str

The script (in taskcluster/scripts/misc) to run.

arguments (optional)

type: list[str]

Arguments to pass to the script.

sparse-profile (required)

type: Any[str | None]

Sparse profile to give to checkout using run-task. If given, a filename in build/sparse-profiles. Defaults to “toolchain-build”, i.e., to build/sparse-profiles/toolchain-build. If None, instructs run-task to not use a sparse profile at all.

resources (optional)

type: list[str]

Paths/patterns pointing to files that influence the outcome of a toolchain build.

toolchain-artifact (required)

type: str

Path to the artifact produced by the toolchain task.

toolchain-alias (optional)

type: Any

An alias that can be used instead of the real toolchain task name in fetch stanzas for tasks.

str

or

list[str]

toolchain-env (optional)

type: dict[str, object]

Additional env variables to add to the worker when using this toolchain.

workdir (required)

type: str

Base work directory used to set up the task.

Module contents#

Convert a run description into a task description.

Run descriptions are similar to task descriptions, but they specify how to run the task at a higher level, using a “run” field that can be interpreted by run-using handlers in taskcluster/taskgraph/transforms/run.

taskgraph.transforms.run.always_optimized(config, task, taskdesc)#
taskgraph.transforms.run.configure_taskdesc_for_run(config, task, taskdesc, worker_implementation)#

Run the appropriate function for this task against the given task description.

This will raise an appropriate error if no function exists, or if the task’s run is not valid according to the schema.

taskgraph.transforms.run.get_attribute(dict, key, attributes, attribute_name)#

Get attribute_name from the given attributes dict, and if there is a corresponding value, set key in dict to that value.

taskgraph.transforms.run.make_task_description(config, tasks)#

Given a build description, create a task description

taskgraph.transforms.run.rewrite_when_to_optimization(config, tasks)#
taskgraph.transforms.run.run_description_schema#

extra keys: prevented

name (optional)

type: str

The name of the task. At least one of ‘name’ or ‘label’ must be specified. If ‘label’ is not provided, it will be generated from the ‘name’ by prepending the kind.

label (optional)

type: str

The label of the task. At least one of ‘name’ or ‘label’ must be specified. If ‘label’ is not provided, it will be generated from the ‘name’ by prepending the kind.

description (required)

type: str

priority (optional)

type: Any[highest | very-high | high | medium | low | very-low | lowest]

attributes (optional)

type: dict[str, object]

task-from (optional)

type: str

dependencies (optional)

type: dict[All, object]

soft-dependencies (optional)

type: list[str]

if-dependencies (optional)

type: list[str]

requires (optional)

type: Any[all-completed | all-resolved]

deadline-after (optional)

type: str

expires-after (optional)

type: str

routes (optional)

type: list[str]

scopes (optional)

type: list[str]

tags (optional)

type: dict[str, str]

extra (optional)

type: dict[str, object]

treeherder (optional)

type: Any

True

or

dict[symbol | kind | tier | platform, Optional]

index (optional)

type: dict

product: str

job-name: str

type: str

rank: Any

by-tier

or

int

or

build_date

run-on-projects (optional)

type: list[str]

optionally keyed by: build-platform

run-on-tasks-for (optional)

type: list[str]

run-on-git-branches (optional)

type: list[str]

shipping-phase (optional)

type: Any[None | build | promote | push | ship]

always-target (optional)

type: bool

optimization (exclusive=optimization)

type: Any

None

or

dict

index-search: list

str

or

dict

skip-unless-changed: list

str

needs-sccache (optional)

type: bool

when (exclusive=optimization)

type: dict

The “when” section contains descriptions of the circumstances under which this task should be included in the task graph. This will be converted into an optimization, so it cannot be specified in a run description that also gives ‘optimization’.

files-changed (optional)

type: list[str]

This task only needs to be run if a file matching one of the given patterns has changed in the push. The patterns use the mozpack match function (python/mozbuild/mozpack/path.py).

fetches (optional)

type: dict

A list of artifacts to install from ‘fetch’ tasks.

str: list

str

or

dict

artifact (required)

type: str

dest (optional)

type: str

extract (optional)

type: bool

verify-hash (optional)

type: bool

run (required)

type: dict

A description of how to run this task.

using (required)

type: str

The key to a run implementation in a peer module to this one.

workdir (optional)

type: str

Base work directory used to set up the task.

Extra: object

worker-type (required)

type: str

worker (optional)

type: dict

This object will be passed through to the task description, with additions provided by the task’s run-using function.

taskgraph.transforms.run.run_task_using(worker_implementation, run_using, schema=None, defaults={})#

Register the decorated function as able to set up a task description for tasks with the given worker implementation and run.using property. If schema is given, the task’s run field will be verified to match it.

The decorated function should have the signature using_foo(config, task, taskdesc) and should modify the task description in-place. The skeleton of the task description is already set up, but without a payload.

taskgraph.transforms.run.set_implementation(config, tasks)#
taskgraph.transforms.run.set_label(config, tasks)#
taskgraph.transforms.run.use_fetches(config, tasks)#
Contents