taskgraph.transforms package

Contents

taskgraph.transforms package#

Subpackages#

Submodules#

taskgraph.transforms.base module#

class taskgraph.transforms.base.RepoConfig(prefix: str, name: str, base_repository: str, head_repository: str, head_ref: str, type: str, path: str = '', head_rev: str | None = None, ssh_secret_name: str | None = None)#

Bases: object

base_repository: str#
head_ref: str#
head_repository: str#
head_rev: str | None = None#
name: str#
path: str = ''#
prefix: str#
ssh_secret_name: str | None = None#
type: str#
class taskgraph.transforms.base.TransformConfig(kind: str, path: str, config: dict, params: Parameters, kind_dependencies_tasks: dict[str, Task], graph_config: GraphConfig, write_artifacts: bool)#

Bases: object

A container for configuration affecting transforms. The config argument to transforms is an instance of this class.

config: dict#
graph_config: GraphConfig#
kind: str#
kind_dependencies_tasks: dict[str, Task]#
params: Parameters#
path: str#
property repo_configs#
write_artifacts: bool#
class taskgraph.transforms.base.TransformSequence(_transforms: list = <factory>)#

Bases: object

Container for a sequence of transforms. Each transform is represented as a callable taking (config, items) and returning a generator which will yield transformed items. The resulting sequence has the same interface.

This is convenient to use in a file full of transforms, as it provides a decorator, @transforms.add, that will add the decorated function to the sequence.

add(func)#
add_validate(schema)#
class taskgraph.transforms.base.ValidateSchema(schema: taskgraph.util.schema.Schema)#

Bases: object

schema: Schema#

taskgraph.transforms.from_deps module#

Transforms used to create tasks based on the kind dependencies, filtering on common attributes like the build-type.

These transforms are useful when follow-up tasks are needed for some indeterminate subset of existing tasks. For example, running a signing task after each build task, whatever builds may exist.

taskgraph.transforms.from_deps.FROM_DEPS_SCHEMA#

extra keys: allowed

from-deps (required)

type: dict

kinds (optional)

type: list[str]

Limit dependencies to specified kinds (defaults to all kinds in kind-dependencies). The first kind in the list is the “primary” kind. The dependency of this kind will be used to derive the label and copy attributes (if copy-attributes is True).

set-name (optional)

type: Any

UPDATE ME AND DOCS

None

or

False

or

strip-kind

or

retain-kind

or

dict

Any[strip-kind | retain-kind]: object

with-attributes (optional)

type: dict

Limit dependencies to tasks whose attributes match using attrmatch().

str: Any

list

or

str

group-by (optional)

type: Any

Group cross-kind dependencies using the given group-by function. One task will be created for each group. If not specified, the ‘single’ function will be used which creates a new task for each individual dependency.

None

or

single

or

all

or

attribute

or

dict

Any[single | all | attribute]: object

copy-attributes (optional)

type: bool

If True, copy attributes from the dependency matching the first kind in the kinds list (whether specified explicitly or taken from kind-dependencies).

unique-kinds (optional)

type: bool

If true (the default), there must be only a single unique task for each kind in a dependency group. Setting this to false disables that requirement.

fetches (optional)

type: dict

If present, a fetches entry will be added for each task dependency. Attributes of the upstream task may be used as substitution values in the artifact or dest values of the fetches entry.

str: list

dict

artifact (required)

type: str

dest (optional)

type: str

extract (optional)

type: bool

verify-hash (optional)

type: bool

taskgraph.transforms.from_deps.from_deps(config, tasks)#

taskgraph.transforms.cached_tasks module#

taskgraph.transforms.cached_tasks.cache_task(config, tasks)#
taskgraph.transforms.cached_tasks.format_task_digest(cached_task)#
taskgraph.transforms.cached_tasks.order_tasks(config, tasks)#

Iterate image tasks in an order where parent tasks come first.

taskgraph.transforms.chunking module#

taskgraph.transforms.chunking.CHUNK_SCHEMA#

extra keys: allowed

chunk (optional)

type: dict

chunk can be used to split one task into total-chunks tasks, substituting this_chunk and total_chunks into any fields in substitution-fields.

total-chunks (required)

type: int

The total number of chunks to split the task into.

substitution-fields (optional)

type: list[str]

A list of fields that need to have {this_chunk} and/or {total_chunks} replaced in them.

taskgraph.transforms.chunking.chunk_tasks(config, tasks)#

taskgraph.transforms.code_review module#

Add soft dependencies and configuration to code-review tasks.

taskgraph.transforms.code_review.add_dependencies(config, tasks)#

taskgraph.transforms.docker_image module#

taskgraph.transforms.docker_image.docker_image_schema#

extra keys: prevented

name (required)

type: str

Name of the docker image.

parent (optional)

type: str

Name of the parent docker image.

symbol (optional)

type: str

Treeherder symbol.

task-from (optional)

type: str

Relative path (from config.path) to the file the docker image was defined in.

args (optional)

type: dict[str, str]

Arguments to use for the Dockerfile.

definition (optional)

type: str

Name of the docker image definition under taskcluster/docker, when different from the docker image name.

packages (optional)

type: list[str]

List of package tasks this docker image depends on.

index (optional)

type: dict

Information for indexing this build so its artifacts can be discovered.

product: str

job-name: str

type: str

rank: Any

by-tier

or

int

or

build_date

cache (optional)

type: bool

Whether this image should be cached based on inputs.

taskgraph.transforms.docker_image.fill_template(config, tasks)#

taskgraph.transforms.fetch module#

taskgraph.transforms.fetch.FETCH_SCHEMA#

extra keys: prevented

name (required)

type: str

Name of the task.

task-from (optional)

type: str

Relative path (from config.path) to the file the task was defined in.

description (required)

type: str

Description of the task.

expires-after (optional)

type: str

docker-image (optional)

type: object

fetch-alias (optional)

type: str

An alias that can be used instead of the real fetch task name in fetch stanzas for tasks.

artifact-prefix (optional)

type: str

The prefix of the taskcluster artifact being uploaded. Defaults to public/; if it starts with something other than public/ the artifact will require scopes to access.

attributes (optional)

type: dict[str, object]

fetch (required)

type: dict

type (required)

type: str

Extra: object

class taskgraph.transforms.fetch.FetchBuilder(schema: taskgraph.util.schema.Schema, builder: Callable)#

Bases: object

builder: Callable#
schema: Schema#
taskgraph.transforms.fetch.configure_fetch(config, typ, name, fetch)#
taskgraph.transforms.fetch.create_fetch_url_task(config, name, fetch)#
taskgraph.transforms.fetch.create_git_fetch_task(config, name, fetch)#
taskgraph.transforms.fetch.fetch_builder(name, schema)#
taskgraph.transforms.fetch.make_task(config, tasks)#
taskgraph.transforms.fetch.process_fetch_task(config, tasks)#

taskgraph.transforms.matrix module#

Transforms used to split one task definition into many tasks, governed by a matrix defined in the definition.

taskgraph.transforms.matrix.MATRIX_SCHEMA#

extra keys: allowed

name (required)

type: str

matrix (optional)

type: dict

exclude (optional)

type: list

Exclude the specified combination(s) of matrix values from the final list of tasks. If only a subset of the possible rows are present in the exclusion rule, then all combinations including that subset subset will be excluded.

dict[str, str]

set-name (optional)

type: str

Sets the task name to the specified format string. Useful for cases where the default of joining matrix values by a dash is not desired.

substitution-fields (optional)

type: list[str]

List of fields in the task definition to substitute matrix values into. If not specified, all fields in the task definition will be substituted.

Extra: list

str

taskgraph.transforms.matrix.split_matrix(config, tasks)#

taskgraph.transforms.notify module#

Add notifications to tasks via Taskcluster’s notify service.

See https://docs.taskcluster.net/docs/reference/core/notify/usage for more information.

taskgraph.transforms.notify.NOTIFY_SCHEMA#

extra keys: allowed

notify (exclusive=config)

type: dict

recipients (required)

type: list

Any

dict

type (required)

type: email

address (required)

type: str

optionally keyed by: project, level

status-type (optional)

type: Any[on-completed | on-defined | on-exception | on-failed | on-pending | on-resolved | on-running]

or

dict

type (required)

type: matrix-room

room-id (required)

type: str

status-type (optional)

type: Any[on-completed | on-defined | on-exception | on-failed | on-pending | on-resolved | on-running]

or

dict

type (required)

type: pulse

routing-key (required)

type: str

status-type (optional)

type: Any[on-completed | on-defined | on-exception | on-failed | on-pending | on-resolved | on-running]

or

dict

type (required)

type: slack-channel

channel-id (required)

type: str

status-type (optional)

type: Any[on-completed | on-defined | on-exception | on-failed | on-pending | on-resolved | on-running]

content (optional)

type: dict

email (optional)

type: dict

subject (optional)

type: str

content (optional)

type: str

link (optional)

type: dict

text (required)

type: str

href (required)

type: str

matrix (optional)

type: dict

body (optional)

type: str

formatted-body (optional)

type: str

format (optional)

type: str

msg-type (optional)

type: str

slack (optional)

type: dict

text (optional)

type: str

blocks (optional)

type: list

attachments (optional)

type: list

notifications (exclusive=config)

type: dict

emails (required)

type: list[str]

optionally keyed by: project, level

subject (required)

type: str

message (optional)

type: str

status-types (optional)

type: list

Any[on-completed | on-defined | on-exception | on-failed | on-pending | on-resolved | on-running]

taskgraph.transforms.notify.add_notifications(config, tasks)#

taskgraph.transforms.task module#

These transformations take a task description and turn it into a TaskCluster task definition (along with attributes, label, etc.). The input to these transformations is generic to any kind of task, but abstracts away some of the complexities of worker implementations, scopes, and treeherder annotations.

class taskgraph.transforms.task.PayloadBuilder(schema: taskgraph.util.schema.Schema, builder: Callable)#

Bases: object

builder: Callable#
schema: Schema#
taskgraph.transforms.task.add_generic_index_routes(config, task)#
taskgraph.transforms.task.add_github_checks(config, tasks)#

For git repositories, add checks route to all tasks.

This will be replaced by a configurable option in the future.

taskgraph.transforms.task.add_index_routes(config, tasks)#
taskgraph.transforms.task.build_beetmover_payload(config, task, task_def)#
taskgraph.transforms.task.build_docker_worker_payload(config, task, task_def)#
taskgraph.transforms.task.build_dummy_payload(config, task, task_def)#
taskgraph.transforms.task.build_generic_worker_payload(config, task, task_def)#
taskgraph.transforms.task.build_invalid_payload(config, task, task_def)#
taskgraph.transforms.task.build_task(config, tasks)#
taskgraph.transforms.task.chain_of_trust(config, tasks)#
taskgraph.transforms.task.check_caches_are_volumes(task)#

Ensures that all cache paths are defined as volumes.

Caches and volumes are the only filesystem locations whose content isn’t defined by the Docker image itself. Some caches are optional depending on the task environment. We want paths that are potentially caches to have as similar behavior regardless of whether a cache is used. To help enforce this, we require that all paths used as caches to be declared as Docker volumes. This check won’t catch all offenders. But it is better than nothing.

taskgraph.transforms.task.check_run_task_caches(config, tasks)#

Audit for caches requiring run-task.

run-task manages caches in certain ways. If a cache managed by run-task is used by a non run-task task, it could cause problems. So we audit for that and make sure certain cache names are exclusive to run-task.

IF YOU ARE TEMPTED TO MAKE EXCLUSIONS TO THIS POLICY, YOU ARE LIKELY CONTRIBUTING TECHNICAL DEBT AND WILL HAVE TO SOLVE MANY OF THE PROBLEMS THAT RUN-TASK ALREADY SOLVES. THINK LONG AND HARD BEFORE DOING THAT.

taskgraph.transforms.task.check_task_dependencies(config, tasks)#

Ensures that tasks don’t have more than 100 dependencies.

taskgraph.transforms.task.check_task_identifiers(config, tasks)#

Ensures that all tasks have well defined identifiers: ^[a-zA-Z0-9_-]{1,38}$

taskgraph.transforms.task.get_branch_rev(config)#
taskgraph.transforms.task.get_default_deadline(graph_config, project)#
taskgraph.transforms.task.get_default_priority(graph_config, project)#
taskgraph.transforms.task.index_builder(name)#
taskgraph.transforms.task.payload_builder(name, schema)#
taskgraph.transforms.task.process_treeherder_metadata(config, tasks)#
taskgraph.transforms.task.set_defaults(config, tasks)#
taskgraph.transforms.task.set_implementation(config, tasks)#

Set the worker implementation based on the worker-type alias.

taskgraph.transforms.task.task_description_schema#

extra keys: prevented

label (required)

type: str

The label for this task.

description (required)

type: str

Description of the task (for metadata).

attributes (optional)

type: dict[str, object]

Attributes for this task.

task-from (optional)

type: str

Relative path (from config.path) to the file task was defined in.

dependencies (optional)

type: dict[All, object]

Dependencies of this task, keyed by name; these are passed through verbatim and subject to the interpretation of the Task’s get_dependencies method.

priority (optional)

type: Any[highest | very-high | high | medium | low | very-low | lowest]

Priority of the task.

soft-dependencies (optional)

type: list[str]

Soft dependencies of this task, as a list of task labels.

if-dependencies (optional)

type: list[str]

Dependencies that must be scheduled in order for this task to run.

requires (optional)

type: Any[all-completed | all-resolved]

Specifies the condition for task execution.

expires-after (optional)

type: str

Expiration time relative to task creation, with units (e.g., ‘14 days’). Defaults are set based on the project.

deadline-after (optional)

type: str

Deadline time relative to task creation, with units (e.g., ‘14 days’). Defaults are set based on the project.

routes (optional)

type: list[str]

Custom routes for this task; the default treeherder routes will be added automatically.

scopes (optional)

type: list[str]

Custom scopes for this task; any scopes required for the worker will be added automatically. The following parameters will be substituted in each scope: {level} – the scm level of this push {project} – the project of this push.

tags (optional)

type: dict[str, str]

Tags for this task.

extra (optional)

type: dict[str, object]

Custom ‘task.extra’ content.

treeherder (optional)

type: Any

Treeherder-related information. Can be a simple true to auto-generate information or a dictionary with specific keys.

True

or

dict[symbol | kind | tier | platform, Optional]

index (optional)

type: dict

Information for indexing this build so its artifacts can be discovered. If omitted, the build will not be indexed.

product: str

job-name: str

type: str

rank: Any

by-tier

or

int

or

build_date

run-on-projects (optional)

type: list[str]

optionally keyed by: build-platform

The run_on_projects attribute, defaulting to ‘all’. Dictates the projects on which this task should be included in the target task set. See the attributes documentation for details.

run-on-tasks-for (optional)

type: list[str]

Specifies tasks for which this task should run.

run-on-git-branches (optional)

type: list[str]

Specifies git branches for which this task should run.

shipping-phase (optional)

type: Any[None | build | promote | push | ship]

The shipping_phase attribute, defaulting to None. Specifies the release promotion phase that this task belongs to.

always-target (required)

type: bool

The always-target attribute will cause the task to be included in the target_task_graph regardless of filtering. Tasks included in this manner will be candidates for optimization even when optimize_target_tasks is False, unless the task was also explicitly chosen by the target_tasks method.

optimization (required)

type: Any

Optimization to perform on this task during the optimization phase. Defined in taskcluster/taskgraph/optimize.py.

None

or

dict

index-search: list

str

or

dict

skip-unless-changed: list

str

worker-type (required)

type: str

The provisioner-id/worker-type for the task. The following parameters will be substituted in this string: {level} – the scm level of this push.

needs-sccache (required)

type: bool

Whether the task should use sccache compiler caching.

worker (optional)

type: dict

Information specific to the worker implementation that will run this task.

implementation (required)

type: str

The worker implementation type.

Extra: object

taskgraph.transforms.task.task_name_from_label(config, tasks)#
taskgraph.transforms.task.validate(config, tasks)#
taskgraph.transforms.task.verify_index(config, index)#

taskgraph.transforms.task_context module#

taskgraph.transforms.task_context.SCHEMA#

extra keys: allowed

name (optional)

type: str

task-context (optional)

type: dict

task-context can be used to substitute values into any field in a task with data that is not known until taskgraph runs. This data can be provided via from-parameters or from-file, which can pull in values from parameters and a defined yml file respectively. Data may also be provided directly in the from-object section of task-context. This can be useful in kinds that define most of their contents in task-defaults, but have some values that may differ for various concrete tasks in the kind. If the same key is found in multiple places the order of precedence is as follows: - Parameters - from-object keys - File That is to say: parameters will always override anything else.

from-parameters (optional)

type: dict

Retrieve task context values from parameters. A single parameter may be provided or a list of parameters in priority order. The latter can be useful in implementing a “default” value if some other parameter is not provided.

str: Any

list[str]

or

str

from-file (optional)

type: str

Retrieve task context values from a yaml file. The provided file should usually only contain top level keys and values (eg: nested objects will not be interpolated - they will be substituted as text representations of the object).

from-object (optional)

type: object

Key/value pairs to be used as task context

substitution-fields (required)

type: list[str]

A list of fields in the task to substitute the provided values into.

taskgraph.transforms.task_context.render_task(config, tasks)#

Module contents#

Contents