taskgraph.transforms package

taskgraph.transforms package#

Submodules#

taskgraph.transforms.base module#

class taskgraph.transforms.base.RepoConfig(prefix: str, name: str, base_repository: str, head_repository: str, head_ref: str, type: str, path: str = '', head_rev: str | None = None, ssh_secret_name: str | None = None)#

Bases: object

base_repository: str#

head_ref: str#

head_repository: str#

head_rev: str | None = None#

name: str#

path: str = ''#

prefix: str#

ssh_secret_name: str | None = None#

type: str#

class taskgraph.transforms.base.TransformConfig(kind: str, path: str, config: Dict, params: Parameters, kind_dependencies_tasks: Dict[str, Task], graph_config: GraphConfig, write_artifacts: bool)#

Bases: object

A container for configuration affecting transforms. The config argument to transforms is an instance of this class.

config: Dict#

graph_config: GraphConfig#

kind: str#

kind_dependencies_tasks: Dict[str, Task]#

params: Parameters#

path: str#

property repo_configs#

write_artifacts: bool#

class taskgraph.transforms.base.TransformSequence(_transforms: ~typing.List = <factory>)#

Bases: object

Container for a sequence of transforms. Each transform is represented as a callable taking (config, items) and returning a generator which will yield transformed items. The resulting sequence has the same interface.

This is convenient to use in a file full of transforms, as it provides a decorator, @transforms.add, that will add the decorated function to the sequence.

add(func)#

add_validate(schema)#

class taskgraph.transforms.base.ValidateSchema(schema: taskgraph.util.schema.Schema)#

Bases: object

schema: Schema#

taskgraph.transforms.from_deps module#

Transforms used to create tasks based on the kind dependencies, filtering on common attributes like the build-type.

These transforms are useful when follow-up tasks are needed for some indeterminate subset of existing tasks. For example, running a signing task after each build task, whatever builds may exist.

taskgraph.transforms.from_deps.FROM_DEPS_SCHEMA = <Schema({'from-deps': {'kinds': <class 'list'>, 'set-name': Any(None, 'strip-kind', 'retain-kind', {Any('strip-kind', 'retain-kind', msg=None): <class 'object'>}, msg=None), 'with-attributes': {<class 'str'>: Any(<class 'list'>, <class 'str'>, msg=None)}, 'group-by': Any(None, 'single', 'all', 'attribute', {Any('single', 'all', 'attribute', msg=None): <class 'object'>}, msg=None), 'copy-attributes': <class 'bool'>, 'unique-kinds': <class 'bool'>, 'fetches': {<class 'str'>: [{'artifact': <class 'str'>, 'dest': <class 'str'>, 'extract': <class 'bool'>, 'verify-hash': <class 'bool'>}]}}, <function Extra>: <class 'object'>}, extra=PREVENT_EXTRA, required=False) object>#: Schema for from_deps transforms.

taskgraph.transforms.from_deps.from_deps(config, tasks)#

taskgraph.transforms.cached_tasks module#

taskgraph.transforms.cached_tasks.cache_task(config, tasks)#

taskgraph.transforms.cached_tasks.format_task_digest(cached_task)#

taskgraph.transforms.cached_tasks.order_tasks(config, tasks)#: Iterate image tasks in an order where parent tasks come first.

taskgraph.transforms.code_review module#

Add soft dependencies and configuration to code-review tasks.

taskgraph.transforms.code_review.add_dependencies(config, tasks)#

taskgraph.transforms.docker_image module#

taskgraph.transforms.docker_image.fill_template(config, tasks)#

taskgraph.transforms.fetch module#

class taskgraph.transforms.fetch.FetchBuilder(schema: taskgraph.util.schema.Schema, builder: Callable)#

Bases: object

builder: Callable#

schema: Schema#

taskgraph.transforms.fetch.configure_fetch(config, typ, name, fetch)#

taskgraph.transforms.fetch.create_fetch_url_task(config, name, fetch)#

taskgraph.transforms.fetch.create_git_fetch_task(config, name, fetch)#

taskgraph.transforms.fetch.fetch_builder(name, schema)#

taskgraph.transforms.fetch.make_task(config, tasks)#

taskgraph.transforms.fetch.process_fetch_task(config, tasks)#

taskgraph.transforms.notify module#

Add notifications to tasks via Taskcluster’s notify service.

See https://docs.taskcluster.net/docs/reference/core/notify/usage for more information.

taskgraph.transforms.notify.NOTIFY_SCHEMA = <Schema({'notify': {'recipients': [Any({'type': 'email', 'address': <function optionally_keyed_by.<locals>.validator>, 'status-type': Any('on-completed', 'on-defined', 'on-exception', 'on-failed', 'on-pending', 'on-resolved', 'on-running', msg=None)}, {'type': 'matrix-room', 'room-id': <class 'str'>, 'status-type': Any('on-completed', 'on-defined', 'on-exception', 'on-failed', 'on-pending', 'on-resolved', 'on-running', msg=None)}, {'type': 'pulse', 'routing-key': <class 'str'>, 'status-type': Any('on-completed', 'on-defined', 'on-exception', 'on-failed', 'on-pending', 'on-resolved', 'on-running', msg=None)}, {'type': 'slack-channel', 'channel-id': <class 'str'>, 'status-type': Any('on-completed', 'on-defined', 'on-exception', 'on-failed', 'on-pending', 'on-resolved', 'on-running', msg=None)}, msg=None)], 'content': {'email': {'subject': <class 'str'>, 'content': <class 'str'>, 'link': {'text': <class 'str'>, 'href': <class 'str'>}}, 'matrix': {'body': <class 'str'>, 'formatted-body': <class 'str'>, 'format': <class 'str'>, 'msg-type': <class 'str'>}, 'slack': {'text': <class 'str'>, 'blocks': <class 'list'>, 'attachments': <class 'list'>}}}, 'notifications': {'emails': <function optionally_keyed_by.<locals>.validator>, 'subject': <class 'str'>, 'message': <class 'str'>, 'status-types': [Any('on-completed', 'on-defined', 'on-exception', 'on-failed', 'on-pending', 'on-resolved', 'on-running', msg=None)]}}, extra=ALLOW_EXTRA, required=False) object>#: Notify schema.

taskgraph.transforms.notify.add_notifications(config, tasks)#

taskgraph.transforms.task module#

These transformations take a task description and turn it into a TaskCluster task definition (along with attributes, label, etc.). The input to these transformations is generic to any kind of task, but abstracts away some of the complexities of worker implementations, scopes, and treeherder annotations.

class taskgraph.transforms.task.PayloadBuilder(schema: taskgraph.util.schema.Schema, builder: Callable)#

Bases: object

builder: Callable#

schema: Schema#

taskgraph.transforms.task.add_generic_index_routes(config, task)#

taskgraph.transforms.task.add_github_checks(config, tasks)#

For git repositories, add checks route to all tasks.

This will be replaced by a configurable option in the future.

taskgraph.transforms.task.add_index_routes(config, tasks)#

taskgraph.transforms.task.build_beetmover_payload(config, task, task_def)#

taskgraph.transforms.task.build_docker_worker_payload(config, task, task_def)#

taskgraph.transforms.task.build_dummy_payload(config, task, task_def)#

taskgraph.transforms.task.build_generic_worker_payload(config, task, task_def)#

taskgraph.transforms.task.build_invalid_payload(config, task, task_def)#

taskgraph.transforms.task.build_task(config, tasks)#

taskgraph.transforms.task.chain_of_trust(config, tasks)#

taskgraph.transforms.task.check_caches_are_volumes(task)#

Ensures that all cache paths are defined as volumes.

Caches and volumes are the only filesystem locations whose content isn’t defined by the Docker image itself. Some caches are optional depending on the task environment. We want paths that are potentially caches to have as similar behavior regardless of whether a cache is used. To help enforce this, we require that all paths used as caches to be declared as Docker volumes. This check won’t catch all offenders. But it is better than nothing.

taskgraph.transforms.task.check_run_task_caches(config, tasks)#

Audit for caches requiring run-task.

run-task manages caches in certain ways. If a cache managed by run-task is used by a non run-task task, it could cause problems. So we audit for that and make sure certain cache names are exclusive to run-task.

IF YOU ARE TEMPTED TO MAKE EXCLUSIONS TO THIS POLICY, YOU ARE LIKELY CONTRIBUTING TECHNICAL DEBT AND WILL HAVE TO SOLVE MANY OF THE PROBLEMS THAT RUN-TASK ALREADY SOLVES. THINK LONG AND HARD BEFORE DOING THAT.

taskgraph.transforms.task.check_task_dependencies(config, tasks)#: Ensures that tasks don’t have more than 100 dependencies.

taskgraph.transforms.task.check_task_identifiers(config, tasks)#: Ensures that all tasks have well defined identifiers: ^[a-zA-Z0-9_-]{1,38}$

taskgraph.transforms.task.get_branch_rev(config)#

taskgraph.transforms.task.get_default_deadline(graph_config, project)#

taskgraph.transforms.task.get_default_priority(graph_config, project)#

taskgraph.transforms.task.index_builder(name)#

taskgraph.transforms.task.payload_builder(name, schema)#

taskgraph.transforms.task.process_treeherder_metadata(config, tasks)#

taskgraph.transforms.task.set_defaults(config, tasks)#

taskgraph.transforms.task.set_implementation(config, tasks)#: Set the worker implementation based on the worker-type alias.

taskgraph.transforms.task.task_name_from_label(config, tasks)#

taskgraph.transforms.task.validate(config, tasks)#

taskgraph.transforms.task.verify_index(config, index)#

taskgraph.transforms package

Contents

taskgraph.transforms package#

Subpackages#

Submodules#

taskgraph.transforms.base module#

taskgraph.transforms.from_deps module#

taskgraph.transforms.cached_tasks module#

taskgraph.transforms.code_review module#

taskgraph.transforms.docker_image module#

taskgraph.transforms.fetch module#

taskgraph.transforms.notify module#

taskgraph.transforms.task module#

Module contents#