taskgraph package

taskgraph package#

Subpackages#

Submodules#

taskgraph.config module#

class taskgraph.config.GraphConfig(_config: Dict, root_dir: str)#

Bases: object

register()#: Add the project’s taskgraph directory to the python path, and register any extensions present.

root_dir: str#

property taskcluster_yml#

property vcs_root#

taskgraph.config.graph_config_schema = <Schema({'trust-domain': <class 'str'>, 'task-priority': <function optionally_keyed_by.<locals>.validator>, 'task-deadline-after': <function optionally_keyed_by.<locals>.validator>, 'workers': {'aliases': {<class 'str'>: {'provisioner': <function optionally_keyed_by.<locals>.validator>, 'implementation': <class 'str'>, 'os': <class 'str'>, 'worker-type': <function optionally_keyed_by.<locals>.validator>}}}, 'taskgraph': {'register': <class 'str'>, 'decision-parameters': <class 'str'>, 'cached-task-prefix': <class 'str'>, 'index-path-regexes': [<class 'str'>], 'repositories': All({<class 'str'>: {'name': <class 'str'>, 'project-regex': <class 'str'>, 'ssh-secret-name': <class 'str'>, <function Extra>: <class 'str'>}}, Length(min=1, max=None), msg=None)}, <function Extra>: <class 'object'>}, extra=PREVENT_EXTRA, required=False) object>#: Schema for GraphConfig

taskgraph.config.load_graph_config(root_dir)#

taskgraph.config.validate_graph_config(config)#

taskgraph.create module#

taskgraph.create.create_task(session, task_id, label, task_def)#

taskgraph.create.create_tasks(graph_config, taskgraph, label_to_taskid, params, decision_task_id)#

taskgraph.decision module#

taskgraph.decision.full_task_graph_to_runnable_tasks(full_task_json)#

taskgraph.decision.get_decision_parameters(graph_config, options)#: Load parameters from the command-line options for ‘taskgraph decision’. This also applies per-project parameters, based on the given project.

taskgraph.decision.read_artifact(filename)#

taskgraph.decision.rename_artifact(src, dest)#

taskgraph.decision.set_try_config(parameters, task_config_file)#

taskgraph.decision.taskgraph_decision(options, parameters=None)#

Run the decision task. This function implements mach taskgraph decision, and is responsible for

processing decision task command-line options into parameters

running task-graph generation exactly the same way the other mach taskgraph commands do

generating a set of artifacts to memorialize the graph

calling TaskCluster APIs to create the graph

taskgraph.decision.write_artifact(filename, data)#

taskgraph.docker module#

taskgraph.docker.build_context(name, outputFile, args=None)#: Build a context.tar for image with specified name.

taskgraph.docker.build_image(name, tag, args=None)#

Build a Docker image of specified name.

Output from image building process will be printed to stdout.

taskgraph.docker.get_image_digest(image_name)#

taskgraph.docker.load_image(url, imageName=None, imageTag=None)#

Load docker image from URL as imageName:tag, if no imageName or tag is given it will use whatever is inside the zstd compressed tarball.

Returns an object with properties ‘image’, ‘tag’ and ‘layer’.

taskgraph.docker.load_image_by_name(image_name, tag=None)#

taskgraph.docker.load_image_by_task_id(task_id, tag=None)#

taskgraph.files_changed module#

Support for optimizing tasks based on the set of files that have changed.

taskgraph.files_changed.check(params, file_patterns)#: Determine whether any of the files changed between 2 revisions match any of the given file patterns.

taskgraph.files_changed.get_changed_files(head_repository_url, head_rev, base_rev=None)#: Get the set of files changed between revisions. Responses are cached, so multiple calls with the same arguments are OK.

taskgraph.filter_tasks module#

taskgraph.filter_tasks.filter_target_tasks(graph, parameters, graph_config)#

Proxy filter to use legacy target tasks code.

This should go away once target_tasks are converted to filters.

taskgraph.filter_tasks.filter_task(name)#: Generator to declare a task filter function.

taskgraph.generator module#

class taskgraph.generator.Kind(name: str, path: str, config: Dict, graph_config: taskgraph.config.GraphConfig)#

Bases: object

config: Dict#

graph_config: GraphConfig#

classmethod load(root_dir, graph_config, kind_name)#

load_tasks(parameters, loaded_tasks, write_artifacts)#

name: str#

path: str#

exception taskgraph.generator.KindNotFound#

Bases: Exception

Raised when trying to load kind from a directory without a kind.yml.

class taskgraph.generator.TaskGraphGenerator(root_dir, parameters, decision_task_id='DECISION-TASK', write_artifacts=False)#

Bases: object

The central controller for taskgraph. This handles all phases of graph generation. The task is generated from all of the kinds defined in subdirectories of the generator’s root directory.

Access to the results of this generation, as well as intermediate values at various phases of generation, is available via properties. This encourages the provision of all generation inputs at instance construction time.

property full_task_graph#

the full task set, with edges representing dependencies.

@type: TaskGraph

Type:: The full task graph

property full_task_set#

all tasks defined by any kind (a graph without edges)

@type: TaskGraph

Type:: The full task set

property graph_config#

The configuration for this graph.

@type: TaskGraph

property label_to_taskid#

A dictionary mapping task label to assigned taskId. This property helps in interpreting optimized_task_graph.

@type: dictionary

property morphed_task_graph#

The optimized task graph, with any subsequent morphs applied. This graph will have the same meaning as the optimized task graph, but be in a form more palatable to TaskCluster.

@type: TaskGraph

property optimized_task_graph#

The set of targeted tasks and all of their dependencies; tasks that have been optimized out are either omitted or replaced with a Task instance containing only a task_id.

@type: TaskGraph

property parameters#

The properties used for this graph.

@type: Properties

property target_task_graph#

The set of targeted tasks and all of their dependencies

@type: TaskGraph

property target_task_set#

The set of targeted tasks (a graph without edges)

@type: TaskGraph

verify(name, obj, *args, **kwargs)#

taskgraph.generator.load_tasks_for_kind(parameters, kind, root_dir=None)#

Get all the tasks of a given kind.

This function is designed to be called from outside of taskgraph.

taskgraph.graph module#

class taskgraph.graph.Graph(nodes: FrozenSet, edges: FrozenSet)#

Bases: object

Generic representation of a directed acyclic graph with labeled edges connecting the nodes. Graph operations are implemented in a functional manner, so the data structure is immutable.

It permits at most one edge of a given name between any set of nodes. The graph is not checked for cycles, and methods may hang or otherwise fail if given a cyclic graph.

The nodes and edges attributes may be accessed in a read-only fashion. The nodes attribute is a set of node names, while edges is a set of (left, right, name) tuples representing an edge named name going from node left to node right..

edges: FrozenSet#

links_dict()#: Return a dictionary mapping each node to a set of the nodes it links to (omitting edge names)

named_links_dict()#: Return a two-level dictionary mapping each node to a dictionary mapping edge names to labels.

nodes: FrozenSet#

reverse_links_dict()#: Return a dictionary mapping each node to a set of the nodes linking to it (omitting edge names)

transitive_closure(nodes, reverse=False)#

Return the transitive closure of <nodes>: the graph containing all specified nodes as well as any nodes reachable from them, and any intervening edges.

If reverse is true, the “reachability” will be reversed and this will return the set of nodes that can reach the specified nodes.

Example:

a ------> b ------> c
          |
          `-------> d

transitive_closure([b]).nodes == set([a, b]) transitive_closure([c]).nodes == set([c, b, a]) transitive_closure([c], reverse=True).nodes == set([c]) transitive_closure([b], reverse=True).nodes == set([b, c, d])

visit_postorder()#

Generate a sequence of nodes in postorder, such that every node is visited after any nodes it links to.

Behavior is undefined (read: it will hang) if the graph contains a cycle.

visit_preorder()#: Like visit_postorder, but in reverse: evrey node is visited before any nodes it links to.

taskgraph.main module#

class taskgraph.main.Command(func, args, kwargs, defaults)#

Bases: tuple

args#: Alias for field number 1

defaults#: Alias for field number 3

func#: Alias for field number 0

kwargs#: Alias for field number 2

taskgraph.main.action_callback(options)#

taskgraph.main.argument(*args, **kwargs)#

taskgraph.main.build_image(args)#

taskgraph.main.command(*args, **kwargs)#

taskgraph.main.create_parser()#

taskgraph.main.decision(options)#

taskgraph.main.dump_output(out, path=None, params_spec=None)#

taskgraph.main.format_taskgraph(options, parameters, logfile=None)#

taskgraph.main.format_taskgraph_json(taskgraph)#

taskgraph.main.format_taskgraph_labels(taskgraph)#

taskgraph.main.format_taskgraph_yaml(taskgraph)#

taskgraph.main.generate_taskgraph(options, parameters, logdir)#

taskgraph.main.get_filtered_taskgraph(taskgraph, tasksregex, exclude_keys)#: Filter all the tasks on basis of a regular expression and returns a new TaskGraph object

taskgraph.main.get_taskgraph_generator(root, parameters)#: Helper function to make testing a little easier.

taskgraph.main.image_digest(args)#

taskgraph.main.init_taskgraph(options)#

taskgraph.main.load_image(args)#

taskgraph.main.main(args=['-T', '-E', '-b', 'html', '-d', '_build/doctrees', '-D', 'language=en', '.', '/home/docs/checkouts/readthedocs.org/user_builds/taskcluster-taskgraph/checkouts/latest/_readthedocs//html'])#

taskgraph.main.setup_logging()#

taskgraph.main.show_taskgraph(options)#

taskgraph.main.test_action_callback(options)#

taskgraph.main.validate_docker()#

taskgraph.morph module#

Graph morphs are modifications to task-graphs that take place after the optimization phase.

These graph morphs are largely invisible to developers running ./mach locally, so they should be limited to changes that do not modify the meaning of the graph.

taskgraph.morph.amend_taskgraph(taskgraph, label_to_taskid, to_add)#: Add the given tasks to the taskgraph, returning a new taskgraph

taskgraph.morph.derive_index_task(task, taskgraph, label_to_taskid, parameters, graph_config)#: Create the shell of a task that depends on task and on the given docker image.

taskgraph.morph.make_index_task(parent_task, taskgraph, label_to_taskid, parameters, graph_config)#

taskgraph.morph.morph(taskgraph, label_to_taskid, parameters, graph_config)#: Apply all morphs

taskgraph.morph.register_morph(func)#

taskgraph.parameters module#

exception taskgraph.parameters.ParameterMismatch#

Bases: Exception

Raised when a parameters.yml has extra or missing parameters.

class taskgraph.parameters.Parameters(strict=True, repo_root=None, **kwargs)#

Bases: ReadOnlyDict

An immutable dictionary with nicer KeyError messages on failure

check()#

file_url(path, pretty=False)#

Determine the VCS URL for viewing a file in the tree, suitable for viewing by a human.

Parameters:

path (str) – The path, relative to the root of the repository.
pretty (bool) – Whether to return a link to a formatted version of the file, or the raw file version.

Return str:

The URL displaying the given path.

static format_spec(spec)#

Get a friendly identifier from a parameters specifier.

Parameters:: spec (str) – Parameters specifier.
Returns:: Name to identify parameters by.
Return type:: str

property id#

is_try()#: Determine whether this graph is being built on a try project or for mach try fuzzy.

property moz_build_date#

taskgraph.parameters.extend_parameters_schema(schema, defaults_fn=None)#

Extend the schema for parameters to include per-project configuration.

This should be called by the taskgraph.register function in the graph-configuration.

Parameters:

schema (Schema) – The voluptuous.Schema object used to describe extended parameters.
defaults_fn (function) – A function which takes no arguments and returns a dict mapping parameter name to default value in the event strict=False (optional).

taskgraph.parameters.get_contents(path)#

taskgraph.parameters.get_version(repo_path)#

taskgraph.parameters.load_parameters_file(spec, strict=True, overrides=None, trust_domain=None, repo_root=None)#

Load parameters from a path, url, decision task-id or project.

Examples

task-id=fdtgsD5DQUmAQZEaGMvQ4Q project=mozilla-central

taskgraph.parameters.parameters_loader(spec, strict=True, overrides=None)#

taskgraph.target_tasks module#

taskgraph.target_tasks.filter_for_git_branch(task, parameters)#: Filter tasks by git branch. If run_on_git_branch is not defined, then task runs on all branches

taskgraph.target_tasks.filter_for_project(task, parameters)#: Filter tasks by project. Optionally enable nightlies.

taskgraph.target_tasks.filter_for_tasks_for(task, parameters)#

taskgraph.target_tasks.filter_out_cron(task, parameters)#: Filter out tasks that run via cron.

taskgraph.target_tasks.filter_out_shipping_phase(task, parameters)#

taskgraph.target_tasks.get_method(method)#: Get a target_task_method to pass to a TaskGraphGenerator.

taskgraph.target_tasks.register_target_task(name)#

taskgraph.target_tasks.standard_filter(task, parameters)#

taskgraph.target_tasks.target_tasks_codereview(full_task_graph, parameters, graph_config)#: Target the tasks which have indicated they should be run on this project via the run_on_projects attributes.

taskgraph.target_tasks.target_tasks_default(full_task_graph, parameters, graph_config)#: Target the tasks which have indicated they should be run on this project via the run_on_projects attributes.

taskgraph.target_tasks.target_tasks_nothing(full_task_graph, parameters, graph_config)#: Select nothing, for DONTBUILD pushes

taskgraph.task module#

class taskgraph.task.Task(kind: str, label: str, attributes: ~typing.Dict, task: ~typing.Dict, description: str = '', optimization: ~typing.Optional[~typing.Dict[str, ~typing.Any]] = None, dependencies: ~typing.Dict = <factory>, soft_dependencies: ~typing.List = <factory>, if_dependencies: ~typing.List = <factory>)#

Bases: object

Representation of a task in a TaskGraph. Each Task has, at creation:

kind: the name of the task kind
label; the label for this task
attributes: a dictionary of attributes for this task (used for filtering)
task: the task definition (JSON-able dictionary)
optimization: optimization to apply to the task (see taskgraph.optimize)
dependencies: tasks this one depends on, in the form {name: label}, for example {‘build’: ‘build-linux64/opt’, ‘docker-image’: ‘build-docker-image-desktop-test’}
soft_dependencies: tasks this one may depend on if they are available post optimisation. They are set as a list of tasks label.
if_dependencies: only run this task if at least one of these dependencies are present.

And later, as the task-graph processing proceeds:

task_id – TaskCluster taskId under which this task will be created

This class is just a convenience wrapper for the data type and managing display, comparison, serialization, etc. It has no functionality of its own.

attributes: Dict#

dependencies: Dict#

description: str = ''#

classmethod from_json(task_dict)#: Given a data structure as produced by taskgraph.to_json, re-construct the original Task object. This is used to “resume” the task-graph generation process, for example in Action tasks.

if_dependencies: List#

kind: str#

label: str#

optimization: Optional[Dict[str, Any]] = None#

soft_dependencies: List#

task: Dict#

task_id: Optional[str] = None#

to_json()#

taskgraph.taskgraph module#

class taskgraph.taskgraph.TaskGraph(tasks: List[Task], graph: Graph)#

Bases: object

Representation of a task graph.

A task graph is a combination of a Graph and a dictionary of tasks indexed by label. TaskGraph instances should be treated as immutable.

In the graph, tasks are said to “link to” their dependencies. Whereas tasks are “linked from” their dependents.

for_each_task(f, *args, **kwargs)#

classmethod from_json(tasks_dict)#: This code is used to generate the a TaskGraph using a dictionary which is representative of the TaskGraph.

graph: Graph#

tasks: List[Task]#

to_json()#: Return a JSON-able object representing the task graph, as documented

taskgraph package

Contents

taskgraph package#

Subpackages#

Submodules#

taskgraph.config module#

taskgraph.create module#

taskgraph.decision module#

taskgraph.docker module#

taskgraph.files_changed module#

taskgraph.filter_tasks module#

taskgraph.generator module#

taskgraph.graph module#

taskgraph.main module#

taskgraph.morph module#

taskgraph.parameters module#

taskgraph.target_tasks module#

taskgraph.task module#

taskgraph.taskgraph module#

Module contents#