taskgraph package#
Subpackages#
- taskgraph.actions package
- taskgraph.loader package
- taskgraph.optimize package
- taskgraph.transforms package
- Subpackages
- Submodules
- taskgraph.transforms.base module
- taskgraph.transforms.from_deps module
- taskgraph.transforms.cached_tasks module
- taskgraph.transforms.code_review module
- taskgraph.transforms.docker_image module
- taskgraph.transforms.fetch module
- taskgraph.transforms.notify module
- taskgraph.transforms.task module
PayloadBuilder
add_generic_index_routes()
add_github_checks()
add_index_routes()
build_beetmover_payload()
build_docker_worker_payload()
build_dummy_payload()
build_generic_worker_payload()
build_invalid_payload()
build_task()
chain_of_trust()
check_caches_are_volumes()
check_run_task_caches()
check_task_dependencies()
check_task_identifiers()
get_branch_rev()
get_default_deadline()
get_default_priority()
index_builder()
payload_builder()
process_treeherder_metadata()
set_defaults()
set_implementation()
task_name_from_label()
validate()
verify_index()
- Module contents
- taskgraph.util package
- Submodules
- taskgraph.util.archive module
- taskgraph.util.attributes module
- taskgraph.util.cached_tasks module
- taskgraph.util.dependencies module
- taskgraph.util.docker module
- taskgraph.util.hash module
- taskgraph.util.keyed_by module
- taskgraph.util.memoize module
- taskgraph.util.parameterization module
- taskgraph.util.path module
- taskgraph.util.python_path module
- taskgraph.util.readonlydict module
- taskgraph.util.schema module
- taskgraph.util.shell module
- taskgraph.util.taskcluster module
cancel_task()
find_task_id()
get_artifact()
get_artifact_from_index()
get_artifact_path()
get_artifact_prefix()
get_artifact_url()
get_current_scopes()
get_index_url()
get_purge_cache_url()
get_root_url()
get_session()
get_task_definition()
get_task_url()
list_artifacts()
list_task_group_incomplete_tasks()
list_tasks()
parse_time()
purge_cache()
requests_retry_session()
rerun_task()
send_email()
state_task()
status_task()
- taskgraph.util.taskgraph module
- taskgraph.util.templates module
- taskgraph.util.time module
- taskgraph.util.treeherder module
- taskgraph.util.vcs module
GitRepository
GitRepository.all_remote_names
GitRepository.base_rev
GitRepository.branch
GitRepository.default_branch
GitRepository.default_remote_name
GitRepository.does_revision_exist_locally()
GitRepository.find_latest_common_revision()
GitRepository.get_changed_files()
GitRepository.get_commit_message()
GitRepository.get_outgoing_files()
GitRepository.get_url()
GitRepository.head_rev
GitRepository.remote_name
GitRepository.tool
GitRepository.update()
GitRepository.working_directory_clean()
HgRepository
HgRepository.all_remote_names
HgRepository.base_rev
HgRepository.branch
HgRepository.default_branch
HgRepository.default_remote_name
HgRepository.does_revision_exist_locally()
HgRepository.find_latest_common_revision()
HgRepository.get_changed_files()
HgRepository.get_commit_message()
HgRepository.get_outgoing_files()
HgRepository.get_url()
HgRepository.head_rev
HgRepository.remote_name
HgRepository.tool
HgRepository.update()
HgRepository.working_directory_clean()
Repository
Repository.NULL_REVISION
Repository.all_remote_names
Repository.base_rev
Repository.branch
Repository.default_branch
Repository.default_remote_name
Repository.does_revision_exist_locally()
Repository.find_latest_common_revision()
Repository.get_changed_files()
Repository.get_commit_message()
Repository.get_outgoing_files()
Repository.get_url()
Repository.head_rev
Repository.remote_name
Repository.run()
Repository.tool
Repository.update()
Repository.working_directory_clean()
get_repository()
- taskgraph.util.verify module
- taskgraph.util.workertypes module
- taskgraph.util.yaml module
- Module contents
Submodules#
taskgraph.config module#
- class taskgraph.config.GraphConfig(_config: Dict, root_dir: str)#
Bases:
object
- register()#
Add the project’s taskgraph directory to the python path, and register any extensions present.
- root_dir: str#
- property taskcluster_yml#
- property vcs_root#
- taskgraph.config.graph_config_schema = <Schema({'trust-domain': <class 'str'>, 'task-priority': <function optionally_keyed_by.<locals>.validator>, 'task-deadline-after': <function optionally_keyed_by.<locals>.validator>, 'workers': {'aliases': {<class 'str'>: {'provisioner': <function optionally_keyed_by.<locals>.validator>, 'implementation': <class 'str'>, 'os': <class 'str'>, 'worker-type': <function optionally_keyed_by.<locals>.validator>}}}, 'taskgraph': {'register': <class 'str'>, 'decision-parameters': <class 'str'>, 'cached-task-prefix': <class 'str'>, 'index-path-regexes': [<class 'str'>], 'repositories': All({<class 'str'>: {'name': <class 'str'>, 'project-regex': <class 'str'>, 'ssh-secret-name': <class 'str'>, <function Extra>: <class 'str'>}}, Length(min=1, max=None), msg=None)}, <function Extra>: <class 'object'>}, extra=PREVENT_EXTRA, required=False) object>#
Schema for GraphConfig
- taskgraph.config.load_graph_config(root_dir)#
- taskgraph.config.validate_graph_config(config)#
taskgraph.create module#
- taskgraph.create.create_task(session, task_id, label, task_def)#
- taskgraph.create.create_tasks(graph_config, taskgraph, label_to_taskid, params, decision_task_id)#
taskgraph.decision module#
- taskgraph.decision.full_task_graph_to_runnable_tasks(full_task_json)#
- taskgraph.decision.get_decision_parameters(graph_config, options)#
Load parameters from the command-line options for ‘taskgraph decision’. This also applies per-project parameters, based on the given project.
- taskgraph.decision.read_artifact(filename)#
- taskgraph.decision.rename_artifact(src, dest)#
- taskgraph.decision.set_try_config(parameters, task_config_file)#
- taskgraph.decision.taskgraph_decision(options, parameters=None)#
Run the decision task. This function implements mach taskgraph decision, and is responsible for
processing decision task command-line options into parameters
running task-graph generation exactly the same way the other mach taskgraph commands do
generating a set of artifacts to memorialize the graph
calling TaskCluster APIs to create the graph
- taskgraph.decision.write_artifact(filename, data)#
taskgraph.docker module#
- taskgraph.docker.build_context(name, outputFile, args=None)#
Build a context.tar for image with specified name.
- taskgraph.docker.build_image(name, tag, args=None)#
Build a Docker image of specified name.
Output from image building process will be printed to stdout.
- taskgraph.docker.get_image_digest(image_name)#
- taskgraph.docker.load_image(url, imageName=None, imageTag=None)#
Load docker image from URL as imageName:tag, if no imageName or tag is given it will use whatever is inside the zstd compressed tarball.
Returns an object with properties ‘image’, ‘tag’ and ‘layer’.
- taskgraph.docker.load_image_by_name(image_name, tag=None)#
- taskgraph.docker.load_image_by_task_id(task_id, tag=None)#
taskgraph.files_changed module#
Support for optimizing tasks based on the set of files that have changed.
- taskgraph.files_changed.check(params, file_patterns)#
Determine whether any of the files changed between 2 revisions match any of the given file patterns.
- taskgraph.files_changed.get_changed_files(head_repository_url, head_rev, base_rev=None)#
Get the set of files changed between revisions. Responses are cached, so multiple calls with the same arguments are OK.
taskgraph.filter_tasks module#
- taskgraph.filter_tasks.filter_target_tasks(graph, parameters, graph_config)#
Proxy filter to use legacy target tasks code.
This should go away once target_tasks are converted to filters.
- taskgraph.filter_tasks.filter_task(name)#
Generator to declare a task filter function.
taskgraph.generator module#
- class taskgraph.generator.Kind(name: str, path: str, config: Dict, graph_config: taskgraph.config.GraphConfig)#
Bases:
object
- config: Dict#
- graph_config: GraphConfig#
- classmethod load(root_dir, graph_config, kind_name)#
- load_tasks(parameters, loaded_tasks, write_artifacts)#
- name: str#
- path: str#
- exception taskgraph.generator.KindNotFound#
Bases:
Exception
Raised when trying to load kind from a directory without a kind.yml.
- class taskgraph.generator.TaskGraphGenerator(root_dir, parameters, decision_task_id='DECISION-TASK', write_artifacts=False)#
Bases:
object
The central controller for taskgraph. This handles all phases of graph generation. The task is generated from all of the kinds defined in subdirectories of the generator’s root directory.
Access to the results of this generation, as well as intermediate values at various phases of generation, is available via properties. This encourages the provision of all generation inputs at instance construction time.
- property full_task_graph#
the full task set, with edges representing dependencies.
@type: TaskGraph
- Type:
The full task graph
- property full_task_set#
all tasks defined by any kind (a graph without edges)
@type: TaskGraph
- Type:
The full task set
- property graph_config#
The configuration for this graph.
@type: TaskGraph
- property label_to_taskid#
A dictionary mapping task label to assigned taskId. This property helps in interpreting optimized_task_graph.
@type: dictionary
- property morphed_task_graph#
The optimized task graph, with any subsequent morphs applied. This graph will have the same meaning as the optimized task graph, but be in a form more palatable to TaskCluster.
@type: TaskGraph
- property optimized_task_graph#
The set of targeted tasks and all of their dependencies; tasks that have been optimized out are either omitted or replaced with a Task instance containing only a task_id.
@type: TaskGraph
- property parameters#
The properties used for this graph.
@type: Properties
- property target_task_graph#
The set of targeted tasks and all of their dependencies
@type: TaskGraph
- property target_task_set#
The set of targeted tasks (a graph without edges)
@type: TaskGraph
- verify(name, obj, *args, **kwargs)#
- taskgraph.generator.load_tasks_for_kind(parameters, kind, root_dir=None)#
Get all the tasks of a given kind.
This function is designed to be called from outside of taskgraph.
taskgraph.graph module#
- class taskgraph.graph.Graph(nodes: FrozenSet, edges: FrozenSet)#
Bases:
object
Generic representation of a directed acyclic graph with labeled edges connecting the nodes. Graph operations are implemented in a functional manner, so the data structure is immutable.
It permits at most one edge of a given name between any set of nodes. The graph is not checked for cycles, and methods may hang or otherwise fail if given a cyclic graph.
The nodes and edges attributes may be accessed in a read-only fashion. The nodes attribute is a set of node names, while edges is a set of (left, right, name) tuples representing an edge named name going from node left to node right..
- edges: FrozenSet#
- links_dict()#
Return a dictionary mapping each node to a set of the nodes it links to (omitting edge names)
- named_links_dict()#
Return a two-level dictionary mapping each node to a dictionary mapping edge names to labels.
- nodes: FrozenSet#
- reverse_links_dict()#
Return a dictionary mapping each node to a set of the nodes linking to it (omitting edge names)
- transitive_closure(nodes, reverse=False)#
Return the transitive closure of <nodes>: the graph containing all specified nodes as well as any nodes reachable from them, and any intervening edges.
If reverse is true, the “reachability” will be reversed and this will return the set of nodes that can reach the specified nodes.
Example:
a ------> b ------> c | `-------> d
transitive_closure([b]).nodes == set([a, b]) transitive_closure([c]).nodes == set([c, b, a]) transitive_closure([c], reverse=True).nodes == set([c]) transitive_closure([b], reverse=True).nodes == set([b, c, d])
- visit_postorder()#
Generate a sequence of nodes in postorder, such that every node is visited after any nodes it links to.
Behavior is undefined (read: it will hang) if the graph contains a cycle.
- visit_preorder()#
Like visit_postorder, but in reverse: evrey node is visited before any nodes it links to.
taskgraph.main module#
- class taskgraph.main.Command(func, args, kwargs, defaults)#
Bases:
tuple
- args#
Alias for field number 1
- defaults#
Alias for field number 3
- func#
Alias for field number 0
- kwargs#
Alias for field number 2
- taskgraph.main.action_callback(options)#
- taskgraph.main.argument(*args, **kwargs)#
- taskgraph.main.build_image(args)#
- taskgraph.main.command(*args, **kwargs)#
- taskgraph.main.create_parser()#
- taskgraph.main.decision(options)#
- taskgraph.main.dump_output(out, path=None, params_spec=None)#
- taskgraph.main.format_taskgraph(options, parameters, logfile=None)#
- taskgraph.main.format_taskgraph_json(taskgraph)#
- taskgraph.main.format_taskgraph_labels(taskgraph)#
- taskgraph.main.format_taskgraph_yaml(taskgraph)#
- taskgraph.main.generate_taskgraph(options, parameters, logdir)#
- taskgraph.main.get_filtered_taskgraph(taskgraph, tasksregex, exclude_keys)#
Filter all the tasks on basis of a regular expression and returns a new TaskGraph object
- taskgraph.main.get_taskgraph_generator(root, parameters)#
Helper function to make testing a little easier.
- taskgraph.main.image_digest(args)#
- taskgraph.main.init_taskgraph(options)#
- taskgraph.main.load_image(args)#
- taskgraph.main.main(args=['-T', '-E', '-b', 'html', '-d', '_build/doctrees', '-D', 'language=en', '.', '/home/docs/checkouts/readthedocs.org/user_builds/taskcluster-taskgraph/checkouts/latest/_readthedocs//html'])#
- taskgraph.main.setup_logging()#
- taskgraph.main.show_taskgraph(options)#
- taskgraph.main.test_action_callback(options)#
- taskgraph.main.validate_docker()#
taskgraph.morph module#
Graph morphs are modifications to task-graphs that take place after the optimization phase.
These graph morphs are largely invisible to developers running ./mach locally, so they should be limited to changes that do not modify the meaning of the graph.
- taskgraph.morph.amend_taskgraph(taskgraph, label_to_taskid, to_add)#
Add the given tasks to the taskgraph, returning a new taskgraph
- taskgraph.morph.derive_index_task(task, taskgraph, label_to_taskid, parameters, graph_config)#
Create the shell of a task that depends on task and on the given docker image.
- taskgraph.morph.make_index_task(parent_task, taskgraph, label_to_taskid, parameters, graph_config)#
- taskgraph.morph.morph(taskgraph, label_to_taskid, parameters, graph_config)#
Apply all morphs
- taskgraph.morph.register_morph(func)#
taskgraph.parameters module#
- exception taskgraph.parameters.ParameterMismatch#
Bases:
Exception
Raised when a parameters.yml has extra or missing parameters.
- class taskgraph.parameters.Parameters(strict=True, repo_root=None, **kwargs)#
Bases:
ReadOnlyDict
An immutable dictionary with nicer KeyError messages on failure
- check()#
- file_url(path, pretty=False)#
Determine the VCS URL for viewing a file in the tree, suitable for viewing by a human.
- Parameters:
path (str) – The path, relative to the root of the repository.
pretty (bool) – Whether to return a link to a formatted version of the file, or the raw file version.
- Return str:
The URL displaying the given path.
- static format_spec(spec)#
Get a friendly identifier from a parameters specifier.
- Parameters:
spec (str) – Parameters specifier.
- Returns:
Name to identify parameters by.
- Return type:
str
- property id#
- is_try()#
Determine whether this graph is being built on a try project or for mach try fuzzy.
- property moz_build_date#
- taskgraph.parameters.extend_parameters_schema(schema, defaults_fn=None)#
Extend the schema for parameters to include per-project configuration.
This should be called by the taskgraph.register function in the graph-configuration.
- Parameters:
schema (Schema) – The voluptuous.Schema object used to describe extended parameters.
defaults_fn (function) – A function which takes no arguments and returns a dict mapping parameter name to default value in the event strict=False (optional).
- taskgraph.parameters.get_contents(path)#
- taskgraph.parameters.get_version(repo_path)#
- taskgraph.parameters.load_parameters_file(spec, strict=True, overrides=None, trust_domain=None, repo_root=None)#
Load parameters from a path, url, decision task-id or project.
Examples
task-id=fdtgsD5DQUmAQZEaGMvQ4Q project=mozilla-central
- taskgraph.parameters.parameters_loader(spec, strict=True, overrides=None)#
taskgraph.target_tasks module#
- taskgraph.target_tasks.filter_for_git_branch(task, parameters)#
Filter tasks by git branch. If run_on_git_branch is not defined, then task runs on all branches
- taskgraph.target_tasks.filter_for_project(task, parameters)#
Filter tasks by project. Optionally enable nightlies.
- taskgraph.target_tasks.filter_for_tasks_for(task, parameters)#
- taskgraph.target_tasks.filter_out_cron(task, parameters)#
Filter out tasks that run via cron.
- taskgraph.target_tasks.filter_out_shipping_phase(task, parameters)#
- taskgraph.target_tasks.get_method(method)#
Get a target_task_method to pass to a TaskGraphGenerator.
- taskgraph.target_tasks.register_target_task(name)#
- taskgraph.target_tasks.standard_filter(task, parameters)#
- taskgraph.target_tasks.target_tasks_codereview(full_task_graph, parameters, graph_config)#
Target the tasks which have indicated they should be run on this project via the run_on_projects attributes.
- taskgraph.target_tasks.target_tasks_default(full_task_graph, parameters, graph_config)#
Target the tasks which have indicated they should be run on this project via the run_on_projects attributes.
- taskgraph.target_tasks.target_tasks_nothing(full_task_graph, parameters, graph_config)#
Select nothing, for DONTBUILD pushes
taskgraph.task module#
- class taskgraph.task.Task(kind: str, label: str, attributes: ~typing.Dict, task: ~typing.Dict, description: str = '', optimization: ~typing.Optional[~typing.Dict[str, ~typing.Any]] = None, dependencies: ~typing.Dict = <factory>, soft_dependencies: ~typing.List = <factory>, if_dependencies: ~typing.List = <factory>)#
Bases:
object
Representation of a task in a TaskGraph. Each Task has, at creation:
kind: the name of the task kind
label; the label for this task
attributes: a dictionary of attributes for this task (used for filtering)
task: the task definition (JSON-able dictionary)
optimization: optimization to apply to the task (see taskgraph.optimize)
dependencies: tasks this one depends on, in the form {name: label}, for example {‘build’: ‘build-linux64/opt’, ‘docker-image’: ‘build-docker-image-desktop-test’}
soft_dependencies: tasks this one may depend on if they are available post optimisation. They are set as a list of tasks label.
if_dependencies: only run this task if at least one of these dependencies are present.
And later, as the task-graph processing proceeds:
task_id – TaskCluster taskId under which this task will be created
This class is just a convenience wrapper for the data type and managing display, comparison, serialization, etc. It has no functionality of its own.
- attributes: Dict#
- dependencies: Dict#
- description: str = ''#
- classmethod from_json(task_dict)#
Given a data structure as produced by taskgraph.to_json, re-construct the original Task object. This is used to “resume” the task-graph generation process, for example in Action tasks.
- if_dependencies: List#
- kind: str#
- label: str#
- optimization: Optional[Dict[str, Any]] = None#
- soft_dependencies: List#
- task: Dict#
- task_id: Optional[str] = None#
- to_json()#
taskgraph.taskgraph module#
- class taskgraph.taskgraph.TaskGraph(tasks: List[Task], graph: Graph)#
Bases:
object
Representation of a task graph.
A task graph is a combination of a Graph and a dictionary of tasks indexed by label. TaskGraph instances should be treated as immutable.
In the graph, tasks are said to “link to” their dependencies. Whereas tasks are “linked from” their dependents.
- for_each_task(f, *args, **kwargs)#
- classmethod from_json(tasks_dict)#
This code is used to generate the a TaskGraph using a dictionary which is representative of the TaskGraph.
- to_json()#
Return a JSON-able object representing the task graph, as documented