Multi-scenario workflows (workflow
)
Concept & design
Research with MESSAGEix models often involves multiple scenarios that are related to one another or derived from one another by certain modifications. Together, the solutions/reported information from these scenarios provide the output data used in research products, e.g. a plot comparing total emissions in a policy scenario to a reference scenario.
model.build
provides tools to build models or scenarios based on (possibly empty) base scenarios; and tools
provides tools for manipulating scenarios or model input data (parameters).
The Workflow
API provided in this module allows researchers to use these pieces and atomic, reusable functions to define arbitrarily complex workflows involving many, related scenarios; and then to solve, report, or otherwise operate on those scenarios.
The generic pattern for workflows is:
Each scenario has zero or 1 (or more?) base/precursor/antecedent scenarios. These must exist before the target scenario can be created.
A workflow ‘step’ includes:
The precursor scenario is cloned to the target scenario name.
Modifications are applied. These can be modifications to structure or to data. For example:
Setting up a model variant, e.g. adding the MESSAGEix-Materials structure to a base MESSAGEix-GLOBIOM model.
Changing policy variables via constraint parameters.
Any other possible modification.
The target scenario is optionally solved.
The target scenario is optionally reported.
A workflow can consist of any number of scenarios and steps.
The same precursor scenario can be used as the basis for multiple target scenarios.
A workflow is ‘run’ starting with the earliest precursor scenario, ending with 1-to-many target scenarios.
The implementation is based on the observation that these form a graph (DAG) of nodes (scenarios) and edges (steps), in the same way that message_ix.reporting
calculations do; and so the dask
DAG features (via genno
) can be used to organize the workflow.
Usage
General
Define a workflow using ordinary Python functions, each handling the modifications/manipulations in a single, atomic workflow step. These functions they must:
Accept 1 argument: either the precursor scenario, or
None
.Return either:
a
Scenario
object: required if the argument isNone
.None
. In this case, the modifications are reflected in theScenario
given as an argument.
The functions may call any other code, and can be from one to many lines; they should be reusable, i.e. respond in simple and obvious ways to a few clearly-defined arguments.
def base_scenario(arg=None) -> Scenario:
"""Generate a base scenario."""
return testing.bare_res(request, test_context, solved=False)
def changes_a(s: Scenario) -> None:
"""Change a scenario by modifying structure data, but not data."""
with s.transact():
s.add_set("technology", "test_tech")
# Here, invoke other code to further modify `s`
def changes_b(s: Scenario, value=100.0) -> None:
"""Change a scenario by modifying parameter data, but not structure.
This function takes an extra argument, `values`, so functools.partial()
can be used to supply different values when it is used in different
workflow steps. See below.
"""
with s.transact():
s.add_par(
"technical_lifetime",
make_df(
"technical_lifetime",
node_loc=s.set("node")[0],
year_vtg=s.set("year")[0],
technology="test_tech",
value=100.0,
unit="y",
),
)
# Here, invoke other code to further modify `s`
With the steps defined, the workflow is composed using a Workflow
instance.
Call Workflow.add()
to define each target model with its precursor and the function that will create the former from the latter:
from message_ix_models import Context, Workflow
# Create the workflow
ctx = Context.get_instance()
wf = Workflow(ctx)
# "Model/base" is created from nothing by calling base_scenario()
wf.add("Model/base", None, base_scenario)
# "Model/A" is created from "Model/base" by calling changes_a()
wf.add("Model/A", "Model/base", changes_a)
# "Model/B1" is created from "Model/A" by calling changes_b() with the
# default value
wf.add("Model/B1", "Model/A", changes_b)
# "Model/B2" is similar, but uses a different value
wf.add("Model/B2", "Model/A", partial(changes_b, value=200.0))
Finally, the workflow is triggered using Workflow.run()
, giving either one scenario identifier or a list of identifiers.
The indicated scenarios are created and solved; if this requires any precursor scenarios, those are first created and solved, etc. as required.
Other, unrelated scenarios/steps are not created.
s1, s2 = wf.run(["Model/B1", "Model/B2"])
Common use cases
Todo
Expand with discussion of workflow patterns common in research projects using MESSAGEix, e.g.:
Run the same scenario with multiple emissions budgets.
API reference
Tools for modeling workflows.
- class message_ix_models.workflow.Workflow(context: Context, solve=True)[source]
Workflow containing multiple
Scenarios
.- Parameters
context (.Context) – Context object with settings common to the entire workflow.
solve (bool, optional) – Passed to every
WorkflowStep
created usingadd()
.
- class message_ix_models.workflow.WorkflowStep(name: str, callback: Callable, solve=True)[source]
Single step in a multi-scenario workflow.
- Parameters
name (str) –
"model name/scenario name"
for theScenario
produced by the step.callback (Callable) – Function to be executed to modify the base into the target Scenario.
solve (bool, optional) – If
True
, the created scenario is solved when it is created.report (bool, optional) – If
True
, the created scenario is reported after it is created and maybe (according to solve) solved.