Postprocessing and reporting

The MESSAGEix framework provides zero-configuration reporting of models built on the framework. The word “reporting” refers to calculations and other post-processing performed after a Scenario has been solved by the associated optimization model: first the model solution is obtained, and then things are “reported” based on that solution.

message_ix.report is developed on the basis of ixmp.report and genno. It provides a basis for other code and packages—such as message_ix_models—that perform reporting calculations tailored to specific model structures. Each layer of this “stack” builds on the features in the level below:

Package	Role	Core features	Reporting features
message_ix_models	MESSAGEix-GLOBIOM models	Specific model structure (coal_ppl in t)	Calculations for specific technologies
message_ix	Energy model framework	Common sets/parameters (output)	Derived quantities (tom)
ixmp	Optimization models & data	Scenario with sets, parameters, variables	Reporter auto-populated with sets etc.
genno	Structured calculations	Computer, Key, Quantity	—

These features are accessible through Reporter, which can produce multiple reports from one or more Scenarios. A report and the quantities that enter it is identified by a key, and may…

• perform arbitrarily complex calculations while intelligently handling units;

• read and make use of data that is ‘exogenous’ to (not included in) a Scenario;

• produce output as Python or R objects (in code), or write to files or databases;

• calculate only a requested subset of quantities; and

• much, much more!

Contents:

Concepts

See Concepts and usage in the genno documentation for an introduction to concepts including quantity, key, computation, task, graph, and operator. In message_ix.report:

• The Reporter class is an extended version of the genno.Computer class.

• ixmp parameters, scalars, equations, and time-series data all become quantities for the purpose of reporting.

• For example, the MESSAGEix parameter resource_cost, defined with the dimensions (node n, commodity c, grade g, year y) is identified by the key resource_cost:n-c-g-y. When summed across the grade/g dimension, it has dimensions n, c, y and is identified by the key resource_cost:n-c-y.

• Reporter.from_scenario() automatically sets up keys and tasks (such as resource_cost:n-c-g-y) that simply retrieve raw/unprocessed data from a Scenario and return it as a genno.Quantity.

• Operators are defined as functions in modules including: message_ix.report.operator, ixmp.report.operator, and genno.operator. These are documented below.

Usage

A MESSAGEix reporting workflow has the following steps:

1. Obtain a Scenario object from an Platform.

2. Use Reporter.from_scenario() to prepare a Reporter object with many calculations automatically prepared.

3. (optionally) Use the built-in features of Reporter to describe additional calculations.

4. Use get() 1 or more times to execute tasks, including all the calculations on which they depend:

from ixmp import Platform
from message_ix import Scenario, Reporter

mp = Platform()
scen = Scenario(scen)
rep = Reporter.from_scenario(scen)
rep.get("all")

Note that keys and tasks are described in steps (2–3), but they are not executed until get() is called—or the results of one task are required by another. This design allows the Reporter to skip unneeded (and potentially slow) computations and deliver good performance. The Reporter’s graph may contain thousands of tasks for retrieving model quantities and calculating derived quantities, but a particular call to get() may only execute a few of these.

Customization

A Reporter prepared with from_scenario() always contains a key scenario, referring to the Scenario to be reported.

The method Reporter.add() can be used to add arbitrary Python code that operates directly on the Scenario object:

def my_custom_report(scenario):
    """Function with custom code that manipulates the `scenario`."""
    print("Model name:", scenario.model)

# Add a task at the key "custom". The task executes my_custom_report().
# The key "scenario" means that the Scenario object is retrieved and
# passed as an argument to the function.
rep.add("custom", (my_custom_report, "scenario"))
rep.get("custom")

In this example, the function my_custom_report() could run to thousands of lines; read to and write from multiple files; invoke other programs or Python scripts; etc. In order to take advantage of the performance-optimizing features of the Reporter, such calculations can instead be composed from atomic (i.e. small, indivisible) operators or functions. See the genno documentation for more.

API reference

Top-level classes and functions

message_ix.report provides:

Reporter(*args, **kwargs)

MESSAGEix Reporter.

The following objects from genno may also be imported from message_ix.report. Their documentation is repeated below for convenience.

ComputationError(exc)	Wrapper to print intelligible exception information for Computer.get().
Key(name_or_value[, dims, tag, _fast])	A hashable key for a quantity that includes its dimensionality.
KeyExistsError	Raised by Computer.add() when the target key exists.
MissingKeyError	Raised by Computer.add() when a required input key is missing.
Quantity	alias of AttrSeries
configure([path])	Configure genno globally.

ixmp.Reporter.from_scenario automatically adds keys based on the contents of the Scenario argument; that is, every ixmp set, parameter, variable, and equation available in the Scenario. message_ix.Reporter.from_scenario extends this to add additional keys for derived quantities specific to the MESSAGEix model framework. These include:

Tip

Use full_key() to retrieve the full-dimensionality Key for any of these quantities.

• out = output × ACT; that is, the product of output (output efficiency) and ACT (activity)

• out_hist = output × ref_activity (historical reference activity)

• in = input × ACT

• in_hist = input × ref_activity

• emi = emission_factor × ACT

• emi_hist = emission_factor × ref_activity

• inv = inv_cost × CAP_NEW

• inv_hist = inv_cost × ref_new_capacity

• fom = fix_cost × CAP; the name refers to “Fixed Operation and Maintenance costs”

• fom_hist = fix_cost × ref_capacity

• vom = var_cost × ACT; “Variable Operation and Maintenance costs”

• vom_hist = var_cost × ref_activity

• tom = fom + vom; “Total Operation and Maintenance costs”

• land_out = land_output × LAND

• land_use_qty = land_use × LAND

• land_emi = land_emission × LAND

• addon conversion, the model parameter addon_conversion (note space versus underscore), except broadcast across individual add-on technologies (ta) rather than add-on types (type_addon).

• addon up, which is addon_up similarly broadcast.

• addon ACT = addon conversion × ACT

• addon in = input × addon ACT

• addon out = output × addon ACT

• addon potential = addon up × addon ACT, the maximum potential activity by add-on technology.

• price emission, the model variable PRICE_EMISSION broadcast across emission species (e) and technologies (t) rather than types (type_emission, type_tec).

Other added keys include:

• message_ix adds the standard short symbols for MESSAGEix dimensions (sets) based on common.DIMS. Each of these is also available in a Reporter: for example rep.get("n") returns a list with the elements of the MESSAGEix set named “node”; rep.get("t") returns the elements of the set “technology”, and so on. These keys can be used as input to other computations.

• y0 = the firstmodelyear or \(y_0\) (int).

• y::model = only the periods in the year set (y) that are equal to or greater than y0.

• Computations to convert internal Quantity data format to the IAMC data format, specifically as pyam.IamDataFrame objects. These include:

  • <name>::pyam for most of the above derived quantities.

  • CAP::pyam (from CAP)

  • CAP_NEW::pyam (from CAP_NEW)

• map_<name> as “one-hot” or indicator quantities for the respective MESSAGEix mapping sets cat_<name>.

• Standard reports message::system, message::costs, and message::emissions per TASKS1.

• The report message::default, collecting all of the above reports.

These automatic contents are prepared using:

TASKS0	Common reporting tasks.
PYAM_CONVERT	Quantities to automatically convert to IAMC format using as_pyam().
TASKS1	Automatic reports that concat() quantities converted to IAMC format.

class message_ix.report.Reporter(*args, **kwargs)

Bases: Reporter

MESSAGEix Reporter.

add(data, *args, **kwargs)	General-purpose method to add computations.
add_queue(queue[, max_tries, fail])	Add tasks from a list or queue.
add_sankey(year, node[, exclude])	Add the tasks required to produce a Sankey diagram.
add_single(key, *computation[, strict, index])	Add a single computation at key.
apply(generator, *keys, **kwargs)	Add computations by applying generator to keys.
check_keys(*keys[, predicate, action])	Check that keys are in the Computer.
configure([path, fail, config])	Configure the Computer.
describe([key, quiet])	Return a string describing the computations that produce key.
eval(expr)	Evaluate expr to add tasks and keys.
finalize(scenario)	Prepare the Reporter to act on scenario.
from_scenario(scenario, **kwargs)	Create a Reporter by introspecting scenario.
full_key(name_or_key)	Return the full-dimensionality key for name_or_key.
get([key])	Execute and return the result of the computation key.
infer_keys(key_or_keys[, dims])	Infer complete key_or_keys.
keys()	Return the keys of graph.
set_filters(**filters)	Apply filters ex ante (before computations occur).
visualize(filename[, key, optimize_graph])	Generate an image describing the Computer structure.
write(key, path, **kwargs)	Compute key and write the result directly to path.

add_file(*args, **kwargs)	Deprecated.
add_product(*args, **kwargs)	Deprecated.
aggregate(qty, tag, dims_or_groups[, ...])	Deprecated.
convert_pyam(*args, **kwargs)	Deprecated.
disaggregate(qty, new_dim[, method, args])	Deprecated.

add(data, *args, **kwargs) → KeyLike | tuple[KeyLike, ...]

General-purpose method to add computations.

add() can be called in several ways; its behaviour depends on data; see below. It chains to methods such as add_single(), add_queue(), and/or apply(); each can also be called directly.

Returns:

Some or all of the keys added to the Computer.

Return type:

KeyLike or tuple of KeyLike

See also

add_single, add_queue, apply, iter_keys, single_key

add_aggregate(qty: KeyLike, tag: str, dims_or_groups: Mapping | str | Sequence[str], weights: DataArray | None = None, keep: bool = True, sums: bool = False, fail: str | int | None = None)

Deprecated.

Add a computation that aggregates qty.

Deprecated since version 1.18.0: Instead, for a mapping/dict dims_or_groups, use:

c.add(qty, "aggregate", groups=dims_or_groups, keep=keep, ...)

Or, for str or sequence of str dims_or_groups, use:

c.add(None, "sum", qty, dimensions=dims_or_groups, ...)

Parameters:

• qty (Key or str) – Key of the quantity to be aggregated.

• tag (str) – Additional string to add to the end the key for the aggregated quantity.

• dims_or_groups (str or collections.abc.Iterable of str or dict) – Name(s) of the dimension(s) to sum over, or nested dict.

• weights (xarray.DataArray, optional) – Weights for weighted aggregation.

• keep (bool, optional) – Passed to operator.aggregate.

• sums (bool, optional) – Passed to add().

• fail (str or int, optional) – Passed to add_queue() via add().

Returns:

The key of the newly-added node.

Return type:

Key

add_file(*args, **kwargs)

Deprecated.

Deprecated since version 1.18.0: Instead use add_load_file() via:

c.add(..., "load_file", ...)

add_product(*args, **kwargs)

Deprecated.

Deprecated since version 1.18.0: Instead use add_binop() via:

c.add(..., "mul", ...)

add_queue(queue: Iterable[tuple], max_tries: int = 1, fail: str | int | None = None) → tuple[KeyLike, ...]

Add tasks from a list or queue.

Parameters:

• queue (collections.abc.Iterable of tuple) – Each item is either a N-tuple of positional arguments to add(), or a 2-tuple of (tuple of positional arguments, dict of keyword arguments).

• max_tries (int, optional) – Retry adding elements up to this many times.

• fail ("raise" or str or logging level, optional) – Action to take when a computation from queue cannot be added after max_tries: “raise” an exception, or log messages on the indicated level and continue.

add_sankey(year: int, node: str, exclude: list[str] = []) → str

Add the tasks required to produce a Sankey diagram.

See map_for_sankey() for the meaning of the node, and exclude parameters.

Parameters:

year (int) – The period (year) to be plotted.

Returns:

A key like "sankey figure a1b2c", where the last part is a unique hash of the arguments year, node, and exclude. Calling Reporter.get() with this key triggers generation of a plotly.Figure with the Sankey diagram.

Return type:

str

See also

map_for_sankey, pyam.figures.sankey

add_single(key: KeyLike, *computation, strict=False, index=False) → KeyLike

Add a single computation at key.

Parameters:

• key (str or Key or hashable) – A string, Key, or other value identifying the output of computation.

• computation (object) – Any computation. See graph.

• strict (bool, optional) – If True, key must not already exist in the Computer, and any keys referred to by computation must exist.

• index (bool, optional) – If True, key is added to the index as a full-resolution key, so it can be later retrieved with full_key().

Raises:

• KeyExistsError – If strict is True and either (a) key already exists; or (b) sums is True and the key for one of the partial sums of key already exists.

• MissingKeyError – If strict is True and any key referred to by computation does not exist.

add_tasks(fail_action: int | str = 'raise') → None

Add the pre-defined MESSAGEix reporting tasks to the Reporter.

Parameters:

fail_action ("raise" or int) – logging level or level name, passed to the fail argument of Reporter.add_queue().

aggregate(qty: KeyLike, tag: str, dims_or_groups: Mapping | str | Sequence[str], weights: DataArray | None = None, keep: bool = True, sums: bool = False, fail: str | int | None = None)

Deprecated.

Add a computation that aggregates qty.

Deprecated since version 1.18.0: Instead, for a mapping/dict dims_or_groups, use:

c.add(qty, "aggregate", groups=dims_or_groups, keep=keep, ...)

Or, for str or sequence of str dims_or_groups, use:

c.add(None, "sum", qty, dimensions=dims_or_groups, ...)

Parameters:

• qty (Key or str) – Key of the quantity to be aggregated.

• tag (str) – Additional string to add to the end the key for the aggregated quantity.

• dims_or_groups (str or collections.abc.Iterable of str or dict) – Name(s) of the dimension(s) to sum over, or nested dict.

• weights (xarray.DataArray, optional) – Weights for weighted aggregation.

• keep (bool, optional) – Passed to operator.aggregate.

• sums (bool, optional) – Passed to add().

• fail (str or int, optional) – Passed to add_queue() via add().

Returns:

The key of the newly-added node.

Return type:

Key

apply(generator: Callable, *keys, **kwargs) → KeyLike | tuple[KeyLike, ...]

Add computations by applying generator to keys.

Parameters:

• generator (callable) –

  Function to apply to keys. This function may take a first positional argument annotated with Computer or a subtype; if so, then it is provided with a reference to self.

  The function may:

  • yield or return an iterable of (key, computation). These are used to directly update the graph, and then apply() returns the added keys.

  • If it is provided with a reference to the Computer, call add() or any other method to update the graph. In this case, it should return a Key or sequence of keys, indicating what was added; these are in turn returned by apply().

• keys (Hashable) – The starting key(s). These are provided as positional arguments to generator.

• kwargs – Keyword arguments to generator.

cache(func)

Decorate func so that its return value is cached.

See also

cache

check_keys(*keys: str | Key, predicate=None, action='raise') → list[KeyLike]

Check that keys are in the Computer.

Parameters:

• keys (KeyLike) – Some Keys or strings.

• predicate (callable, optional) – Function to run on each of keys; see below.

• action ("raise" or str) – Action to take on missing keys.

Returns:

One item for each item k in keys:

1. k itself, unchanged, if predicate is given and predicate(k) returns True.

2. Graph.unsorted_key(), that is, k but with its dimensions in a specific order that already appears in graph.

3. Graph.full_key(), that is, an existing key with the name k with its full dimensionality.

4. None otherwise.

Return type:

list of KeyLike

Raises:

MissingKeyError – If action is “raise” and 1 or more of keys do not appear (either in different dimension order, or full dimensionality) in the graph.

configure(path: Path | str | None = None, fail: str | int = 'raise', config: Mapping[str, Any] | None = None, **config_kw)

Configure the Computer.

Accepts a path to a configuration file and/or keyword arguments. Configuration keys loaded from file are superseded by keyword arguments. Messages are logged at level logging.INFO if config contains unhandled sections.

See config for a list of all configuration sections and keys, and details of the configuration file format.

Parameters:

• path (pathlib.Path, optional) – Path to a configuration file in JSON or YAML format.

• fail ("raise" or str or logging level, optional) – Passed to add_queue(). If not “raise”, then log messages are generated for config handlers that fail. The Computer may be only partially configured.

• config – Configuration keys/sections and values, as a mapping. Use this if any of the keys/sections are not valid Python names, for instance if they contain “-” or “ “.

• **config_kw – Configuration keys/sections and values, as keyword arguments.

convert_pyam(*args, **kwargs)

Deprecated.

Deprecated since version 1.18.0: Instead use add_as_pyam() via:

c.require_compat("pyam")
c.add(..., "as_pyam", ...)

default_key: 'KeyLike' | None = None

The default key to get() with no argument.

describe(key=None, quiet=True)

Return a string describing the computations that produce key.

If key is not provided, all keys in the Computer are described.

Unless quiet, the string is also printed to the console.

Returns:

Description of computations. If a malformed graph is detected (one key is its own direct ancestor), the text “← CYCLE DETECTED” is shown, and recursion stops.

Return type:

str

disaggregate(qty, new_dim, method='shares', args=[])

Deprecated.

Deprecated since version 1.18.0: Instead, for method = “disaggregate_shares”, use:

c = Computer()
c.add(qty.append(new_dim), "mul", qty, ..., strict=True)

Or for a callable() method, use:

c.add(qty.append(new_dim), method, qty, ..., strict=True)

duplicate(key: TKeyLike, tag: str) → TKeyLike

Duplicate the task at key and all of its inputs.

Re

Parameters:

• key – Starting key to duplicate.

• tag – tag to add to duplicated keys.

eval(expr: str) → tuple[Key, ...]

Evaluate expr to add tasks and keys.

Parse a statement or block of statements using ast from the Python standard library. expr may include:

• Constants.

• References to existing keys in the Computer by their name; these are expanded using full_key().

• Multiple statements on separate lines or separated by “;”.

• Python arithmetic operators including +, -, *, /, **; these are mapped to the corresponding operator.

• Function calls, also mapped to the corresponding operator via get_operator(). These may include simple positional (constants or key references) or keyword (constants only) arguments.

Parameters:

expr (str) – Expression to be evaluated.

Returns:

One key for the left-hand side of each expression.

Return type:

tuple of Key

Raises:

• NotImplementedError – For complex expressions not supported; if any of the statements is anything other than a simple assignment.

• NameError – If a function call references a non-existent computation.

finalize(scenario: Scenario) → None

Prepare the Reporter to act on scenario.

The TimeSeries (thus also Scenario or message_ix.Scenario) object scenario is stored with the key 'scenario'. All subsequent processing will act on data from this Scenario.

classmethod from_scenario(scenario, **kwargs) → Reporter

Create a Reporter by introspecting scenario.

Warnings are logged if scenario does not have a solution. In this case, any keys/computations based on model output (ixmp variables and equations) may return an empty Quantity, fail, or behave unpredictably. Keys/computations based only on model input (ixmp sets and parameters) should function normally.

Returns:

A reporter for scenario.

Return type:

Reporter

full_key(name_or_key: KeyLike) → KeyLike

Return the full-dimensionality key for name_or_key.

An quantity ‘foo’ with dimensions (a, c, n, q, x) is available in the Computer as 'foo:a-c-n-q-x'. This Key can be retrieved with:

c.full_key("foo")
c.full_key("foo:c")
# etc.

Raises:

KeyError – if name_or_key is not in the graph.

get(key=None)

Execute and return the result of the computation key.

Only key and its dependencies are computed.

Parameters:

key (str, optional) – If not provided, default_key is used.

Raises:

ValueError – If key and default_key are both None.

get_comp(name) → Callable | None

Return a function, Operator, or callable for use in a task.

get_operator() checks each of the modules for a callable with the given name. Modules at the end of the list take precedence over those earlier in the list.

Returns:

• callable

• None – If there is no callable with the given name in any of modules.

get_operator(name) → Callable | None

Return a function, Operator, or callable for use in a task.

get_operator() checks each of the modules for a callable with the given name. Modules at the end of the list take precedence over those earlier in the list.

Returns:

• callable

• None – If there is no callable with the given name in any of modules.

graph: genno.core.graph.Graph = {'config': {}}

A dask-format graph (see 1, 2).

infer_keys(key_or_keys: KeyLike | Iterable[KeyLike], dims: Iterable[str] = [])

Infer complete key_or_keys.

Each return value is one of:

• a Key with either

  • dimensions dims, if any are given, otherwise

  • its full dimensionality (cf. full_key())

• str, the same as input, if the key is not defined in the Computer.

Parameters:

• key_or_keys (KeyLike or list of KeyLike)

• dims (list of str, optional) – Drop all but these dimensions from the returned key(s).

Returns:

• KeyLike – If key_or_keys is a single KeyLike.

• list of KeyLike – If key_or_keys is an iterable of KeyLike.

insert(key: KeyLike, *args, tag: str = 'pre', **kwargs) → None

Insert a task before key, using args, kwargs.

The existing task at key is moved to key + tag. The args and kwargs are passed to add() to insert a new task at key. The args must include at least 2 items:

1. the new callable or Operator, and

2. the .Ellipsis (...), which is replaced by the shifted key + tag.

If there are more than 2 items, each instance of the Ellipsis is replaced per (2); all other items (and kwargs) are passed on as-is.

The effect is that all existing tasks to which key are input will receive, instead, the output of the added task.

One way to use insert() is with a ‘pass-through’ operation that, for instance, performs logging, assertions, or other steps, then returns its input unchanged. It is also possible to insert a new task that mutates its input in certain ways.

keys()

Return the keys of graph.

modules: MutableSequence[types.ModuleType] = []

List of modules containing operators.

By default, this includes the genno built-in operators in genno.operator. require_compat() appends additional modules, for instance genno.compat.plotnine, to this list. User code may also add modules to this list directly.

require_compat(pkg: str | ModuleType)

Register a module for get_operator().

The specified module is appended to modules.

Parameters:

pkg (str or module) –

One of:

• the name of a package (for instance “plotnine”), corresponding to a submodule of genno.compat (genno.compat.plotnine). genno.compat.{pkg}.operator is added.

• the name of any importable module, for instance “foo.bar”.

• a module object that has already been imported.

Raises:

ModuleNotFoundError – If the required packages are missing.

Examples

Operators packaged with genno for compatibility:

>>> c = Computer()
>>> c.require_compat("pyam")

Operators in another module, using the module name:

>>> c.require_compat("ixmp.reporting.computations")

or using imported module object directly:

>>> import ixmp.reporting.computations as mod
>>> c.require_compat(mod)

set_filters(**filters: Any) → None

Apply filters ex ante (before computations occur).

See the description of filters() under Configuration.

property unit_registry

The pint.UnitRegistry used by the Computer.

visualize(filename, key=None, optimize_graph=False, **kwargs)

Generate an image describing the Computer structure.

This is similar to dask.visualize(); see compat.graphviz.visualize(). Requires graphviz.

write(key, path, **kwargs)

Compute key and write the result directly to path.

message_ix.report.TASKS0: tuple[tuple, ...] = (('map_addon', 'map_as_qty', 'cat_addon', 't'), ('map_emission', 'map_as_qty', 'cat_emission', 'e'), ('map_tec', 'map_as_qty', 'cat_tec', 't'), ('map_year', 'map_as_qty', 'cat_year', 'y'), ('out', 'mul', 'output', 'ACT'), ('in', 'mul', 'input', 'ACT'), ('in_cap_new', 'mul', 'input_cap_new', 'CAP_NEW'), ('out_cap_new', 'mul', 'output_cap_new', 'CAP_NEW'), ('rel', 'mul', 'relation_activity', 'ACT'), ('emi', 'mul', 'emission_factor', 'ACT'), ('inv', 'mul', 'inv_cost', 'CAP_NEW'), ('fom', 'mul', 'fix_cost', 'CAP'), ('vom', 'mul', 'var_cost', 'ACT'), ('land_out', 'mul', 'land_output', 'LAND'), ('land_in', 'mul', 'land_input', 'LAND'), ('land_use_qty', 'mul', 'land_use', 'LAND'), ('land_emi', 'mul', 'land_emission', 'LAND'), ('y::model', 'model_periods', 'y', 'cat_year'), ('y0', operator.itemgetter(0), 'y::model'), ('tom', 'add', 'fom:nl-t-yv-ya', 'vom:nl-t-yv-ya'), (('addon conversion:nl-t-yv-ya-m-h-ta', <function broadcast_map>, 'addon_conversion:n-t-yv-ya-m-h-type_addon', 'map_addon'), {'rename': {'n': 'nl'}}), ('addon ACT', 'mul', 'addon conversion', 'ACT'), ('addon in', 'mul', 'input', 'addon ACT'), ('addon out', 'mul', 'output', 'addon ACT'), (('addon up:nl-t-ya-m-h-ta', <function broadcast_map>, 'addon_up:n-t-ya-m-h-type_addon', 'map_addon'), {'rename': {'n': 'nl'}}), ('addon potential', 'mul', 'addon up', 'addon ACT'), ('price emission:n-e-t-y', <function broadcast_map>, (<function broadcast_map>, 'PRICE_EMISSION:n-type_emission-type_tec-y', 'map_emission'), 'map_tec'))

Common reporting tasks. These include:

1. MESSAGE mapping sets, converted to reporting quantities via map_as_qty().

   For instance, the mapping set cat_addon is available at the reporting key map_addon, which produces a genno.Quantity with the two dimensions type_addon and ta (short form of technology_addon). This Quantity contains the value 1 at every valid (type_addon, ta) location, and 0 elsewhere.

2. Simple products of 2 or mode quantities.

3. Other derived quantities.

message_ix.report.PYAM_CONVERT: list[tuple[str, CollapseMessageColsKw]] = [('out:nl-t-ya-m-nd-c-l', {'kind': 'ene', 'var': 'out'}), ('in:nl-t-ya-m-no-c-l', {'kind': 'ene', 'var': 'in'}), ('CAP:nl-t-ya', {'var': 'capacity'}), ('CAP_NEW:nl-t-yv', {'var': 'new capacity'}), ('inv:nl-t-yv', {'var': 'inv cost'}), ('fom:nl-t-ya', {'var': 'fom cost'}), ('vom:nl-t-ya', {'var': 'vom cost'}), ('tom:nl-t-ya', {'var': 'total om cost'}), ('emi:nl-t-ya-m-e', {'kind': 'emi', 'var': 'emis'})]

Quantities to automatically convert to IAMC format using as_pyam().

message_ix.report.TASKS1 = (('message::system', 'concat', 'out::pyam', 'in::pyam', 'CAP::pyam', 'CAP_NEW::pyam'), ('message::costs', 'concat', 'inv::pyam', 'fom::pyam', 'vom::pyam', 'tom::pyam'), ('message::emissions', 'concat', 'emi::pyam'), ('message::default', 'concat', 'message::system', 'message::costs', 'message::emissions'))

Automatic reports that concat() quantities converted to IAMC format.

exception message_ix.report.ComputationError(exc)

Wrapper to print intelligible exception information for Computer.get().

In order to aid in debugging, this helper:

• Omits the parts of the stack trace that are internal to Dask, and

• Gives the key in the Computer.graph and the computation/task that caused the exception.

class message_ix.report.Key(name_or_value: str | Key | AnyQuantity, dims: Iterable[str] = [], tag: str | None = None, _fast: bool = False)

A hashable key for a quantity that includes its dimensionality.

add_tag(tag: str | None) → Key

Return a new Key with tag appended.

append(*dims: str) → Key

Return a new Key with additional dimensions dims.

classmethod bare_name(value) → str | None

If value is a bare name (no dims or tags), return it; else None.

property dims: tuple[str, ...]

Dimensions of the quantity, tuple of str.

drop(*dims: str | bool) → Key

Return a new Key with dims dropped.

drop_all() → Key

Return a new Key with all dimensions dropped / zero dimensions.

classmethod from_str_or_key(value: str | Key | AnyQuantity, drop: Iterable[str] | bool = [], append: Iterable[str] = [], tag: str | None = None) → Key

Return a new Key from value.

Changed in version 1.18.0: Calling from_str_or_key() with a single argument is no longer necessary; simply give the same value as an argument to Key.

The class method is retained for convenience when calling with multiple arguments. However, the following are equivalent and may be more readable:

k1 = Key("foo:a-b-c:t1", drop="b", append="d", tag="t2")
k2 = Key("foo:a-b-c:t1").drop("b").append("d)"

Parameters:

• value (str or Key) – Value to use to generate a new Key.

• drop (list of str or True, optional) – Existing dimensions of value to drop. See drop().

• append (list of str, optional) – New dimensions to append to the returned Key. See append().

• tag (str, optional) – Tag for returned Key. If value has a tag, the two are joined using a ‘+’ character. See add_tag().

Return type:

Key

iter_sums() → Generator[tuple[Key, Callable, Key], None, None]

Generate (key, task) for all possible partial sums of the Key.

property name: str

Name of the quantity, str.

classmethod product(new_name: str, *keys, tag: str | None = None) → Key

Return a new Key that has the union of dimensions on keys.

Dimensions are ordered by their first appearance:

1. First, the dimensions of the first of the keys.

2. Next, any additional dimensions in the second of the keys that were not already added in step 1.

3. etc.

Parameters:

• new_name (str) – Name for the new Key. The names of keys are discarded.

• keys – May include instances of Key, str (converted to Key), or Quantity (the dimensions of the quantity are used directly).

remove_tag(*tags: str) → Key

Return a key with any of tags dropped.

Raises:

ValueError – If none of tags are in tags.

rename(name: str) → Key

Return a Key with a replaced name.

property sorted: Key

A version of the Key with its dims sorted().

property tag: str | None

Quantity tag, str or None.

exception message_ix.report.KeyExistsError

Raised by Computer.add() when the target key exists.

exception message_ix.report.MissingKeyError

Raised by Computer.add() when a required input key is missing.

message_ix.report.Quantity

alias of AttrSeries

message_ix.report.configure(path: Path | str | None = None, **config)

Configure genno globally.

Modifies global variables that affect the behaviour of all Computers and operators. Configuration keys loaded from file are superseded by keyword arguments. Messages are logged at level logging.INFO if config contains unhandled sections.

Parameters:

• path (pathlib.Path, optional) – Path to a configuration file in JSON or YAML format.

• **config – Configuration keys/sections and values.

Operators

message_ix.report provides a small number of operators. Two of these (plot_cumulative() and stacked_bar()) are currently only used in the tutorials to produce simple plots; for more flexible plotting, genno.compat.plotnine is recommended instead.

as_message_df()	Convert qty to an add_par()-ready data frame using make_df().
model_periods(y, cat_year)	Return the elements of y beyond the firstmodelyear of cat_year.
plot_cumulative(x, y, labels)	Plot a supply curve.
stacked_bar(qty[, dims, units, title, cf, ...])	Plot qty as a stacked bar chart.

Other operators are provided by ixmp.report:

data_for_quantity(ix_type, name, column, ...)	Retrieve data from scenario.
from_url(url[, cls])	Return a ixmp.TimeSeries or subclass instance, given its url.
get_ts(ts[, filters, iamc, subannual])	Retrieve timeseries data from ts.
map_as_qty(set_df, full_set)	Convert set_df to a Quantity.
remove_ts(ts[, data, after])	Remove all time series data from ts.
store_ts(scenario, *data[, strict])	Store time series data on scenario.
update_scenario(scenario, *quantities[, params])	Update scenario with computed data from reporting quantities.

…and by genno.operator and its compatibility modules. See the package documentation for details.

Plot()	Class for plotting using plotnine.
add	Sum across multiple quantities.
aggregate(quantity, groups, keep)	Aggregate quantity by groups.
apply_units(qty, units)	Apply units to qty.
as_pyam	Return a pyam.IamDataFrame containing the data from quantity.
broadcast_map(quantity, map[, rename, strict])	Broadcast quantity using a map.
combine(*quantities[, select, weights])	Sum distinct quantities by weights.
concat()	Concatenate Quantity objs.
div	Compute the ratio numerator / denominator.
drop_vars(qty, names, *[, errors])	Return a Quantity with dropped variables (coordinates).
group_sum(qty, group, sum)	Group by dimension group, then sum across dimension sum.
index_to(qty, dim_or_selector[, label])	Compute an index of qty against certain of its values.
interpolate(qty[, coords, method, ...])	Interpolate qty.
load_file	Read the file at path and return its contents as a Quantity.
mul	Compute the product of any number of quantities.
pow(a, b)	Compute a raised to the power of b.
relabel(qty[, labels])	Replace specific labels along dimensions of qty.
rename_dims(qty[, name_dict])	Returns a new Quantity with renamed dimensions or a new name.
round(qty, *args, **kwargs)	Like xarray.DataArray.round().
select(qty, indexers, *[, inverse, drop])	Select from qty based on indexers.
sub	Subtract b from a.
sum	Sum quantity over dimensions, with optional weights.
write_report(-> None  -> None  -> None)	Write a quantity to a file.

disaggregate_shares(quantity, shares)	Deprecated: Disaggregate quantity by shares.
product	Alias of mul(), for backwards compatibility.
ratio	Alias of div(), for backwards compatibility.

message_ix.report.operator.as_message_df(qty: AnyQuantity, name: str, dims: Mapping, common: Mapping, wrap: Literal[True] = True) → dict[str, DataFrame]

message_ix.report.operator.as_message_df(qty: AnyQuantity, name: str, dims: Mapping, common: Mapping, wrap: Literal[False]) → DataFrame

Convert qty to an add_par()-ready data frame using make_df().

The resulting data frame has:

• A “value” column populated with the values of qty.

• A “unit” column with the string representation of the units of qty.

• Other dimensions/key columns filled with labels of qty according to dims.

• Other dimensions/key columns filled with uniform values from common.

Parameters:

• qty (genno.Quantity)

• name (str) – Name of the MESSAGEix parameter to prepare.

• dims (collections.abc.Mapping) – Each key corresponds to a dimension of the target parameter name, for instance “node_loc”; the label corresponds to a dimension of qty, for instance “nl”.

• common (collections.abc.Mapping) – Each key corresponds to a dimension of the target parameter; values are used literally, as if passed to make_df().

• wrap (bool, optional) – See below.

Returns:

• dict – if wrap is True (the default): length 1, mapping from name to pandas.DataFrame containing the converted data.

• pandas.DataFrame – if wrap is False.

message_ix.report.operator.model_periods(y: list[int], cat_year: DataFrame) → list[int]

Return the elements of y beyond the firstmodelyear of cat_year.

message_ix.report.operator.plot_cumulative(x: AnyQuantity, y: AnyQuantity, labels: tuple[str, str, str])

Plot a supply curve.

• x and y must share the first two dimensions.

• The first dimension must contain unique values.

• One rectangle is plotted for each unique value in the second dimension.

Parameters:

• x (genno.Quantity) – e.g. <resource_volume:n-g>.

• y (genno.Quantity) – e.g. <resource_cost:n-g-y>. The mean() is taken across the third dimension.

message_ix.report.operator.stacked_bar(qty: AnyQuantity, dims: tuple[str, ...] = ('nl', 't', 'ya'), units: str = '', title: str = '', cf: float = 1.0, stacked: bool = True)

Plot qty as a stacked bar chart.

Parameters:

• qty (genno.Quantity) – Data to plot.

• dims (tuple of str) –

  3 or more dimensions for, respectively:

  • 1 dimension: The node/region.

  • 1 or more dimensions: to stack in bars of different colour.

  • 1 dimension: The ordinate (x-axis); typically the year.

• units (str) – Units to display on the plot.

• title (str) – Title fragment; the plot title is “{node} {title}”.

• cf (float, optional) – Conversion factor to apply to data.

Utilities

message_ix.report.pyam.collapse_message_cols(df: pandas.DataFrame, var: str | None = None, kind: str | None = None, var_cols=[]) → pandas.DataFrame

genno.compat.pyam collapse=… callback for MESSAGEix quantities.

Wraps collapse() with arguments particular to MESSAGEix.

Parameters:

• var (str) – Name for ‘variable’ column.

• kind (None or 'ene' or 'emi', optional) –

  Determines which other columns are combined into the ‘region’ and ‘variable’ columns:

  • ’ene’: ‘variable’ is '<var>|<level>|<commodity>|<technology>|<mode>' and ‘region’ is '<region>|<node_dest>' (if var=’out’) or '<region>|<node_origin>' (if ‘var=’in’).

  • ’emi’: ‘variable’ is '<var>|<emission>|<technology>|<mode>'.

  • Otherwise: ‘variable’ is '<var>|<technology>'.