Postprocessing and reporting¶

The ix modeling platform provides powerful features to perform calculations and other postprocessing after a message_ix.Scenario has been solved by the associated model. The MESSAGEix framework uses these features to provide zero-configuration reporting of models built on the framework.

These features are accessible through Reporter, which can produce multiple reports from one or more Scenarios. A report is identified by a key (usually a string), and may…

perform arbitrarily complex calculations while intelligently handling units;
read and make use of data that is ‘exogenous’ to (not included in) a Scenario;
produce output as Python or R objects (in code), or to files or databases;
calculate only a requested subset of quantities; and
much, much more!

Contents:

Terminology
Basic usage
Customization
Reporters
Computations
- Computations from ixmp
Configuration
Utilities

Terminology ¶

ixmp.reporting handles numerical quantities, which are scalar (0-dimensional) or array (1 or more dimensions) data with optional associated units. ixmp parameters, scalars, equations, and time-series data all become quantities for the purpose of reporting.

Every quantity and report is identified by a key, which is a str or other hashable object. Special keys are used for multidimensional quantities. For instance: the MESSAGEix parameter resource_cost, defined with the dimensions (node n, commodity c, grade g, year y) is identified by the key 'resource_cost:n-c-g-y'. When summed across the grade/g dimension, it has dimensions n, c, y and is identified by the key 'resource_cost:n-c-y'.

Non-model 1 quantities and reports are produced by computations, which are atomic tasks that build on other computations. The most basic computations—for instance, resource_cost:n-c-g-y—simply retrieve raw/unprocessed data from a message_ix.Scenario and return it as a Quantity. Advanced computations can depend on many quantities, and/or combine quantities together into a structure like a document or spreadsheet. Computations are defined in ixmp.reporting.computations and message_ix.reporting.computations, but most common computations can be added using the methods of Reporter.

1: i.e. quantities that do not exist within the mathematical formulation of the model itself, and do not affect its solution.

Basic usage ¶

A basic reporting workflow has the following steps:

Obtain a message_ix.Scenario object from an ixmp.Platform.
Use from_scenario() to create a Reporter object.
(optionally) Use Reporter built-in methods or advanced features to add computations to the reporter.
Use get() to retrieve the results (or trigger the effects) of one or more computations.

>>> from ixmp import Platform
>>> from message_ix import Scenario, Reporter
>>>
>>> mp = Platform()
>>> scen = Scenario(scen)
>>> rep = Reporter.from_scenario(scen)
>>> rep.get('all')

Note

Reporter stores defined computations, but these are not executed until get() is called—or the results of one computation are required by another. This allows the Reporter to skip unneeded (and potentially slow) computations. A Reporter may contain computations for thousands of model quantities and derived quantities, but a call to get() may only execute a few of these.

Customization ¶

A Reporter prepared with from_scenario() always contains a key scenario, referring to the Scenario to be reported.

The method Reporter.add() can be used to add arbitrary Python code that operates directly on the Scenario object:

>>> def my_custom_report(scenario):
>>>     """Function with custom code that manipulates the *scenario*."""
>>>     print('foo')
>>>
>>> rep.add('custom', (my_custom_report, 'scenario'))
>>> rep.get('custom')
foo

In this example, the function my_custom_report() could run to thousands of lines; read to and write from multiple files; invoke other programs or Python scripts; etc.

In order to take advantage of the performance-optimizing features of the Reporter, however, such calculations can be instead composed from atomic (i.e. small, indivisible) computations.

Reporters ¶

`message_ix.reporting.Reporter`(**kwargs)	MESSAGEix Reporter.
`ixmp.reporting.Reporter`(**kwargs)	Class for generating reports on `ixmp.Scenario` objects.
`ixmp.reporting.Key`(name[, dims, tag])	A hashable key for a quantity that includes its dimensionality.

The ixmp.Reporter automatically adds keys based on the contents of the ixmp.Scenario argument. The message_ix.reporting.Reporter adds additional keys for derived quantities specific to the MESSAGEix model framework. These include:

out: the product of output (output efficiency) and ACT (activity).
out_hist = output × ref_activity (historical reference activity),
in = input × ACT,
in_hist = input × ref_activity,
emi = emission_factor × ACT,
emi_hist = emission_factor × ref_activity,
inv = inv_cost × CAP_NEW,
inv_hist = inv_cost × ref_new_capacity,
fom = fix_cost × CAP,
fom_hist = fix_cost × ref_capacity,
vom = var_cost × ACT, and
vom_hist = var_cost × ref_activity.
tom = fom + vom.
land_out = land_output × LAND,
land_use_qty = land_use × LAND,
land_emi = land_emission × LAND,
addon conversion, the model parameter addon_conversion (note space versus underscore), except broadcast across individual add-on technologies (ta) rather than add-on types (type_addon),
addon up, which is addon_up similarly broadcast.,
addon ACT = addon conversion × ACT,
addon in = input × addon ACT,
addon out = output × addon ACT, and
addon potential = addon up × addon ACT, the maximum potential activity by add-on technology.
price emission, the model variable PRICE_EMISSION broadcast across emission species (e) and technologies (t) rather than types (type_emission, type_tec).

Tip

Use full_key() to retrieve the full-dimensionality Key for any of these quantities.

Other added keys include:

<name>:pyam for the above quantities, plus:
- CAP:pyam (from CAP)
- CAP_NEW:pyam (from CAP_NEW)
These keys return the values in the IAMC data format, as pyam objects.
map_<name> as ‘indicator’ quantities for the mapping sets cat_<name>.
Standard reports message:system, message_costs, and message:emissions.
The report message:default, collecting all of the above reports.

These automatic features of Reporter are controlled by:

`PRODUCTS`	Automatic quantities that are the `product()` of two others.
`DERIVED`	Automatic quantities derived by other calculations.
`MAPPING_SETS`	MESSAGE mapping sets, converted to reporting quantities via `map_as_qty()`.
`PYAM_CONVERT`	Quantities to automatically convert to IAMC format using `as_pyam()`.
`REPORTS`	Automatic reports that `concat()` quantities converted to IAMC format.

class message_ix.reporting.Reporter(**kwargs)¶

Bases: ixmp.reporting.Reporter

MESSAGEix Reporter.

convert_pyam(quantities, year_time_dim, tag='iamc', drop={}, collapse=None, unit=None, replace_vars=None)¶

Add conversion of one or more quantities to IAMC format.

Parameters

quantities (str or Key or list of (str, Key)) – Quantities to transform to pyam/IAMC format.
year_time_dim (str) – Label of the dimension use for the ‘Year’ or ‘Time’ column of the resulting pyam.IamDataFrame. The column is labelled ‘Time’ if year_time_dim=='h', otherwise ‘Year’.
tag (str, optional) – Tag to append to new Keys.
drop (iterable of str, optional) – Label of additional dimensions to drop from the resulting data frame. Dimensions h, y, ya, yr, and yv— except for the one named by year_time_dim—are automatically dropped.
collapse (callable, optional) – Callback to handle additional dimensions of the quantity. A pandas.DataFrame is passed as the sole argument to collapse, which must return a modified dataframe.
unit (str or pint.Unit, optional) – Convert values to these units.
replace_vars (str or Key) – Other reporting key containing a dict mapping variable names to replace.

Returns

Each key converts a Quantity into a pyam.IamDataFrame.

Return type

list of Key

The IAMC data format includes columns named ‘Model’, ‘Scenario’, ‘Region’, ‘Variable’, ‘Unit’; one of ‘Year’ or ‘Time’; and ‘value’.

Using convert_pyam():

‘Model’ and ‘Scenario’ are populated from the attributes of the Scenario returned by the Reporter key scenario;
‘Variable’ contains the name(s) of the quantities;
‘Unit’ contains the units associated with the quantities; and
‘Year’ or ‘Time’ is created according to year_time_dim.

A callback function (collapse) can be supplied that modifies the data before it is converted to an IamDataFrame; for instance, to concatenate extra dimensions into the ‘Variable’ column. Other dimensions can simply be dropped (with drop). Dimensions that are not collapsed or dropped will appear as additional columns in the resulting IamDataFrame; this is valid, but non-standard IAMC data.

For example, here the values for the MESSAGEix technology and mode dimensions are appended to the ‘Variable’ column:

def m_t(df):
    """Callback for collapsing ACT columns."""
    # .pop() removes the named column from the returned row
    df['variable'] = 'Activity|' + df['t'] + '|' + df['m']
    return df

ACT = rep.full_key('ACT')
keys = rep.convert_pyam(ACT, 'ya', collapse=m_t, drop=['t', 'm'])

write(key, path)¶

Compute key and write its value to the file at path.

In addition to the formats handled by ixmp.Reporter.write(), this version will write pyam.IamDataFrame to CSV or Excel files using built-in methods.

class ixmp.reporting.Reporter(**kwargs)¶

Class for generating reports on ixmp.Scenario objects.

A Reporter is used to postprocess data from from one or more ixmp.Scenario objects. The get() method can be used to:

Retrieve individual quantities. A quantity has zero or more dimensions and optional units. Quantities include the ‘parameters’, ‘variables’, ‘equations’, and ‘scalars’ available in an ixmp.Scenario.
Generate an entire report composed of multiple quantities. A report may:
- Read in non-model or exogenous data,
- Trigger output to files(s) or a database, or
- Execute user-defined methods.

Every report and quantity (including the results of intermediate steps) is identified by a utils.Key; all the keys in a Reporter can be listed with keys().

Reporter uses a graph data structure to keep track of computations, the atomic steps in postprocessing: for example, a single calculation that multiplies two quantities to create a third. The graph allows get() to perform only the requested computations. Advanced users may manipulate the graph directly; but common reporting tasks can be handled by using Reporter methods:

`add`(data, args, *kwargs)	General-purpose method to add computations.
`add_file`(path[, key])	Add exogenous quantities from path.
`add_product`(key, *quantities[, sums])	Add a computation that takes the product of quantities.
`aggregate`(qty, tag, dims_or_groups[, …])	Add a computation that aggregates qty.
`apply`(generator, keys, *kwargs)	Add computations by applying generator to keys.
`check_keys`(*keys)	Check that keys are in the Reporter.
`configure`([path])	Configure the Reporter.
`describe`([key, quiet])	Return a string describing the computations that produce key.
`disaggregate`(qty, new_dim[, method, args])	Add a computation that disaggregates qty using method.
`finalize`(scenario)	Prepare the Reporter to act on scenario.
`full_key`(name_or_key)	Return the full-dimensionality key for name_or_key.
`get`([key])	Execute and return the result of the computation key.
`keys`()	Return the keys of `graph`.
`set_filters`(**filters)	Apply filters ex ante (before computations occur).
`visualize`(filename, **kwargs)	Generate an image describing the reporting structure.
`write`(key, path)	Write the report key to the file path.

graph = {'config': {}}¶: A dask-format graph.

add(data, *args, **kwargs)¶

General-purpose method to add computations.

add() can be called in several ways; its behaviour depends on data; see below. It chains to methods such as add_single(), add_queue(), and apply(), which can also be called directly.

Parameters: args (data,) –
Other Parameters: sums (bool, optional) – If True, all partial sums of the key data are also added to the Reporter.
Returns: Some or all of the keys added to the Reporter.
Return type: list of Key-like
Raises: KeyError – If a target key is already in the Reporter; any key referred to by a computation does not exist; or sums=True and the key for one of the partial sums of key is already in the Reporter.

add() may be used to:

Provide an alias from one key to another:

>>> r.add('aliased name', 'original name')

Define an arbitrarily complex computation in a Python function that operates directly on the ixmp.Scenario:

>>> def my_report(scenario):
>>>     # many lines of code
>>>     return 'foo'
>>> r.add('my report', (my_report, 'scenario'))
>>> r.finalize(scenario)
>>> r.get('my report')
foo

Note

Use care when adding literal str values (2); these may conflict with keys that identify the results of other computations.

add_file(path, key=None, **kwargs)¶

Add exogenous quantities from path.

Reporting the key or using it in other computations causes path to be loaded and converted to Quantity.

Parameters

path (os.PathLike) – Path to the file, e.g. ‘/path/to/foo.ext’.
key (str or Key, optional) – Key for the quantity read from the file.

Other Parameters

dims (dict or list or set) – Either a collection of names for dimensions of the quantity, or a mapping from names appearing in the input to dimensions.
units (str or pint.Unit) – Units to apply to the loaded Quantity.

Returns

Either key (if given) or e.g. file:foo.ext based on the path name, without directory components.

Return type

Key

add_load_file(path, key=None, **kwargs)¶

Add exogenous quantities from path.

Reporting the key or using it in other computations causes path to be loaded and converted to Quantity.

Parameters

path (os.PathLike) – Path to the file, e.g. ‘/path/to/foo.ext’.
key (str or Key, optional) – Key for the quantity read from the file.

Other Parameters

dims (dict or list or set) – Either a collection of names for dimensions of the quantity, or a mapping from names appearing in the input to dimensions.
units (str or pint.Unit) – Units to apply to the loaded Quantity.

Returns

Either key (if given) or e.g. file:foo.ext based on the path name, without directory components.

Return type

Key

add_product(key, *quantities, sums=True)¶

Add a computation that takes the product of quantities.

Parameters

key (str or Key) – Key of the new quantity. If a Key, any dimensions are ignored; the dimensions of the product are the union of the dimensions of quantities.
sums (bool, optional) – If True, all partial sums of the new quantity are also added.

Returns

The full key of the new quantity.

Return type

Key

add_queue(queue, max_tries=1, fail='raise')¶

Add tasks from a list or queue.

Parameters

queue (list of 2-tuple) – The members of each tuple are the arguments (i.e. a list or tuple) and keyword arguments (i.e. a dict) to add().
max_tries (int, optional) – Retry adding elements up to this many times.
fail ('raise' or log level, optional) – Action to take when a computation from queue cannot be added after max_tries.

add_single(key, *computation, strict=False, index=False)¶

Add a single computation at key.

Parameters

key (str or Key or hashable) – A string, Key, or other value identifying the output of task.
computation (object) –
Any dask computation, i.e. one of:
1. any existing key in the Reporter.
2. any other literal value or constant.
3. a task, i.e. a tuple with a callable followed by one or more computations.
4. A list containing one or more of #1, #2, and/or #3.
strict (bool, optional) – If True, key must not already exist in the Reporter, and any keys referred to by computation must exist.
index (bool, optional) – If True, key is added to the index as a full-resolution key, so it can be later retrieved with full_key().

aggregate(qty, tag, dims_or_groups, weights=None, keep=True, sums=False)¶

Add a computation that aggregates qty.

Parameters

qty (Key or str) – Key of the quantity to be aggregated.
tag (str) – Additional string to add to the end the key for the aggregated quantity.
dims_or_groups (str or iterable of str or dict) – Name(s) of the dimension(s) to sum over, or nested dict.
weights (xarray.DataArray, optional) – Weights for weighted aggregation.
keep (bool, optional) – Passed to computations.aggregate.
sums (bool, optional) – Passed to add().

Returns

The key of the newly-added node.

Return type

Key

apply(generator, *keys, **kwargs)¶

Add computations by applying generator to keys.

Parameters

generator (callable) – Function to apply to keys.
keys (hashable) – The starting key(s).
kwargs – Keyword arguments to generator.

check_keys(*keys)¶

Check that keys are in the Reporter.

If any of keys is not in the Reporter, KeyError is raised. Otherwise, a list is returned with either the key from keys, or the corresponding full_key().

configure(path=None, **config)¶

Configure the Reporter.

Accepts a path to a configuration file and/or keyword arguments. Configuration keys loaded from file are replaced by keyword arguments.

Valid configuration keys include:

default: the default reporting key; sets default_key.
filters: a dict, passed to set_filters().
files: a list where every element is a dict of keyword arguments to add_file().
alias: a dict mapping aliases to original keys.

Warns: UserWarning – If config contains unrecognized keys.

default_key = None¶: The default reporting key.

describe(key=None, quiet=True)¶

Return a string describing the computations that produce key.

If key is not provided, all keys in the Reporter are described.

The string can be printed to the console, if not quiet.

disaggregate(qty, new_dim, method='shares', args=[])¶

Add a computation that disaggregates qty using method.

Parameters

qty (hashable) – Key of the quantity to be disaggregated.
new_dim (str) – Name of the new dimension of the disaggregated variable.
method (callable or str) – Disaggregation method. If a callable, then it is applied to var with any extra args. If then a method named ‘disaggregate_{method}’ is used.
args (list, optional) – Additional arguments to the method. The first element should be the key for a quantity giving shares for disaggregation.

Returns

The key of the newly-added node.

Return type

Key

finalize(scenario)¶

Prepare the Reporter to act on scenario.

The Scenario object scenario is associated with the key 'scenario'. All subsequent processing will act on data from this scenario.

classmethod from_scenario(scenario, **kwargs)¶

Create a Reporter by introspecting scenario.

Parameters

scenario (ixmp.Scenario) – Scenario to introspect in creating the Reporter.
kwargs (optional) – Passed to Scenario.configure().

Returns

A Reporter instance containing:

A ‘scenario’ key referring to the scenario object.
Each parameter, equation, and variable in the scenario.
All possible aggregations across different sets of dimensions.
Each set in the scenario.

Return type

Reporter

full_key(name_or_key)¶

Return the full-dimensionality key for name_or_key.

An ixmp variable ‘foo’ with dimensions (a, c, n, q, x) is available in the Reporter as 'foo:a-c-n-q-x'. This Key can be retrieved with:

rep.full_key('foo')
rep.full_key('foo:c')
# etc.

get(key=None)¶

Execute and return the result of the computation key.

Only key and its dependencies are computed.

Parameters: key (str, optional) – If not provided, default_key is used.
Raises: ValueError – If key and default_key are both None.

keys()¶: Return the keys of graph.

set_filters(**filters)¶

Apply filters ex ante (before computations occur).

Filters are stored in the reporter at the 'filters' key, and are passed to ixmp.Scenario.par() and similar methods. All quantity values read from the Scenario are filtered before any other computations take place.

Parameters

filters (mapping of str → (list of str or None)) –

Argument names are dimension names; values are lists of allowable labels along the respective dimension, or None to clear any existing filters for the dimension.

If no arguments are provided, all filters are cleared.

property unit_registry¶: The pint.UnitRegistry() used by the Reporter.

visualize(filename, **kwargs)¶

Generate an image describing the reporting structure.

This is a shorthand for dask.visualize(). Requires graphviz.

write(key, path)¶: Write the report key to the file path.

class ixmp.reporting.Key(name, dims=[], tag=None)¶

A hashable key for a quantity that includes its dimensionality.

Quantities in a Scenario can be indexed by one or more dimensions. Keys refer to quantities, using three components:

a string name,
zero or more ordered dimensions dims, and
an optional tag.

For example, an ixmp parameter with three dimensions can be initialized with:

>>> scenario.init_par('foo', ['a', 'b', 'c'], ['apple', 'bird', 'car'])

Key allows a specific, explicit reference to various forms of “foo”:

in its full resolution, i.e. indexed by a, b, and c:

>>> k1 = Key('foo', ['a', 'b', 'c'])
>>> k1
<foo:a-b-c>

in a partial sum over one dimension, e.g. summed across dimension c, with remaining dimensions a and b:
```
>>> k2 = k1.drop('c')
>>> k2
<foo:a-b>
```

in a partial sum over multiple dimensions, etc.:

>>> k1.drop('a', 'c') == k2.drop('a') == 'foo:b'
True

after it has been manipulated by different reporting computations, e.g.

>>> k3 = k1.add_tag('normalized')
>>> k3
<foo:a-b-c:normalized>
>>> k4 = k3.add_tag('rescaled')
>>> k4
<foo:a-b-c:normalized+rescaled>

Notes:

A Key has the same hash, and compares equal to its str representation. repr(key) prints the Key in angle brackets (‘<>’) to signify that it is a Key object.

>>> str(k1)
'foo:a-b-c'
>>> repr(k1)
'<foo:a-b-c>'
>>> hash(k1) == hash('foo:a-b-c')
True

Keys are immutable: the properties name, dims, and tag are read-only, and the methods append(), drop(), and add_tag() return new Key objects.

Keys may be generated concisely by defining a convenience method:

>>> def foo(dims):
>>>     return Key('foo', dims.split())
>>> foo('a b c')
<foo:a-b-c>

add_tag(tag)¶: Return a new Key with tag appended.

append(*dims)¶: Return a new Key with additional dimensions dims.

property dims¶: Dimensions of the quantity, tuple of str.

drop(*dims)¶: Return a new Key with dims dropped.

classmethod from_str_or_key(value, drop=[], append=[], tag=None)¶

Return a new Key from value.

Parameters

value (str or Key) – Value to use to generate a new Key.
drop (list of str or True, optional) – Existing dimensions of value to drop. See drop().
append (list of str, optional.) – New dimensions to append to the returned Key. See append().
tag (str, optional) – Tag for returned Key. If value has a tag, the two are joined using a ‘+’ character. See add_tag().

Returns

Return type

Key

iter_sums()¶: Generate (key, task) for all possible partial sums of the Key.

property name¶: Name of the quantity, str.

classmethod product(new_name, *keys, tag=None)¶

Return a new Key that has the union of dimensions on keys.

Dimensions are ordered by their first appearance:

First, the dimensions of the first of the keys.
Next, any additional dimensions in the second of the keys that were not already added in step 1.
etc.

Parameters: new_name (str) – Name for the new Key. The names of keys are discarded.

property tag¶: Quantity tag, str.

Computations ¶

message_ix.reporting.computations.add(a, b, fill_value=0.0)¶: Sum of a and b.

message_ix.reporting.computations.as_pyam(scenario, quantity, replace_vars=None, year_time_dim=None, drop=[], collapse=None, unit=None)¶

Return a pyam.IamDataFrame containing quantity.

Warnings are logged if the arguments result in additional, unhandled columns in the resulting data frame that are not part of the IAMC spec.

Raises: ValueError – If the resulting data frame has duplicate values in the standard IAMC index columns. pyam.IamDataFrame cannot handle this data.

message_ix.reporting.computations.broadcast_map(quantity, map, rename={})¶

Broadcast quantity using a map.

The map must be a 2-dimensional quantity, such as returned by map_as_qty().

quantity is ‘broadcast’ by multiplying it with the 2-dimensional map, and then dropping the common dimension. The result has the second dimension of map instead of the first.

Parameters: rename (dict (str -> str), optional) – Dimensions to rename on the result.

message_ix.reporting.computations.concat(*args)¶: Concatenate args, which must all be pyam.IamDataFrame.

message_ix.reporting.computations.map_as_qty(set_df, full_set)¶

Convert set_df to a Quantity.

For the MESSAGE sets named cat_* (see Category types and mappings) ixmp.Scenario.set() returns a DataFrame with two columns: the category set (S1) elements and the category member set (S2, also required as the argument full_set) elements.

map_as_qty converts such a DataFrame (set_df) into a Quantity with two dimensions. At the coordinates (s₁, s₂), the value is 1 if s₂ is a mapped from s₁; otherwise 0.

An category named ‘all’, containing all elements of full_set, is added automatically.

See also

broadcast_map()

message_ix.reporting.computations.write_report(quantity, path)¶

Write the report identified by key to the file at path.

If quantity is a pyam.IamDataFrame and path ends with ‘.csv’ or ‘.xlsx’, use pyam methods to write the file to CSV or Excel format, respectively. Otherwise, equivalent to ixmp.reporting.computations.write_report().

Computations from ixmp ¶

Elementary computations for reporting.

Unless otherwise specified, these methods accept and return Quantity objects for data arguments/return values.

Calculations:

`aggregate`(quantity, groups, keep)	Aggregate quantity by groups.
`disaggregate_shares`(quantity, shares)	Disaggregate quantity by shares.
`product`(*quantities)	Return the product of any number of quantities.
`ratio`(numerator, denominator)	Return the ratio numerator / denominator.
`sum`(quantity[, weights, dimensions])	Sum quantity over dimensions, with optional weights.

Input and output:

`load_file`(path[, dims, units])	Read the file at path and return its contents as a `Quantity`.
`write_report`(quantity, path)	Write a quantity to a file.

Data manipulation:

concat(*objs, **kwargs)

Concatenate Quantity objs.

ixmp.reporting.computations.aggregate(quantity, groups, keep)¶

Aggregate quantity by groups.

Parameters

quantity (Quantity) –
groups (dict of dict) – Top-level keys are the names of dimensions in quantity. Second-level keys are group names; second-level values are lists of labels along the dimension to sum into a group.
keep (bool) – If True, the members that are aggregated into a group are returned with the group sums. If False, they are discarded.

Returns

Same dimensionality as quantity.

Return type

Quantity

ixmp.reporting.computations.apply_units(qty, units, quiet=False)¶

Simply apply units to qty.

Logs on level WARNING if qty already has existing units.

Parameters

qty (Quantity) –
units (str or pint.Unit) – Units to apply to qty
quiet (bool, optional) – If True log on level DEBUG.

ixmp.reporting.computations.concat(*objs, **kwargs)¶

Concatenate Quantity objs.

Any strings included amongst args are discarded, with a logged warning; these usually indicate that a quantity is referenced which is not in the Reporter.

ixmp.reporting.computations.data_for_quantity(ix_type, name, column, scenario, config)¶

Retrieve data from scenario.

Parameters

ix_type ('equ' or 'par' or 'var') – Type of the ixmp object.
name (str) – Name of the ixmp object.
column ('mrg' or 'lvl' or 'value') – Data to retrieve. ‘mrg’ and ‘lvl’ are valid only for ix_type='equ', and ‘level’ otherwise.
scenario (ixmp.Scenario) – Scenario containing data to be retrieved.
config (dict of (str -> dict)) – The key ‘filters’ may contain a mapping from dimensions to iterables of allowed values along each dimension. The key ‘units’/’apply’ may contain units to apply to the quantity; any such units overwrite existing units, without conversion.

Returns

Data for name.

Return type

Quantity

ixmp.reporting.computations.disaggregate_shares(quantity, shares)¶: Disaggregate quantity by shares.

ixmp.reporting.computations.load_file(path, dims={}, units=None)¶

Read the file at path and return its contents as a Quantity.

Some file formats are automatically converted into objects for direct use in reporting code:

.csv:: Converted to Quantity. CSV files must have a ‘value’ column; all others are treated as indices, except as given by dims. Lines beginning with ‘#’ are ignored.

Parameters

path (pathlib.Path) – Path to the file to read.
dims (collections.abc.Collection or collections.abc.Mapping, optional) – If a collection of names, other columns besides these and ‘value’ are discarded. If a mapping, the keys are the column labels in path, and the values are the target dimension names.
units (str or pint.Unit) – Units to apply to the loaded Quantity.

ixmp.reporting.computations.product(*quantities)¶: Return the product of any number of quantities.

ixmp.reporting.computations.ratio(numerator, denominator)¶

Return the ratio numerator / denominator.

Parameters

numerator (Quantity) –
denominator (Quantity) –

ixmp.reporting.computations.select(qty, indexers, inverse=False)¶

Select from qty based on indexers.

Parameters

qty (Quantity) –
select (dict (str -> list of str)) – Elements to be selected from qty. Mapping from dimension names to labels along each dimension.
inverse (bool, optional) – If True, remove the items in indexers instead of keeping them.

ixmp.reporting.computations.sum(quantity, weights=None, dimensions=None)¶

Sum quantity over dimensions, with optional weights.

Parameters

quantity (Quantity) –
weights (Quantity, optional) – If dimensions is given, weights must have at least these dimensions. Otherwise, any dimensions are valid.
dimensions (list of str, optional) – If not provided, sum over all dimensions. If provided, sum over these dimensions.

ixmp.reporting.computations.write_report(quantity, path)¶

Write a quantity to a file.

Parameters: path (str or Path) – Path to the file to be written.

Configuration ¶

`ixmp.reporting.configure`([path])	Configure reporting globally.
`ixmp.reporting.utils.RENAME_DIMS`	Dimensions to rename when extracting raw data from Scenario objects.
`ixmp.reporting.utils.REPLACE_UNITS`	Replacements to apply to quantity units before parsing by pint.

reporting.configure(**config)¶

Configure reporting globally.

Modifies global variables that affect the behaviour of all Reporters and computations, namely RENAME_DIMS and REPLACE_UNITS.

Valid configuration keys—passed as config keyword arguments—include:

Other Parameters

units (mapping) – Configuration for handling of units. Valid sub-keys include:
- replace (mapping of str -> str): replace units before they are parsed by pint. Added to REPLACE_UNITS.
- define (str): block of unit definitions, added to the pint application registry so that units are recognized. See the pint documentation on defining units.
rename_dims (mapping of str -> str) – Update RENAME_DIMS.

Warns

UserWarning – If config contains unrecognized keys.

message_ix.reporting.PRODUCTS = (('out', ('output', 'ACT')), ('in', ('input', 'ACT')), ('rel', ('relation_activity', 'ACT')), ('emi', ('emission_factor', 'ACT')), ('inv', ('inv_cost', 'CAP_NEW')), ('fom', ('fix_cost', 'CAP')), ('vom', ('var_cost', 'ACT')), ('land_out', ('land_output', 'LAND')), ('land_use_qty', ('land_use', 'LAND')), ('land_emi', ('land_emission', 'LAND')), ('addon ACT', ('addon conversion', 'ACT')), ('addon in', ('input', 'addon ACT')), ('addon out', ('output', 'addon ACT')), ('addon potential', ('addon up', 'addon ACT')))¶: Automatic quantities that are the product() of two others.

message_ix.reporting.DERIVED = [('tom:nl-t-yv-ya', (<function add>, 'fom:nl-t-yv-ya', 'vom:nl-t-yv-ya')), ('addon conversion:nl-t-yv-ya-m-h-ta', (functools.partial(<function broadcast_map>, rename={'n': 'nl'}), 'addon_conversion:n-t-yv-ya-m-h-type_addon', 'map_addon')), ('addon up:nl-t-ya-m-h-ta', (functools.partial(<function broadcast_map>, rename={'n': 'nl'}), 'addon_up:n-t-ya-m-h-type_addon', 'map_addon')), ('price emission:n-e-t-y', (<function broadcast_map>, (<function broadcast_map>, 'PRICE_EMISSION:n-type_emission-type_tec-y', 'map_emission'), 'map_tec'))]¶: Automatic quantities derived by other calculations.

message_ix.reporting.PYAM_CONVERT = [('out:nl-t-ya-m-nd-c-l', 'ya', {'kind': 'ene', 'var': 'out'}), ('in:nl-t-ya-m-no-c-l', 'ya', {'kind': 'ene', 'var': 'in'}), ('CAP:nl-t-ya', 'ya', {'var': 'capacity'}), ('CAP_NEW:nl-t-yv', 'yv', {'var': 'new capacity'}), ('inv:nl-t-yv', 'yv', {'var': 'inv cost'}), ('fom:nl-t-ya', 'ya', {'var': 'fom cost'}), ('vom:nl-t-ya', 'ya', {'var': 'vom cost'}), ('tom:nl-t-ya', 'ya', {'var': 'total om cost'}), ('emi:nl-t-ya-m-e', 'ya', {'kind': 'emi', 'var': 'emis'})]¶: Quantities to automatically convert to IAMC format using as_pyam().

message_ix.reporting.REPORTS = {'message:costs': ['inv:pyam', 'fom:pyam', 'vom:pyam', 'tom:pyam'], 'message:emissions': ['emi:pyam'], 'message:system': ['out:pyam', 'in:pyam', 'CAP:pyam', 'CAP_NEW:pyam']}¶: Automatic reports that concat() quantities converted to IAMC format.

message_ix.reporting.MAPPING_SETS = [('addon', 't'), ('emission', 'e'), ('tec', 't'), ('year', 'y')]¶

MESSAGE mapping sets, converted to reporting quantities via map_as_qty().

For instance, the mapping set cat_addon is available at the reporting key map_addon, which produces a Quantity with the two dimensions type_addon and ta (short form of technology_addon). This Quantity contains the value 1 at every valid (type_addon, ta) location, and 0 elsewhere.

ixmp.reporting.utils.RENAME_DIMS = {'commodity': 'c', 'emission': 'e', 'grade': 'g', 'land_scenario': 's', 'land_type': 'u', 'level': 'l', 'mode': 'm', 'node': 'n', 'node_dest': 'nd', 'node_loc': 'nl', 'node_origin': 'no', 'node_rel': 'nr', 'node_share': 'ns', 'rating': 'q', 'relation': 'r', 'technology': 't', 'technology_addon': 'ta', 'technology_primary': 'tp', 'time': 'h', 'time_dest': 'hd', 'time_origin': 'ho', 'year': 'y', 'year_act': 'ya', 'year_rel': 'yr', 'year_vtg': 'yv'}¶: Dimensions to rename when extracting raw data from Scenario objects. Mapping from Scenario dimension name -> preferred dimension name. message_ix adds the standard short symbols for MESSAGE sets to this variable.

ixmp.reporting.utils.REPLACE_UNITS = {'%': 'percent'}¶: Replacements to apply to quantity units before parsing by pint. Mapping from original unit -> preferred unit.

Utilities ¶

class ixmp.reporting.quantity.AttrSeries(data=None, *args, name=None, units=None, attrs=None, **kwargs)¶

pandas.Series subclass imitating xarray.DataArray.

Future versions of ixmp.reporting will use xarray.DataArray as Quantity; however, because xarray currently lacks sparse matrix support, ixmp quantities may be too large for available memory.

The AttrSeries class provides similar methods and behaviour to xarray.DataArray, so that ixmp.reporting.computations methods can use xarray-like syntax.

Parameters

units (str or pint.Unit, optional) – Set the units attribute. The value is converted to pint.Unit and added to attrs.
attrs (Mapping, optional) – Set the attrs of the AttrSeries. This attribute was added in pandas 1.0, but is not currently supported by the Series constructor.

ixmp.reporting.utils.clean_units(input_string)¶

Tolerate messy strings for units.

Handles two specific cases found in MESSAGEix test cases:

Dimensions enclosed in ‘[]’ have these characters stripped.
The ‘%’ symbol cannot be supported by pint, because it is a Python operator; it is translated to ‘percent’.

ixmp.reporting.utils.collect_units(*args)¶: Return an list of ‘_unit’ attributes for args.

ixmp.reporting.utils.dims_for_qty(data)¶

Return the list of dimensions for data.

If data is a pandas.DataFrame, its columns are processed; otherwise it must be a list.

ixmp.reporting.RENAME_DIMS is used to rename dimensions.

ixmp.reporting.utils.filter_concat_args(args)¶

Filter out str and Key from args.

A warning is logged for each element removed.

ixmp.reporting.utils.parse_units(units_series)¶: Return a pint.Unit for a pd.Series of strings.

message_ix.reporting.pyam.collapse_message_cols(df, var, kind=None)¶

as_pyam() collapse=… callback for MESSAGE quantities.

Parameters

var (str) – Name for ‘variable’ column.
kind (None or 'ene' or 'emi', optional) –
Determines which other columns are combined into the ‘region’ and ‘variable’ columns:
- ’ene’: ‘variable’ is '<var>|<level>|<commodity>|<technology>|<mode>' and ‘region’ is '<region>|<node_dest>' (if var=’out’) or '<region>|<node_origin>' (if ‘var=’in’).
- ’emi’: ‘variable’ is '<var>|<emission>|<technology>|<mode>'.
- Otherwise: ‘variable’ is '<var>|<technology>'.
The referenced columns are also dropped, so it is not necessary to provide the drop argument of as_pyam().