Reporting¶
Warning
ixmp.reporting
is experimental in ixmp 2.0.
The API and functionality may change without advance notice or a deprecation period in subsequent releases.
Top-level methods and classes:
configure ([path]) |
Configure reporting globally. |
Reporter (**kwargs) |
Class for generating reports on ixmp.Scenario objects. |
Key (name[, dims, tag]) |
A hashable key for a quantity that includes its dimensionality. |
Others:
-
reporting.
configure
(**config)¶ Configure reporting globally.
Modifies global variables that affect the behaviour of all Reporters and computations, namely
RENAME_DIMS
,REPLACE_UNITS
, andUNITS
.Valid configuration keys—passed as config keyword arguments—include:
Other Parameters: - units (mapping) – Configuration for handling of units. Valid sub-keys include:
- replace (mapping of str -> str): replace units before they are
parsed by pint. Added to
REPLACE_UNITS
. - define (
str
): block of unit definitions, added toUNITS
so that units are recognized by pint. See the pint documentation.
- replace (mapping of str -> str): replace units before they are
parsed by pint. Added to
- rename_dims (mapping of str -> str) – Update
RENAME_DIMS
.
Warns: UserWarning – If config contains unrecognized keys.
- units (mapping) – Configuration for handling of units. Valid sub-keys include:
-
class
ixmp.reporting.
Reporter
(**kwargs)¶ Class for generating reports on
ixmp.Scenario
objects.A Reporter is used to postprocess data from from one or more
ixmp.Scenario
objects. Theget()
method can be used to:Retrieve individual quantities. A quantity has zero or more dimensions and optional units. Quantities include the ‘parameters’, ‘variables’, ‘equations’, and ‘scalars’ available in an
ixmp.Scenario
.Generate an entire report composed of multiple quantities. A report may:
- Read in non-model or exogenous data,
- Trigger output to files(s) or a database, or
- Execute user-defined methods.
Every report and quantity (including the results of intermediate steps) is identified by a
utils.Key
; all the keys in a Reporter can be listed withkeys()
.Reporter uses a graph data structure to keep track of computations, the atomic steps in postprocessing: for example, a single calculation that multiplies two quantities to create a third. The graph allows
get()
to perform only the requested computations. Advanced users may manipulate the graph directly; but common reporting tasks can be handled by using Reporter methods:add
(key, computation[, strict, index, sums])Add computation to the Reporter under key. add_file
(path[, key])Add exogenous quantities from path. add_product
(name, *quantities[, sums])Add a computation that takes the product of quantities. aggregate
(qty, tag, dims_or_groups[, …])Add a computation that aggregates qty. apply
(generator, *keys)Add computations from generator applied to key. check_keys
(*keys)Check that keys are in the Reporter. configure
([path])Configure the Reporter. describe
([key, quiet])Return a string describing the computations that produce key. disaggregate
(qty, new_dim[, method, args])Add a computation that disaggregates qty using method. finalize
(scenario)Prepare the Reporter to act on scenario. full_key
(name_or_key)Return the full-dimensionality key for name_or_key. get
([key])Execute and return the result of the computation key. keys
()read_config
(path)Configure the Reporter with information from a YAML file at path. set_filters
(**filters)Apply filters ex ante (before computations occur). visualize
(filename, **kwargs)Generate an image describing the reporting structure. write
(key, path)Write the report key to the file path. -
add
(key, computation, strict=False, index=False, sums=False)¶ Add computation to the Reporter under key.
Parameters: - key (hashable) – A string, Key, or other value identifying the output of task.
- computation (object) –
One of:
- any existing key in the Reporter.
- any other literal value or constant.
- a task, i.e. a tuple with a callable followed by one or more computations.
- A list containing one or more of #1, #2, and/or #3.
- strict (bool, optional) – If True, key must not already exist in the Reporter, and any keys referred to by computation must exist.
- index (bool, optional) – If True, key is added to the index as a full-resolution key, so
it can be later retrieved with
full_key()
. - sums (bool, optional) – If True, all partial sums of key are also added to the Reporter.
Raises: KeyError
– If key is already in the Reporter; any key referred to by computation does not exist; orsums=True
and the key for one of the partial sums of key is already in the Reporter.add()
may be used to:Provide an alias from one key to another:
>>> from message_ix.reporting import Reporter >>> rep = Reporter() # Create a new Reporter object >>> rep.add('aliased name', 'original name')
Define an arbitrarily complex computation in a Python function that operates directly on the
ixmp.Scenario
:>>> def my_report(scenario): >>> # many lines of code >>> return 'foo' >>> rep.add('my report', (my_report, 'scenario')) >>> rep.finalize(scenario) >>> rep.get('my report') foo
Note
Use care when adding literal
str()
values as a computation argument foradd()
; these may conflict with keys that identify the results of other computations.
-
add_file
(path, key=None)¶ Add exogenous quantities from path.
A file at a path like ‘/path/to/foo.ext’ is added at the key
'file:foo.ext'
.
-
add_product
(name, *quantities, sums=True)¶ Add a computation that takes the product of quantities.
Parameters: Returns: The full key of the new quantity.
Return type:
-
aggregate
(qty, tag, dims_or_groups, weights=None, keep=True, sums=False)¶ Add a computation that aggregates qty.
Parameters: - qty (
Key
or str) – Key of the quantity to be aggregated. - tag (str) – Additional string to add to the end the key for the aggregated quantity.
- dims_or_groups (str or iterable of str or dict) – Name(s) of the dimension(s) to sum over, or nested dict.
- weights (
xarray.DataArray
, optional) – Weights for weighted aggregation. - keep (bool, optional) – Passed to
computations.aggregate
. - sums (bool, optional) – Passed to
add()
.
Returns: The key of the newly-added node.
Return type: - qty (
-
apply
(generator, *keys)¶ Add computations from generator applied to key.
Parameters: - generator (callable) – Function to apply to keys. Must yield a sequence (0 or more) of
(key, computation), which are added to the
graph
. - keys (hashable) – The starting key(s)
- generator (callable) – Function to apply to keys. Must yield a sequence (0 or more) of
(key, computation), which are added to the
-
check_keys
(*keys)¶ Check that keys are in the Reporter.
If any of keys is not in the Reporter, KeyError is raised. Otherwise, a list is returned with either the key from keys, or the corresponding
full_key()
.
-
configure
(path=None, **config)¶ Configure the Reporter.
Accepts a path to a configuration file and/or keyword arguments. Configuration keys loaded from file are replaced by keyword arguments.
Valid configuration keys include:
- default: the default reporting key; sets
default_key
. - filters: a
dict
, passed toset_filters()
. - files: a
dict
mapping keys to file paths. - alias: a
dict
mapping aliases to original keys.
Warns: UserWarning – If config contains unrecognized keys. - default: the default reporting key; sets
-
default_key
= None¶ The default reporting key.
-
describe
(key=None, quiet=False)¶ Return a string describing the computations that produce key.
If key is not provided, all keys in the Reporter are described.
The string is also printed. If quiet, do not print to the console.
-
disaggregate
(qty, new_dim, method='shares', args=[])¶ Add a computation that disaggregates qty using method.
Parameters: - qty (hashable) – Key of the quantity to be disaggregated.
- new_dim (str) – Name of the new dimension of the disaggregated variable.
- method (callable or str) – Disaggregation method. If a callable, then it is applied to var with any extra args. If then a method named ‘disaggregate_{method}’ is used.
- args (list, optional) – Additional arguments to the method. The first element should be the key for a quantity giving shares for disaggregation.
Returns: The key of the newly-added node.
Return type:
-
finalize
(scenario)¶ Prepare the Reporter to act on scenario.
The
Scenario
object scenario is associated with the key'scenario'
. All subsequent processing will act on data from this scenario.
-
classmethod
from_scenario
(scenario, **kwargs)¶ Create a Reporter by introspecting scenario.
Parameters: - scenario (ixmp.Scenario) – Scenario to introspect in creating the Reporter.
- kwargs (optional) – Passed to
Scenario.configure()
.
Returns: A Reporter instance containing:
- A ‘scenario’ key referring to the scenario object.
- Each parameter, equation, and variable in the scenario.
- All possible aggregations across different sets of dimensions.
- Each set in the scenario.
Return type:
-
full_key
(name_or_key)¶ Return the full-dimensionality key for name_or_key.
An ixmp variable ‘foo’ with dimensions (a, c, n, q, x) is available in the Reporter as
'foo:a-c-n-q-x'
. ThisKey
can be retrieved with:rep.full_key('foo') rep.full_key('foo:c') # etc.
-
get
(key=None)¶ Execute and return the result of the computation key.
Only key and its dependencies are computed.
Parameters: key (str, optional) – If not provided, default_key
is used.Raises: ValueError
– If key anddefault_key
are bothNone
.
-
read_config
(path)¶ Configure the Reporter with information from a YAML file at path.
See
configure()
.
-
set_filters
(**filters)¶ Apply filters ex ante (before computations occur).
Filters are stored in the reporter at the
'filters'
key, and are passed toixmp.Scenario.par()
and similar methods. All quantity values read from the Scenario are filtered before any other computations take place.Parameters: filters (mapping of str → (list of str or None)) – Argument names are dimension names; values are lists of allowable labels along the respective dimension, or None to clear any existing filters for the dimension.
If no arguments are provided, all filters are cleared.
-
visualize
(filename, **kwargs)¶ Generate an image describing the reporting structure.
This is a shorthand for
dask.visualize()
. Requires graphviz.
-
write
(key, path)¶ Write the report key to the file path.
-
class
ixmp.reporting.
Key
(name, dims=[], tag=None)¶ A hashable key for a quantity that includes its dimensionality.
Quantities in a
Scenario
can be indexed by one or more dimensions. A Key refers to a quantity using three components:For example, an ixmp parameter with three dimensions can be initialized with:
>>> scenario.init_par('foo', ['a', 'b', 'c'], ['apple', 'bird', 'car'])
Key allows a specific, explicit reference to various forms of “foo”:
in its full resolution, i.e. indexed by a, b, and c:
>>> k1 = Key('foo', ['a', 'b', 'c']) >>> k1 == 'foo:a-b-c' True
Notice that a Key has the same hash, and compares equal (==) to its
str()
.in a partial sum over one dimension, e.g. summed along c with dimensions a and b:
>>> k2 = k1.drop('c') >>> k2 == 'foo:a-b' True
in a partial sum over multiple dimensions, etc.:
>>> k1.drop('a', 'c') == k2.drop('a') == 'foo:b' True
Note
Some remarks:
repr(key)
prints the Key in angle brackets (‘<>’) to signify it is a Key object.>>> repr(k1) <foo:a-b-c>
Keys are immutable: the properties
name
,dims
, andtag
are read-only, and the methodsappend()
,drop()
, andadd_tag()
return new Key objects.Keys may be generated concisely by defining a convenience method:
>>> def foo(dims): >>> return Key('foo', dims.split()) >>> foo('a b c') foo:a-b-c
-
add_tag
(tag)¶ Return a new Key with tag appended.
-
append
(*dims)¶ Return a new Key with additional dimensions dims.
-
drop
(*dims)¶ Return a new Key with dims dropped.
-
classmethod
from_str_or_key
(value, drop=[], append=[], tag=None)¶ Return a new Key from value.
Parameters: - value (str or Key) – Value to use to generate a new Key.
- drop (list of str, optional) – Existing dimensions of value to drop. See
drop()
. - append (list of str, optional.) – New dimensions to append to the returned Key. See
append()
. - tag (str, optional) – Tag for returned Key. If value has a tag, the two are joined
using a ‘+’ character. See
add_tag()
.
Returns: Return type:
-
iter_sums
()¶ Generate (key, task) for all possible partial sums of the Key.
-
classmethod
product
(new_name, *keys)¶ Return a new Key that has the union of dimensions on keys.
Dimensions are ordered by their first appearance:
- First, the dimensions of the first of the keys.
- Next, any additional dimensions in the second of the keys that were not already added in step 1.
- etc.
Parameters: new_name (str) – Name for the new Key. The names of keys are discarded.
Computations¶
Elementary computations for reporting.
Unless otherwise specified, these methods accept and return
Quantity
objects for data
arguments/return values.
Calculations:
aggregate (quantity, groups, keep) |
Aggregate quantity by groups. |
disaggregate_shares (quantity, shares) |
Disaggregate quantity by shares. |
product (*quantities) |
Return the product of any number of quantities. |
ratio (numerator, denominator) |
Return the ratio numerator / denominator. |
sum (quantity[, weights, dimensions]) |
Sum quantity over dimensions, with optional weights. |
Input and output:
load_file (path) |
Read the file at path and return its contents. |
write_report (quantity, path) |
Write a quantity to a file. |
Data manipulation:
concat (*objs, **kwargs) |
Concatenate Quantity objs. |
-
ixmp.reporting.computations.
aggregate
(quantity, groups, keep)¶ Aggregate quantity by groups.
Parameters: - quantity (
Quantity
) – - groups (dict of dict) – Top-level keys are the names of dimensions in quantity. Second-level keys are group names; second-level values are lists of labels along the dimension to sum into a group.
- keep (bool) – If True, the members that are aggregated into a group are returned with the group sums. If False, they are discarded.
Returns: Same dimensionality as quantity.
Return type: Quantity
- quantity (
-
ixmp.reporting.computations.
concat
(*objs, **kwargs)¶ Concatenate Quantity objs.
Any strings included amongst args are discarded, with a logged warning; these usually indicate that a quantity is referenced which is not in the Reporter.
Disaggregate quantity by shares.
-
ixmp.reporting.computations.
load_file
(path)¶ Read the file at path and return its contents.
Some file formats are automatically converted into objects for direct use in reporting code:
- csv: converted to
xarray.DataArray
. CSV files must have a ‘value’ column; all others are treated as indices.
- csv: converted to
-
ixmp.reporting.computations.
product
(*quantities)¶ Return the product of any number of quantities.
-
ixmp.reporting.computations.
ratio
(numerator, denominator)¶ Return the ratio numerator / denominator.
-
ixmp.reporting.computations.
sum
(quantity, weights=None, dimensions=None)¶ Sum quantity over dimensions, with optional weights.
Utilities¶
-
class
ixmp.reporting.attrseries.
AttrSeries
(*args, **kwargs)¶ pandas.Series
subclass imitatingxarray.DataArray
.Future versions of
ixmp.reporting
will usexarray.DataArray
asQuantity
; however, becausexarray
currently lacks sparse matrix support, ixmp quantities may be too large for available memory.The AttrSeries class provides similar methods and behaviour to
xarray.DataArray
, such as an attrs dictionary for metadata, so thatixmp.reporting.computations
methods can use xarray-like syntax.
-
ixmp.reporting.utils.
RENAME_DIMS
= {}¶ Dimensions to rename when extracting raw data from Scenario objects. Mapping from Scenario dimension name -> preferred dimension name.
-
ixmp.reporting.utils.
REPLACE_UNITS
= {'%': 'percent'}¶ Replacements to apply to quantity units before parsing by pint. Mapping from original unit -> preferred unit.
-
ixmp.reporting.utils.
UNITS
= <pint.registry.UnitRegistry object>¶ pint unit registry for processing quantity units. All units handled by
ixmp.reporting
must be either standard SI units, or added to this registry.
-
ixmp.reporting.utils.
as_attrseries
(obj)¶ Convert obj to an AttrSeries.
-
ixmp.reporting.utils.
as_quantity
(obj)¶ Convert obj to an AttrSeries.
-
ixmp.reporting.utils.
as_sparse_xarray
(obj)¶ Convert obj to an
xarray.DataArray
with sparse.COO storage.
-
ixmp.reporting.utils.
clean_units
(input_string)¶ Tolerate messy strings for units.
Handles two specific cases found in MESSAGEix test cases:
- Dimensions enclosed in ‘[]’ have these characters stripped.
- The ‘%’ symbol cannot be supported by pint, because it is a Python operator; it is translated to ‘percent’.
-
ixmp.reporting.utils.
collect_units
(*args)¶ Return an list of ‘_unit’ attributes for args.
-
ixmp.reporting.utils.
data_for_quantity
(ix_type, name, column, scenario, filters=None)¶ Retrieve data from scenario.
Parameters: - ix_type ('equ' or 'par' or 'var') – Type of the ixmp object.
- name (str) – Name of the ixmp object.
- column ('mrg' or 'lvl' or 'value') – Data to retrieve. ‘mrg’ and ‘lvl’ are valid only for
ix_type='equ'
, and ‘level’ otherwise. - scenario (ixmp.Scenario) – Scenario containing data to be retrieved.
- filters (dict, optional) – Mapping from dimensions to iterables of allowed values along each dimension.
Returns: Data for name.
Return type: Quantity
-
ixmp.reporting.utils.
dims_for_qty
(data)¶ Return the list of dimensions for data.
If data is a
pandas.DataFrame
, its columns are processed; otherwise it must be a list.ixmp.reporting.RENAME_DIMS is used to rename dimensions.
-
ixmp.reporting.utils.
filter_concat_args
(args)¶ Filter out str and Key from args.
A warning is logged for each element removed.
-
ixmp.reporting.utils.
keys_for_quantity
(ix_type, name, scenario)¶ Iterate over keys for name in scenario.