Reporting (`report`)

Introduction 

See the discussion in the MESSAGEix docs about the stack. In short, message_ix must not contain reporting code that references coal_ppl, because not every model built on the MESSAGE framework will have a technology with this name. Any reporting specific to coal_ppl must be in message_ix_models, since all models in the MESSAGEix-GLOBIOM family will have this technology.

The basic design pattern of message_ix_models.report is:

A global.yaml file (i.e. in YAML format) that contains a concise yet explicit description of the reporting computations needed for a MESSAGE-GLOBIOM model.
prepare_reporter() reads the file and a Scenario object, and uses it to populate a new Reporter. This function mostly relies on the configuration handlers built in to Genno to handle the different sections of the file.

Features 

By combining these genno, ixmp, message_ix, and message_ix_models features, the following functionality is provided.

Note

If any of this does not appear to work as advertised, file a bug!

Units 

Are read automatically for ixmp parameters.
Pass through calculations/are derived automatically.
Are recognized based on the definitions of non-SI units from IAMconsortium/units.
Are discarded when inconsistent.
Can be overridden for entire parameters:
```
units:
  apply:
    inv_cost: USD
```

Can be set explicitly when converting data to IAMC format:

iamc:
# 'value' will be in kJ; 'units' will be the string 'kJ'
- variable: Variable Name
  base: example_var:a-b-c
  units: kJ

Continuous reporting 

Note

This section is no longer current.

The IIASA TeamCity build server is configured to automatically run the full (global.yaml) reporting on the following scenarios:

This takes place:

every morning at 07:00 IIASA time, and
for every commit on every pull request branch, if the branch name includes report anywhere, e.g. feature/improve-reporting.

The results are output to Excel files that are preserved and made available as ‘build artifacts’ via the TeamCity web interface.

API reference 

`prepare_reporter`(context[, scenario, reporter])	Return a `Reporter` and key prepared to report a `Scenario`.
`register`(name_or_callback)	Register a callback function for `prepare_reporter()`.
`report`(context, args, *kwargs)	Run complete reporting on a `message_ix.Scenario`.

message_ix_models.report.prepare_reporter(context: Context, scenario: Scenario | None = None, reporter: Reporter | None = None) → Tuple[Reporter, Key][source]

Return a Reporter and key prepared to report a Scenario.

Parameters:

context (The code responds to the following settings on) – Containing settings in the report/* tree.
scenario (message_ix.Scenario, optional) – Scenario to report. If not given, Context.get_scenario() is used to retrieve a Scenario.
reporter (.Reporter, optional) – Existing reporter to extend with computations. If not given, it is created using Reporter.from_scenario().
context –
list-table:: (..) –

width:

100%

widths:

25 25 50

header-rows:

1
- - Setting
  - Type
  - Description
- - scenario_info
  - Identifies the (solved) scenario to be reported.
- - report/key
  - str or ixmp.reporting.Key
  - Quantity or node to compute. The computation is not triggered (i.e. get is not called); but the corresponding, full-resolution Key, if any, is returned.
- - report/config
  - dict or Path-like or None
  - If dict, then this is passed to Reporter.configure(). If Path-like, then this is the path to the reporting configuration file. If not given, defaults to report/global.yaml.
- - report/output_path
  - Path-like, optional
  - Path to write reporting outputs. If given, a computation cli-output is added to the Reporter which writes report/key to this path.

Returns:

.Reporter – Reporter prepared with MESSAGEix-GLOBIOM calculations; if reporter is given, this is a reference to the same object.
.Key – Same as context.report["key"] if any, but in full resolution; else one of default or cli-output according to the other settings.

message_ix_models.report.register(name_or_callback: Callable | str) → str | None[source]

Register a callback function for prepare_reporter().

Each registered function is called by prepare_reporter(), in order to add or modify reporting keys. Specific model variants and projects can register a callback to extend the reporting graph.

Callback functions must take two arguments: the Reporter, and a Context:

from message_ix.reporting import Reporter
from message_ix_models import Context
from message_ix_models.report import register

def cb(rep: Reporter, ctx: Context):
    # Modify `rep` by calling its methods ...
    pass

register(cb)

Parameters:: name_or_callback – If a string, this may be a submodule of message_ix_models, or message_data, in which case the function {message_data,message_ix_models}.{name}.report.callback is used. Or, it may be a fully-resolved package/module name, in which case {name}.callback is used. If a callable (function), it is used directly.

message_ix_models.report.report(context: Context, *args, **kwargs)[source]

Run complete reporting on a message_ix.Scenario.

This function provides a single, common interface to call both the ‘new’ (reporting) and ‘legacy’ (tools.post_processing) reporting codes.

The code responds to the following settings on context:

Setting	Type	Description
scenario_info		Identifies the (solved) scenario to be reported.
report/dry_run	bool	Only show what would be done. Default: `False`.
report/legacy	dict or None	If given, the old-style reporting in `iamc_report_hackathon` is used, with legacy as keyword arguments.

As well:

report/key is set to default, if not set.
report/config is set to report/globa.yaml, if not set.

Operators 

Atomic reporting operations for MESSAGEix-GLOBIOM.

message_ix_models provides the following:

`from_url`(url)	Return a `ixmp.TimeSeries` given its url.
`get_ts`(scenario[, filters, iamc, subannual])	Retrieve timeseries data from scenario.
`gwp_factors`()	Use `iam_units` to generate a Quantity of GWP factors.
`make_output_path`(config, name)	Return a path under the "output_dir" Path from the reporter configuration.
`model_periods`(y, cat_year)	Return the elements of y beyond the firstmodelyear of cat_year.
`remove_ts`(scenario, config[, after, dump])	Remove all time series data from scenario.
`share_curtailment`(curt, *parts)	Apply a share of curt to the first of parts.

Other operators are provided by:

message_ix_models.report.computations.from_url(url: str) → TimeSeries[source]: Return a ixmp.TimeSeries given its url.

message_ix_models.report.computations.get_ts(scenario: Scenario, filters: dict | None = None, iamc: bool = False, subannual: bool | str = 'auto')[source]

Retrieve timeseries data from scenario.

Corresponds to ixmp.Scenario.timeseries().

Todo

Move upstream, e.g. to ixmp alongside store_ts().

message_ix_models.report.computations.gwp_factors()[source]

Use iam_units to generate a Quantity of GWP factors.

The quantity is dimensionless, e.g. for converting [mass] to [mass], andhas dimensions:

‘gwp metric’: the name of a GWP metric, e.g. ‘SAR’, ‘AR4’, ‘AR5’. All metrics are
on a 100-year basis.
‘e’: emissions species, as in MESSAGE. The entry ‘HFC’ is added as an alias for the species ‘HFC134a’ from iam_units.
‘e equivalent’: GWP-equivalent species, always ‘CO2’.

message_ix_models.report.computations.make_output_path(config, name)[source]: Return a path under the “output_dir” Path from the reporter configuration.

message_ix_models.report.computations.model_periods(y: List[int], cat_year: DataFrame) → List[int][source]: Return the elements of y beyond the firstmodelyear of cat_year.

Todo

Move upstream, to message_ix.

message_ix_models.report.computations.remove_ts(scenario: Scenario, config: dict, after: int | None = None, dump: bool = False) → None[source]: Remove all time series data from scenario.

Todo

Improve to provide the option to remove only those periods in the model horizon.

Todo

Move upstream, e.g. to ixmp alongside store_ts().

message_ix_models.report.computations.share_curtailment(curt, *parts)[source]

Apply a share of curt to the first of parts.

If this is being used, it usually will indicate the need to split curt into multiple technologies; one for each of parts.

Utilities 

`add_replacements`(dim, codes)	Update `REPLACE_DIMS` for dimension dim with values from codes.
`as_quantity`(info)	Convert values from a `dict` to Quantity.
`collapse`(df[, var])	Callback for the collapse argument to `convert_pyam()`.
`collapse_gwp_info`(df, var)	`collapse()` helper for emissions data with GWP dimensions.
`copy_ts`(rep, other, filters)	Prepare rep to copy time series data from other to scenario.

message_ix_models.report.util.REPLACE_DIMS: Dict[str, Dict[str, str]] = {'c': {'Agri_Ch4': 'GLOBIOM|Emissions|CH4 Emissions Total'}, 'l': {'Final Energy': 'Final Energy|Residential'}, 't': {}}

Replacements used in collapse(). These are applied using pandas.DataFrame.replace() with regex=True; see the documentation of that method.

Applied to whole strings along each dimension.
These columns have str.title() applied before these replacements.

message_ix_models.report.util.REPLACE_VARS = {'(Emissions\\|CH4)\\|((Gases|Liquids|Solids|Elec|Heat)(.*))': '\\1|Energy|Supply|\\3|Fugitive\\4', '(Emissions\\|CH4)\\|Fugitive': '\\1|Energy|Supply|Fugitive', '(Secondary Energy\\|Solids)\\|Solids': '\\1', 'Import Energy\\|(Liquids\\|(Biomass|Oil))': 'Secondary Energy|\\1', 'Import Energy\\|Coal': 'Primary Energy|Coal', 'Import Energy\\|Lh2': 'Secondary Energy|Hydrogen', 'Import Energy\\|Lng': 'Primary Energy|Gas', 'Import Energy\\|Oil': 'Primary Energy|Oil', 'Residential\\|(Biomass|Coal)': 'Residential|Solids|\\1', 'Residential\\|Gas': 'Residential|Gases|Natural Gas', '^(land_out CH4.*\\|)Awm': '\\1Manure Management', '^land_out CH4\\|': '', '^land_out CH4\\|Emissions\\|Ch4\\|Land Use\\|Agriculture\\|': 'Emissions|CH4|AFOLU|Agriculture|Livestock|'}: Replacements used in collapse() after the ‘variable’ column is assembled. These are applied using pandas.DataFrame.replace() with regex=True; see the documentation of that method. For documentation of regular expressions, see https://docs.python.org/3/library/re.html and https://regex101.com.

Todo

These may be particular or idiosyncratic to a single “template”. The strings used to collapse multiple conceptual dimensions into the IAMC “variable” column are known to vary in poorly-documented ways across these templates.

This setting is currently applied universally. To improve, specify a different mapping with the replacements needed for each individual template, and load the correct one when reporting scenarios to that template.

message_ix_models.report.util.add_replacements(dim: str, codes: Iterable[Code]) → None[source]: Update REPLACE_DIMS for dimension dim with values from codes.

message_ix_models.report.util.as_quantity(info: dict | float | str) → Quantity[source]: Convert values from a dict to Quantity.

Todo

move upstream, to genno.

message_ix_models.report.util.collapse(df: DataFrame, var=[]) → DataFrame[source]

Callback for the collapse argument to convert_pyam().

Replacements from REPLACE_DIMS and REPLACE_VARS are applied. The dimensions listed in the var arguments are automatically dropped from the returned pyam.IamDataFrame. If var[0] contains the word “emissions”, then collapse_gwp_info() is invoked.

Adapted from genno.compat.pyam.collapse().

Parameters:: var (list of str, optional) – Strings or dimensions to concatenate to the ‘Variable’ column. The first of these is usually a string value used to populate the column. These are joined using the pipe (‘|’) character.

Command-line interface 

$ mix-models report --help

Usage: mix-models report [OPTIONS] [KEY]

  Postprocess results.

  KEY defaults to the comprehensive report 'message::default', but may also be
  the name of a specific model quantity, e.g. 'output'.

  --config can give either the absolute path to a reporting configuration
  file, or the stem (i.e. name without .yaml extension) of a file in
  data/report.

  With --from-file, read multiple Scenario identifiers from FILE, and report
  each one. In this usage, --output-path may only be a directory.

Options:
  --dry-run             Only show what would be done.
  --config TEXT         Path or stem for reporting config file.  [default:
                        global]
  -L, --legacy          Invoke legacy reporting.
  -m, --module MODULES  Add extra reporting for MODULES.
  -o, --output PATH     Write output to file instead of console.
  --from-file FILE      Report multiple Scenarios listed in FILE.
  --help                Show this message and exit.

Reporting (report)

Reporting (`report`)