Low-level utilities (util)

Submodules:

click	Command-line utilities.
context	Context and settings for message_ix_models code.
importlib	Load model and project code from message_data.
_logging	Logging utilities.
scenarioinfo	ScenarioInfo class.

Commonly used:

as_codes(data)	Convert data to a list of Code objects.
broadcast(df, **kwargs)	Fill missing data in df by broadcasting.
copy_column(column_name)	For use with pandas.DataFrame.assign().
ffill(df, dim, values[, expr])	Forward-fill df on dim to cover values.
load_package_data(*parts[, suffix])	Load a message_ix_models package data file and return its contents.
load_private_data(*parts)	Load a private data file from message_data and return its contents.
make_io(src, dest, efficiency[, on])	Return input and output data frames for a 1-to-1 technology.
make_matched_dfs(base, **par_value)	Return data frames derived from base for multiple parameters.
make_source_tech(info, common, **values)	Return parameter data for a ‘source’ technology.
merge_data(base, *others)	Merge dictionaries of DataFrames together into base.
package_data_path(*parts)	Construct a path to a file under message_ix_models/data/.
private_data_path(*parts)	Construct a path to a file under data/ in message_data.
same_node(df)	Fill ‘node_origin’/’node_dest’ in df from ‘node_loc’.
Context(*args, **kwargs)	Context and settings for message_ix_models code.
ScenarioInfo([scenario])	Information about a Scenario object.

message_ix_models.util.PACKAGE_DATA: Dict[str, Any] = {}

Package data already loaded with load_package_data().

message_ix_models.util.PRIVATE_DATA: Dict[str, Any] = {}

Data already loaded with load_private_data().

message_ix_models.util.add_par_data(scenario: message_ix.core.Scenario, data: Mapping[str, pandas.core.frame.DataFrame], dry_run: bool = False)

Add data to scenario.

Parameters

• data – Dict with keys that are parameter names, and values are pd.DataFrame or other arguments

• dry_run (optional) – Only show what would be done.

See also

strip_par_data

message_ix_models.util.as_codes(data: Union[List[str], Dict[str, Dict]]) → List[sdmx.model.Code]

Convert data to a list of Code objects.

Various inputs are accepted:

• list of str.

• dict, in which keys are Code.id and values are further dict with keys matching other Code attributes.

message_ix_models.util.broadcast(df, **kwargs)

Fill missing data in df by broadcasting.

Parameters

kwargs – Keys are dimensions. Values are labels along that dimension to fill.

message_ix_models.util.copy_column(column_name)

For use with pandas.DataFrame.assign().

Examples

Modify df by filling the column ‘baz’ with the value 3, and copying the column ‘bar’ into column ‘foo’.

>>> df.assign(foo=copy_column('bar'), baz=3)

message_ix_models.util.eval_anno(obj: sdmx.model.AnnotableArtefact, id: str)

Retrieve the annotation id from obj, run eval() on its contents.

This can be used for unpacking Python values (e.g. dict) stored as an annotation on a Code.

Returns None if no attribute exists with the given id.

message_ix_models.util.ffill(df: pandas.core.frame.DataFrame, dim: str, values: Sequence[Union[str, sdmx.model.Code]], expr: Optional[str] = None) → pandas.core.frame.DataFrame

Forward-fill df on dim to cover values.

Parameters

• df (DataFrame) – Data to fill forwards.

• dim (str) – Dimension to fill along. Must be a column in df.

• values (list of str) – Labels along dim that must be present in the returned data frame.

• expr (str, optional) – If provided, DataFrame.eval() is called. This can be used to assign one column to another. For instance, if dim == “year_vtg” and expr is “year_act = year_vtg”, then forward filling is performed along the “year_vtg” dimension/ column, and then the filled values are copied to the “year_act” column.

message_ix_models.util.iter_parameters(set_name)

Iterate over MESSAGEix parameters with set_name as a dimension.

Parameters

set_name (str) – Name of a set.

Yields

str – Names of parameters that have set_name indexing ≥1 dimension.

message_ix_models.util.load_package_data(*parts: str, suffix: Optional[str] = '.yaml') → Any

Load a message_ix_models package data file and return its contents.

Data is re-used if already loaded.

Example

The single call:

>>> info = load_package_data("node", "R11")

1. loads the metadata file data/node/R11.yaml, parsing its contents,

2. stores those values at PACKAGE_DATA["node R11"] for use by other code, and

3. returns the loaded values.

Parameters

• parts (iterable of str) – Used to construct a path under message_ix_models/data/.

• suffix (str, optional) – File name suffix, including, the “.”, e.g. .yaml.

Returns

Configuration values that were loaded.

Return type

dict

message_ix_models.util.load_private_data(*parts: str) → Mapping

Load a private data file from message_data and return its contents.

Analogous to load_package_data, but for non-public data.

Parameters

parts (iterable of str) – Used to construct a path under data/ in the message_data repository.

Returns

Configuration values that were loaded.

Return type

dict

Raises

RuntimeError – if message_data is not installed.

message_ix_models.util.make_io(src, dest, efficiency, on='input', **kwargs)

Return input and output data frames for a 1-to-1 technology.

Parameters

• src (tuple (str, str, str)) – Input commodity, level, unit.

• dest (tuple (str, str, str)) – Output commodity, level, unit.

• efficiency (float) – Conversion efficiency.

• on ('input' or 'output') – If ‘input’, efficiency applies to the input, and the output, thus the activity level of the technology, is in dest[2] units. If ‘output’, the opposite.

• kwargs – Passed to make_df().

Returns

Keys are ‘input’ and ‘output’; values are data frames.

Return type

dict (str -> pd.DataFrame)

message_ix_models.util.make_matched_dfs(base, **par_value)

Return data frames derived from base for multiple parameters.

par_values maps from parameter names (e.g. ‘fix_cost’) to values. make_matched_dfs returns a dict of pandas.DataFrame, one for each parameter in par_value. The contents of base are used to populate the columns of each data frame, and the values of par_value overwrite the ‘value’ column. Duplicates—which occur when the target parameter has fewer dimensions than base—are dropped.

Examples

>>> input = make_df('input', ...)
>>> cf_tl = make_matched_dfs(
>>>     input,
>>>     capacity_factor=1,
>>>     technical_lifetime=1,
>>> )

message_ix_models.util.make_source_tech(info, common, **values) → Dict[str, pandas.core.frame.DataFrame]

Return parameter data for a ‘source’ technology.

The technology has no inputs; its output commodity and/or level are determined by common; either single values, or None if the result will be pipe()’d through broadcast().

Parameters

• info (ScenarioInfo) –

• common (dict) – Passed to make_df().

• **values – Values for ‘capacity_factor’ (optional; default 1.0), ‘output’, ‘var_cost’, and optionally ‘technical_lifetime’.

Returns

Suitable for add_par_data().

Return type

dict

message_ix_models.util.merge_data(base, *others)

Merge dictionaries of DataFrames together into base.

message_ix_models.util.package_data_path(*parts) → pathlib.Path

Construct a path to a file under message_ix_models/data/.

message_ix_models.util.private_data_path(*parts) → pathlib.Path

Construct a path to a file under data/ in message_data.

message_ix_models.util.same_node(df)

Fill ‘node_origin’/’node_dest’ in df from ‘node_loc’.

message_ix_models.util.strip_par_data(scenario, set_name, value, dry_run=False, dump: Optional[Dict[str, pandas.core.frame.DataFrame]] = None)

Remove data from parameters of scenario where value in set_name.

Returns

Return type

Total number of rows removed across all parameters.

See also

add_par_data

util.click

Command-line utilities.

These are used for building CLIs using click.

message_ix_models.util.click.PARAMS = {'dest': <Option dest>, 'dry_run': <Option dry_run>, 'force': <Option force>, 'output_model': <Option output_model>, 'platform_dest': <Option platform_dest>, 'policy_path': <Option policy_path>, 'quiet': <Option quiet>, 'regions': <Option regions>, 'rep_out_path': <Option rep_out_path>, 'rep_template': <Option rep_template>, 'run_reporting_only': <Option run_reporting_only>, 'ssp': <Argument ssp>, 'verbose': <Option verbose>}

Common command-line parameters (arguments and options). See common_params().

message_ix_models.util.click.common_params(param_names: str)

Decorate a click.command with common parameters param_names.

param_names must be a space-separated string of names appearing in PARAMS, e.g. "ssp force output_model". The decorated function receives keyword arguments with these names:

@click.command()
@common_params("ssp force output_model")
def mycmd(ssp, force, output_model)
    # ...

message_ix_models.util.click.default_path_cb(*default_parts)

Return a callback function for click.Option handling.

If no option value is given, the callback uses Context.get_local_path() and default_parts to provide a path that is relative to local data directory, e.g. the current working directory (see Data, metadata, and configuration).

message_ix_models.util.click.store_context(context, param, value)

Callback that simply stores a value on the Context object.

Use this for parameters that are not used directly in a @click.command() function, but need to be carried by the Context for later use.

util.context

Context and settings for message_ix_models code.

class message_ix_models.util.context.Context(*args, **kwargs)

Context and settings for message_ix_models code.

Context is a subclass of dict, so common methods like copy() and setdefault() may be used to handle settings. To be forgiving, it also provides attribute access; context.foo is equivalent to context["foo"].

Context provides additional methods to do common tasks that depend on configurable settings:

clone_to_dest()	Return a scenario based on the --dest command-line option.
close_db()
delete()	Hide the current Context from future get_instance() calls.
get_cache_path(*parts)	Return a path to a local cache file.
get_local_path(*parts[, suffix])	Return a path under local_data.
get_platform([reload])	Return a ixmp.Platform from platform_info.
get_scenario()	Return a message_ix.Scenario from scenario_info.
handle_cli_args([url, platform, model_name, …])	Handle command-line arguments.
only()	Return the only Context instance.
use_defaults(settings)	Update from settings.

The following Context methods and attribute are deprecated:

get_config_file(*parts[, ext])	Return a path under metadata_path.
get_path(*parts)	Return a path under message_data_path by joining parts.
load_config(*parts[, suffix])	Load configuration from message_ix_models.
units	Access the unit registry.

clone_to_dest() → Tuple[message_ix.core.Scenario, ixmp.core.Platform]

Return a scenario based on the --dest command-line option.

Returns

To prevent the scenario from being garbage collected. Keep a reference to its Platform:

Return type

Scenario

See also

create_res

To use this method, either decorate a command with common_params():

from message_data.tools.cli import common_params

@click.command()
@common_params("dest")
@click.pass_obj
def foo(context, dest):
    scenario, mp = context.clone_to_dest()

or, store the settings dest_scenario and dest_platform on context:

c = Context.get_instance()

c.dest_scenario = dict(model="foo model", scenario="foo scenario")
scenario_mp = context.clone_to_dest()

The resulting scenario has the indicated model- and scenario names.

If --url (or --platform, --model, --scenario, and optionally --version) are given, the identified scenario is used as a ‘base’ scenario, and is cloned. If --url/--platform and --dest refer to different Platform instances, then this is a two-platform clone.

If no base scenario can be loaded, bare.create_res() is called to generate a base scenario.

delete()

Hide the current Context from future get_instance() calls.

get_cache_path(*parts) → pathlib.Path

Return a path to a local cache file.

get_config_file(*parts, ext='yaml') → pathlib.Path

Return a path under metadata_path.

The suffix “.{ext}” is added; defaulting to “.yaml”.

Deprecated since version 2021.2.28: Use package_data_path() instead. Will be removed on or after 2021-05-28.

classmethod get_instance(index=0) → message_ix_models.util.context.Context

Return a Context instance; by default, the first created.

Parameters

index (int, optional) – Index of the Context instance to return, e.g. -1 for the most recently created.

get_local_path(*parts, suffix=None)

Return a path under local_data.

get_path(*parts) → pathlib.Path

Return a path under message_data_path by joining parts.

parts may include directory names, or a filename with extension.

Deprecated since version 2021.2.28: Use private_data_path() instead. Will be removed on or after 2021-05-28.

get_platform(reload=False) → ixmp.core.Platform

Return a ixmp.Platform from platform_info.

When used through the CLI, platform_info is a ‘base’ platform as indicated by the –url or –platform options.

If a Platform has previously been instantiated with get_platform(), the same object is returned unless reload=True.

get_scenario() → message_ix.core.Scenario

Return a message_ix.Scenario from scenario_info.

When used through the CLI, scenario_info is a ‘base’ scenario for an operation, indicated by the --url or --platform/--model/--scenario options.

handle_cli_args(url=None, platform=None, model_name=None, scenario_name=None, version=None, local_data=None, _store_as=('platform_info', 'scenario_info'))

Handle command-line arguments.

May update the data_path, platform_info, scenario_info, and/or url settings.

load_config(*parts, suffix=None)

Load configuration from message_ix_models.

Deprecated since version 2021.2.28: Use load_package_data() instead. Will be removed on or after 2021-05-28.

classmethod only() → message_ix_models.util.context.Context

Return the only Context instance.

Raises

IndexError – If there is more than one instance.

property units

Access the unit registry.

Deprecated since version 2021.2.28: Instead, use:

from iam_units import registry

Will be removed on or after 2021-05-28.

use_defaults(settings)

Update from settings.

util.importlib

Load model and project code from message_data.

class message_ix_models.util.importlib.MessageDataFinder

Load model and project code from message_data.

util._logging

Logging utilities.

class message_ix_models.util._logging.Formatter(colorama)

Formatter for log records.

Parameters

colorama (module) – If provided, colorama is used to colour log messages printed to stdout.

format(record)

Format record.

Records are formatted like:

model.transport.data.add_par_data  220 rows in 'input'
...add_par_data:  further messages

…with the calling function name (e.g. ‘add_par_data’) coloured for legibility on first occurrence, then dimmed when repeated.

message_ix_models.util._logging.make_formatter()

Return a Formatter instance for the message_ix_models logger.

See also

setup

message_ix_models.util._logging.setup(level='NOTSET', console=True)

Initialize logging.

Parameters

• level (str, optional) – Log level for message_ix_models and message_data.

• console (bool, optional) – If True, print all messages to console using a Formatter.

message_ix_models.util._logging.silence_log()

Context manager to temporarily silence log output.

Examples

>>> with silence_log():
>>>     log.warning("This message is not recorded.")

util.scenarioinfo

ScenarioInfo class.

class message_ix_models.util.scenarioinfo.ScenarioInfo(scenario=None)

Information about a Scenario object.

Code that prepares data for a target Scenario can accept a ScenarioInfo instance. This avoids the need to load a Scenario, which can be slow under some conditions.

ScenarioInfo objects can also be used (e.g. by apply_spec()) to describe the contents of a Scenario before it is created.

ScenarioInfo objects have the following convenience attributes:

set	Elements of ixmp/message_ix sets.
is_message_macro	True if a MESSAGE-MACRO scenario.
N	Elements of the set ‘node’.
Y	Elements of the set ‘year’ that are >= the first model year.
y0	First model year, if set, else Y[0].
yv_ya	pandas.DataFrame with valid year_vtg, year_act pairs.

Parameters

scenario (message_ix.Scenario) – If given, set is initialized from this existing scenario.

property N

Elements of the set ‘node’.

property Y

Elements of the set ‘year’ that are >= the first model year.

is_message_macro: bool = False

True if a MESSAGE-MACRO scenario.

par: Dict[str, pandas.core.frame.DataFrame] = {}

Elements of ixmp/message_ix parameters.

set: Dict[str, List] = {}

Elements of ixmp/message_ix sets.

y0: int = -1

First model year, if set, else Y[0].

year_from_codes(codes: List[sdmx.model.Code])

Update using a list of codes.

The following are updated:

• set year

• set cat_year, with the first model year.

• par duration_period

Any existing values are discarded.

After this, the attributes y0 and Y give the first model year and model years, respectively.

Examples

Get a particular code list, create a ScenarioInfo instance, and update using the codes:

>>> years = get_codes("year/A")
>>> info = ScenarioInfo()
>>> info.year_from_codes(years)

Use populated values:

>>> info.y0
2020
>>> info.Y[:3]
[2020, 2030, 2040]
>>> info.Y[-3:]
[2090, 2100, 2110]

property yv_ya

pandas.DataFrame with valid year_vtg, year_act pairs.