Low-level utilities (util)

Submodules:

click	Command-line utilities.
context	Context and settings for message_ix_models code.
importlib	Load model and project code from message_data.
_logging	Logging utilities.
scenarioinfo	ScenarioInfo class.

Commonly used:

adapt_R11_R12	Adapt data using mappings for 1 or more dimension(s).
adapt_R11_R14	Adapt data using mappings for 1 or more dimension(s).
as_codes(data)	Convert data to a list of Code objects.
broadcast(df[, labels])	Fill missing data in df by broadcasting.
cached(func)	Decorator to cache the return value of a function func.
check_support(context[, settings, desc])	Check whether a Context is compatible with certain settings.
convert_units(s, unit_info[, store])	Convert units of s, for use with apply().
copy_column(column_name)	For use with pandas.DataFrame.assign().
ffill(df, dim, values[, expr])	Forward-fill df on dim to cover values.
identify_nodes(scenario)	Return the ID of a node codelist given the contents of scenario.
load_package_data(*parts[, suffix])	Load a message_ix_models package data file and return its contents.
load_private_data(*parts)	Load a private data file from message_data and return its contents.
local_data_path(*parts)	Construct a path for local data.
make_io(src, dest, efficiency[, on])	Return input and output data frames for a 1-to-1 technology.
make_matched_dfs(base, **par_value)	Return data frames derived from base for multiple parameters.
make_source_tech(info, common, **values)	Return parameter data for a ‘source’ technology.
maybe_query(series, query)	Apply pandas.Series.query() if the query arg is not None.
merge_data(base, *others)	Merge dictionaries of DataFrames together into base.
package_data_path(*parts)	Construct a path to a file under message_ix_models/data/.
private_data_path(*parts)	Construct a path to a file under data/ in message_data.
same_node(df)	Fill 'node_origin'/'node_dest' in df from 'node_loc'.
series_of_pint_quantity(*args, **kwargs)	Suppress a spurious warning.
Context(*args, **kwargs)	Context and settings for message_ix_models code.
ScenarioInfo([scenario])	Information about a Scenario object.
Spec(add, remove, require)	A specification for the structure of a model or variant.

message_ix_models.util.as_codes(data: Union[List[str], Dict[str, Union[Dict, Code]]]) → List[Code]

Convert data to a list of Code objects.

Various inputs are accepted:

• list of str.

• dict, in which keys are Code.id and values are further dict with keys matching other Code attributes.

message_ix_models.util.broadcast(df: DataFrame, labels: Optional[DataFrame] = None, **kwargs) → DataFrame

Fill missing data in df by broadcasting.

broadcast() is suitable for use with partly-filled data frames returned by message_ix.util.make_df(), with 1 column per dimension, plus a “value” column. It is also usable with pandas.DataFrame.pipe() for chained operations.

labels (if any) are handled first: one copy or duplicate of df is produced for each row (set of labels) in this argument. Then, kwargs are handled; broadcast() returns one copy for each element in the cartesian product of the dimension labels given by kwargs.

Parameters

• labels (pandas.DataFrame) – Each column (dimension) corresponds to one in df. Each row represents one matched set of labels for those dimensions.

• kwargs – Keys are dimensions. Values are labels along that dimension to fill.

Returns

The length is either 1 or an integer multiple of the length of df.

Return type

pandas.DataFrame

Raises

ValueError – if any of the columns in labels or kwargs are not present in df, or if those columns are present but not empty.

Examples

>>> from message_ix import make_df
>>> from message_ix_models.util import broadcast
# Create a base data frame with some empty columns
>>> base = make_df("input", technology="t", value=[1.1, 2.2])
# Broadcast (duplicate) the data across 2 dimensions
>>> df = base.pipe(broadcast, node_loc=["node A", "node B"], mode=["m0", "m1"])
# Show part of the result
>>> df.dropna(axis=1)
  mode node_loc technology  value
0   m0   node A          t    1.1
1   m0   node A          t    2.2
2   m0   node B          t    1.1
3   m0   node B          t    2.2
4   m1   node A          t    1.1
5   m1   node A          t    2.2
6   m1   node B          t    1.1
7   m1   node B          t    2.2

message_ix_models.util.cached(func: Callable) → Callable

Decorator to cache the return value of a function func.

On a first call, the data requested is returned and also cached under Context.get_cache_path(). On subsequent calls, if the cache exists, it is used instead of calling the (possibly slow) func.

When SKIP_CACHE is true, func is always called.

See also

Caching in the genno documentation

message_ix_models.util.check_support(context, settings={}, desc: str = '') → None

Check whether a Context is compatible with certain settings.

Raises

• NotImplementedError – if any context value for a key of settings is not among the values in settings.

• KeyError – if the key is not set on context at all.

See also

Advertise and check compatibility

message_ix_models.util.convert_units(s: Series, unit_info: Mapping[str, Tuple[float, str, Optional[str]]], store='magnitude') → Series

Convert units of s, for use with apply().

s.name is used to retrieve a tuple of (factor, input_unit, output_unit) from unit_info. The (float) values of s are converted to pint.Quantity with the input_unit and factor; then cast to output_unit, if provided.

Parameters

• s (pandas.Series) –

• unit_info (dict (str -> tuple)) – Mapping from quantity name (matched to s.name) to 3-tuples of (factor, input_unit, output_unit). output_unit may be None. For example, see ikarus.UNITS.

• store ("magnitude" or "quantity") – If “magnitude”, the values of the returned series are the magnitudes of the results, with no output units. If “quantity”, the values are scalar Quantity objects.

Returns

Same shape, index, and values as s, with output units.

Return type

pandas.Series

message_ix_models.util.eval_anno(obj: AnnotableArtefact, id: str)

Retrieve the annotation id from obj, run eval() on its contents.

This can be used for unpacking Python values (e.g. dict) stored as an annotation on a Code.

Returns None if no attribute exists with the given id.

message_ix_models.util.identify_nodes(scenario: Scenario) → str

Return the ID of a node codelist given the contents of scenario.

Returns

The ID of the Node code lists containing the regions of scenario.

Return type

str

Raises

ValueError – if no codelist can be identified, or the nodes in the scenario do not match the children of the “World” node in the codelist.

message_ix_models.util.load_package_data(*parts: str, suffix: Optional[str] = '.yaml') → Any

Load a message_ix_models package data file and return its contents.

Data is re-used if already loaded.

Example

The single call:

>>> info = load_package_data("node", "R11")

1. loads the metadata file data/node/R11.yaml, parsing its contents,

2. stores those values at PACKAGE_DATA["node R11"] for use by other code, and

3. returns the loaded values.

Parameters

• parts (iterable of str) – Used to construct a path under message_ix_models/data/.

• suffix (str, optional) – File name suffix, including, the “.”, e.g. .yaml.

Returns

Configuration values that were loaded.

Return type

dict

message_ix_models.util.load_private_data(*parts: str) → Mapping

Load a private data file from message_data and return its contents.

Analogous to load_package_data, but for non-public data.

Parameters

parts (iterable of str) – Used to construct a path under data/ in the message_data repository.

Returns

Configuration values that were loaded.

Return type

dict

Raises

RuntimeError – if message_data is not installed.

message_ix_models.util.local_data_path(*parts) → Path

Construct a path for local data.

The setting message local data in the user’s ixmp configuration file is used as a base path. If this is not configured, the current working directory is used.

Parameters

parts (sequence of str or Path) – Joined to the base path using Path.joinpath().

See also

Choose locations for data

message_ix_models.util.make_io(src, dest, efficiency, on='input', **kwargs)

Return input and output data frames for a 1-to-1 technology.

Parameters

• src (tuple (str, str, str)) – Input commodity, level, unit.

• dest (tuple (str, str, str)) – Output commodity, level, unit.

• efficiency (float) – Conversion efficiency.

• on ('input' or 'output') – If ‘input’, efficiency applies to the input, and the output, thus the activity level of the technology, is in dest[2] units. If ‘output’, the opposite.

• kwargs – Passed to make_df().

Returns

Keys are ‘input’ and ‘output’; values are data frames.

Return type

dict (str -> pd.DataFrame)

message_ix_models.util.maybe_query(series: Series, query: Optional[str]) → Series

Apply pandas.Series.query() if the query arg is not None.

query() is not chainable (pandas-dev/pandas#37941). Use this function with pandas.Series.pipe(), passing an argument that may be None, to have a chainable query operation that can be a no-op.

message_ix_models.util.package_data_path(*parts) → Path

Construct a path to a file under message_ix_models/data/.

Use this function to access data packaged and installed with message_ix_models.

Parameters

parts (sequence of str or Path) – Joined to the base path using Path.joinpath().

See also

Choose locations for data

message_ix_models.util.private_data_path(*parts) → Path

Construct a path to a file under data/ in message_data.

Use this function to access non-public (e.g. embargoed or proprietary) data stored in the message_data repository.

Parameters

parts (sequence of str or Path) – Joined to the base path using Path.joinpath().

See also

Choose locations for data

message_ix_models.util.series_of_pint_quantity(*args, **kwargs) → Series

Suppress a spurious warning.

Creating a pandas.Series with a list of pint.Quantity triggers a warning “The unit of the quantity is stripped when downcasting to ndarray,” even though the entire object is being stored and the unit is not stripped. This function suppresses this warning.

message_ix_models.util.cache.SKIP_CACHE = False

Controls whether cached data is returned for functions decorated with cached(). Set to True to force reload.

util.click

Command-line utilities.

These are used for building CLIs using click.

message_ix_models.util.click.PARAMS = {'dest': <Option dest>, 'dry_run': <Option dry_run>, 'force': <Option force>, 'nodes': <Option nodes>, 'output_model': <Option output_model>, 'platform_dest': <Option platform_dest>, 'policy_path': <Option policy_path>, 'quiet': <Option quiet>, 'regions': <Option regions>, 'rep_out_path': <Option rep_out_path>, 'rep_template': <Option rep_template>, 'run_reporting_only': <Option run_reporting_only>, 'ssp': <Argument ssp>, 'verbose': <Option verbose>, 'years': <Option years>}

Common command-line parameters (arguments and options). See common_params().

message_ix_models.util.click.common_params(param_names: str)

Decorate a click.command with common parameters param_names.

param_names must be a space-separated string of names appearing in PARAMS, e.g. "ssp force output_model". The decorated function receives keyword arguments with these names:

@click.command()
@common_params("ssp force output_model")
def mycmd(ssp, force, output_model)
    # ...

message_ix_models.util.click.default_path_cb(*default_parts)

Return a callback function for click.Option handling.

If no option value is given, the callback uses Context.get_local_path() and default_parts to provide a path that is relative to local data directory, e.g. the current working directory (see Data, metadata, and configuration).

message_ix_models.util.click.store_context(context: Union[Context, Context], param, value)

Callback that simply stores a value on the Context object.

Use this for parameters that are not used directly in a @click.command() function, but need to be carried by the Context for later use.

util.context

Context and settings for message_ix_models code.

class message_ix_models.util.context.Context(*args, **kwargs)

Context and settings for message_ix_models code.

Context is a subclass of dict, so common methods like copy() and setdefault() may be used to handle settings. To be forgiving, it also provides attribute access; context.foo is equivalent to context["foo"].

Context provides additional methods to do common tasks that depend on configurable settings:

clone_to_dest([create])	Return a scenario based on the --dest command-line option.
close_db()
delete()	Hide the current Context from future get_instance() calls.
get_cache_path(*parts)	Return a path to a local cache file.
get_local_path(*parts[, suffix])	Return a path under local_data.
get_platform([reload])	Return a ixmp.Platform from platform_info.
get_scenario()	Return a message_ix.Scenario from scenario_info.
handle_cli_args([url, platform, model_name, ...])	Handle command-line arguments.
only()	Return the only Context instance.
use_defaults(settings)	Update from settings.

clone_to_dest(create=True) → Scenario

Return a scenario based on the --dest command-line option.

Parameters

create (bool, optional) – If True (the default) and the base scenario does not exist, a bare RES scenario is created. Otherwise, an exception is raised.

Returns

To prevent the scenario from being garbage collected, keep a reference to its Platform:

Return type

Scenario

See also

create_res

To use this method, either decorate a command with common_params():

from message_data.tools.cli import common_params

@click.command()
@common_params("dest")
@click.pass_obj
def foo(context, dest):
    scenario, mp = context.clone_to_dest()

or, store the settings dest_scenario and dest_platform on context:

c = Context.get_instance()

c.dest_scenario = dict(model="foo model", scenario="foo scenario")
scenario_mp = context.clone_to_dest()

The resulting scenario has the indicated model- and scenario names.

If --url (or --platform, --model, --scenario, and optionally --version) are given, the identified scenario is used as a ‘base’ scenario, and is cloned. If --url/--platform and --dest refer to different Platform instances, then this is a two-platform clone.

If no base scenario can be loaded, bare.create_res() is called to generate a base scenario.

delete()

Hide the current Context from future get_instance() calls.

get_cache_path(*parts) → Path

Return a path to a local cache file.

classmethod get_instance(index=0) → Context

Return a Context instance; by default, the first created.

Parameters

index (int, optional) – Index of the Context instance to return, e.g. -1 for the most recently created.

get_local_path(*parts, suffix=None)

Return a path under local_data.

get_platform(reload=False) → Platform

Return a ixmp.Platform from platform_info.

When used through the CLI, platform_info is a ‘base’ platform as indicated by the –url or –platform options.

If a Platform has previously been instantiated with get_platform(), the same object is returned unless reload=True.

get_scenario() → Scenario

Return a message_ix.Scenario from scenario_info.

When used through the CLI, scenario_info is a ‘base’ scenario for an operation, indicated by the --url or --platform/--model/--scenario options.

handle_cli_args(url=None, platform=None, model_name=None, scenario_name=None, version=None, local_data=None, verbose=False, _store_as=('platform_info', 'scenario_info'))

Handle command-line arguments.

May update the data_path, platform_info, scenario_info, and/or url settings.

classmethod only() → Context

Return the only Context instance.

Raises

IndexError – If there is more than one instance.

set_scenario(scenario: Scenario) → None

Update scenario_info to match an existing scenario.

use_defaults(settings)

Update from settings.

util.importlib

Load model and project code from message_data.

class message_ix_models.util.importlib.MessageDataFinder

Load model and project code from message_data.

util._logging

Logging utilities.

class message_ix_models.util._logging.Formatter(colorama)

Formatter for log records.

Parameters

colorama (module) – If provided, colorama is used to colour log messages printed to stdout.

format(record)

Format record.

Records are formatted like:

model.transport.data.add_par_data  220 rows in 'input'
...add_par_data:  further messages

…with the calling function name (e.g. ‘add_par_data’) coloured for legibility on first occurrence, then dimmed when repeated.

message_ix_models.util._logging.make_formatter()

Return a Formatter instance for the message_ix_models logger.

See also

setup

message_ix_models.util._logging.setup(level='NOTSET', console=True)

Initialize logging.

Parameters

• level (str, optional) – Log level for message_ix_models and message_data.

• console (bool, optional) – If True, print all messages to console using a Formatter.

message_ix_models.util._logging.silence_log()

Context manager to temporarily silence log output.

Examples

>>> with silence_log():
>>>     log.warning("This message is not recorded.")

util.node

Utilities for nodes.

message_ix_models.util.node.NODE_DIMS = ['n', 'node', 'node_loc', 'node_origin', 'node_dest', 'node_rel', 'node_share']

Names of dimensions indexed by ‘node’.

Todo

to be robust to changes in message_ix, read these names from that package.

message_ix_models.util.node.R11_R12 = (('R11_AFR', 'R12_AFR'), ('R11_CPA', 'R12_CHN'), ('R11_EEU', 'R12_EEU'), ('R11_FSU', 'R12_FSU'), ('R11_LAM', 'R12_LAM'), ('R11_MEA', 'R12_MEA'), ('R11_NAM', 'R12_NAM'), ('R11_PAO', 'R12_PAO'), ('R11_PAS', 'R12_PAS'), ('R11_CPA', 'R12_RCPA'), ('R11_SAS', 'R12_SAS'), ('R11_WEU', 'R12_WEU'))

Mapping from R11 to R12 node IDs.

message_ix_models.util.node.R11_R14 = (('R11_AFR', 'R14_AFR'), ('R11_FSU', 'R14_CAS'), ('R11_CPA', 'R14_CPA'), ('R11_EEU', 'R14_EEU'), ('R11_LAM', 'R14_LAM'), ('R11_MEA', 'R14_MEA'), ('R11_NAM', 'R14_NAM'), ('R11_PAO', 'R14_PAO'), ('R11_PAS', 'R14_PAS'), ('R11_FSU', 'R14_RUS'), ('R11_SAS', 'R14_SAS'), ('R11_FSU', 'R14_SCS'), ('R11_FSU', 'R14_UBM'), ('R11_WEU', 'R14_WEU'))

Mapping from R11 to R14 node IDs.

message_ix_models.util.node.adapt_R11_R12 = <message_ix_models.util.common.MappingAdapter object>

Adapt data from the R11 to the R14 node list.

The data is adapted using the mappings in R11_R12 for each of the dimensions in NODE_DIMS.

message_ix_models.util.node.adapt_R11_R14 = <message_ix_models.util.common.MappingAdapter object>

Adapt data from the R11 to the R14 node list.

The data is adapted using the mappings in R11_R14 for each of the dimensions in NODE_DIMS.

message_ix_models.util.node.identify_nodes(scenario: Scenario) → str

Return the ID of a node codelist given the contents of scenario.

Returns

The ID of the Node code lists containing the regions of scenario.

Return type

str

Raises

ValueError – if no codelist can be identified, or the nodes in the scenario do not match the children of the “World” node in the codelist.

util.scenarioinfo

ScenarioInfo class.

class message_ix_models.util.scenarioinfo.ScenarioInfo(scenario=None)

Information about a Scenario object.

Code that prepares data for a target Scenario can accept a ScenarioInfo instance. This avoids the need to load a Scenario, which can be slow under some conditions.

ScenarioInfo objects can also be used (e.g. by apply_spec()) to describe the contents of a Scenario before it is created.

ScenarioInfo objects have the following convenience attributes:

set	Elements of ixmp/message_ix sets.
io_units(technology, commodity[, level])	Return units for the MESSAGE input or output parameter.
is_message_macro	True if a MESSAGE-MACRO scenario.
N	Elements of the set 'node'.
units_for(set_name, id)	Return the units associated with code id in MESSAGE set set_name.
Y	Elements of the set 'year' that are >= the first model year.
y0	First model year, if set, else Y[0].
yv_ya	pandas.DataFrame with valid year_vtg, year_act pairs.

Parameters

scenario (message_ix.Scenario) – If given, set is initialized from this existing scenario.

See also

Spec

property N

Elements of the set ‘node’.

property Y

Elements of the set ‘year’ that are >= the first model year.

io_units(technology: str, commodity: str, level: Optional[str] = None) → Unit

Return units for the MESSAGE input or output parameter.

These are implicitly determined as the ratio of:

• The units for the origin (for input) or destination commodity, per units_for().

• The units of activity for the technology.

Parameters

level (str) – Placeholder for future functionality, i.e. to use different units per (commodity, level). Currently ignored. If given, a debug message is logged.

is_message_macro: bool = False

True if a MESSAGE-MACRO scenario.

par: Dict[str, DataFrame] = {}

Elements of ixmp/message_ix parameters.

set: Dict[str, List] = {}

Elements of ixmp/message_ix sets.

units_for(set_name: str, id: str) → Unit

Return the units associated with code id in MESSAGE set set_name.

ixmp (or the sole JDBCBackend, as of v3.5.0) does not handle unit information for variables and equations (unlike parameter values), such as MESSAGE decision variables ACT, CAP, etc. In message_ix_models and message_data, the following conventions are (generally) followed:

• The units of ACT and others are consistent for each technology.

• The units of COMMODITY_BALANCE, STOCK, commodity_stock, etc. are consistent for each commodity.

Thus, codes for elements of these sets (e.g. Commodities (commodity.yaml)) can be used to carry the standard units for the corresponding quantities. units_for() retrieves these units, for use in model-building and reporting.

Todo

Expand this discussion and transfer to the message_ix docs.

See also

io_units

update(other: ScenarioInfo)

Update with the set elements of other.

y0: int = -1

First model year, if set, else Y[0].

year_from_codes(codes: List[Code])

Update using a list of codes.

The following are updated:

• set year

• set cat_year, with the first model year.

• par duration_period

Any existing values are discarded.

After this, the attributes y0 and Y give the first model year and model years, respectively.

Examples

Get a particular code list, create a ScenarioInfo instance, and update using the codes:

>>> years = get_codes("year/A")
>>> info = ScenarioInfo()
>>> info.year_from_codes(years)

Use populated values:

>>> info.y0
2020
>>> info.Y[:3]
[2020, 2030, 2040]
>>> info.Y[-3:]
[2090, 2100, 2110]

property yv_ya

pandas.DataFrame with valid year_vtg, year_act pairs.

class message_ix_models.util.scenarioinfo.Spec(add: ~message_ix_models.util.scenarioinfo.ScenarioInfo = <factory>, remove: ~message_ix_models.util.scenarioinfo.ScenarioInfo = <factory>, require: ~message_ix_models.util.scenarioinfo.ScenarioInfo = <factory>)

A specification for the structure of a model or variant.

A Spec collects 3 ScenarioInfo instances at the attributes add, remove, and require. This is the type that is accepted by apply_spec(); Building models (model.build) describes how a Spec is used to modify a Scenario. A Spec may also be used to express information about the target structure of data to be prepared; like ScenarioInfo, this can happen before the target Scenario exists.

Spec also provides:

• Dictionary-style access, e.g. s["add"] is equivalent to s.add..

• Error checking; setting keys other than add/remove/require results in an error.

• merge(), a helper method.

add: ScenarioInfo

Structure to be added to a base scenario.

static merge(a: Spec, b: Spec) → Spec

Merge Specs a and b together.

Returns a new Spec where each member is a union of the respective members of a and b.

remove: ScenarioInfo

Structure to be removed from a base scenario.

require: ScenarioInfo

Structure that must be present in a base scenario.