Low-level utilities (util)

Submodules:

click	Command-line utilities.
context	Context and settings for message_ix_models code.
importlib	Load model and project code from message_data.
_logging	Logging utilities.
node	Utilities for nodes.
pooch	Utilities for using Pooch.
scenarioinfo	ScenarioInfo class.
sdmx	Utilities for handling objects from sdmx.

Commonly used:

Config(cache_path, debug_paths, dest, ...)	Core/top-level settings for message_ix_models and message_data.
ConfigHelper()	Mix-in for dataclass-based configuration classes.
Context(*args, **kwargs)	Context and settings for message_ix_models code.
ScenarioInfo(scenario_obj, empty, ...)	Information about a Scenario object.
Spec(add, remove, require)	A specification for the structure of a model or variant.
broadcast(df[, labels])	Fill missing data in df by broadcasting.
cached(func)	Decorator to cache the return value of a function func.
check_support(context[, settings, desc])	Check whether a Context is compatible with certain settings.
convert_units(...)	Convert units of data.
copy_column(column_name)	For use with pandas.DataFrame.assign().
datetime_now_with_tz()	Current date and time with time zone information.
ffill(df, dim, values[, expr])	Forward-fill df on dim to cover values.
identify_nodes(scenario)	Return the ID of a node codelist given the contents of scenario.
iter_keys(base)	Return an iterator over a sequence of keys starting with base_key.
load_package_data(*parts[, suffix])	Load a message_ix_models package data file and return its contents.
load_private_data(*parts)	Load a private data file from message_data and return its contents.
local_data_path(*parts)	Construct a path for local data.
make_io(src, dest, efficiency[, on])	Return input and output data frames for a 1-to-1 technology.
make_matched_dfs(base, **par_value)	Return data frames derived from base for multiple parameters.
make_source_tech(info, common, **values)	Return parameter data for a ‘source’ technology.
maybe_query(series, query)	Apply pandas.DataFrame.query() if the query arg is not None.
merge_data(base, *others)	Merge dictionaries of DataFrames together into base.
minimum_version(expr[, raises])	Decorator for functions that require a minimum version of some upstream package.
nodes_ex_world(nodes)	Exclude "World" and anything containing "GLB" from nodes.
package_data_path(*parts)	Construct a path to a file under message_ix_models/data/.
private_data_path(*parts)	Construct a path to a file under data/ in message_data.
same_node()	Fill 'node_{,dest,loc,origin,rel,share}' in df from from_col.
same_time()	Fill 'time_origin'/'time_dest' in df from 'time'.
series_of_pint_quantity(*args, **kwargs)	Suppress a spurious warning.
show_versions()	Output of ixmp.show_versions(), as a str.

class message_ix_models.util.Adapter

Adapt data.

Adapter is an abstract base class for tools that adapt data in any way, e.g. between different code lists for certain dimensions. An instance of an Adapter can be called with any of the following as data:

• genno.Quantity,

• pandas.DataFrame, or

• dict mapping str parameter names to values (either of the above types).

…and will return data of the same type.

Subclasses can implement different adapter logic by overriding the abstract adapt() method.

abstract adapt(qty: AttrSeries) → AttrSeries

Adapt data.

class message_ix_models.util.MappingAdapter(maps: Mapping[str, Sequence[tuple[str, str]]])

Adapt data using mappings for 1 or more dimension(s).

Parameters:

maps (dict of Sequence of tuple) – Keys are names of dimensions. Values are sequences of 2-tuples; each tuple consists of an original label and a target label.

Examples

>>> a = MappingAdapter({"foo": [("a", "x"), ("a", "y"), ("b", "z")]})
>>> df = pd.DataFrame(
...     [["a", "m", 1], ["b", "n", 2]], columns=["foo", "bar", "value"]
... )
>>> a(df)
  foo  bar  value
0   x    m      1
1   y    m      1
2   z    n      2

adapt(qty: AttrSeries) → AttrSeries

Adapt data.

message_ix_models.util.add_par_data(scenario: Scenario, data: ParameterData, dry_run: bool = False) → int

Add data to scenario.

Parameters:

• data – Any mapping with keys that are valid message_ix parameter names, and values that are pd.DataFrame or other arguments valid for message_ix.Scenario.add_par().

• dry_run (optional) – Only show what would be done.

See also

strip_par_data

message_ix_models.util.aggregate_codes(df: DataFrame, dim: str, codes)

Aggregate df along dimension dim according to codes.

message_ix_models.util.broadcast(df: DataFrame, labels: DataFrame | None = None, **kwargs) → DataFrame

Fill missing data in df by broadcasting.

broadcast() is suitable for use with partly-filled data frames returned by message_ix.util.make_df(), with 1 column per dimension, plus a “value” column. It is also usable with pandas.DataFrame.pipe() for chained operations.

labels (if any) are handled first: one copy or duplicate of df is produced for each row (set of labels) in this argument. Then, kwargs are handled; broadcast() returns one copy for each element in the cartesian product of the dimension labels given by kwargs.

Parameters:

• labels (pandas.DataFrame) – Each column (dimension) corresponds to one in df. Each row represents one matched set of labels for those dimensions.

• kwargs – Keys are dimensions. Values are labels along that dimension to fill.

Returns:

The length is either 1 or an integer multiple of the length of df.

Return type:

pandas.DataFrame

Raises:

ValueError – if any of the columns in labels or kwargs are not present in df, or if those columns are present but not empty.

Examples

>>> from message_ix import make_df
>>> from message_ix_models.util import broadcast
# Create a base data frame with some empty columns
>>> base = make_df("input", technology="t", value=[1.1, 2.2])
# Broadcast (duplicate) the data across 2 dimensions
>>> df = base.pipe(broadcast, node_loc=["node A", "node B"], mode=["m0", "m1"])
# Show part of the result
>>> df.dropna(axis=1)
  mode node_loc technology  value
0   m0   node A          t    1.1
1   m0   node A          t    2.2
2   m0   node B          t    1.1
3   m0   node B          t    2.2
4   m1   node A          t    1.1
5   m1   node A          t    2.2
6   m1   node B          t    1.1
7   m1   node B          t    2.2

message_ix_models.util.cached(func: Callable) → Callable

Decorator to cache the return value of a function func.

On a first call, the data requested is returned and also cached under Context.get_cache_path(). On subsequent calls, if the cache exists, it is used instead of calling the (possibly slow) func.

When SKIP_CACHE is true, func is always called.

See also

Caching in the genno documentation

message_ix_models.util.check_support(context, settings={}, desc: str = '') → None

Check whether a Context is compatible with certain settings.

Raises:

• NotImplementedError – if any context value for a key of settings is not among the values in settings.

• KeyError – if the key is not set on context at all.

See also

Advertise and check compatibility

message_ix_models.util.convert_units(data: Any, **kwargs)

message_ix_models.util.convert_units(data: Series, unit_info: Mapping[str, tuple[float, str, str | None]], store='magnitude') → Series

message_ix_models.util.convert_units(data: DataFrame, info: ScenarioInfo) → DataFrame

message_ix_models.util.convert_units(data: dict[str, DataFrame], info: ScenarioInfo) → dict[str, DataFrame]

Convert units of data.

With pandas.Series: for use with apply().

data.name` is used to retrieve a tuple of (factor, input_unit, output_unit) from unit_info. The (float) values of data are converted to pint.Quantity with the input_unit and factor; then cast to output_unit, if provided.

Parameters:

• data (pandas.Series)

• unit_info (dict (str -> tuple)) – Mapping from quantity name (matched to s.name) to 3-tuples of (factor, input_unit, output_unit). output_unit may be None. For example, see ikarus.UNITS.

• store ("magnitude" or "quantity") – If “magnitude”, the values of the returned series are the magnitudes of the results, with no output units. If “quantity”, the values are scalar Quantity objects.

Returns:

Same shape, index, and values as s, with output units.

Return type:

pandas.Series

message_ix_models.util.copy_column(column_name)

For use with pandas.DataFrame.assign().

Examples

Modify df by filling the column ‘baz’ with the value 3, and copying the column ‘bar’ into column ‘foo’.

>>> df.assign(foo=copy_column('bar'), baz=3)

Note that a similar assignment can be achieved with eval():

>>> df.eval("foo = bar")

copy_column() is useful in the context of more complicated calls to assign().

message_ix_models.util.datetime_now_with_tz() → datetime

Current date and time with time zone information.

message_ix_models.util.ffill(df: DataFrame, dim: str, values: Sequence[str | Code], expr: str | None = None) → DataFrame

Forward-fill df on dim to cover values.

Parameters:

• df (pandas.DataFrame) – Data to fill forwards.

• dim (str) – Dimension to fill along. Must be a column in df.

• values (list of str) – Labels along dim that must be present in the returned data frame.

• expr (str, optional) – If provided, DataFrame.eval() is called. This can be used to assign one column to another. For instance, if dim == “year_vtg” and expr is “year_act = year_vtg”, then forward filling is performed along the “year_vtg” dimension/ column, and then the filled values are copied to the “year_act” column.

message_ix_models.util.identify_nodes(scenario: Scenario) → str

Return the ID of a node codelist given the contents of scenario.

Returns:

The ID of the Node code lists containing the regions of scenario.

Return type:

str

Raises:

ValueError – if no codelist can be identified, or the nodes in the scenario do not match the children of the “World” node in the codelist.

message_ix_models.util.iter_keys(base: genno.Key) → KeyIterator

Return an iterator over a sequence of keys starting with base_key.

This can be used for shorthand when constructing sequences of genno computations.

Example

>>> base_key = genno.Key("foo:a-b-c")
>>> k = iter_keys(base_key)
>>> k()
<foo:a-b-c:0>
>>> k()
<foo:a-b-c:1>
>>> k()
<foo:a-b-c:2>

message_ix_models.util.load_package_data(*parts: str, suffix: str | None = '.yaml') → Any

Load a message_ix_models package data file and return its contents.

Data is re-used if already loaded.

Example

The single call:

>>> info = load_package_data("node", "R11")

1. loads the metadata file data/node/R11.yaml, parsing its contents,

2. stores those values at PACKAGE_DATA["node R11"] for use by other code, and

3. returns the loaded values.

Parameters:

• parts (Iterable of str) – Used to construct a path under message_ix_models/data/.

• suffix (str, optional) – File name suffix, including, the “.”, e.g. .yaml.

Returns:

Configuration values that were loaded.

Return type:

dict

message_ix_models.util.load_private_data(*parts: str) → Mapping

Load a private data file from message_data and return its contents.

Analogous to load_package_data(), but for non-public data.

Parameters:

parts (Iterable of str) – Used to construct a path under data/ in the message_data repository.

Returns:

Configuration values that were loaded.

Return type:

dict

Raises:

RuntimeError – if message_data is not installed.

message_ix_models.util.local_data_path(*parts) → Path

Construct a path for local data.

The setting message local data in the user’s ixmp configuration file is used as a base path. If this is not configured, the current working directory is used.

Parameters:

parts (Sequence of str or Path) – Joined to the base path using Path.joinpath().

See also

Choose locations for data

message_ix_models.util.make_io(src: tuple[str, str, str], dest: tuple[str, str, str], efficiency: float, on: Literal['input', 'output'] = 'input', **kwargs)

Return input and output data frames for a 1-to-1 technology.

Parameters:

• src (tuple of str) – Input (commodity, level, unit)

• dest (tuple of str) – Output (commodity, level, unit)

• efficiency (float) – Conversion efficiency.

• on ('input' or 'output') – If ‘input’, efficiency applies to the input, and the output, thus the activity level of the technology, is in dest[2] units. If ‘output’, the opposite.

• kwargs – Passed to make_df().

Returns:

Keys are ‘input’ and ‘output’; values are data frames.

Return type:

dict (str -> pd.DataFrame)

message_ix_models.util.make_matched_dfs(base: MutableMapping | DataFrame, **par_value: float | Quantity | dict) → MutableParameterData

Return data frames derived from base for multiple parameters.

Creates one data frame per keyword argument.

Parameters:

• base (pandas.DataFrame, dict, etc.) – Used to populate other columns of each data frame. Duplicates—which occur when the target parameter has fewer dimensions than base—are dropped.

• par_values – Argument names (e.g. ‘fix_cost’) are passed to make_df(). If the value is float, it overwrites the “value” column; if pint.Quantity, its magnitude overwrites “value” and its units the “units” column, as a formatted string.

Returns:

one for each parameter in par_values.

Return type:

dict of pandas.DataFrame

Examples

>>> input = make_df("input", ...)
>>> cf_tl = make_matched_dfs(
>>>     input,
>>>     capacity_factor=1,
>>>     technical_lifetime=pint.Quantity(8, "year"),
>>> )

message_ix_models.util.make_source_tech(info: Scenario | ScenarioInfo, common, **values) → MutableParameterData

Return parameter data for a ‘source’ technology.

The technology has no inputs; its output commodity and/or level are determined by common; either single values, or None if the result will be pipe()’d through broadcast().

Parameters:

• info (Scenario or ScenarioInfo)

• common (dict) – Passed to make_df().

• **values – Values for ‘capacity_factor’ (optional; default 1.0), ‘output’, ‘var_cost’, and optionally ‘technical_lifetime’.

Returns:

Suitable for add_par_data().

Return type:

dict

message_ix_models.util.mark_time(quiet: bool = False) → None

Record and log (if quiet is True) a time mark.

message_ix_models.util.maybe_query(series: Series, query: str | None) → Series

Apply pandas.DataFrame.query() if the query arg is not None.

query() is not chainable (pandas-dev/pandas#37941). Use this function with pandas.Series.pipe(), passing an argument that may be None, to have a chainable query operation that can be a no-op.

message_ix_models.util.merge_data(base: MutableParameterData, *others: ParameterData) → None

Merge dictionaries of DataFrames together into base.

message_ix_models.util.minimum_version(expr: str, raises: Iterable[type[Exception]] | None = None) → Callable

Decorator for functions that require a minimum version of some upstream package.

If the decorated function is called and the condition in expr is not met, NotImplementedError is raised with an informative message.

The decorated function gains an attribute .minimum_version, a pytest MarkDecorator that can be used on associated test code. This marks the test as XFAIL, raising NotImplementedError (directly); RuntimeError or AssertionError (for instance, via click test utilities), or any of the classes given in the raises argument.

See prepare_reporter() / test_prepare_reporter() for a usage example.

Parameters:

expr – Like “example 1.2.3.post0”. The condition for the decorated function is that the installed version must be equal to or greater than this version.

message_ix_models.util.package_data_path(*parts) → Path

Construct a path to a file under message_ix_models/data/.

Use this function to access data packaged and installed with message_ix_models.

Parameters:

parts (Sequence of str or Path) – Joined to the base path using joinpath().

See also

Choose locations for data

message_ix_models.util.path_fallback(*parts: str | Path, where: str | list[str | Path] = '') → Path

Locate a path constructed from parts found in the first of several directories.

This allows to implement ‘fallback’ behaviour in which files or directories in certain locations are used preferentially.

Parameters:

• parts – Path parts or fragments such as directory names and a final file name.

• where –

  Either:

  • str containing one or more of:

    • ”cache”: locate parts in the message_ix_models cache directory.

    • ”local”: locate parts in the user’s local data directory (same as :func:`local_data_path).

    • ”package”: locate parts in message_ix_models package data (same as package_data_path()).

    • ”private”: locate parts in the message_data /data/ directory (same as private_data_path()).

    • ”test”: locate test data in package_data_path("test", ...)

  • list where each element is str (one of the above) or a pathlib.Path.

Returns:

The first of the locations indicated by where in which the file or directory parts exists.

Return type:

pathlib.Path

Raises:

ValueError – If where is empty or parts are not found in any of the indicated locations.

message_ix_models.util.preserve_log_level()

Context manager to preserve the level of the message_ix_models logger.

message_ix_models.util.private_data_path(*parts) → Path

Construct a path to a file under data/ in message_data.

Use this function to access non-public (for instance, embargoed or proprietary) data stored in the message_data repository.

If the repository is not available, the function falls back to Context.get_local_path(), where users may put files obtained through other messages.

Parameters:

parts (Sequence of str or Path) – Joined to the base path using joinpath().

See also

Choose locations for data

message_ix_models.util.replace_par_data(scenario: Scenario, parameters: str | Sequence[str], filters: Mapping[str, str | int | Collection[str] | Collection[int]], to_replace: Mapping[str, Mapping[str, str] | Mapping[int, int]]) → None

Replace data in parameters of scenario.

Parameters:

• scenario – Scenario in which to replace data.

• parameters (str or Sequence of str) – Name(s) of parameters in which to replace data.

• filters – Passed to Scenario.par() argument of the same name.

• to_replace – Passed to pandas.DataFrame.replace() argument of the same name.

Examples

Replace data in the “relation_activity” parameter for a particular technology and relation: assign the same values as entries in a different relation name for the same technology.

>>> replace_par_data(
...     scenario,
...     "relation_activity",
...     dict(technology="hp_gas_i", relation="CO2_r_c"),
...     dict(relation={"CO2_r_c": "CO2_ind"}),
... )

message_ix_models.util.same_node(data: DataFrame, from_col: str = 'node_loc') → DataFrame

message_ix_models.util.same_node(data: MutableParameterData, from_col: str = 'node_loc') → MutableParameterData

Fill ‘node_{,dest,loc,origin,rel,share}’ in df from from_col.

message_ix_models.util.same_time(data: DataFrame) → DataFrame

message_ix_models.util.same_time(data: MutableParameterData) → MutableParameterData

Fill ‘time_origin’/’time_dest’ in df from ‘time’.

message_ix_models.util.series_of_pint_quantity(*args, **kwargs) → Series

Suppress a spurious warning.

Creating a pandas.Series with a list of pint.Quantity triggers a warning “The unit of the quantity is stripped when downcasting to ndarray,” even though the entire object is being stored and the unit is not stripped. This function suppresses this warning.

message_ix_models.util.show_versions() → str

Output of ixmp.show_versions(), as a str.

message_ix_models.util.silence_log(names: str | None = None, level: int = 40)

Context manager to temporarily quiet 1 or more loggers.

Parameters:

• names (str, optional) – Space-separated names of loggers to quiet.

• level (int, optional) – Minimum level of log messages to allow.

Examples

>>> with silence_log():
>>>     log.warning("This message is not recorded.")

message_ix_models.util.strip_par_data(scenario: Scenario, set_name: str, element: str, dry_run: bool = False, dump: MutableParameterData | None = None) → int

Remove element from set_name in scenario, optionally dumping to dump.

Parameters:

• dry_run (bool, optional) – If True, only show what would be done.

• dump (dict, optional) – If provided, stripped data are stored in this dictionary. Otherwise, they are discarded.

Returns:

Total number of rows removed across all parameters.

Return type:

int

See also

add_par_data

message_ix_models.util.cache.SKIP_CACHE = False

Controls whether cached data is returned for functions decorated with cached(). Set to True to force reload.

util.click

Command-line utilities.

These are used for building CLIs using click.

PARAMS contains, among others:

• --urls-from-file=… Path to a file containing scenario URLs, one per line. These are parsed and stored on Config.scenarios.

class message_ix_models.util.click.CliRunner(cli_cmd: ~click.core.Command, cli_module: str, env: ~collections.abc.Mapping[str, str] = <factory>, charset: str = 'utf-8', method: ~typing.Literal['click', 'subprocess'] = 'click')

Similar to click.testing.CliRunner, with extra features.

assert_exit_0(*args, **kwargs) → Result

Assert a result has exit_code 0, or print its traceback.

If any args or kwargs are given, invoke() is first called. Otherwise, the result from the last call of invoke() is used.

Raises:

AssertionError – if the result exit code is not 0.

cli_cmd: Command

CLI entry point

cli_module: str

CLI module

invoke_subprocess(*args, **kwargs) → Result

Invoke the CLI in a subprocess.

method: Literal['click', 'subprocess'] = 'click'

Method for invoking the command

message_ix_models.util.click.PARAMS = {'dest': <Option dest>, 'dry_run': <Option dry_run>, 'force': <Option force>, 'nodes': <Option nodes>, 'output_model': <Option output_model>, 'platform_dest': <Option platform_dest>, 'policy_path': <Option policy_path>, 'quiet': <Option quiet>, 'regions': <Option regions>, 'rep_out_path': <Option rep_out_path>, 'rep_template': <Option rep_template>, 'run_reporting_only': <Option run_reporting_only>, 'ssp': <Argument ssp>, 'urls_from_file': <Option urls_from_file>, 'verbose': <Option verbose>, 'years': <Option years>}

Common command-line parameters (arguments and options). See common_params().

message_ix_models.util.click.common_params(param_names: str)

Decorate a click.command with common parameters param_names.

param_names must be a space-separated string of names appearing in PARAMS, for instance "ssp force output_model". The decorated function receives keyword arguments with these names; some are also stored on the

Example

>>> @click.command
... @common_params("ssp force output_model")
... @click.pass_obj
... def mycmd(context, ssp, force, output_model):
...     assert context.force == force

message_ix_models.util.click.default_path_cb(*default_parts)

Return a callback function for click.Option handling.

If no option value is given, the callback uses Context.get_local_path() and default_parts to provide a path that is relative to local data directory, e.g. the current working directory (see Data, metadata, and configuration).

message_ix_models.util.click.exec_cb(expression: str) → Callable

Return a callback that exec()-utes an expression.

The expression is executed in a limited context that has only two names available:

• context: the Context instance.

• value: the value passed to the click.Parameter.

Example

>>> @click.command
... @click.option(
...     "--myopt", callback=exec_cb("context.my_mod.my_opt = value + 3")
... )
... def cmd(...):
...     ...

message_ix_models.util.click.format_sys_argv() → str

Format sys.argv in a readable manner.

message_ix_models.util.click.scenario_param(param_decls: str | list[str], *, values: list[str] = None, default: str | None = None)

Add an SSP or scenario option or argument to a click.Command.

The parameter uses store_context() to store the given value (if any) on the Context.

Parameters:

• param_decls – "--ssp" (or any other name prefixed by --) to generate a click.Option; "ssp" to generate a click.Argument. Click-style declarations are also supported; see below.

• values – Allowable values. If not given, the allowable values are [“LED”, “SSP1”, “SSP2”, “SSP3”, “SSP4”, “SSP5”].

• default – Default value.

Raises:

ValueError – if default is given with param_decls that indicate a click.Argument.

Examples

Add a (mandatory, positional) click.Argument. This is nearly the same as using common_params("ssp"), except the decorated function does not receive an ssp argument. The value is still stored on context automatically.

>>> @click.command
... @scenario_param("ssp")
... @click.pass_obj
... def mycmd(context):
...     print(context.ssp)

Add a click.Option with certain, limited values and a default:

>>> @click.command
... @scenario_param("--ssp", values=["SSP1", "SSP2", "SSP3"], default="SSP3")
... @click.pass_obj
... def mycmd(context):
...     print(context.ssp)

An option given by the user as --scenario but stored as Context.ssp:

>>> @click.command
... @scenario_param(["--scenario", "ssp"])
... @click.pass_obj
... def mycmd(context):
...     print(context.ssp)

message_ix_models.util.click.store_context(context: Context | Context, param, value)

Callback that simply stores a value on the Context object.

Use this for parameters that are not used directly in a @click.command() function, but need to be carried by the Context for later use.

message_ix_models.util.click.temporary_command(group: Group, command: Command)

Temporarily attach command command to group.

message_ix_models.util.click.unique_id() → str

Return a unique ID for a CLI invocation.

The return value resembles “mix-models-debug-3332d415ef65bf2a-2023-02-02T162931” and contains:

• The CLI name and (sub)commands.

• A hash of all the CLI parameters (options and arguments).

• The current date and time, in ISO format with Windows-incompatible “:” removed.

message_ix_models.util.click.urls_from_file(context: Context | Context, param, value) → list[ScenarioInfo]

Callback to parse scenario URLs from value.

util.config

class message_ix_models.util.config.ConfigHelper

Mix-in for dataclass-based configuration classes.

This provides methods read_file(), replace(), and from_dict() that help to use dataclass classes for handling message_ix_models configuration.

All 3 methods take advantage of name manipulations: the characters “-” and “ “ are replaced with underscores (“_”). This allows to write the names of attributes in legible ways—e.g. “attribute name” or “attribute-name” instead of “attribute_name”— in configuration files and/or code.

It also add hexdigest().

classmethod from_dict(data: Mapping)

Construct an instance from data with name manipulation.

hexdigest(length: int = -1) → str

Return a hex digest that is unique for distinct settings on the instance.

Returns:

If length is non-zero, a string of this length; otherwise a 32-character string from blake2s.hexdigest().

Return type:

str

read_file(path: Path, fail='raise') → None

Update configuration from file.

Parameters:

• path – to a .yaml file containing a top-level mapping.

• fail (str) – if “raise” (the default), any names in path which do not match attributes of the dataclass raise a ValueError. Ottherwise, a message is logged.

replace(**kwargs)

Like dataclasses.replace() with name manipulation.

update(**kwargs)

Update attributes in-place.

Raises:

AttributeError – Any of the kwargs are not fields in the data class.

util.context

Context and settings for message_ix_models code.

message_ix_models.util.context.MODULE_WITH_CONFIG_DATACLASS: tuple[tuple[str, str | None, bool, bool], ...] = (('message_ix_models.util.config', 'core', True, True), ('message_ix_models.model.config', 'model', True, True), ('message_ix_models.report.config', 'report', True, False), ('message_ix_models.transport.config', 'transport', False, False))

Tuples containing:

1. Full name of module that contains a dataclass named Config.

2. Context key where an instance of the Config class is stored.

3. True if such an instance should be created by default for every Context instance.

   This should be False if creation of the Config instance is slow or has side effects, as this will occur even where the module (1) is not in use.

4. True if the dataclass fields of the Config class should be ‘aliased’, or directly available on Context instances.

   This should be False for all new modules/Config classes. Aliasing is provided only for backwards-compatible support of earlier code.

class message_ix_models.util.context.Context(*args, **kwargs)

Context and settings for message_ix_models code.

Context is dict-like, so dict methods including copy() and setdefault() may be used to handle settings. It also provides attribute access: context.foo is equivalent to context["foo"].

A Context instance always has the following members:

1. core: an instance of message_ix_models.Config.

2. model: an instance of message_ix_models.model.Config.

3. report: an instance of message_ix_models.report.Config.

Attributes of (1) and (2) may be accessed by shorthand/aliases. For instance, context.regions is an alias for context.model.regions. However, for clarity and to support type checking, explicit reference to the configuration class and its attributes should be used.

Context provides additional methods to do common tasks that depend on configurable settings:

clone_to_dest([create])	Return a scenario based on the --dest command-line option.
close_db()	Shorthand for Config.close_db().
delete()	Hide the current Context from future get_instance() calls.
get_cache_path(*parts)	Return a path to a local cache file, i.e. within cache_path.
get_local_path(*parts[, suffix])	Return a path under Config.local_data.
get_platform([reload])	Return a Platform from Config.platform_info.
get_scenario()	Return a Scenario from scenario_info.
handle_cli_args([url, platform, model_name, ...])	Handle command-line arguments.
only()	Return the only Context instance.
use_defaults(settings)	Update from settings.

clone_to_dest(create=True) → message_ix.Scenario

Return a scenario based on the --dest command-line option.

Parameters:

create (bool, optional) – If True (the default) and the base scenario does not exist, a bare RES scenario is created. Otherwise, an exception is raised.

Returns:

To prevent the scenario from being garbage collected, keep a reference to its Platform:

Return type:

Scenario

See also

create_res

To use this method, either decorate a command with common_params():

from message_data.tools.cli import common_params

@click.command()
@common_params("dest")
@click.pass_obj
def foo(context, dest):
    scenario, mp = context.clone_to_dest()

or, store the settings Config.dest_scenario and optionally Config.dest_platform on context:

c = Context.get_instance()

c.dest_scenario = dict(model="foo model", scenario="foo scenario")
scenario_mp = context.clone_to_dest()

The resulting scenario has the indicated model- and scenario names.

If --url (or --platform, --model, --scenario, and optionally --version) are given, the identified scenario is used as a ‘base’ scenario, and is cloned. If --url/--platform and --dest refer to different Platform instances, then this is a two-platform clone.

If no base scenario can be loaded, bare.create_res() is called to generate a base scenario.

asdict() → dict

Return a deepcopy() of the Context’s values as a dict.

close_db() → None

Shorthand for Config.close_db().

property core: message_ix_models.util.config.Config

An instance of util.config.Config.

delete()

Hide the current Context from future get_instance() calls.

get(key: str, default: Optional = None)

Retrieve the value for key.

get_cache_path(*parts) → Path

Return a path to a local cache file, i.e. within cache_path.

Shorthand for Config.get_cache_path().

classmethod get_instance(index=0) → Context

Return a Context instance; by default, the first created.

Parameters:

index (int, optional) – Index of the Context instance to return, e.g. -1 for the most recently created.

get_local_path(*parts: str, suffix=None) → Path

Return a path under Config.local_data.

Shorthand for Config.get_local_path().

get_platform(reload: bool = False) → Platform

Return a Platform from Config.platform_info.

Shorthand for Config.get_platform().

get_scenario() → message_ix.Scenario

Return a Scenario from scenario_info.

Shorthand for Config.get_scenario().

handle_cli_args(url=None, platform=None, model_name=None, scenario_name=None, version=None, local_data=None, verbose: bool = False, _store_as: tuple[str, str] = ('platform_info', 'scenario_info'))

Handle command-line arguments.

Shorthand for Config.handle_cli_args().

property model: message_ix_models.model.config.Config

An instance of model.config.Config.

classmethod only() → Context

Return the only Context instance.

Raises:

IndexError – If there is more than one instance.

property report: message_ix_models.report.config.Config

An instance of report.config.Config.

set(key: str, value: Any) → None

Change the stored value for key.

set_scenario(scenario: message_ix.Scenario) → None

Update Config.scenario_info to match an existing scenario.

Shorthand for Config.set_scenario().

use_defaults(settings)

Update from settings.

write_debug_archive() → None

Write an archive containing the files listed in Config.debug_paths.

Shorthand for Config.write_debug_archive().

util.importlib

Load model and project code from message_data.

class message_ix_models.util.importlib.MessageDataFinder

Load model and project code from message_data.

util._logging

Logging utilities.

class message_ix_models.util._logging.Formatter(use_colour: bool = True)

Formatter for log records.

Parameters:

use_color (bool, optional) – If True, colorama is used to colour log messages.

format(record)

Format record.

Records are formatted like:

model.transport.data.add_par_data  220 rows in 'input'
...add_par_data  further messages

…with the calling function name (e.g. ‘add_par_data’) coloured for legibility on first occurrence, then dimmed when repeated.

class message_ix_models.util._logging.QueueListener(queue, *handlers, respect_handler_level=False)

logging.QueueListener with a flush() method.

flush()

Flush the queue: join the listener/monitor thread and then restart.

class message_ix_models.util._logging.SilenceFilter(names: str, level: int)

Log filter that only allows records from names that are at or above level.

filter(record) → bool

Determine if the specified record is to be logged.

Returns True if the record should be logged, or False otherwise. If deemed appropriate, the record may be modified in-place.

class message_ix_models.util._logging.StreamHandler(stream_name: str)

Like logging.StreamHandler, but retrieve the stream on each access.

This avoids the case that click, pytest, or something else adjusts sys.stdout temporarily, but the handler’s stored reference to the original is not updated.

stream_name: str

Name of the sys stream to use, as str rather than a direct reference.

message_ix_models.util._logging.setup(level: str | int = 99, console: bool = True, *, file: bool = False) → None

Initialize logging.

Parameters:

• level (str, optional) – Log level for the console log handler.

• console (bool, optional) – If False, do not print any messages to console.

• file (bool, optional) – If False, do not print any messages to file.

util.node

Utilities for nodes.

message_ix_models.util.node.NODE_DIMS = ['n', 'node', 'node_loc', 'node_origin', 'node_dest', 'node_rel', 'node_share']

Names of dimensions indexed by ‘node’.

Todo

to be robust to changes in message_ix, read these names from that package.

message_ix_models.util.node.R11_R12 = (('R11_AFR', 'R12_AFR'), ('R11_CPA', 'R12_CHN'), ('R11_EEU', 'R12_EEU'), ('R11_FSU', 'R12_FSU'), ('R11_LAM', 'R12_LAM'), ('R11_MEA', 'R12_MEA'), ('R11_NAM', 'R12_NAM'), ('R11_PAO', 'R12_PAO'), ('R11_PAS', 'R12_PAS'), ('R11_CPA', 'R12_RCPA'), ('R11_SAS', 'R12_SAS'), ('R11_WEU', 'R12_WEU'))

Mapping from R11 to R12 node IDs.

message_ix_models.util.node.R11_R14 = (('R11_AFR', 'R14_AFR'), ('R11_FSU', 'R14_CAS'), ('R11_CPA', 'R14_CPA'), ('R11_EEU', 'R14_EEU'), ('R11_LAM', 'R14_LAM'), ('R11_MEA', 'R14_MEA'), ('R11_NAM', 'R14_NAM'), ('R11_PAO', 'R14_PAO'), ('R11_PAS', 'R14_PAS'), ('R11_FSU', 'R14_RUS'), ('R11_SAS', 'R14_SAS'), ('R11_FSU', 'R14_SCS'), ('R11_FSU', 'R14_UBM'), ('R11_WEU', 'R14_WEU'))

Mapping from R11 to R14 node IDs.

message_ix_models.util.node.adapt_R11_R12 = <message_ix_models.util.common.MappingAdapter object>

Adapt data from the R11 to the R14 node list.

The data is adapted using the mappings in R11_R12 for each of the dimensions in NODE_DIMS.

message_ix_models.util.node.adapt_R11_R14 = <message_ix_models.util.common.MappingAdapter object>

Adapt data from the R11 to the R14 node list.

The data is adapted using the mappings in R11_R14 for each of the dimensions in NODE_DIMS.

message_ix_models.util.node.nodes_ex_world(nodes: Sequence[str | Code]) → list[str | Code]

Exclude “World” and anything containing “GLB” from nodes.

May also be used as a genno (reporting) operator.

util.pooch

Utilities for using Pooch.

class message_ix_models.util.pooch.Extract(members=None, extract_dir=None)

Similar to pooch.Unzip, streamlined using pathlib.

This version supports:

• Absolute or relative paths for the extract_dir parameter.

• .zip or .tar.xz archives.

message_ix_models.util.pooch.GH_MAIN = 'https://github.com/iiasa/message-ix-models/raw/main/message_ix_models/data'

Base URL portion for files stored in the message-ix-models GitHub repository.

message_ix_models.util.pooch.SOURCE: Mapping[str, Mapping[str, Any]] = {'MESSAGEix-Materials': {'pooch_args': {'base_url': 'https://github.com/iiasa/message-ix-models/raw/main/message_ix_models/data/material/', 'registry': {'material.tar.gz': 'sha1:b93f4390cb00749c2316fcdddf901a0eab896774'}}, 'processor': <message_ix_models.util.pooch.Extract object>}, 'MESSAGEix-Nexus': {'pooch_args': {'base_url': 'https://github.com/iiasa/message-ix-models/raw/main/message_ix_models/data/water/', 'registry': {'water.tar.xz': 'sha1:ec9e0655af90ca844c0158968bb03a194b8fa6c6'}}, 'processor': <message_ix_models.util.pooch.Extract object>}, 'PRIMAP': {'pooch_args': {'base_url': 'ftp://datapub.gfz-potsdam.de/download/10.5880.PIK.2019.001/', 'registry': {'PRIMAP-hist_v2.0_11-Dec-2018.zip': 'md5:f28d58abef4ecfc36fc8ce3e9eef2871'}}, 'processor': <message_ix_models.util.pooch.Extract object>}, 'SSP-Update-3.0': {'pooch_args': {'base_url': 'https://github.com/iiasa/message-ix-models/raw/main/message_ix_models/data/ssp/', 'registry': {'1706548837040-ssp_basic_drivers_release_3.0_full.csv.gz': 'sha1:e2af7a88aeed7d0e44ceaf2dff60f891cf551517'}}}, 'SSP-Update-3.0.1': {'pooch_args': {'base_url': 'https://github.com/iiasa/message-ix-models/raw/main/message_ix_models/data/ssp/', 'registry': {'1710759470883-ssp_basic_drivers_release_3.0.1_full.csv.gz': 'sha1:e5c24c27ee743e79dac5a578235b35a68cd64183'}}}, 'snapshot-0': {'pooch_args': {'base_url': 'doi:10.5281/zenodo.5793870', 'registry': {'MESSAGEix-GLOBIOM_1.1_R11_no-policy_baseline.xlsx': 'md5:222193405c25c3c29cc21cbae5e035f4'}}, 'processor': <message_ix_models.util.pooch.UnpackSnapshot object>}, 'snapshot-1': {'pooch_args': {'base_url': 'doi:10.5281/zenodo.10514052', 'registry': {'MESSAGEix-GLOBIOM_1.1_R11_no-policy_baseline.xlsx': 'md5:e7c0c562843e85c643ad9d84fecef979'}}}}

Supported remote sources of data.

class message_ix_models.util.pooch.UnpackSnapshot

Pooch processor that calls snapshot.unpack().

message_ix_models.util.pooch.fetch(pooch_args: dict, *, extra_cache_path: str | None = None, **fetch_kwargs) → tuple[Path, ...]

Create a Pooch instance and fetch a single file.

Files are stored under the directory identified by Context.get_cache_path(), unless args provides another location.

Parameters:

• args – Passed to pooch.create().

• kwargs – Passed to pooch.Pooch.fetch().

Returns:

Path to the fetched file.

Return type:

Path

See also

snapshot.load()

util.pycountry

message_ix_models.util.pycountry.COUNTRY_NAME = {'Korea': 'Korea, Republic of', 'Republic of Korea': 'Korea, Republic of', 'Russia': 'Russian Federation', 'South Korea': 'Korea, Republic of', 'Turkey': 'Türkiye'}

Mapping from common, non-standard country names to exact field values occurring in the ISO 3166-1 database.

Other code may extend this mapping before calling iso_3166_alpha_3().

message_ix_models.util.pycountry.iso_3166_alpha_3(name: str) → str | None

Return an ISO 3166 alpha-3 code for a country name.

Parameters:

name (str) – Country name. This is looked up in the pycountry ‘name’, ‘official_name’, or ‘common_name’ field. Values in COUNTRY_NAME are supported.

Return type:

str or None

util.scenarioinfo

ScenarioInfo class.

class message_ix_models.util.scenarioinfo.ScenarioInfo(scenario_obj: dataclasses.InitVar[typing.Optional[ForwardRef('Scenario')]] = None, empty: dataclasses.InitVar[bool] = False, platform_name: str | None = None, model: str | None = None, scenario: str | None = None, version: int | None = None, set: dict[str, list] = <factory>, par: dict[str, ~pandas.core.frame.DataFrame] = <factory>, y0: int = -1, is_message_macro: bool = False, _yv_ya: ~pandas.core.frame.DataFrame | None = None)

Information about a Scenario object.

Code that prepares data for a target Scenario can accept a ScenarioInfo instance. This avoids the need to create or load an actual Scenario or its data, which can be a performance-limiting step.

ScenarioInfo objects can also be used (for instance, by apply_spec()) to describe the contents of a Scenario before it is created.

ScenarioInfo objects have the following convenience attributes:

set	Elements of ixmp/message_ix sets.
io_units(technology, commodity[, level])	Return units for the MESSAGE input or output parameter.
is_message_macro	True if a MESSAGE-MACRO scenario.
N	Elements of the set 'node'.
units_for(set_name, id)	Return the units associated with code id in MESSAGE set set_name.
Y	Elements of the set 'year' that are >= the first model year.
y0	First model year, if set, else Y[0].
yv_ya	pandas.DataFrame with valid year_vtg, year_act pairs.

Parameters:

scenario_obj (message_ix.Scenario) – If given, set is initialized from this existing scenario.

Examples

Iterating over an instance gives “model”, “scenario”, “version” and the values of the respective attributes: >>> si = ScenarioInfo.from_url(“model name/scenario name#123”) >>> dict(si) {‘model’: ‘model name’, ‘scenario’: ‘scenario name’, ‘version’: 123}

See also

Spec

property N

Elements of the set ‘node’.

See also

nodes_ex_world

property Y: list[int]

Elements of the set ‘year’ that are >= the first model year.

classmethod from_url(url: str) → ScenarioInfo

Create an instance using only an url.

io_units(technology: str, commodity: str, level: str | None = None) → Unit

Return units for the MESSAGE input or output parameter.

These are implicitly determined as the ratio of:

• The units for the origin (for input) or destination commodity, per units_for().

• The units of activity for the technology.

Parameters:

level (str) – Placeholder for future functionality, i.e. to use different units per (commodity, level). Currently ignored. If given, a debug message is logged.

Raises:

ValueError – if either technology or commodity lack defined units.

is_message_macro: bool = False

True if a MESSAGE-MACRO scenario.

model: str | None = None

Model name; equivalent to TimeSeries.model.

par: dict[str, DataFrame]

Elements of ixmp/message_ix parameters.

property path: str

A valid file system path name similar to url.

Characters invalid in Windows paths are replaced with “_”.

scenario: str | None = None

Scenario name; equivalent to TimeSeries.scenario.

set: dict[str, list]

Elements of ixmp/message_ix sets.

substitute_codes() → None

Update the members of set using get_codelist().

Members of the following set(s) that are str are replaced with codes loaded from the corresponding code list(s), including all annotations:

• commodity

Todo

Extend for other sets and cases where there are multiple code lists (for instance year, region).

units_for(set_name: str, id: str) → Unit

Return the units associated with code id in MESSAGE set set_name.

ixmp (or the sole JDBCBackend, as of v3.5.0) does not handle unit information for variables and equations (unlike parameter values), such as MESSAGE decision variables ACT, CAP, etc. In message_ix_models and message_data, the following conventions are (generally) followed:

• The units of ACT and others are consistent for each technology.

• The units of COMMODITY_BALANCE, STOCK, commodity_stock, etc. are consistent for each commodity.

Thus, codes for elements of these sets (e.g. Commodities (commodity.yaml)) can be used to carry the standard units for the corresponding quantities. units_for() retrieves these units, for use in model-building and reporting.

Todo

Expand this discussion and transfer to the message_ix docs.

See also

io_units

update(other: ScenarioInfo)

Update with the set elements of other.

property url: str

Identical to TimeSeries.url.

version: int | None = None

Scenario version; equivalent to TimeSeries.version.

y0: int = -1

First model year, if set, else Y[0].

year_from_codes(codes: list[Code])

Update using a list of codes.

The following are updated:

• set year

• set cat_year, with the first model year.

• par duration_period

Any existing values are discarded.

After this, the attributes y0 and Y give the first model year and model years, respectively.

Examples

Get a particular code list, create a ScenarioInfo instance, and update using the codes:

>>> years = get_codes("year/A")
>>> info = ScenarioInfo()
>>> info.year_from_codes(years)

Use populated values:

>>> info.y0
2020
>>> info.Y[:3]
[2020, 2030, 2040]
>>> info.Y[-3:]
[2090, 2100, 2110]

property yv_ya

pandas.DataFrame with valid year_vtg, year_act pairs.

class message_ix_models.util.scenarioinfo.Spec(add: ~message_ix_models.util.scenarioinfo.ScenarioInfo = <factory>, remove: ~message_ix_models.util.scenarioinfo.ScenarioInfo = <factory>, require: ~message_ix_models.util.scenarioinfo.ScenarioInfo = <factory>)

A specification for the structure of a model or variant.

A Spec collects 3 ScenarioInfo instances at the attributes add, remove, and require. This is the type that is accepted by apply_spec(); Building models (model.build) describes how a Spec is used to modify a Scenario. A Spec may also be used to express information about the target structure of data to be prepared; like ScenarioInfo, this can happen before the target Scenario exists.

Spec also provides:

• Dictionary-style access, e.g. s["add"] is equivalent to s.add..

• Error checking; setting keys other than add/remove/require results in an error.

• merge(), a helper method.

add: ScenarioInfo

Structure to be added to a base scenario.

static merge(a: Spec, b: Spec) → Spec

Merge Specs a and b together.

Returns a new Spec where each member is a union of the respective members of a and b.

remove: ScenarioInfo

Structure to be removed from a base scenario.

require: ScenarioInfo

Structure that must be present in a base scenario.

util.sdmx

Utilities for handling objects from sdmx.

class message_ix_models.util.sdmx.URNLookupEnum(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)

Enum subclass that allows looking up members using a URN.

classmethod by_urn(urn: str)

Return the Enum member given its urn.

message_ix_models.util.sdmx.as_codes(data: list[str] | dict[str, str | Code]) → list[Code]

Convert data to a list of Code objects.

Various inputs are accepted:

• list of str.

• dict, in which keys are id and values are further dict with keys matching other Code attributes.

message_ix_models.util.sdmx.eval_anno(obj: AnnotableArtefact, id: str)

Retrieve the annotation id from obj, run eval() on its contents.

Deprecated since version 2023.9.12: Use sdmx.model.common.AnnotableArtefact.eval_annotation(), which provides the same functionality.

message_ix_models.util.sdmx.make_enum(urn, base=<enum 'URNLookupEnum'>)

Create an enum.Enum (or base) with members from codelist urn.

message_ix_models.util.sdmx.read(urn: str, base_dir: PathLike | None = None)

Read SDMX object from package data given its urn.

message_ix_models.util.sdmx.register_agency(agency: Agency) → AgencyScheme

Add agency to the AgencyScheme “IIASA_ECE:AGENCIES”.

message_ix_models.util.sdmx.write(obj, base_dir: PathLike | None = None, basename: str | None = None)

Store an SDMX object as package data.

types

Types for hinting.

message_ix_models.types.MutableParameterData

Mutable collection of message_ix or ixmp parameter data.

alias of MutableMapping[Hashable, DataFrame]

message_ix_models.types.ParameterData

Collection of message_ix or ixmp parameter data. Keys should be parameter names (str), and values should be data frames with the same structure returned by make_df(). See also add_par_data().

alias of Mapping[Hashable, DataFrame]