Low-level utilities (util)

Submodules:

click	Command-line utilities.
context	Context and settings for message_ix_models code.
importlib	Load model and project code from message_data.
_logging	Logging utilities.
scenarioinfo	ScenarioInfo class.

Commonly used:

adapt_R11_R14(data)	Adapt data from R11 to R14 node list.
as_codes(data)	Convert data to a list of Code objects.
broadcast(df, **kwargs)	Fill missing data in df by broadcasting.
cached(func)	Decorator to cache the return value of a function func.
check_support(context[, settings, desc])	Check whether a Context is compatible with certain settings.
convert_units(s, unit_info[, store])	Convert units of s, for use with apply().
copy_column(column_name)	For use with pandas.DataFrame.assign().
ffill(df, dim, values[, expr])	Forward-fill df on dim to cover values.
identify_nodes(scenario)	Return the ID of a node codelist given the contents of scenario.
load_package_data(*parts[, suffix])	Load a message_ix_models package data file and return its contents.
load_private_data(*parts)	Load a private data file from message_data and return its contents.
make_io(src, dest, efficiency[, on])	Return input and output data frames for a 1-to-1 technology.
make_matched_dfs(base, **par_value)	Return data frames derived from base for multiple parameters.
make_source_tech(info, common, **values)	Return parameter data for a ‘source’ technology.
maybe_query(series, query)	Apply pandas.Series.query() if the query arg is not None.
merge_data(base, *others)	Merge dictionaries of DataFrames together into base.
package_data_path(*parts)	Construct a path to a file under message_ix_models/data/.
private_data_path(*parts)	Construct a path to a file under data/ in message_data.
same_node(df)	Fill ‘node_origin’/’node_dest’ in df from ‘node_loc’.
series_of_pint_quantity(*args, **kwargs)	Suppress a spurious warning.
Context(*args, **kwargs)	Context and settings for message_ix_models code.
ScenarioInfo([scenario])	Information about a Scenario object.

message_ix_models.util.adapt_R11_R14(data: Dict[str, Union[pandas.core.frame.DataFrame, genno.core.quantity.Quantity]])

Adapt data from R11 to R14 node list.

The data is adapted by:

• Renaming regions such as R11_NAM to R14_NAM.

• Copying the data for R11_FSU to R14_CAS, R14_RUS, R14_SCS, and R14_UBM.

…wherever these appear in a column/dimension named ‘node’, ‘node_*’, or ‘n’.

The function may be called with:

• pandas.DataFrame,

• genno.Quantity, or

• dict mapping str parameter names to values (either of the above types).

message_ix_models.util.cached(func: Callable) → Callable

Decorator to cache the return value of a function func.

On a first call, the data requested is returned and also cached under Context.get_cache_path(). On subsequent calls, if the cache exists, it is used instead of calling the (possibly slow) func.

When SKIP_CACHE is true, func is always called.

See also

Caching in the genno documentation

message_ix_models.util.check_support(context, settings={}, desc: str = '') → None

Check whether a Context is compatible with certain settings.

Raises

• NotImplementedError – if any context value for a key of settings is not among the values in settings.

• KeyError – if the key is not set on context at all.

See also

Advertise and check compatibility

message_ix_models.util.convert_units(s: pandas.core.series.Series, unit_info: Mapping[str, Tuple[float, str, Optional[str]]], store='magnitude') → pandas.core.series.Series

Convert units of s, for use with apply().

s.name is used to retrieve a tuple of (factor, input_unit, output_unit) from unit_info. The (float) values of s are converted to pint.Quantity with the input_unit and factor; then cast to output_unit, if provided.

Parameters

• s (pandas.Series) –

• unit_info (dict (str -> tuple)) – Mapping from quantity name (matched to s.name) to 3-tuples of (factor, input_unit, output_unit). output_unit may be None. For example, see ikarus.UNITS.

• store ("magnitude" or "quantity") – If “magnitude”, the values of the returned series are the magnitudes of the results, with no output units. If “quantity”, the values are scalar Quantity objects.

Returns

Same shape, index, and values as s, with output units.

Return type

pandas.Series

message_ix_models.util.identify_nodes(scenario: message_ix.core.Scenario) → str

Return the ID of a node codelist given the contents of scenario.

Returns

The ID of the Node code lists containing the regions of scenario.

Return type

str

Raises

ValueError – if no codelist can be identified, or the nodes in the scenario do not match the children of the “World” node in the codelist.

message_ix_models.util.load_package_data(*parts: str, suffix: Optional[str] = '.yaml') → Any

Load a message_ix_models package data file and return its contents.

Data is re-used if already loaded.

Example

The single call:

>>> info = load_package_data("node", "R11")

1. loads the metadata file data/node/R11.yaml, parsing its contents,

2. stores those values at PACKAGE_DATA["node R11"] for use by other code, and

3. returns the loaded values.

Parameters

• parts (iterable of str) – Used to construct a path under message_ix_models/data/.

• suffix (str, optional) – File name suffix, including, the “.”, e.g. .yaml.

Returns

Configuration values that were loaded.

Return type

dict

message_ix_models.util.load_private_data(*parts: str) → Mapping

Load a private data file from message_data and return its contents.

Analogous to load_package_data, but for non-public data.

Parameters

parts (iterable of str) – Used to construct a path under data/ in the message_data repository.

Returns

Configuration values that were loaded.

Return type

dict

Raises

RuntimeError – if message_data is not installed.

message_ix_models.util.maybe_query(series: pandas.core.series.Series, query: Optional[str]) → pandas.core.series.Series

Apply pandas.Series.query() if the query arg is not None.

query() is not chainable (pandas-dev/pandas#37941). Use this function with pandas.Series.pipe(), passing an argument that may be None, to have a chainable query operation that can be a no-op.

message_ix_models.util.package_data_path(*parts) → pathlib.Path

Construct a path to a file under message_ix_models/data/.

message_ix_models.util.private_data_path(*parts) → pathlib.Path

Construct a path to a file under data/ in message_data.

message_ix_models.util.series_of_pint_quantity(*args, **kwargs) → pandas.core.series.Series

Suppress a spurious warning.

Creating a pandas.Series with a list of pint.Quantity triggers a warning “The unit of the quantity is stripped when downcasting to ndarray,” even though the entire object is being stored and the unit is not stripped. This function suppresses this warning.

util.click

Command-line utilities.

These are used for building CLIs using click.

message_ix_models.util.click.PARAMS = {'dest': <Option dest>, 'dry_run': <Option dry_run>, 'force': <Option force>, 'output_model': <Option output_model>, 'platform_dest': <Option platform_dest>, 'policy_path': <Option policy_path>, 'quiet': <Option quiet>, 'regions': <Option regions>, 'rep_out_path': <Option rep_out_path>, 'rep_template': <Option rep_template>, 'run_reporting_only': <Option run_reporting_only>, 'ssp': <Argument ssp>, 'verbose': <Option verbose>}

Common command-line parameters (arguments and options). See common_params().

message_ix_models.util.click.common_params(param_names: str)

Decorate a click.command with common parameters param_names.

param_names must be a space-separated string of names appearing in PARAMS, e.g. "ssp force output_model". The decorated function receives keyword arguments with these names:

@click.command()
@common_params("ssp force output_model")
def mycmd(ssp, force, output_model)
    # ...

message_ix_models.util.click.default_path_cb(*default_parts)

Return a callback function for click.Option handling.

If no option value is given, the callback uses Context.get_local_path() and default_parts to provide a path that is relative to local data directory, e.g. the current working directory (see Data, metadata, and configuration).

message_ix_models.util.click.store_context(context, param, value)

Callback that simply stores a value on the Context object.

Use this for parameters that are not used directly in a @click.command() function, but need to be carried by the Context for later use.

util.context

Context and settings for message_ix_models code.

class message_ix_models.util.context.Context(*args, **kwargs)

Context and settings for message_ix_models code.

Context is a subclass of dict, so common methods like copy() and setdefault() may be used to handle settings. To be forgiving, it also provides attribute access; context.foo is equivalent to context["foo"].

Context provides additional methods to do common tasks that depend on configurable settings:

clone_to_dest([create])	Return a scenario based on the --dest command-line option.
close_db()
delete()	Hide the current Context from future get_instance() calls.
get_cache_path(*parts)	Return a path to a local cache file.
get_local_path(*parts[, suffix])	Return a path under local_data.
get_platform([reload])	Return a ixmp.Platform from platform_info.
get_scenario()	Return a message_ix.Scenario from scenario_info.
handle_cli_args([url, platform, model_name, …])	Handle command-line arguments.
only()	Return the only Context instance.
use_defaults(settings)	Update from settings.

The following Context methods and attribute are deprecated:

clone_to_dest(create=True) → message_ix.core.Scenario

Return a scenario based on the --dest command-line option.

Parameters

create (bool, optional) – If True (the default) and the base scenario does not exist, a bare RES scenario is created. Otherwise, an exception is raised.

Returns

To prevent the scenario from being garbage collected, keep a reference to its Platform:

Return type

Scenario

See also

create_res

To use this method, either decorate a command with common_params():

from message_data.tools.cli import common_params

@click.command()
@common_params("dest")
@click.pass_obj
def foo(context, dest):
    scenario, mp = context.clone_to_dest()

or, store the settings dest_scenario and dest_platform on context:

c = Context.get_instance()

c.dest_scenario = dict(model="foo model", scenario="foo scenario")
scenario_mp = context.clone_to_dest()

The resulting scenario has the indicated model- and scenario names.

If --url (or --platform, --model, --scenario, and optionally --version) are given, the identified scenario is used as a ‘base’ scenario, and is cloned. If --url/--platform and --dest refer to different Platform instances, then this is a two-platform clone.

If no base scenario can be loaded, bare.create_res() is called to generate a base scenario.

delete()

Hide the current Context from future get_instance() calls.

get_cache_path(*parts) → pathlib.Path

Return a path to a local cache file.

classmethod get_instance(index=0) → message_ix_models.util.context.Context

Return a Context instance; by default, the first created.

Parameters

index (int, optional) – Index of the Context instance to return, e.g. -1 for the most recently created.

get_local_path(*parts, suffix=None)

Return a path under local_data.

get_platform(reload=False) → ixmp.core.Platform

Return a ixmp.Platform from platform_info.

When used through the CLI, platform_info is a ‘base’ platform as indicated by the –url or –platform options.

If a Platform has previously been instantiated with get_platform(), the same object is returned unless reload=True.

get_scenario() → message_ix.core.Scenario

Return a message_ix.Scenario from scenario_info.

When used through the CLI, scenario_info is a ‘base’ scenario for an operation, indicated by the --url or --platform/--model/--scenario options.

handle_cli_args(url=None, platform=None, model_name=None, scenario_name=None, version=None, local_data=None, _store_as=('platform_info', 'scenario_info'))

Handle command-line arguments.

May update the data_path, platform_info, scenario_info, and/or url settings.

classmethod only() → message_ix_models.util.context.Context

Return the only Context instance.

Raises

IndexError – If there is more than one instance.

use_defaults(settings)

Update from settings.

util.importlib

Load model and project code from message_data.

class message_ix_models.util.importlib.MessageDataFinder

Load model and project code from message_data.

util._logging

Logging utilities.

class message_ix_models.util._logging.Formatter(colorama)

Formatter for log records.

Parameters

colorama (module) – If provided, colorama is used to colour log messages printed to stdout.

format(record)

Format record.

Records are formatted like:

model.transport.data.add_par_data  220 rows in 'input'
...add_par_data:  further messages

…with the calling function name (e.g. ‘add_par_data’) coloured for legibility on first occurrence, then dimmed when repeated.

message_ix_models.util._logging.make_formatter()

Return a Formatter instance for the message_ix_models logger.

See also

setup

message_ix_models.util._logging.setup(level='NOTSET', console=True)

Initialize logging.

Parameters

• level (str, optional) – Log level for message_ix_models and message_data.

• console (bool, optional) – If True, print all messages to console using a Formatter.

message_ix_models.util._logging.silence_log()

Context manager to temporarily silence log output.

Examples

>>> with silence_log():
>>>     log.warning("This message is not recorded.")

util.scenarioinfo

ScenarioInfo class.

class message_ix_models.util.scenarioinfo.ScenarioInfo(scenario=None)

Information about a Scenario object.

Code that prepares data for a target Scenario can accept a ScenarioInfo instance. This avoids the need to load a Scenario, which can be slow under some conditions.

ScenarioInfo objects can also be used (e.g. by apply_spec()) to describe the contents of a Scenario before it is created.

ScenarioInfo objects have the following convenience attributes:

set	Elements of ixmp/message_ix sets.
is_message_macro	True if a MESSAGE-MACRO scenario.
N	Elements of the set ‘node’.
Y	Elements of the set ‘year’ that are >= the first model year.
y0	First model year, if set, else Y[0].
yv_ya	pandas.DataFrame with valid year_vtg, year_act pairs.

Parameters

scenario (message_ix.Scenario) – If given, set is initialized from this existing scenario.

property N

Elements of the set ‘node’.

property Y

Elements of the set ‘year’ that are >= the first model year.

is_message_macro: bool = False

True if a MESSAGE-MACRO scenario.

par: Dict[str, pandas.core.frame.DataFrame] = {}

Elements of ixmp/message_ix parameters.

set: Dict[str, List] = {}

Elements of ixmp/message_ix sets.

y0: int = -1

First model year, if set, else Y[0].

year_from_codes(codes: List[sdmx.model.Code])

Update using a list of codes.

The following are updated:

• set year

• set cat_year, with the first model year.

• par duration_period

Any existing values are discarded.

After this, the attributes y0 and Y give the first model year and model years, respectively.

Examples

Get a particular code list, create a ScenarioInfo instance, and update using the codes:

>>> years = get_codes("year/A")
>>> info = ScenarioInfo()
>>> info.year_from_codes(years)

Use populated values:

>>> info.y0
2020
>>> info.Y[:3]
[2020, 2030, 2040]
>>> info.Y[-3:]
[2090, 2100, 2110]

property yv_ya

pandas.DataFrame with valid year_vtg, year_act pairs.