Low-level utilities (util)

Submodules:

click	Command-line utilities.
context	Context and settings for message_ix_models code.
importlib	Load model and project code from message_data.
_logging	Logging utilities.
scenarioinfo	ScenarioInfo class.

Commonly used:

adapt_R11_R14(-> pandas.core.frame.DataFrame)	Adapt data from R11 to R14 node list.
as_codes(data)	Convert data to a list of Code objects.
broadcast(df, **kwargs)	Fill missing data in df by broadcasting.
copy_column(column_name)	For use with pandas.DataFrame.assign().
ffill(df, dim, values[, expr])	Forward-fill df on dim to cover values.
identify_nodes(scenario)	Return the ID of a node codelist given the contents of scenario.
load_package_data(*parts[, suffix])	Load a message_ix_models package data file and return its contents.
load_private_data(*parts)	Load a private data file from message_data and return its contents.
make_io(src, dest, efficiency[, on])	Return input and output data frames for a 1-to-1 technology.
make_matched_dfs(base, **par_value)	Return data frames derived from base for multiple parameters.
make_source_tech(info, common, **values)	Return parameter data for a ‘source’ technology.
merge_data(base, *others)	Merge dictionaries of DataFrames together into base.
package_data_path(*parts)	Construct a path to a file under message_ix_models/data/.
private_data_path(*parts)	Construct a path to a file under data/ in message_data.
same_node(df)	Fill ‘node_origin’/’node_dest’ in df from ‘node_loc’.
Context(*args, **kwargs)	Context and settings for message_ix_models code.
ScenarioInfo([scenario])	Information about a Scenario object.

message_ix_models.util.adapt_R11_R14(data: Dict[str, Union[pandas.core.frame.DataFrame, genno.core.quantity.Quantity]])

message_ix_models.util.adapt_R11_R14(df: pandas.core.frame.DataFrame) → pandas.core.frame.DataFrame

message_ix_models.util.adapt_R11_R14(qty: genno.core.quantity.Quantity) → genno.core.quantity.Quantity

Adapt data from R11 to R14 node list.

The data is adapted by:

• Renaming regions such as R11_NAM to R14_NAM.

• Copying the data for R11_FSU to R14_CAS, R14_RUS, R14_SCS, and R14_UBM.

…wherever these appear in a column/dimension named ‘node’, ‘node_*’, or ‘n’.

The function may be called with:

• pandas.DataFrame,

• genno.Quantity, or

• dict mapping str parameter names to values (either of the above types).

message_ix_models.util.identify_nodes(scenario: message_ix.core.Scenario) → str

Return the ID of a node codelist given the contents of scenario.

Returns

The ID of the Node code lists containing the regions of scenario.

Return type

str

Raises

ValueError – if no codelist can be identified, or the nodes in the scenario do not match the children of the “World” node in the codelist.

util.click

Command-line utilities.

These are used for building CLIs using click.

message_ix_models.util.click.PARAMS = {'dest': <Option dest>, 'dry_run': <Option dry_run>, 'force': <Option force>, 'output_model': <Option output_model>, 'platform_dest': <Option platform_dest>, 'policy_path': <Option policy_path>, 'quiet': <Option quiet>, 'regions': <Option regions>, 'rep_out_path': <Option rep_out_path>, 'rep_template': <Option rep_template>, 'run_reporting_only': <Option run_reporting_only>, 'ssp': <Argument ssp>, 'verbose': <Option verbose>}

Common command-line parameters (arguments and options). See common_params().

message_ix_models.util.click.common_params(param_names: str)

Decorate a click.command with common parameters param_names.

param_names must be a space-separated string of names appearing in PARAMS, e.g. "ssp force output_model". The decorated function receives keyword arguments with these names:

@click.command()
@common_params("ssp force output_model")
def mycmd(ssp, force, output_model)
    # ...

message_ix_models.util.click.default_path_cb(*default_parts)

Return a callback function for click.Option handling.

If no option value is given, the callback uses Context.get_local_path() and default_parts to provide a path that is relative to local data directory, e.g. the current working directory (see Data, metadata, and configuration).

message_ix_models.util.click.store_context(context, param, value)

Callback that simply stores a value on the Context object.

Use this for parameters that are not used directly in a @click.command() function, but need to be carried by the Context for later use.

util.context

Context and settings for message_ix_models code.

class message_ix_models.util.context.Context(*args, **kwargs)

Context and settings for message_ix_models code.

Context is a subclass of dict, so common methods like copy() and setdefault() may be used to handle settings. To be forgiving, it also provides attribute access; context.foo is equivalent to context["foo"].

Context provides additional methods to do common tasks that depend on configurable settings:

clone_to_dest()	Return a scenario based on the --dest command-line option.
close_db()
delete()	Hide the current Context from future get_instance() calls.
get_cache_path(*parts)	Return a path to a local cache file.
get_local_path(*parts[, suffix])	Return a path under local_data.
get_platform([reload])	Return a ixmp.Platform from platform_info.
get_scenario()	Return a message_ix.Scenario from scenario_info.
handle_cli_args([url, platform, model_name, …])	Handle command-line arguments.
only()	Return the only Context instance.
use_defaults(settings)	Update from settings.

The following Context methods and attribute are deprecated:

get_config_file(*parts[, ext])	Return a path under metadata_path.
get_path(*parts)	Return a path under message_data_path by joining parts.
load_config(*parts[, suffix])	Load configuration from message_ix_models.
units	Access the unit registry.

clone_to_dest() → Tuple[message_ix.core.Scenario, ixmp.core.Platform]

Return a scenario based on the --dest command-line option.

Returns

To prevent the scenario from being garbage collected. Keep a reference to its Platform:

Return type

Scenario

See also

create_res

To use this method, either decorate a command with common_params():

from message_data.tools.cli import common_params

@click.command()
@common_params("dest")
@click.pass_obj
def foo(context, dest):
    scenario, mp = context.clone_to_dest()

or, store the settings dest_scenario and dest_platform on context:

c = Context.get_instance()

c.dest_scenario = dict(model="foo model", scenario="foo scenario")
scenario_mp = context.clone_to_dest()

The resulting scenario has the indicated model- and scenario names.

If --url (or --platform, --model, --scenario, and optionally --version) are given, the identified scenario is used as a ‘base’ scenario, and is cloned. If --url/--platform and --dest refer to different Platform instances, then this is a two-platform clone.

If no base scenario can be loaded, bare.create_res() is called to generate a base scenario.

delete()

Hide the current Context from future get_instance() calls.

get_cache_path(*parts) → pathlib.Path

Return a path to a local cache file.

get_config_file(*parts, ext='yaml') → pathlib.Path

Return a path under metadata_path.

The suffix “.{ext}” is added; defaulting to “.yaml”.

Deprecated since version 2021.2.28: Use package_data_path() instead. Will be removed on or after 2021-05-28.

classmethod get_instance(index=0) → message_ix_models.util.context.Context

Return a Context instance; by default, the first created.

Parameters

index (int, optional) – Index of the Context instance to return, e.g. -1 for the most recently created.

get_local_path(*parts, suffix=None)

Return a path under local_data.

get_path(*parts) → pathlib.Path

Return a path under message_data_path by joining parts.

parts may include directory names, or a filename with extension.

Deprecated since version 2021.2.28: Use private_data_path() instead. Will be removed on or after 2021-05-28.

get_platform(reload=False) → ixmp.core.Platform

Return a ixmp.Platform from platform_info.

When used through the CLI, platform_info is a ‘base’ platform as indicated by the –url or –platform options.

If a Platform has previously been instantiated with get_platform(), the same object is returned unless reload=True.

get_scenario() → message_ix.core.Scenario

Return a message_ix.Scenario from scenario_info.

When used through the CLI, scenario_info is a ‘base’ scenario for an operation, indicated by the --url or --platform/--model/--scenario options.

handle_cli_args(url=None, platform=None, model_name=None, scenario_name=None, version=None, local_data=None, _store_as=('platform_info', 'scenario_info'))

Handle command-line arguments.

May update the data_path, platform_info, scenario_info, and/or url settings.

load_config(*parts, suffix=None)

Load configuration from message_ix_models.

Deprecated since version 2021.2.28: Use load_package_data() instead. Will be removed on or after 2021-05-28.

classmethod only() → message_ix_models.util.context.Context

Return the only Context instance.

Raises

IndexError – If there is more than one instance.

property units

Access the unit registry.

Deprecated since version 2021.2.28: Instead, use:

from iam_units import registry

Will be removed on or after 2021-05-28.

use_defaults(settings)

Update from settings.

util.importlib

Load model and project code from message_data.

class message_ix_models.util.importlib.MessageDataFinder

Load model and project code from message_data.

util._logging

Logging utilities.

class message_ix_models.util._logging.Formatter(colorama)

Formatter for log records.

Parameters

colorama (module) – If provided, colorama is used to colour log messages printed to stdout.

format(record)

Format record.

Records are formatted like:

model.transport.data.add_par_data  220 rows in 'input'
...add_par_data:  further messages

…with the calling function name (e.g. ‘add_par_data’) coloured for legibility on first occurrence, then dimmed when repeated.

message_ix_models.util._logging.make_formatter()

Return a Formatter instance for the message_ix_models logger.

See also

setup

message_ix_models.util._logging.setup(level='NOTSET', console=True)

Initialize logging.

Parameters

• level (str, optional) – Log level for message_ix_models and message_data.

• console (bool, optional) – If True, print all messages to console using a Formatter.

message_ix_models.util._logging.silence_log()

Context manager to temporarily silence log output.

Examples

>>> with silence_log():
>>>     log.warning("This message is not recorded.")

util.scenarioinfo

ScenarioInfo class.

class message_ix_models.util.scenarioinfo.ScenarioInfo(scenario=None)

Information about a Scenario object.

Code that prepares data for a target Scenario can accept a ScenarioInfo instance. This avoids the need to load a Scenario, which can be slow under some conditions.

ScenarioInfo objects can also be used (e.g. by apply_spec()) to describe the contents of a Scenario before it is created.

ScenarioInfo objects have the following convenience attributes:

set	Elements of ixmp/message_ix sets.
is_message_macro	True if a MESSAGE-MACRO scenario.
N	Elements of the set ‘node’.
Y	Elements of the set ‘year’ that are >= the first model year.
y0	First model year, if set, else Y[0].
yv_ya	pandas.DataFrame with valid year_vtg, year_act pairs.

Parameters

scenario (message_ix.Scenario) – If given, set is initialized from this existing scenario.

property N

Elements of the set ‘node’.

property Y

Elements of the set ‘year’ that are >= the first model year.

is_message_macro: bool = False

True if a MESSAGE-MACRO scenario.

par: Dict[str, pandas.core.frame.DataFrame] = {}

Elements of ixmp/message_ix parameters.

set: Dict[str, List] = {}

Elements of ixmp/message_ix sets.

y0: int = -1

First model year, if set, else Y[0].

year_from_codes(codes: List[sdmx.model.Code])

Update using a list of codes.

The following are updated:

• set year

• set cat_year, with the first model year.

• par duration_period

Any existing values are discarded.

After this, the attributes y0 and Y give the first model year and model years, respectively.

Examples

Get a particular code list, create a ScenarioInfo instance, and update using the codes:

>>> years = get_codes("year/A")
>>> info = ScenarioInfo()
>>> info.year_from_codes(years)

Use populated values:

>>> info.y0
2020
>>> info.Y[:3]
[2020, 2030, 2040]
>>> info.Y[-3:]
[2090, 2100, 2110]

property yv_ya

pandas.DataFrame with valid year_vtg, year_act pairs.