Low-level utilities (util
)
Submodules:
Command-line utilities. |
|
Context and settings for |
|
Load model and project code from |
|
Logging utilities. |
|
|
Commonly used:
|
Adapt data using mappings for 1 or more dimension(s). |
|
Adapt data using mappings for 1 or more dimension(s). |
|
Convert data to a |
|
Fill missing data in df by broadcasting. |
|
Decorator to cache the return value of a function func. |
|
Check whether a Context is compatible with certain settings. |
|
Convert units of s, for use with |
|
For use with |
|
Forward-fill df on dim to cover values. |
|
Return the ID of a node codelist given the contents of scenario. |
|
Load a |
|
Load a private data file from |
|
Construct a path for local data. |
|
Return input and output data frames for a 1-to-1 technology. |
|
Return data frames derived from base for multiple parameters. |
|
Return parameter data for a ‘source’ technology. |
|
Apply |
|
Merge dictionaries of DataFrames together into base. |
|
Construct a path to a file under |
|
Construct a path to a file under |
|
Fill 'node_origin'/'node_dest' in df from 'node_loc'. |
|
Suppress a spurious warning. |
|
Context and settings for |
|
Information about a |
|
A specification for the structure of a model or variant. |
- message_ix_models.util.as_codes(data: Union[List[str], Dict[str, Union[Dict, Code]]]) List[Code] [source]
Convert data to a
list
ofCode
objects.Various inputs are accepted:
- message_ix_models.util.broadcast(df: DataFrame, labels: Optional[DataFrame] = None, **kwargs) DataFrame [source]
Fill missing data in df by broadcasting.
broadcast()
is suitable for use with partly-filled data frames returned bymessage_ix.util.make_df()
, with 1 column per dimension, plus a “value” column. It is also usable withpandas.DataFrame.pipe()
for chained operations.labels (if any) are handled first: one copy or duplicate of df is produced for each row (set of labels) in this argument. Then, kwargs are handled;
broadcast()
returns one copy for each element in the cartesian product of the dimension labels given by kwargs.- Parameters
labels (pandas.DataFrame) – Each column (dimension) corresponds to one in df. Each row represents one matched set of labels for those dimensions.
kwargs – Keys are dimensions. Values are labels along that dimension to fill.
- Returns
The length is either 1 or an integer multiple of the length of df.
- Return type
- Raises
ValueError – if any of the columns in labels or kwargs are not present in df, or if those columns are present but not empty.
Examples
>>> from message_ix import make_df >>> from message_ix_models.util import broadcast # Create a base data frame with some empty columns >>> base = make_df("input", technology="t", value=[1.1, 2.2]) # Broadcast (duplicate) the data across 2 dimensions >>> df = base.pipe(broadcast, node_loc=["node A", "node B"], mode=["m0", "m1"]) # Show part of the result >>> df.dropna(axis=1) mode node_loc technology value 0 m0 node A t 1.1 1 m0 node A t 2.2 2 m0 node B t 1.1 3 m0 node B t 2.2 4 m1 node A t 1.1 5 m1 node A t 2.2 6 m1 node B t 1.1 7 m1 node B t 2.2
- message_ix_models.util.cached(func: Callable) Callable [source]
Decorator to cache the return value of a function func.
On a first call, the data requested is returned and also cached under
Context.get_cache_path()
. On subsequent calls, if the cache exists, it is used instead of calling the (possibly slow) func.When
SKIP_CACHE
is true, func is always called.See also
Caching in the
genno
documentation
- message_ix_models.util.check_support(context, settings={}, desc: str = '') None [source]
Check whether a Context is compatible with certain settings.
- Raises
NotImplementedError – if any context value for a key of settings is not among the values in settings.
KeyError – if the key is not set on context at all.
See also
- message_ix_models.util.convert_units(s: Series, unit_info: Mapping[str, Tuple[float, str, Optional[str]]], store='magnitude') Series [source]
Convert units of s, for use with
apply()
.s.name
is used to retrieve a tuple of (factor, input_unit, output_unit) from unit_info. The (float
) values of s are converted topint.Quantity
with the input_unit and factor; then cast to output_unit, if provided.- Parameters
s (pandas.Series) –
unit_info (dict (str -> tuple)) – Mapping from quantity name (matched to
s.name
) to 3-tuples of (factor, input_unit, output_unit). output_unit may beNone
. For example, seeikarus.UNITS
.store ("magnitude" or "quantity") – If “magnitude”, the values of the returned series are the magnitudes of the results, with no output units. If “quantity”, the values are scalar
Quantity
objects.
- Returns
Same shape, index, and values as s, with output units.
- Return type
- message_ix_models.util.eval_anno(obj: AnnotableArtefact, id: str)[source]
Retrieve the annotation id from obj, run
eval()
on its contents.This can be used for unpacking Python values (e.g.
dict
) stored as an annotation on aCode
.Returns
None
if no attribute exists with the given id.
- message_ix_models.util.identify_nodes(scenario: Scenario) str [source]
Return the ID of a node codelist given the contents of scenario.
- Returns
The ID of the Node code lists containing the regions of scenario.
- Return type
- Raises
ValueError – if no codelist can be identified, or the nodes in the scenario do not match the children of the “World” node in the codelist.
- message_ix_models.util.load_package_data(*parts: str, suffix: Optional[str] = '.yaml') Any [source]
Load a
message_ix_models
package data file and return its contents.Data is re-used if already loaded.
Example
The single call:
>>> info = load_package_data("node", "R11")
loads the metadata file
data/node/R11.yaml
, parsing its contents,stores those values at
PACKAGE_DATA["node R11"]
for use by other code, andreturns the loaded values.
- message_ix_models.util.load_private_data(*parts: str) Mapping [source]
Load a private data file from
message_data
and return its contents.Analogous to
load_package_data
, but for non-public data.- Parameters
parts (iterable of str) – Used to construct a path under
data/
in themessage_data
repository.- Returns
Configuration values that were loaded.
- Return type
- Raises
RuntimeError – if
message_data
is not installed.
- message_ix_models.util.local_data_path(*parts) Path [source]
Construct a path for local data.
The setting
message local data
in the user’s ixmp configuration file is used as a base path. If this is not configured, the current working directory is used.- Parameters
parts (sequence of str or Path) – Joined to the base path using
Path.joinpath()
.
See also
- message_ix_models.util.make_io(src, dest, efficiency, on='input', **kwargs)[source]
Return input and output data frames for a 1-to-1 technology.
- Parameters
dest (tuple (str, str, str)) – Output commodity, level, unit.
efficiency (float) – Conversion efficiency.
on ('input' or 'output') – If ‘input’, efficiency applies to the input, and the output, thus the activity level of the technology, is in dest[2] units. If ‘output’, the opposite.
kwargs – Passed to
make_df()
.
- Returns
Keys are ‘input’ and ‘output’; values are data frames.
- Return type
dict (str -> pd.DataFrame)
- message_ix_models.util.maybe_query(series: Series, query: Optional[str]) Series [source]
Apply
pandas.Series.query()
if the query arg is notNone
.query()
is not chainable (pandas-dev/pandas#37941). Use this function withpandas.Series.pipe()
, passing an argument that may beNone
, to have a chainable query operation that can be a no-op.
- message_ix_models.util.package_data_path(*parts) Path [source]
Construct a path to a file under
message_ix_models/data/
.Use this function to access data packaged and installed with
message_ix_models
.- Parameters
parts (sequence of str or Path) – Joined to the base path using
Path.joinpath()
.
See also
- message_ix_models.util.private_data_path(*parts) Path [source]
Construct a path to a file under
data/
inmessage_data
.Use this function to access non-public (e.g. embargoed or proprietary) data stored in the
message_data
repository.- Parameters
parts (sequence of str or Path) – Joined to the base path using
Path.joinpath()
.
See also
- message_ix_models.util.series_of_pint_quantity(*args, **kwargs) Series [source]
Suppress a spurious warning.
Creating a
pandas.Series
with a list ofpint.Quantity
triggers a warning “The unit of the quantity is stripped when downcasting to ndarray,” even though the entire object is being stored and the unit is not stripped. This function suppresses this warning.
- message_ix_models.util.cache.SKIP_CACHE = False
Controls whether cached data is returned for functions decorated with
cached()
. Set toTrue
to force reload.
util.click
Command-line utilities.
These are used for building CLIs using click
.
- message_ix_models.util.click.PARAMS = {'dest': <Option dest>, 'dry_run': <Option dry_run>, 'force': <Option force>, 'nodes': <Option nodes>, 'output_model': <Option output_model>, 'platform_dest': <Option platform_dest>, 'policy_path': <Option policy_path>, 'quiet': <Option quiet>, 'regions': <Option regions>, 'rep_out_path': <Option rep_out_path>, 'rep_template': <Option rep_template>, 'run_reporting_only': <Option run_reporting_only>, 'ssp': <Argument ssp>, 'verbose': <Option verbose>, 'years': <Option years>}
Common command-line parameters (arguments and options). See
common_params()
.
- message_ix_models.util.click.common_params(param_names: str)[source]
Decorate a click.command with common parameters param_names.
param_names must be a space-separated string of names appearing in
PARAMS
, e.g."ssp force output_model"
. The decorated function receives keyword arguments with these names:@click.command() @common_params("ssp force output_model") def mycmd(ssp, force, output_model) # ...
- message_ix_models.util.click.default_path_cb(*default_parts)[source]
Return a callback function for click.Option handling.
If no option value is given, the callback uses
Context.get_local_path()
and default_parts to provide a path that is relative to local data directory, e.g. the current working directory (see Data, metadata, and configuration).
util.context
Context and settings for message_ix_models
code.
- class message_ix_models.util.context.Context(*args, **kwargs)[source]
Context and settings for
message_ix_models
code.Context is a subclass of
dict
, so common methods likecopy()
andsetdefault()
may be used to handle settings. To be forgiving, it also provides attribute access;context.foo
is equivalent tocontext["foo"]
.Context provides additional methods to do common tasks that depend on configurable settings:
clone_to_dest
([create])Return a scenario based on the
--dest
command-line option.close_db
()delete
()Hide the current Context from future
get_instance()
calls.get_cache_path
(*parts)Return a path to a local cache file.
get_local_path
(*parts[, suffix])Return a path under
local_data
.get_platform
([reload])Return a
ixmp.Platform
fromplatform_info
.Return a
message_ix.Scenario
fromscenario_info
.handle_cli_args
([url, platform, model_name, ...])Handle command-line arguments.
only
()Return the only
Context
instance.use_defaults
(settings)Update from settings.
- clone_to_dest(create=True) Scenario [source]
Return a scenario based on the
--dest
command-line option.- Parameters
create (bool, optional) – If
True
(the default) and the base scenario does not exist, a bare RES scenario is created. Otherwise, an exception is raised.- Returns
To prevent the scenario from being garbage collected, keep a reference to its Platform:
- Return type
Scenario
See also
create_res
To use this method, either decorate a command with
common_params()
:from message_data.tools.cli import common_params @click.command() @common_params("dest") @click.pass_obj def foo(context, dest): scenario, mp = context.clone_to_dest()
or, store the settings
dest_scenario
anddest_platform
on context:c = Context.get_instance() c.dest_scenario = dict(model="foo model", scenario="foo scenario") scenario_mp = context.clone_to_dest()
The resulting scenario has the indicated model- and scenario names.
If
--url
(or--platform
,--model
,--scenario
, and optionally--version
) are given, the identified scenario is used as a ‘base’ scenario, and is cloned. If--url
/--platform
and--dest
refer to different Platform instances, then this is a two-platform clone.If no base scenario can be loaded,
bare.create_res()
is called to generate a base scenario.
- delete()[source]
Hide the current Context from future
get_instance()
calls.
- classmethod get_instance(index=0) Context [source]
Return a Context instance; by default, the first created.
- Parameters
index (int, optional) – Index of the Context instance to return, e.g.
-1
for the most recently created.
- get_platform(reload=False) Platform [source]
Return a
ixmp.Platform
fromplatform_info
.When used through the CLI,
platform_info
is a ‘base’ platform as indicated by the –url or –platform options.If a Platform has previously been instantiated with
get_platform()
, the same object is returned unless reload=True.
- get_scenario() Scenario [source]
Return a
message_ix.Scenario
fromscenario_info
.When used through the CLI,
scenario_info
is a ‘base’ scenario for an operation, indicated by the--url
or--platform/--model/--scenario
options.
- handle_cli_args(url=None, platform=None, model_name=None, scenario_name=None, version=None, local_data=None, verbose=False, _store_as=('platform_info', 'scenario_info'))[source]
Handle command-line arguments.
May update the
data_path
,platform_info
,scenario_info
, and/orurl
settings.
- classmethod only() Context [source]
Return the only
Context
instance.- Raises
IndexError – If there is more than one instance.
util.importlib
Load model and project code from message_data
.
util._logging
Logging utilities.
- class message_ix_models.util._logging.Formatter(colorama)[source]
Formatter for log records.
- Parameters
colorama (module) – If provided,
colorama
is used to colour log messages printed to stdout.
- message_ix_models.util._logging.make_formatter()[source]
Return a
Formatter
instance for themessage_ix_models
logger.See also
util.node
Utilities for nodes.
- message_ix_models.util.node.NODE_DIMS = ['n', 'node', 'node_loc', 'node_origin', 'node_dest', 'node_rel', 'node_share']
Names of dimensions indexed by ‘node’.
Todo
to be robust to changes in
message_ix
, read these names from that package.
- message_ix_models.util.node.R11_R12 = (('R11_AFR', 'R12_AFR'), ('R11_CPA', 'R12_CHN'), ('R11_EEU', 'R12_EEU'), ('R11_FSU', 'R12_FSU'), ('R11_LAM', 'R12_LAM'), ('R11_MEA', 'R12_MEA'), ('R11_NAM', 'R12_NAM'), ('R11_PAO', 'R12_PAO'), ('R11_PAS', 'R12_PAS'), ('R11_CPA', 'R12_RCPA'), ('R11_SAS', 'R12_SAS'), ('R11_WEU', 'R12_WEU'))
Mapping from R11 to R12 node IDs.
- message_ix_models.util.node.R11_R14 = (('R11_AFR', 'R14_AFR'), ('R11_FSU', 'R14_CAS'), ('R11_CPA', 'R14_CPA'), ('R11_EEU', 'R14_EEU'), ('R11_LAM', 'R14_LAM'), ('R11_MEA', 'R14_MEA'), ('R11_NAM', 'R14_NAM'), ('R11_PAO', 'R14_PAO'), ('R11_PAS', 'R14_PAS'), ('R11_FSU', 'R14_RUS'), ('R11_SAS', 'R14_SAS'), ('R11_FSU', 'R14_SCS'), ('R11_FSU', 'R14_UBM'), ('R11_WEU', 'R14_WEU'))
Mapping from R11 to R14 node IDs.
- message_ix_models.util.node.adapt_R11_R12 = <message_ix_models.util.common.MappingAdapter object>
Adapt data from the R11 to the R14 node list.
The data is adapted using the mappings in
R11_R12
for each of the dimensions inNODE_DIMS
.
- message_ix_models.util.node.adapt_R11_R14 = <message_ix_models.util.common.MappingAdapter object>
Adapt data from the R11 to the R14 node list.
The data is adapted using the mappings in
R11_R14
for each of the dimensions inNODE_DIMS
.
- message_ix_models.util.node.identify_nodes(scenario: Scenario) str [source]
Return the ID of a node codelist given the contents of scenario.
- Returns
The ID of the Node code lists containing the regions of scenario.
- Return type
- Raises
ValueError – if no codelist can be identified, or the nodes in the scenario do not match the children of the “World” node in the codelist.
util.scenarioinfo
ScenarioInfo
class.
- class message_ix_models.util.scenarioinfo.ScenarioInfo(scenario=None)[source]
Information about a
Scenario
object.Code that prepares data for a target Scenario can accept a ScenarioInfo instance. This avoids the need to load a Scenario, which can be slow under some conditions.
ScenarioInfo objects can also be used (e.g. by
apply_spec()
) to describe the contents of a Scenario before it is created.ScenarioInfo objects have the following convenience attributes:
Elements of
ixmp
/message_ix
sets.io_units
(technology, commodity[, level])Return units for the MESSAGE
input
oroutput
parameter.True
if a MESSAGE-MACRO scenario.Elements of the set 'node'.
units_for
(set_name, id)Return the units associated with code id in MESSAGE set set_name.
Elements of the set 'year' that are >= the first model year.
First model year, if set, else
Y[0]
.pandas.DataFrame
with validyear_vtg
,year_act
pairs.- Parameters
scenario (message_ix.Scenario) – If given,
set
is initialized from this existing scenario.
See also
- property N
Elements of the set ‘node’.
- property Y
Elements of the set ‘year’ that are >= the first model year.
- io_units(technology: str, commodity: str, level: Optional[str] = None) Unit [source]
Return units for the MESSAGE
input
oroutput
parameter.These are implicitly determined as the ratio of:
The units for the origin (for
input
) or destination commodity, perunits_for()
.The units of activity for the technology.
- Parameters
level (str) – Placeholder for future functionality, i.e. to use different units per (commodity, level). Currently ignored. If given, a debug message is logged.
- units_for(set_name: str, id: str) Unit [source]
Return the units associated with code id in MESSAGE set set_name.
ixmp
(or the soleJDBCBackend
, as of v3.5.0) does not handle unit information for variables and equations (unlike parameter values), such as MESSAGE decision variablesACT
,CAP
, etc. Inmessage_ix_models
andmessage_data
, the following conventions are (generally) followed:The units of
ACT
and others are consistent for eachtechnology
.The units of
COMMODITY_BALANCE
,STOCK
,commodity_stock
, etc. are consistent for eachcommodity
.
Thus, codes for elements of these sets (e.g. Commodities (commodity.yaml)) can be used to carry the standard units for the corresponding quantities.
units_for()
retrieves these units, for use in model-building and reporting.Todo
Expand this discussion and transfer to the
message_ix
docs.See also
- update(other: ScenarioInfo)[source]
Update with the set elements of other.
- year_from_codes(codes: List[Code])[source]
Update using a list of codes.
The following are updated:
Any existing values are discarded.
After this, the attributes
y0
andY
give the first model year and model years, respectively.Examples
Get a particular code list, create a ScenarioInfo instance, and update using the codes:
>>> years = get_codes("year/A") >>> info = ScenarioInfo() >>> info.year_from_codes(years)
Use populated values:
>>> info.y0 2020 >>> info.Y[:3] [2020, 2030, 2040] >>> info.Y[-3:] [2090, 2100, 2110]
- property yv_ya
pandas.DataFrame
with validyear_vtg
,year_act
pairs.
- class message_ix_models.util.scenarioinfo.Spec(add: ~message_ix_models.util.scenarioinfo.ScenarioInfo = <factory>, remove: ~message_ix_models.util.scenarioinfo.ScenarioInfo = <factory>, require: ~message_ix_models.util.scenarioinfo.ScenarioInfo = <factory>)[source]
A specification for the structure of a model or variant.
A Spec collects 3
ScenarioInfo
instances at the attributesadd
,remove
, andrequire
. This is the type that is accepted byapply_spec()
; Building models (model.build) describes how a Spec is used to modify aScenario
. A Spec may also be used to express information about the target structure of data to be prepared; likeScenarioInfo
, this can happen before the targetScenario
exists.Spec also provides:
Dictionary-style access, e.g.
s["add"]
is equivalent tos.add.
.Error checking; setting keys other than add/remove/require results in an error.
merge()
, a helper method.
- add: ScenarioInfo
Structure to be added to a base scenario.
- static merge(a: Spec, b: Spec) Spec [source]
Merge Specs a and b together.
Returns a new Spec where each member is a union of the respective members of a and b.
- remove: ScenarioInfo
Structure to be removed from a base scenario.
- require: ScenarioInfo
Structure that must be present in a base scenario.