Python & R API¶
The application programming interface (API) for MESSAGEix model developers is implemented in Python:
Support for R usage of the core classes is provided through the reticulate package. For instance:
> library(reticulate)
> ixmp <- import('ixmp')
> message_ix <- import('message_ix')
> mp <- ixmp$Platform(...)
> scen <- message_ix$Scenario(mp, ...)
ixmp
package¶
ixmp
provides three classes. These are fully described by the ixmp documentation, which is cross-linked from many places in the MESSAGEix documentation.
|
Instance of the modeling platform. |
|
Collection of data in time series format. |
|
Collection of model-related data. |
ixmp
also provides some utility classes and methods:
|
Configuration for ixmp. |
|
Mapping from names to available models. |
|
Return a model for name (or the default) with model_options. |
|
message_ix
package¶
MESSAGEix models are created using the message_ix.Scenario
class. Several utility methods are also provided in the module message_ix.utils
.
-
class
message_ix.
Scenario
(mp, model, scenario=None, version=None, annotation=None, scheme=None, **kwargs)¶ Bases:
ixmp.core.Scenario
MESSAGEix Scenario.
See
ixmp.TimeSeries
for the meaning of arguments mp, model, scenario, version, and annotation. The scheme of a newly-created Scenario is always ‘MESSAGE’.This class extends
ixmp.Scenario
andixmp.TimeSeries
and inherits all their methods. Documentation of these inherited methods is included here for convenience.message_ix.Scenario
defines additional methods specific to MESSAGEix:Changed in version 3.0:
read_excel()
andto_excel()
are now methods ofixmp.Scenario
, but continue to work with message_ix.Scenario.add_cat
(name, cat, keys[, is_unique])Map elements from keys to category cat within set name.
add_horizon
(data)Add sets related to temporal dimensions of the model.
add_macro
(data[, scenario, check_convergence])Add MACRO parametrization to the Scenario and calibrate.
add_spatial_sets
(data)Add sets related to spatial dimensions of the model.
cat
(name, cat)Return a list of all set elements mapped to a category.
cat_list
(name)Return a list of all categories for a mapping set.
equ
(name[, filters])Return equation data.
The first model year of the scenario.
par
(name[, filters])Return parameter data.
read_excel
(path[, add_units, init_items, …])Read a Microsoft Excel file into the Scenario.
rename
(name, mapping[, keep])Rename an element in a set
to_excel
(path[, items, max_row])Write Scenario to a Microsoft Excel file.
var
(name[, filters])Return variable data.
vintage_and_active_years
([ya_args, in_horizon])Return sets of vintage and active years for use in data input.
years_active
(node, tec, yr_vtg)Return years in which tec of yr_vtg can be active in node.
-
add_macro
(data, scenario=None, check_convergence=True, **kwargs)¶ Add MACRO parametrization to the Scenario and calibrate.
Warning
MACRO support via
add_macro()
is experimental in message_ix 3.0 and may not function as expected on all possible MESSAGEix models. See a list of known and pending issues on GitHub.
-
add_cat
(name, cat, keys, is_unique=False)¶ Map elements from keys to category cat within set name.
- Parameters
-
add_geodata
(df)¶ Add geodata (layers) to the TimeSeries.
- Parameters
df (
pandas.DataFrame
) –Data to add. df must have the following columns:
region
variable
subannual
unit
year
value
meta
-
add_horizon
(data)¶ Add sets related to temporal dimensions of the model.
- Parameters
data (dict-like) – Year sets. “year” is a required key. “firstmodelyear” is optional; if not provided, the first element of “year” is used.
Examples
>>> s = message_ix.Scenario() >>> s.add_horizon({'year': [2010, 2020]}) >>> s.add_horizon({'year': [2010, 2020], 'firstmodelyear': 2020})
-
add_par
(name, key_or_data=None, value=None, unit=None, comment=None)¶ Set the values of a parameter.
- Parameters
name (str) – Name of the parameter.
key_or_data (str or iterable of str or range or dict or) –
pandas.DataFrame
Element(s) to be added.value (numeric or iterable of numeric, optional) – Values.
unit (str or iterable of str, optional) – Unit symbols.
comment (str or iterable of str, optional) – Comment(s) for the added values.
-
add_set
(name, key, comment=None)¶ Add elements to an existing set.
- Parameters
name (str) – Name of the set.
key (str or iterable of str or dict or
pandas.DataFrame
) – Element(s) to be added. If name exists, the elements are appended to existing elements.comment (str or iterable of str, optional) – Comment describing the element(s). If given, there must be the same number of comments as elements.
- Raises
KeyError – If the set name does not exist.
init_set()
must be called beforeadd_set()
.ValueError – For invalid forms or combinations of key and comment.
-
add_spatial_sets
(data)¶ Add sets related to spatial dimensions of the model.
- Parameters
data (dict) –
Mapping of level → member. Each member may be:
A single label for elements.
An iterable of labels for elements.
A recursive
dict
following the same convention, defining sub-levels and their members.
Examples
>>> s = message_ix.Scenario() >>> s.add_spatial_sets({'country': 'Austria'}) >>> s.add_spatial_sets({'country': ['Austria', 'Germany']}) >>> s.add_spatial_sets({'country': { ... 'Austria': {'state': ['Vienna', 'Lower Austria']}}})
-
add_timeseries
(df, meta=False, year_lim=None, None)¶ Add data to the TimeSeries.
- Parameters
df (
pandas.DataFrame
) –Data to add. df must have the following columns:
region or node
variable
unit
Additional column names may be either of:
year and value—long, or ‘tabular’, format.
one or more specific years—wide, or ‘IAMC’ format.
To support subannual temporal resolution of timeseries data, a column subannual is optional in df. The entries in this column must have been defined in the Platform instance using
add_timeslice()
beforehand. If no column subannual is included in df, the data is assumed to contain yearly values. Seetimeslices()
for a detailed description of the feature.meta (bool, optional) – If
True
, store df as metadata. Metadata is treated specially whenScenario.clone()
is called for Scenarios created withscheme='MESSAGE'
.year_lim (tuple of (int or None, int or None`), optional) – Respectively, minimum and maximum years to add from df; data for other years is ignored.
-
cat
(name, cat)¶ Return a list of all set elements mapped to a category.
-
cat_list
(name)¶ Return a list of all categories for a mapping set.
- Parameters
name (str) – Name of the set.
-
change_scalar
(name, val, unit, comment=None)¶ Set the value and unit of a scalar.
-
check_out
(timeseries_only=False)¶ Check out the Scenario.
- Raises
ValueError – If
has_solution()
isTrue
.
See also
TimeSeries.check_out()
,utils.maybe_check_out()
-
clone
(*args, **kwargs)¶ Clone the current scenario and return the clone.
See
ixmp.Scenario.clone()
for other parameters.- Parameters
keep_solution (bool, optional) – If
True
, include all timeseries data and the solution (vars and equs) from the source Scenario in the clone. Otherwise, only timeseries data marked as meta=True (seeTimeSeries.add_timeseries()
) or prior to first_model_year (seeTimeSeries.add_timeseries()
) are cloned.shift_first_model_year (int, optional) – If given, the values of the solution are transfered to parameters historical_*, parameter resource_volume is updated, and the first_model_year is shifted. The solution is then discarded, see
TimeSeries.remove_solution()
.
-
commit
(comment)¶ Commit all changed data to the database.
If the TimeSeries was newly created (with
version='new'
),version
is updated with a new version number assigned by the backend. Otherwise,commit()
does not change theversion
.- Parameters
comment (str) – Description of the changes being committed.
See also
utils.maybe_commit()
-
delete_meta
(name)¶ Delete scenario meta.
- Parameters
name (str or list of str) – Either single meta key or list of keys.
-
discard_changes
()¶ Discard all changes and reload from the database.
-
equ
(name, filters=None)¶ Return equation data.
Same as
ixmp.Scenario.equ()
, except columns indexed by the MESSAGEix setyear
are returned withint
dtype.
-
equ_list
()¶ List all defined equations.
-
classmethod
from_url
(url, errors='warn')¶ Instantiate a Scenario given an ixmp-scheme URL.
The following are equivalent:
from ixmp import Platform, Scenario mp = Platform(name='example') scen = Scenario(mp 'model', 'scenario', version=42)
and:
from ixmp import Scenario scen, mp = Scenario.from_url('ixmp://example/model/scenario#42')
- Parameters
url (str) – See
parse_url
.errors ('warn' or 'raise') – If ‘warn’, a failure to load the Scenario is logged as a warning, and the platform is still returned. If ‘raise’, the exception is raised.
- Returns
scenario, platform – The Scenario and Platform referred to by the URL.
- Return type
2-tuple of (Scenario,
Platform
)
-
get_geodata
()¶ Fetch geodata and return it as dataframe.
- Returns
Specified data.
- Return type
-
has_equ
(name)¶ Check whether the scenario has an equation with that name.
-
has_par
(name)¶ Check whether the scenario has a parameter with that name.
-
has_set
(name)¶ Check whether the scenario has a set name.
-
has_solution
()¶ Return
True
if the Scenario has been solved.If
has_solution() == True
, model solution data exists in the db.
-
has_var
(name)¶ Check whether the scenario has a variable with that name.
-
idx_names
(name)¶ Return the list of index names for an item (set, par, var, equ).
- Parameters
name (str) – name of the item
-
idx_sets
(name)¶ Return the list of index sets for an item (set, par, var, equ).
- Parameters
name (str) – name of the item
-
init_equ
(name, idx_sets=None, idx_names=None)¶ Initialize a new equation.
-
init_par
(name, idx_sets, idx_names=None)¶ Initialize a new parameter.
-
init_scalar
(name, val, unit, comment=None)¶ Initialize a new scalar.
-
init_set
(name, idx_sets=None, idx_names=None)¶ Initialize a new set.
- Parameters
- Raises
ValueError – If the set (or another object with the same name) already exists.
RuntimeError – If the Scenario is not checked out (see
check_out()
).
-
init_var
(name, idx_sets=None, idx_names=None)¶ Initialize a new variable.
-
last_update
()¶ Get the timestamp of the last update/edit of this TimeSeries.
-
load_scenario_data
()¶ Load all Scenario data into memory.
- Raises
ValueError – If the Scenario was instantiated with
cache=False
.
-
par
(name, filters=None)¶ Return parameter data.
Same as
ixmp.Scenario.par()
, except columns indexed by the MESSAGEix setyear
are returned withint
dtype.
-
par_list
()¶ List all defined parameters.
-
preload_timeseries
()¶ Preload timeseries data to in-memory cache. Useful for bulk updates.
-
read_excel
(path, add_units=False, init_items=False, commit_steps=False)¶ Read a Microsoft Excel file into the Scenario.
- Parameters
path (os.PathLike) – File to read. Must have suffix ‘.xlsx’.
add_units (bool, optional) – Add missing units, if any, to the Platform instance.
init_items (bool, optional) – Initialize sets and parameters that do not already exist in the Scenario.
commit_steps (bool, optional) – Commit changes after every data addition.
See also
excel-data-format,
TimeSeries.read_file()
,to_excel()
-
read_file
(path, firstyear=None, lastyear=None)¶ Read time series data from a CSV or Microsoft Excel file.
- Parameters
path (os.PathLike) – File to read. Must have suffix ‘.csv’ or ‘.xlsx’.
firstyear (int, optional) – Only read data from years equal to or later than this year.
lastyear (int, optional) – Only read data from years equal to or earlier than this year.
See also
-
remove_geodata
(df)¶ Remove geodata from the TimeSeries instance.
- Parameters
df (
pandas.DataFrame
) –Data to remove. df must have the following columns:
region
variable
unit
subannual
year
-
remove_par
(name, key=None)¶ Remove parameter values or an entire parameter.
- Parameters
name (str) – Name of the parameter.
key (dataframe or key list or concatenated string, optional) – elements to be removed
-
remove_set
(name, key=None)¶ Delete set elements or an entire set.
- Parameters
name (str) – Name of the set to remove (if key is
None
) or from which to remove elements.key (
pandas.DataFrame
or list of str, optional) – Elements to be removed from set name.
-
remove_solution
(first_model_year=None)¶ Remove the solution from the scenario
This function removes the solution (variables and equations) and timeseries data marked as meta=False from the scenario (see
TimeSeries.add_timeseries()
).- Parameters
first_model_year (int, optional) – If given, timeseries data marked as meta=False is removed only for years from first_model_year onwards.
- Raises
ValueError – If Scenario has no solution or if first_model_year is not int.
-
remove_timeseries
(df)¶ Remove timeseries data from the TimeSeries instance.
- Parameters
df (
pandas.DataFrame
) –Data to remove. df must have the following columns:
region or node
variable
unit
year
-
rename
(name, mapping, keep=False)¶ Rename an element in a set
-
run_id
()¶ Get the run id of this TimeSeries.
-
scalar
(name)¶ Return the value and unit of a scalar.
- Parameters
name (str) – Name of the scalar.
- Returns
{‘value’
- Return type
value, ‘unit’: unit}
-
set
(name, filters=None)¶ Return elements of a set.
Same as
ixmp.Scenario.set()
, except columns for multi-dimensional sets indexed by the MESSAGEix setyear
are returned withint
dtype.- Parameters
name (str) – Name of the set.
filters (dict (str -> list of str), optional) – Mapping of dimension_name → elements, where dimension_name is one of the idx_names given when the set was initialized (see
init_set()
), and elements is an iterable of labels to include in the return value.
- Returns
pd.Series – If name is an index set.
pd.DataFrame – If name is a set defined over one or more other, index sets.
-
set_as_default
()¶ Set the current
version
as the default.
-
set_list
()¶ List all defined sets.
-
set_meta
(name_or_dict, value=None)¶ Set scenario meta.
-
solve
(model='MESSAGE', solve_options={}, **kwargs)¶ Solve MESSAGE or MESSAGE-MACRO for the Scenario.
By default,
ixmp.Scenario.solve()
is called with ‘MESSAGE’ as the model argument. model may also be overwritten, e.g.:>>> s.solve(model='MESSAGE-MACRO')
- Parameters
model ('MESSAGE' or 'MACRO' or 'MESSAGE-MACRO', optional) – Model to solve.
solve_options (dict (str -> str), optional) – Name to value mapping to use for GAMS CPLEX solver options file. See the
MESSAGE
class andDEFAULT_CPLEX_OPTIONS
.kwargs – Many other options control the execution of the underlying GAMS code; see the
MESSAGE_MACRO
class andGAMSModel
.
-
timeseries
(region=None, variable=None, unit=None, year=None, iamc=False, subannual='auto')¶ Retrieve timeseries data.
- Parameters
iamc (bool, optional) – Return data in wide/’IAMC’ format. If
False
, return data in long/’tabular’ format; seeadd_timeseries()
.region (str or list of str, optional) – Regions to include in returned data.
variable (str or list of str, optional) – Variables to include in returned data.
unit (str or list of str, optional) – Units to include in returned data.
year (str or int or list of (str or int), optional) – Years to include in returned data.
subannual (bool or 'auto', optional) – Whether to include column for sub-annual specification (if
bool
); if ‘auto’, include column if sub-annual data (other than ‘Year’) exists in returned dataframe.
- Raises
ValueError – If subannual is
False
but Scenario has (filtered) sub-annual data.- Returns
Specified data.
- Return type
-
to_excel
(path, items=<ItemType.PAR|SET: 6>, max_row=None)¶ Write Scenario to a Microsoft Excel file.
- Parameters
path (os.PathLike) – File to write. Must have suffix
.xlsx
.items (ItemType, optional) – Types of items to write. Either
SET
|PAR
(i.e. only sets and parameters), orMODEL
(also variables and equations, i.e. model solution data).max_row (int, optional) – Maximum number of rows in each sheet. If the number of elements in an item exceeds this number or
EXCEL_MAX_ROWS
, then an item is written to multiple sheets named, e.g. ‘foo’, ‘foo(2)’, ‘foo(3)’, etc.
See also
excel-data-format,
read_excel()
-
var
(name, filters=None)¶ Return variable data.
Same as
ixmp.Scenario.var()
, except columns indexed by the MESSAGEix setyear
are returned withint
dtype.
-
var_list
()¶ List all defined variables.
-
vintage_and_active_years
(ya_args=None, in_horizon=True)¶ Return sets of vintage and active years for use in data input.
For a valid pair (year_vtg, year_act), the following conditions are satisfied:
Both the vintage year (year_vtg) and active year (year_act) are in the model’s
year
set.year_vtg <= year_act.
year_act <= the model’s first year or year_act is in the smaller subset
ixmp.Scenario.years_active()
for the given ya_args.
- Parameters
ya_args (tuple of (node, tec, yr_vtg), optional) – Arguments to
years_active()
.in_horizon (bool, optional) – Only return years within the model horizon (
firstmodelyear
or later).
- Returns
with columns ‘year_vtg’ and ‘year_act’, in which each row is a valid pair.
- Return type
-
years_active
(node, tec, yr_vtg)¶ Return years in which tec of yr_vtg can be active in node.
The parameters
duration_period
andtechnical_lifetime
are used to determine which periods are partly or fully within the lifetime of the technology.
-
Model classes¶
|
Model class for MESSAGE. |
|
Model class for MACRO. |
|
Model class for MESSAGE_MACRO. |
|
Extended |
Solver options used by |
|
List of ixmp items for MESSAGE. |
-
message_ix.models.
DEFAULT_CPLEX_OPTIONS
= {'advind': 0, 'epopt': 1e-06, 'lpmethod': 2, 'threads': 4}¶ Solver options used by
message_ix.Scenario.solve()
. These configure the GAMS CPLEX solver (or another solver, if selected); see the solver documentation for possible values.
-
class
message_ix.models.
MESSAGE
(name=None, **model_options)¶ Bases:
message_ix.models.GAMSModel
Model class for MESSAGE.
The MESSAGE Python class encapsulates the GAMS code for the core MESSAGE mathematical formulation. The model_options arguments are received from
Scenario.solve()
, and—except for solve_options—are passed on to the parent classGAMSModel
; see there for a full list of options.-
name
= 'MESSAGE'¶
-
defaults
= dict(...)¶ The paths to MESSAGE GAMS source files use the
MODEL_PATH
configuration setting.MODEL_PATH
, in turn, defaults to “message_ix/model” inside the directory wheremessage_ix
is installed.Key
Value
MESSAGE defaults
model_file
'{MODEL_PATH}/{model_name}_run.gms'
in_file
'{MODEL_PATH}/data/MsgData_{case}.gdx'
out_file
'{MODEL_PATH}/output/MsgOutput_{case}.gdx'
solve_args
['--in="{in_file}"', '--out="{out_file}"', '--iter="{MODEL_PATH}/output/MsgIterationReport_{case}.gdx"']
Inherited from
GAMSModel
case
'{scenario.model}_{scenario.scenario}'
gams_args
['LogOption=4']
check_solution
comment
equ_list
var_list
-
classmethod
initialize
(scenario)¶ Set up scenario with required sets and parameters for MESSAGE.
See also
-
-
class
message_ix.models.
MACRO
(*args, **kwargs)¶ Bases:
message_ix.models.GAMSModel
Model class for MACRO.
-
name
= 'MACRO'¶
-
GAMS_min_version
= '24.8.1'¶ MACRO uses the GAMS
break;
statement, and thus requires GAMS 24.8.1 or later.
-
classmethod
initialize
(scenario, with_data=False)¶ Initialize the model structure.
-
-
class
message_ix.models.
MESSAGE_MACRO
(*args, **kwargs)¶ Bases:
message_ix.models.MACRO
Model class for MESSAGE_MACRO.
MESSAGE_MACRO solves the MESSAGE and MACRO models iteratively, connecting changes in technology activity and resource demands (from MESSAGE) to changes in final demands and prices (from MACRO). This iteration continues until the solution converges; i.e. the two models reach a stable point for the values of these parameters.
MESSAGE_MACRO accepts three additional model_options that control the behaviour of this iteration algorithm:
max_adjustment (
float
, default 0.2): the maximum absolute relative change in final demands between iterations. If MACRO returns demands that have changed by more than a factor outside the range (1 - max_adjustment, 1 + max_adjustment) since the previous iteration, then the change is confined to the limits of that range for the next run of MESSAGE.convergence_criterion (
float
, default 0.01): threshold for model convergence. This option applies to the same value as max_adjustment: the relative change in final demands between two iterations. If the absolute relative change is less than convergence_criterion, the linked model run is complete.max_iteration (
int
, default 50): the maximum number of iterations between the two models. If the solution does not converge after this many iterations, the linked model run fails and no valid result is produced.
See also
-
name
= 'MESSAGE-MACRO'¶
-
class
message_ix.models.
GAMSModel
(name=None, **model_options)¶ Bases:
ixmp.model.gams.GAMSModel
Extended
ixmp.model.gams.GAMSModel
for MESSAGE & MACRO.-
classmethod
initialize
(scenario)¶ Set up scenario with required sets and parameters for MESSAGE.
See also
-
run
(scenario)¶ Execute the model.
GAMSModel creates a file named
cplex.opt
in the model directory containing the options inDEFAULT_CPLEX_OPTIONS
, or any overrides passed tosolve()
.Warning
GAMSModel can solve Scenarios in two or more Python processes simultaneously; but using different CPLEX options in each process may produced unexpected results.
-
classmethod
-
message_ix.models.
MESSAGE_ITEMS
= {'level_storage': {'ix_type': 'set'}, 'map_tec_storage': {'idx_sets': ['node', 'technology', 'storage_tec', 'level', 'commodity'], 'ix_type': 'set'}, 'storage_initial': {'idx_sets': ['node', 'technology', 'level', 'commodity', 'year', 'time'], 'ix_type': 'par'}, 'storage_self_discharge': {'idx_sets': ['node', 'technology', 'level', 'commodity', 'year', 'time'], 'ix_type': 'par'}, 'storage_tec': {'ix_type': 'set'}, 'time_order': {'idx_sets': ['lvl_temporal', 'time'], 'ix_type': 'par'}}¶ List of ixmp items for MESSAGE. Currently, these only include items required for storage, in order to ensure that all scenarios are compatible with the MESSAGE GAMS implementation.
See also
Utility methods¶
-
message_ix.utils.
make_df
(base, **kwargs)¶ Extend or overwrite base with new values from kwargs.
- Parameters
base (dict,
pandas.Series
, orpandas.DataFrame
) – Existing dataset to append to.**kwargs – Additional values to append to base.
- Returns
base modified with kwargs.
- Return type
Examples
Scalar values in base or kwargs are broadcast. The number of rows in the returned
pandas.DataFrame
equals the length of the longest item in either argument.>>> base = {'foo': 'bar'} >>> make_df(base, baz=[42, 43, 44]) foo baz 0 bar 42 1 bar 43 2 bar 44
Testing utilities¶
-
message_ix.testing.
make_dantzig
(mp, solve=False, multi_year=False, **solve_opts)¶ Return an
message_ix.Scenario
for Dantzig’s canning problem.
-
message_ix.testing.
make_westeros
(mp, emissions=False, solve=False)¶ Return an
message_ix.Scenario
for the Westeros model.This is the same model used in the
westeros_baseline.ipynb
tutorial.