Storage back ends (ixmp.backend)

ixmp includes ixmp.backend.jdbc.JDBCBackend, which can store data in many types of relational database management systems (RDBMS) that have Java DataBase Connector (JDBC) interfaces—hence its name.

ixmp is extensible to support other methods of storing data: in non-JDBC RDBMS, non-relational databases, local files, memory, or other ways. Developers wishing to add such capabilities may subclass ixmp.backend.base.Backend and implement its methods.

Provided backends

ixmp.backend.BACKENDS: Dict[str, Type] = {'jdbc': <class 'ixmp.backend.jdbc.JDBCBackend'>}

Mapping from names to available backends. To register additional backends, add entries to this dictionary.

class ixmp.backend.jdbc.JDBCBackend(jvmargs=None, **kwargs)

Backend using JPype/JDBC to connect to Oracle and HyperSQL databases.

This backend is based on the third-party JPype Python package that allows interaction with Java code.

Parameters:

• driver ('oracle' or 'hsqldb') – JDBC driver to use.

• path (os.PathLike, optional) – Path to the HyperSQL database.

• url (str, optional) – Partial or complete JDBC URL for the Oracle or HyperSQL database, e.g. database-server.example.com:PORT:SCHEMA. See Configuration.

• user (str, optional) – Database user name.

• password (str, optional) – Database user password.

• cache (bool, optional) – If True (the default), cache Python objects after conversion from Java objects.

• jvmargs (str, optional) – Java Virtual Machine arguments. See start_jvm().

• dbprops (os.PathLike, optional) – With driver='oracle', the path to a database properties file containing driver, url, user, and password information.

JDBCBackend supports:

• Databases in local files (HyperSQL) using driver='hsqldb' and the path argument.

• Remote, Oracle databases using driver='oracle' and the url, username and password arguments.

• Temporary, in-memory databases using driver='hsqldb' and the url argument. Use the url parameter with the format jdbc:hsqldb:mem:[NAME], where [NAME] is any string:

  mp = ixmp.Platform(
      backend="jdbc",
      driver="hsqldb",
      url="jdbc:hsqldb:mem:temporary platform",
  )
  

JDBCBackend caches values in memory to improve performance when repeatedly reading data from the same items with par(), equ(), or var().

Tip

If repeatedly accessing the same item with different filters:

1. First, access the item by calling e.g. par() without any filters. This causes the full contents of the item to be loaded into cache.

2. Then, access by making multiple par() calls with different filters arguments. The cache value is filtered and returned without further access to the database.

Tip

Modifying an item by adding or deleting elements invalidates its cache.

JDBCBackend has the following limitations:

• The comment argument to Platform.add_unit() is limited to 64 characters.

• Infinite floating-point values (numpy.inf, math.inf) cannot be stored using TimeSeries.add_timeseries() when using an Oracle database via driver='oracle'.

JDBCBackend’s implementation allows the following kinds of file input and output:

read_file(path, item_type, **kwargs)	Read Platform, TimeSeries, or Scenario data from file.
write_file(path, item_type, **kwargs)	Write Platform, TimeSeries, or Scenario data to file.

classmethod handle_config(args, kwargs)

Handle platform/backend config arguments.

args will overwrite any kwargs, and may be one of:

• (“oracle”, url, user, password, [jvmargs]) for an Oracle database.

• (“hsqldb”, path, [jvmargs]) for a file-backed HyperSQL database.

• (“hsqldb”,) with “url” supplied via kwargs, e.g. “jdbc:hsqldb:mem://foo” for an in-memory database.

read_file(path, item_type: ItemType, **kwargs)

Read Platform, TimeSeries, or Scenario data from file.

JDBCBackend supports reading from:

• path='*.gdx', item_type=ItemType.MODEL. The keyword arguments check_solution, comment, equ_list, and var_list are required.

Parameters:

• check_solution (bool) – If True, raise an exception if the GAMS solver did not reach optimality. (Only for MESSAGE-scheme Scenarios.)

• comment (str) – Comment added to Scenario when importing the solution.

• equ_list (list of str) – Equations to be imported.

• var_list (list of str) – Variables to be imported.

• filters (dict of dict of str) – Restrict items read.

See also

Backend.read_file

write_file(path, item_type: ItemType, **kwargs)

Write Platform, TimeSeries, or Scenario data to file.

JDBCBackend supports writing to:

• path='*.gdx', item_type=ItemType.SET | ItemType.PAR.

• path='*.csv', item_type=TS. The default keyword argument is required.

Parameters:

filters (dict of dict of str) –

Restrict items written. The following filters may be used:

• model : str

• scenario : str

• variable : list of str

• default : bool. If True, only data from TimeSeries versions with TimeSeries.set_as_default() are written.

See also

Backend.write_file

ixmp.backend.jdbc.start_jvm(jvmargs=None)

Start the Java Virtual Machine via JPype.

Parameters:

jvmargs (str or list of str, optional) –

Additional arguments for launching the JVM, passed to jpype.startJVM().

For instance, to set the maximum heap space to 4 GiB, give jvmargs=['-Xmx4G']. See the JVM documentation for a list of options.

Backend API

ixmp.backend.base.Backend()	Abstract base class for backends.
ixmp.backend.base.CachingBackend([cache_enabled])	Backend with additional features for caching data.
ixmp.backend.ItemType(value[, names, ...])	Type of data items in ixmp.TimeSeries and ixmp.Scenario.
ixmp.backend.FIELDS	Lists of field names for tuples returned by Backend API methods.
ixmp.backend.IAMC_IDX	Partial list of dimensions for the IAMC data structure, or “IAMC format”.

• ixmp.Platform implements a user-friendly API for scientific programming. This means its methods can take many types of arguments, check, and transform them—in a way that provides modeler-users with easy, intuitive workflows.

• In contrast, Backend has a very simple API that accepts arguments and returns values in basic Python data types and structures.

• As a result:

  • Platform code is not affected by where and how data is stored; it merely handles user arguments and then makes, usually, a single Backend call.

  • Backend code does not need to perform argument checking; merely store and retrieve data reliably.

• Additional Backends may inherit from Backend or CachingBackend.

class ixmp.backend.base.Backend

Abstract base class for backends.

In the following, the bold-face words required, optional, etc. have specific meanings as described in IETF RFC 2119.

Backend is an abstract class; this means it must be subclassed. Most of its methods are decorated with abc.abstractmethod; this means they are required and must be overridden by subclasses.

Others, marked below with “OPTIONAL:”, are not so decorated. For these methods, the behaviour in the base Backend—often, nothing—is an acceptable default behaviour. Subclasses may extend or replace this behaviour as desired, so long as the methods still perform the actions described in the description.

Backends:

• must only raise standard Python exceptions.

• must implement the data model as described, or raise NotImplementedError for not implemented parts of the data model.

Methods related to ixmp.Platform:

add_model_name	Add (register) new model name.
add_scenario_name	Add (register) new scenario name.
close_db	OPTIONAL: Close database connection(s).
get_auth	OPTIONAL: Return user authorization for models.
get_doc	Read documentation from database
get_log_level	OPTIONAL: Get logging level for the backend and other code.
get_meta	Retrieve all metadata attached to a specific target.
get_model_names	List existing model names.
get_nodes	Iterate over all nodes stored on the Platform.
get_scenarios	Iterate over TimeSeries stored on the Platform.
get_scenario_names	List existing scenario names.
get_units	Return all registered symbols for units of measurement.
handle_config	OPTIONAL: Handle platform/backend config arguments.
open_db	OPTIONAL: (Re-)open database connection(s).
read_file	OPTIONAL: Read Platform, TimeSeries, or Scenario data from file.
remove_meta	Remove metadata attached to a target.
set_doc	Save documentation to database
set_log_level	OPTIONAL: Set logging level for the backend and other code.
set_meta	Set metadata on a target.
set_node	Add a node name to the Platform.
set_unit	Add a unit of measurement to the Platform.
write_file	OPTIONAL: Write Platform, TimeSeries, or Scenario data to file.

Methods related to ixmp.TimeSeries:

• Each method has an argument ts, a reference to the TimeSeries object being manipulated.

• ‘Geodata’ is otherwise identical to regular timeseries data, except value are str rather than float.

check_out	Check out ts for modification.
commit	Commit changes to ts.
delete	Remove time series data.
delete_geo	Remove 'geodata' values.
discard_changes	Discard changes to ts since the last check_out().
get	Retrieve the existing TimeSeries (or Scenario) ts.
get_data	Retrieve time series data.
get_geo	Retrieve time-series 'geodata'.
init	Create a new TimeSeries (or Scenario) ts.
is_default	Return True if ts is the default version for its (model, scenario).
last_update	Return the date of the last modification of the ts, if any.
preload	OPTIONAL: Load ts data into memory.
run_id	Return the run ID of the ts.
set_data	Store data.
set_as_default	Set the current TimeSeries.version as the default.
set_geo	Store time series geodata.

Methods related to ixmp.Scenario:

• Each method has an argument s, a reference to the Scenario object being manipulated.

clone	Clone s.
delete_item	Remove an item name of type.
get_meta	Retrieve all metadata attached to a specific target.
has_solution	Return True if Scenario s has been solved.
init_item	Initialize an item name of type.
item_delete_elements	Remove elements of item name.
item_get_elements	Return elements of item name.
item_set_elements	Add keys or values to item name.
item_index	Return the index sets or names of item name.
list_items	Return a list of names of items of type.
remove_meta	Remove metadata attached to a target.
set_meta	Set metadata on a target.

Methods related to message_ix.Scenario:

• Each method has an argument ms, a reference to the Scenario object being manipulated.

Warning

These methods may be moved to ixmp in a future release.

cat_get_elements	Get elements of a category mapping.
cat_list	Return list of categories in mapping name.
cat_set_elements	Add elements to category mapping.

abstract add_model_name(name: str) → None

Add (register) new model name.

Parameters:

name (str) – New model name

abstract add_scenario_name(name: str) → None

Add (register) new scenario name.

Parameters:

name (str) – New scenario name

abstract cat_get_elements(ms: Scenario, name: str, cat: str) → List[str]

Get elements of a category mapping.

Parameters:

• name (str) – Name of the category mapping set.

• cat (str) – Name of the category within name.

Returns:

All set elements mapped to cat in name.

Return type:

list of str

abstract cat_list(ms: Scenario, name: str) → List[str]

Return list of categories in mapping name.

Parameters:

name (str) – Name of the category mapping set.

Returns:

All categories in name.

Return type:

list of str

abstract cat_set_elements(ms: Scenario, name: str, cat: str, keys: str | Sequence[str], is_unique: bool) → None

Add elements to category mapping.

Parameters:

• name (str) – Name of the category mapping set.

• cat (str) – Name of the category within name.

• keys (collections.abc.Iterable of str or list of str) – Keys to add to cat.

• is_unique (bool) –

  If True:

  • keys must contain only one key.

  • The Backend must remove any existing member of cat, so that it has only one element.

abstract check_out(ts: TimeSeries, timeseries_only: bool) → None

Check out ts for modification.

Parameters:

timeseries_only (bool) –

???

abstract clear_solution(s: Scenario, from_year=None)

Remove data associated with a model solution.

Todo

Document.

abstract clone(s: Scenario, platform_dest: Platform, model: str, scenario: str, annotation: str, keep_solution: bool, first_model_year: int | None = None) → Scenario

Clone s.

Parameters:

• platform_dest (Platform) – Target backend. May be the same as s.platform.

• model (str) – New model name.

• scenario (str) – New scenario name.

• annotation (str) – Description for the creation of the new scenario.

• keep_solution (bool) – If True, model solution data is also cloned. If False, it is discarded.

• first_model_year (int or None) – If int, must be greater than the first model year of s.

Returns:

The cloned Scenario. If s is an instance of a subclass of ixmp.Scenario, the returned object must be of the same subclass.

Return type:

Scenario

close_db() → None

OPTIONAL: Close database connection(s).

Close any database connection(s), if open.

abstract commit(ts: TimeSeries, comment: str) → None

Commit changes to ts.

ts_init may modify the version attribute of ts.

Parameters:

comment (str) – Description of the changes being committed.

del_ts(ts: TimeSeries) → None

OPTIONAL: Free memory associated with the TimeSeries ts.

The default implementation has no effect.

abstract delete(ts: TimeSeries, region: str, variable: str, subannual: str, years: Iterable[int], unit: str) → None

Remove time series data.

Parameters:

• region (str) – Region name.

• variable (str) – Variable name.

• years (Iterable of int) – Years.

• unit (str) – Unit symbol.

• subannual (str) – Name of time slice.

abstract delete_geo(ts: TimeSeries, region: str, variable: str, subannual: str, years: Iterable[int], unit: str) → None

Remove ‘geodata’ values.

Parameters:

• region (str) – Region name.

• variable (str) – Variable name.

• subannual (str) – Name of time slice.

• years (Iterable of int) – Years.

• unit (str) – Unit symbol.

abstract delete_item(s: Scenario, type: str, name: str) → None

Remove an item name of type.

Parameters:

• type ('set' or 'par' or 'equ') –

• name (str) – Name of the item to delete.

abstract discard_changes(ts: TimeSeries) → None

Discard changes to ts since the last check_out().

abstract get(ts: TimeSeries) → None

Retrieve the existing TimeSeries (or Scenario) ts.

The TimeSeries is identified based on the unique combination of the attributes of ts:

• model,

• scenario, and

• version.

If version is None, the Backend must return the version marked as default, and must set the attribute value.

If ts is a Scenario, get() must set the scheme attribute with the value previously passed to init().

Raises:

ValueError – If model or scenario does not exist on the Platform.

See also

is_default, set_as_default

get_auth(user: str, models: Sequence[str], kind: str) → Dict[str, bool]

OPTIONAL: Return user authorization for models.

If the Backend implements access control, this method must indicate whether user has permission kind for each of models.

kind may be ‘read’/’view’, ‘write’/’modify’, or other values; get_auth() should raise exceptions on invalid values.

Parameters:

• user (str) – User name or identifier.

• models (list of str) – Model names.

• kind (str) – Type of permission being requested

Returns:

Mapping of model name (str) → bool; True if the user is authorized for the model.

Return type:

dict

abstract get_data(ts: TimeSeries, region: Sequence[str], variable: Sequence[str], unit: Sequence[str], year: Sequence[str]) → Iterable[Tuple[str, str, str, int, float]]

Retrieve time series data.

Parameters:

• region (list of str) – Region names to filter results.

• variable (list of str) – Variable names to filter results.

• unit (list of str) – Unit symbols to filter results.

• year (list of str) – Years to filter results.

Yields:

tuple – The members of each tuple are:

ID	Type	Description
region	str	Region name
variable	str	Variable name
unit	str	Unit symbol
year	int	Year
value	float	Data value

abstract get_doc(domain: str, name: str | None = None) → str | Dict

Read documentation from database

Parameters:

• domain (str) – Documentation domain, e.g. model, scenario etc

• name (str, optional) – Name of domain entity (e.g. model name).

Returns:

String representing fragment of documentation if name is passed as parameter or dictionary containing mapping between name of domain object (e.g. model name) and string representing fragment when name parameter is omitted.

Return type:

str or dict

abstract get_geo(ts: TimeSeries) → Iterable[Tuple[str, str, int, str, str, str, bool]]

Retrieve time-series ‘geodata’.

Yields:

tuple – The members of each tuple are:

ID	Type	Description
region	str	Region name
variable	str	Variable name
year	int	Year
value	str	Value
unit	str	Unit symbol
subannual	str	Name of time slice
meta	bool	True if the data is marked as metadata

get_log_level() → str

OPTIONAL: Get logging level for the backend and other code.

The default implementation returns the effective level of the “ixmp.backend.base” logger; usually the same as “ixmp” or “ixmp.backend” (if set).

Returns:

Name of a Python logging level.

Return type:

str

See also

set_log_level

abstract get_meta(model: str | None, scenario: str | None, version: int | None, strict: bool) → Dict[str, Any]

Retrieve all metadata attached to a specific target.

Depending on which of model, scenario, version are None, metadata attached to one of the four kinds of metadata targets (see Metadata) is returned.

If strict is False, then get_meta() must also return metadata attached to less specific or “higher level” targets:

• For (model, scenario, version), these are (model, scenario); (model,); and (scenario).

• For (model, scenario), these are (model,) and (scenario,).

• For (model,) or (scenario,), there are no less specific targets.

Parameters:

• model (str, optional) – Model name of metadata target.

• scenario (str, optional) – Scenario name of metadata target.

• version (int, optional) – TimeSeries.version of metadata target.

• strict (bool) – Only retrieve metadata from the specified target.

Returns:

Mapping from metadata names/identifiers (str) to values (Any).

Return type:

dict

Raises:

ValueError – on unsupported (model, scenario, version) combinations.

abstract get_model_names() → Iterable[str]

List existing model names.

Returns:

List of the retrieved model names.

Return type:

list of str

abstract get_nodes() → Iterable[Tuple[str, str | None, str, str]]

Iterate over all nodes stored on the Platform.

Yields:

tuple – The members of each tuple are:

ID	Type	Description
region	str	Node name or synonym for node
mapped_to	str or None	Node name
parent	str	Parent node name
hierarchy	str	Node hierarchy ID

See also

set_node

abstract get_scenario_names() → Iterable[str]

List existing scenario names.

Returns:

List of the retrieved scenario names.

Return type:

list of str

abstract get_scenarios(default: bool, model: str | None, scenario: str | None) → Iterable[Tuple[str, str, str, bool, bool, str, str, str, str, str, str, str, int]]

Iterate over TimeSeries stored on the Platform.

Scenarios, as subclasses of TimeSeries, are also included.

Parameters:

• default (bool) – True to include only TimeSeries versions marked as default.

• model (str or None) – Model name to filter results.

• scenario (str or None) – Scenario name to filter results.

Yields:

tuple – The members of each tuple are:

ID	Type	Description
model	str	Model name
scenario	str	Scenario name
scheme	str	Scheme name
is_default	bool	True if version is the default
is_locked	bool	True if read-only
cre_user	str	Name of user who created the TimeSeries
cre_date	str	Creation datetime
upd_user	str	Name of user who last modified the TimeSeries
upd_date	str	Modification datetime
lock_user	str	Name of user who locked the TimeSeries
lock_date	str	Lock datetime
annotation	str	Description of the TimeSeries
version	int	Version

abstract get_timeslices() → Iterable[Tuple[str, str, float]]

Iterate over subannual timeslices defined on the Platform instance.

Yields:

tuple – The members of each tuple are:

ID	Type	Description
name	str	Time slice name
category	str	Time slice category
duration	float	Time slice duration (fraction of year)

See also

set_timeslice

abstract get_units() → List[str]

Return all registered symbols for units of measurement.

Return type:

list of str

See also

set_unit

classmethod handle_config(args: Sequence, kwargs: MutableMapping) → Dict[str, Any]

OPTIONAL: Handle platform/backend config arguments.

Returns a dict to be stored in the configuration file. This dict must be valid as keyword arguments to the __init__() method of a Backend subclass.

The default implementation expects both args and kwargs to be empty.

See also

Config.add_platform

abstract has_solution(s: Scenario) → bool

Return True if Scenario s has been solved.

If True, model solution data is available from the Backend.

abstract init(ts: TimeSeries, annotation: str) → None

Create a new TimeSeries (or Scenario) ts.

init may modify the version attribute of ts.

If ts is a Scenario; the Backend must store the Scenario.scheme attribute.

Parameters:

annotation (str) – If ts is newly-created, the Backend must store this annotation with the TimeSeries.

abstract init_item(s: Scenario, type: str, name: str, idx_sets: Sequence[str], idx_names: Sequence[str] | None) → None

Initialize an item name of type.

Parameters:

• type ('set' or 'par' or 'equ' or 'var') –

• name (str) – Name for the new item.

• idx_sets (collections.abc.Sequence of str) – If empty, a 0-dimensional/scalar item is initialized. Otherwise, a 1+-dimensional item is initialized.

• idx_names (collections.abc.Sequence of str or None) – Optional names for the dimensions. If not supplied, the names of the idx_sets (if any) are used. If supplied, idx_names and idx_sets must be the same length.

Raises:

ValueError – if any of the idx_sets is not an existing set in the Scenario; if idx_names and idx_sets are not the same length.

abstract is_default(ts: TimeSeries) → bool

Return True if ts is the default version for its (model, scenario).

See also

get, set_as_default

abstract item_delete_elements(s: Scenario, type: str, name: str, keys) → None

Remove elements of item name.

Parameters:

• type ('par' or 'set') –

• name (str) –

• keys (collections.abc.Iterable of collections.abc.Iterable of str) – If name is indexed by other set(s), then the number of elements of each key in keys, and their contents, must match the index set(s). If name is a basic set, then each key must be a list containing a single str, which must exist in the set.

See also

init_item, item_set_elements

abstract item_get_elements(s: Scenario, type: Literal['equ', 'par', 'set', 'var'], name: str, filters: Dict[str, List[Any]] | None = None) → Dict[str, Any] | Series | DataFrame

Return elements of item name.

Parameters:

• type (str) – Type of the item.

• name (str) – Name of the item.

• filters (dict, optional) –

  If provided, a mapping from dimension names (class:str) to allowed values along that dimension (list).

  item_get_elements must silently accept values that are not members of the set indexing a dimension. Elements which are not str must be handled as equivalent to their string representation; that is, item_get_elements must return the same data for filters={'foo': [42]} and filters={'foo': ['42']}.

Returns:

• pandas.Series – When type is ‘set’ and name an index set (not indexed by other sets).

• dict – When type is ‘equ’, ‘par’, or ‘var’ and name is scalar (zero- dimensional). The value has the keys ‘value’ and ‘unit’ (for ‘par’) or ‘lvl’ and ‘mrg’ (for ‘equ’ or ‘var’).

• pandas.DataFrame – For mapping sets, or all 1+-dimensional values. The dataframe has one column per index name with dimension values; plus the columns ‘value’ and ‘unit’ (for ‘par’) or ‘lvl’ and ‘mrg’ (for ‘equ’ or ‘var’).

Raises:

KeyError – If name does not exist in s.

abstract item_index(s: Scenario, name: str, sets_or_names: str) → List[str]

Return the index sets or names of item name.

Parameters:

sets_or_names ('sets' or 'names') –

Return type:

list of str

abstract item_set_elements(s: Scenario, type: str, name: str, elements: Iterable[Tuple[Any, float | None, str | None, str | None]]) → None

Add keys or values to item name.

Parameters:

• type ('par' or 'set') –

• name (str) – Name of the items.

• elements (collections.abc.Iterable of tuple) –

  The members of each tuple are:

  ID	Type	Description
  key	str or list of str or None	Set elements or value indices
  value	float or None	Parameter value
  unit	str or None	Unit symbol
  comment	str or None	Description of the change

  If name is indexed by other set(s), then the number of elements of each key, and their contents, must match the index set(s). When type is ‘set’, value and unit must be None.

Raises:

• ValueError – If elements contain invalid values, e.g. key values not in the index set(s).

• Exception – If the Backend encounters any error adding the elements.

See also

init_item, item_delete_elements

abstract last_update(ts: TimeSeries) → str | None

Return the date of the last modification of the ts, if any.

abstract list_items(s: Scenario, type: str) → List[str]

Return a list of names of items of type.

Parameters:

type ('set' or 'par' or 'equ') –

open_db() → None

OPTIONAL: (Re-)open database connection(s).

A backend may connect to a database server. This method opens the database connection if it is closed.

preload(ts: TimeSeries) → None

OPTIONAL: Load ts data into memory.

read_file(path: PathLike, item_type: ItemType, **kwargs) → None

OPTIONAL: Read Platform, TimeSeries, or Scenario data from file.

A backend may implement read_file for one or more combinations of the path and item_type methods. For all other combinations, it must raise NotImplementedError.

The default implementation supports:

• path ending in ‘.xlsx’, item_type is ItemType.MODEL: read a single Scenario given by kwargs['filters']['scenario'] from file, using s_read_excel().

Parameters:

• path (os.PathLike) –

  File for input. The filename suffix determines the input format:

  Suffix	Format
  .csv	Comma-separated values
  .gdx	GAMS data exchange
  .xlsx	Microsoft Office Open XML spreadsheet

• item_type (ItemType) – Type(s) of items to read.

Raises:

• ValueError – If ts is not None and ‘scenario’ is a key in filters.

• NotImplementedError – If input of the specified items from the file format is not supported.

See also

write_file

abstract remove_meta(names: list, model: str | None, scenario: str | None, version: int | None) → None

Remove metadata attached to a target.

Parameters:

• names (list of str) – Metadata names/identifiers to remove.

• model (str or None) – Model name of metadata target.

• scenario (str or None) – Scenario name of metadata target.

• version (int or None) – TimeSeries.version of metadata target.

Raises:

ValueError – on unsupported (model, scenario, version) combinations.

See also

get_meta

abstract run_id(ts: TimeSeries) → int

Return the run ID of the ts.

abstract set_as_default(ts: TimeSeries) → None

Set the current TimeSeries.version as the default.

See also

get, is_default

abstract set_data(ts: TimeSeries, region: str, variable: str, data: Dict[int, float], unit: str, subannual: str, meta: bool) → None

Store data.

Parameters:

• region (str) – Region name.

• variable (str) – Variable name.

• subannual (str) – Name of time slice.

• unit (str) – Unit symbol.

• data – Mapping from year (int) to value (float).

• meta (bool) – True to mark data as metadata.

abstract set_doc(domain: str, docs) → None

Save documentation to database

Parameters:

• domain (str) – Documentation domain, e.g. model, scenario, etc.

• docs (dict or collections.abc.Iterable of tuple) – Dictionary or tuple array containing mapping between name of domain object (e.g. model name) and string representing fragment of documentation.

abstract set_geo(ts: TimeSeries, region: str, variable: str, subannual: str, year: int, value: str, unit: str, meta: bool) → None

Store time series geodata.

Parameters:

• region (str) – Region name.

• variable (str) – Variable name.

• subannual (str) – Name of time slice.

• year (int) – Year.

• value (str) – Data value.

• unit (str) – Unit symbol.

• meta (bool) – True to mark data as metadata.

set_log_level(level: int) → None

OPTIONAL: Set logging level for the backend and other code.

The default implementation has no effect.

Parameters:

level (int) – A Python logging level.

See also

get_log_level

abstract set_meta(meta: dict, model: str | None, scenario: str | None, version: int | None) → None

Set metadata on a target.

Parameters:

• meta (dict) – Mapping from metadata names/identifiers to values.

• model (str or None) – Model name of metadata target.

• scenario (str or None) – Scenario name of metadata target.

• version (int or None) – TimeSeries.version of metadata target.

Raises:

ValueError – on unsupported (model, scenario, version) combinations.

See also

get_meta

abstract set_node(name: str, parent: str | None = None, hierarchy: str | None = None, synonym: str | None = None) → None

Add a node name to the Platform.

This method must have one of two effects, depending on the arguments:

• With parent and hierarchy: name is added as a child of parent in the named hierarchy.

• With synonym: synonym is added as an alias for name.

Parameters:

• name (str) – Node name.

• parent (str, optional) – Parent node name.

• hierarchy (str, optional) – Node hierarchy ID.

• synonym (str, optional) – Synonym for node.

See also

get_nodes

abstract set_timeslice(name: str, category: str, duration: float) → None

Add a subannual time slice to the Platform.

Parameters:

• name (str) – Node name.

• category (str) – Time slice category.

• duration (float) – Time slice duration (a fraction of a year).

See also

get_timeslices

abstract set_unit(name: str, comment: str) → None

Add a unit of measurement to the Platform.

Parameters:

• name (str) – Symbol of the unit.

• comment (str) – Description of the change or of the unit.

See also

get_units

write_file(path: PathLike, item_type: ItemType, **kwargs) → None

OPTIONAL: Write Platform, TimeSeries, or Scenario data to file.

A backend may implement write_file for one or more combinations of the path and item_type methods. For all other combinations, it must raise NotImplementedError.

The default implementation supports:

• path ending in ‘.xlsx’, item_type is either MODEL or SET | PAR: write a single Scenario given by kwargs['filters']['scenario'] to file using s_write_excel().

Parameters:

• path (os.PathLike) – File for output. The filename suffix determines the output format.

• item_type (ItemType) – Type(s) of items to write.

Raises:

• ValueError – If ts is not None and ‘scenario’ is a key in filters.

• NotImplementedError – If output of the specified items to the file format is not supported.

See also

read_file

class ixmp.backend.base.CachingBackend(cache_enabled=True)

Backend with additional features for caching data.

CachingBackend stores cache values for multiple TimeSeries/Scenario objects, and for multiple values of a filters argument.

Subclasses must call cache(), cache_get(), and cache_invalidate() as appropriate to manage the cache; CachingBackend does not enforce any such logic.

_cache: Dict[Tuple, object] = {}

Cache of values. Keys are given by _cache_key(); values depend on the subclass’ usage of the cache.

_cache_hit: Dict[Tuple, int] = {}

Count of number of times a value was retrieved from cache successfully using cache_get().

classmethod _cache_key(ts: TimeSeries, ix_type: str | None, name: str | None, filters: Dict[str, Hashable] | None = None) → Tuple[Hashable, ...]

Return a hashable cache key.

ixmp filters (a dict of list) are converted to a unique id that is hashable.

Returns:

A hashable key with 4 elements for ts, ix_type, name, and filters.

Return type:

tuple

cache(ts: TimeSeries, ix_type: str, name: str, filters: Dict, value: Any) → bool

Store value in cache.

Returns:

True if the key was already in the cache and its value was overwritten.

Return type:

bool

cache_enabled = True

True if caching is enabled.

cache_get(ts: TimeSeries, ix_type: str, name: str, filters: Dict) → Any | None

Retrieve value from cache.

The value in _cache is copied to avoid cached values being modified by user code. _cache_hit is incremented.

Raises:

KeyError – If the key for ts, ix_type, name and filters is not in the cache.

cache_invalidate(ts: TimeSeries, ix_type: str | None = None, name: str | None = None, filters: Dict | None = None) → None

Invalidate cached values.

With all arguments given, single key/value is removed from the cache. Otherwise, multiple keys/values are removed:

• ts only: all cached values associated with the TimeSeries or Scenario object.

• ts, ix_type, and name: all cached values associated with the item, whether filtered or unfiltered.

del_ts(ts: TimeSeries)

Invalidate cache entries associated with ts.

Backend API.

ixmp.backend.FIELDS = {'get_nodes': ('region', 'mapped_to', 'parent', 'hierarchy'), 'get_scenarios': ('model', 'scenario', 'scheme', 'is_default', 'is_locked', 'cre_user', 'cre_date', 'upd_user', 'upd_date', 'lock_user', 'lock_date', 'annotation', 'version'), 'get_timeslices': ('name', 'category', 'duration'), 'ts_get': ('region', 'variable', 'unit', 'subannual', 'year', 'value'), 'ts_get_geo': ('region', 'variable', 'subannual', 'year', 'value', 'unit', 'meta'), 'write_file': ('MODEL', 'SCENARIO', 'VERSION', 'VARIABLE', 'UNIT', 'REGION', 'META', 'SUBANNUAL', 'YEAR', 'VALUE')}

Lists of field names for tuples returned by Backend API methods.

The key “write_file” refers to the columns appearing in the CSV output from export_timeseries_data() when using JDBCBackend.

Todo

Make this consistent with other dimension orders and with IAMC_IDX.

ixmp.backend.IAMC_IDX: List[str | int] = ['model', 'scenario', 'region', 'variable', 'unit']

Partial list of dimensions for the IAMC data structure, or “IAMC format”. This omits “year” and “subannual” which appear in some variants of the structure, but not in others.

class ixmp.backend.ItemType(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)

Type of data items in ixmp.TimeSeries and ixmp.Scenario.

TS = 1

T = 1

Time series data variable.

SET = 2

S = 2

Set.

PAR = 4

P = 4

Parameter.

VAR = 8

V = 8

Model variable.

EQU = 16

E = 16

Equation.

MODEL = 30

M = 30

All kinds of model-related data, i.e. SET, PAR, VAR and EQU.

SOLUTION = 24

Model solution data, i.e. VAR and EQU.

ALL = 31

A = 31

All data, i.e. MODEL and TS.

Common input/output routines for backends

ixmp.backend.io.EXCEL_MAX_ROWS = 1048576

Maximum number of rows supported by the Excel file format. See to_excel() and Scenario/model data.

ixmp.backend.io.maybe_init_item(scenario, ix_type, name, new_idx, path)

Call init_set(), init_par(), etc. if possible.

Logs an intelligible warning and then raises ValueError in two cases:

• the new_idx is ambiguous, e.g. containing index names that cannot be used to infer index sets, or

• an existing item has index names that are different from new_idx.

ixmp.backend.io.s_read_excel(be, s, path, add_units=False, init_items=False, commit_steps=False)

Read data from a Microsoft Excel file at path into s.

See also

Scenario.read_excel

ixmp.backend.io.s_write_excel(be, s, path, item_type, filters=None, max_row=None)

Write s to a Microsoft Excel file at path.

See also

Scenario.to_excel

ixmp.backend.io.ts_read_file(ts, path, firstyear=None, lastyear=None)

Read data from a CSV or Microsoft Excel file at path into ts.

See also

TimeSeries.add_timeseries, TimeSeries.read_file