File formats and input/output

In addition to the data management features provided by Storage back ends (ixmp.backend), ixmp is able to write and read TimeSeries and Scenario data to and from files. This page describes those options and formats.

Time series data

Time series data can be:

Both CSV and Excel files in the IAMC time-series format are supported.

Scenario/model data

Scenario data can be read from/written to Microsoft Excel files using Scenario.read_excel() and to_excel(), and the CLI commands ixmp import scenario FILE and ixmp export FILE. The files have the following structure:

  • One sheet named ‘ix_type_mapping’ with two columns:

    • ‘item’: the name of an ixmp item.

    • ‘ix_type’: the item’s type as a length-3 string: ‘set’, ‘par’, ‘var’, or ‘equ’.

  • One or more sheet per item. If the length of data is greater than the maximum number of rows per sheet supported by the Excel file format (EXCEL_MAX_ROWS), the item is split across multiple sheets named, e.g., ‘foo’, ‘foo(2)’, ‘foo(3)’.

  • Sets:

    • Sheets for one-dimensional indexed sets have one column, with a header cell that is the index set name.

    • Sheets for multi-dimensional indexed sets have multiple columns.

    • Sets with no elements are represented by empty sheets.

  • Parameters, variables, and equations:

    • Sheets have zero (for scalar items) or more columns with headers that are the index names (not necessarily sets; see below) for those dimensions.

    • Parameter sheets have ‘value’ and ‘unit’ columns.

    • Variable and equation sheets have ‘lvl’ and ‘mrg’ columns.

    • Items with no elements are not included in the file.

Limitations

Reading variables and equations

The ixmp API provides no way to set the data of variables and equations, because these are considered model solution data.

Thus, while to_excel() will write files containing variable and equation data, read_excel() can not add these to a Scenario, and only emits log messages indicating that they are ignored.

Multiple dimensions indexed by the same set

read_excel() provides the init_items argument to create new sets and parameters when reading a file. However, the file format does not capture information needed to reconstruct the original data in all cases.

For example:

scenario.init_set('foo')
scenario.add_set('foo', ['a', 'b', 'c'])
scenario.init_par(name='bar', idx_sets=['foo'])
scenario.init_par(
    name='baz',
    idx_sets=['foo', 'foo'],
    idx_names=['foo', 'another_dimension'])
scenario.to_excel('file.xlsx')

file.xlsx will contain sheets named ‘bar’ and ‘baz’. The sheet ‘bar’ will have column headers ‘foo’, ‘value’, and ‘unit’, which are adequate to reconstruct the parameter. However, the sheet ‘baz’ will have column headers ‘foo’ and ‘another_dimension’; this information does not allow ixmp to infer that ‘another_dimension’ is indexed by ‘foo’.

To work around this limitation, initialize ‘baz’ with the correct dimensions before reading its data:

new_scenario.init_par(
    name='baz',
    idx_sets=['foo', 'foo'],
    idx_names=['foo', 'another_dimension'])
new_scenario.read_excel('file.xlsx', init_items=True)
File formats other than .xlsx

The .xlsx (Office Open XML) file format is preferred for input and output. ixmp uses openpyxl and pandas in order to read and write this format. For other Excel file formats, including .xls and .xlsb, see the Pandas documentation.