Reproducibility

On this page:

Elsewhere:

• A high-level introduction, to how testing supports validity, reproducibility, interoperability, and reusability, in message_ix_models and related packages.

• Test utilities and fixtures (testing) (message_ix_models.testing), on a separate page.

Strategy

The code in model.bare generates a “bare” reference energy system. This is a Scenario that has the same structure (ixmp ‘sets’) as actual instances of the MESSAGEix-GLOBIOM global model, but contains no data (ixmp ‘parameter’ values). Code that operates on the global model can be tested on the bare RES; if it works on that scenario, this is one indication (necessary, but not always sufficient) that it should work on fully-populated scenarios. Such tests are faster and lighter than testing on fully-populated scenarios, and make it easier to isolate errors in the code that is being tested.

Test suite (message_ix_models.tests)

message_ix_models.tests contains a suite of tests written using Pytest.

The following is automatically generated documentation of all modules, test classes, functions, and fixtures in the test suite. Each test should have a docstring explaining what it checks.

tests

Test suite for message_ix_models.

Running the test suite

Some notes for running the test suite.

cached() is used to cache the data resulting from slow operations, like parsing large input files. Data are stored in a location described by the Context setting cache_path. The test suite interacts with caches in two ways:

• --local-cache: Giving this option causes pytest to use whatever cache directory is configured for normal runs/usage of message_ix_models or mix-models.

• By default (without --local-cache), the test suite uses pytest’s built-in cache fixture. This creates and uses a temporary directory, usually .pytest_cache/d/cache/ within the repository root. This location is used only by tests, and not by normal runs/usage of the code.

In either case:

• The tests use existing cached data in these locations and skip over code that generates this data. If the generating code is changed, the cached data must be deleted in order to actually check that the code runs properly.

• Running the test suite with --local-cache causes the local cache to be populated, and this will affect subsequent runs.

• The continuous integration (below) services don’t preserve caches, so code always runs.

Continuous testing

The test suite (message_ix_models.tests) is run using GitHub Actions for new commits on the main branch; new commits on any branch associated with a pull request; and on a daily schedule. These ensure that the code is functional and produces expected outputs, even as upstream dependencies evolve. Workflow runs and their outputs can be viewed here.

Because it is closed-source and requires access to internal IIASA resources, including databases, continuous integration for message_data is handled by GitHub Actions self-hosted runners running on IIASA systems.

Prepare data for testing

Use the export-test-data CLI command:

mix-models --url="ixmp://ixmp-dev/ENGAGE_SSP2_v4.1.7/baseline" export-test-data

See also the documentation for export_test_data(). Use the --exclude, --nodes, and --techs options to control the content of the resulting file.