Test utilities and fixtures (testing)

Fixtures:

mix_models_cli(session_context, tmp_env)	A CliRunner object that invokes the mix-models CLI.
session_context(pytestconfig, tmp_env)	A Context connected to a temporary, in-memory database.
test_context(request, session_context)	A copy of session_context() scoped to one test function.
user_context(request)	Context which can access user's configuration, e.g. platform names.

Marks:

NIE	Shorthand for marking a parametrized test case that is expected to fail because it is not implemented.
not_ci([reason, action])	Mark a test as xfail or skipif if on CI infrastructure.

Others:

EXPORT_OMIT	Items with names that match (partially or fully) these names are omitted by export_test_data().
bare_res(request, context[, solved])	Return or create a Scenario containing the bare RES for use in testing.
export_test_data(context)	Export a subset of data from a scenario, for use in tests.
pytest_addoption(parser)	Add two command-line options to pytest:

message_ix_models.testing.EXPORT_OMIT = ['aeei', 'cost_MESSAGE', 'demand_MESSAGE', 'demand', 'depr', 'esub', 'gdp_calibrate', 'grow', 'historical_gdp', 'kgdp', 'kpvs', 'lakl', 'land', 'lotol', 'mapping_macro_sector', 'MERtoPPP', 'prfconst', 'price_MESSAGE', 'ref_', 'sector']

Items with names that match (partially or fully) these names are omitted by export_test_data().

message_ix_models.testing.GHA = False

True if tests occur on GitHub Actions.

message_ix_models.testing.KEY = {'cache-path': <_pytest.stash.StashKey object>, 'user-local-data': <_pytest.stash.StashKey object>}

Keys for pytestconfig.stash.

message_ix_models.testing.MARK: dict[str, MarkDecorator | MarkFactory] = {'GEA_not_implemented': MarkDecorator(mark=Mark(name='xfail', args=(), kwargs={'reason': 'Not implemented with GEA input data'})), 'ISR_no_data': MarkDecorator(mark=Mark(name='xfail', args=(), kwargs={'raises': <class 'FileNotFoundError'>, 'reason': 'Missing ISR/mer-to-ppp.csv + not supported by MaybeAdaptR11Source'})), 'R12_only': MarkDecorator(mark=Mark(name='skip', args=(), kwargs={'reason': 'Currently only possible with node=R12 input data/config'})), 'R14_no_data': MarkDecorator(mark=Mark(name='xfail', args=(), kwargs={'reason': 'Missing R14 input data/config'})), 'SHAPE_not_implemented': MarkDecorator(mark=Mark(name='xfail', args=(), kwargs={'reason': 'Not implemented with SHAPE input data'})), 'ci_db_access': MarkDecorator(mark=Mark(name='xfail', args=(), kwargs={'condition': False, 'reason': 'GitHub-hosted runner has no access to IIASA-internal databases'})), 'ci_linux_only': MarkDecorator(mark=Mark(name='skipif', args=(), kwargs={'condition': False, 'reason': 'Skip on non-Linux GitHub Actions runners, for performance'})), 'ci_not_macos_intel': MarkDecorator(mark=Mark(name='skipif', args=(), kwargs={'condition': False, 'reason': 'R not installed on macos-15-intel GitHub Actions runners'})), 'ci_timeout': MarkDecorator(mark=Mark(name='skipif', args=(False,), kwargs={'reason': 'Stalls or times out on GitHub Actions'})), 'gh_375': MarkDecorator(mark=Mark(name='flaky', args=(), kwargs={'reruns': 3, 'rerun_delay': 2, 'condition': False, 'reason': 'https://github.com/iiasa/message-ix-models/issues/375'})), 'gh_471': MarkDecorator(mark=Mark(name='xfail', args=(), kwargs={'condition': False, 'reason': 'Temporary, for https://github.com/iiasa/message-ix-models/pull/471'})), 'ixmp_595': MarkDecorator(mark=Mark(name='xfail', args=(), kwargs={'condition': True, 'reason': 'https://github.com/iiasa/ixmp#595'})), 'ixmp_600': MarkDecorator(mark=Mark(name='xfail', args=(), kwargs={'condition': False, 'reason': 'https://github.com/iiasa/ixmp/issues/600'})), 'message_ix_cap_comm': MarkDecorator(mark=Mark(name='xfail', args=(True,), kwargs={'reason': 'Requires MESSAGE cap-comm parameters, only available in message-ix >=3.12'})), 'needs_message_data': MarkDecorator(mark=Mark(name='xfail', args=(), kwargs={'condition': True, 'reason': 'Not yet migrated from message_data'})), 'no_data': <message_ix_models.util.pytest.MarkFactory object>, 'non_public_data': <message_ix_models.util.pytest.MarkFactory object>, 'sdmx_230': MarkDecorator(mark=Mark(name='xfail', args=(), kwargs={'condition': False, 'reason': 'https://github.com/khaeru/sdmx/issues/230', 'raises': <class 'sdmx.exceptions.XMLParseError'>})), 'transport_build_data': MarkDecorator(mark=Mark(name='usefixtures', args=('advance_test_data', 'iea_eweb_test_data', 'ssp_user_data'), kwargs={})), 'zenodo_timeout': MarkDecorator(mark=Mark(name='xfail', args=(), kwargs={'condition': False, 'reason': 'Connection to Zenodo times out', 'raises': <class 'requests.exceptions.ReadTimeout'>}))}

Common marks for tests. Use short, informative keys that are valid identifiers; these can then be applied using, for instance, @pytest.mark.ci_db_access.

message_ix_models.testing.NIE = MarkDecorator(mark=Mark(name='xfail', args=(), kwargs={'raises': <class 'NotImplementedError'>}))

Shorthand for marking a parametrized test case that is expected to fail because it is not implemented.

message_ix_models.testing.advance_test_data(monkeypatch) → None

Temporarily allow path_fallback() to find test data.

message_ix_models.testing.bare_res(request: FixtureRequest, context: Context, solved: bool = False) → Scenario

Return or create a Scenario containing the bare RES for use in testing.

The Scenario has a model name like “MESSAGEix-GLOBIOM [regions] Y[years]”, for instance “MESSAGEix-GLOBIOM R14 YB” (see bare.name()) and a scenario name either from request.node.name or “baseline” plus a random string of 5 characters.

This function should:

• only be called from within test code, i.e. in message_data.tests.

• be called once for each test function, so that each test receives a fresh copy of the RES scenario.

Parameters:

• request (FixtureRequest or None) – The pytest request fixture. If provided the pytest test node name is used for the scenario name of the returned Scenario.

• context (Context) – Passed to testing.bare_res().

• solved (bool, optional) – Return a solved Scenario.

Returns:

The scenario is a fresh clone, so can be modified freely without disturbing other tests.

Return type:

Scenario

message_ix_models.testing.export_test_data(context: Context)

Export a subset of data from a scenario, for use in tests.

The context settings export_nodes (default: “R11_AFR” and “R11_CPA”) and export_techs (default: “coal_ppl”) are used to filter the data exported. In addition, any item (set, parameter, variable, or equation) with a name matching EXPORT_OMIT or the context setting export_exclude is discarded.

The output is stored at data/tests/model name_scenario name_techs.xlsx in message_data.

See also

Prepare data for testing

message_ix_models.testing.iea_eei_user_data(pytestconfig, monkeypatch) → None

Temporarily allow IEA_EEI to find user data.

message_ix_models.testing.iea_eweb_test_data(monkeypatch, session_context) → None

Temporarily allow path_fallback() to find test data.

session_context is required in order to set the cache path in the test environment.

message_ix_models.testing.iea_eweb_user_data(pytestconfig, monkeypatch) → None

Temporarily allow IEA_EWEB to find user data.

message_ix_models.testing.mix_models_cli(session_context, tmp_env)

A CliRunner object that invokes the mix-models CLI.

NB this requires:

• The ixmp tmp_env() fixture. This sets IXMP_DATA to a temporary directory managed by pytest.

• The session_context() fixture. This (a) sets Config.local_data to a temporary directory within IXMP_DATA and (b) ensures changes to Context made by invoked commands do not reach other tests.

message_ix_models.testing.not_ci(reason=None, action='skip')

Mark a test as xfail or skipif if on CI infrastructure.

Checks the GITHUB_ACTIONS environment variable; returns a pytest mark.

message_ix_models.testing.pytest_addoption(parser)

Add two command-line options to pytest:

--local-cache

Use existing, local cache files in tests. This option can speed up tests that use the results of slow data loading/parsing. However, if cached values are not up to date with the current code, unexpected failure may occur.

--jvmargs

Additional arguments to give for the Java Virtual Machine used by ixmp’s JDBCBackend. Used by session_context().

message_ix_models.testing.pytest_collection_modifyitems(session: Session, config: Config, items: list[Item]) → None

Substitute or add marks from MARK.

message_ix_models.testing.pytest_configure(config: Config) → None

Register markers for keys in MARK.

message_ix_models.testing.pytest_report_header(config, start_path) → str

Add the ixmp configuration to the pytest report header.

message_ix_models.testing.pytest_sessionstart(session: Session) → None

Identify the path to be used for cached local data.

If pytest --local-cache was given, pick up the existing setting from the user’s environment. Otherwise, use a pytest-managed cache directory that persists across test sessions.

message_ix_models.testing.session_context(pytestconfig, tmp_env)

A Context connected to a temporary, in-memory database.

This Context is suitable for modifying and running test code that does not affect the user/developer’s filesystem and configured ixmp databases.

Uses the tmp_env() fixture from ixmp. This fixture also sets:

• Config.cache_path, depending on whether the --local-cache CLI option was given:

  • If not given: pytest’s standard cache directory.

  • If given: the /cache/ directory under the user’s “message local data” directory.

• the “message local data” config key to a temporary directory /data/ under the pytest tmp_path directory.

message_ix_models.testing.ssp_test_data(monkeypatch) → None

Temporarily allow path_fallback() to find test data.

message_ix_models.testing.ssp_user_data(pytestconfig, monkeypatch) → None

Temporarily allow SSPOriginal/SSPUpdate to find user data.

message_ix_models.testing.test_context(request, session_context)

A copy of session_context() scoped to one test function.

message_ix_models.testing.test_data_path() → Traversable

Path to the test data directory, message_ix_models/data/tests/.

message_ix_models.testing.unpack_snapshot_data(context: Context, snapshot_id: int)

Already-unpacked data for a snapshot.

This copies the .csv.gz files from message_ix_models/data/test/… to the directory where they would be unpacked by .model.snapshot._unpack. This causes the code to skip unpacking them, which can be very slow.

message_ix_models.testing.user_context(request)

Context which can access user’s configuration, e.g. platform names.

Checks

In-line checks for genno graphs.

class message_ix_models.testing.check.Check

Class representing a single check.

recurse_parameter_data(obj) → tuple[bool, str]

run() recursively on .ParameterData.

abstractmethod run(obj) → bool | tuple[bool, str]

Run the check on obj and return either True or False.

class message_ix_models.testing.check.CheckResult

Accumulator for the results of multiple checks.

This class’ __call__() method can be used as the result_cb argument to apply_checks(). After doing so, bool(check_result) will give the overall passage or failure of the check suite.

assert_all_passed() → None

Raise AssertionError if not all checks passed.

class message_ix_models.testing.check.ContainsDataForParameters(parameter_names: set[str])

parameter_names: set[str]

Collection of parameter names that should be present in the object.

run(obj)

Run the check on obj and return either True or False.

class message_ix_models.testing.check.Dump(base_path: Path)

Dump to a temporary path for inspection.

This always returns True.

recurse_parameter_data(obj) → tuple[bool, str]

run() recursively on .ParameterData.

run(obj, *, name: str | None = None)

Run the check on obj and return either True or False.

class message_ix_models.testing.check.HasCoords(coords: dict[str, Collection[Hashable]], inverse: bool = False)

Object has/omits certain coordinates.

run(obj)

Run the check on obj and return either True or False.

class message_ix_models.testing.check.HasUnits(units: str | dict | pint.registry.Quantity | pint.registry.Unit | None)

Quantity has the expected units.

run(obj)

Run the check on obj and return either True or False.

class message_ix_models.testing.check.InRange(min: float = -inf, max: float = inf, atol: float = 1e-08)

Values are in expected range.

atol: float = 1e-08

Absolute tolerance.

max: float = inf

Maximum value, if any.

min: float = -inf

Minimum value, if any.

run(obj)

Run the check on obj and return either True or False.

class message_ix_models.testing.check.Log(rows: int | None = 7)

Log contents.

This always returns True.

recurse_parameter_data(obj) → tuple[bool, str]

run() recursively on .ParameterData.

rows: int | None = 7

Number of rows to log.

run(obj)

Run the check on obj and return either True or False.

class message_ix_models.testing.check.NoDuplicates(setting: None = None, rows: int = 7)

No duplicate indices in parameter data.

run(obj)

Run the check on obj and return either True or False.

class message_ix_models.testing.check.NonNegative

No negative values.

Todo

Add args so the check can be above or below any threshold value.

run(obj)

Run the check on obj and return either True or False.

class message_ix_models.testing.check.NoneMissing(setting: None = None)

No missing values.

run(obj)

Run the check on obj and return either True or False.

class message_ix_models.testing.check.Size(setting: dict[str, int])

Quantity has expected size on certain dimensions.

run(obj)

Run the check on obj and return either True or False.

message_ix_models.testing.check.apply_checks(value: T, checks: Collection[Check], *, key: KeyLike, result_cb: Callable[[bool, str], None]) → T

Apply some checks to value.

Parameters:

• value – Anything.

• checks – 0 or more Check instances. Each is called on value.

• key – Used to log information about the checks performed.

• result_cb – Callback function that is passed the result of each Check call.

Returns:

value exactly as passed.

Return type:

Any

message_ix_models.testing.check.insert_checks(computer: genno.Computer, target: KeyLike, check_map: Mapping[KeyLike, Collection[Check]], check_common: Collection[Check]) → CheckResult

Insert some Checks into computer.

Parameters:

• computer

• target – A new key added to trigger all the tasks and checks.

• check_map – A mapping from existing keys (that must appear in computer) to collections of Check instances to be applied to those keys.

• check_common – A collection of common Check instances, to be applied to every key in check_map.

Returns:

after the checks are triggered (for instance, with computer.get(target)), this object will contain the overall check pass/fail result.

Return type:

CheckResult

message_ix_models.testing.check.verbose_check(verbosity: int, tmp_path: Path | None = None) → list[Check]

Return 0 or more checks that display the data to which they are applied.

These may be appended to collections passed as inputs to insert_checks().

Parameters:

verbosity (int) –

0. Don’t log anything

1. Log Log.rows values at the start/end of each quantity.

2. Log all data. This can produce large logs, e.g. more than 1 GiB of text for tests.model.transport.test_build.test_debug().

3. Dump all data to files in tmp_path.