MESSAGEix-Transport

Warning

MESSAGEix-Transport is under development.

  • The code and data documented on these pages were recently migrated from message_data. The text may still contain references to the old location.

  • For details, see the project board.

message_ix_models.model.transport adds a technology-rich representation of transport to models in the MESSAGEix-GLOBIOM family. The resulting “model variant” is variously referred to as:

  • MESSAGEix-Transport,

  • “MESSAGEix-GLOBIOM ‘T’” or, with other variants like buildings and material, “MESSAGEix-GLOBIOM BMT”, or

  • “MESSAGEix-XX-Transport” where built on a single-country base model (again, in the MESSAGEix-GLOBIOM family) named like “MESSAGEix-XX”.

MESSAGEix-Transport extends the formulation described by McCollum et al. (2017) [55] for the older, MESSAGE V framework that predated MESSAGEix. Some inherited information about the older model is collected at MESSAGE V-Transport.

Information about MESSAGEix-Transport, its inputs, configuration, implementation, and output, are organized according to this diagram:

https://raw.githubusercontent.com/khaeru/doc/main/image/data-stages.svg

On this page:

Implementation

Summary

The code:

  • Operates on a base model with a particular structure, the standard MESSAGEix-GLOBIOM representation of transport. See Structure of base scenarios.

  • Builds MESSAGEix-Transport on the base model:

    • Use the apply_spec() pattern, with a Spec that identifies:

      • Required set elements in the base model, for instance commodity elements representing the fuels used by MESSAGEix-Transport technologies.

      • Set elements to be removed, for instance the technology elements for base model/aggregate transport technologies. (Removing these elements also removes all parameter data indexed by these elements.)

      • Set elements to be added. These are generated dynamically based on configuration setting from files and code; see Configuration.

    • Use a genno.Computer (from build.get_computer()) to:

      • Read data from Input data files and other sources,

      • Prepares the parametrization of MESSAGEix-Transport through an extensive set of computations, and

      • Add these data to the target Scenario.

  • Solves the Scenario.

  • Provides message_ix_models.report extensions to report or post-process the model solution data and prepare detailed transport outputs in various formats (see Reporting).

Details

On other page(s): Disutility costs.

  • For light-duty vehicle technologies annotated with historical-only: True, parameter data for bound_new_capacity_up are created with a value of 0.0. These prevent new capacity of these technologies from being created during the model horizon, but allow pre-horizon installed capacity (represented by historical_new_capacity) to continue to be operated within its technical lifetime. (PR #441)

Usage

Automated workflow

transport.workflow.generate() returns a Workflow instance. This can be invoked or modified by other code, or through the command-line:

$ mix-models transport run --help
Usage: mix-models transport run [OPTIONS] TARGET

  Run the MESSAGEix-Transport workflow up to step TARGET.

  Unless --go is given, the workflow is only displayed. --from is interpreted
  as a regular expression.

Options:
  --future TEXT                   Transport futures scenario.
  --fast                          Skip removing data for removed set elements.
  --model-extra TEXT              Model name suffix.
  --scenario-extra TEXT           Scenario name suffix.
  --key TEXT                      Key to report.
  --dest TEXT                     Destination URL for created scenario(s).
  --dry-run                       Only show what would be done.
  --nodes [ADVANCE|B210-R11|ISR|R11|R12|R14|R17|R20|R32|RCP|ZMB]
                                  Code list to use for 'node' dimension.
  --quiet                         Show less or no output.
  --go                            Actually run the workflow.
  --from TEXT                     Truncate workflow at matching step(s).
  --help                          Show this message and exit.

This workflow can be used in the following ways:

Run the entire workflow

This is the method used by the transport.yaml GitHub Actions workflow (see Testing and validation) on a daily schedule, and thus always known to work for the combinations of arguments used in that workflow.

For the same reason, it should not be necessary to run this manually; simply trigger the workflow.

mix-models \
  --platform=ixmp-dev \
  transport run \
  --base=auto --nodes=R12 --model-extra="ci nightly" \
  --from="" \
  "SSP2 policy reported" \
  --go

The options result in the following behaviour:

  • --platform=ixmp-dev: store MESSAGEix-Transport scenarios on the ixmp platform named “ixmp-dev”.

  • --base=auto: identify the base scenario URLs using base_scenario_url() / the file base-scenario-url.json, according to other config settings.

  • --model-extra="ci nightly": append the string ” ci nightly” to the model name of any created Scenario. This avoids accidentally producing new versions of ‘production’ (model name, scenario name) combinations.

  • --from="": start from the very first step in the workflow—load the identified base scenario—and perform all subsequent workflow steps, up to and including…

  • "SSP2 policy reported": the target step to reach.

  • --go: Actually execute the workflow. Omit this to show a preview/dry-run.

Debug the build for a single scenario

This emulates the build step, but does not use any ‘real’ base scenario; only a bare RES scenario that contains structure but no data.

mix-models \
  transport run \
  --nodes=R12 \
  "SSP2 debug build" \
  --go

This produces output files (.csv, .pdf) for debugging the build. The output for this particular command is in the directory local-data-path/transport/debug-ICONICS:SSP(2024).2-R12-B/. With a different target or --nodes option, the directory name will differ accordingly.

Debug the build for all scenarios

This performs the above debug build step for all SSPs, and then runs debug_multi() to generate plots that compare the contents of the debug .csv files for all 5 SSPs. The plots are output to the directory local-data-path/transport/.

mix-models \
  transport run \
  --nodes=R12 \
  "debug build" \
  --go
Run only reporting
export URL="ixmp://ixmp-dev/MESSAGEix-GLOBIOM 1.1-T-R12 ci nightly/SSP_2024.2 baseline#154"
mix-models \
  --url="$URL" \
  transport run \
  --nodes=R12 --model-extra="ci nightly" \
  --from="SSP2 solved" \
  "SSP2 reported" \
  --key="in:nl-t-ya-c:transport+units" \
  --go

This:

  • First sets a shell environment variable ($URL) that points to an (existing model name, scenario name, version)—for instance, one produced by the GitHub Actions workflow.

  • --url="$URL": runs the workflow with this as a base URL.

  • --from="SSP2 solved": starts after the step SSP2 solved. In other words, assume that all steps up to and including this step have already run, and have produced a scenario with the given $URL.

  • "SSP2 reported": run the reporting step.

  • --key="in:nl-t-ya-c:transport+units": compute only this reporting key; display; and exit. This can be changed to any other key, or omitted entirely for the default action, which is to produce all output files (IAMC-structured output in .csv and .xlsx formats, plots in .pdf format, and files for base MESSAGEix-GLOBIOM calibration in .csv format) and also store the IAMC-structured output as time series data with the scenario.

Manual

This subsection contains an older, manual, step-by-step workflow.

Preliminaries. Check the list of pre-requisite knowledge for working with message_ix_models.

Note

One pre-requisite is basic familiarity with using a shell/command line.

Specifically: export BASE="…", seen below, is a built-in command of the Bash shell (Linux or macOS) to set an environment variable. $BASE refers to this variable. In the Windows Command Prompt, use set BASE="…" to set and %BASE% to reference. Variables with values containing spaces must be quoted when referencing, as in the example commands below.

To avoid using environment variables altogether, insert the URL directly in the command, for instance:

$ mix-models --url="ixmp://mt/Bare RES/baseline" res create-bare

Choose a platform. This example uses a platform named mt. If not already configured on your system, create the configuration for the platform to be used; something like:

$ ixmp platform add mt jdbc hsqldb /path/to/db

Note

See the ixmp documentation for how to use the ixmp command to add or edit configuration for specific platforms and databases.

Identify the base scenario. One option is to create the ‘bare’ RES; the following is equivalent to calling bare.create_res():

$ export BASE="ixmp://mt/Bare RES/baseline"
$ mix-models --url="$BASE" res create-bare

For other possibilities, see Base scenarios.

Build the model. The following is equivalent to cloning BASE to URL, and then calling transport.build.main() on the scenario stored at URL:

$ export URL=ixmp://mt/MESSAGEix-Transport/baseline
$ mix-models --url="$BASE" transport build --dest="$URL"

Solve the model. The following is equivalent to calling message_ix.Scenario.solve():

$ message-ix --url="$URL" solve

Report the results. The -m model.transport option indicates that additional reporting calculations from model.transport.report should be added to the base reporting configuration for MESSAGEix-GLOBIOM:

$ mix-models --url="$URL" report -m model.transport "transport plots"

Utilities

There are several other sub-commands of mix-models transport available. Use --help overall or for a particular command to learn more.

gen-activity

Generate projected activity data without building a full scenario:

$ mix-models transport gen-demand --ssp-update=2

This command produces:

  • Files in MESSAGE_LOCAL_DATA/transport/gen-activity/scenario (see local data), where scenario reflects the command options:

    • pdt.csv, pdt-cap.csv: projected activity data.

    • demand-exo.pdf, demand-exo-cap.pdf: plots of the same.

  • MESSAGE_LOCAL_DATA/transport/gen-activity/compare-[pdt,pdt-cap].pdf: plots that contrast the data output by the current command invocation and any others in other sub-directories of …/gen-demand, that is, from previous invocations.

Thus, to prepare compare-pdt.pdf containing projections for multiple scenarios, invoke the command repeatedly, for instance:

$ mix-models transport gen-demand --ssp=2
$ mix-models transport gen-demand --ssp-update=1
$ mix-models transport gen-demand --ssp-update=2
$ mix-models transport gen-demand --ssp-update=3
$ mix-models transport gen-demand --ssp-update=4
$ mix-models transport gen-demand --ssp-update=5
refresh

Deprecated since version 2023-11-21: Use ixmp platform copy from the ixmp Command-line interface instead.

Scenarios

Base scenarios

The following existing scenarios are targets for the MESSAGEix-Transport code to operate on:

ixmp://ixmp-dev/MESSAGEix-GLOBIOM 1.1-R12/baseline_DEFAULT#21

nodes=R12, years=B.

Current development target as of 2023-12-13, and used in the transport.yaml CI workflow.

ixmp://ene-ixmp/CD_Links_SSP2_v2/baseline

regions=R11, years=A.

ixmp://ixmp-dev/ENGAGE_SSP2_v4.1.7/baseline#3

regions=R11, years=B.

ixmp://ixmp-dev/ENGAGE_SSP2_v4.1.7_ar5_gwp100/EN_NPi2020_1000_emif_new#5

regions=R11, years=B. This scenario has a “hybrid” or “dual” implementation of emissions accounting: it includes both:

  • the ‘old’ structure, in which emissions are accounted using message_ix relation_activity and related parameter, but emission_factor is unused/empty, and

  • a ‘new’ structure in which the emission_factor parameter is actually used.

ixmp://ixmp-dev/MESSAGEix-GLOBIOM_R12_CHN/baseline#17

years=B. Based on ENGAGE, without MACRO calibration. This scenario has a non-standard node code list: there are 12 nodes, as in the R12 list, but their IDs are e.g. R11_CHN, R11_RCPA, etc.

ixmp://ixmp-dev/MESSAGEix-GLOBIOM_R12_CHN/baseline_macro#3

regions=R12, years=B. Includes MACRO calibration

ixmp://ixmp-dev/MESSAGEix-Materials/NoPolicy_GLOBIOM_R12_s#1

regions=R12, years=B. Includes MESSAGEix-Materials detail.

ixmp://ixmp-dev/MESSAGEix-Materials/NoPolicy_2305#?

regions=R12, years=B. Includes MESSAGEix-Materials detail.

Structure of base scenarios

The MESSAGEix-GLOBIOM RES (e.g. model.bare or message_data.model.create) contains a representation of transport with lower resolution. Some documentation is in the base-model documentation (message_doc; see also iiasa/message-ix-models#107). This section gives additional details missing there, which are relevant to MESSAGEix-Transport.

  • Demand (commodity=transport, level=useful) is denoted in energy units, i.e. GWa.

  • Technologies producing this output; all at m=M1, except where noted. This is the same set as in MESSAGE V, i.e. in MESSAGE V, the aggregate transport representation is inactive but still present.

    • coal_trp

    • foil_trp

    • loil_trp

    • gas_trp

    • elec_trp

    • meth_ic_trp

    • eth_ic_trp

    • meth_fc_trp

    • eth_fc_trp

    • h2_fc_trp

    • back_trp — at modes M1, M2, M3

    • Trans_1

    • Trans_2

    • Trans_3

    • Trans_4

    • Trans_5

  • historical_activity and ref_activity indicates which of these technologies were active in the model base year. - Some, e.g. back_trp, are not (zero values) - Disaggregated technologies must match these totals.

SSP scenarios (2024 update)

The code responds to transport.Config.ssp (equivalently, context["transport"].ssp) by computing and applying a variety of factors. These are defined in factor.COMMON. Each Factor is built-up from 1 or more layers that each represent a different assumption or choice. When MESSAGEix-Transport is built, these assumptions are quantified and combined into a single, concrete Quantity object with at least the dimensions \((n, y)\), sometimes \((n, y, t)\). These specific values are applied in (usually) multiplicative or other ways to other values produced by the model build process.

Here we explain one example:

LMH = Map(
    "setting", L=Constant(0.8, "n y"), M=Constant(1.0, "n y"), H=Constant(1.2, "n y")
)
OMIT_2020 = Omit(y=[2020])

...

    "ldv load factor": Factor(
        [
            LMH,
            OMIT_2020,
            ScenarioSetting.of_enum(SSP_2024, "1=H 2=M 3=M 4=L 5=L", default="M"),
        ]
    ),

This example has three layers. The first two are reused from variables, because they also appear in other factors.

  • The first layer sets constant values (the same for every label on the dimensions \((n, y)\)) for three different labels on a ‘setting’ dimension. These labels are merely str: their meaning or interpretation must be clearly explained in code comments or by linked documentation. Otherwise they may be ambiguous (“‘H’igh energy intensity” means the same thing as “‘L’ow efficiency”: what precisely is measured by the quantity to which the factor should apply?) The ‘M’ setting has a value of 1.0. Because this particular factor is used multiplicatively, in effect choosing the ‘M’ setting results in value * 1.0 = value: that is, no change or a no-op.

  • The second layer indicates to omit or mask values for \(y \in \{2020\}\). In effect, this results in values of 1.0 for this period, with the same no-op effect described above.

  • The last layer is a “scenario setting”. In effect, this transforms a ‘scenario’ identifier from an enumeration (something like SSP_2024["2"]) into one of the ‘setting’ labels from the first layer. This allows the same setting to be specified for multiple scenarios: in this example, SSP2 and SSP3 have the same setting. If the constant values in the first layer are changed, the values applied for SSP2 and SSP3 will both change.

    The string "1=H …" must contain every member of (in this case) SSP_2024; every setting label that appears must be provided by the previous layers of the factor. (To be clear: this does not mean that all defined settings must be used; it is valid to use, for instance, "1=M 2=M 3=M 4=M 5=M".)

To change the assumptions that are modeled via any particular factor:

  • Add or remove layers.

  • Change the Constant values that appear.

  • Change the ScenarioSetting mapping.

  • Adjust where and how the factor is applied in the transport build process. This option requires more technical skill.

Testing and validation

MESSAGEix-Transport includes a GitHub Actions workflow defined in the file .github/workflows.transport.yaml. A list of past runs of this workflow appears here. This workflow:

  • Runs on a schedule trigger, daily.

  • Runs on a GitHub Actions self-hosted runner. This is hosted on a server within the IIASA network, so is able to access the ixmp-dev ixmp Oracle database.

  • Uses, as its starting point, the first scenario listed under Base scenarios, above.

  • Runs multiple jobs; currently, one job for each SSP_2024. Each job takes about 30 minutes, and the jobs run in sequence, so the entire workflow takes 2.5 hours to run.

  • Produces an artifact: aside from the logs, certain files generated during the run are combined in a ZIP archive and stored by GitHub. This artifact contains, inter alia:

    • One directory per job.

    • In each directory, files transport.csv and transport.xlsx containing MESSAGEix-Transport reporting output.

    • In each directory, files demand.csv and bound_activity_lo,up.csv containing data suitable for parametrizing the base MESSAGEix-GLOBIOM model.

  • May be triggered manually. Use the “Run workflow” button and choose a branch; the code and data on this branch will be the ones used to build, solve, and report MESSAGEix-Transport.

  • May be altered to aid with development:

    • Run on every commit on a pull request branch:

      # Uncomment these lines for debugging, but leave them commented on 'main'/'dev'
pull_request:
branches: [ main, dev ]

    • Run only some steps; not the full build–solve–report sequence. For instance:

      env:
  # Starting point of the workflow.
  # Use this value to build from a certain scenario:
  # base: --url="ixmp://ixmp-dev/MESSAGEix-GLOBIOM 1.1-R12/baseline_DEFAULT#21"
  # Use this value to allow the workflow to determine model & scenario names
  # and versions:
  base: --platform="ixmp-dev"

  # Set this to a particular step to truncate the workflow
  from-step: ".* solved"

    Per the comments, do not merge such changes to dev or main. Instead, make them with a commit message like “TEMPORARY Adjust ‘transport’ CI workflow for PR”; then later git rebase -i and drop the temporary commit.

Code reference

The entire module and its contents are documented recursively:

message_ix_models.model.transport

MESSAGEix-Transport.

Other documents