message_ix_models coexists with the private repository/package currently named
The latter is the location for code related to new research that has not yet been completed and published, data that must remain closed-source permanently, etc.
All other code will be migrated from
Code and data for individual projects will be moved from
message_ix_modelsat a suitable point during the process of publication. (This point may vary from project to project.)
message_datamay be renamed.
Using both packages together
This section gives some practices and tips for using the two packages together.
- Always import via
message_ix_models.project.project_nameis imported, and
This module does not actually exist in
Then the code will instead file the respective modules
Even when using code that currently or temporarily lives in
message_data, access it like this:
# Code in message_data/model/mymodelvariant.py from message_ix_models.model import mymodelvariant mymodelvariant.build(...)
This code is future-proof: it will not need adjustment if/when “mymodelvariant” is eventually moved from
- Use the mix-models command-line interface (CLI)
All CLI commands and subcommands defined in
message_dataare also made available through the
message_ix_modelsCLI, the executable mix-models.
Use this program in documentation examples and in scripts. In a similar manner to the point above, these documents and scripts will remain correct if/when code is moved.
- Don’t import from
The open-source code should not depend on any private code. If this appears necessary, the code in
message_datacan probably be moved to
The former have stricter quality standards and are more transparent, which is better for reproducibility.
At some points, similar code may appear in both packages as it is being migrated. In such cases, always import and use the code in
message_ix_models, making any adjustments that are necessary.
Migrating individual modules
This section describes a general process for migrating (sub)modules of
Using this process preserves the commit and development history of code and data.
This is useful for future development, and can contain important methodological and research insights that would be lost with a simple copy.
Uses the scripts in the
Has been tested on Ubuntu Linux. It uses bash shell scripts that may not run in the same way on other systems.
May need modification depending on the target code.
Install git lfs.
Install git-filter-repo. This is a Python package, so e.g.:
$ pip install git-filter-repo
Read through all the steps first before starting.
Create a temporary directory:
$ mkdir tmp $ cd tmp
This directory will hold new clones of both repositories. We use new clones to avoid interacting with local settings, uncommitted (ignored) files, or history from other work, and so we can safely delete an incomplete migration and start again.
Copy all files from the
This can be done from an existing clone, or from GitHub.
Use the comments in the file as a guide to the necessary changes.
message_databranch from which the migrated code and data will be taken.
Fetches all available Git LFS objects associated with any commit in
message_data. These are needed as the history is replayed in the next step.
This may download up to 6 GB of data from GitHub, so it can be a slow step.
Perform the initial transfer (
Among other things, this script:
Copies the Git LFS objects from the
message_dataclone in step (4), so that they are available for use in the target location.
Uses git-filter-repo (docs) to:
Move code. For example, all commits pertaining to a file like
message_data/model/foo/bar.pyare preserved, except they now appear to describe changes to
Move data. Data is moved from the unpackaged, private, top-level
message_data, to the packageable
message_ix_models/data/directory. There are further considerations; see Data, metadata, and configuration and below.
Discard everything else relating to
message_data, especially other code and data that are not going to be migrated, according to your settings in step (3).
Partly clean up commit messages that do not match the code style, e.g. by ensuring they start with a capital letter.
Writes two files,
duplicates.txt, to be used in the next step.
Prepare for git rebase.
Make a copy of the file
rebase-todo.txt, and open the copy. This file contains a list of commands for the rebase. You can edit this list before using it in step (7); if needed, restore the list by making a fresh copy of the original.
To help with this,
duplicates.txtcontains a list of identical commit messages that appear more than once in the history. These commits may—not necessarily—be indication of a non-linear history. This can occur when branches with similar commit names but different contents are merged together; despite our best efforts, this sometimes happens on
Some changes you can make to your copy of
Remove lines for duplicated commits, per
duplicates.txt. This avoids commanding git to apply the same changes more than once, which can lead to conflicts. You could:
Keep only the first of two or more occurrences of duplicate commits.
Keep only the last of two or more occurrences of duplicate commits.
Use any other strategy to minimize conflicts.
Remove lines for merge commits. These are ignored by git rebase and git filter-repo, but you may need to manually skip them if you do not remove them at this step.
Add blank lines and comments to help yourself read the history.
Perform the rebase. Run:
$ git rebase --interactive --empty=drop main
In the editor that opens, delete everything.
Paste in the contents of
rebase-todo.txtfrom step (6).
Save the file and exit.
The interactive rebase begins.
Resolve any conflicts that arise in the usual way. After resolving, perhaps run:
$ git add --update && git rebase --continue
If you see a message like the following:
error: commit 47db89c0128e6edf19ebb9ffbcea1d5da4d25176 is a merge but no -m option was given. hint: Could not execute the todo command hint: hint: pick 47db89c0128e6edf19ebb9ffbcea1d5da4d25176 Merge pull request #123 from iiasa/foo/bar hint: hint: It has been rescheduled; To edit the command before continuing, please hint: edit the todo list first: hint: hint: git rebase --edit-todo hint: git rebase --continue
…follow the instructions:
Give git rebase --edit-todo.
Delete the line/command related to the merge commit.
Save and exit.
Give git rebase --continue.
If many conflicts occur, you may run:
$ git rebase --abort
Then, return to step (6) to adjust the list of commands, considering the history and apparent conflicts.
$ git push --set-upstream=origin migrate-example
…and open a pull request.
This can be initially a “draft” state, until you complete step (9). The pull request is partly to help you diagnose whether the above steps produced a reasonable result. The branch can also be inspected by others, e.g. to compare it to the source in
This can be done directly on the branch from (8). However, a better option to create a secondary branch from the HEAD of (8), named like
migrate-example-tidy, and make clean-up commits to this branch. Create a second pull request to merge this manual clean-up branch into the branch from (8). (This way, if the semi-automated process needs to be repeated, it can be re-pushed to
migrate-example, and then the manual clean-up branch can be rebased on the newly updated
migrate-examplebranch, with little disturbance.)
Push further changes to the clean-up branch to:
Modify imports and references.
For example, when migrating
message_data.model.foo, statements like:
from message_data.model.foo.bar import baz
…must be modified to:
from message_ix_models.model.foo.bar import baz
Similar changes must be made to intersphinx references in the documentation.
Adjust data handling.
For example, usage of
private_data_path()to locate data files must be modified to
package_data_path()if the data files were moved during the migration. Tests can help to ensure that these changes are effective.
Address CI checks. For example:
Add tests, or exclude files from test coverage.
Lint files, or exclude files from linting.
It is important to avoid scope creep: do not try to include large modifications, improvements, or refactoring of code in this step. This will greatly increase the complexity of the task and make it harder to complete. Instead, do these things either after or before migrating the code.
Invite review of your PR.
Merge the clean-up branch from (9) into (8), and then (8) into the target branch (currently
At any time:
Run ./reset.sh from your temporary directory to delete the clone of
message_ix_modelsand all other changes from step (4). Then you can restart from step (4).
Some follow-up actions that may or should take place after the migration is complete:
Discuss with the
message_ix_modelsmaintainers about releasing a new version of the package, so that the code is available in a released version.
Open (an) additional issue(s) or PR(s) to record or immediately address missing items—for example, documentation, tests, or small enhancements for reusability—that were identified during the migration.
Open a PR to remove the migrated code from
message_data. This is important because future development should target the code in its new home in
message_ix_models; other projects, workflows, and colleagues should be discouraged to depend on the old code in
message_data, where it may not receive updates.
The simplest way to do this is to delete the code entirely and adjust any other code that imports it to import from the new location in
message_ix_models. For temporary compatibility, it is also possible to use
git and git filter-repo are both flexible programs with plenty of power and flexibility. The above is one suggested way of using them to achieve a clean, history-preserving migration, but there are alternate options.
The description of PR #86 describes an alternate process.
PRs that used this process include: