Upgrade step guidelines

This is a draft for comment.

This document is written for Totara developers, but shared publicly for transparency.

The upgrade subsystem in Totara is largely unchanged from Moodle, please review Plugin Upgrades | Moodle Developer Resources for basics of how it works.

A customer’s data is the most important and valuable part of their system.

A process that causes lengthy downtime does not spark joy.

Upgrades must be repeatable / idempotent

Totara currently enforces a linear upgrade path, meaning each successive plugin upgrade must be to a higher version.

However it is likely, because of the way that we manage versions across major releases, for the same upgrade step to have two or more different versions. For example, in Totara 18.6 an upgrade step that is part of a bug fix might have the version 2023112801. In Totara 19, that same step would have the version 2025012301. So a site upgrading from Totara 17, through 19, to 20 would execute the step twice.

For this reason, upgrade steps must be written so that they use conditional statements to avoid trying to make the same change twice. As a simple example, consider an upgrade that adds an index to a table:

// Define index unique_issuer_username (unique) to be added to auth_oauth2_linked_login.
$table = new xmldb_table('auth_oauth2_linked_login');
$index = new xmldb_index('unique_issuer_username', XMLDB_INDEX_UNIQUE, array('issuerid', 'username'));
if (!$dbman->index_exists($table, $index)) {
    $dbman->add_index($table, $index);
}

At line 4, the conditional statement ensures that the index does not already exist, allowing this upgrade step to be run multiple times without error.

Upgrades must not use plugin code, including local models and entities

As a rule we do not use model methods, helper classes, or other plugin-specific code during upgrades.

Plugin code always assumes that the database is at the latest savepoint, so it is dangerous to use when upgrading because the database the code is meant to operate on may be several versions out of date. Databases don’t like to load non-existent fields. Use core code only.

It is okay to use query builder in upgrade steps, but not ORM (entity, repository, or model classes).

Upgrades must not delete or transform data or files in a way that is not reversible

It should go without saying that an upgrade step should not result in data loss, but in practice this has happened before due to bugs and unintended consequences from repeated upgrade steps.

The safest way to prevent data loss during upgrade is to not transform or delete data in-place, but rather to use a new column or table so that old and new data can coexist in the system at the same time. A later task or upgrade (in a future major version) can clean up old data once the migration is proven complete and an admin approves the action.

Conceptually, this is similar to our deprecation guidelines: we allow old, unmaintained code to coexist with new implementations for a period of time before it is removed completely.

Situations where this happens are rare, and we don’t have great examples in code already.

Upgrade logic should be testable and tested

For simple changes, such as database schema updates or capability cloning, the logic can be self-contained within the upgrade step in db/upgrade.php.

As an upgrade step becomes more complex, however, it is necessary to abstract the steps to one or more standalone functions in db/upgradelib.php that have unit test coverage proving that they work and handle edge cases appropriately.

There are numerous examples of this throughout the product.

Upgrades should be tested at scale

This is especially important for patch-level upgrades in stable branches, because these may be installed by automated processes during short maintenance / downtime windows.

If a site upgrade takes longer than the available maintenance window, the admin has to stop it and restore the previous version of the site from backup. This is lost time and money for our partners and it produces ill-will from customers.

When making a change to database structure or looping through records for any reason, consider what will happen if there are millions of records in the table. Take time to test upgrade with a scaled-up database to prove that it is efficient. If not? We may need to hold off until the next major version, or create a manual upgrade experience.

Lengthy upgrades must be avoided

Some upgrades take a long time. Adding an index to a table with millions of records is not trivial, nor is backing up and transforming a large collection of objects.

This is a big problem at scale because larger sites can least afford the downtime required by Totara’s upgrade process.

We should go to great lengths to avoid lengthy upgrade steps in patch and minor releases, including implementation of temporary manual upgrade steps that can be triggered by an informed administrator after the upgrade is complete. These same upgrades can be automatic in the next major release.

The patterns for this are not well-established; we have used adhoc tasks to defer performance intensive upgrades in the past but this has do be done carefully and in small batches to prevent overloading the site on the next cron run.

A manual trigger (CLI script or in-product admin action) may also a possibility, with a check API check to detect that the manual upgrade hasn’t been actioned yet.

If you are concerned about upgrade performance, please raise the issue early your dev lead and/or an architect so we can collaborate on the best solution.