This is a draft for comment.
The upgrade subsystem in Totara is largely unchanged from Moodle, please review https://moodledev.io/docs/5.0/guides/upgrade for basics of how it works.
Plugin upgrade steps must be reviewed by a dev lead or architect due to previous upgrades which resulted in data loss or which performed poorly, causing the upgrade to overrun the time window allotted to it.
Upgrades must be repeatable / idempotent
Totara currently enforces a linear upgrade path, meaning each successive plugin upgrade must be to a higher version. However it is possible to for sites to get into situations where a given upgrade step is run more than once – consider the simple scenario where an upgrade step fails and is restarted before the savepoint is reached.
For this reason, upgrade steps must be written so that they use conditional statements to avoid trying to make the same change twice. As a simple example, consider an upgrade that adds an index to a table:
// Define index unique_issuer_username (unique) to be added to auth_oauth2_linked_login.
$table = new xmldb_table('auth_oauth2_linked_login');
$index = new xmldb_index('unique_issuer_username', XMLDB_INDEX_UNIQUE, array('issuerid', 'username'));
if (!$dbman->index_exists($table, $index)) {
$dbman->add_index($table, $index);
}
At line 4, the conditional statement ensures that the index does not already exist, allowing this upgrade step to be run multiple times without error.
Upgrades must not use plugin code / model methods
As a rule we do not use model methods, helper classes, or other plugin-specific code during upgrades.
Plugin code always assumes that the database is at the latest savepoint, so it is dangerous to use when upgrading because the database may be several versions out of date. Use core functions only.
It is okay to use query builder in upgrade steps, but not ORM (entity, repository, or model classes).
Upgrades must not delete or transform data or files in a way that is not reversible
It should go without saying that an upgrade step should not result in data loss, but in practice this has happened before due to bugs and unintended consequences from repeated upgrade steps.
The safest way to prevent data loss during upgrade is to not transform or delete data in-place, but rather to use a new column or table so that old and new data can coexist in the system at the same time. A later task or upgrade (in a future major version) can clean up old data once the migration is proven complete and an admin approves the action.
Conceptually, this is similar to our deprecation guidelines: we allow old, unmaintained code to coexist with new implementations for a period of time before it is removed completely.
Applying this rule in all situations may not be possible or desirable – there are legitimate reasons to change records and files during an upgrade.
Situations where this happens are rare, and we don’t have great examples in code already.
Please consult an architect before proceeding with this type of upgrade.
Upgrade logic should be testable and tested
For simple changes, such as database schema updates or capability cloning, the logic can be self-contained within the upgrade step in db/upgrade.php.
As an upgrade step becomes more complex, however, it is important to abstract the steps to one or more standalone functions in db/upgradelib.php that have unit test coverage proving that they work and handle edge cases appropriately.
There are numerous examples of this throughout the product.
Upgrades should be tested at scale
This is especially important for patch-level upgrades in stable branches, because these may be installed by automated processes during short maintenance / downtime windows.
If a site upgrade takes longer than the available maintenance window, the admin has to stop it and restore the previous version of the site from backup. This is lost time and money for our partners and it produces ill-will from customers.
When making a change to database structure or looping through records for any reason, consider what will happen if there are millions of records in the table. Take time to test upgrade with a scaled-up database to prove that it is efficient. If not? We may need to hold off until the next major version, or split the upgrade into batches that run in the background over time.
Long-running upgrades should be split into post-upgrade tasks
TBD