| Note |
|---|
This is a draft for comment. This document is written for Totara developers, but shared publicly for transparency. |
The upgrade subsystem in Totara is largely unchanged from Moodle, please review https://moodledev.io/docs/5.0/guides/upgrade for basics of how it works.Plugin upgrade steps must be reviewed by a dev lead or architect due to previous upgrades which resulted in data loss or which performed poorly, causing the upgrade to overrun the time window allotted to it
| Tip |
|---|
A customer’s data is the most important and valuable part of their system. |
| Warning |
|---|
A process that causes lengthy downtime does not spark joy. |
Upgrades must be repeatable / idempotent
Totara currently enforces a linear upgrade path, meaning each successive plugin upgrade must be to a higher version.
However it is possible to for sites to get into situations where a given upgrade step is run more than once – consider the simple scenario where an upgrade step fails and is restarted before the savepoint is reachedlikely, because of the way that we manage versions across major releases, for the same upgrade step to have two or more different versions. For example, in Totara 18.6 an upgrade step that is part of a bug fix might have the version 2023112801. In Totara 19, that same step would have the version 2025012301. So a site upgrading from Totara 17, through 19, to 20 would execute the step twice.
For this reason, upgrade steps must be written so that they use conditional statements to avoid trying to make the same change twice. As a simple example, consider an upgrade that adds an index to a table:
...
At line 4, the conditional statement ensures that the index does not already exist, allowing this upgrade step to be run multiple times without error.
Upgrades must not use plugin code
...
, including local models and entities
As a rule we do not use model methods, helper classes, or other plugin-specific code during upgrades.
Plugin code always assumes that the database is at the latest savepoint, so it is dangerous to use when upgrading because the database the code is meant to operate on may be several versions out of date. Databases don’t like to load non-existent fields. Use core functions code only.
It is okay to use query builder in upgrade steps, but not ORM (entity, repository, or model classes).
...
Conceptually, this is similar to our deprecation guidelines: we allow old, unmaintained code to coexist with new implementations for a period of time before it is removed completely. Applying this rule in all situations may not be possible or desirable – there are legitimate reasons to change records and files during an upgrade.
Situations where this happens are rare, and we don’t have great examples in code already.
Please consult an architect before proceeding with this type of upgrade.
Upgrade logic should be testable and tested
...
As an upgrade step becomes more complex, however, it is important necessary to abstract the steps to one or more standalone functions in db/upgradelib.php that have unit test coverage proving that they work and handle edge cases appropriately.
...
When making a change to database structure or looping through records for any reason, consider what will happen if there are millions of records in the table. Take time to test upgrade with a scaled-up database to prove that it is efficient. If not? We may need to hold off until the next major version, or split the upgrade into batches that run in the background over time.
Long-running upgrades should be split into post-upgrade tasks
TBDcreate a manual upgrade experience.
Lengthy upgrades must be avoided
Some upgrades take a long time. Adding an index to a table with millions of records is not trivial, nor is backing up and transforming a large collection of objects.
This is a big problem at scale because larger sites can least afford the downtime required by Totara’s upgrade process.
We should go to great lengths to avoid lengthy upgrade steps in patch and minor releases, including implementation of temporary manual upgrade steps that can be triggered by an informed administrator after the upgrade is complete. These same upgrades can be automatic in the next major release.
The patterns for this are not well-established; we have used adhoc tasks to defer performance intensive upgrades in the past but this has do be done carefully and in small batches to prevent overloading the site on the next cron run.
A manual trigger (CLI script or in-product admin action) may also a possibility, with a check API check to detect that the manual upgrade hasn’t been actioned yet.
If you are concerned about upgrade performance, please raise the issue early your dev lead and/or an architect so we can collaborate on the best solution.