Testing processes

Owned by Former user (Deleted)
Last updated: Jan 11, 2024 by Fabian D.

All code submitted to Totara must pass through multiple stages of QA designed to catch any bugs prior to release. We are continually refining this process to improve our QA and testing processes based.

Pre-submission testing

We expect the developer to fully test their own code prior to submitting to us.
It is also expected that they have covered their code to a significant degree with automated tests. Both behat and unit tests, where applicable.

Review and testing prior to inclusion in the product

All changes to our product, whether they are to fix bugs, make improvements, or implement entirely new features must pass through a strict QA process.
This process has the following main stages:

  1. Automated testing.
    We install a real test site with the new code included and run a battery of automated tests to check for errors:
    • PHPUnit execution.
      Over 13'000 automated unit tests consisting of over 270'000 individual assertions.
    • Code syntax checker.
    • Product version checks.
    • Database schema checks.
    • Installation tests.
    • Upgrade tests from recent and past releases.
    • Database comparison install vs upgrade to detect and inconsistencies.
    • Running grunt and verifying builds are exactly correct.
  2. Manual code review.
    A developer must read the code and check for correctness, bugs and other issues.
  3. Manual code testing.
    A developer must install a site with the code and test that it works as expected.
    They also run relevant behat tests.
    This is done on their local machine, in their local environment. Although if platform issues are a possibility testing occurs in all supported systems.

Our code review platform is found here: https://git.totaralearning.com

Release testing

Each Totara point release is independently tested prior to release with all proposed code in place.
During this testing process a code freeze is in effect ensuring that nothing changes while testing occurs.

There are two main focus areas for release testing:

  1. Automated testing
    As well as the testing above, more thorough tests are run before releases:
    1. Complete run of the entire behat suite
    2. Complete unit test run on every major supported PHP version for the release as applicable on every database
    3. Installation on all supported databases against every major supported PHP version for the release as applicable
    4. Upgrade from all previous Totara versions to the current release
    5. Major upgrade steps (e.g. from Totara 12 → 13 → 18)
    6. Upgrade from Moodle
  2. Manual testing
    As well as the testing above, more thorough tests are run before releases:
    • Installation and operation testing on all supported databases.
    • Upgrade and operation testing on all supported databases.
    • Upgrade from the applicable Moodle release and basic operation testing.
    • Complete run of the entire behat suite
    • Complete unit test run on every major supported PHP version for the release as applicable.
    • Testing is completed on a variety of host operating systems, and while its not a complete matrix test we ensure we cover testing under all supported environments:
      • Both Apache and IIS
      • All supported databases
      • Various versions of PHP

The main focus here is on testing the areas modified during the release, ensuring upgrades and installs work correctly with existing data and looking out for any issues not detected by the automated tests.