Performance testing with PHPUnit

From Totara 19.1.0 it is possible to write tests specifically designed to measure performance. These tests are run independently from normal tests and are intended for use when you have an existing dataset to measure against.

1 Creating a performance test
- 1.1 Group tag
- 1.2 Assertion structure
2 Executing performance tests
- 2.1 PHPUnit configuration file
- 2.2 Running the tests
- 2.3 Output
3 Using an existing database

Creating a performance test

Group tag

Performance tests must be tagged with the group @performance. This ensures they are excluded from normal PHPUnit runs and only included when specifically requested. You can apply the tag at either the class or method level.

Tagging a whole class

If the @performance tag is added to a test class, all test methods in that class will be included in a performance test run:

/**
 * @group performance
 */
class test_performance extends testcase {
    public function test_1(): void {
        // This will run    
    }
    
    public function test_2(): void {
        // This will also run    
    }
}

Tagging individual methods

If the @performance tag is added only to specific methods, only those methods will be included:

class test_performance extends testcase {
    public function test_1(): void {
        // This will not run
    }
    
    /**
     * @group performance
     */
    public function test_2(): void {
        // This will run
    }
}

Assertion structure

Before each functionality that needs timing, the clock must be started by calling $this->blockTimingStart();. Immediately afterward, include any functional calls or code that should be timed, and finally end with a call to $this->assertBlockExecutesWithinTimeLimit('Assert expected', 3);.

You may call blockTimingStart and assertBlockExecutesWithinTimeLimit multiple times in one test case, as long as they are in pairs. Start one, stop one, start another, stop the other. Don’t try to start a second timing block without finishing the first.

It’s advisable to move any setup outside of the timing block so it isn’t included in the time measurements.

public function test_assert_block_executes_within_time_limit_pass() {
	$this->blockTimingStart();
	
	// Some code here we want to measure
	sleep(1);
	
	$this->assertBlockExecutesWithinTimeLimit('Assert expected', 3);
}

Executing performance tests

PHPUnit configuration file

To make performance tests available, your PHPUnit XML configuration file must include the new extension. When you run the built-in PHPUnit install script, the template PHPUnit file provided by Totara will be copied across, including the new extension. Otherwise, you will need to manually add the settings in.

<extensions>
    <bootstrap class="\core_phpunit\extension\totara_framework"/>
    <bootstrap class="\core_phpunit\extension\performance">
        <parameter name="console" value="true" />
        <!--<parameter name="console_colours" value="true" /> Uncomment to control whether the performance messages log with colour or not-->
        <!--<parameter name="log_file" value="performance_timings.csv" /> Uncomment to enable writing the performance results to file-->
    </bootstrap>
</extensions>

There are three available settings.

Setting Name	Default Value	Notes

Setting Name	Default Value	Notes
`console`	true	The results of the timed tests are printed to screen. Default.
`console_colours`	true	If false, no ansi codes will be used for colouring output. Defaults to true (has colours).
`log_file`	performance_timing.csv	Name of the csv file to write the results to (filename only, not a path). Defaults to disabled, uncomment to enable.

Be sure to remove the plain text settings descriptions if uncommenting, to avoid a PHPUnit warning.

Running the tests

Running PHPUnit with the group performance is enough to activate the performance extension and execute the new tests. You can also apply further filters as needed:

php phpunit.php run --group="performance" --filter="regular-filter"

Or if using docker:

unit --group performance --filter regular-filter

Output

CLI

The results will be presented in table form:

❯ php test/phpunit/phpunit.php run --test-suffix='test.php' --group=performance --filter=test_has_capability_in_any_context_with_timing
Totara 19.1.0-dev (Build: 20250327.00)
PHP: 8.2.28, pgsql: 15.12 (Debian 15.12-1.pgdg120+1), OS: Linux 5.15.167.4-microsoft-standard-WSL2 x86_64
Started Thu, 10 Apr 2025 09:06:06 +0800
PHPUnit 10.5.20 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.2.28
Configuration: test/phpunit/phpunit.xml


totara_core_accesslib_test::test_has_capability_in_any_context_with_timing
  Guest capability check for moodle/backup:backupsection in context level CONTEXT_SYSTEM........ ✓ 0.00005 (10.00)
  Guest capability check for moodle/backup:backupsection in context level CONTEXT_TENANT........ ✓ 0.00006 (10.00)
  Guest capability check for moodle/backup:backupsection in context level CONTEXT_USER.......... ✓ 0.00001 (10.00)
  Guest capability check for moodle/backup:backupsection in context level CONTEXT_COURSECAT..... ✓ 0.00001 (10.00)
  Guest capability check for moodle/backup:backupsection in context level CONTEXT_PROGRAM....... ✓ 0.00001 (10.00)
  Guest capability check for moodle/backup:backupsection in context level CONTEXT_COURSE........ ✓ 0.00001 (10.00)

Time: 00:02.793, Memory: 654.90 MB

OK (1 test, 6 assertions)

CSV

If the log_file option is enabled, the results will be written in raw form to a CSV file, and the file’s name will be printed to screen.

Performance log file can be found in "/path/to/phpunit/performance_timings.csv"

Time: 00:02.836, Memory: 654.90 MB

OK (1 test, 126 assertions)

The log file can be used to compare performance across previous runs to determine whether it is improving, degrading, or staying the same.

Example log file (note: the measured time is in nanoseconds):

test,block,time,expected
test_class_name::test_method_name,"Assert expected",2000168402,1

Using an existing database

With performance testing, it is useful to run tests against the same database each time, one pre-filled with testing data.

Using an existing database cannot be used with Paratest / parallel test runs.

This option cannot be used on a production site as it is destructive

In your config.php file, define the following setting:

$CFG->use_site_database_for_testing = true;
$CFG->phpunit_prefix = $CFG->prefix;

The active database configured will then be used as the base of the unit tests, instead of starting from an empty database.