Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.

Code coverage is not the mark of good test coverage - in fact you can have fantastic code coverage, yet terrible test coverage.

However it can be a helpful to understand what is covered and what is not.

Generating a code coverage report is easy, albeit potentially time consuming.

Understanding code coverage in PHPUnit

PHPUnit supports generating code coverage reports and has good documentation on generating code coverage analysis.

To keep it simple there are really two command line arguments that you will need to know about:

HTML code coverage analysis

The following will generate an HTML code coverage analysis that you can explore in your browser - just use the wwwroot in your path.

Code Block
titleTotara 13 and above
test/phpunit/phpunit.php run --coverage-html --coverage-html /var/www/html/coverage_report

Code Block
titleTotara 12 and below
vendor/bin/phpunit --coverage-html /var/www/html/coverage_report

Clover code coverage analysis

The following generates a code coverage analysis report in the Clover XML format.

Code Block
titleTotara 13 and above
test/phpunit/phpunit.php run --coverage-clover /tmp/coverage_report.xml

Code Block
titleTotara 12 and below
vendor/bin/phpunit --coverage-clover /tmp/coverage_report.xml

The reason to mention this is PHPStorm (and probably all other popular IDEs) has the ability to import these files and reflect coverage within your IDE.

Whitelists and Totara

In order to produce a code coverage analysis you need to provide a whitelist configuration.

Failure to do so will result in the following error (aka warning):


Error: Incorrect whitelist config, no code coverage will be generated.

Defining a whitelist is easily done via a small modification to the phpunit.xml that is generated after you have initialised PHPUnit.

Make sure you init phpunit first!

Code Block
titleTotara 13 and above
php test/phpunit/phpunit.php init

Code Block
titleTotara 12 and below
php admin/tool/phpunit/cli/init.php

Add the following to your phpunit.xml to whitelist a group of files within Totara.

Code Block
        <whitelist processUncoveredFilesFromWhitelist="false" addUncoveredFilesFromWhitelist="false">
            <directory suffix=".php">totara/core/classes/local</directory>
            <directory suffix=".php">totara/core/classes/task</directory>

This should be added after the <php> tag.

Once the whitelist is in place you are ready to go.

If you wanted to run code coverage for the whole of Totara the following is the filter block we would recommend using:

Code Block
        <whitelist processUncoveredFilesFromWhitelist="false" addUncoveredFilesFromWhitelist="true">
            <directory suffix=".php">./admin</directory>
            <directory suffix=".php">auth</directory>
            <directory suffix=".php">availability</directory>
            <directory suffix=".php">backup</directory>
            <directory suffix=".php">badges</directory>
            <directory suffix=".php">blocks</directory>
            <directory suffix=".php">blog</directory>
            <directory suffix=".php">cache</directory>
            <directory suffix=".php">calendar</directory>
            <directory suffix=".php">cohort</directory>
            <directory suffix=".php">comment</directory>
            <directory suffix=".php">completion</directory>
            <directory suffix=".php">course</directory>
            <directory suffix=".php">dataformat</directory>
            <directory suffix=".php">elementlibrary</directory>
            <directory suffix=".php">enrol</directory>
            <directory suffix=".php">error</directory>
            <directory suffix=".php">files</directory>
            <directory suffix=".php">filter</directory>
            <directory suffix=".php">grade</directory>
            <directory suffix=".php">group</directory>
            <directory suffix=".php">iplookup</directory>
            <directory suffix=".php">lib</directory>
            <directory suffix=".php">local</directory>
            <directory suffix=".php">login</directory>
            <directory suffix=".php">media</directory>
            <directory suffix=".php">message</directory>
            <directory suffix=".php">mnet</directory>
            <directory suffix=".php">mod</directory>
            <directory suffix=".php">my</directory>
            <directory suffix=".php">notes</directory>
            <directory suffix=".php">plagiarism</directory>
            <directory suffix=".php">portfolio</directory>
            <directory suffix=".php">question</directory>
            <directory suffix=".php">rating</directory>
            <directory suffix=".php">report</directory>
            <directory suffix=".php">repository</directory>
            <directory suffix=".php">rss</directory>
            <directory suffix=".php">search</directory>
            <directory suffix=".php">tag</directory>
            <directory suffix=".php">totara</directory>
            <directory suffix=".php">user</directory>
            <directory suffix=".php">webservice</directory>

In general the more you whitelist and test the longer the execution will take.


Generating a code coverage analysis requires more than just PHP and PHPUnit.

The three most commonly talked about are:

  • PHPDBG is a SAPI module bundled with PHP since 5.6
  • XDebug is a PHP debugging extension that supports code coverage analysis
  • pcov is a PHP 7 code coverage extension

All allow us to generate a code coverage analysis, albeit with different results. Xdebug is commonly installed by developers here as not only does it facilitate code coverage but it integrates well with many IDEs and is a full-blown debugging extension with many great offerings. It is however much slower when generating a code coverage analysis, by an order of magnitude.

PHPDBG is very poorly documented, and provides some really interesting functionality if you are keen to wade through some deep waters. However it is reasonably simple to use and pretty quick. pcov requires a bit more to get running because we use PHPUnit 7, however it appears to perform best of all.

Here are some sample timings from running code coverage on a visibility refactor patch using PHPDBG, XDebug and pcov:

ToolNo coverageTest execution time w/ restricted whitelist
PHPDBG1m 46s2m 44s
XDebug14m 32s
XDebug with prepared filters file11m 25s
pcov1m 59s

Notice the difference in time whitelist has as well.

In addition preparing the actual coverage analysis report takes time, and each extension has its own timings.

They tend to be representative of the test execution time, increasing it by around 25%.

Code coverage with XDebug

  1. Make sure XDebug is installed and enabled.
    php -m | grep xdebug
    If that returns xdebug then it's ready to go, otherwise you will need to install and enable it.
  2. Ensure you have modified phpunit.xml to add your whitelist.

  3. Code Block
    titleTotara 13 and above
    php test/phpunit/phpunit.php run --filter=totara_core --coverage-clover /tmp/coverage.xml

    Code Block
    titleTotara 12 and below
    vendor/bin/phpunit --filter=totara_core --coverage-clover /tmp/coverage.xml

When run directly like this XDebug code coverage can take a very long time.

One method of speeding up the processing is to delegate whitelist functionality to XDebug. This vastly improves code coverage analysis.

To do this:

  1. Prepare a filter file for XDebug.

    Code Block
    titleTotara 13 and above
    php test/phpunit/phpunit.php run --dump-xdebug-filter /tmp/xdebug_filter.php

    Code Block
    titleTotara 12 and below
    vendor/bin/phpunit --dump-xdebug-filter /tmp/xdebug_filter.php

  2. Execute coverage with the filter file included.

    Code Block
    titleTotara 13 and above
    php test/phpunit/phpunit.php run --prepend /tmp/xdebug_filter.php --filter=totara_core --coverage-clover /tmp/coverage.xml

    Code Block
    titleTotara 12 and below
    vendor/bin/phpunit --prepend /tmp/xdebug_filter.php --filter=totara_core --coverage-clover /tmp/coverage.xml


If you are switching from pcov to XDebug you will need to remove your vendor directory and reinitialise.

Code coverage with PHPDBG

This is done on the command line:

Code Block
phpdbg -qrr -d memory_limit=-1 vendor/bin/phpunit --filter=totara_core --coverage-clover /tmp/coverage.xml

Code coverage with pcov

pcov is an alternative to XDebug and PHPDBG, with one big advantage being its speed.

It's designed for PHPUnit 8, however there is a composer package that helps by enabling compatibility with PHPUnit 7 (which we use).

To get it running:

  1. Disable XDebug if you have it enabled presently.
  2. pecl install pcov.
  3. If you are running coverage on Totara 12 or below you will need to install pcov/clobber. If testing 13 or above skip this step.

    Code Block
    composer require pcov/clobber
    vendor/bin/pcov clobber

  4. Ensure PHP has enough memory in php.ini. Give it heaps, as code coverage is memory intensive.

    Code Block

  5. Configure pcov (within php.ini):

    Code Block

  6. Run the tests:

    Code Block
    titleTotara 13 and above
    cd test/phpunit
    php -d`pwd` vendor/bin/phpunit --filter=totara_core --coverage-clover /tmp/coverage.xml

    Code Block
    titleTotara 12 and below
    cd /path/to/code
    php -d`pwd` vendor/bin/phpunit --filter=totara_core --coverage-clover /tmp/coverage.xml

The really nice thing about pcov is that it works with the code coverage integration within PHPStorm.

Alternative installation methods exist if you are using Fedora, Ubuntu or Debian. See pcov

PHPStorm and XDebug/pcov

PHPStorm has the ability to run code coverage anaylsis in product using XDebug or pcov. The analysis is automatically loaded into the platform and displayed both in the Coverage tool, and in-editor when looking at files. Their documentation on code coverage explains how to set it up.

A few pointers and tips to help however:

  • You still need to define that whitelist
  • As a product we support several different versions of PH - if you're using php-fpm and have them installed you can configure PHPStorm so that you can choose which version you run tests and coverage on
  • When looking at a testcase file you can click on the little arrows next to test case classes and test case methods and choose to run specific tests


PHPStorm does not have the ability to use PHPDBG for code coverage.

However you can import clover format code coverage analysis reports into PHPStorm.

To do so:

  1. Go to Tools > Show code coverage data.
  2. Click the + icon (top left).
  3. Locate your clover report and click Open.
  4. Make sure your file is checked.
  5. Click Show selected.

Practical coverage within Totara

titleSam Hemelryk

I find running pcov gives me what I am after. It executes quickly, and plays nicely with PHPStorm.

I use XDebug to its full potential very rarely, whereas I run tests frequently every day; as such keeping XDebug disabled and pcov enabled for regular development works well for me. When I need to run profiling (which is much less frequently than I run tests) I just swap out the PHP extensions.

In regards to regular test execution I prefer to use a limited whitelist, and filter my results using either a testcase pattern, or group injected for development purpose (I use my name in the group, to make it obvious I don't intend it to ever be integrated).

When executing a full run the only extension that I have found that works is pcov. XDebug without the prepared filter file doesn't make it to 1% for me, and PHPDBG doesn't make it to 2%. pcov on the other hand gets the whole way there, and in around 90 minutes.