Page Comparison

Table of Contents

This document describes what Behat is to Totara and was originally used for a workshop run at Totara HQ 26th March 2015 by Sam Hemelryk. This workshop was recorded and you can [https://totara.box.com/s/xs2aijbgadn6w1fxvtq8w2fwvm5qu2p3 download the screencast of this workshop].

Behat, acceptance testing, Totara and Moodle

Behat is the system Moodle uses to implement its own form of acceptance testing.
Behat testing is black box user simulation testing and allows us to test:
- Workflows through our software.
- Business logic in our software.
- Frontend code - JavaScript
At each stage of the process behat is watching for:
- PHP errors.
- Notices.
- Calls to debugging.
Its important to recognise that it doesn’t do the following:
- It can not check the page looks “ok”.
- Think like a user - the simulation is programmatic and without vision.
- It does not test usability or accessibility.
Behat differs in Totara from Moodle because of our default settings - there is a step for this mentioned later.

Why write behat test

They are a write and forget means QA, a properly covered feature is going to be tested every time anyone runs the behat tests, greatly increasing the chance of bringing regressions to the surface early on.
Low cost testing, they are easy to write. You are simulating the user experience, often this is as simple as documenting your testing.
They can facilitate faster, easier development when working on large change sets.
Can be written to prove workflow and business logic flaws reported by customers and be used to test for the resulting fix. Much like PHPUnit but for process.
Our software is huge - QA is hit or miss - more tests equals more confidence.

The bits you need to write and run behat

The test site with behat configuration variables in your config.php. It has to be available via a web server.
- You can use PHP’s built in web server (5.4 and up)
- Configure your existing web server to serve the same site on a different URL
The behat admin tool - located in admin/tool/behat. The cli script really is what we are most interested in.
- ‘‘admin/tool/behat/cli/init.php’’ initialises a new site
- ‘‘admin/tool/behat/cli/util.php’’ – drop" drops the current configuration and you’ll need to run init again.
- ‘‘admin/tool/behat/cli/util.php’’ --enable" triggers behat to re-find behat resources to ensure they are included.
Selenium - browser automation - user simulation
- [http://www.seleniumhq.org/download/ Selenium standalone server]
- Drivers Firefox by default but has extensions for Chrome, IE and Safari (perhaps Opera - but unsure)
- [https://sites.google.com/a/chromium.org/chromedriver/ Chrome driver]
The behat executable - vendor/bin/behat
- Is a composer dependency.
- You only need to have composer installed - when you call behat to initialise it will download what it needs.

Running behat

When init finishes it gives you the command you need to run for behat.
A full behat run of both Moodle and Totara tests can take more than 6 hours.
There are several ways to run just a specific test or set of tests:
- Tags: --tags @tagname will run just the tests with that tag.
- By name: --name “The name of a feature or scenario” will run that
When you run behat there are several other options you are going to encounter:
- --profile sets which behat profile will be used. Profile determine the browser and any defaults for the run.
- --format sets how you get displayed data, by default this is progress but there are reasons to change it.
- --out determines where output goes
- --stop-on-failure behat will exit if it hits a problem.

Directing output

If you are running behat overnight then you may way to direct output to both the screen and the file. This can be done in the following manner:

--format=html,progress --out=report.html,

This will create an HTML report and save it to report.txt, it will also show the default progress output on the screen at the same time.

Debugging behat

Can be tricky but there are a very tricks that will help you along the way:
Set ‘’$CFG->behat_faildump_path = ‘/tmp’;’’ When behat encounters a fail it will create a new directory there and add into it a screenshot and HTML source for the page that it was on when the failure happened. This way you can visually see what the browser was showing and inspect the source.
Run just the failing scenario (using --name) and set ‘’–format=pretty’’. This will cause behat to print out each line it runs in a colour and will identify exactly which step is failing. Useful if you have the same step more than once.
Add ‘‘And I pause’’ to the scenario right before the step that fails. This will cause behat to pause a this line until you press enter in the console. You can then browse to your behat site on the behat_wwwroot URL and explore the state of things (admin, admin).
Remember behat doesn’t have real eyes, it relies on the browser and it can see the source. You need to be specific when looking for something with an expectation. What you are looking for may exist in several places and it is important you look for it in an expected space.

Writing behat tests

There are several core components to know about when writing behat tests and its important to understand the terminology.

Features

This is commonly seen as the feature file ‘‘component/tests/behat/featurename.feature’’
Each feature file should test a specific feature, because of the nature of our software usually there is overlap, this is both acceptable and expected.
The first line should be the tags that are relevant to the component being covered by this feature.
- There should be a tag for Totara if its a Totara feature being tested: @totara
- There should be a tag for the component, or plugin type and plugin name: ‘’@totara @totara_core’’ or ‘’@mod @mod_facetoface’’
The second line starts with ‘‘Feature:’’ identifying this as a feature. There should then be a summary of the feature being written.
After this you must have a sentence or two describing what this feature is going to be testing and how.
Finally an empty line between this and the first scenario.
You can now run this feature with ‘’–name “demonstrate behat testing phase 1”’’

Scenario

A scenario tests a specific objective relating to the feature it is within.
One or more scenarios can exist for a feature.
Each scenario should test something explicit so that it is clear when a test fails what aspect of the feature is failing.
There is often a lot of duplication in scenarios within a feature - this is acceptable and expected.
Any tags relating to the scenario should appear on the line before. @javascript is the most commonly seeing one here.
The scenario starts with ‘‘Scenario:’’ and should have a summary of what this scenario is testing.
Immediately follow this are the steps that must be run to test the scenario.

Background

Background is a special kind of scenario that gets executed BEFORE every scenario in a feature file.
There can only be one background scenario in a feature file.
It has to appear BEFORE all scenarios.
It functions exactly like a scenario and is used to perform the same set up for several scenarios rather than duplicating the same setup for each.
You don’t have to use a background.

Scenario Outline

Scenario Outlines are a special kind of scenario that allows for variables and a table of example data.
When behat executes it runs the scenario once for each row of example data.
Everything about these is the same as a normal scenario.
See the sam4.feature example.

Steps

These are the individual steps that must be executed in order to complete the background or scenario. If you're using PhpStorm, it's worth installing the Behat plugin as this will help you find and use Behat steps.
Every step must start with one of four keywords:
- Given - this should be the first keyword used in the background / scenario and should only be used once. Steps within the ‘‘Given’’ block should bring the site to a state where we are ready to start testing. Typically generators are used to quickly set up a site here.
- When - like saying ‘’When I do the following‘’. The when section should contain the steps building to and completing an action prior to checking the outcome.
- Then - like saying ‘’Then I should see‘’. The Then section contains the assertions that check the results of the proceeding ‘‘When’’ action(s).
- And - used with any of the above, if needed, where more than one step is required. e.g. Given, And, And, When, And, And, Then, And... etc.
When and Then can be used in pairs several times within a scenario. e.g. Given, When, Then, When, Then.
Maintaining the Given, When, Then order helps keep our tests readable.

Comments

You can add comments to your behat tests using ‘’# some comment’’
Comments are not common at present due to the descriptive nature of the language.
You can add comments if you feel there is something additional to communicate.

Generators

Generators can be used to quickly generate required content (courses, users, etc) for a scenario.
Moodle provides a basic set of generators for content, these can be used with :
- And the following “users” exist:
- | username | firstname | lastname | email |
- | samh | Sam | Hemelryk | sam.hemelryk@example.com |
- Thanks to David and Petr you can also use Totara data generators:
- Given the following “programs” exist in “totara_program” plugin:
- | fullname | shortname |
- | Generator Program Tests | gentest |
Using a generator does not count as testing the creation of content.
If you write a generator you should also write a behat test to cover it and tag it with @totara_generator

Definitions

Definitions are the code that gets executed for a step. Each step has a definition.
Every component of plugin file can have a class that adds definitions:
- For a component this is ‘’/component/tests/behat/behat_component.php’’
- For a plugin this is ‘’/type/name/tests/behat/behat_type_name.php’’
Allows for you to handle complex situations with a single nice step - useful to yourself and others.

Tips and tricks

If you add generators, change generators, or behat can’t find your tests try running ‘‘php admin/tool/behat/cli/util.php --enable’’. This will trigger behat to re-parse the site for tests and the like. It shouldn’t normally be needed.
Look at existing feature files - chances are someone has already done what you are trying to do and a quick search can save you a lot of time. ‘‘There is also Home > Site administration > Development > Acceptance testing’’. Take a look there and see what it does, some find it useful - not me I find grep more useful but each to their own.
You will likely need to interact with Totara dialogues, there are steps to make this easier and tests that cover the steps for you to reference in totara_core.
Make sure you login with a user with the most relevant role for a particular test. So if you are testing the submission of an assignment, login as a learner. Logging in as an administrator might conceal permissions errors. In addition we rely on the user's roles in behat tests in some code which attempts to automatically classify language strings by role.
Always use the latest selenium standalone server and firefox - Firefox often breaks selenium and this can be a trouble to debug - using the latest will help.
Set up chromedriver and chrome in case, if you find something is failing and you don’t know why test it in another browser as perhaps thats the reason. These can be tricky and you may need to ask for help if you hit these when you first start out.
Submit each feature file for review when you finish it, don’t write all the features and then submit it. I’d rather have several on the go there than get one at the end and find a problem that is persistent through the whole thing.
In-house behat gotos: Petr, Brian, David and myself. Maria, Val and Simon are also up with the play largely.

Useful steps

The following is a list of useful steps, its a not a complete list, for that you can browse to Home > Site administration > Development > Acceptance testing.

I am on a totara site
I should see “’‘text’’”
I should not see “’‘text’’”
I should see “’‘text’’” in the “’‘text or selector’’” “’‘type’’” - type can be:
- activity
- block
- button
- css_element
- dialogue
- link
- link_or_button
- table
- table_row
- xpath_element
- totara_dialogue
- + more
“’‘text’’” “’‘type’’” should be visible
“’‘text’’” “’‘type’’” should not be visible
“’‘text’’” “’‘type’’” in the “’‘text or selector’’” “’‘type’’” should be visible
“’‘text’’” “’‘type’’” in the “’‘text or selector’’” “’‘type’’” should not be visible
The “’‘attribute’’” attribute of “’‘selector’’” “’‘type’’” should contain “’‘text’’”
“’‘text’’” “’‘type’’” should not exist
I click on “’‘text’’”
I click on “’‘text’’” in the “’‘text or selector’’” “’‘type’’”
I click on “’‘text’’” “’‘type’’” confirming the dialogue
I click on “’‘text’’” in the totara menu
I should see “’‘text’’” in the totara menu
I hover “’‘text’’” “’‘type’’”
I press “’‘button text’’”

A special mention on tables - there are generic steps for interacting with a table, however they are inaccurate as each table often has its own way of presenting data. Its usually required to write a definition that looks for the data that identifies the row you are interested in and finds what you are looking for in that row. XPath is the native selector for behat and is the preferred way to navigate the source structure.

Common Gotchas

I should see"’‘text’’" in the “’‘text or selector’’” “’‘type’’” :

Searches for the text within the given element, it doesn’t search the properties of the current element. This is encountered when searching text inside a property of an element such as with ‘‘I should see “Button text” in the “#some-id” “button”’’

There should only be one Given per Background, Scenario, and Scenario outline :

The given keyword is used to define what is essentially “setup” of the environment so that it is suitable for testing.

Assertions should be minimal as we should have upmost confidence that this will always work, it is still testing the process of course but it should be very basic setup and data generation only.

As such there should only be one Given per block. After the given (the first when) then the test should be focused on walking through process and making the necessary assertions to check the outcome.

Pairing when and then :

The when and then keywords should always be used in a pair, think of them as “When I do something Then I see something”

Having when…when is redundant, its like saying “When I do this and when I do that” really we need that then assertion to make sure the first action succeeded.

Same idea with then.

Parallel runs

With the merge of Moodle 2.9, it is now possible to run behat in multiple threads (decreasing the time it takes to do large runs). To do this you will need to set up the $CFG->behat_parallel_run variable in your config.php. This is a nested array with each element in the top level array needing the following keys (minimum):

behat_prefix. This is the database prefix for this instance and needs to be unique (I use ‘bht1_’, ‘bht2_’, ‘bht3_’, etc).
behat_wwwroot. This is where the site is hosted and needs to be unique (I use ’http://<ipnumber>/behat1/main', ’http://<ipnumber>/behat2/main', ’http://<ipnumber>/behat3/main', etc where behat<x> is a soft link to my sites directory - this means that they are all looking at the same code, although they think otherwise).
behat_dataroot. This is the equivalent of the site data directory and needs to be unique (on the filesystem).

Other entries that I am aware of (although I haven’t tested them in great detail) are:

dbtype. This would replace $CFG->dbtype.
dblibrary. This would replace $CFG->dblibrary.
dbhost. This would replace $CFG->dbhost.
dbname. This would replace $CFG->dbname.
dbuser. This would replace $CFG->dbuser.
dbpass. This would replace $CFG->dbpass.
wd_host. This would replace the wd_host as specified in the behat_config variable

Once these have been set up in your config.php file, run admin/tool/behat/cli/init.php -j=<x> where x is the number of threads you want to run (and has to be less than the number of elements in $CFG->behat_parallel_run). Once this is complete, you can run the tests (assuming Selenium is up and running) with admin/tool/behat/run.php. I’ve had 5 threads running quite happily (on an 8 core machine) averaging around 75% CPU utilisation, and still been able to do other work. I would not consider going above the number of cores you have available. The existing single threaded implementation is still available and I prefer to use it when working on small tests as I find the output easier to read.

Behat profiles

It is possible to set up different profiles to test different browsers. This is done by setting up the $CFG->behat_config array. An example element is:

$CFG->behat_config = array(
    'default' => array(
        'extensions' => array(
           'Behat\MinkExtension\Extension' => array(
                'selenium2' => array(
                    'browser' => 'chrome',
                    'wd_host' => 'http://localhost:4444/wd/hub',
                    'capabilities' => array(
                        'version' => '',
                        'platform' => 'LINUX'
                    )
               )
           )
       )
    ),
    'firefox' => array(
        'extensions' => array(
            'Behat\MinkExtension\Extension' => array(
               'selenium2' => array(
                   'browser' => 'firefox',
                   'wd_host' => 'http://localhost:4444/wd/hub',
                   'capabilities' => array(
                       'version' => '',
                       'platform' => 'LINUX'
                   )
               )
           )
       )
   )
);

The key variable here is browser This will need to be specified, and ensure that the driver is attached to your running selenium install (by using ‘-Dwebdriver.chrome.driver=path/to/chrome/driver’ and/or ‘-Dwebdriver.ie.driver=c:\path\to\IE\driver’). wd_host is where the selenium host is (for example if the apache server is a linux box and you want to test IE) and can be ignored.

Headless testing

Useful for a number of reasons but more of a necessity if you use Vagrant to develop on headless VMs or somewhere else without a display. The following assumes installing on an Ubuntu (14.04) Vagrant box but the same should work on your local Linux.

Note that I understand it’s also possible to run Chrome this way - although I’ve not yet tested.

Install requirements

If you’re using Vagrant / a VM for development this installation takes place there. Assuming that you’ve already downloaded the Selenium standalone Jar and done the necessary installation / config which Totara requires to run the Behat tests using ‘’‘Firefox’’’.

First up install Firefox and XVFB (this is a virtual frame-buffer which acts as a screen ‘placebo’ to stop the browser from exiting with an error).

<pre>% sudo apt-get install xvfb firefox</pre>

Running

It turns out that the <code>xvfb</code> package bundles in a very handy tool in the form of <code>xvfb-run</code> which does all the setting of environment variables for the virtual frame buffer so you don’t have to. All you need to do is pass it the command you want to run once that’s done. As the Selenium Jar starts Firefox for us all we need to do is:

<pre>% xvfb-run java -jar /path/to/your/selenium-server-standalone-XX.X.X.jar</pre> Once that’s running you should be able to run your Behat scenarios from the command-line as usual.

Behat Integration with PHPStorm

JetBrains provide some useful Behat support for PHPStorm / IntelliJ and while I’m generally a command-line kind of person it ‘‘really’’ is worth checking out. It includes the ability to create run configurations for Behat which show each scenario / step in the IDE with failure and success as a collapsible / filterable tree. On top of this it provides <code><ctrl>|<command> b</code> ‘go to definition’ support for Gherkin step definitions from <code>.feature</code> files which is also pretty killer IMHO). This is baked right in to PHPStorm - IntelliJ users this is available via a plugin as you will be accustomed.

Requirements

To configure this to work you’ll need to have already set up and selected a PHP interpreter for the project you’re setting up Behat for. This also works with a remote interpreter located on a VM (I use this way to run on my Vagrant development boxes from IntelliJ). Note that if you use a remote interpreter you ‘’‘must’’’ also set up SFTP deployment (<code>Settings > Build, Extension, Deployment</code>) as by default the IDE uses the path mappings and credentials configured there to access the remote box (the same process as when setting up PHPUnit to run on a remote box from the IDE). See the notes in the dedicated section on remote Behat testing for gotchas.

It is also assumes that you have installed and configured Totara for Behat already.

Settings

‘’‘Behat configuration for the interpreter’’’ <code>Languages & Frameworks > PHP > Behat</code> and select your interpreter from the list on the left (if its not shown add it). In the section ‘Behat library’ under ‘Path to Behat directory of phar file’ enter <code>/<path to totara dirroot>/vendor/bin/behat</code>. If you are using a remote interpreter this should be the path on the ‘’‘remote’’’ box. In the ‘Test runner’ section check the ‘Default configuration file’ checkbox and enter <code>/<path you assigned to $CFG->behat_dataroot>/behat/behat.yml</code>. Again remote interpreters use the path on the ‘’‘remote’’’ box. Click OK to Save your settings.

‘’‘Add a new run configuration’’’ In the run configurations dropdown in the top right of your IDE (next to the run / debug buttons) select ‘Edit configurations’. In the dialogue which follows click the plus to add a new run configuration And select ‘Behat’. Enter a name and in the section ‘Test Runner’ use the radio button to select how you want to define the scenarios to be run - this is useful if you need to isolate just a single feature you’re working on. If you want to run all Totara scenarios select ‘Defined in the configuration file’. Click OK to save.

Run the tests

Fire up your selenium Jar as usual and select the run configuration you just created in step 2. of ‘’‘Settings’’’ using the run configuration drop-down. Click the play button next to it and the IDE will open up the run panel and start testing. By default it omits passed tests from the heirarcy on the left - you can toggle this behaviour using the filter buttons above it.

Remote interpreter gotchas

‘’‘Vagrant users’’’ When configuring the SFTP deployment SSH settings ‘’‘do not’’’ use the IDE’s ‘Vagrant’ option as this uses vagrant commands to connect which are horribly slow. Instead choose the SSH option and manually configure for your Vagrant box (by default <code>IP 127.0.0.1</code> port <code>2222</code> password will work - if you want to use the private key it can be found in <code>/<path to your local Vagrant box dir>/.vagrant/machines/default/<provider>/private_key</code>.
‘’‘Missing helper’’’ IDE complains about a missing file in your remote <code>~/.phpstorm_helpers</code> directory and exits when trying to run. Looks like a bug when the directory was created [by PHPUnit] ‘‘before’’ the Behat plugin was installed. Delete that directory and re-run and the IDE will re-deploy its helpers.

Versions Compared

Old Version 3

New Version 4

Key