Totara 13 code reorganisation

Owned by Former user (Deleted)
Last updated: Aug 25, 2020Version comment

Overview

The code base in Totara 13 has been reorganised. This document looks at the changes that were made, and why, as well as noting how we expect this will impact our stakeholders, and what needs to be considered with regard to this when upgrading.

It is important to note that while the code has been reorganised, and by consequence will affect many aspects of your hosting, deployment and development processes we have still tried to keep everything as backwards compatible as possible. For many of the operations your undertake, simply shifting the current working directory from the top level directory to the server directory will return things to normal for you. In addition to this we do not intend to reorganise the code like this again in the future.

What changed

In Totara 13 we reorganised the structure of our git repository, and by consequence the distributed deployment packages.
These changes were only made in Totara 13, the stable release code structure remains exactly as it was.

Prior to Totara 13 the top level directory contained the entirety of the product code, and web servers were pointed directly at the top level directory.

Directory changes

As of Totara 13 there is a new top level directory structure, as follows:

  • server
    The server directory contains the Totara Core product code. When configuring your web server it should use the server directory as the web root. No other directories should be web accessible.
    Within the server directory the structure is exactly as it was in Totara 12 at the top level.
  • client
    The client directory contains our Tui, our new front end framework. Within it, it has two directories, the first is tooling which contains the build process for Tui as well as all of the tooling surrounding the production of Tui such as webpack configuration, linting etc. The second directory is component which contains all of the components Tui currently provides.
    This directory does not need to be web accessible. Required files are mediated through the totara_tui plugin (located at server/totara/tui)
    For more information on our new front end framework please see our Tui organisation and implementation overview.
  • libraries
    The libraries directory contains third party libraries used by Totara. Initially a couple of libraries used by Totara Core have been moved here. In the future we will add all new libraries here, and continue to move libraries out to here when updated.
    Wrapper methods for these libraries continue to exist within the server directory, preferably as auto-loaded classes. Libraries can be included via composer, or can be directly committed to core. There is now also the concept of required and optional libraries.
    Existing libraries will all be considered required. Newly added libraries may now be optional. The first use of an optional library will be with cloud file support, a feature that has also arrived in Totara 13.
  • extensions
    The extensions directory contains Totara product extensions. Initially it will contain just the recommendations engine used by Totara Engage, but further extensions may be introduced in the future.
    These are external applications, that should not be web accessible, and that have their own running environments.
  • test
    The test directory contains phpunit and behat, now separated from one another. In the future we expect to be reviewing our automated acceptance test framework and likely will be changing the libraries that facilitate it. Likewise we may introduce further automated testing suites in the future.
  • dev
    The dev directory contains scripts and tools used by Totara developers. It will be excluded from the packaged distributions in the future. Initially the scripts located at totara/core/dev will be moved into this top level directory.

The config.php file

The other notable change is that config.php, originally located in the top level directory, will continue to be located in the top level directory. Do NOT put it into the server directory.
This will shift the config.php file outside of the web accessible directory which was one of our goals as it is more secure. A server/config.php file exists and has been committed to the repository. This file includes your config.php from the top level directory, and sets up the runtime environment. As all entry pages include config.php relative to the old web root (now server) this keeps everything backwards compatible without requiring any changes.

Other minor changes

The following minor changes have also been made.

  • /config.example.php replaces server/config-dist.php which has now been removed
  • /composer.json now initialises for our build environment only, it no longer initialises phpunit and behat
  • The original JS/CSS build process remains in the server directory, and must be run from there

Why were these changes made

There were two driving factors that led us to reorganise our code base.

The first was a desire to move towards an open core model. Tui, our new front end framework is not being made available under the GPL license. It is instead being made available under our proprietary license. The recommendations engine also is made available under our proprietary license.
This required us to fully separate the Totara Core and Totara Enterprise Extensions within the source code, and to ensure that any interaction and dependence between them occurs through publicly accessible API, outside of runtime code for the products, that could be consumed by any product, application or solution.

For this reason Totara Core which is made available under the GPL license has moved into its own top level directory "server". Likewise Tui our front end framework, and our recommendations engine are located in their own directories, client, and extensions respectively. For more information on the licensing changes to Totara 13 please see Front end architecture, licensing and distribution changes.

The second driving factor was a desire to shift the web accessible directory from the top level directory. This in turn will enable us over time to move code outside of the web root if it is not required in a production environment, or if it is not required to be web accessible.

As we shift and grow the technology base of the product we introduce more and more dependencies on third party libraries and code, nearly all of which do not need to be web accessible. Likewise for our own development activities, we generate ever more code that does not need to be web accessible, and code that supports our product rather than just driving it.

This split will help improve the security of the product by reducing the surface of the code exposed to the web, and allows us to better organise our code without compromising that surface.

In the future we expect to further explore and develop this new structure. The development team is excited by the possibilities that this brings. Rest assured though we will not be performing any further such reorganisations, nor introducing breaking changes any time in the near future.

How this impacts our stakeholders

This impacts all parties who are familiar with Totara 12 and below. The most significant impact is the newly added server directory, and that the Totara Core product code has moved from the top level directory into the server directory.

For those deploying and administering Totara sites:

  • When deploying Totara 13 and above the web server configuration will need to change so that the document root of the web server is the "server" directory within the codebase. An environment check has been added to the product to warn for web roots that do not point to the server directory.
  • When executing scripts such as admin/cli/cron.php you will need to change the path to the script by prepending "server", such that you would now execute server/admin/cli/cron.php.
  • When installing plugins within Totara 13, rather than working from the top level directory, you will need to be working from within the server directory. The plugin structure within the server directory now, is identical the plugin structure in the root directory prior to Totara 13.
  • Config.php still gets placed into the top level directory. Do NOT put it into the server directory. You also need to remove the global $CFG line, and the line that requires lib/setup.php. See config.example.php as an example.

For those developing in Totara the following applies in addition to the above:

  • PHPUnit and Behat execution has now changed. The original tools and scripts for both are still where they previously were and can be used. However if you were manually initialising PHPUnit or Behat you will need to change how you do this. See our PHPUnit developer documentation, and Behat developer documentation for more information.
  • Grunt, used to build the old front end JavaScript and CSS has moved into the server directory. It can be run as previous by changing your working directory to the server directory and then running the same commands as you used to. For details see the documentation on Working With LESS in Themes.

Finally, the introduction of code released under our proprietary license will have an impact on those who are developing plugins and customisations for distribution. More information with regard to this can be at Front end architecture, licensing and distribution changes.