Front end architecture, licensing and distribution changes

Overview

The release of Totara 13 brought with it a number of changes, which apply to subsequent releases.

Importantly we released the two enterprise extensions in Totara 13:

• Tui: Our new front end framework
• Recommendations engine: Machine learning-driven content recommendations within Totara Engage

Both extensions are released under the Totara Enterprise Extensions proprietary licence.

One of the driving factors in reorganising the code base in Totara 13 (and for future versions) was to accommodate these two extensions by providing separation between the solutions that make up the Totara product.

• The existing Totara Core code base, using the GPLv3 licence has been moved into a new top-level directory called server
• Tui code base has been put into a new top-level directory called client
• The recommendations engine has been put into a new top-level directory called extensions

This document looks at how these changes impact the Totara ecosystem.

Interaction between Totara Core and Totara Enterprise Extensions

All interaction between Totara Core and Totara Enterprise Extensions must occur through channels that would be usable by other means. This includes the likes of web services, executable programs, and data interchange via the import and export of data in common formats such as CSV.

The use of intimate APIs, such as having an extension script include a library file from Totara Core, is not permitted. Nor is the use of licensed resources such as images and fonts.

Tui front end framework

Tui achieves this separation by interacting with Totara Core through the Web API, GraphQL services it presently. Tui is utilised in the page by writing minimal HTML markup that signals Tui is wanted for a page, and then on the client Tui which has also been loaded picks up and replaces the minimal markup with the Tui component for the page.

The following two lists explain what is permitted with regards to licensing now when running in the client, and what is not. 

Tui can:

• Replace the initial markup on a page.
• Enhance basic HTML tags that are in the markup (where the HTML tag is from the HTML specification). For example a textarea can be enhanced with an editor.
• Interact with user data embedded into the page. This is data that comes from the server, but belongs to the site. As an example if resolved query data is embedded into the page as JavaScript then Tui can pick this up. Additionally as an example while we opted to introduce an alternative to M.cfg for Tui, we could have in theory used M.cfg in Tui.
• Enhance and manipulate any markup that it introduces on the page.
• Load its own resources into the page (css, javascript, images etc)

Tui must not:

• Enhance HTML markup generated by Totara Core that provides more than basic markup e.g. it cannot convert list structures into a treeview (it is also not designed to do this).
• Interact with the GPL licensed JavaScript running on the client. This includes all AMD modules, and global API introduced by JavaScript static.
• Make use of library files from within Totara Core such as by including or requiring them. This includes PHP and JavaScript.
• Make use of resources coming from Totara Core, this includes images, icons and fonts.
• Know anything about the server directory structure.

Totara Core can:

• Read files from a predicted location in the client directory in order to mediate them to the client.
• Perform mutations on mediated files providing there is nothing specific to Totara Core being performed. Translating SCSS to CSS is a permitted mutation as the SCSS is entirely unrelated to Totara. Ideally we want it to perform as little processing as possible.
• Read JSON assets produced by the build process, providing those assets are not specific to Totara Core, but describe generally Tui attributes and structure. The use of dependency files generated by Totara Core is one such example of this.

Totara Core can not:

• Interact with the Tui JavaScript when running on the client.
• Make use of library files from within Tui such as by including or requiring them.
• Make use of assets in any capacity other than mediating them back to the client when requested.
• Know any intimate details of the Tui bundles or components, any information it needs must be read for common descriptive sources such as JSON files that cannot be Totara Core specific.

Additionally the following points explain the integration between Tui and Totara Core:

• A totara_tui plugin within Totara Core facilitates the integration.
• Totara Core pages wishing to use Tui send an empty tag span to the client which has data attributes that Tui will look for and replace with Tui content.
• All data required by Tui is requested via GraphQL queries from the client, or injected into the minimal markup as props. This includes user data, and site data such as translations.
• JavaScript required by Tui is mediated by the totara_tui plugin. It does not know anything about the Tui bundles, but it uses a basic pattern to look for the build files in a specific location, and delivers them to the client when requested.
• CSS required by Tui is mediated by the totara_tui plugin. Because Tui uses SCSS totara_tui will translate the SCSS to CSS before returning it to the client. As above, it uses a basic pattern to find the file to return to the client.
• Dependencies between Tui bundles are written to a JSON file in each bundles directory by the Tui build process. The totara_tui mediation layer knows to look at the list of dependencies when requiring resources and to additionally require any listed bundles.
• GraphQL schema locations are record in a JSON file in the top level directory, Tui uses this to find GraphQL files used by components as it does not know anything about the server directory structure due to its isolation.

For more information on the Tui implementation please see our Tui organisation and implementation overview document.

Recommendations engine

The recommendations engine is a little more clear cut. It serves its purpose nearly entirely independent of all other code. It requires site and user data with which it calculate recommendations, which is exported by Totara Core server to CSV. Once recommendations have been made they are exported by the recommendations engine as CSV and then imported into Totara Core. The same CSV import/export functionality could be used by other recommendations engines.

Licensing, and distributing plugins

As noted above all developers who work with Totara Enterprise Extension code must adhere to the same strict level of isolation noted above so as to not violate either our proprietary licence or the GPL.

When developing plugins for Totara that make use of Tui you will need need to separate your front end code from your server side code, and use WebAPI GraphQL services as we have with Totara Core.

If you are keeping your plugin in a single repository then separating the two libraries by top level directories is the path forwards. Otherwise multiple repositories can be used.

When deploying you will need to ensure the compiled Tui front end code goes into the client/component/{component_name}/build directory, the source code if provided will need to go into the client/component/{component_name}/src directory, and the server side component into the appropriate directory within the server directory.

How third parties license their plugins will also need to change. The server side code can remain with the licence you have been using previously, providing there are no existing conflicts with the GPL. Your Tui client side code however must use a licence that is compatible with our proprietary licence.

Our recommendation if you have been using GPL in the past is to switch to the MIT licence, another open source licence with less restriction.

Using third-party libraries open source libraries, source code, and assets

Third-party libraries and source code can still be used within the product, however the licence of the third-party code must be compatible with the system in which you are using it.

If it is being used within Totara Core then the previous rules applied, the licence of the source code you are using must be compatible with the GPLv3 licence.

If it is being used within Totara Enterprise Extensions, then the licence must be compatible with our proprietary licence.