Totara 13 notable technical changes

Overview

Totara 13 brings with it a number of significant changes, including but limited to three new products; Totara Perform, Totara Engage, and Totara Mobile. There is also:

• A new front end framework called Tui that is based upon Vue
• A new MVC and ORM backend componentry
• A brand new GraphQL service layer
• Alteration to the permission and capability system to introduce multitenancy
• Significant changes to the structure of the code base in order to facilitate the implementation of new technologies

Backwards compatibility

The changes in Totara 13 are nearly entirely backwards compatible. Most plugins will require no further modification than you would experience with our past major releases. and the same goes for customisations. As long as you are using API's provided by the system the only significant change you will need to account for is the code reorganisation changes discussed below, which will require you to adjust how you install the plugin and customisation code.

New technologies including GraphQL, Tui, MVC and ORM are used extensively within the new products. However they are opt-in technology. The existing output rendering, less, SCSS, and AMD JS is still in place and working exactly as it does in Totara 12.

Code reorganisation

Within Totara 12 web server executable code exists within the root directory.

Within Totara 13 we have introduced a new root directory server and all web server executable code has been moved within this root directory. The structure within the server directory in Totara 13 is identical to the root directory in Totara 12.

This change was made for a number of reasons, the most significant of which are:

• It enables us to completely separate Tui, our new front end framework, from the existing code base. Tui code exists in a root level directory called client.
• It enables us to move extension code, and code that is not required for a server in a production environment out of the web accessible path.  We moved several libraries and extensions, and will continue to move more of these in the future.
• It enabled us to separate and better abstract our automated unit and acceptance test environment. Our acceptance testing architecture is set to be greatly improved in the near future and this separation affords us more freedom in how we can improve it.
• It enables us to move the config.php file out of the web accessible directory. Config.php continues to be located in the root directory.

Changes for those deploying code

When deploying Totara sites, plugins, and customisations you will need to know that:

• The web server should now be configured to serve just the server directory.
• Plugin and customisations are now deployed within the server directory. Previously you put the auth_saml2 plugin into auth/saml2  whereas now you put it into server/auth/saml2.
• Your config.php file remains in the root level directory, don't move it into the server directory. The config.php file needs to be changed to no longer declare $CFG as global, and to remove the line including lib/setup.php. See config.example.php as an example.
• When running CLI scripts you can run them from the root directory php server/admin/cli/purge_caches.php or you can shift into the server directory and run them as you did previously cd server; php admin/cli/purge_caches.php.

For more information see our Totara 13 code reorganisation developer documentation.

Distribution of the code

There are no changes to how the code is distributed. You will be emailed links to release packages that can be downloaded from our subscription portal, and the code will be available through the current git repositories.

One of the key drivers in reorganising the code was to ensure a separation by directory of Totara Core and Totara Enterprise Extensions (Tui, Recommendations engine) whilst still facilitating a simple deployment via git.

Tui front end framework

Totara has a new Vue.js based front end framework that we call Tui. It enables us to create rich, modern, and reactive experiences within our product.

It is separated from the main body of the code base as discussed above in code reorganisation and is distributed under a proprietary license.

Tui is a front end framework, consisting of Vue.js components which embrace modern development practices when manipulating the DOM and working with data. Tui requires a build process during development that takes the Vue component files, which include JavaScript, SCSS, HTML and Language strings, and compiles them into bundles for functionality and style. These compiled files are referred to as build files.

A new server plugin called totara_tui facilitates the use of Tui on pages by outputting a minimal block of HTML markup that will be replaced by Tui pages and experiences when the page loads on the client. The totara_tui plugin also mediates the compiled build files when requested by the client.

Entire pages and experiences are created using Tui.

Tui then communicates with the server via WebAPI and more specifically using GraphQL queries and mutations.

Importantly, and as noted in the backwards compatibility section Tui is opt-in for developers. While the new products (Totara Engage and Totara Perform) use it extensively, if you are working with existing plugins you can continue to do exactly what you did before. As a bonus in that situation you can additionally tap into the WebAPI GraphQL queries and mutations if you choose, from your plugin, without needing to use Tui.

For more information on Vue.js we recommend you read the official Vue.js documentation. Our implementation of Vue.js within Tui follows their guidelines.

In addition to this we use the Apollo client and Vue Apollo libraries in order to extend Vue.js, and to enable easy integration and communication between Tui and Totara core via the WebAPI GraphQL services.

For detailed information on the architecture, principles and how to's of Tui please see our Tui front-end framework developer documentation.

WebAPI and our new GraphQL services

Totara 13 has a brand new service layer we call WebAPI. At its heart Web API uses the GraphQL query language. This enables rich description of the APIs and makes a range of developer tools immediately available, making it easier to work with the new service layer.

The rich description includes not just the queries and mutations used to get information from the system, and manipulate it, but also provides a description of the data that enables the explanation of object types, fields, interfaces, and others.

The WebAPI implementation of GraphQL follows the official specification that can be found at: https://graphql.org. For those interested we recommend first learning about GraphQL from the official site.

The following are the notable decisions made when in our Web API GraphQL implementation:

• Each component and plugin can introduce its own schema, at runtime these schema files from all components and plugins are built into a complete schema file for the site that is cached and mediated by Totara Core.
• We only support named queries (sometimes referred to as persistent queries). This means you can only call pre-defined queries and mutations. You cannot write you own query on the client and execute it.
• Queries can be batched, in order to support the bulk loading of information.
• Each consumer should have its own endpoint, and can have its own named queries and mutations. Totara core ships with two such endpoints, the AJAX endpoint that can be used from a browser, and a Mobile endpoint used by our new mobile app.
• The server side implementation of the resolvers marries up with the intended flexibility of the query language, each individual field is independently resolvable. 
• We use an open source PHP GraphQL library as the base of our implementation. Read more at https://github.com/webonyx/graphql-php.

You can find more on developing with Web API and GraphQL in Totara Core in our developer documentation.

Recommendations engine

Totara Engage comes with an implicit feedback hybrid recommendations engine built in Cython. The recommendations engine utilises collaborative filtering (i.e. interactions that users have with content) and content filtering (i.e. meta-data of the content itself) to make personalised recommendations to users and to suggest related content for each piece of Totara Engage content.

The recommendations engine itself does not talk directly with Totara Core, or with the database. Instead, data required by the recommendations engine is prepared by Totara Core and exported into a collection of CSV files (user data, content data and user-item interactions data). This data is read into and processed by the recommendations engine and the resulting recommendations are output in CSV format for automatic upload into the Totara database.

For more information on the recommendations engine, its architecture, and how it operates see our developer documentation on Machine learning and recommenders.

System requirements and instructions for working with the recommendation engine can be found in our Help documentation.

Mobile implementation

Totara Mobile was released at the same time as Totara 13, and is the first mobile app released by Totara Learning Solutions.

The purpose of our Totara Mobile app is not to reproduce the behaviour you can get through the browser (which is responsive) but instead to leverage the power of the mobile device. For the first release of Totara Mobile this includes push notifications and offline SCORM.

It has been developed in React Native, for both iOS and Android platforms, and internally uses Apollo Client. When communicating with Totara Core it communicates with a new server side plugin totara_tui, that presents a GraphQL endpoint specifically for the mobile app.

The code base for the mobile app will be made available to partners so that they can customise and further develop it to meet their subscribers needs.

For more information on Totara Mobile development please see its developer documentation on Totara Mobile.

Multitenancy changes

Multitenancy within Totara 13 brings the ability to separate, and if desired isolate users and content, based upon the tenant that a user is a member of.

Multitenancy is an advanced feature and must be enabled for the site if wanted. Once enabled the tenant management interfaces become available, as does an experimental setting to isolate tenants.

The following provides a summary of the most notable changes to the platform made to support this functionality.

Tenant context and its effect on the user context

With this change we have introduced a new context level CONTEXT_TENANT that must have the system context as its parent. One is created for each tenant.

A user is either a system user, or a tenant member. If they are a tenant member their context level will use the tenant context as its parent. Contexts for system users continue to use the system context as a parent.

Tenant category

Each tenant has a tenant category that is created when the tenant is created. The tenant category is always a top level category. It cannot be edited like other categories, and cannot be manually deleted.

Content within the category is considered to belong to the tenant. This includes subcategories, courses, activities, and blocks.

Permission resolution

The behaviour of has_capability and fiends has changed in order to support tenants and each context now tracks the tenant that it belongs to. When multitenancy is enabled a tenant member cannot pass capability checks if they are made against a context belonging to a different tenant. In other words a member of one tenant cannot access content or users belonging to a different tenant.

A tenant member can still access system content and users (contexts that do not belong to a tenant) providing isolation mode has not been turned on. When isolation mode has been turned on tenant members can only access content and users within their tenant.

Containers

Totara 13 introduces the concept of containers. Internally this reuses the course table, course context, and all associated components. Containers enable us to deliver functionality akin to a course, but with control over every aspect of the experience and API integration. They are used by Totara Perform in order to present performance activities, and by Totara Engage in order to facilitate workspace functionality.

Courses continue to function exactly as they did previously, as do course formats, activities and other componentry contained within courses.

For those upgrading to Totara 13 with plugins and customisations that select courses providing you use the methods and functions provided by the course API then it is likely no changes will be required to your plugins and customisations. If however you select directly from the course table you may now need to exclude container records.

The course table has a new field course.containertype which can be used to filter course records as follows:

SELECT c.id, c.fullname, c.shortname, ...
FROM {course} c
WHERE c.containertype = 'container_course'

Container functionality itself is facilitated by a series of hooks added to existing functionality that enable routes to be redirected and to provide a means to fundamentally hijack the behaviour a course would otherwise have.

For more information on the container implementation please see our container developer documentation.

Weka editor and JSON editor format

A new editor has arrived in Totara 13. Weka is based upon the ProseMirror rich text editor.

Unlike Atto and TinyMCE, the Weka editor does not generate HTML markup. Instead it builds a JSON structure that describes the structure of the document and its content. The JSON structure can then be rendered into HTML, or any other output document format. It is more restrictive than HTML, styling is not nearly as free, and only some of the HTML tags are available to use. It is more flexible than Markdown in that the JSON structure enables rich meta information that describes the content and structure of the document. It is also much easier to extend than Markdown and HTML.

The decision to integrate a new editor was driven by requirements to present a more tailored, functional experience for users creating content in Totara Engage, and by the need to support the creation of rich content that can be expressed easily in platforms that do not work in HTML, namely in our case our new mobile app.

Weka has been implemented as a standard editor within Totara Core, and using it simply requires using the standard editor API.

What should be noted is that a new format has been added to the platform in order to enable support and output of content described by JSON.

For more information on the JSON format and the Weka editor please see our Weka editor developer documentation.

MVC and ORM componentry

As part of our work on Totara Perform and Totara Engage we implemented two new components to aid us in developing consistently and securely. These two components are used extensively throughout our new product code, and are a fundamental part of their architecture.

They are opt in - you don't need to use them if you don't want. However they are now there for those who do.

If you are familiar with development in Laravel and Symphony frameworks you will likely already be somewhat familiar with the inner workings of these two.

For more information on the MVC and ORM components please see our developer documentation.

FAQs