Catalogue 1

Internally, the 'Catalogue' is referred to as the 'catalog' or 'totara_catalog'.

How to extend the Grid catalog

The grid catalog has been designed in a way which allows Totara devs and third parties to extend it easily. There are several levels of enhancement and customisation available.

Providers can be created, which add a new type of learning item to the catalog.
Dataholders can be added, which allow pieces of data to be displayed in the Item or Details of a learning item, included in the full text search (FTS) index or be sorted on.
Dataformatters can be added, which use data retrieved by dataholders and can reformat the data as required.
Filters can be defined, which allow users to restrict the items displayed in the catalog.
Features can be defined, which allow learning items to be marked as featured in relation to different data sources.
Observers can be added, which ensure the catalog index is kept up to date when related data is changed.
Merge select objects can be defined, which can allow use of different front end filter elements, and connect them with the related filters.
Datasearch filter classes can be defined, which allow different types of sql filter statements to be defined.

Each of the features above are described in detail below.

Providers

The core of the grid catalog is built around providers. Each provider defines a type of learning item that can appear in the catalog. Providers define collections of dataholders, filters and features (and, frequently, dataformatters and observers will also be associated with a particular provider).

To define a new provider, simply implement a new subclass of totara_catalog\provider, and fill in all required methods. These include (but are not limited to):

get_name, which is displayed to admins and users, as a label for the type of learning items.
get_object_type, which must be unique across the site, and is used to uniquely identify the provider and associated data.
get_data_holder_config, which defines what dataholders are used for sorting and FTS.
get_all_objects_sql, which should return an array of all the learning items that should be in the catalog.
get_manage_link and get_details_link, which define what links the user should see in the catalog in relation to a given learning item.
get_buttons and get_create_buttons, which define what buttons the user should see in relation to the learning type.
can_see, which can be used to prevent the current user from seeing a learning item which is in the catalog.

All items which might appear in the catalog for some user should be returned by get_all_objects_sql. Visibility restrictions are then handled by can_see. Any learning item which should never be displayed in the catalog should be excluded in get_all_objects_sql. For example, all courses except the site course are included by the course provider.

Dataholders

A provider will have a collection of dataholders, which define how the data stored in the database is retrieved and displayed. A provider implements a number of dataholder_factorys, which return one or more dataholders. A dataholder is defined as:

A key which is unique within the provider.
A name, which is displayed to admins and users.
An array of formatters, which define how the data should be formatted for a given purpose.
An array of datajoins, which define sql joins, which allow the data to be retrieved from the database.
An array of dataparams, which define sql params which are needed by the datajoins.
A category, which defines grouping of the dataholder in drop-down menus in the admin interface.

Code reuse can be achieved by having dataholder factories call code in other areas. For example, courses, programs and certs all have custom field dataholders, which they create by calling customfield_dataholder_factory::get_dataholders.

Dataformatters

A dataformatter defines how a given set of data should be manipulated in order to prepare it for use. For example, the user_date formatter takes a database date in integer format and converts it to a user-readable string. Each dataformatter can have a bespoke constructor, which should define the list of required data, and can record any other information which might be needed during formatting. For example, the ordered_list constructor takes a database field which indicates what database field contains the list, a source delimiter which indicates what delimiter was used within the data, and a result delimiter which defines how the resulting ordered list should be formatted.

Dataformatters also define and restrict how they can be used, in get_suitable_types. For example, the duration dataformatter can be used to display durations in text placeholders, but cannot be used by dataholders to put data into the FTS index (because it doesn't make sense to do FTS on a duration).

The get_formatted_value function may perform arbitrarily complex operations to format the raw data into the appropriate output format. This can include database operations, or calling other functions. The performance of this function should be carefully considered. While dataformatters are usually used to format data for display to a user, which limits the number of calls to a few dozen at most (the 'Load more' setting goes up to 60), if the dataformatter is used to populate the FTS index then it will be called for ever learning item in the index during a full index refresh.

A set of generic dataformatters are defined by the catalog, and include classes such as text, textarea, static_text (which use no database data, it just displays the text which was supplied in the constructor) and fts (which makes text suitable for inclusion in the FTS index). Each provider can define a collection of dataformatters which are specific to their data. For example, courses have a 'format' dataformatter, which calls get_string using the 'format' text key from the course table.

Filters

Filters are used by users to restrict the items displayed in the catalog. For example, a user can select 'Courses' in the 'Learning type' filter to see only courses. There are some core catalog filters, but most filters are defined by, and specific to, providers.

Like dataholders, providers implement a number of filter_factorys, which return one or more filters. A filter is defined as:

A key. If more than one filter has the same key (within or across providers), then an attempt will be made to automatically merge the filters.
A region in the catalog interface where the filter can be displayed, either the top-left Browse drop-down, or the left Panel.
A datafilter, which defines how an sql statement can be constructed to filter the catalog results to those specified by the form element that the user has 'activated'. These are explained in detail later.
A selector, subclassed from merge_select, which defines the form element which can used by the user to control the results.
A category, which defines grouping of the filter in drop-down menus in the admin interface.

If two filters have the same key then the catalog will attempt to merge them. The purpose of merging is to allow developers to provider a single filter which applies to more the one provider. For example, if courses have a 'menu' custom field called 'Colour' with options 'red, green, blue', and programs has the same custom field containing 'red, orange, yellow', then the catalog will present a single filter containing the options from both courses and programs combined, 'blue, green, orange, red, yellow'. Any combination of colours can be selected, with the query putting 'OR' between each option, e.g. selecting 'blue' and 'red' will result in all courses containing 'blue' (there are no programs with 'blue') as well as all courses and programs with 'red'.

Merging happens by first verifying that all properties of the filters are compatible (the regions must match, and datafilters and selectors must be mergeable). Then it merges the filters by merging the datafilters and selectors. More details about merging are explained in the sections for datafilters and merge_select objects.

Like dataholders, code reuse can be achieved by calling another function which returns a set of filters configured for the specific provider.

Features

Features are used to define a subset of learning items which should be marked with the 'Featured' label, and for sorting. Like filters, there are some core catalog features, but most features are defined by, and specific to, providers.

Features are functionally very similar to filters. A feature is defined as:

A key. If more than one feature has the same key (within or across providers), then an attempt will be made to automatically merge the features.
A name, which is displayed to admins and users.
A datafilter, which defines how an sql statement can be constructed to filter the catalog results to those specified by the admin in the catalog settings. These are explained in detail later.
A category, which defines grouping of the feature in drop-down menus in the admin interface.

Instead of a selector, features directly define a list of related options, by calling the add_options_loader function during feature setup. In the catalog settings, when an admin selects a feature, they are presented with a list of all the possible options for that feature. All learning items which match the selected feature option will have the label 'Featured' and will be sorted first when sorting by Featured.

Merging happens in basically the same way as for filters. When two features with matching keys are merged, options from both features are merged. It is important to consider what keys and values are used when merging options for features - are they compatible?

Observers

Catalog observers are used to make sure that the catalog index is up-to-date. Standard observers can also be used to schedule either provider_active_task (refreshes the specified provider's data) or refresh_catalog_adhoc (refreshes all providers' data). These adhoc tasks can also be scheduled within catalog observers. Catalog observers are defined by specifying the event that they observer, adding an entry in db/events.php, and calling register_for_update or register_for_delete from init_change_objects. The 'register' functions can be called multiple times from within one observer (if more than one catalog item is affected by the event), or not at all (if the event related to an item that doesn't need adding, updating or removing from the index).

Observers should be defined when there are new events which can result in learning items being added or removed from the catalog index, or when a data change event can result in information in the FTS index needing to be updated. For example, if a new dataholder is defined, and it provides data which can be included in the FTS index, and the provider dataholder configuration is updated to include the new data, then observers (and, where they are missing, events) should be created which will trigger update of the catalog. The type and nature of the observer will depend on the situation.

Merge selectors

Subclasses of merge_select are used to manage totara_core\output\select form elements (although it should be possible to use them to manage any form element which can return a core\output\template). The totara select form elements are simple classes that are used to produce template objects (which link to a template file and contain the data needed to fill in the template).

When a catalog filter is defined, you need to specify a selector. A selector needs a key and label.

Merge_select objects can be merged together. If two catalog filters match then their selectors are merged. The result of merging depends on the subclass. Catalog comes with the following merge_select subclasses:

search_text: Displays a text input using the select_search_text template object. Merging just results in one object.
multi: Displays a list of items that can be selected using the select_multi template object. When created, options can specified by calling add_options_loader. When merged, the options from the two objects are merged.
single: Displays a drop-down list of items using the select_tree template object. When created, options can specified by calling add_options_loader, and an 'All' option can be added. When merged, the options from the two objects are merged.
tree: Displays a drop-down list of items using the select_tree template object. Options can be specified when calling the constructor, and an 'All' option can be added. Can only be merged if the two objects are identical, including options.

If you want to create a new type of form element for users to interact with, you would first create a template class (maybe based on the totara select class). Then define a merge_select subclass, which returns an instance of the template, and define how the objects should behave if selectors are merged. In some circumstances, it might not make sense to allow selectors to be merged at all, in which case the can_merge function would just return false and make sure that all uses of the new class have unique keys.

Datasearch filter classes

These classes are used to specify how sql filter snippets can be built. Each datasearch filter object contains information about the joins, parameters and conditions that need to be put together to add the particular filter to the catalog search query.

The main function in datasearch filters is make_compare. It returns an sql WHERE clause and some parameters. Included are:

All: A special subclass which does not result in any criteria (the same as if you added "1=1" to your query).
Equal: Creates a basic criteria "<somefield> = :<someuniqueparam>" and the current data value as the param.
In_or_equal: Takes a list of values and constructs an IN clause, and returns a list of uniquely identified params.
Like: Create a basic LIKE criteria, and allows control over how '%' prefixes and suffixes are applied.
Like_or: The same as 'like', creates a LIKE criteria for each provided value and joins them with ORs e.g. "(somefield LIKE :uq1 OR somefield LIKE :uq2)" and params ['uq1' => '%'.value1.'%', 'uq2' => '%'.value2.'%'].
Full_text_search: Constructs a full text search criteria, using the weights specified when the object is constructed.

Additional subclasses of filter could be defined to allow other WHERE clauses to be constructed.

The base filter class contains the following properties:

An alias, which is used to group filters for merging, and as the alias of the subquery that will be built.
The basealias of the table that the filter will join to.
Joinonbasefields which define what fields need to be linked between each 'source' and the base table.
Jointype which defaults to "JOIN" but can be changed to "LEFT JOIN" in order to add the filter subquery into the main query without actually removing any non-matching results (used for marking 'Featured learning').
A list of 'sources'. Each source further specified details of the subquery. The main reason for multiple sources is to allow several datasearch filters to be merged together.

Sources contain the following:

The filterfield which will be used by the filter subclass to construct the final criteria.
A table which needs to be joined to. Can be a complex subquery.
An alias for the joined table.
Joinons which map between the base table and the joined table, and must match the list of joinonbasefields.
Additionalcriteria which specify some more sql which should be included in the where clause. For example, this could be used to restrict the records considered from the joined table (e.g. exclude/ignore course set non-zero program completion records).
Additionalparams which might be needed in the table or additonalcriteria.
Additionalselect is used to 'float' data out from the joined table and make it available in the main query (used to allow the featured learning boolean to be included in the main query's select and return it in the record results).

Like selectors above, datasearch filter objects can be merged together. Merging is attempted if two filters have the same alias. In this case, they must also have matching basealias, joinonbasefields and jointype. If merging is possible, the sources from one filter are added to the other and the first filter is discarded.

When the filter is used, it constructs a subquery using the data provided when it was constructed, and all sources that it contains. The subquery is slightly different depending on whether there is one source or more. More information about how this happens is available in the other technical documentation. The end result is that calling make_sql will return a join, some where conditions and some parameters. The catalog pieces together these string from each of the active filters, and adds the base table information, in order to retrieve the appropriate results.

How the catalog works

The purpose of this section is to help developers who have to debug the catalog. It might also be useful as background for developers who are trying to implement new providers or extend existing ones.

Provider handler

This class manages and instantiates the providers. Only active providers should be instantiated, and the handler ensures this is the case.

The class contains the function get_data_for_objects. This function is used to retrieve data associated with the given list of objects. This data might be needed for various reasons, such as for populating the FTS index, displaying to users in the grid, or in the details popup. The following steps are performed:

Function get_sql_from_dataholders constructs an sql query by combining the sql snippets provided by each required dataholder.
Records are retrieved in bulk for each separate provider/objecttype.
The original list of objects is iterated over. For each object, a 'data' property is added, which contains all of the data formatted and ready to be used.

Function get_formatted_value_from_dataholder is used to transform the raw data obtained in the main sql query into the final formatted data, by calling the get_formatted_data function of the appropriate dataholder. Be aware that these formatter functions usually contain only simple php code, but they could also contain complex logic, function calls, or even database interactions.

Provider base class

The base class contains a number of functions which are used to:

Change the status of a provider, active or inactive.
Load filters, features and dataholders relating to the provider.

Filter handler

This class manages various lists of filters, including (but not limited to):

All filters: All filters from active providers, plus core catalog filters.
Active filters: All filters which are displayed on the catalog (not to be confused with 'active' when a filter has been activated by the current user by making some selection).
Region filters: The filters which need to be displayed in a particular catalog region (browse, panel or FTS).

The filter handler also does the merging of catalog filters as they are loaded.

Feature handler

This class manages the list of features. Like the filter handler, it merges features as they are loaded.

Adding data to the index

The class catalog_storage manages data going into (and being deleted from) the catalog. This class is called from various observers and adhoc and scheduled tasks.

Function update_records takes a list of objects and inserts or updates the catalog, in bulk. These objects are stdClass instances containing:

objecttype: A string matching a provider's get_object_type, e.g. 'course'.
objectid: An integer referencing some object belonging to the related provider, e.g. a course id.
contextid: An integer, 0 where there is no applicable contextid for the object.

Getting data out of the index

The catalog_retrieval class is used to get records, by calling get_page_of_objects. It constructs an sql query based on the active filters, which must be activated by calling the filter_handler (see below). The query is executed (perhaps several times) and the function iterates over the resulting records, checking whether the resulting learning items are visible, and continues until a full page of objects has been retrieved, or the end of the search results is reached. The result is a list of objects containing catalog id, objecttype, objectid and contextid.

Displaying data in the catalog

The totara_catalog\output\catalog class is used to display the main catalog page. It is a template class, so an instance contains a link to a template file and the data needed to fill in the template. This template displays the whole catalog page, including:

The grid, containing all of the items.
The 'Load more' button.
Results count.
Buttons used by admins to manage the catalog and learning items (e.g. Create course, edit catalog settings).
The item style toggle (grid/list).
The primary region, containing the browse and FTS filters.
The panel region, containing all the left-side filters.
The order by selector.
Debugging data.

This template is used in two ways.

When the page is first loaded, through index.php, the param_processor takes filters and other parameters specified in the url and passes them to the catalog::create function. The resulting catalog object is rendered in index.php.
When a user clicks on an interactive element within the catalog (e.g. filter, sorting, 'Load more'), a call is made to get_catalog_template_data in external.php. This function passes all the provided params to catalog::create, and then returns only the data the catalog object contains.

Catalog template class

The 'create' function takes a bunch of parameters and returns a new instance of the catalog template class. The parameters include details about how the grid items should be displayed, which page of records should be shown, and what filters should be applied to the grid results.

Note that it is possible to request only the catalog template data needed to refresh the 'results' of the current filters, paging, sorting and layout. In this case the data needed to render the filters, sorting options, debugging and management buttons is skipped.

The most significant procedure that the catalog template class performs is to retrieve the catalog items. It does the following:

The appropriate filters are loaded by the filter_handler and set_current_data is called with the provided filter params.
A catalog_retrieval object is created and used to retrieve the appropriate page of objects.
An item template object is created for each object, containing all the formatted data needed to display the item.
A grid template object is created containing all of the item template objects.

Details template class

This class is used to retrieve the data needed to display the details popup for a learning item. It is created in get_details_template_data in external.php, and the template data is returned. The create function works in a similar way to the catalog template class, when the item template objects are created.

Config class

This class is used to manage catalog and provider settings. Settings are stored using the standard set_config function, so end up in the config_plugins table.