Cloud file storage

Using cloud storage

In Totara you can now use cloud storage with your site. The main use cases for cloud storage are:

• Regular, scheduled backups to keep copies of your files online
• Limiting the size of shared dataroot in web clusters
• Sharing content between different environments (e.g. staging and production sites)

The production server in the local hosting facility uses the regular filedir in dataroot. Whenever new content is uploaded to your Totara site a copy of each file is sent to the S3 cloud store. Content is never deleted from the S3 cloud store.

Sharing of filedir in web cluster via cloud

Site is configured to use a cluster of web servers in order to improve scalability and performance. Each node is using local filedir as a fast cache and the shared cloud store is updated after all changes in any cluster node. When adding a new node local filedir starts empty and is gradually filled during file access.

Sharing of file contents between different environments

Any cloud file content store can be configured to be read-only and used for restoring of missing content files only. This eliminates the need to make a full copy of local filedir when testing upgrades or staging production server changes.

Offloading file contents

Some operating system file systems allow tracking of the last access time for each file. If there is a full backup in the cloud, then the administrator can create a script that deletes large files that have not been accessed recently. If the missing files are accessed then the contents are automatically downloaded from the cloud store.

Setting up cloud storage

Follow these steps to set up cloud storage for your site:

1. Install PHP libraries for cloud providers.
2. Add settings to config.php.
3. Upgrade or install site if necessary.
4. Push existing content files to cloud storage using ```totara/cloudfiledir/cli/store.php --push```

Installing PHP libraries

Before using this plugin you must install optional libraries via composer in /libraries/directory.

Cloud store options

All configuration is stored directly in config.php files in the $CFG->totara_cloudfiledir_stores array. The administration interface can be used to review a list of configured stores.

Here is an example of these settings once they have been configured:

```
$CFG->totara_cloudfiledir_stores = [
    [
       'idnumber' => 'shared_storage',
        'provider' => 's3',
        'bucket' => 'sharedstorage',
        'options' => [
            'region' => 'us-west-2',
            'profile' => 'default',
        ],
        'add' => true,
        'delete' => true,
        'restore' => true,
        'active' => true,
        'maxinstantuploadsize' => -1, // default, means all new files are uploaded to cloud asap.
    ],
    [
       'idnumber' => 'persistent_backup',
        'provider' => 's3',
        'bucket' => 'persistentbackup',
        'options' => [
            'region' => 'us-west-2',
            'profile' => 'backup',
        ],
        'add' => true,
        'delete' => false,
        'restore' => true,
        'active' => true,
        'maxinstantuploadsize' => 0, // 0 means upload via cron/CLI only.
    ],
];

```

Cloud storage options

Amazon S3

The Amazon S3 API is supported by the vast majority of cloud storage solutions. Totara supports arbitrary connection options.

Supported connection options are described in Amazon's SDK documentation:

Here are some alternative storage solutions that should be compatible with this plugin:

• MinIO - https://docs.min.io/docs/how-to-use-aws-sdk-for-php-with-minio-server.html
• OpenStack Swift - https://docs.openstack.org/mitaka/config-reference/object-storage/configure-s3.html
• Google Storage - https://cloud.google.com/storage/docs/migrating#migration-simple
• OpenIO - https://docs.openio.io/latest/source/integrations/cookbook_nextcloud.html

Azure Blob Storage

Microsoft Azure Blob Storage is a notable exception because it does not support the S3 API de facto standard. Connection configuration options are described at https://docs.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string.

Custom store providers

Support for different API can be easily implemented in a small custom PHP class extending \totara_cloudfiledir\local\provider\base, see \totara_cloudfiledir\local\provider\s3 and \totara_cloudfiledir\local\provider\azure classes.

Additional tasks

Cron sync task

If any store has a maxinstantuploadsize value set, then the system administrator should enable push task in scheduled tasks settings.

CLI script for maintenance of cloud filedir

The command line script /totara/cloudfiledir/cli/store.php can be used to do the following:

• List all available stores
• Fetch list of content files when connecting a new cloud store with existing content
• push local file contents to newly connected empty cloud store or upload missing contents to existing cloud store
• Reset flags used for skipping of invalid local content
• Print list of problems related to cloud store

CLI script for maintenance of local filedir

There is a new CLI script for maintenance of local filedir directory, see /admin/cli/check_filedir.php.

These are the expected use cases for this script:

• Checking consistency of all content files in local filedir
• Restoring missing content files from external stores
• Deleting orphaned files from local filedir