Creating a New Storage Provider (C#Bot)

Introduction

In C#Bot, files are stored in storage providers, which are an abstraction over any system that can store files. More information can be seen [here](). In this article we will go over the process of extending C#Bot to create a new storage provider.

Code Structure

Before creating a new provider, it is important to understand how a storage provider works. All storage providers are implemented in serverside/src/Services/Files/Providers and all implement the IUploadStorageProvider interface. This interface contains a limited set of storage operations which can be interacted with by the rest of the application. Another key location to consider is the registration of the provider in Startup and adding configuration in serverside/src/Configuration.

Data in a storage provider is stored in a Container and has a FileName. The implementation of a container is up to the storage provider. For example, in a file system based storage provider, a container could be implemented by folders. Conversely, in a key value store based provider, containers could be implemented by prefixing keys with the container name.

Creating a New Provider

For this tutorial we are going to be constructing a basic in-memory storage provider to store our files in. In this implementation of a storage provider, we are going to be implementing the concept of containers with the container name concatenated to the file name separated by a /. This means a key will have the format of Container/FileName.

It is important to note since this is going to be an in-memory store in the source code, it is not suitable for use in a production application and is for tutorial purposes only.

The first thing we must do is create a file for our new provider at serverside/src/Services/Files/Providers/InMemoryStorageProvider.cs. With this file we can add the following contents.

using System.Collections.Concurrent;

namespace Climateaction.Services.Files.Providers
{
    public class InMemoryStorageProvider : IUploadStorageProvider
    {
        /// <summary>
        /// The data store for our files. This is just a simple dictionary that stores byte arrays against keys.
        /// </summary>
        private static readonly ConcurrentDictionary<string, byte[]> _contents = new ConcurrentDictionary<string, byte[]>();
    }
}

Now we have our class skeleton, we can implement the rest of the interface methods we require.

GetAsync

The GetAsync method is used to retrieve a single file from the storage provider. This will return a Task which contains a stream of bytes representing the file.

public Task<Stream> GetAsync(StorageGetOptions options, CancellationToken cancellationToken = default)
{
    var bytes = _contents[$"{options.Container}/{options.FileName}"];
    return Task.FromResult(new MemoryStream(bytes) as Stream);
}

ListAsync

This function will list the contents of a container. Due to our container implementation, we must get all keys that start with Container/.

public Task<IEnumerable<string>> ListAsync(StorageListOptions options, CancellationToken cancellationToken = default)
{
    var keys = _contents.Keys.Where(x => x.StartsWith($"{options.Container}/"));
    return Task.FromResult(keys);
}

ExistsAsync

This method will check if the file exists in the specified container:

public Task<bool> ExistsAsync(StorageExistsOptions options, CancellationToken cancellationToken = default)
{
    return Task.FromResult(_contents.ContainsKey($"{options.Container}/{options.FileName}"));
}

PutAsync

This method is used to create a new file. For this implementation we are reading the stream into a MemoryStream which can then be converted into a byte array and saved to the hashmap. In the case of the overwrite option is set to false, then we should throw an IOException if a file with this name already exists in the container.

public Task PutAsync(StoragePutOptions options, CancellationToken cancellationToken = default)
{
    if (!options.Overwrite && _contents.ContainsKey($"{options.Container}/{options.FileName}"))
    {
        throw new IOException("File already exists");
    }

    var stream = new MemoryStream();
    options.Content.CopyTo(stream);
    _contents[$"{options.Container}/{options.FileName}"] = stream.ToArray();
    return Task.CompletedTask;
}

DeleteAsync

This method is used to delete a file in the provider. For this implementation all we have to do is remove the key from the dictionary.

public Task DeleteAsync(StorageDeleteOptions options, CancellationToken cancellationToken = default)
{
    _contents.Remove($"{options.Container}/{options.FileName}", out _);
    return Task.CompletedTask;
}

ContainerExistsAsync

This method checks to see if a container exists in the provider. For providers which do not have the concept of physical containers (such as this one), this function should return whether any files exist in this container.

public Task<bool> ContainerExistsAsync(StorageContainerExistsOptions options, CancellationToken cancellationToken = default)
{
    var hasFiles = _contents.Keys.Any(x => x.StartsWith($"{options.Container}/"));
    return Task.FromResult(hasFiles);
}

DeleteContainerAsync

This method will delete all files in a container and the container itself, if it is physically manifested.

public Task DeleteContainerAsync(StorageDeleteContainerOptions options, CancellationToken cancellationToken = default)
{
    var keys = _contents.Keys.Where(x => x.StartsWith($"{options.Container}/"));

    foreach (var key in keys)
    {
        _contents.Remove(key, out _);
    }

    return Task.CompletedTask;
}

OnFetch

This function is used to implement a way for a storage provider to manually handle serving the client from the API. If this function returns null, the file controller will fetch the file using GetAsync and return the contents of that function. Otherwise, this function will return a new function which will be executed by the file controller. This is useful if the backing provider already has a method to serve files over http; an example of this is Amazon S3 using presigned urls.

public Func<CancellationToken, Task<IActionResult>> OnFetch(StorageOnFetchOptions options)
{
    return async token =>
    {
        var file = await GetAsync(new StorageGetOptions
        {
            Container = options.File.Container,
            FileName = options.File.FileId,
        }, token);

        var cd = new ContentDispositionHeaderValue(options.Download ? "attachment" : "inline")
        {
            Name = options.File.FileName,
            FileNameStar = options.File.FileName,
            Size = file.Length,
            FileName = options.File.FileName,
        };

        options.HttpContext.Response.Headers["Content-Disposition"] = cd.ToString();
        options.HttpContext.Response.Headers["Content-Type"] = options.File.ContentType;

        return new FileStreamResult(file, options.File.ContentType)
        {
            LastModified = options.File.Modified,
        };
    };
}

It is important to note in this implementation of the storage provider, doing this manually provides no benefits and it ids only done for demonstration purposes. The recommended approach to take would be to return null.

public Func<CancellationToken, Task<IActionResult>> OnFetch(StorageOnFetchOptions options)
{
    return null;
}

Dispose

Since there is nothing we will need to dispose of with this provider, we can provide and empty Dispose method.

public void Dispose()
{
}

Registering the Provider

The provider must now be registered for use in the application. Firstly, we will add this new provider as an option in the configuration. In serverside/src/Configuration/StorageProviderConfiguration.cs change the enum to look like the following:

public enum StorageProviders
{
    FILE_SYSTEM,
    S3,
    // % protected region % [Add any extra storage provider enum entries here] on begin
    IN_MEMORY,
    // % protected region % [Add any extra storage provider enum entries here] end
}

Now we must register this provider in dependency injection. In serverside/src/Startup.cs locate the protected region labelled Configure storage provider services here and change it to the following.

// % protected region % [Configure storage provider services here] on begin
// Configure the file system provider to use
var storageOptions = new StorageProviderConfiguration();
Configuration.GetSection("StorageProvider").Bind(storageOptions);
switch (storageOptions.Provider)
{
    case StorageProviders.S3:
        services.TryAddScoped<IUploadStorageProvider, S3StorageProvider>();
        break;
    case StorageProviders.IN_MEMORY:
        services.TryAddScoped<IUploadStorageProvider, InMemoryStorageProvider>();
        break;
    case StorageProviders.FILE_SYSTEM:
    default:
        services.TryAddScoped<IUploadStorageProvider, FileSystemStorageProvider>();
        break;
}
// % protected region % [Configure storage provider services here] end

Configuring the Provider

Finally we have to change our appsettings to use the provider we made. In serverside/src/appsettings.Development.xml locate the protected region labelled Add any extra app configurations here and add the following.

<StorageProvider>
    <Provider>IN_MEMORY</Provider>
</StorageProvider>

Now, if you run your server you will be storing files in your new provider.