Async File Uploads

Overview

The Async Upload feature in alliance-platform-storage provides a seamless way to handle asynchronous file uploads in Django applications. In this context, the async in “Async Upload” means the file is immediately uploaded to the server (e.g. S3, Azure, local server) while the user continues to fill out a form. A reference (the key) to the uploaded file is stored in the form and handled in the Django model save when processing the form submission. See How it works for more specifics.

It’s primarily implemented through the AsyncFileField and AsyncImageField classes.

Key Components:

AsyncFileField: The main field for handling async file uploads.
AsyncImageField: Similar to AsyncFileField, but specifically for image files.
AsyncUploadStorage: The base storage class for handling async uploads.
GenerateUploadUrlView: View for generating upload URLs.
DownloadRedirectView: View for handling file downloads.
AsyncFieldRegistry: Registry for managing async fields.

Storage backends:

Storage backends handle the logic for generating signed upload & download URLs, and for moving files to the final location. The following backends are provided:

S3AsyncUploadStorage - Uploads to Amazon S3
AzureAsyncUploadStorage - Uploads to Azure blob storage
FileSystemAsyncUploadStorage - Uploads to Django directly and stores the file in the local filesystem (e.g. media files).

Other implementations can be provided by extending the AsyncUploadStorage class and implementing the relevant methods.

Installation

Backend Configuration

The chosen backend class can be set globally in the STORAGES setting:

STORAGES = {
    "default": {
        "BACKEND": "<your chosen backend class here>"
    },
}

Alternatively, you can pass a storage class instance to the storage argument on the model field .

Amazon S3

To use with Amazon S3 django-storages with S3 is required. If you installed alliance_platform_storage with -E s3 this will be installed, otherwise run:

poetry add django-storages -E s3

To make it the default for fields set the STORAGES setting:

STORAGES = {
    "default": {
        "BACKEND": "alliance_platform.storage.async_uploads.storage.s3.S3AsyncUploadStorage"
    },
}

See the S3 authentication documentation for what other settings will need to be set.

Azure Blob Storage

To use with Azure django-storages with Azure is required. If you installed alliance_platform_storage with -E azure this will be installed, otherwise run:

poetry add django-storages -E azure

To make it the default for fields set the STORAGES setting:

STORAGES = {
    "default": {
        "BACKEND": "alliance_platform.storage.async_uploads.storage.azure.AzureAsyncUploadStorage"
    },
}

See the Azure authentication documentation for what other settings will need to be set.

File System

To use with the local filesystem you can use FileSystemAsyncUploadStorage.

To make it the default for fields set the STORAGES setting:

STORAGES = {
    "default": {
        "BACKEND": "alliance_platform.storage.async_uploads.storage.filesystem.FileSystemAsyncUploadStorage"
    },
}

Register URLs

To facilitate async uploads, some URLs need to be registered. This is crucial for generating upload URLs and handling downloads. You can register the URLs by calling get_url_patterns().

from alliance_platform.storage.async_uploads.registry import default_async_field_registry

urlpatterns = [
    # ... other patterns ...
    path("async-uploads/", include(default_async_field_registry.get_url_patterns())),
]

Note

If you use multiple registries, you will need to do this for each registry. In most cases the default registry is sufficient.

Cleanup command

Intermediate files are stored in the alliance_platform.storage.async_uploads.models.AsyncTempFile table. Periodically clean up these files by running the cleanup_async_temp_files command:

python manage.py cleanup_async_temp_files

How it works

The AsyncFile feature works in conjunction with GenerateUploadUrlView. The view generates a URL (e.g., a signed URL when using S3) that the frontend can then use for direct uploads. Each view is tied to a specific registry, which you can specify using async_field_registry (defaults to default_async_field_registry). In most cases, a single registry is fine and you don’t need to explicitly reference it.

The flow for async file uploads is as follows:

When a form is rendered on the frontend (e.g., using AsyncFileField), it knows the async_field_id from the registry and the generate_upload_url endpoint.
When an upload occurs, the frontend first hits the generate_upload_url endpoint, passing the async_field_id, filename, and optionally an instance_id for updates.
GenerateUploadUrlView looks up the registry for the async_field_id, checks permissions, and creates an AsyncTempFile record.
The frontend receives the upload URL and uploads the file directly to the storage backend. The key for the AsyncTempFile is stored in the form for submission.
Upon form submission, the backend moves the file from its temporary location to its final destination, and cleans up the AsyncTempFile record.
If form submission never occurs, for example the user abandons the form after uploading a file, then the file will be retained until the cleanup_async_temp_files command is run.

Usage

Add a AsyncFileField: or AsyncImageField to a model, optionally passing the storage option if you need to use a different backend from the project STORAGES setting.

from alliance_platform.storage.async_uploads.models import AsyncFileField
from alliance_platform.storage.async_uploads.storage.s3 import S3AsyncUploadStorage

storage = S3AsyncUploadStorage()

class MyModel(models.Model):
    file = AsyncFileField()
    # Optionally pass storage
    image = AsyncImageField(storage=storage)

Form Usage:

By default, the AsyncFileField is used to handle uploads from Django forms. The default widget is AsyncFileInput.

async_uploads.rest_framework Integration:

For Django Rest Framework, use the async_uploads.rest_framework fields alliance_platform.storage.async_uploads.rest_framework.AsyncFileField or alliance_platform.storage.async_uploads.rest_framework.AsyncImageField.

You can set this as the default for the corresponding model fields by adding entries to the serializer_field_mapping on a custom ModelSerializer base class:

from alliance_platform.storage.async_uploads.rest_framework import AsyncFileField
from alliance_platform.storage.async_uploads.rest_framework import AsyncImageField
import alliance_platform.storage.async_uploads.models as async_file_fields

class XenopusFrogAppModelSerializer(ModelSerializer):
    serializer_field_mapping = {
        **ModelSerializer.serializer_field_mapping,
        async_file_fields.AsyncFileField: AsyncFileField,
        async_file_fields.AsyncImageField: AsyncImageField,
    }

Permissions

Permissions for file operations can be specified using perm_create and perm_update. If not provided, they default to the value returned by resolve_perm_name() for the ‘create’ and ‘update’ actions respectively. To disable permission checks, pass None.

Advanced Usage

For more advanced usage, including custom storage backends, modifying temporary file paths, and handling file overwrites, refer to the API documentation of individual classes and the installation guide.

Note on File Length

The key for the file is stored in the database as a CharField with a default max_length of 500. Ensure this is sufficient for your use case, especially when considering temporary file paths and upload_to configurations. You can pass a different max_length as a kwarg to the field.