API

Storage classes

class alliance_platform.storage.async_uploads.storage.base.AsyncUploadStorage

Base storage class for use with AsyncFileField

Provides generate_upload_url to return a URL to upload a file to (eg. a signed URL) and a move_file method to move a file from a temporary location to the permanent location. The key used for temporary files is returned by generate_temporary_path and is_temporary_path should return whether a file is in the temporary location and needs to be moved to a permanent location.

See AsyncFileMixin for a detailed explanation of how all the pieces fit together.

generate_download_url(key, field_id, **kwargs)

Should return a URL that the specified key should be downloadable from

In S3 this would be a signed URL.

Parameters:

key (str)
field_id (str)

generate_temporary_path(filename, max_length=None)

Generates a unique key to upload filename to

This generates a string like async-temp-files/2021/03/03/fVy5cSVBQpOb-test.png where test.png is the filename passed in. The part after - will be truncated to fit within max_length but will retain the file extension. If there is insufficient length to accommodate the temporary path prefix (up to the - and the file extension an error will be thrown.

generate_upload_url(name, field_id, *args, **kwargs)

Should return a URL that a file can be uploaded directly to

In S3 this would be a signed URL.

Parameters:

name (str)
field_id (str)

Return type:

GenerateUploadUrlResponse

get_url_patterns(registry)

Return the URL patterns for any views required by the storage class

When extending AsyncUploadStorage, this method can be implemented if any custom views are required to support the implementation. By default, two views are supplied:

DownloadRedirectView to support downloading an existing file. This is attached to the “download-file/” path.
GenerateUploadUrlView to generate a URL that can be uploaded to directly from the frontend. This is attached to the “generate-upload-url/” path.

This method is called by get_url_patterns().

Parameters:: registry (AsyncFieldRegistry)

is_temporary_path(filename)

Is the specified path that of a temporary file?

This is used to determine whether a file should be moved to a permanent location. If this returns true it’s expected an AsyncTempFile exists with the matching key.

Default implementation checks if path begins with temporary_key_prefix

Parameters:: filename (str)
Return type:: bool

move_file(from_key, to_key)

Move a file from one location to another

The details of this depend on the storage solution. In S3 this involves copying the file and then deleting the original file.

temporary_key_prefix = 'async-temp-files': The prefix to use on files when first uploaded before they are moved. This is used to identify a file as a temporary file to know whether to move them and so must be unique to temporary files.

class alliance_platform.storage.async_uploads.storage.s3.S3AsyncUploadStorage(*args, **kwargs)

S3 implementation of AsyncUploadStorage

Uses signed URLs for uploading.

class alliance_platform.storage.async_uploads.storage.azure.AzureAsyncUploadStorage(*args, **kwargs)

Azure implementation of AsyncUploadStorage

Uses signed URLs for uploading.

class alliance_platform.storage.async_uploads.storage.filesystem.FileSystemAsyncUploadStorage(*args, **kwargs)

Implementation of AsyncUploadStorage that uploads directly to the local server

This is useful in local dev, or when you still want the behaviour of uploading immediately rather than waiting until the whole form is submitted.

To use this by default, set the STORAGES setting:

STORAGES = {
    "default": {
        "BACKEND": "alliance_platform.storage.async_uploads.storage.filesystem.FileSystemAsyncUploadStorage"
    },
}

Alternatively, pass an instance of the class to the storage argument on AsyncFileField or AsyncImageField.

Staticfiles Storage

class alliance_platform.storage.staticfiles.storage.ExcludingManifestStaticFilesStorage(*args, **kwargs)

Custom static files storage that excludes specified files from being processed by ManifestStaticFilesStorage (i.e., they won’t have their names hashed). This is useful for files that are already hashed, or are never cached by the browser.

To use this class, set it in the STORAGES setting. Pass the exclude_patterns option as shown below. This can be set to patterns accepted by fnmatch()

Note

The patterns accepted are not the same as regular expressions - see fnmatch for details.

Usage:

STORAGES = {
    "staticfiles": {
        "BACKEND": "alliance_platform.storage.staticfiles.storage.ExcludingManifestStaticFilesStorage",
        "OPTIONS": {"exclude_patterns": ["frontend/build/*", "*.pdf"]},
    }
}

Models

class alliance_platform.storage.async_uploads.models.AsyncTempFile(*args, **kwargs)

Model to track files that are being uploaded to a temporary location

GenerateUploadUrlView is used to generate a URL to directly upload a file to. When this URL is generated an AsyncTempFile is created to track the new key that is used (eg. /temp/2020/01/04/abc123-myfile.png), the original filename (eg. myfile.png) and the specific field it came from (this is done via alliance_platform.storage.async_uploads.registry.AsyncFieldRegistry).

Once a file has been uploaded and the form saved the key recorded here will be saved against the underlying file field (either AsyncFileField or AsyncImageField) on the target model which will check if that key is a temporary file using an is_temporary_path(). If so the file will be moved to its permanent location using move_file() and the AsyncTempFile record will have the moved_to_location value set. See AsyncFileMixin for where this happens and more details.

You must run the cleanup_async_temp_files command periodically to cleanup this table. This handles two cases: the success case where moved_to_location is set but the record is being kept around for a while to detect duplicate submissions, and the other case where upload occurred on the frontend but the form was never submitted and the file was never moved.

Fields

class alliance_platform.storage.async_uploads.models.AsyncFileMixin(*args, perm_create=<object object>, perm_update=<object object>, perm_detail=<object object>, max_size=100, file_restrictions=None, async_field_registry=<alliance_platform.storage.async_uploads.registry.AsyncFieldRegistry object>, max_length=500, download_params=None, **kwargs)

Mixin for file fields that works with AsyncUploadStorage to handle uploading directly to external service like S3 or Azure.

This field works in conjunction with GenerateUploadUrlView. The view will generate a URL (eg. a signed URL when using S3) that the frontend can then upload to. Each view is tied to a specific registry which you can specify in async_field_registry (defaults to default_async_field_registry).

The permissions used by GenerateUploadUrlView can be specified in perm_create and perm_update. If not provided they default to the value returned by alliance_platform.auth.resolve_perm_name() for the action ‘create’ and ‘update’ respectively. To disable checking permissions, you can pass None - this will mean any user, including anonymous users, can generate upload urls.

When using django forms AsyncFileField provides a widget for handling the upload from the frontend. This is the default formfield provided by AsyncFileField.

Note

The key for the file is stored in the database as a CharField and as such has a max_length. The default for this is 500. This must be sufficient to accommodate the temporary file value which looks something like async-temp-files/2021/03/03/fVy5cSVBQpOb-test.png by default (see generate_temporary_path() for how to customise this).

Anything after the - will be truncated if necessary for the purposes of the temporary key. The actual filename will still be used for the final field value after the temporary file is moved to the permanent location. When moving to the permanent location the filename may be truncated if the length is too long when upload_to is taken into account.

If using S3AsyncUploadStorage you may wish to set AWS_S3_FILE_OVERWRITE = False to avoid overwriting files if the same key is used. This is not necessary if your files generate unique paths (eg. upload_to takes into account the record ID and filename).

The flow for how this works is as follows:

When a form is rendered on the frontend (eg. using AsyncFileField) it knows the async_field_id from the registry and generate_upload_url which is to the GenerateUploadUrlView view for the registry. This URL will be used to generated a specific URL for each file to upload to.

When an upload occurs on the frontend it first hits generate_upload_url and passes the async_field_id, the filename and optionally an instance_id if it’s an update for an existing record. GenerateUploadUrlView looks up the registry for the async_field_id to get the field and checks permissions on it (perm_update if instance_id is passed otherwise perm_create). If the permission check passes it will then create a AsyncTempFile record to track the filename passed in and a generated key that will be used to upload the file to a temporary location on the storage backend. The view will return an upload url by calling generate_upload_url() along with the generated key.

The frontend will receive the upload_url and key and proceed to upload to it. When the form is submitted and saved the key (the value returned from generated key) is the what will be stored in the database.

On save a post_save signal will be called which will check if the file needs to be moved to a permanent location. It does this by calling is_temporary_path() and move_file() is called to move the file to its permanent location at which point the AsyncTempFile is deleted.

If an upload occurs but the form isn’t submitted AsyncTempFile will be created but never deleted. The cleanup_async_temp_files should be run periodically to clean these up.

The url returned when accessing the url property, eg. model.file_field.url will always return the URL for DownloadRedirectView. This view will check the user has permission to download the file by checking the perm_download permission. If not provided this defaults to the value returned by alliance_platform.auth.resolve_perm_name() with the action “detail” (ie. if they have permission to view the record they can download the file). This view will then redirect to the URL provided by generate_download_url().

You can specify download_params on fields to control what arguments are passed through to generate_download_url(). For example:

expire_in_seconds = 60 * 60 * 6
download_params = {
    "expire": expire_in_seconds,
    "parameters": {"cache_control": f"public,max-age={expire_in_seconds}"},
}

# pass to a field on the model
image_file = AsyncFileField(download_params=download_params)

Note

You must you this with a storage class that implements AsyncUploadStorage. Either set DEFAULT_FILE_STORAGE to a class (eg. S3AsyncUploadStorage) or pass an instance in the storage kwarg.

For use with a Presto form use AsyncFileField or AsyncImageField on your serializer. This is handled by default when extending xenopus_frog_app.base.XenopusFrogAppModelSerializer. Codegen will create this as a AsyncFileField or AsyncImageField on the frontend Presto model. getWidgetForField will map these fields to the UploadWidget.

See AsyncFileField and AsyncFileField for Field classes that use this.

Warning

User input should be assigned using the AsyncFileInputData class which has validation to disallow things like moving the file by changing the key that passed from the frontend. This is handled for you when using AsyncFileField, AsyncImageField, AsyncFileField or AsyncImageField

If a file cannot be moved for any reason (eg. any exception is raised) then the error will be saved in the error field on the AsyncTempFile record and the temporary key will be retained as the field value. The error will also be logged to the alliance_platform.storage logger. This cannot be resolved automatically and so requires manual intervention to cleanup.

File size can be restricted by passing in max_size (in MB)

The file extension/file type can be restricted by passing in file_restrictions for example:

["image/png"]
["image/*"]
[".doc", "docx"]
["application/vnd.openxmlformats-officedocument.wordprocessingml.document"]

Parameters:: download_params (dict | None)

class alliance_platform.storage.async_uploads.models.AsyncFileField(*args, perm_create=<object object>, perm_update=<object object>, perm_detail=<object object>, max_size=100, file_restrictions=None, async_field_registry=<alliance_platform.storage.async_uploads.registry.AsyncFieldRegistry object>, max_length=500, download_params=None, **kwargs)

A FileField that works with AsyncUploadStorage to directly upload somewhere (eg. S3) from the frontend

See AsyncFileMixin for more details on how this field works.

Parameters:: download_params (dict | None)

class alliance_platform.storage.async_uploads.models.AsyncImageField(*args, suppress_pillow_check=False, **kwargs)

A ImageField that works with AsyncUploadStorage to directly upload somewhere (eg. S3) from the frontend

This supports width_field and height_field in two ways

1) (preferred) The frontend passes the width & height with the form submission. This is supported with the AsyncImageField form field (the default). This works in conjunction with the UploadWidget on the frontend.

2) (slow) Same behaviour as ImageField which requires the file to be downloaded and processed with Pillow.

See AsyncFileMixin for more details on how this field works.

Serializer Fields

To automatically map model fields to serializer fields add this to the base ModelSerializer class (this has already been done in xenopus_frog_app.base.XenopusFrogAppModelSerializer):

from rest_framework.serializers import ModelSerializer
from alliance_platform.storage.async_uploads.rest_framework import AsyncFileField
from alliance_platform.storage.async_uploads.rest_framework import AsyncImageField
import alliance_platform.storage.async_uploads.models as async_file_fields


class BaseModelSerializer(ModelSerializer):
    serializer_field_mapping = {
        **ModelSerializer.serializer_field_mapping,
        async_file_fields.AsyncFileField: AsyncFileField,
        async_file_fields.AsyncImageField: AsyncImageField,
    }

class alliance_platform.storage.async_uploads.rest_framework.AsyncFileField(*args, **kwargs)

Field that works with AsyncUploadStorage to handle uploading directly to external service like S3 or Azure.

Unlike most fields this field must be backed by an underlying model field specified in the model_field kwarg. This is inferred automatically if using a base ModelSerializer class with the following:

import alliance_platform.storage.async_uploads.models as async_file_fields
class BaseModelSerializer(ModelSerializer):
    serializer_field_mapping = {
        **ModelSerializer.serializer_field_mapping,
        async_file_fields.AsyncFileField: AsyncFileField,
        async_file_fields.AsyncImageField: AsyncImageField,
    }

This field expects to receive data in the shape { “key”: “/storage/file.png”, “name”: “file.png” }. It will extract the key to be set on the file field. See AsyncFileMixin for more details. For AsyncImageField the width and height keys may also exist in the data. See AsyncFileInputData

Serialized data is in shape { “key”: “/storage/file.png”, “name”: “file.png”, “url”: “/download/?field_id=…” } which is supported by the UploadWidget on the frontend.

class alliance_platform.storage.async_uploads.rest_framework.AsyncImageField(*args, **kwargs)

Same behaviour as AsyncFileField

This exists as a separate class to allow codegen to differentiate for the purpose of customising the field & widget on the frontend.

Forms

class alliance_platform.storage.async_uploads.forms.AsyncFileField(*args, async_field_registry, async_field_id, storage, max_length=None, **kwargs)

Form field that renders a AsyncFileInput

This is the default form field for alliance_platform.storage.async_uploads.models.AsyncFileField

Parameters:

async_field_registry (AsyncFieldRegistry)
async_field_id (str)
storage (AsyncUploadStorage)

__init__(*args, async_field_registry, async_field_id, storage, max_length=None, **kwargs)

Parameters:

*args – Any additional arguments to pass through to django.forms.Field
async_field_registry (AsyncFieldRegistry) – The async field registry that should used on the frontend to create unique upload urls for files. This typically comes from field.async_field_registry where field is an AsyncFileField.
async_field_id (str) – The string ID of the field used in the registry. Generated with field.async_field_registry.generate_id(field).
storage (AsyncUploadStorage) – The storage class. This comes from the field field.storage.
max_length – Max length for the filename
**kwargs – ny additional keyword arguments to pass through to django.forms.Field

class alliance_platform.storage.async_uploads.forms.AsyncFileInput(attrs=None)

Input for handling async uploads

This handles the submission value from UploadWidget and converts it to an instance of AsyncFileInputData. This is then handled on the descriptor classes for AsyncFileField and AsyncImageField.

To customise the widget rendered on the frontend you can override the template alliance_platform/storage/widgets/async_file_input.html.

class alliance_platform.storage.async_uploads.forms.AsyncFileInputData(key, name, width=None, height=None, error=None)

The data we receive from the frontend gets converted to an instance of this

width and height can optionally be included for images to avoid having to calculate dimensions manually when an AsyncImageField uses the width_field and height_field options.

Parameters:

key (str)
name (str)
width (int | None)
height (int | None)
error (str | None)

classmethod create_from_user_input(values)

Create instance from user input.

Throws if invalid keys present in values. url is accepted but ignored.

Parameters:: values (dict[str, str | int])

update_dimension_cache(field)

Updates the dimension_cache on AsyncImageField

Parameters:: field (Field)

validate_key(field, existing_value)

Validate the key should be accepted

This checks to make sure the key is a temporary path or matches the existing field value. This prevents manipulation of keys which could result in eg. taking over another records file

Parameters:

field (AsyncFileField)
existing_value (str | AsyncFieldFile | None)

class alliance_platform.storage.async_uploads.forms.AsyncImageField(*args, async_field_registry, async_field_id, storage, max_length=None, **kwargs)

Parameters:

async_field_registry (AsyncFieldRegistry)
async_field_id (str)
storage (AsyncUploadStorage)

Views

class alliance_platform.storage.async_uploads.views.GenerateUploadUrlView(**kwargs)

View to generate a URL using generate_upload_url().

Each view is tied to a specific registry which you can specify in the registry kwarg (defaults to default_async_field_registry).

This view expects a async_field_id, a filename and optionally an instance_id if it’s an update for an existing record. async_field_id is used to look up in registry the matching field instance. From the retrieved field instance the corresponding AsyncUploadStorage is retrieved along with the permissions to check (see AsyncFileMixin for where to specify this).

If the permission check passes a AsyncTempFile record is created which stores the filename and a generated key that will be used as a temporary location to upload the file to. An upload URL returned from generate_upload_url() and the temporary key is returned to the frontend. The frontend will submit the returned temporary key from the form and save it on the target model. On save the file is then moved to the permanent location using move_file() (this happens as part of AsyncFileMixin)

See AsyncFileMixin for a detailed explanation of how all the pieces fit together.

This view is automatically registered if you have followed the Register URLs guide.

class alliance_platform.storage.async_uploads.views.DownloadRedirectView(**kwargs)

View that checks permissions and redirects to a download URL using generate_download_url().

Each view is tied to a specific registry which you can specify in the registry kwarg (defaults to default_async_field_registry).

This view expects a async_field_id and instance_id which is the primary key of the record the file field is attached to. record. async_field_id is used to look up in registry the matching field instance. From the retrieved field instance the corresponding AsyncUploadStorage is retrieved along with the permission (perm_download) to check (see AsyncFileMixin for where to specify this).

Once the permission check has passed the view will redirect to the URL returned from generate_download_url().

See AsyncFileMixin for a detailed explanation of how all the pieces fit together.

This view is automatically registered if you have followed the Register URLs guide.

class alliance_platform.storage.async_uploads.views.filesystem.FileSystemAsyncStorageUploadView(**kwargs)

View to handle a direct upload to the filesystem when using FileSystemAsyncUploadStorage.

Each view is tied to a specific registry which you can specify in the registry kwarg (defaults to default_async_field_registry).

Note that this view does works off of signed data generated by generate_upload_url, and does not itself check any permissions. It does verify the signed data, including enforcing an expiry based on UPLOAD_URL_EXPIRY. This is similar to a signed URL generated by the S3 or Azure backends.

The permission checking is done in GenerateUploadUrlView, which calls generate_upload_url.

The URL for this view is automatically registered when FileSystemAsyncUploadStorage is used if you have followed the Register URLs guide.

class alliance_platform.storage.async_uploads.views.filesystem.FileSystemAsyncStorageDownloadView(**kwargs)

View to handle a direct download from the filesystem when using FileSystemAsyncUploadStorage.

Each view is tied to a specific registry which you can specify in the registry kwarg (defaults to default_async_field_registry).

Note that this view does works off of signed data generated by generate_download_url, and does not itself check any permissions. It does verify the signed data, including enforcing an expiry based on DOWNLOAD_URL_EXPIRY. This is similar to a signed URL generated by the S3 or Azure backends.

The permission checking is done in GenerateUploadUrlView, which calls generate_upload_url.

The URL for this view is automatically registered when FileSystemAsyncUploadStorage is used if you have followed the Register URLs guide.

Registry

class alliance_platform.storage.async_uploads.registry.AsyncFieldRegistry(name)

A registry for async fields. This allows looking up a field by its ID.

In most cases the default registry default_async_field_registry is suitable and more than one registry is not necessary.

generate_id(field)

Generate a string ID for a field to be used to lookup into fields_by_id.

This is passed from the frontend to uniquely identify a field in a registry.

Parameters:: field (Field)

get_url_patterns()

Get the required URL patterns for the storage classes used by the registered fields

This calls get_url_patterns() on each storage instance and combines the results.

Usage:

from django.urls import path
from django.urls import include
from alliance_platform.storage.async_uploads.registry import default_async_field_registry

urlpatterns = [path("async-uploads/", include(default_async_field_registry.get_url_patterns()))]

register_field(field)

Register a field in this registry.

Parameters:: field (AsyncFileField | AsyncImageField)

alliance_platform.storage.async_uploads.registry.default_async_field_registry: The default registry that is used when one is not explicitly specified