evennia.contrib.awsstorage package

Intended to be a collecting folder for Django-specific contribs that do not have observable effects to players.

Submodules

evennia.contrib.awsstorage.aws_s3_cdn module

AWS Storage System The Right Honourable Reverend (trhr) 2020

ABOUT THIS PLUGIN:

This plugin migrates the Web-based portion of Evennia, namely images, javascript, and other items located inside staticfiles into Amazon AWS (S3) for hosting.

Files hosted on S3 are “in the cloud,” and while your personal server may be sufficient for serving multimedia to a minimal number of users, the perfect use case for this plugin would be:

1. Servers supporting heavy web-based traffic (webclient, etc)

2. With a sizable number of users

3. Where the users are globally distributed

4. Where multimedia files are served to users as a part of gameplay

Bottom line - if you’re sending an image to a player every time they traverse a map, the bandwidth reduction will be substantial. If not, probably skip this one.

Note that storing and serving files via S3 is not technically free outside of Amazon’s “free tier” offering, which you may or may not be eligible for; evennia’s base install currently requires 1.5MB of storage space on S3, making the current total cost to install this plugin ~$0.0005 per year. If you have substantial media assets and intend to serve them to many users, caveat emptor on a total cost of ownership - check AWS’s pricing structure.

See the ./README.md file for details and install instructions.

class evennia.contrib.awsstorage.aws_s3_cdn.S3Boto3Storage(acl=None, bucket=None, **settings)

Bases: django.core.files.storage.Storage

Amazon Simple Storage Service using Boto3 This storage backend supports opening files in read or write mode and supports streaming(buffering) data in chunks to S3 when writing.

__init__(acl=None, bucket=None, **settings)

Check if some of the settings we’ve provided as class attributes need to be overwritten with values passed in here.

_clean_name(name)

Cleans the name so that Windows style paths work

_compress_content(content)

Gzip a given string content.

_decode_name(name)

_encode_name(name)

_get_access_keys()

Gets the access keys to use when accessing S3. If none is provided in the settings then get them from the environment variables.

_get_or_create_bucket(name)

Retrieves a bucket if it exists, otherwise creates it.

_get_security_token()

Gets the security token to use when accessing S3. Get it from the environment variables.

_get_write_parameters(name, content=None)

_normalize_name(name)

Normalizes the name so that paths like /path/to/ignored/../something.txt work. We check to make sure that the path pointed to is not outside the directory specified by the LOCATION setting.

_open(name, mode='rb')

Opens the file, if it exists.

_save(name, content)

Stitches and cleans multipart uploads; normalizes file paths.

_strip_signing_parameters(url)

Boto3 does not currently support generating URLs that are unsigned. Instead we take the signed URLs and strip any querystring params related to signing and expiration. Note that this may end up with URLs that are still invalid, especially if params are passed in that only work with signed URLs, e.g. response header params. The code attempts to strip all query parameters that match names of known parameters from v2 and v4 signatures, regardless of the actual signature version used.

access_key = ''

access_key_names = ['AWS_S3_ACCESS_KEY_ID', 'AWS_ACCESS_KEY_ID']

addressing_style = None

auto_create_bucket = False

property bucket

Get the current bucket. If there is no current bucket object create it.

bucket_acl = 'public-read'

bucket_name = None

config = None

property connection

Creates the actual connection to S3

custom_domain = None

deconstruct()

Return a 3-tuple of class import path, positional arguments, and keyword arguments.

default_acl = 'public-read'

default_content_type = 'application/octet-stream'

delete(name)

Deletes a file from S3.

encryption = False

endpoint_url = None

property entries

Get the locally cached files for the bucket.

exists(name)

Checks if file exists.

file_name_charset = 'utf-8'

file_overwrite = True

get_available_name(name, max_length=None)

Overwrite existing file with the same name.

get_modified_time(name)

Returns an (aware) datetime object containing the last modified time if USE_TZ is True, otherwise returns a naive datetime in the local timezone.

get_object_parameters(name)

Returns a dictionary that is passed to file upload. Override this method to adjust this on a per-object basis to set e.g ContentDisposition. By default, returns the value of AWS_S3_OBJECT_PARAMETERS. Setting ContentEncoding will prevent objects from being automatically gzipped.

gzip = False

gzip_content_types = ('text/css', 'text/javascript', 'application/javascript', 'application/x-javascript', 'image/svg+xml')

listdir(name)

Translational function to go from S3 file paths to the format Django’s listdir expects.

location = ''

max_memory_size = 0

modified_time(name)

Returns a naive datetime object containing the last modified time. If USE_TZ=False then get_modified_time will return a naive datetime so we just return that, else we have to localize and strip the tz

object_parameters = {}

preload_metadata = False

proxies = None

querystring_auth = True

querystring_expire = 3600

reduced_redundancy = False

region_name = None

secret_key = ''

secret_key_names = ['AWS_S3_SECRET_ACCESS_KEY', 'AWS_SECRET_ACCESS_KEY']

secure_urls = True

security_token = None

security_token_names = ['AWS_SESSION_TOKEN', 'AWS_SECURITY_TOKEN']

signature_version = None

size(name)

Gets the filesize of a remote file.

url(name, parameters=None, expire=None)

Returns the URL of a remotely-hosted file

url_protocol = 'http:'

use_ssl = True

verify = None

class evennia.contrib.awsstorage.aws_s3_cdn.S3Boto3StorageFile(name, mode, storage, buffer_size=None)

Bases: django.core.files.base.File

The default file object used by the S3Boto3Storage backend. This file implements file streaming using boto’s multipart uploading functionality. The file can be opened in read or write mode. This class extends Django’s File class. However, the contained data is only the data contained in the current buffer. So you should not access the contained file object directly. You should access the data via this class. Warning: This file must be closed using the close() method in order to properly write the file to S3. Be sure to close the file in your application.

__init__(name, mode, storage, buffer_size=None)

Initializes the File object.

Parameters

• name (str) – The name of the file

• mode (str) – The access mode (‘r’ or ‘w’)

• storage (Storage) – The Django Storage object

• buffer_size (int) – The buffer size, for multipart uploads

property _buffer_file_size

_create_empty_on_close()

Attempt to create an empty file for this key when this File is closed if no bytes have been written and no object already exists on S3 for this key. This behavior is meant to mimic the behavior of Django’s builtin FileSystemStorage, where files are always created after they are opened in write mode:

f = storage.open(“file.txt”, mode=”w”) f.close()

Raises:

Exception: Raised if a 404 error occurs

_flush_write_buffer()

Flushes the write buffer.

_get_file()

Helper function to manage zipping and temporary files

_set_file(value)

buffer_size = 5242880

close()

Manages file closing after multipart uploads

deconstruct()

Return a 3-tuple of class import path, positional arguments, and keyword arguments.

property file

Helper function to manage zipping and temporary files

read(*args, **kwargs)

Checks if file is in read mode; then continues to boto3 operation

readline(*args, **kwargs)

Checks if file is in read mode; then continues to boto3 operation

property size

Helper property to return filesize

write(content)

Checks if file is in write mode or needs multipart handling, then continues to boto3 operation.

evennia.contrib.awsstorage.aws_s3_cdn.check_location(storage)

Helper function to make sure that the storage location is configured correctly.

Parameters

storage (Storage) – A Storage object (Django)

Raises

ImproperlyConfigured – If the storage location is not configured correctly, this is raised.

evennia.contrib.awsstorage.aws_s3_cdn.get_available_overwrite_name(name, max_length)

Helper function indicating files that will be overwritten during trunc.

Parameters

• name (str) – The name of the file

• max_length (int) – The maximum length of a filename

Returns

A joined path including directory, file, and extension

Return type

joined (path)

evennia.contrib.awsstorage.aws_s3_cdn.lookup_env(names)

Helper function for looking up names in env vars. Returns the first element found.

Parameters

names (str) – A list of environment variables

Returns

The value of the found environment variable.

Return type

value (str)

evennia.contrib.awsstorage.aws_s3_cdn.safe_join(base, *paths)

Helper function, a version of django.utils._os.safe_join for S3 paths. Joins one or more path components to the base path component intelligently. Returns a normalized version of the final path. The final path must be located inside of the base path component (otherwise a ValueError is raised). Paths outside the base path indicate a possible security sensitive operation.

Parameters

• base (str) – A path string to the base of the staticfiles

• *paths (list) – A list of paths as referenced from the base path

Returns

A joined path, base + filepath

Return type

final_path (str)

evennia.contrib.awsstorage.aws_s3_cdn.setting(name, default=None)

Helper function to get a Django setting by name. If setting doesn’t exist it will return a default.

Parameters

name (str) – A Django setting name

Returns

The value of the setting variable by that name

evennia.contrib.awsstorage.tests module

class evennia.contrib.awsstorage.tests.S3Boto3StorageTests(methodName='runTest')

Bases: evennia.contrib.awsstorage.tests.S3Boto3TestCase

_test_storage_mtime(use_tz)

test_auto_creating_bucket()

test_auto_creating_bucket_with_acl()

test_clean_name()

Test the base case of _clean_name

test_clean_name_normalize()

Test the normalization of _clean_name

test_clean_name_trailing_slash()

Test the _clean_name when the path has a trailing slash

test_clean_name_windows()

Test the _clean_name when the path has a trailing slash

test_compress_content_len()

Test that file returned by _compress_content() is readable.

test_connection_threading()

test_content_type()

Test saving a file with a None content type.

test_generated_url_is_encoded()

test_location_leading_slash()

test_override_class_variable()

test_override_init_argument()

test_pickle_with_bucket()

Test that the storage can be pickled with a bucket attached

test_pickle_without_bucket()

Test that the storage can be pickled, without a bucket instance

test_special_characters()

test_storage_delete()

test_storage_exists()

test_storage_exists_doesnt_create_bucket()

test_storage_exists_false()

test_storage_listdir_base()

test_storage_listdir_subdir()

test_storage_mtime()

test_storage_open_no_overwrite_existing()

Test opening an existing file in write mode and closing without writing.

test_storage_open_no_write()

Test opening file in write mode and closing without writing.

A file should be created as by obj.put(…).

test_storage_open_write()

Test opening a file in write mode

test_storage_save()

Test saving a file

test_storage_save_gzip()

Test saving a file with gzip enabled.

test_storage_save_gzip_twice()

Test saving the same file content twice with gzip enabled.

test_storage_save_gzipped()

Test saving a gzipped file

test_storage_save_with_acl()

Test saving a file with user defined ACL.

test_storage_size()

test_storage_url()

test_storage_url_slashes()

Test URL generation.

test_storage_write_beyond_buffer_size()

Test writing content that exceeds the buffer size

test_strip_signing_parameters()

class evennia.contrib.awsstorage.tests.S3Boto3TestCase(methodName='runTest')

Bases: django.test.testcases.TestCase

setUp()

Hook method for setting up the test fixture before exercising it.