​
AmazonS3Storage

class AmazonS3Storage(BaseObjectStorage):

A class to connect with AWS S3 object storage to put and get objects from one S3 bucket. The class will first try to use the credentials passed as arguments, if not provided, it will look for the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. If none of these are provided, it will try to use the local credentials (will be created if logged in with AWS CLI).

Parameters:

  • bucket_name (str): The name of the S3 bucket.
  • create_if_not_exists (bool, optional): Whether to create the bucket if it does not exist. Defaults to True.
  • access_key_id (Optional[str], optional): The AWS access key ID. Defaults to None.
  • secret_access_key (Optional[str], optional): The AWS secret access key. Defaults to None.
  • anonymous (bool, optional): Whether to use anonymous access. Defaults to False.
  • References:
  • https: //aws.amazon.com/pm/serv-s3/
  • https: //aws.amazon.com/cli/

​
init

def __init__(
    self,
    bucket_name: str,
    create_if_not_exists: bool = True,
    access_key_id: Optional[str] = None,
    secret_access_key: Optional[str] = None,
    anonymous: bool = False
):

​
_prepare_and_check

def _prepare_and_check(self):

Check privileges and existence of the bucket.

​
canonicalize_path

def canonicalize_path(file_path: PurePath):

Canonicalize file path for Amazon S3.

Parameters:

  • file_path (PurePath): The path to be canonicalized.

Returns:

Tuple[str, str]: The canonicalized file key and file name.

​
_put_file

def _put_file(self, file_key: str, file: File):

Put a file to the Amazon S3 bucket.

Parameters:

  • file_key (str): The path to the object in the bucket.
  • file (File): The file to be uploaded.

​
_get_file

def _get_file(self, file_key: str, filename: str):

Get a file from the Amazon S3 bucket.

Parameters:

  • file_key (str): The path to the object in the bucket.
  • filename (str): The name of the file.

Returns:

File: The object from the S3 bucket.

​
_upload_file

def _upload_file(self, local_file_path: Path, remote_file_key: str):

Upload a local file to the Amazon S3 bucket.

Parameters:

  • local_file_path (Path): The path to the local file to be uploaded.
  • remote_file_key (str): The path to the object in the bucket.

​
_download_file

def _download_file(self, local_file_path: Path, remote_file_key: str):

Download a file from the Amazon S3 bucket to the local system.

Parameters:

  • local_file_path (Path): The path to the local file to be saved.
  • remote_file_key (str): The key of the object in the bucket.

​
_object_exists

def _object_exists(self, file_key: str):

Check if the object exists in the Amazon S3 bucket.

Parameters:

  • file_key: The key of the object in the bucket.

Returns:

bool: Whether the object exists in the bucket.