How to Build Open-Source AWS S3-Compatible Storage on Docker?

AWS S3 compatible storage is one of the emerging technologies in the enterprise storage medium. Initially, it was used only by Amazon in public cloud environments. However, today it has been commonly used by all the storage & cloud vendors in on-premises and private cloud environments.

‘S3 compatible storage’ offers rich Amazon S3 API complaint interfaces.

Use-cases:

1 Backup & Disaster Recovery:

S3 compatible storage is suitable for storing and archiving mission-critical data on-premises providing maximum availability, reliability, and durability.

2 Storing large data-sets over network:

S3 compatible storage is ideal when you want to store all kinds of documents and unstructured data: images, materials like PDFs and Excel docs, music, videos, backup files, database dumps, log files and render with faster performance.

3 File sharing solutions:

S3 compatible storage can also be used as a file-sharing solution or a network drive and be integrated into your environment.

4 Pricing:

Lesser cost than public cloud: With industry-standard hardware/ VMs, S3 compatible storage solutions can be installed and these solutions deliver the highest value.

5 Secured & Performance:

S3 compatible storage is deployed on industry-standard hardware/ VMs, which is in your data center and has secured data access. It also delivers higher throughput and lesser latencies.

Open-source AWS S3-compatible storage solutions:

Below, I am going to explain 2 solutions as an example of open-source AWS S3-compatible storage built on the Docker platform.

1. Scality/s3server.
2. MinIO Object Storage.

Solution1: Scality/s3server

About Scality:

Scality is an open-source AWS S3 compatible storage solution that provides an S3-compliant interface for IT professionals. It allows using there S3-compatible storage applications, develop there S3 compliant apps faster by doing testing and integration locally or against any remote S3 compatible cloud.

Quick Start: Used Centos-7 VM

[root@localhost ~]# docker run --name AWS_S3 -p 8000:8000 -e SCALITY_ACCESS_KEY_ID=accessKey1 -e SCALITY_SECRET_ACCESS_KEY=verySecretKey1 scality/s3server

[root@localhost ~]# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
bc290f05ad5c        scality/s3server    "/usr/src/app/dock..."   8 hours ago         Up 8 hours          0.0.0.0:8000->8000/tcp   AWS_S3

Testing – Create buckets on Scality S3server using CYBERDUCK UI and create/upload files on the bucket.

Solution2: MinIO Object Storage

About MinIO:

MinIO is a 100 percent open-source, distributed object storage system. It is software-defined, runs on industry-standard hardware, and API compatible with Amazon S3 cloud storage service.

Quick Start: Used Centos-7 VM

[root@localhost ~]# docker run -p 9000:9000 --name S3_minio -e "MINIO_ACCESS_KEY=accessKey1" -e "MINIO_SECRET_KEY=verySecretKey1"  minio/minio server /mnt/data

[root@localhost ~]# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS              PORTS                    NAMES
13f2fc802ec9        minio/minio         "/usr/bin/docker-e..."   About a minute ago   Up About a minute   0.0.0.0:9000->9000/tcp   S3_minio

Testing – Create buckets on MinIO Object Storage using AWS CLI and create/upload files on the bucket.

References:
https://min.io/
https://www.scality.com/topics/what-is-s3-compatible-storage/