Yesterday I had a customer asking me how to make a simple directory listing like you have with apache server, but with the content stored in an S3 bucket. This way they can replace their git lfs + apache setup with something possibly simpler and more suited to binary storage.
As it turns out it’s not that complicated. You need to get the permissions right, and to be aware of a listing option in S3 API. Let’s see how to make this work on Clever Cloud.
Create a Cellar Addon
Clever Cloud is a Platform as a Service. We provide many managed services and one of them is Cellar, our S3 compatible storage. Assuming you already have an account, here’s how to create a Cellar add-on.
Using the CLI
Assuming you are already using clever-tools, all you have to do is run clever addon create cellar-addon --plan S myCellarService.
Using the Web Console
You can login to our console on https://console.clever-cloud.com/. From there click on Create, an add-on and choose Cellar. There is only one plan available since Cellar is billed based on consumption so click next. We have no need to link it to an existing application so click next again, give it a name and click on create.
Setup a Bucket
Now that you have your S3 add-on, the easiest way to manage buckets is probably to use s3cmd or Rclone. Here’s how to create a bucket, make it public and store some files.
First get your configuration file. It’s waiting for you on the add-on dashboard. Browse your Cellar add-on from the console, you should see some details and a link to Download a pre-filled s3cfg file.. Download it and place it under your home directory. If you are a linux user it should be ~/.s3cfg.
Now s3cmd will use this configuration. Nothing else to do. Typing s3cmd la list all resources available. If you just created the Cellar add-on, you should see nothing listed. To create your bucket run s3cmd mb s3://your-unique-bucket-name/. The chosen name should be lower case and unique. If I insist on the uniqueness of that name, it’s because it can’t be already taken by someone else. All buckets are made available by default on http://your-unique-bucket-name.cellar-c2.services.clever-cloud.com. You can see why they have to be unique.
You can also name it after your domain name, and put a CNAME redirection to cellar-c2.services.clever-cloud.com.. This way it will be available from your domain name directly. You will find more information on our documenation.
Once you have created your bucket, you are ready to upload your files. Run s3cmd sync ./ s3://your-unique-bucket-name/ and the content of the current folder will be automatically updated to your bucket.
By default everything is private so if you browse http://your-unique-bucket-name.cellar-c2.services.clever-cloud.com you should get an XML error saying access denied. To change this and make it readable by everyone, run s3cmd setacl --acl-public s3://your-unique-bucket-name/*. This will grant read access to all anonymous users.
Each file should be accessible separately but you still cannot list the content of the bucket. This is because the bucket itself has to be set to public. Run s3cmd setacl --acl-public s3://your-unique-bucket-name and it will allow you to see the list of objects as XML. You can get a simpler list by adding type-list=2 as query param. So by browsing http://your-unique-bucket-name.cellar-c2.services.clever-cloud.com/?type-list=2, you should see something like:
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name>ldoguinbucket</Name>
<Prefix></Prefix>
<MaxKeys>1000</MaxKeys>
<IsTruncated>false</IsTruncated>
<Contents>
<Key>hello.txt</Key>
<LastModified>2020-10-05T13:07:08.664Z</LastModified>
<ETag>"b1946ac92492d2347c6235b4d2611184"</ETag>
<Size>6</Size>
<StorageClass>STANDARD</StorageClass>
<Owner>
<ID>cellar_0e275449-c02a-4e07-b100-2e357b532e3b</ID>
<DisplayName>cellar_0e275449-c02a-4e07-b100-2e357b532e3b@user_51f7098f-604e-4d9e-8fb6-4026fd1767bb</DisplayName>
</Owner>
<Type>Normal</Type>
</Contents>
<Marker></Marker>
</ListBucketResult>
You can of course adjust the permissions to your liking. Not everything has to be public. You can make that listing look much better and be actually browsable. Here’s how.
Add the Listing
As you have seen it’s easy to get an XML list of your bucket content. You will find many examples on GitHub of single page HTML that fetch the XML list with an XHR request and then renders it nicely. All you have to do is usually upload that files at the root of your bucket and everything should work.
I have tried with https://github.com/qoomon/aws-s3-bucket-browser and it works great. Simply clone this repo and upload all the files at the root of your bucket with the public ACL.
git clone https://github.com/qoomon/aws-s3-bucket-browser/
s3cmd put --acl-public aws-s3-bucket-browser/favicon.ico s3://your-unique-bucket-name/favicon.ico
s3cmd put --acl-public aws-s3-bucket-browser/index.html s3://your-unique-bucket-name/index.html
s3cmd put --acl-public aws-s3-bucket-browser/logo.png s3://your-unique-bucket-name/logo.png
From there you should see a very nice listing when you browse http://your-unique-bucket-name.cellar-c2.services.clever-cloud.com/index.html. Thanks to Bengt Brodersen for providing this 🙂