S3 is Great

Amazon S3 is a pretty cool way to store data online, for just 15 cents a GB/month. The bandwidth's pretty expensive at 10c per GB up, and 17c per GB down but still viable depending on what you're planning on doing with it.

They also have a really slick API to get files to and from the server. One in particular allows clients to directly upload files to your storage space from their machines via HTTP post (so via the browser, Flash, etc.) This is great since your server doesn't have to proxy them to S3. But wait! If you have to authenticate into the service with a secret key on the client side, then you're basically giving away your key. No good.

But this is not the case, and their solution is really well implemented. To do this, you essentially sign an policy that specifies what can be uploaded -- maximum file size, key name, etc. The policy is in a standard format, you base64 encode the policy and send it with a HMAC-SHA-1 hash encrypted with your secret key and encoded using base64.

Perl is Great

Athleon's written in Perl, which I've been extremely happy with on this project. The code below is to encode a policy, generate the SHA-1 hash, and base64 encode it. It uses the MIME::Base64 and Digest::HMAC_SHA1 modules, and looks like this.

my $key = 'my_secret_key_from_amazon'; utf8::encode($acl);

my $base64 = encode_base64($acl); $base64 =~ s/

//igs; #Perl adds newlines

my $hmac = Digest::HMAC_SHA1->new($key); $hmac->add($base64); #encrypt

my $signature = $hmac->b64digest() ; #generate signature

S3 + Perl is Almost Great

Quick and easy, right? Alas, though. This code doesn't work. What gives? I didn't see any documentation anywhere that says this, but looking inside some modules on CPAN, everyone's adding a '=' to the end of the signature.

So I . '=' on the end of the signature, and everything works great. Maybe it has something to do with a spec I don't know about. Or maybe its something obvious that I missed. If anyone knows why, I'd appreciate a comment.

Still though, Amazon did a good job with AWS, and its been an absolute pleasure working with the S3 API so far.

Just a random technical post...