This page is an archive of my old blog. Please visit for my current blog.
This site is no longer being maintained and commenting is disabled.
Currently Browsing: Amazon Webservices

Amazon Developing Persisent Storage for EC2

Amazon just officially announced that they are indeed developing persistent storage for EC2.

This new feature provides reliable, persistent storage volumes, for use with Amazon EC2 instances. These volumes exist independently from any Amazon EC2 instances, and will behave like raw, unformatted hard drives or block devices, which may then be formatted and configured based on the needs of your application. The volumes will be significantly more durable than the local disks within an Amazon EC2 instance. Additionally, our persistent storage feature will enable you to automatically create snapshots of your volumes and back them up to Amazon S3 for even greater reliability.

You will be able to create volumes ranging in size from 1 GB to 1 TB, and will be able to attach multiple volumes to a single instance. Volumes are designed for high throughput, low latency access from Amazon EC2, and can be attached to any running EC2 instance where they will show up as a device inside of the instance. This feature will make it even easier to run everything from relational databases to distributed file systems to Hadoop processing clusters using Amazon EC2.

Amazon has now cleared the second big hurdle for EC (with the first one being static IP’s – which they recently remedied). Now developers will have to start asking the question – “Why am I not using EC2 for my RIA’s?”.

Amazon EC2 Now with Great New Features

Amazon EC2 just received two very important updates: Elastic IP Addresses and Availability Zones.

Elastic IP Addresses are static IP addresses designed for dynamic cloud computing, and now make it easy to host web sites, web services and other online applications in Amazon EC2. Elastic IP addresses are associated with your AWS account, not with your instances, and can be programmatically mapped to any of your instances. This allows you to easily recover from instance and other failures while presenting your users with a static IP address.

This fixes the first major problem that people had with EC2, non-static IP addresses. I have not yet played with this new feature, but I expect to try and give it a demo this week.

Availability Zones give you the ability to easily and inexpensively operate a highly available internet application. Each Amazon EC2 Availability Zone is a distinct location that is engineered to be insulated from failures in other Availability Zones. Previously, only very large companies had the scale to be able to distribute an application across multiple locations, but now it is as easy as changing a parameter in an API call. You can choose to run your application across multiple Availability Zones to be prepared for unexpected events such as power failures or network connectivity issues, or you can place instances in the same Availability Zone to take advantage of free data transfer and the lowest latency communication.

This feature also will have the potential to help many of those high availability sites that use EC2. We recently learned that Amazon Web Services are not perfect (as the S3 system was down for most of a day), but this hopefully should help insulate EC2 customers from such a fate.

Elastic IP Addresses
Availability Zones

Learning from the S3 Outage

Today’s S3 outage taught me quite a bit. I don’t currently have any sites or services running on AWS, but I will have multiple ones up shortly. I am always big on redundancy, but I admit that when I comes to things like EC2 and S3 – I assumed that Amazon had already implemented the redundancy (trust me, I realize that this wasn’t the best viewpoint to take). When it boils down to it – here is the truth: Everything can fail. Second, if it isn’t your service/server/network – you can’t fix it. You just simply have to wait for it to be fixed.

Every site needs a ‘what if’ plan: what if my database server crashes, what if S3 goes down, what if EC2 goes down and I lose all of my server instances? In a world that is moving so fast toward hosted services – we need to remember these things and architect our applications to function (even if not at a full level) when these things fail. Oh, and don’t kid yourself – they will fail.

Amazon’s Newest Webservice – SimpleDB

Amazon has announced their newest webservice, SimpleDB. This service could fill in the gap with EC2 and S3 to offer a completely redundant and scalable solution for robust Internet applications. Amazon has only released a few details, but the public beta will be opening within the next few weeks.

Traditionally, this type of functionality has been accomplished with a clustered relational database that requires a sizable upfront investment, brings more complexity than is typically needed, and often requires a DBA to maintain and administer. In contrast, Amazon SimpleDB is easy to use and provides the core functionality of a database – real-time lookup and simple querying of structured data – without the operational complexity.

Get more information here.