You may not know this, but each EC2 instance has available to it lots of “local” storage. For example, the Large instances that Ive been using have two 420 GiB volumes available to them. HOWEVER, when you launch the typical “EBS-backed” AMIs from the management console these volumes wont be available to you.
There are up- and downsides to both EBS and Ephemeral volumes, but for me the big win for Ephemeral is that its I/O speed is at least twice as fast as EBS and I dont have to allocate the extra space I usually need. The catch is you have to use the command-line tools to start the instances. Heres the link to the original blog post that clued me in: http://www.webadminblog.com/index.php/2010/03/23/amazon-ec2-ebs-instances-and-ephemeral-storage/