I was consistently hitting the 512MB "limit" immediately, even with lightweight processes (not sure how tbh), and I frequently would see processes complaining that they were at 450% or more of their memory usage.
I'm pretty confident that Heroku virtualizes EC2 instances which potentially makes them more resource strapped than they appear.
They do; if you look at Herou's "dedicated" database instances the RAM numbers correlate perfectly with EC2 instance sizes. Even a "dedicated" Heroku database instance runs on shared EC2 hardware, making both its latency and throughput very unpredictable under any kind of serious workload.
Combined with limitations in how their dyno instances work, like the inability to have more than one slug version deployed at once to take advantage of Heroku's routing fabric for hot-deployment, I only recommend Heroku for early-stage systems. Thankfully it's not very hard to migrate off of and still offers an awesome time savings in the early stages - it's just not something to build out on IMO.
dotCloud (http://dotcloud.com) offers the hot-deployment feature you describe with its new "granite" system. It works for an arbitrary number of parallel containers, too, so you can hot-deploy even when scaled out horizontally.
Heroku virtualizes dynos with linux container (lxc) [1]. The good thing is that lxc is very lightweight because it shares the host's kernel, so dynos aren't slowed by another virtualization abstraction in addition to ec2 xen. The bad thing for heroku users is that memory does not need to provisioned upfront for dynos, so spikes in memory usage of neighboring dynos can eat up all the memory in a physical machine.
I was consistently hitting the 512MB "limit" immediately, even with lightweight processes (not sure how tbh), and I frequently would see processes complaining that they were at 450% or more of their memory usage.
I'm pretty confident that Heroku virtualizes EC2 instances which potentially makes them more resource strapped than they appear.