A good cloud developer could do something like monitor the server load on a web head and submit an API call to spin up another server behind the load balancer if load goes over 1 (or connection count exceeds 1000, or apache processes are within 5% of MaxClients). This is something you just can't do with dedicated hardware.
You don't have to be a good developer at all, for that. Any code monkey could get that set up.
The bottleneck there is that you have to decide
to pay for the added servers, which is not a developer issue. The issue there is that you need your main management, bean counter(s), admins, programmers,
and infrastructure providers to all be on the same page wrt to the issue, and make a distinct point to implement it as an automatic feature. It's not a problem that would be hard for a developer, so much as it is being willing to make a decision in which program behavior can add to your costs,
without management making a specific decision to spend that money.
I'm sure some SaaS providers have exactly that going on, once they're sure that the load will only be increased when it involves more paying customers.
With dedicated hardware, you tune your storage so that you don't need to do that, with low-utilization servers, and/or cold/warm spares, if keeping them on and ready all the time costs too much. Once you're paying for redundant A/C units, a dedicated room, redundant internet with a SLA, etc., having more server capacity than you think you'll need is
cheap. It's not that you can't handle higher load (marketing crap!), but that handling real loads in-house involves a significant expenditure up-front (possibly so with a colo, as well, even beyond the servers), where IaaS providers will give you operational costs only (capex v. opex).