If this is a business project, it sounds like speaking with an experienced technical person could save a great deal of pain.
Here's some examples:
1) Reverse proxy grab bag:
- Remove dead backends automatically. Duh.
- Never show internal error pages publicly, show entertaining pages instead that mean different things internally. Graceful degradation/feature darkmode for +2 overcapacity awesome.
- Sanitized X-Backend-Server host header is often useful.
- Use a combination of haproxy, nginx, varnish and/or ipvs (ldirectord/keepalived) up front. If resources are plentiful, opt for fancy stuff netscaler or f5. I like HA pfsense for cost-containment. Also, I can't say enough good things about varnish for awesomeness.
2) For anything substantial, consider a J2EE container that supports jopr/jon or similar monitoring tool that does J2EE well.
3) Hosted HA MySQL can have latency issues and should be stress tested.
Non-technical:
0) The number of servers doesn't matter as long as the number is always greater than 1 and no resource is completely saturated. Scale with a capacity load-predicting cloud controller automatically.
1) Architecture should be allowed to evolve and respond to pain-points rather than being planned and optimized too soon (except when it comes to security and disaster recovery). Idle servers are wasted money by definition.
2) Start small. Don't go for big solutions right away unless your team works with it on a daily basis. Complicated breaks and is insane to diagnose. (Any guess why I know this?)
3) Don't promote boxes from dev to qa to prod. Promote build artifacts and configuration management scripts that are the same (except passwords, ip, lun wwn, etc.) across all environments. Deploy fresh everything every time. (Many reasons for this.)
4) Run nonfunctional tests on infrastructure, including performance, automatically. Fail the build if below thresholds. Must mention versioning code and system infrastructure for this to work effectively.
Barry