In a previous article, I detailed the open source projects that I used to implement a PaaS infrastructure.
Since that time the number of instances in the infrastructure has grown by 2.5X and several of the components needed to be rethought.
|Reasons:||The collectd backend was too slow and I/O heavy
Graphite graphs are easily to embed in dashboard applications
Ability to easily transform metrics, such as average CPU across a cluster of servers
|Reasons:||Selenium tests failed too often for undiscernable reasons
False positives slowed development too often
|Reasons:||Mongodb too slow in EC2 for storing and searching
Logstash offers better parsing and indexing of logs with powerful filtersElasticSearch is super fast and scales horizontally on EC2
Kibana is simple to use and allows Developers to quickly find the relevant information
All of these components are easily integrated into our dashboard application
These changes not only allow the infrastructure to scale, but provide APIs that allow easy integration with custom dashboards.