A How to Guide on Modern Monitoring and Alerting
At our organization (CCIN2P3) we are building an event-based infrastructure to push structured messages to different subsystems for alerting, reporting and storage. Using syslog-ng, each message is normalized into a structured event, optionally correlated with other messages, and conditionally routed to the next systems, including:
- a synchronous web-dashboard,
- different asynchronous alerting systems, and
- a searchable storage backend.
The events which are collected are essentially system and application logs. Here’s a few examples of interesting messages:
