Ganglia

Monitoring clusters and Grids since the year 2000

Introducing Overlay Events

One of the commonly asked Ganglia feature requests has been the ability to overlay events as vertical lines e.g. to show deploys. Unfortunately there was no built in functionality in Ganglia to do that but it had to be “hacked in”. For example in this blog post there is a description of one approach. Fortunately that is now history as we have added “Overlay Events”. This is a generic feature that allows you to specify a list of events including time (unix timestamp) and description as well as grid, cluster and host regex that this event applies to. This way you can limit an overlay event to a subset of hosts e.g. DB backup affects only the DB slave server. You will end up with something like this Overlay events To enable overlay events add following to your conf.php
$conf['overlay_events'] = true;
Events are configured using a simple JSON array. By default events are stored in the following file
$conf['overlay_events_file'] = $conf['conf_dir'] . "/events.json";
If you are using defaults that is /var/lib/ganglia/conf/events.json Example of the events JSON file used to create the above overlay looks like this
[
 {"start_time":1308496361,
 "summary":"DB Backup",
 "description":"Prod daily db backup",
 "grid":"*",
 "cluster":"*",
 "host_regex":"centos1"},
 {"start_time":1308497211,
 "summary":"FS cleanup",
 "grid":"*",
 "cluster":"*",
 "host_regex":"centos1"}
]
Currently only host_regex is supported but we are working on adding filtering by grid and cluster. All you now need to do is decide which events to include. Example events you can include
  • Start time of particular periodic jobs such as DB backups, DB clean ups
  • Deploys
  • Nagios alerts sent
Alternatively you can try the Events API e.g. I have added following command to be executed before my critical jobs start
wget -O /dev/null -q "http://mygangliahost.com/ganglia/api/events_api.php?action=add&start_time=now&summary=Prod DB Backup&host_regex=db02"
Change the start_time to a UNIX timestamp or a well formed date. To download the latest release with Overlay events please visit https://github.com/vvuksan/ganglia-misc