One of the commonly asked Ganglia feature requests has been the ability to overlay events as vertical lines e.g. to show deploys. Unfortunately there was no built in functionality in Ganglia to do that but it had to be “hacked in”. For example
in this blog post there is a description of one approach. Fortunately that is now history as we have added “Overlay Events”. This is a generic feature that allows you to specify a list of events including time (unix timestamp) and description as well as grid, cluster and host regex that this event applies to. This way you can limit an overlay event to a subset of hosts e.g. DB backup affects only the DB slave server. You will end up with something like this
To enable overlay events add following to your conf.php
$conf['overlay_events'] = true;
Events are configured using a simple JSON array. By default events are stored in the following file
$conf['overlay_events_file'] = $conf['conf_dir'] . "/events.json";
If you are using defaults that is
/var/lib/ganglia/conf/events.json
Example of the events JSON file used to create the above overlay looks like this
[
{"start_time":1308496361,
"summary":"DB Backup",
"description":"Prod daily db backup",
"grid":"*",
"cluster":"*",
"host_regex":"centos1"},
{"start_time":1308497211,
"summary":"FS cleanup",
"grid":"*",
"cluster":"*",
"host_regex":"centos1"}
]
Currently only host_regex is supported but we are working on adding filtering by grid and cluster. All you now need to do is decide which events to include. Example events you can include
- Start time of particular periodic jobs such as DB backups, DB clean ups
- Deploys
- Nagios alerts sent
Alternatively you can try the Events API e.g. I have added following command to be executed before my critical jobs start
wget -O /dev/null -q "http://mygangliahost.com/ganglia/api/events_api.php?action=add&start_time=now&summary=Prod DB Backup&host_regex=db02"
Change the start_time to a UNIX timestamp or a well formed date.
To download the latest release with Overlay events please visit
https://github.com/vvuksan/ganglia-misc
This provides you with immediate context and allows you to better correlate metrics. It may also provide you with additional insight. Let’s say you saw something like this
It may not be DB backup that is causing the load and you may want to investigate.
To use event time line all you need to do is supply both start and end time of the event e.g.
Stacked graph
Next steps
We need to add more error checking and bug fixes. Better composer UI and ability to add aggregate graphs to views. Stay tuned.
If you’d like to play with you can try it on our