Corrade metrics can be obtained via the heartbeat notification. To be able to watch the evolution over time for the heartbeat metrics, one can use the Apache Elasticsearch backend to collect the metrics and then Apache Kibana to plot the metrics. Perhaps, the reason for using Elasticsearch and Kibana is due to the popularity of the software combination as they are used in various large scale computer clustering scenarios.

The tutorial on this page uses a small Node.JS script to ferry the metrics from Corrade via MQTT over to elasticsearch via a HTTP connection. Kibana is then used as the interfaces, where the ferried data can be indexed and graphs generated to monitor Corrade's performance.

• Corrade (continuous preferred),
• Node.JS,
• Elasticsearch,
• Kibana

The setup is split into two parts:

• installing the Corrade Elasticsearch template,
• installing Corrade, Elasticsearch and Kibana.

The final step is to install Node.JS using the package manager and checkout the Corrade Elasticsearch template via Suversion from the address:

• http://svn.grimore.org/corrade-elasticsearch-heartbeat

either using a Subversion client such as TortoiseSVN (Windows) or by issuing the command:

svn co http://svn.grimore.org/corrade-elasticsearch-heartbeat

on Linux systems with Subversion installed.

The corrade-elasticsearch-heartbeat template requires Node.JS modules to be installed. Change directory to corrade-elasticsearch-heartbeat:

cd corrade-elasticsearch-heartbeat

and issue the command:

npm install

which should pull all the required Node.JS modules for the template.

Now, copy the config.yml.dist to config.yml inside the corrade-elasticsearch-heartbeat folder and edit config.yml to adjust the settings. The following settings have to be made:

• for Corrade:
  • group - this should be set to the group name configured with Corrade,
  • password - the group password,
  • mqtt - this should be set to the MQTT server enabled via Corrade-Nucleus,
  • name - this is an identifier to track the data, which can conveniently be set to the account name for the Corrade scripted agent.
• for Elasticsearch:
  • connect - this should be set to the elasticsearch API URL (including any created users and passwords),
  • index - should be set to the name of the index to be created on Elasticsearch (this can be left as-is, for now).

The template setup is now complete.

Corrade can be obtained from the download website and requires to be setup with the heartbeat notification enabled for a configured group using Nucleus. Similarly, the MQTT server has to be enabled and configured to listen on a chosen address and port. For the tutorial, it can be assumed that the configured address and port are 127.0.0.1 respectively 8095 for the MQTT server.

Moving on, the Elasticsearch and Kibana packages can usually be obtained via the Linux distribution package manager or downloaded from their official websites. Elasticsearch and Kibana are configured out of the box to listen on 127.0.0.1 and the default Elasticsearch port. Next, loading the Kibana interface by browsing to: http://127.0.0.1:5601 on the machine where Kibana and Elasticsearch are installed, locate the settings button on the left-hand side and select Import.

Using the slide-in pane, upload a file with the following contents:

export.ndjson

{"attributes":{"fields":"[{\"name\":\"AgentName\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"AgentName.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"parent\":\"AgentName\",\"subType\":\"multi\"},{\"name\":\"AverageCPUUsage\",\"type\":\"number\",\"esTypes\":[\"long\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"AverageRAMUsage\",\"type\":\"number\",\"esTypes\":[\"long\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"AverageThreadsUsage\",\"type\":\"number\",\"esTypes\":[\"long\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"ExecutingCommands\",\"type\":\"number\",\"esTypes\":[\"long\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"ManifestingRLVBehaviours\",\"type\":\"number\",\"esTypes\":[\"long\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"ProcessedCommands\",\"type\":\"number\",\"esTypes\":[\"long\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"ProcessedRLVBehaviours\",\"type\":\"number\",\"esTypes\":[\"long\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"StartTime\",\"type\":\"date\",\"esTypes\":[\"date\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"TimeNow\",\"type\":\"date\",\"esTypes\":[\"date\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"Uptime\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"Uptime.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"parent\":\"Uptime\",\"subType\":\"multi\"},{\"name\":\"Version\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"Version.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"parent\":\"Version\",\"subType\":\"multi\"},{\"name\":\"_id\",\"type\":\"string\",\"esTypes\":[\"_id\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"_index\",\"type\":\"string\",\"esTypes\":[\"_index\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"_score\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_source\",\"type\":\"_source\",\"esTypes\":[\"_source\"],\"count\":0,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_type\",\"type\":\"string\",\"esTypes\":[\"_type\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false}]","timeFieldName":"TimeNow","title":"corrade-heartbeat"},"id":"5f660a00-b65c-11e9-bed3-1dfd40700947","migrationVersion":{"index-pattern":"6.5.0"},"references":[],"type":"index-pattern","updated_at":"2019-08-04T02:05:47.751Z","version":"WzU0LDhd"}
{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"language\":\"kuery\",\"query\":\"\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"title":"Ectogram Resident (CPU Usage Distribution)","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Ectogram Resident (CPU Usage Distribution)\",\"type\":\"histogram\",\"params\":{\"type\":\"histogram\",\"grid\":{\"categoryLines\":true,\"valueAxis\":\"ValueAxis-1\"},\"categoryAxes\":[{\"id\":\"CategoryAxis-1\",\"type\":\"category\",\"position\":\"bottom\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\"},\"labels\":{\"show\":true,\"truncate\":100},\"title\":{}}],\"valueAxes\":[{\"id\":\"ValueAxis-1\",\"name\":\"LeftAxis-1\",\"type\":\"value\",\"position\":\"left\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\",\"mode\":\"normal\"},\"labels\":{\"show\":true,\"rotate\":0,\"filter\":false,\"truncate\":100},\"title\":{\"text\":\"Hits\"}}],\"seriesParams\":[{\"show\":true,\"mode\":\"stacked\",\"type\":\"histogram\",\"drawLinesBetweenPoints\":true,\"showCircles\":true,\"data\":{\"id\":\"2\",\"label\":\"Hits\"},\"valueAxis\":\"ValueAxis-1\"}],\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"bottom\",\"times\":[],\"addTimeMarker\":false,\"dimensions\":{\"x\":{\"accessor\":0,\"format\":{\"id\":\"terms\",\"params\":{\"id\":\"number\",\"otherBucketLabel\":\"Other\",\"missingBucketLabel\":\"Missing\"}},\"params\":{},\"aggType\":\"terms\"},\"y\":[{\"accessor\":1,\"format\":{\"id\":\"number\"},\"params\":{},\"aggType\":\"count\"}]},\"orderBucketsBySum\":false},\"aggs\":[{\"id\":\"2\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{\"customLabel\":\"Hits\"}},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"AverageCPUUsage\",\"order\":\"asc\",\"size\":20,\"orderBy\":\"_key\",\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"CPU (%)\"}}]}"},"id":"f9752040-b65c-11e9-bed3-1dfd40700947","migrationVersion":{"visualization":"7.2.0"},"references":[{"id":"5f660a00-b65c-11e9-bed3-1dfd40700947","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"visualization","updated_at":"2019-08-04T02:15:14.991Z","version":"WzU5LDhd"}
{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"language\":\"kuery\",\"query\":\"\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"title":"Ectogram Resident (RAM Usage over Time)","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Ectogram Resident (RAM Usage over Time)\",\"type\":\"histogram\",\"params\":{\"addLegend\":true,\"addTimeMarker\":true,\"addTooltip\":true,\"categoryAxes\":[{\"id\":\"CategoryAxis-1\",\"labels\":{\"show\":true,\"truncate\":100,\"rotate\":75},\"position\":\"bottom\",\"scale\":{\"type\":\"linear\"},\"show\":true,\"style\":{},\"title\":{},\"type\":\"category\"}],\"dimensions\":{\"x\":{\"accessor\":0,\"format\":{\"id\":\"date\",\"params\":{\"pattern\":\"HH:mm\"}},\"params\":{\"date\":true,\"interval\":\"PT1M\",\"format\":\"HH:mm\",\"bounds\":{\"min\":\"2019-08-04T04:31:57.513Z\",\"max\":\"2019-08-04T04:46:57.513Z\"}},\"aggType\":\"date_histogram\"},\"y\":[{\"accessor\":1,\"format\":{\"id\":\"number\"},\"params\":{},\"aggType\":\"avg\"}]},\"grid\":{\"categoryLines\":true,\"valueAxis\":\"ValueAxis-1\"},\"legendPosition\":\"bottom\",\"seriesParams\":[{\"data\":{\"id\":\"2\",\"label\":\"RAM (bytes)\"},\"drawLinesBetweenPoints\":true,\"lineWidth\":5,\"mode\":\"stacked\",\"show\":true,\"showCircles\":true,\"type\":\"line\",\"valueAxis\":\"ValueAxis-1\"}],\"times\":[],\"type\":\"histogram\",\"valueAxes\":[{\"id\":\"ValueAxis-1\",\"labels\":{\"filter\":false,\"rotate\":0,\"show\":true,\"truncate\":100},\"name\":\"LeftAxis-1\",\"position\":\"left\",\"scale\":{\"defaultYExtents\":false,\"max\":500000000,\"min\":500000,\"mode\":\"normal\",\"setYExtents\":true,\"type\":\"square root\"},\"show\":true,\"style\":{},\"title\":{\"text\":\"RAM (bytes)\"},\"type\":\"value\"}]},\"aggs\":[{\"id\":\"2\",\"enabled\":true,\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"AverageRAMUsage\",\"customLabel\":\"RAM (bytes)\"}},{\"id\":\"3\",\"enabled\":true,\"type\":\"date_histogram\",\"schema\":\"segment\",\"params\":{\"field\":\"TimeNow\",\"timeRange\":{\"from\":\"now-15m\",\"to\":\"now\"},\"useNormalizedEsInterval\":true,\"interval\":\"m\",\"drop_partials\":false,\"min_doc_count\":1,\"extended_bounds\":{},\"customLabel\":\"Time\"}}]}"},"id":"738b6600-b65d-11e9-bed3-1dfd40700947","migrationVersion":{"visualization":"7.2.0"},"references":[{"id":"5f660a00-b65c-11e9-bed3-1dfd40700947","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"visualization","updated_at":"2019-08-04T04:47:53.843Z","version":"WzYyLDhd"}
{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"title":"Ectogram Resident (Total Commands and Behaviours)","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Ectogram Resident (Total Commands and Behaviours)\",\"type\":\"metric\",\"params\":{\"addTooltip\":true,\"addLegend\":false,\"type\":\"metric\",\"metric\":{\"percentageMode\":false,\"useRanges\":false,\"colorSchema\":\"Green to Red\",\"metricColorMode\":\"None\",\"colorsRange\":[{\"from\":0,\"to\":10000}],\"labels\":{\"show\":true},\"invertColors\":false,\"style\":{\"bgFill\":\"#000\",\"bgColor\":false,\"labelColor\":false,\"subText\":\"\",\"fontSize\":60}},\"dimensions\":{\"metrics\":[{\"accessor\":0,\"format\":{\"id\":\"number\"},\"params\":{},\"aggType\":\"max\"}]}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"max\",\"schema\":\"metric\",\"params\":{\"field\":\"ProcessedCommands\",\"customLabel\":\"Commands\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"max\",\"schema\":\"metric\",\"params\":{\"field\":\"ProcessedRLVBehaviours\",\"customLabel\":\"Behaviours\"}}]}"},"id":"15d234f0-b674-11e9-bed3-1dfd40700947","migrationVersion":{"visualization":"7.2.0"},"references":[{"id":"5f660a00-b65c-11e9-bed3-1dfd40700947","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"visualization","updated_at":"2019-08-04T04:55:31.007Z","version":"WzYzLDhd"}
{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Ectogram Resident (RAM Moving Average)","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Ectogram Resident (RAM Moving Average)\",\"type\":\"timelion\",\"params\":{\"expression\":\".es(index=corrade-heartbeat, timefield='TimeNow', metric='max:AverageRAMUsage').mvavg(60).label('RAM Moving Average').yaxis(label=\\\"RAM (bytes)\\\").color('red')\",\"interval\":\"auto\"},\"aggs\":[]}"},"id":"4c0ddb80-b67b-11e9-8ca2-dd0fee42a4dc","migrationVersion":{"visualization":"7.2.0"},"references":[],"type":"visualization","updated_at":"2019-08-04T05:55:49.638Z","version":"WzY4LDhd"}
{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"language\":\"kuery\",\"query\":\"\"},\"filter\":[]}"},"title":"Ectogram Resident (Moving Average Theads Usage)","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Ectogram Resident (Moving Average Theads Usage)\",\"type\":\"timelion\",\"params\":{\"expression\":\".es(index=corrade-heartbeat, timefield='TimeNow', metric='max:AverageThreadsUsage').mvavg(60).label('Threads Moving Average').yaxis(label=\\\"Threads\\\").color('red')\",\"interval\":\"auto\"},\"aggs\":[]}"},"id":"4c065af0-b7fb-11e9-bbd7-6db8670da417","migrationVersion":{"visualization":"7.2.0"},"references":[],"type":"visualization","updated_at":"2019-08-06T03:36:50.691Z","version":"WzgxLDld"}
{"attributes":{"description":"","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"language\":\"kuery\",\"query\":\"\"},\"filter\":[]}"},"optionsJSON":"{\"hidePanelTitles\":false,\"useMargins\":true}","panelsJSON":"[{\"embeddableConfig\":{},\"gridData\":{\"x\":0,\"y\":0,\"w\":24,\"h\":15,\"i\":\"1\"},\"panelIndex\":\"1\",\"version\":\"7.2.0\",\"panelRefName\":\"panel_0\"},{\"embeddableConfig\":{},\"gridData\":{\"x\":24,\"y\":0,\"w\":24,\"h\":15,\"i\":\"2\"},\"panelIndex\":\"2\",\"version\":\"7.2.0\",\"panelRefName\":\"panel_1\"},{\"embeddableConfig\":{},\"gridData\":{\"x\":0,\"y\":30,\"w\":24,\"h\":15,\"i\":\"3\"},\"panelIndex\":\"3\",\"version\":\"7.2.0\",\"panelRefName\":\"panel_2\"},{\"embeddableConfig\":{},\"gridData\":{\"x\":24,\"y\":15,\"w\":24,\"h\":15,\"i\":\"4\"},\"panelIndex\":\"4\",\"version\":\"7.2.0\",\"panelRefName\":\"panel_3\"},{\"embeddableConfig\":{},\"gridData\":{\"x\":0,\"y\":15,\"w\":24,\"h\":15,\"i\":\"5\"},\"panelIndex\":\"5\",\"version\":\"7.2.0\",\"panelRefName\":\"panel_4\"}]","refreshInterval":{"pause":false,"value":60000},"timeFrom":"now-6h","timeRestore":true,"timeTo":"now","title":"Ectogram Resident (Performance)","version":1},"id":"3c4cbda0-b672-11e9-bed3-1dfd40700947","migrationVersion":{"dashboard":"7.0.0"},"references":[{"id":"f9752040-b65c-11e9-bed3-1dfd40700947","name":"panel_0","type":"visualization"},{"id":"738b6600-b65d-11e9-bed3-1dfd40700947","name":"panel_1","type":"visualization"},{"id":"15d234f0-b674-11e9-bed3-1dfd40700947","name":"panel_2","type":"visualization"},{"id":"4c0ddb80-b67b-11e9-8ca2-dd0fee42a4dc","name":"panel_3","type":"visualization"},{"id":"4c065af0-b7fb-11e9-bbd7-6db8670da417","name":"panel_4","type":"visualization"}],"type":"dashboard","updated_at":"2019-08-06T03:39:02.618Z","version":"WzgyLDld"}

which should create:

• an index named corrade-heartbeat,
• a few visualizations, and
• a dashboard.

Going back to the template in the corrade-elasticsearch-heartbeat folder, copy the file at contrib/linux/systemd/corrade-elasticsearch-heatbeat.service into /etc/systemd/system and issue the command:

systemctl enable corrade-elasticsearch-heatbeat

to enable the service file, followed by:

systemctl start corrade-elasticsearch-heartbeat

to start the template.

If everything has been setup correctly, selecting the dashboard button, should show a few pre-made visualizations tracking various statistics such as (from left to right):

• a CPU usage distribution,
• an overall RAM usage over time,
• a moving average of used threads,
• a moving average of RAM usage,
• a display showing the total number of executed commands and RLV behaviours.

• Elasticsearch and Kibana are made to track very large amounts of data such that, counter-intuitively, you will not be able to generate discrete plots. In other words, Kibana works with averages, sumations, median values and cannot, for instance, plot RAM usage over time for discrete points - this is by design, since discrete plots for very large amounts of data would not be feasible.
• More advanced maps can be created with Kibana using the Timelion plot that allows for formulas and manipulation of data directly without using interface elements to build a visualization.
• Tip Corrade has a map notification that can transmit map tiles which can probably be coupled with Elasticsearch and Kibana to show a map overview of Corrade's location on the grid.