Corrade metrics can be obtained via the heartbeat notification. To be able to watch the evolution over time for the heartbeat metrics, one can use the Apache Elasticsearch backend to collect the metrics and then Apache Kibana to plot the metrics. Perhaps, the reason for using Elasticsearch and Kibana is due to the popularity of the software combination as they are used in various large scale computer clustering scenarios.
The tutorial on this page uses a small Node.JS script to ferry the metrics from Corrade via MQTT over to elasticsearch via a HTTP connection. Kibana is then used as the interfaces, where the ferried data can be indexed and graphs generated to monitor Corrade's performance.
The setup is split into two parts:
The final step is to install Node.JS using the package manager and checkout the Corrade Elasticsearch template via Suversion from the address:
either using a Subversion client such as TortoiseSVN (Windows) or by issuing the command:
svn co http://svn.grimore.org/corrade-elasticsearch-heartbeat
on Linux systems with Subversion installed.
The corrade-elasticsearch-heartbeat
template requires Node.JS modules to be installed. Change directory to corrade-elasticsearch-heartbeat
:
cd corrade-elasticsearch-heartbeat
and issue the command:
npm install
which should pull all the required Node.JS modules for the template.
Now, copy the config.yml.dist
to config.yml
inside the corrade-elasticsearch-heartbeat
folder and edit config.yml
to adjust the settings. The following settings have to be made:
group
- this should be set to the group name configured with Corrade,password
- the group password,mqtt
- this should be set to the MQTT server enabled via Corrade-Nucleus,name
- this is an identifier to track the data, which can conveniently be set to the account name for the Corrade scripted agent.connect
- this should be set to the elasticsearch API URL (including any created users and passwords),index
- should be set to the name of the index to be created on Elasticsearch (this can be left as-is, for now).The template setup is now complete.
Corrade can be obtained from the download website and requires to be setup with the heartbeat
notification enabled for a configured group using Nucleus. Similarly, the MQTT server has to be enabled and configured to listen on a chosen address and port. For the tutorial, it can be assumed that the configured address and port are 127.0.0.1
respectively 8095
for the MQTT server.
Moving on, the Elasticsearch and Kibana packages can usually be obtained via the Linux distribution package manager or downloaded from their official websites. Elasticsearch and Kibana are configured out of the box to listen on 127.0.0.1
and the default Elasticsearch port. Next, loading the Kibana interface by browsing to: http://127.0.0.1:5601
on the machine where Kibana and Elasticsearch are installed, locate the settings button on the left-hand side and select Import
.
Using the slide-in pane, upload a file with the following contents:
{"attributes":{"fields":"[{\"name\":\"AgentName\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"AgentName.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"parent\":\"AgentName\",\"subType\":\"multi\"},{\"name\":\"AverageCPUUsage\",\"type\":\"number\",\"esTypes\":[\"long\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"AverageRAMUsage\",\"type\":\"number\",\"esTypes\":[\"long\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"AverageThreadsUsage\",\"type\":\"number\",\"esTypes\":[\"long\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"ExecutingCommands\",\"type\":\"number\",\"esTypes\":[\"long\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"ManifestingRLVBehaviours\",\"type\":\"number\",\"esTypes\":[\"long\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"ProcessedCommands\",\"type\":\"number\",\"esTypes\":[\"long\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"ProcessedRLVBehaviours\",\"type\":\"number\",\"esTypes\":[\"long\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"StartTime\",\"type\":\"date\",\"esTypes\":[\"date\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"TimeNow\",\"type\":\"date\",\"esTypes\":[\"date\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"Uptime\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"Uptime.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"parent\":\"Uptime\",\"subType\":\"multi\"},{\"name\":\"Version\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"Version.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"parent\":\"Version\",\"subType\":\"multi\"},{\"name\":\"_id\",\"type\":\"string\",\"esTypes\":[\"_id\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"_index\",\"type\":\"string\",\"esTypes\":[\"_index\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"_score\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_source\",\"type\":\"_source\",\"esTypes\":[\"_source\"],\"count\":0,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_type\",\"type\":\"string\",\"esTypes\":[\"_type\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false}]","timeFieldName":"TimeNow","title":"corrade-heartbeat"},"id":"5f660a00-b65c-11e9-bed3-1dfd40700947","migrationVersion":{"index-pattern":"6.5.0"},"references":[],"type":"index-pattern","updated_at":"2019-08-04T02:05:47.751Z","version":"WzU0LDhd"} {"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"language\":\"kuery\",\"query\":\"\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"title":"Ectogram Resident (CPU Usage Distribution)","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Ectogram Resident (CPU Usage Distribution)\",\"type\":\"histogram\",\"params\":{\"type\":\"histogram\",\"grid\":{\"categoryLines\":true,\"valueAxis\":\"ValueAxis-1\"},\"categoryAxes\":[{\"id\":\"CategoryAxis-1\",\"type\":\"category\",\"position\":\"bottom\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\"},\"labels\":{\"show\":true,\"truncate\":100},\"title\":{}}],\"valueAxes\":[{\"id\":\"ValueAxis-1\",\"name\":\"LeftAxis-1\",\"type\":\"value\",\"position\":\"left\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\",\"mode\":\"normal\"},\"labels\":{\"show\":true,\"rotate\":0,\"filter\":false,\"truncate\":100},\"title\":{\"text\":\"Hits\"}}],\"seriesParams\":[{\"show\":true,\"mode\":\"stacked\",\"type\":\"histogram\",\"drawLinesBetweenPoints\":true,\"showCircles\":true,\"data\":{\"id\":\"2\",\"label\":\"Hits\"},\"valueAxis\":\"ValueAxis-1\"}],\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"bottom\",\"times\":[],\"addTimeMarker\":false,\"dimensions\":{\"x\":{\"accessor\":0,\"format\":{\"id\":\"terms\",\"params\":{\"id\":\"number\",\"otherBucketLabel\":\"Other\",\"missingBucketLabel\":\"Missing\"}},\"params\":{},\"aggType\":\"terms\"},\"y\":[{\"accessor\":1,\"format\":{\"id\":\"number\"},\"params\":{},\"aggType\":\"count\"}]},\"orderBucketsBySum\":false},\"aggs\":[{\"id\":\"2\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{\"customLabel\":\"Hits\"}},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"AverageCPUUsage\",\"order\":\"asc\",\"size\":20,\"orderBy\":\"_key\",\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"CPU (%)\"}}]}"},"id":"f9752040-b65c-11e9-bed3-1dfd40700947","migrationVersion":{"visualization":"7.2.0"},"references":[{"id":"5f660a00-b65c-11e9-bed3-1dfd40700947","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"visualization","updated_at":"2019-08-04T02:15:14.991Z","version":"WzU5LDhd"} {"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"language\":\"kuery\",\"query\":\"\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"title":"Ectogram Resident (RAM Usage over Time)","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Ectogram Resident (RAM Usage over Time)\",\"type\":\"histogram\",\"params\":{\"addLegend\":true,\"addTimeMarker\":true,\"addTooltip\":true,\"categoryAxes\":[{\"id\":\"CategoryAxis-1\",\"labels\":{\"show\":true,\"truncate\":100,\"rotate\":75},\"position\":\"bottom\",\"scale\":{\"type\":\"linear\"},\"show\":true,\"style\":{},\"title\":{},\"type\":\"category\"}],\"dimensions\":{\"x\":{\"accessor\":0,\"format\":{\"id\":\"date\",\"params\":{\"pattern\":\"HH:mm\"}},\"params\":{\"date\":true,\"interval\":\"PT1M\",\"format\":\"HH:mm\",\"bounds\":{\"min\":\"2019-08-04T04:31:57.513Z\",\"max\":\"2019-08-04T04:46:57.513Z\"}},\"aggType\":\"date_histogram\"},\"y\":[{\"accessor\":1,\"format\":{\"id\":\"number\"},\"params\":{},\"aggType\":\"avg\"}]},\"grid\":{\"categoryLines\":true,\"valueAxis\":\"ValueAxis-1\"},\"legendPosition\":\"bottom\",\"seriesParams\":[{\"data\":{\"id\":\"2\",\"label\":\"RAM (bytes)\"},\"drawLinesBetweenPoints\":true,\"lineWidth\":5,\"mode\":\"stacked\",\"show\":true,\"showCircles\":true,\"type\":\"line\",\"valueAxis\":\"ValueAxis-1\"}],\"times\":[],\"type\":\"histogram\",\"valueAxes\":[{\"id\":\"ValueAxis-1\",\"labels\":{\"filter\":false,\"rotate\":0,\"show\":true,\"truncate\":100},\"name\":\"LeftAxis-1\",\"position\":\"left\",\"scale\":{\"defaultYExtents\":false,\"max\":500000000,\"min\":500000,\"mode\":\"normal\",\"setYExtents\":true,\"type\":\"square root\"},\"show\":true,\"style\":{},\"title\":{\"text\":\"RAM (bytes)\"},\"type\":\"value\"}]},\"aggs\":[{\"id\":\"2\",\"enabled\":true,\"type\":\"avg\",\"schema\":\"metric\",\"params\":{\"field\":\"AverageRAMUsage\",\"customLabel\":\"RAM (bytes)\"}},{\"id\":\"3\",\"enabled\":true,\"type\":\"date_histogram\",\"schema\":\"segment\",\"params\":{\"field\":\"TimeNow\",\"timeRange\":{\"from\":\"now-15m\",\"to\":\"now\"},\"useNormalizedEsInterval\":true,\"interval\":\"m\",\"drop_partials\":false,\"min_doc_count\":1,\"extended_bounds\":{},\"customLabel\":\"Time\"}}]}"},"id":"738b6600-b65d-11e9-bed3-1dfd40700947","migrationVersion":{"visualization":"7.2.0"},"references":[{"id":"5f660a00-b65c-11e9-bed3-1dfd40700947","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"visualization","updated_at":"2019-08-04T04:47:53.843Z","version":"WzYyLDhd"} {"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"title":"Ectogram Resident (Total Commands and Behaviours)","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Ectogram Resident (Total Commands and Behaviours)\",\"type\":\"metric\",\"params\":{\"addTooltip\":true,\"addLegend\":false,\"type\":\"metric\",\"metric\":{\"percentageMode\":false,\"useRanges\":false,\"colorSchema\":\"Green to Red\",\"metricColorMode\":\"None\",\"colorsRange\":[{\"from\":0,\"to\":10000}],\"labels\":{\"show\":true},\"invertColors\":false,\"style\":{\"bgFill\":\"#000\",\"bgColor\":false,\"labelColor\":false,\"subText\":\"\",\"fontSize\":60}},\"dimensions\":{\"metrics\":[{\"accessor\":0,\"format\":{\"id\":\"number\"},\"params\":{},\"aggType\":\"max\"}]}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"max\",\"schema\":\"metric\",\"params\":{\"field\":\"ProcessedCommands\",\"customLabel\":\"Commands\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"max\",\"schema\":\"metric\",\"params\":{\"field\":\"ProcessedRLVBehaviours\",\"customLabel\":\"Behaviours\"}}]}"},"id":"15d234f0-b674-11e9-bed3-1dfd40700947","migrationVersion":{"visualization":"7.2.0"},"references":[{"id":"5f660a00-b65c-11e9-bed3-1dfd40700947","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"visualization","updated_at":"2019-08-04T04:55:31.007Z","version":"WzYzLDhd"} {"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Ectogram Resident (RAM Moving Average)","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Ectogram Resident (RAM Moving Average)\",\"type\":\"timelion\",\"params\":{\"expression\":\".es(index=corrade-heartbeat, timefield='TimeNow', metric='max:AverageRAMUsage').mvavg(60).label('RAM Moving Average').yaxis(label=\\\"RAM (bytes)\\\").color('red')\",\"interval\":\"auto\"},\"aggs\":[]}"},"id":"4c0ddb80-b67b-11e9-8ca2-dd0fee42a4dc","migrationVersion":{"visualization":"7.2.0"},"references":[],"type":"visualization","updated_at":"2019-08-04T05:55:49.638Z","version":"WzY4LDhd"} {"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"language\":\"kuery\",\"query\":\"\"},\"filter\":[]}"},"title":"Ectogram Resident (Moving Average Theads Usage)","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Ectogram Resident (Moving Average Theads Usage)\",\"type\":\"timelion\",\"params\":{\"expression\":\".es(index=corrade-heartbeat, timefield='TimeNow', metric='max:AverageThreadsUsage').mvavg(60).label('Threads Moving Average').yaxis(label=\\\"Threads\\\").color('red')\",\"interval\":\"auto\"},\"aggs\":[]}"},"id":"4c065af0-b7fb-11e9-bbd7-6db8670da417","migrationVersion":{"visualization":"7.2.0"},"references":[],"type":"visualization","updated_at":"2019-08-06T03:36:50.691Z","version":"WzgxLDld"} {"attributes":{"description":"","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"language\":\"kuery\",\"query\":\"\"},\"filter\":[]}"},"optionsJSON":"{\"hidePanelTitles\":false,\"useMargins\":true}","panelsJSON":"[{\"embeddableConfig\":{},\"gridData\":{\"x\":0,\"y\":0,\"w\":24,\"h\":15,\"i\":\"1\"},\"panelIndex\":\"1\",\"version\":\"7.2.0\",\"panelRefName\":\"panel_0\"},{\"embeddableConfig\":{},\"gridData\":{\"x\":24,\"y\":0,\"w\":24,\"h\":15,\"i\":\"2\"},\"panelIndex\":\"2\",\"version\":\"7.2.0\",\"panelRefName\":\"panel_1\"},{\"embeddableConfig\":{},\"gridData\":{\"x\":0,\"y\":30,\"w\":24,\"h\":15,\"i\":\"3\"},\"panelIndex\":\"3\",\"version\":\"7.2.0\",\"panelRefName\":\"panel_2\"},{\"embeddableConfig\":{},\"gridData\":{\"x\":24,\"y\":15,\"w\":24,\"h\":15,\"i\":\"4\"},\"panelIndex\":\"4\",\"version\":\"7.2.0\",\"panelRefName\":\"panel_3\"},{\"embeddableConfig\":{},\"gridData\":{\"x\":0,\"y\":15,\"w\":24,\"h\":15,\"i\":\"5\"},\"panelIndex\":\"5\",\"version\":\"7.2.0\",\"panelRefName\":\"panel_4\"}]","refreshInterval":{"pause":false,"value":60000},"timeFrom":"now-6h","timeRestore":true,"timeTo":"now","title":"Ectogram Resident (Performance)","version":1},"id":"3c4cbda0-b672-11e9-bed3-1dfd40700947","migrationVersion":{"dashboard":"7.0.0"},"references":[{"id":"f9752040-b65c-11e9-bed3-1dfd40700947","name":"panel_0","type":"visualization"},{"id":"738b6600-b65d-11e9-bed3-1dfd40700947","name":"panel_1","type":"visualization"},{"id":"15d234f0-b674-11e9-bed3-1dfd40700947","name":"panel_2","type":"visualization"},{"id":"4c0ddb80-b67b-11e9-8ca2-dd0fee42a4dc","name":"panel_3","type":"visualization"},{"id":"4c065af0-b7fb-11e9-bbd7-6db8670da417","name":"panel_4","type":"visualization"}],"type":"dashboard","updated_at":"2019-08-06T03:39:02.618Z","version":"WzgyLDld"}
which should create:
corrade-heartbeat
,
Going back to the template in the corrade-elasticsearch-heartbeat
folder, copy the file at contrib/linux/systemd/corrade-elasticsearch-heatbeat.service
into /etc/systemd/system
and issue the command:
systemctl enable corrade-elasticsearch-heatbeat
to enable the service file, followed by:
systemctl start corrade-elasticsearch-heartbeat
to start the template.
If everything has been setup correctly, selecting the dashboard button, should show a few pre-made visualizations tracking various statistics such as (from left to right):
Timelion
plot that allows for formulas and manipulation of data directly without using interface elements to build a visualization.