How to: Monitor Intelligence Hub Component Health
Introduction
Monitoring Connections, Flows and Pipelines can be a vital aspect of manufacturing and Industrial IoT architecture. This guide will detail the step-by-step process to monitor Connections, Flows and Pipelines and send their state data to your desired output. In this guide we will send them to MQTT so you can view the output in via our UNS Client. However, you can use this guide to send the payloads to another connection within the HighByte Intelligence Hub, access the instance with our REST data server or to third-party tools like Splunk or Datadog.
Step-by Step Guide
- Create an MQTT Output
Determine the destination where you want to send the status of the Flow or Connection to be sent. Create an MQTT output accordingly.
- Note - The example project uses Status/{{this.name}} for the MQTT topic path. If you emulate this is your own project, the name of your flow and/or connections will appear in MQTT
- Navigate to Flows
Once the output is established, go to the flows section in the interface.
- Create A Flow that Will Monitor Connections and other Flows
-
Click the New Flow Button
-
Name your flow
-
Go to the reference panel.
-
Select “system” as your type.
-
Under the “system” category, choose “Input”.
-
Select the specific Flow, Connection or Pipeline you want to monitor as the source of the flow.
- If you wish to monitor ALL Connections and/or Flows drag the ‘Connections’/‘Flows’/‘Pipelines’ root value into the source of your flow. This will dynamically send all inputs and outputs to your target.
- Choose the Output
- Select the MQTT output previously created to send the information.
Example Output Data
Flow Data
{
"_name": "flowStatus",
"_model": "FlowStatus",
"_timestamp": 1697488855354,
"name": "ToBeMonitored_Instance_Flow",
"lastError": "",
"enabled": true,
"status": "Good",
"statistics": {
"totalExecutionCount": 839,
"lastCompleteTime": "2023-10-16T20:40:54.720Z",
"executingTimes": {
"meanMS": 6,
"minMS": 3,
"maxMS": 8,
"stdDevMS": 1
},
"waitingTimes": {
"meanMS": 1000,
"minMS": 996,
"maxMS": 1005,
"stdDevMS": 2
}
},
"healthy": true
}Connection Data
{
"_name": "connectionStatus",
"_model": "ConnectionStatus",
"_timestamp": 1697488915351,
"name": "MQTT",
"lastError": "",
"status": "Good",
"statistics": {
"pendingWrites": 0
},
"healthy": true
}Pipeline Data
{
"Pipelines": {
"Pipeline_Example": {
"name": "Pipeline_Example",
"lastError": "",
"state": "Good",
"stageStatus": [
{
"name": "Breakup",
"lastError": "",
"pipelineName": "Pipeline_Example",
"state": "Good",
"healthy": true
},
{
"name": "InputsOutputsFromInstance",
"lastError": "",
"pipelineName": "Pipeline_Example",
"state": "Good",
"healthy": true
},
{
"name": "Transform",
"lastError": "",
"pipelineName": "Pipeline_Example",
"state": "Good",
"healthy": true
},
{
"name": "Write",
"lastError": "",
"pipelineName": "Pipeline_Example",
"state": "Good",
"healthy": true
},
{
"name": "WriteNew",
"lastError": "",
"pipelineName": "Pipeline_Example",
"state": "Good",
"healthy": true
},
{
"name": "WriteNew_1",
"lastError": "",
"pipelineName": "Pipeline_Example",
"state": "Good",
"healthy": true
},
{
"name": "Transform_1",
"lastError": "",
"pipelineName": "Pipeline_Example",
"state": "Good",
"healthy": true
},
{
"name": "Breakup_1",
"lastError": "",
"pipelineName": "Pipeline_Example",
"state": "Good",
"healthy": true
}
],
"statistics": {
"idle": false,
"totalRuns": 3776,
"totalErrors": 0,
"queuedWrites": 1,
"lastCompleteTime": "2023-10-24T15:05:43.001Z",
"executingSamples": {
"meanMS": 114,
"minMS": 91,
"maxMS": 152,
"stdDevMS": 17
},
"waitingSamples": {
"meanMS": 0,
"minMS": 0,
"maxMS": 0,
"stdDevMS": 0
}
},
"healthy": true
}
}
}Utilizing the REST Data Server to Obtain Component Health
You can utilize the REST data server to access component Health data.
Use the /data/v1/instances/{instanceName}/Value endpoint with to get a response body based on the configuration of your instance.
Utilizing the intelligencehub-events.log with an External Log Consumer
The Intelligence Hub captures errors in a JSON format within the intelligencehub-events.log. For example, if a particular flow encounters an error, this will be documented in the log. By default Intelligence Hub does not store system metadata for flows and connections in the log. To send system metadata to the log, navigate to your reference panel, set the ‘Type’ to ‘system’, ‘System Type’ to ‘Output’, and drag the ‘Info’ reference to your flow target.
By forwarding system metadata to the events log, third-party tools can access and analyze this log. This allows for monitoring the status of your connections and flows, regardless of whether they are in an error state. Therefore, this could provide a holistic view of the instance’s health.
It’s important to note that within your settings, under the ‘Logging’ section, there’s an option to modify the ‘Max File Size’. Adjusting this ensures that your log consumer retrieves all the necessary data from the file before any old data gets overwritten by new entries.
Attached Project
We’ve attached a project that demonstrates this process. Copy the code block below to your clipboard. Within the Intelligence Hub go to Manage -> Project. Set the Full Project flag to off and change Import Type to ‘JSON’. Paste the copied code block and click ‘Import’.
Within this project you will notice that there is a model with three attributes that is being used in an instance called ‘Example_Instance’. This instance is used in the ‘Flow_Example’ that flows this data to MQTT.
‘StatusFlow’ is a flow that uses the system references of the ‘Example_Flow’ and all system connections as a source and flows it to MQTT. If you go to the UNS Client and connect to MQTT, you will see a Topic called ‘Status’ that contains state date on the ‘Example_Flow’ and any other connection set up in your project.
{
"productInfo": {
"company": "HighByte",
"product": "IntelligenceHub",
"version": "3.2.0",
"build": "2023.9.6.1007",
"stage": "Released"
},
"project": {
"version": 5,
"connections": [
{
"name": "MQTT",
"uri": "mqtt://localhost:1885",
"tags": [],
"writes": {
"flattenModeledValues": false
},
"storeForward": {
"enabled": false,
"maxEntries": 100,
"waitOnFailureInterval": {
"duration": 1,
"units": "Seconds"
}
},
"settings": {
"connectionTimeoutSeconds": 10,
"keepAliveSeconds": 60,
"requestTimeoutMS": 5000,
"ssl": false
}
}
],
"inputs": [],
"outputs": [
{
"name": "Status",
"connection": "MQTT",
"type": "mqtt",
"qualifier": {
"qos": 0,
"namedRoot": false,
"retained": false,
"breakupArrays": false,
"topic": "Status/{{this.name}}"
}
},
{
"name": "ToBeMonitored_Output",
"connection": "MQTT",
"type": "mqtt",
"qualifier": {
"qos": 0,
"namedRoot": false,
"retained": false,
"breakupArrays": false,
"topic": "ToBeMonitored"
}
}
],
"modeling": {
"models": [
{
"name": "Model_Example",
"tags": [],
"attributes": [
{
"name": "StringData",
"type": "Any",
"array": false,
"required": false
},
{
"name": "Time",
"type": "Any",
"array": false,
"required": false
},
{
"name": "Numbers",
"type": "Any",
"array": false,
"required": false
}
]
}
],
"instances": [
{
"name": "Example_Instance",
"tags": [],
"model": "Model_Example",
"rootValueAs": "Object",
"attributes": [
{
"name": "StringData",
"default": "\"Data\""
},
{
"name": "Time",
"expression": "Date.now()"
},
{
"name": "Numbers",
"expression": "Math.random()"
}
]
}
]
},
"flows": [
{
"name": "Example_Flow",
"description": "This flow represents the flow you want to monitor",
"tags": [],
"inReferences": [
"{{Instance.Example_Instance}}"
],
"outReferences": [
"{{Connection.MQTT.ToBeMonitored_Output}}"
],
"enabled": true,
"publishMode": "All",
"trigger": {
"type": "Polled",
"interval": {
"duration": 1,
"units": "Seconds"
},
"mode": "Always",
"delay": {
"duration": 0,
"units": "Seconds"
}
}
},
{
"name": "StatusFlow",
"description": "This flows the data about the status of the flow to be monitored to MQTT.",
"tags": [],
"inReferences": [
"{{System.Flows.Example_Flow}}",
"{{System.Connections.MQTT}}"
],
"outReferences": [
"{{Connection.MQTT.Status}}"
],
"enabled": true,
"publishMode": "All",
"trigger": {
"type": "Polled",
"interval": {
"duration": 1,
"units": "Minutes"
},
"mode": "Always",
"delay": {
"duration": 0,
"units": "Seconds"
}
}
}
],
"conditions": [],
"functions": [],
"tags": [],
"pipelines": []
},
"network": {
"groups": [],
"hubs": []
}
}Conclusion
Monitoring Connections, Flows and Pipelines via MQTT is an efficient method to track and report statuses. The process outlined in this article can be adapted to various outputs and can be an integral part of your digital transformation strategy.