Communication - MQTT Sparkplug

What is MQTT Sparkplug? And does it have anything to do with MQTT? Or should he replace it? You will also find the answers to these questions from today's blog.

In the blog MQTT in practice, which is almost three years old, I described the basics of how the MQTT protocol works. Messages in the MQTT protocol have a topic (Topic) and a body (Payload).

Topic must be a UTF8-encoded string. It can have any structure defined by slashes (e.g. 'plant A/operation B/line C/plc D/Temperature of environment) and MQTT clients (those interested in the message) can request to send messages with a specific Topic (or with a mask, e.g. mask 'plant A/+/+/+/Temperature of environment' corresponds to all devices on all lines in plant A).

The body (Payload) can be arbitrary data – it is not defined by the MQTT protocol.

In the past, as a first step, we implemented the MQTT protocol in such a way that the Topic/Payload were received into the text measured points with the addresses IN_TOPIC and IN_DATA - and all processing and possible data extraction had to be done within the ESL script (or in the evaluated tags).

Later, when we encountered clients sending JSON Payload, we supported the parsing of JSON messages directly in the D2000 KOM process - through I/O tags with the address JA=json_address (e.g. the address JA=rx.current parses the JSON element current found in the structure rx).

The MQTT Sparkplug standard seeks to "supplement" the MQTT for the needs of industrial devices (IIoT). So that it is not necessary for each device to define e.g. JSON or XML schema - and at the same time so that the data is encoded more efficiently than text formats can.

Device types (or application types)

First, let's see what types of applications Sparkplug defines. I would write device types, but I can't, since Device is one of the defined types. So:

• Edge Node - supports the MQTT protocol and connects to the MQTT server. It publishes data obtained from the Device or its own or aggregated data.
• Device/Sensor - represents a physical or logical device connected to the Edge Node and providing data, process data or metrics (the standard calls a specific data, e.g. measured temperature, a metric).
• Host Application - represents the data consumer (SCADA/MES system, historian, analytical tool) which connects to the MQTT server, receives MQTT data from the Edge Node/Device and possibly sends commands. (D2000 application is a Host Application). A specific Edge Node/Device can set one specific Host Application as Primary - if it is not available, it does not send any PUBLISH messages, but temporarily stores all values ​​locally and sends them only after the Primary Host Application connects (this is called Store & Forward).

One physical device can contain an Edge Node and several logical Devicesat the same time.

Definitions of topics and messages

Sparkplug specifies the exact Topic format for the Edge Node/Device. Topic is in the form namespace/group_id/message_type/edge_node_id/[device_id], where:

• namespace is a constant spBv1.0 (for Sparkplug B version 1.0)
• group_id is an arbitrary name of a logical group (e.g. by type of device, operation, organizational structure, etc.)
• message_typeis the type of message
• edge_node_idis the identifier of the Edge Node
• device_id is the identifier of the Device (only if the message is from/for the Device)

Sparkplug also specifies the types of messages that individual applications send:

• NBIRTH – Birth certificate for Sparkplug Edge Nodes (information about Edge Node connection to MQTT Server containing all Edge Node metrics).
• NDEATH – Death certificate for Sparkplug Edge Nodes (information about the loss of the connection between the MQTT Server and the Edge Node).
• DBIRTH – Birth certificate for Devices (information on establishing communication between Device and Edge Node, which contains all metrics of a specific Device).
• DDEATH – Death certificate for Devices (information about the loss of Edge Node connection with Device).
• NDATA – Edge Node data message (data that contains Edge Node changed metrics).
• DDATA – Device data message (data that contains Device changed metrics).
• NCMD – Edge Node command message (Host Application command for Edge Node).
• DCMD – Device command message (Host Application command for Device).
• STATE – Sparkplug Host Application state message (information from the state of the Host Application for the Edge Node/Device [online/offline]).

When the Edge Node connects to the MQTT server, it generates NBIRTH and DBIRTH messages that contains all metrics (sent data). Then it sends only the changed data using NDATA/DDATA messages. When the connection of Edge Node with the MQTT server is lost, it automatically generates an NDEATH message (using the WILL mechanism of the MQTT protocol). When the connection is intentionally and correctly terminated, the Edge Node must generate the NDEATH message explicitly. The interested party (Host Application) thus knows that there has been a loss of communication (and e.g. the D2000 KOM process changes the status of the stations with the address representing the Edge Node and its subordinate Device to StHARDERR).

If the Edge Node loses connection with the Device, it sends a DDEATH message.

NCMD and DCMD messages are generated by the Host Application and are used for commanding, i.e. writing to the Edge Node/Device. For the sake of interest - after writing to the metric named 'Node Control/Rebirth', the Edge Node should respond by sending an NBIRTH/DBIRTH message. This happens when the Host Application starts/connects to the MQTT server and needs to read the current values of metrics.

The STATE message (in JSON format) is generated by the Host Application and announces its connection to the MQTT server (or the MQTT server announces the disconnection of the Host Application). If the application is a Primary one for a particular Edge Node, the Edge Node can, for example, disconnect from the current MQTT server and try to connect to another MQTT server to find its Primary Host Application (this functionality is useful for redundant MQTT systems).

Payload

The payload of all message types (with the exception of STATE messages) is in a binary format defined by Google Protocol Buffers. The binary format enables more efficient encoding of data than e.g. JSON. This is an advantage on narrowband lines, or in situations where you pay for transferred data - e.g. in the network of mobile operators.

The payload contains a sending timestamp and a series of metrics. Each metric contains a name (or a numeric alias), a value type, the value itself, optionally the time of obtaining the value and other flags (e.g. the is_null flag says that the value is null - in D2000 we map this flag to Invalid).

If numeric aliases are used, then the NBIRTH/DBIRTH message must contain both metric names and aliases. Based on this, the Host Application (e.g. D2000) creates a mapping table.

Sparkplug supports simple values, arrays of values, and complex data structures. For now, the implementation in D2000 supports reading simple values ​​and fields and writing simple values.

One of the connected clients was the IIoT gateway Moxa AIG-301 (datasheet), which also supports the MQTT Sparkplug standard.

Testing

Testing and development of the MQTT Sparkplug protocol were carried out using the test.mosquitto.org and broker.hivemq.com websites, which provide MQTT servers with both simulated and real Sparkplug clients and servers.

Browsing

To facilitate the configuration of I/O tags in D2000, we have added MQTT to the list of protocols supporting browsing (see a blog).

It is possible to choose a Topic from the received list as the station address for the MQTT protocol. If the Topic is in Sparkplug format, its shortened version is offered (in the form of group_id/edge_node_id/[device_id], i.e. the constant spBv1.0 and message_type are omitted).

For the Sparkplug payload, metric names are offered along with metric type, current value, timestamp (this is the timestamp when the message was sent, or the timestamp sent with the metric - if neither is present, then the current time when the message was received). The last column contains the name of the I/O tag, if the metric is already configured.

Templates

Templates allow working with structured values ​​(equivalent to structures in the C language), which are also called UDTs (User Defined Types). These structures can also be nested multiple times. We have supported in D2000 reading structure items (i.e. values ​​with simple types) into measured points.

The following figure shows a simple metric "sec" (of type Int32) as well as a metric "secUDT" (of type Template) and its item "sec" (of type Int32) with the address "secUDT->sec". Addressing in D2000 consists in specifying all levels of structures up to the end item, while they are separated by a pair of characters ("->"). This pair is a configuration parameter of the line, since it can happen that some metrics could have such a combination of characters in the name, in which case it would be necessary to select another separator.

Datasets

A dataset is something very similar to a structured variable in D2000. It has named columns (each of which has a simple type defined) and rows. A dataset can be part of a template or be a metric directly. So even addressing in D2000 is intuitive using the structure[row]^column syntax, e.g. MyMetric[2]^Temperature. If we enter an asterisk [*] instead of a specific row number and configure the I/O tag  with the destination column of the structure, we can use one the I/O tag  to fill the entire column of the structured variable with values ​​from the dataset column. In the following figure, there are several such I/O tags (e.g. with the address SA=DHS/Formation Data->Reservoir Parameter[*]^Layer).

Why MQTT?

MQTT is a simple communication protocol (an order of magnitude simpler than OPC UA), so it is easy to implement. It supports both authentication (username and password) and encryption. MQTTS is MQTT wrapped in a TLS layer, which can be implemented easily, e.g. using the stunnel utility, as described in my MQTT blog (encryption implementation in OPC UA is much more complicated).

The advantage of the MQTT protocol over common communication protocols is that the created "data hub", i.e. the MQTT server, is independent of a specific application. So, adding a new application (e.g. analytical tool or an MES system) does not require making data available e.g. from the SCADA system (which may be associated with license requirements, the price for the contractor's work, etc.), but only connecting another client to the existing MQTT server (and configuring its access rights).

Of course, there are two sides to every coin - the price for such flexibility is the necessity to operate an MQTT server - as well as to ensure its connection to all necessary devices (PLC, RTU) using Edge Node devices (unless they directly support the MQTT protocol).

If a customer has e.g. a SCADA system built on D2000 technology, the available data can be published to other systems in several other ways - e.g. using a transparent gateway (if the system is also built on D2000 technology), via OPC DA server or OPC UA server, or via various communication protocols (IEC 870-5-104 Server, Modbus server), via D2000 REST API or D2000 OBJApi.

Conclusion

The MQTT protocol attracts manufacturers of various devices with its simplicity and performance. Its enhancement, the MQTT Sparkplug standard, makes it possible to unify the communication of industrial devices via MQTT and the reading/writing of values ​​from/to devices. The D2000 KOM process adds MQTT Sparkplug support to text and JSON payload support to further expand the set of supported communication standards.

Ing. Peter Humaj, www.ipesoft.com