Apache Kafka Plug-in Kafka Plug-in

Apache Kafka Connection

The Apache Kafka Software from the Apache Foundation is a free software for streaming data packets. A Kafka system is composed of clusters that connect and interlink to provide a messaging system that allows data to be transported losslessly from sender (producer) to receiver (consumer). The system works according to the logic of Publish/Subscribe. Brokers hold data for consumers so that processing can be controlled by the client. The order of messages remains the same. Messages are marked in Kafka with a topic and optionally with a key. Sent messages contain a payload whose format is not provided, similar to MQTT.

Run a free test now

Example: OPC UA data to Apache Kafka

Example: OPC data of a cutter are retrieved from KEPServerEX, translated into JSON and sent to an Apache Kafka Broker

Example of data transfer from Apache Kafka to MS SQL

Example: Parameter data for a cutter is received as a consumer by Apache Kafka, evaluated (JSON) and transferred to an MS-SQL database so that it is available to the process.

That’s how easy it is.

Connecting Systems

What’s Transfer, Trigger and Browsing?

Transfer Objects

The transfer object maps the Kafka producer in the OPC Router. It offers the fields Topic, Key and Payload. Messages are marked with topic and key so that consumers are informed about new messages if they have subscribed to the respective topic. Any data block can be transferred to the payload in OPC Router Connection. Common sources are the JSON or the XML object.

Trigger

If OPC Router should act as a Kafka Consumer, the Kafka Trigger is used. A topic configured for the trigger, which is used by OPC Router to register as a subscriber in Kafka. The trigger activates its connection, in which it is configured, as soon as a new message is available for the topic. If several messages are available, the trigger is retriggered for each message. Even after a system restart, the trigger retrieves all messages not yet retrieved from broker.

Browsing

Browsing of the topics in the broker is possible in the transfer object and in the trigger. After successful configuration of the broker connection, available topics are loaded and made accessible to the operator in list form for selection.

Apache Kafka Plug-in Application Examples

Master data distribution across all plants

For many processes, material master data is an important basis for the parameterisation of individual production processes. For a fast and autonomous decision, these are kept close to the plant as shadow copies to avoid problems and delays when retrieving central data. With Apache Kafka, master data can be made available securely to all interested “consumers”. A cyclical transmission, for example as a JSON package, ensures that the current status is always available directly in the databases at the plants. There is no dependence on the central system. Current production parameters can also be sent back from the plants in the same way, so that an intelligent evaluation and processing can take place centrally (Big Data/AI).

Kafka Symbol
Symbol Arrows
SQL Server Symbol
Kafka Symbol
Symbol Arrows
OPC UA Symbol

Process Data Streaming

Kafka offers a high degree of security and tolerance towards connection problems for central monitoring over large geographical distances and across a wide variety of network architectures. If OPC Router is used to transfer current status/fault and runtime information from systems to Kafka, the cluster system ensures that data is transferred correctly and seamlessly to the central monitoring system. Compared to lightweight IoT infrastructures such as MQTT, Kafka ensures reliable transmission, as is necessary for industrial messaging systems.

Here you can find more information via the OPC Router, Plug-ins, Add-ons and ETL tools.

Plug-ins & Add-ons
Licensing models
Test now