Building Timely and Reliable Data Pipeline for Time-Series IoT Data Platform
Reengineered IoT data pipeline to reduce message loss by more than 99%.
Overview
KUGU Home is a market leader in digital building management. At their growing scale they face one of the biggest challenges in the IoT world: tracking and analysing data in real time. DataMax helped KUGU refactor, reengineer, and monitor their existing data pipeline to reduce message loss by more than 99%.
Challenge
Messages from numerous sources, arriving at varying rates, converge at the broker. During peak hours the pipeline handles up to 1,000 messages per second — a number that grows as the customer base expands. The most vulnerable point was the transmission of data from the MQTT broker to InfluxDB. The objective was a pipeline capable of reliably, accurately, and timely processing this volume of data while accommodating future capacity growth.
Approach
We reengineered the MQTT-to-InfluxDB transmission layer using modern IoT engineering techniques, and built a monitoring cockpit to give the team real-time visibility into pipeline health.
Solution
We introduced Telegraf — a powerful and adaptable tool widely used in the IoT industry — to handle data reception, transformation, and micro-batching to InfluxDB. We also developed a monitoring cockpit that displays the data reception rate and alerts the team to any issues arising in the pipeline, providing comprehensive insights into its status at all times.
Results
"DataMax demonstrated remarkable expertise in reengineering and monitoring our data pipeline, resulting in a significant improvement in performance and reliability. Their proactive approach, deep understanding of our needs, and exceptional collaboration made them a highly recommended partner for any organisation seeking help in data engineering in cloud. By working with DataMax we were able to blend our engineering together to create a seamless and high impactful piece of work. We hope to work with them more in the future."
Scott Williams — CPTO, KUGU HomeReady to accelerate your AI journey?
Let's talk about your data and AI challenges. We'll help you build the right strategy and execute with speed.
Get in Touch