Building Timely and Reliable Data Pipeline for Time-Series IoT Data Platform
Reengineered IoT data pipeline to reduce message loss by more than 99%.
Überblick
KUGU Home is a market leader in digital building management. At their growing scale they face one of the biggest challenges in the IoT world: tracking and analysing data in real time. DataMax helped KUGU refactor, reengineer, and monitor their existing data pipeline to reduce message loss by more than 99%.
Herausforderung
Messages from numerous sources, arriving at varying rates, converge at the broker. During peak hours the pipeline handles up to 1,000 messages per second — a number that grows as the customer base expands. The most vulnerable point was the transmission of data from the MQTT broker to InfluxDB. The objective was a pipeline capable of reliably, accurately, and timely processing this volume of data while accommodating future capacity growth.
Vorgehen
We reengineered the MQTT-to-InfluxDB transmission layer using modern IoT engineering techniques, and built a monitoring cockpit to give the team real-time visibility into pipeline health.
Lösung
We introduced Telegraf — a powerful and adaptable tool widely used in the IoT industry — to handle data reception, transformation, and micro-batching to InfluxDB. We also developed a monitoring cockpit that displays the data reception rate and alerts the team to any issues arising in the pipeline, providing comprehensive insights into its status at all times.
Ergebnisse
"DataMax demonstrated remarkable expertise in reengineering and monitoring our data pipeline, resulting in a significant improvement in performance and reliability. Their proactive approach, deep understanding of our needs, and exceptional collaboration made them a highly recommended partner for any organisation seeking help in data engineering in cloud. By working with DataMax we were able to blend our engineering together to create a seamless and high impactful piece of work. We hope to work with them more in the future."
Scott Williams — CPTO, KUGU HomeBereit, Ihre KI-Reise zu beschleunigen?
Lassen Sie uns über Ihre Daten- und KI-Herausforderungen sprechen. Wir helfen bei Strategie und schneller Umsetzung.
Kontakt aufnehmen