Building and Scaling Robust Zero-code IoT Streaming Data Pipelines

Oct 26, 2021 · Tel Aviv-Yafo, Israel

This is online tech talk event, register on the website to receive join link:

With the rapid onset of the global Covid-19 Pandemic in 2020 the USA Centers for Disease Control and Prevention (CDC) quickly implemented a new Covid-19 pipeline to collect testing data from all of the USA’s states and territories, and produce multiple consumable results for federal and public agencies. They did this in under 30 days, using Apache Kafka.
Inspired by this story, we built two demonstration streaming pipelines for ingesting, storing, and visualizing public IoT data (Tidal data from NOAA, the National Oceanic and Atmospheric Administration) using multiple open source technologies. The common ingestion technologies were Apache Kafka, Apache Kafka Connect, and Apache Camel Kafka Connector, supplemented with Prometheus and Grafana for monitoring. The initial experiment used Open Distro for Elasticsearch and Kibana as the target storage and visualisation technologies, while the second experiment used PostgreSQL and Apache Superset.

In this talk we introduce each technology and the pipeline architecture, and walk through the steps followed, challenges encountered, and solutions used to build reliable and scalable pipelines, and visualize the results (including Tidal periods, ranges and locations). We compare and contrast the two approaches, focussing on exception handling, scalability, performance and monitoring, and the pros and cons of the two visualization technologies (Kibana and Superset).

Event organizers
  • Seattle Artificial Intelligence Meetup

    Hi Fellow A.I. Enthusiast, Thank you for your interest in this meetup.  I want to start by personally welcoming your input. Please do speak up, your voice is welcomed, appreciated and imperative to the quality of this community. ­ We are committed to making a useful, fun, and effective means of learning about A.I.  We stand for Education, fun and Innovation. If you have event ideas, ideas you'd like to create, or ideas about how to collaborate with others, please send us a message and we'll tell you how to

    Recent Events

Are you organizing Building and Scaling Robust Zero-code IoT Streaming Data Pipelines?

Claim the event and start manage its content.

I am the organizer

based on 0 reviews

Featured Events