The Data Must Flow

Feb 15, 2022 · Belfast, United Kingdom

Speaker: Simon Hewitt
Talk: - The Data Must Flow

This is a sneak peak of a talk Simon will give at DataKind later in the month on his experiences with

It's extremely rare that data sits in one place, or in one state for long. Inevitably it must be gathered, manipulated, augmented, validated, anonymised and a million other things before we can begin to extract any value from it.

In the same way, that processing usually starts with a script, that gets enhanced and built on top of, and refactored, with lines commented out and distributed across everyone's laptop. If we're lucky it might be in source control...

Prefect takes your code and transforms it into a robust, distributed pipeline, taking raw input in one end and reliably transforming into usable data at the other.

Starting with a high level overview, we'll follow the evolution from a one file python script through to a reliable pipeline, available to your whole team.

Event organizers
  • DevOps Belfast

    A group for people interested in DevOps (Culture, Automation, Metrics, Sharing) who want to get together, share ideas and have a beer. We're interested in sharing experiences & knowledge on: • Enabling a sharing culture • Configuration Management & System Automation • Metrics generation, collation, aggregation & interpretation • Cloud/Bare Metal/Private Cloud • Continuous Integration & Delivery • Kanban/Agile Operations Management We're an inclusive group, if you are interested in ho

    Recent Events

Are you organizing The Data Must Flow?

Claim the event and start manage its content.

I am the organizer

based on 0 reviews