Machine Learning Data Pipelines

Categories: Machine Learning, ML, Data, Data Pipelines, Data Engineering, Kafka, Kafka Streaming, Deep Learning

How do we move information realtime and connect machine learning models to make decisions on our business data? This presentation goes through machine learning and Kafka tools that would help achieve that goal.

In this presentation we start with Kafka as our data backplane, how we get information to our pub/sub. As they enter Kafka, how do we sample that data and train our model, then how do we unleash that model on our real time data. In other words, picture extracting samples for credit card approvals for training, then attaching the model for online processing: The moment we receive an application we can either approve or disapprove a credit application based on a machine learning model trained on historical data.