Machine Learning Data Pipelines

Categories: Kafka Kafka Streaming Data Engineering Machine Learning ML Data Data Pipelines Deep Learning

How do we move information realtime and connect machine learning models to make decisions on our business data? This presentation goes through machine learning and Kafka tools that would help achieve that goal.

neural network

In this presentation, we start with Kafka as our data backplane, and how we get information to our pub/sub. As they enter Kafka, how do we sample the data and train our model? Then, how do we unleash that model on our real-time data? In other words, picture-extracting samples for credit card approvals are used for training, and then the model is attached for online processing. The moment we receive an application, we can either approve or disapprove a credit application based on a machine learning model trained on historical data.