Machine Learning Data Pipelines
Categories:
Kafka
Kafka Streaming
Data Engineering
Machine Learning
ML
Data
Data Pipelines
Deep Learning
How do we move information realtime and connect machine learning models to make decisions on our business data? This presentation goes through machine learning and Kafka tools that would help achieve that goal.
In this presentation, we start with Kafka as our data backplane, and how we get information to our pub/sub. As they enter Kafka, how do we sample the data and train our model? Then, how do we unleash that model on our real-time data? In other words, picture-extracting samples for credit card approvals are used for training, and then the model is attached for online processing. The moment we receive an application, we can either approve or disapprove a credit application based on a machine learning model trained on historical data.