Optimizing Real-Time Data Processing with Kafka and Databricks Integration for Scalable Machine Learning Solutions

Trajkovska, Aneta and Ristevski, Blagoj and Veljanovska, Kostandina and Trajkov, Trajche and Rendevski, Nikola (2025) Optimizing Real-Time Data Processing with Kafka and Databricks Integration for Scalable Machine Learning Solutions. In: XV International Conference on Applied Internet and Information Technologies (AIIT 2025), 7 November 2025, Bitola, Macedonia.

Text AIIT2025 Proceedings_Aneta.pdf - Published Version Download (2MB)

Abstract

Integrating Apache Kafka with Databricks enables seamless streaming, analytics, and model deployment, bridging the gap between raw data and actionable insights. Recent trends focus on optimizing performance, reducing latency, and enhancing scalability, allowing organizations to build intelligent, responsive systems. Advances such as event-driven architectures, real-time feature engineering, and automated model orchestration are driving new levels of efficiency in data processing pipelines. Moreover, cloud-native deployments and containerized environments are making these integrations more flexible and resilient, supporting diverse workloads across industries. This paper examines these advancements and their implications for efficient, large-scale machine learning solutions, highlighting both current best practices and emerging innovations that can transform how organizations leverage their data assets.

Item Type:	Conference or Workshop Item (Paper)
Subjects:	Scientific Fields (Frascati) > Natural sciences > Computer and information sciences Scientific Fields (Frascati) > Engineering and Technology > Electrical engineering, electronic engineering,information engineering
Divisions:	Faculty of Information and Communication Technologies
Depositing User:	Prof. d-r. Blagoj Ristevski
Date Deposited:	08 Jan 2026 14:29
Last Modified:	08 Jan 2026 14:29
URI:	https://eprints.uklo.edu.mk/id/eprint/11311

Actions (login required)

View Item