Open Access

Intelligent Framework For Real-Time Anomaly Detection In Distributed Data Engineering Systems

4 Department of Tropical Medicine Bahamas Health Sciences University Nassau, Bahamas
4 Faculty of Clinical Research Caribbean Medical and Research Academy Freeport, Bahamas

Abstract

The rapid expansion of distributed data engineering ecosystems has significantly increased the complexity of real-time anomaly detection across cloud-native infrastructures, Internet of Things (IoT) platforms, enterprise analytics pipelines, and decentralized computational environments. Traditional anomaly detection techniques are increasingly ineffective in addressing high-volume, heterogeneous, and continuously evolving data streams generated by modern distributed architectures. This research proposes an intelligent framework for real-time anomaly detection in distributed data engineering systems by integrating machine learning, adaptive stream analytics, distributed event processing, security-driven architectures, and AI-assisted predictive modeling. The framework focuses on improving detection accuracy, reducing latency, minimizing false positives, and enhancing scalability across multi-source data environments. The study synthesizes recent developments in AI-driven monitoring, blockchain-enabled security, quantum-aware cybersecurity models, predictive maintenance systems, and scalable ETL infrastructures. A layered methodological architecture is introduced to support anomaly identification, automated classification, distributed coordination, and response optimization. The findings indicate that intelligent distributed anomaly detection systems significantly improve operational reliability, cyber resilience, and infrastructure scalability when compared to static rule-based monitoring models. Furthermore, the study identifies critical implementation challenges related to model drift, distributed synchronization, computational overhead, and data governance. The proposed framework contributes to research on intelligent distributed analytics by establishing a scalable, adaptive, and security-oriented anomaly detection paradigm suitable for contemporary enterprise and cloud environments.

How to Cite

Rolle, D. D., & Cartwright, D. M. (2026). Intelligent Framework For Real-Time Anomaly Detection In Distributed Data Engineering Systems. Frontiers in Emerging Computer Science and Information Technology, 3(05), 11–18. Retrieved from https://irjernet.com/index.php/fecsit/article/view/414

References

Agarwal, A. V., & Kumar, S. (2017, October). Intelligent multi-level mechanism of secure data handling of vehicular information for post-accident protocols. In 2017 2nd International Conference on Communication and Electronics Systems (ICCES) (pp. 902-906). IEEE.
Agarwal, A. V., & Kumar, S. (2017, November). Unsupervised data responsive based monitoring of fields. In 2017 International Conference on Inventive Computing and Informatics (ICICI) (pp. 184-188). IEEE.
Agarwal, A. V., Verma, N., & Kumar, S. (2018). Intelligent Decision Making Real-Time Automated System for Toll Payments. In Proceedings of International Conference on Recent Advancement on Computer and Communication: ICRAC 2017 (pp. 223-232). Springer Singapore.
Agarwal, A. V., Verma, N., Saha, S., & Kumar, S. (2018). Dynamic Detection and Prevention of Denial of Service and Peer Attacks with IPAddress Processing. Recent Findings in Intelligent Computing Techniques: Proceedings of the 5th ICACNI 2017, Volume 1, 707, 139.
Al Imran, M., Al Fathah, A., Al Baki, A., Alam, K., Mostakim, M. A., Mahmud, U., & Hossen, M. S. (2023). Integrating IoT and AI For Predictive Maintenance in Smart Power Grid Systems to Minimize Energy Loss and Carbon Footprint. Journal of Applied Optics, 44(1), 27-47.
Alam, K., Mostakim, M. A., & Khan, M. S. I. (2017). Design and Optimization of MicroSolar Grid for Off-Grid Rural Communities. Distributed Learning and Broad Applications in Scientific Research, 3.
Bazemore, K., Permpalung, N., Mathew, J., Lemma, M., Haile, B., Avery, R., ... & Shah, P. (2022). Elevated cell-free DNA in respiratory viral infection and associated lung allograft dysfunction. American Journal of Transplantation, 22(11), 2560-2570.
Chuleerarux, N., Manothummetha, K., Moonla, C., Sanguankeo, A., Kates, O. S., Hirankarn, N., ... & Permpalung, N. (2022). Immunogenicity of SARS-CoV-2 vaccines in patients with multiple myeloma: a systematic review and meta-analysis. Blood Advances, 6(24), 6198-6207.
Dash, S. (2023). Architecting Intelligent Sales and Marketing Platforms: The Role of Enterprise Data Integration and AI for Enhanced Customer Insights. Journal of Artificial Intelligence Research, 3(2), 253-291.
Dash, S. (2023). Designing Modular Enterprise Software Architectures for AI-Driven Sales Pipeline Optimization. Journal of Artificial Intelligence Research, 3(2), 292-334.
Doddipatla, L., Ramadugu, R., Yerram, R. R., & Sharma, T. (2021). Exploring The Role of Biometric Authentication in Modern Payment Solutions. International Journal of Digital Innovation, 2(1).
Ferdinand, J. (2023). Emergence of Dive Paramedics: Advancing Prehospital Care Beyond DMTs.
Ferdinand, J. (2023). Marine Medical Response: Exploring the Training, Role and Scope of Paramedics and Paramedicine (ETRSp). Qeios.
Ferdinand, J. (2023). The Key to Academic Equity: A Detailed Review of EdChat’s Strategies.
Gopinath, S., Ishak, A., Dhawan, N., Poudel, S., Shrestha, P. S., Singh, P., ... & Michel, G. (2022). Characteristics of COVID-19 breakthrough infections among vaccinated individuals and associated risk factors: A systematic review. Tropical medicine and infectious disease, 7(5), 81.
Han, J., Yu, M., Bai, Y., Yu, J., Jin, F., Li, C., ... & Li, L. (2020). Elevated CXorf67 expression in PFA ependymomas suppresses DNA repair and sensitizes to PARP inhibitors. Cancer Cell, 38(6), 844-856.
Integrating solar cells into building materials (Building-Integrated Photovoltaics-BIPV) to turn buildings into self-sustaining energy sources. Journal of Artificial Intelligence Research and Applications, 2(2).
JALA, S., ADHIA, N., KOTHARI, M., JOSHI, D., & PAL, R. SUPPLY CHAIN DEMAND FORECASTING USING APPLIED MACHINE LEARNING AND FEATURE ENGINEERING.
JOSHI, D., SAYED, F., BERI, J., & PAL, R. (2021). An efficient supervised machine learning model approach for forecasting of renewable energy to tackle climate change. Int J Comp Sci Eng Inform Technol Res, 11, 25-32.
Joshi, D., Parikh, A., Mangla, R., Sayed, F., & Karamchandani, S. H. (2021). AI Based Nose for Trace of Churn in Assessment of Captive Customers. Turkish Online Journal of Qualitative Inquiry, 12(6).
Joshi, D., Sayed, F., Jain, H., Beri, J., Bandi, Y., & Karamchandani, S. A Cloud Native Machine Learning based Approach for Detection and Impact of Cyclone and Hurricanes on Coastal Areas of Pacific and Atlantic Ocean.
Joshi, D., Sayed, F., Saraf, A., Sutaria, A., & Karamchandani, S. (2021). Elements of Nature Optimized into Smart Energy Grids using Machine Learning. Design Engineering, 1886-1892.
Khambati, A. (2021). Innovative Smart Water Management System Using Artificial Intelligence. Turkish Journal of Computer and Mathematics Education (TURCOMAT), 12(3), 4726-4734.
Khambaty, A., Joshi, D., Sayed, F., Pinto, K., & Karamchandani, S. (2022, January). Delve into the Realms with 3D Forms: Visualization System Aid Design in an IOT-Driven World. In Proceedings of International Conference on Wireless Communication: ICWiCom 2021 (pp. 335-343). Singapore: Springer Nature Singapore.
Mahmud, U., Alam, K., Mostakim, M. A., & Khan, M. S. I. (2018). AI-driven micro solar power grid systems for remote communities: Enhancing renewable energy efficiency and reducing carbon emissions. Distributed Learning and Broad Applications in Scientific Research, 4.
Malhotra, I., Gopinath, S., Janga, K. C., Greenberg, S., Sharma, S. K., & Tarkovsky, R. (2014). Unpredictable nature of tolvaptan in treatment of hypervolemic hyponatremia: case review on role of vaptans. Case reports in endocrinology, 2014(1), 807054.
Manoharan, A., & Nagar, G. MAXIMIZING LEARNING TRAJECTORIES: AN INVESTIGATION INTO AI-DRIVEN NATURAL LANGUAGE PROCESSING INTEGRATION IN ONLINE EDUCATIONAL PLATFORMS.
Mishra, M. (2017). Reliability-based Life Cycle Management of Corroding Pipelines via Optimization under Uncertainty (Doctoral dissertation).
Mishra, M. (2022). Review of Experimental and FE Parametric Analysis of CFRP-Strengthened Steel-Concrete Composite Beams. Journal of Mechanical, Civil and Industrial Engineering, 3(3), 92-101.
Nagar, G., & Manoharan, A. (2022). Blockchain technology: reinventing trust and security in the digital world. International Research Journal of Modernization in Engineering Technology and Science, 4(5), 6337-6344.
Nagar, G., & Manoharan, A. (2022). THE RISE OF QUANTUM CRYPTOGRAPHY: SECURING DATA BEYOND CLASSICAL MEANS. 04. 6329-6336. 10.56726. IRJMETS24238.
Nagar, G., & Manoharan, A. (2022). ZERO TRUST ARCHITECTURE: REDEFINING SECURITY PARADIGMS IN THE DIGITAL AGE. International Research Journal of Modernization in Engineering Technology and Science, 4, 2686-2693.
Phongkhun, K., Pothikamjorn, T., Srisurapanont, K., Manothummetha, K., Sanguankeo, A., Thongkam, A., ... & Permpalung, N. (2023). Prevalence of ocular candidiasis and Candida endophthalmitis in patients with candidemia: a systematic review and meta-analysis. Clinical Infectious Diseases, 76(10), 1738-1749.
Ramadugu, R., & Doddipatla, L. (2022). Emerging Trends in Fintech: How Technology Is Reshaping the Global Financial Landscape. Journal of Computational Innovation, 2(1).
Ramadugu, R., & Doddipatla, L. (2022). The Role of AI and Machine Learning in Strengthening Digital Wallet Security Against Fraud. Journal of Big Data and Smart Systems, 3(1).
Roh, Y. S., Khanna, R., Patel, S. P., Gopinath, S., Williams, K. A., Khanna, R., ... & Kwatra, S. G. (2021). Circulating blood eosinophils as a biomarker for variable clinical presentation and therapeutic response in patients with chronic pruritus of unknown origin. The Journal of Allergy and Clinical Immunology: In Practice, 9(6), 2513-2516.
Shakibaie, B., Blatz, M. B., & Barootch, S. (2023). Comparación clínica de split rolling flap vestibular (VSRF) frente a double door flap mucoperióstico (DDMF) en la exposición del implante: un estudio clínico prospectivo. Quintessence: Publicación internacional de odontología, 11(4), 232-246.
Shakibaie, B., Blatz, M. B., Conejo, J., & Abdulqader, H. (2023). From Minimally Invasive Tooth Extraction to Final Chairside Fabricated Restoration: A Microscopically and Digitally Driven Full Workflow for Single-Implant Treatment. Compendium of Continuing Education in Dentistry (15488578), 44(10).
Shakibaie, B., Sabri, H., & Blatz, M. (2023). Modified 3-Dimensional Alveolar Ridge Augmentation in the Anterior Maxilla: A Prospective Clinical Feasibility Study. Journal of Oral Implantology, 49(5), 465-472.
Shakibaie-M, B. (2013). Comparison of the effectiveness of two different bone substitute materials for socket preservation after tooth extraction: a controlled clinical study. International Journal of Periodontics & Restorative Dentistry, 33(2).
Singu, S. K. (2021). Designing Scalable Data Engineering Pipelines Using Azure and Databricks. ESP Journal of Engineering & Technology Advancements, 1(2), 176-187.
Singu, S. K. (2021). Real-Time Data Integration: Tools, Techniques, and Best Practices. ESP Journal of Engineering & Technology Advancements, 1(1), 158-172.
Singu, S. K. (2022). ETL Process Automation: Tools and Techniques. ESP Journal of Engineering & Technology Advancements, 2(1), 74-85.
Zeng, J., Han, J., Liu, Z., Yu, M., Li, H., & Yu, J. (2022). Pentagalloylglucose disrupts the PALB2-BRCA2 interaction and potentiates tumor sensitivity to PARP inhibitor and radiotherapy. Cancer Letters, 546, 215851.