Frontiers in Emerging Multidisciplinary Sciences

Open Access Peer Review International
Open Access

AI-Enabled Data Lake Architecture for Efficient Multi-Tenant Big Data Orchestration

4 Department of Artificial Intelligence Arabian Institute of Intelligent Computing Riyadh, Saudi Arabia
4 Department of AI and Data Analytics Gulf Center for Advanced Technology Jeddah, Saudi Arabia

Abstract

The rapid expansion of heterogeneous data, machine learning workloads, and concurrent organizational requirements has increased the need for data lake architectures capable of supporting scalable, efficient, and intelligent multi-tenant orchestration. Conventional data lake designs primarily emphasize storage scalability, while comparatively less attention is given to adaptive workload management, tenant-aware resource allocation, exploration–exploitation decisions, and uncertainty-sensitive orchestration. This research develops a conceptual AI-enabled architecture for multi-tenant data lakes by integrating principles from reinforcement learning, multi-armed bandit optimization, dynamic programming, distributional learning, and entropy-based decision processes. The proposed architecture separates data ingestion, tenant isolation, metadata management, intelligent orchestration, resource allocation, and execution layers while using AI-based decision mechanisms to continuously adapt scheduling and resource policies. The theoretical foundation is derived exclusively from the supplied literature, including work on bandit optimization, dynamic programming, reinforcement learning, Monte-Carlo tree search, entropic regularization, and data-oriented learning environments. The architecture is positioned as a framework for improving workload placement, resource efficiency, fairness, and resilience in heterogeneous multi-tenant environments. The analysis indicates that combining adaptive exploration with value-based decision mechanisms can provide a stronger orchestration model than static policies, although computational overhead, training instability, tenant fairness, and limited empirical validation remain important constraints. The research contributes an integrated conceptual model for applying AI-driven decision intelligence to scalable multi-tenant data lake orchestration.

How to Cite

Dr. Faisal Al-Harbi, & Dr. Sara Al-Qahtani. (2026). AI-Enabled Data Lake Architecture for Efficient Multi-Tenant Big Data Orchestration. Frontiers in Emerging Multidisciplinary Sciences, 3(08), 125–132. https://doi.org/10.64917/fems/Volume03Issue08-07

References

Abernethy, J. D., Lee, C., and Tewari, A. (2015). Fighting bandits with a new kind ofsmoothness.Advances in Neural Information Processing Systems, 28.572
A Unified Perspective on Value Backup and Exploration in Monte-Carlo Tree Search
Auer, P., Cesa-Bianchi, N., and Fischer, P. (2002). Finite-time analysis of the multiarmedbandit problem.Machine learning, 47(2-3):235–256.
Bellemare, M. G., Dabney, W., and Munos, R. (2017). A distributional perspective onreinforcement learning. InProceedings of the 34th International Conference on MachineLearning-Volume 70, pages 449–458. JMLR. org.
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. (2013). The arcade learningenvironment: An evaluation platform for general agents.Journal of Artificial IntelligenceResearch, 47:253–279.
Bellman, R. (1954). The theory of dynamic programming. Technical report, Rand corpsanta monica ca.
Belousov, B. and Peters, J. (2019). Entropic regularization of markov decision processes.Entropy, 21(7):674.
Ben-Tal, A., Charnes, A., and Teboulle, M. (1989). Entropic means.Journal of Mathemat-ical Analysis and Applications, 139(2):537–551.
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., andZaremba, W. (2016). Openai gym.
Buesing, L., Heess, N., and Weber, T. (2020). Approximate inference in discrete distribu-tions with monte carlo tree search and value functions. InInternational Conference onArtificial Intelligence and Statistics, pages 624–634. PMLR.
Bullen, P. S. (2013).Handbook of means and their inequalities. Springer Science & BusinessMedia.
K. K. Goyal, "Scalable Data Lakes for AI Workloads: A Multitenant Architecture for Big Data Orchestration," 2025 IEEE International Conference on Computing (ICOCO), Kuching, Malaysia, 2025, pp. 266-271, doi: 10.1109/ICOCO67189.2025.11334100.