Frontiers in Emerging Multidisciplinary Sciences

Open Access Peer Review International
Open Access

Scalable AI-Based Multi-Agent Systems for Efficient and Resilient Data Stream Processing

4 Department of Artificial Intelligence, Tokyo Institute of Digital Engineering Japan
4 Department of Computer Science and Intelligent Systems, Osaka Technology University Japan

Abstract

The increasing volume, velocity, and heterogeneity of continuously generated data have created a need for intelligent stream-processing architectures capable of adapting to changing computational and operational conditions. Conventional stream-processing systems generally depend on predefined scheduling, centralized coordination, or static optimization strategies, which can become inefficient when workloads fluctuate or processing resources experience failures. This research develops a conceptual framework for scalable AI-based multi-agent systems in which autonomous agents coordinate data acquisition, workload allocation, processing, monitoring, and recovery. The theoretical foundation combines heuristic search, learned heuristic functions, planning, pattern databases, regression-based planning, and neural decision mechanisms. The proposed architecture interprets stream-processing operations as a dynamic planning problem in which agents select actions according to workload state, resource availability, latency requirements, and resilience conditions. The framework is further motivated by recent work on AI-based multi-agent models for optimized data streaming, where scalability and resiliency are treated as simultaneous architectural objectives (Reddy et al., 2026). The analysis indicates that learned heuristics can reduce decision complexity, distributed agents can improve adaptability, and planning-oriented coordination can support graceful recovery from resource degradation. However, the approach introduces challenges involving coordination overhead, non-stationary workloads, communication costs, and the computational expense of learning. The study therefore positions multi-agent AI not merely as an automation layer but as a decision-making mechanism for adaptive stream-processing infrastructures.

How to Cite

Haruto Nakamura, & Yui Tanaka. (2026). Scalable AI-Based Multi-Agent Systems for Efficient and Resilient Data Stream Processing. Frontiers in Emerging Multidisciplinary Sciences, 3(08), 118–124. https://doi.org/10.64917/fems/Volume03Issue08-06

References

Agostinelli, F., McAleer, S., Shmakov, A., & Baldi, P. (2019). Solving the Rubik’s cube with deep reinforcement learning and search. Nature Machine Intelligence,1(8), 356–363.
Alcazar, V., Borrajo, D., Fernandez, S., & Fuentetaja, R. (2013). Revisiting regression in planning. International Joint Conference on Artificial Intelligence, 2254–2260.
Arfaee, S. J., Zilles, S., & Holte, R. C. (2011). Learning heuristic functions for large statespaces. Artificial Intelligence,175(16-17), 2075–2098.
Bäckström, C., & Nebel, B. (1995). Complexity results for SAS+planning.ComputationalIntelligence,11(4), 625–655.
Bonet, B. (2013). An Admissible Heuristic for SAS+Planning Obtained from the StateEquation .International Joint Conference on Artificial Intelligence, 2268–2274.
Bonet, B., & Geffner, H. (2001). Planning as heuristic search.Artificial Intelligence,129(1-2),5–33.
Culberson, J. C., & Schaeffer, J. (1998). Pattern databases. Computational Intelligence,14(3), 318–334.
Dong, H., Mao, J., Lin, T., Wang, C., Li, L., & Zhou, D. (2019). Neural logic machines[ Poster].International Conference on Learning Representations.
Doran, J. E., & Michie, D. (1966). Experiments with the Graph Traverser Program. Proceed-ings of the Royal Society A,294, 235–259.
Edelkamp, S. (2001). Planning with pattern databases. European Conference on Planning,13–24.
Ferber, P., Geißer, F., Trevizan, F., Helmert, M., & Hoffmann, J. (2022). Neural networkheuristic functions for classical planning: Bootstrapping and comparison to othermethods. International Conference on Automated Planning and Scheduling, 583–587.
Ferber, P., Helmert, M., & Hoffmann, J. (2020). Neural network heuristics for classical planning: A study of hyperparameter space .European Conference on ArtificialIntelligence,325, 2346–2353.
R. Reddy, P. Udayaraju, K. K. Goyal, S. C. R. Vudem, R. Sayana and V. Gummadi, "Creating an AI-based Multi-Agent Model for Optimized Data Streaming with Improved Resiliency and Scalability for Event Streaming," 2026 6th International Conference on Image Processing and Capsule Networks (ICIPCN), Dhulikhel, Nepal, 2026, pp. 1372-1379, doi: 10.1109/ICIPCN67432.2026.11438445.