Framework for Scenario-Driven Black-Box Testing of NPC Behavior in Virtual Reality Games

Matahari Bhakti Nendya; Antonius Rachmat Chrismanto; Dan Daniel Pandapotan; Vito  Gautama; Lunchakorn Wuttisittikulkij; Gabriel Indra Widi Tamtama  Tamtama

doi:10.29407/intensif.v10i1.28379

Authors

Matahari Bhakti Nendya Universitas Kristen Duta Wacana https://orcid.org/0000-0002-4708-9087
Antonius Rachmat Chrismanto Universitas Kristen Duta Wacana https://orcid.org/0000-0003-3247-2419
Dan Daniel Pandapotan Universitas Kristen Duta Wacana https://orcid.org/0009-0001-2600-6609
Vito Gautama Universitas Kristen Duta Wacana https://orcid.org/0009-0003-7175-3413
Lunchakorn Wuttisittikulkij Chulalongkorn University https://orcid.org/0000-0002-3033-3020
Gabriel Indra Widi Tamtama Tamtama National Tsing Hua University https://orcid.org/0000-0002-4779-5563

DOI:

https://doi.org/10.29407/intensif.v10i1.28379

Keywords:

Virtual Reality Games, Non-Playable Characters, Scenario-Driven Testing, Black-Box Testing, NPC Behavior Evaluation

Abstract

Background: Non-Playable Characters (NPCs) play an important role in the quality of interaction in Virtual Reality (VR) games. As NPC systems become more complex and multi-role, their behavior must be evaluated not only from a design perspective but also in terms of technical stability and interaction consistency. However, systematic methods for validating NPC behavior in VR games remain limited. Objective: This study proposes and implements a scenario-driven black-box testing framework for evaluating NPC behavior in VR games based on observable interaction outcomes. Methods: NPC behavior was evaluated using normal, boundary, and stress-based interaction scenarios. Four behavioral quality criteria were assessed: determinism, state integrity, role consistency, and recovery stability. Selected dimensions of the Game Experience Questionnaire (GEQ), namely Social Behaviour Activity and Post-game Module, were also evaluated. Results: Six test scenarios were evaluated. Four scenarios (66.7%) met all criteria, while two (33.3%) were categorized as partial due to minor delays in state transitions and temporary idle-state resets. No crashes, system failures, or fatal interaction breakdowns were observed. GEQ results indicated moderate-to-good social behavior involvement (mean = 3.3) and a comfortable post-game experience (mean = 3.5). Conclusion: The proposed framework provides a structured and reproducible approach for assessing NPC behavior in VR games without requiring access to internal implementation details. The findings highlight the importance of technical behavioral testing, as positive player experience does not necessarily indicate fully stable and consistent NPC behavior.

Downloads

Download data is not yet available.

Abstract views: 22 , PDF downloads: 18

References

[1] D. Checa and A. Bustillo, “A review of immersive virtual reality serious games to enhance learning and training,” Multimed. Tools Appl., vol. 79, no. 9–10, pp. 5501–5527, Mar. 2020, doi: 10.1007/s11042-019-08348-9.

[2] G. J. Hwang and S. Y. Chien, “Definition, roles, and potential research issues of the metaverse in education: An artificial intelligence perspective,” Computers and Education: Artificial Intelligence, vol. 3, Jan. 2022, doi: 10.1016/j.caeai.2022.100082.

[3] J. Radianti, T. A. Majchrzak, J. Fromm, and I. Wohlgenannt, “A systematic review of immersive virtual reality applications for higher education: Design elements, lessons learned, and research agenda,” Comput. Educ., vol. 147, p. 103778, Apr. 2020, doi: 10.1016/J.COMPEDU.2019.103778.

[4] M. Coban, Y. I. Bolat, and I. Goksu, “The potential of immersive virtual reality to enhance learning: A meta-analysis,” Educ. Res. Rev., vol. 36, p. 100452, 2022, doi: https://doi.org/10.1016/j.edurev.2022.100452.

[5] V. Souza, A. Maciel, L. Nedel, and R. Kopper, “Measuring Presence in Virtual Environments: A Survey,” ACM Comput. Surv., vol. 54, no. 8, Oct. 2021, doi: 10.1145/3466817.

[6] A. Fraser, R. Hollett, C. Speelman, and S. L. Rogers, “Behavioural Realism and Its Impact on Virtual Reality Social Interactions Involving Self-Disclosure,” Applied Sciences (Switzerland), vol. 15, no. 6, Mar. 2025, doi: 10.3390/app15062896.

[7] C. Kyrlitsias and D. Michael-Grigoriou, “Social Interaction With Agents and Avatars in Immersive Virtual Environments: A Survey,” Jan. 11, 2022, Frontiers Media S.A. doi: 10.3389/frvir.2021.786665.

[8] R. Fauzi, M. Hariadi, S. M. S. Nugroho, and M. Lubis, “Defense behavior of real time strategy games: Comparison between HFSM and FSM,” Indonesian Journal of Electrical Engineering and Computer Science, vol. 13, no. 2, pp. 634–642, Feb. 2019, doi: 10.11591/ijeecs.v13.i2.pp634-642.

[9] E. N. F. Dewi, A. N. Rachman, and R. H. Z. Hartadji, “Implementation of Hierarchical Finite State Machine for Controlling 2D Character Animation in Action Video Game,” in 2023 Eighth International Conference on Informatics and Computing (ICIC), 2023, pp. 1–6. doi: 10.1109/ICIC60109.2023.10381978.

[10] X. Liu, Z. Xie, and S. Jiang, “Personalized Non-Player Characters: A Framework for Character-Consistent Dialogue Generation,” AI (Switzerland), vol. 6, no. 5, May 2025, doi: 10.3390/ai6050093.

[11] M. Iovino, E. Scukins, J. Styrud, P. Ögren, and C. Smith, “A survey of Behavior Trees in robotics and AI,” Aug. 01, 2022, Elsevier B.V. doi: 10.1016/j.robot.2022.104096.

[12] R. C. Jain, “Multimodal Agents: From Vision to Reality,” IEEE MultiMedia, vol. 31, no. 4, pp. 4–16, 2024, doi: 10.1109/MMUL.2024.3485253.

[13] G. Alves Da Silva, M. Wagner, and S. Ribeiro, “Development of Non-Player Character with Believable Behavior: a systematic literature review,” in Anais Estendidos do XX Simpósio Brasileiro de Jogos e Entretenimento Digital, Porto Alegre, 2021, pp. 391–323. doi: doi:10.5753/sbgames_estendido.2021.19660.

[14] Q. Y. Yin et al., “AI in Human-computer Gaming: Techniques, Challenges and Opportunities,” Jun. 01, 2023, Chinese Academy of Sciences. doi: 10.1007/s11633-022-1384-6.

[15] M. Ç. Uludağlı and K. Oğuz, “Non-player character decision-making in computer games,” Artif. Intell. Rev., vol. 56, no. 12, pp. 14159–14191, 2023, doi: 10.1007/s10462-023-10491-7.

[16] M. Billah, A. Seher, A. Bahar, M. Ashfaq, and A. Hamid, “Game Brains: NPCs Intelligence Using Neural Network Brains,” International Journal of Innovations in Science & Technology, vol. 6, no. 5, pp. 74–82, 2024, Accessed: Jan. 31, 2026. [Online]. Available: https://journal.50sea.com/index.php/IJIST/article/view/779

[17] S. K. Shukla, P. Jain, S. Rana, P. Pandey, V. K. Gupta, and R. Mohana, “A Comprehensive Overview of Software Quality Assurance for Computer Gaming,” in 2024 Eighth International Conference on Parallel, Distributed and Grid Computing (PDGC), 2024, pp. 749–754. doi: 10.1109/PDGC64653.2024.10984084.

[18] J. Chueca, J. Verón, J. Font, F. Pérez, and C. Cetina, “The consolidation of game software engineering: A systematic literature review of software engineering for industry-scale computer games,” Inf. Softw. Technol., vol. 165, p. 107330, 2024, doi: https://doi.org/10.1016/j.infsof.2023.107330.

[19] C. Politowski, F. Petrillo, and Y.-G. Guéhéneuc, “A Survey of Video Game Testing,” in 2021 IEEE/ACM International Conference on Automation of Software Test (AST), Madrid, Spain, 2021, pp. 90–99. doi: 10.1109/AST52587.2021.00018.

[20] J. Yu et al., “GameRTS: A Regression Testing Framework for Video Games,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), 2023, pp. 1393–1404. doi: 10.1109/ICSE48619.2023.00122.

[21] J. Bergdahl, C. Gordillo, K. Tollmar, and L. Gisslén, “Augmenting Automated Game Testing with Deep Reinforcement Learning,” in 2020 IEEE Conference on Games (CoG), Osaka, Japan: IEEE, 2020, pp. 600–603. doi: 10.1109/CoG47356.2020.9231552.

[22] R. Tufano, S. Scalabrino, L. Pascarella, E. Aghajani, R. Oliveto, and G. Bavota, “Using reinforcement learning for load testing of video games,” in Proceedings of the 44th International Conference on Software Engineering, in ICSE ’22. New York, NY, USA: Association for Computing Machinery, 2022, pp. 2303–2314. doi: 10.1145/3510003.3510625.

[23] H. Lee, “A unified conceptual model of immersive experience in extended reality,” May 01, 2025, Elsevier B.V. doi: 10.1016/j.chbr.2025.100663.

[24] K. Kreijns, K. Xu, and J. Weidlich, “Social Presence: Conceptualization and Measurement,” Mar. 01, 2022, Springer. doi: 10.1007/s10648-021-09623-8.

[25] D. Nalic, H. Li, A. Eichberger, C. Wellershaus, A. Pandurevic, and B. Rogic, “Stress Testing Method for Scenario-Based Testing of Automated Driving Systems,” IEEE Access, vol. 8, pp. 224974–224984, 2020, doi: 10.1109/ACCESS.2020.3044024.

[26] D. Nalic, T. Mihalj, M. Bäumler, M. Lehmann, A. Eichberger, and S. Bernsteiner, “Scenario based testing of automated driving systems: A literature survey,” in FISITA web Congress, 2020. doi: 10.46720/f2020-acm-096.

[27] F. Finkeldei, C. Thees, J. N. Weghorn, and M. Althoff, “Scenario Factory 2.0: Scenario-Based Testing of Automated Vehicles with CommonRoad,” Automotive Innovation, vol. 8, no. 2, pp. 207–220, May 2025, doi: 10.1007/s42154-025-00360-0.

[28] Y. Chang, C. Xi, and Z. Luo, “Specific Scenario Generation Method for Trustworthiness Testing of Autonomous Vehicles Based on Interaction Coding,” Applied Sciences, vol. 15, no. 19, p. 10656, Oct. 2025, doi: 10.3390/app151910656.

[29] V. Garousi, A. Rainer, P. Lauvås, and A. Arcuri, “Software-testing education: A systematic literature mapping,” Journal of Systems and Software, vol. 165, p. 110570, 2020, doi: https://doi.org/10.1016/j.jss.2020.110570.

[30] K.-J. Stol and B. Fitzgerald, “Guidelines for Conducting Software Engineering Research,” in Contemporary Empirical Methods in Software Engineering, Springer International Publishing, 2020, pp. 27–62. doi: 10.1007/978-3-030-32489-6_2.

[31] S. Pargaonkar, “A Comprehensive Review of Performance Testing Methodologies and Best Practices: Software Quality Engineering,” International Journal of Science and Research (IJSR), vol. 12, no. 8, pp. 2008–2014, Aug. 2023, doi: 10.21275/sr23822111402.

[32] C. Politowski, F. Petrillo, G. C. Ullmann, and Y.-G. Guéhéneuc, “Game industry problems: An extensive analysis of the gray literature,” Inf. Softw. Technol., vol. 134, p. 106538, 2021, doi: https://doi.org/10.1016/j.infsof.2021.106538.

[33] S. Shirzadehhajimahmood, I. S. W. B. Prasetya, D. Prandi, F. Dignum, M. Dastani, and G. Keller, “Automated Game Testing With Online Search Agent and Model Construction, a Study,” Software Testing Verification and Reliability, vol. 35, no. 2, Mar. 2025, doi: 10.1002/stvr.70002.

[34] U. C. Türker, R. M. Hierons, K. El-Fakih, M. R. Mousavi, and I. Y. Tyukin, “Accelerating Finite State Machine-Based Testing Using Reinforcement Learning,” IEEE Trans. Softw. Eng., vol. 50, no. 3, pp. 574–597, Mar. 2024, doi: 10.1109/TSE.2024.3358416.

[35] W. A. Ijsselsteijn, D. Kort, and Y. A. W. & Poels, “The Game Experience Questionnaire,” 2013. Accessed: Jan. 11, 2025. [Online]. Available: https://pure.tue.nl/ws/files/21666907/Game_Experience_Questionnaire_English.pdf

[36] A. Somrak, M. Pogačnik, and J. Guna, “Suitability and comparison of questionnaires assessing virtual reality-induced symptoms and effects and user experience in virtual environments,” Sensors (Switzerland), vol. 21, no. 4, pp. 1–24, Feb. 2021, doi: 10.3390/s21041185.

[37] J. Nielsen and T. K. Landauer, “A mathematical model of the finding of usability problems,” in Proceedings of the INTERACT ’93 and CHI ’93 Conference on Human Factors in Computing Systems, in CHI ’93. New York, NY, USA: Association for Computing Machinery, 1993, pp. 206–213. doi: 10.1145/169059.169166.