BEYOND FLAKY TEST DETECTION: USING THE MODEL CONTEXT PROTOCOL FOR INTELLIGENT TEST FAILURE DIAGNOSIS IN CI/CD

Abstract

The article presents TeamCity QA Intelligence, a system for automated root cause diagnosis of test failures in continuous integration environments. Manual identification of failure causes remains one of the most time-consuming activities in CI/CD workflows, requiring engineers to cross-reference stack traces, commit histories, build logs, and historical test results. Connecting to the TeamCity REST API through 19 read-only tools built on the Model Context Protocol (MCP), the system enables an AI reasoning layer to access structured CI data and produce a root cause classification for each failure. Three diagnostic categories are supported: code change with commit-level attribution, test flakiness, and infrastructure fault. A Cowork plugin with seven pre-configured commands covers routine diagnostic scenarios. The system has been in daily operation on a production TeamCity instance for four months, executing 167 analysis sessions. In documented investigations, diagnosis time decreased from hours or working days of manual analysis to minutes, depending on failure complexity; classifications verified during routine triage were consistently confirmed. QA tests, unit tests, and integration tests are processed through a single pipeline without per-type configuration. An open-source implementation is provided for independent verification.

References

1.                  Rasheed, S., Wahl, T., Elbaum, S., Dwyer, M., & Ahmad, A. (2024). Test flakiness’ causes, detection, impact and responses: A multivocal review. Journal of Systems and Software, 206, 111837. https://doi.org/10.1016/j.jss.2023.111837

2.                  Parry, O., Hilton, M., Kapfhammer, G. M., & McMinn, P. (2025). Systemic flakiness: An empirical analysis of co-occurring flaky test failures. In Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering (EASE 2025) (pp. 476–487). https://doi.org/10.1145/3756681.3756945

3.                  Hou, X., Zhao, Y., Wang, S., & Wang, H. (2025). Model Context Protocol (MCP): Landscape, security threats, and future research directions. arXiv. https://doi.org/10.48550/arXiv.2503.23278

4.                  Liu, Y., Chen, Z., Zhang, H., et al. (2025). LogSage: An LLM-based framework for CI/CD failure detection and remediation with industrial validation. arXiv. https://doi.org/10.48550/arXiv.2506.03691

5.                  Ahmad, A. (2025). MDFlaker: A tool for multi-factor detection and root cause analysis of flaky tests. Software Impacts, 100794. https://doi.org/10.1016/j.simpa.2025.100794

6.                  Silva, D., Gruber, M., Gokhale, S., Arteca, E., Turcotte, A., d’Amorim, M., Lam, W., Winter, S., & Bell, J. (2024). The effects of computational resources on flaky tests. IEEE Transactions on Software Engineering, 50(12), 3104–3121. https://doi.org/10.1109/TSE.2024.3462251

7.                  Sulistyo, H. B., & Riadi, I. (2025). LLM-driven CI failure diagnosis and automated repair: From GitHub Actions logs to patch recommendation. Journal of Technology Informatics and Engineering, 4(1), 190–214. https://doi.org/10.51903/jtie.v4i1.484

8.                  Singh, A., Ehtesham, A., Kumar, S., & Khoei, T. (2025). A survey of the Model Context Protocol (MCP): Standardizing context to enhance large language models. Preprints. https://doi.org/10.20944/preprints202504.0245.v1

9.                  Hughes, L. (2025). Enterprise-grade security for the Model Context Protocol (MCP): Frameworks and mitigation strategies. arXiv. https://doi.org/10.48550/arXiv.2504.08623

10.              Ehtesham, A., Chaudhary, A., & Kumar, S. (2026). The Model Context Protocol (MCP): Standardizing agentic interoperability. Journal of Information Systems Engineering and Management. https://doi.org/10.52783/jisem

11.              Lam, W., Godefroid, P., Nath, S., Santhiar, A., & Thummalapenta, S. (2019). Root causing flaky tests in a large-scale industrial setting. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2019) (pp. 204–215). https://doi.org/10.1145/3293882.3330570

12.              Durieux, T., Le Goues, C., Hilton, M., & Abreu, R. (2020). Empirical study of restarted and flaky builds on Travis CI. In Proceedings of the 17th International Conference on Mining Software Repositories (MSR 2020) (pp. 254–264). https://doi.org/10.1145/3379597.3387460