Failure Isolation And Blast Radius Control In Large-Scale Distributed Systems

Sathwik Rao Sirikonda; Shashidhar Bhat; Rishabh Jain; Vikas Katoch

doi:10.29284/mkwrh621

Authors

Sathwik Rao Sirikonda Site reliability engineer California USA ORCID: 0009-0005-9334-9870
Shashidhar Bhat Site reliability engineer California USA ORCID: 0009-0003-2432-9564
Rishabh Jain Site Reliability Engineer California USA ORCID: 0009-0006-5899-2534
Vikas Katoch Senior Quality Specialist California USA ORCID: 0009-0006-2347-1843

DOI:

https://doi.org/10.29284/mkwrh621

Keywords:

Failure isolation, Blast radius, Cascading failures, Circuit breakers, SLO burn rate, Zonal/cell architecture.

Abstract

The contemporary scale distributed systems microservices, Kubernetes, service meshes, and multi-region platforms can increase the speed of delivery, but increase the paths of failure propagation with the increase of the dependency graphs. This study analyses failure isolation (fault containment boundaries at service, host, zone, region, and organizational boundaries) and blast radius (the scope of impact of users, services, and geographic areas over time). Cascading failures are shown to be driven by retries, shared-control demands, noisy-neighbor contention, and setup errors that are transformed into fleet-wide failures. Empirical motivation is based on recorded events: instability due to changes accounts for the majority of outages; recovery following the 21 June 2022 incident in Cloudflare affected 19 data centers; and the disaster in AWS us-east-1 on 7 December 2021 had a total impact exceeding 8 hours. The research integrates operational instruments for capping propagation, such as bulkheads, circuit breakers, rate limiting, separation of control planes, zonal and cell design, and progressive delivery and rollback. It generates a measurement-based framework based on community-derived MTTD/MTTR, SLO burn rates, and blast-radius indices to direct architecture and operations. The code of methods is 30-50 public incidents (2016-2025), and it statistically compares containment outcomes across layers. The future of research focuses more on uniform reporting, predictive containment, and safe automation of serverless and self-healing systems.

References

[1] Flexera, “RightScale® 2019 State of the Cloud Report from Flexera™,” Flexera, 2019. https://resources.flexera.com/web/media/documents/rightscale-2019-state-of-the-cloud-report-from-flexera.pdf.

[2] Flexera, “Flexera™ 2023 State of the Cloud Report,” Flexera, 2023. https://m3comva1.frb.io/uploads/docs/Flexera-State-of-the-Cloud-Report-2023.pdf.

[3] R. Losio, AWS US-EAST-1 Outage: Postmortem and Lessons Learned. InfoQ, Dec. 2021. https://www.infoq.com/news/2021/12/aws-outage-postmortem/.

[4] T. Strickx and J. Hartman, “Cloudflare outage on June 21, 2022,” Cloudflare Blog, Jun. 21, 2022. https://blog.cloudflare.com/cloudflare-outage-on-june-21-2022/.

[5] S. Wolfe, "Amazon's one hour of downtime on Prime Day may have cost it up to $100 million in lost sales," Business Insider, Jul. 16, 2018. https://www.businessinsider.com/amazon-prime-day-website-issues-cost-it-millions-in-lost-sales-2018-7?r=US&IR=T.

[6] A. Moussa and W. El-Dakhakhni, “Managing interdependence-induced systemic risks in infrastructure projects,” Journal of Management in Engineering, vol. 38, no. 5, p. 04022048, 2022.

[7] S. Durgam and V. Nagaraj, Scalable data-driven engineering for high-performance computing & financial services, International Journal of Intelligent Systems and Applications in Engineering (IJISAE), 2025. https://www.ijisae.org/index.php/IJISAE/article/view/7914.

[8] N. King and A. Jones, Future Energy Options from a Systems Perspective. Cham, Switzerland: Palgrave Macmillan, 2023.

[9] V. R. Chintapalli, B. R. Killi, R. Partani, B. R. Tamma, and C. S. R. Murthy, “Energy-and reliability-aware provisioning of parallelized service function chains with delay guarantees,” IEEE Transactions on Green Communications and Networking, vol. 8, no. 1, pp. 205–223, 2023.

[10] S. K. RaoVanama, AI-Augmented CI/CD Pipeline Optimization for Scalable Cloud-Native Deployment, International Journal of Artificial Intelligence, Data Science and Machine Learning (IJAIDSML), vol. 5, no. 4, pp. 175–187, Dec. 30, 2024. https://ijaidsml.org/index.php/ijaidsml/article/view/368.

[11] K. S. Chadha, Machine Learning–Augmented ETL Pipelines for Fraud–Resistant Insurance Claims Processing, International Journal of Data Science and Machine Learning, vol. 5, no. 01, pp. 410–436, 2025,

[12] S. Zhou, Z. Wang, and Q. Li, “A conceptual framework integrating numerical simulation with system theory based method for quantitative explosion process hazard analysis,” Process Safety and Environmental Protection, vol. 166, pp. 202–211, 2022.

[13] S. K. Vishwakarma, AI-driven predictive risk modelling for aerospace supply chains, International Journal of Innovation in Business & Economics and Applied Journal, 2025. https://www.iibajournal.org/index.php/iibeaj/article/view/64.

[14] Z. Sayyed, Development of a simulator to mimic VMware vCloud Director (VCD) API calls for cloud orchestration testing, International Journal of Computational and Experimental Science and Engineering (IJCESEN), vol. 11, no. 3, pp. 5422–5441, 2025.

[15] W. E. Wong, R. Gao, Y. Li, R. Abreu, F. Wotawa, and D. Li, “Software fault localization: An overview of research, techniques, and tools,” in Handbook of Software Fault Localization: Foundations and Advances, pp. 1–117, 2023.

[16] Internet Research Team, “AWS Outage Analysis: December 7 & 10, 2021,” ThousandEyes Blog, Dec. 2021. https://www.thousandeyes.com/blog/aws-outage-analysis-dec-7-2021.

[17] T. Luxner, “Cloud computing trends and statistics: Flexera 2023 State of the Cloud Report,” Flexera Blog, 2023. [Online]. Available: https://www.flexera.com/blog/finops/cloud-computing-trends-flexera-2023-state-of-the-cloud-report/

[18] S. Rangu, Enterprise digital transformation in financial services: Emerging trends and technologies, Computer Fraud & Security, 2025.https://computerfraudsecurity.com/index.php/journal/article/view/786

[19] K. J. Hsu, “System Support for Fine-Grained Resource Management in Mobile Edge Computing,” Ph.D. dissertation, Georgia Institute of Technology, 2024.

[20] S. Samala, “Architecting multi-instance Jira deployments: Scalability challenges for global enterprises,” International Journal of Engineering Applications, 2025.https://gprjournals.org/journals/index.php/ijea/article/view/353

[21] P. Gannavarapu, Deploying Azure AD federation with SAML for secure enterprise SaaS integration, Computer Fraud & Security, 2025.https://computerfraudsecurity.com/index.php/journal/article/view/782

[22] R. Hariharan, “Automated incident response using AI-based decision trees,” Computer Fraud & Security, 2025.https://computerfraudsecurity.com/index.php/journal/article/view/783

[23] S. Alshayeb, A. Stevanovic, N. Mitrovic, and E. Espino, “Traffic signal optimization to improve sustainability: A literature review,” Energies, vol. 15, no. 22, p. 8452, 2022.

[24] D. N. Shetty, “Minimizing Blast Radius of Chaos Engineering Experiments via Steady-State Metrics Forecasting,” 2023.

[25] T. A. Saurin, S. Wiig, R. Patriarca, and T. O. Grotan, “The cost-effectiveness of resilient healthcare,” International Journal of Health Governance, vol. 28, no. 3, pp. 319–327, 2023.

[26] K. S. Chadha, “Leveraging Advanced Analytics And Ai-Powered Solutions For Transforming Healthcare And Retail E-Commerce: Enhancing Data Security, Personalization, And Compliance,” International Journal of Applied Mathematics, vol. 38, no. 9s, pp. 1310–1334. https://doi.org/10.12732/ijam.v38i9s.862

[27] B. Liu, J. Tong, and Y. Zhuang, “Edgeloc: A communication-adaptive parallel system for real-time localization in infrastructure-assisted autonomous driving,” arXiv preprint arXiv:2405.12120, 2024.

[28] L. A. Haibeh, M. C. Yagoub, and A. Jarray, “A survey on mobile edge computing infrastructure: Design, resource management, and optimization approaches,” IEEE Access, vol. 10, pp. 27591–27610, 2022.

[29] C. D. Nguyen, “High Performance Distributed Transactions for Multi-Region Database Systems,” Ph.D. dissertation, University of Maryland, College Park, 2024.

[30] A. C. Jha, “Automated firewall policy generation with reinforcement learning,” International Journal of Internet of Things, 2025. https://www.academicpublishers.org/journals/index.php/ijiot/article/view/5483.

[31] J. Wang, “High Performance and Dependable Asynchronous Communication on Multi-Core Systems,” 2024.

[32] K. Lulla, “Cross-border compliance and quality assurance in semiconductor manufacturing,” JES, 2025. https://journal.esrgroups.org/jes/article/view/9196/6076

[33] P. R. Vennamaneni, “Modernizing monoliths: Transitioning legacy financial systems to microservices,” JISEM Journal, 2025. https://www.jisem-journal.com/index.php/journal/article/view/8896.

[34] V. K. Enugala, “AI-powered crack propagation predictions,” JES, 2025. https://journal.esrgroups.org/jes/article/view/9203.

[35] Goede, H., Ge, C., & Fransman, W. (2024). Meta-analysis of the quantitative effectiveness of risk management measures (RMM) in the workplace. Annals of work exposures and health, 68(5), 495-509.

[36] Somashekar, G., Tandon, K., Kini, A., Chang, C. C., Husak, P., Bhagwan, R., ... & Natarajan, N. (2024). {OPPerTune}:{Post-Deployment} configuration tuning of services made easy. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24) (pp. 1101-1120).

[37] Filip, F. G., & Leiviskä, K. (2023). Infrastructure and complex systems automation. In Springer Handbook of Automation (pp. 617-640). Cham: Springer International Publishing.