Building Resilient AI-powered Software, And Planning For When Things Go Wrong
AIThis post was created with the assistance of artificial intelligence (AI).

TL;DR

Buying for a business?Offer from Amazon

Get business pricing on networking and server gear

  • Business-only prices and quantity discounts
  • Tax-exempt purchasing
  • Multiple users, one account, clear invoices
As an affiliate, we earn on qualifying purchases.

The Pragmatic Engineer published excerpts from Sam Newman’s 2026 book, Building Resilient Distributed Systems, covering disaster recovery and resilience in AI-powered software. The material emphasizes setting recovery targets, weighing recovery costs and complexity, and planning for changes or disruptions involving AI models and infrastructure.

The Pragmatic Engineer has published excerpts from Sam Newman’s 2026 book, Building Resilient Distributed Systems, outlining how software teams can plan for catastrophic outages and make AI-powered systems more resilient. The material covers disaster recovery targets, recovery patterns, infrastructure constraints and the challenges of relying on changing AI models and providers.

In the excerpt from Chapter 10, Newman describes disaster recovery (DR) as the procedures used when failures affect large parts of a system, including site-wide events such as fires, floods, power problems or cloud infrastructure failures. He frames DR planning as a trade-off: reducing acceptable downtime or data loss generally increases cost and complexity.

Newman recommends defining two measurable targets. The recovery point objective (RPO) sets the maximum tolerable amount of data loss, expressed as the time since data was last backed up or synchronized. The recovery time objective (RTO) sets how long a system can remain unavailable. RTO includes the work of restoring data, making computing resources available and redirecting traffic.

The report says the book also addresses backup-and-restore, pilot light, warm failover and active-active recovery patterns, as well as objectives and service-level indicators for disaster recovery. Its AI-related coverage includes model-provider resilience, the possibility that models change, and the need for observability. The report also points to multi-model and multi-vendor strategies, transparent costs and regular vendor reviews as ways to manage uncertainty.

At a glance
reportWhen: Published in 2026; the source says the…
The developmentThe Pragmatic Engineer published excerpts from Sam Newman’s new book on disaster recovery and resilience for distributed and AI-powered systems.

Recovery Targets Shape Outage Readiness

The excerpts make a practical point for organizations whose software supports important services: resilience requires decisions made before an outage, not only technical fixes during one. RPO and RTO give teams concrete limits for acceptable data loss and downtime, helping them select a recovery design that fits their service and budget.

AI systems add dependencies that can change independently of an application’s own code. If a model provider is unavailable, changes a model, or alters service terms, teams may need alternatives and monitoring to maintain expected behavior. Newman’s recommendations, as summarized by the report, treat provider and model uncertainty as operational risks that belong in resilience planning.

Capacity is another constraint. The source says ongoing GPU and CPU shortages mean teams cannot assume that replacement infrastructure will always be available when needed. Planning must account for vendor limits and prioritize critical services, rather than treating failover capacity as guaranteed.

Amazon

disaster recovery backup software

As an affiliate, we earn on qualifying purchases.

As an affiliate, we earn on qualifying purchases.

A Book Updated for AI Dependencies

The Pragmatic Engineer report says Newman spent three years writing the book, rather than the two he originally planned, and revisited its content to account for AI. The publication shared selected material from two chapters, not the full book; the book contains 16 chapters, according to the report.

The excerpt uses “site” as a general term for the location of computing infrastructure, whether that is a public-cloud region, a private-cloud environment or a traditional data center. This lets the recovery patterns apply across different infrastructure setups. The report notes that questions around using multiple cloud vendors are treated in another chapter.

The Pragmatic Engineer’s author said the recommendation was unpaid and that the links were not affiliate links. The excerpts are reproduced with permission from O’Reilly Media, which published Building Resilient Distributed Systems in 2026.

“The recovery point objective (RPO) defines the maximum amount of time that can have passed since the data was last backed up or synchronized.”

— Sam Newman, in the book excerpt

Amazon

AI model monitoring tools

As an affiliate, we earn on qualifying purchases.

As an affiliate, we earn on qualifying purchases.

Costs, Capacity and Model Changes

The excerpts do not prescribe one recovery design for every organization or provide universal RPO and RTO targets. The appropriate limits depend on how much downtime and data loss a particular service can tolerate, alongside the cost and complexity of its recovery plan.

The source also does not quantify the scale or timing of GPU and CPU shortages, or identify specific vendors and model providers. It presents infrastructure availability and model change as planning concerns, but does not document particular outages or establish that AI-generated code is causing a measured rise in incidents. The report’s introductory comments about more deployments, bugs and outages are framing from the publication, not figures supported by data in the supplied material.

Amazon

cloud infrastructure failover solutions

As an affiliate, we earn on qualifying purchases.

As an affiliate, we earn on qualifying purchases.

Teams Must Set Their Own Recovery Plans

The next step for teams applying the excerpt is to set service-specific RPO and RTO targets, then compare recovery options against those limits, available capacity and cost. Plans also need to address how data is restored and how traffic moves to an alternative site.

For AI-powered software, the report points toward monitoring model behavior and provider availability, making costs visible, and reviewing vendor dependencies regularly. It does not announce a follow-up event or give a timetable for further coverage. The book’s wider treatment of disaster recovery and AI resilience is available in its full 16-chapter edition, according to The Pragmatic Engineer.

Amazon

distributed system resilience tools

As an affiliate, we earn on qualifying purchases.

As an affiliate, we earn on qualifying purchases.

Key Questions

What is the news?

The Pragmatic Engineer published excerpts from Sam Newman’s 2026 book, Building Resilient Distributed Systems, on disaster recovery and resilience in AI-powered software.

What do RPO and RTO mean?

RPO is the maximum acceptable period of potential data loss, based on when data was last backed up or synchronized. RTO is the maximum acceptable time a service remains unavailable after an incident.

Which disaster recovery approaches does the report mention?

The report lists backup-and-restore, pilot light failover, warm failover and multisite active-active as approaches discussed in the book.

The report says the book covers model-provider resilience, model changes and observability, and discusses measures such as using multiple models or vendors, tracking costs and reviewing vendors regularly.

Does the source identify a specific outage caused by AI?

No. The supplied material discusses outage and infrastructure risks generally, but does not document a particular incident or provide data proving that AI-generated code has caused a measured increase in outages.

Source: rss

FALL

Fall Picks

As an affiliate, we earn on qualifying purchases.

You May Also Like

Why Fanless Industrial PCs Are More Than Niche Edge Devices

Investigating why fanless industrial PCs surpass niche roles reveals how their durability and efficiency drive strategic advantages across industries.

Delphi 13 Community Edition Is Now Available

Embarcadero has announced the availability of Delphi 13 Community Edition, offering free access to developers for creating native Windows, macOS, iOS, and Android apps.

SteamOS 0.3.0 For Steam Frame Released

Valve has officially released SteamOS 0.3.0 for the Steam Frame, marking a significant update for the handheld gaming device. Details remain limited.

Google surges in global coverage

Google’s coverage has surged globally, with GDELT reporting 44 mentions in the recent window, indicating increased focus on international news.