Plan A Fails. Plan B Fails. Now What?
Production AI needs three layers of fallback.
The Architecture Series · 9 minute read · By Ed | Founder & CEO, DealiOS
Production systems don't fail in binary. They degrade.
The API doesn't return an error. It returns a 503 for 4 seconds and then recovers. The vector database isn't down. It's slow enough that the 2-second timeout fires. The webhook didn't fail. It just hasn't arrived yet.
Most architectures treat these as failures. The result: the feature breaks, the user sees an error, the system waits for a human to investigate. Meanwhile, a perfectly acceptable reduced version of the feature was available the whole time. Nobody built it because nobody asked the question: "What's the useful version of this feature when the primary path isn't available?"
That question changed how we build everything at DealiOS. Not as an afterthought. As the starting point. Before building any feature that depends on an external service, a runtime condition, or a multi-step operation, we design the degraded version first. Then we build upward.
Because the degraded experience is your floor. Everything above it is upside. If your floor is "nothing works," your users will find it. If your floor is "the essential workflow continues," they might never know anything was wrong.
The degraded experience is your floor. Everything above it is upside.
The Fallback Catalog
Seven documented fallback chains run through the DealiOS codebase. Each represents a conversation that happened before the primary feature was built: "What if this doesn't work?"
RAG retrieval. Primary: pgvector cosine similarity search against the embedded knowledge base. Fallback one: PostgreSQL full-text search against the raw document content. Fallback two: most recent documents by recency. The user asked the AI a question about their account. The vector database is slow. The system doesn't hang. It falls back to full-text search, which returns slightly less semantically precise results but still answers the question. If full-text search also fails, the system retrieves the most recent documents, because recent context is usually relevant context. Three tiers. Each less precise than the last. All better than an error message.
DocuSign integration. Primary: webhook notification when a contract is signed, validated with HMAC-SHA256. Fallback one: polling reconciliation that checks DocuSign status on a schedule, with different audit annotations ("reconciled via polling") to distinguish the source. Fallback two: manual void for signatures that need to be canceled. The webhook is the ideal path. It's also the path that depends on DocuSign's infrastructure delivering a callback to your server. Any engineer who trusts a webhook unconditionally hasn't been in production long enough. The polling fallback catches what the webhook misses.
Email sending. Tier 2 email: sent from the user's verified domain, with their name, their email identity. Professional. Branded. Ideal. Tier 1 fallback: sent from the platform address, formatted as "User Name via DealiOS," with reply-to set to the user's email. Not as professional. Not branded. But the customer gets the email. The proposal arrives. The deal moves forward.
A degraded experience is the food truck. You still eat. You just don't get the tablecloth. And when the alternative is starving, nobody cares about the tablecloth.
The difference between a great restaurant and a good food truck isn't the food. It's the tablecloth. A degraded experience is the food truck. You still eat. You just don't get the tablecloth. And when the alternative is starving, nobody cares about the tablecloth.
Brand extraction. When a new workspace is created, the system extracts the company's logo from their website. Primary: apple-touch-icon (highest resolution, most reliable). Fallback one: favicon. Fallback two: og:image from social meta tags. Fallback three: null (the workspace works without a logo). Four attempts to get a single image. Because a workspace with a logo feels polished and a workspace without one still functions.
Logo resolution. Even after extraction, logos can fail at render time. Primary: URL reference. Fallback one: base64-encoded inline data. Fallback two: alternate variant (if the workspace has multiple logo formats). Fallback three: null (the document renders without a logo). A proposal without the client's logo isn't broken. It's just slightly less impressive. And slightly less impressive is infinitely better than a 500 error.
Calculator state. Primary: active calculator template with all financial models loaded. Fallback one: legacy format detection (for older calculators created before the current schema). Fallback two: null. The proposal generates without a calculator section. The user can add financial data manually. The feature degrades. The workflow continues.
Workspace creation. Primary: full 10-step seed cascade with industry-specific configuration, membership, execution types, taxonomies, billing, knowledge packs, calculator templates, clause library, contract templates, and brand extraction. Fallback: non-fatal partial seeds. Step 2 (membership) fails, the entire workspace creation rolls back. Can't have a workspace without an owner. But steps 3 through 10 log errors and continue. Calculator template failed to seed? The workspace still works. Clause library didn't load? The user adds their own. Knowledge pack timed out? Try again later. A partially seeded workspace is infinitely better than no workspace.
The RAG Fallback Chain in Detail
The RAG retrieval chain is worth examining more closely because it illustrates how each fallback level trades precision for reliability.
pgvector cosine similarity is the ideal. The system embeds the user's query, searches the vector store for semantically similar content, and returns the most relevant passages. The results understand meaning, not just keywords. A query about "pricing strategy" returns documents about margin analysis and deal economics even if those exact words don't appear.
But pgvector depends on the embedding model being available and the vector store being responsive within the timeout window. If the embedding model is slow (429 from the provider, network latency spike, cold start on the inference endpoint), the 2-second timeout fires. The feature is unavailable.
Fallback one: PostgreSQL full-text search. No embeddings needed. No external model call. Pure text matching against the raw document content stored in the same database that's already connected. The results are less semantically precise. "Pricing strategy" returns documents that contain those words, not documents about related concepts. But the query executes in milliseconds against a local index, and the results are still useful.
Fallback two: most recent documents by recency. No search at all. If full-text search also fails (which would require the primary database itself to be degraded), the system retrieves the five most recently modified documents in the workspace. The logic: recent documents are usually relevant documents. A user asking the AI for help with a proposal is likely working with materials they touched in the last few days. Recency is a crude proxy for relevance, but it's a proxy, not a void.
The result: the user asks the AI a question. The AI answers using the best available context. If the vector store is responsive, the answer is excellent. If it's slow, the answer is good. If everything is degraded, the answer is based on recent context. In no scenario does the user see an error message. In no scenario is the AI unable to respond.
The fallback chain doesn't just provide alternatives. It defines a gradient of quality. The user's experience moves smoothly down that gradient as dependencies degrade, rather than falling off a cliff from "working perfectly" to "nothing works."
The Tier 1 Email Pattern in Detail
This fallback chain is worth examining closely because it illustrates a design tension that every degradation plan encounters: the gap between ideal and acceptable.
Tier 2 email is the ideal. The customer receives an email from sarah@clientcompany.com. It looks like Sarah wrote it. The domain is verified. The formatting matches the company's email standards. It's professional, personal, and branded.
But Tier 2 email requires a verified sending domain. Not every workspace has one configured. DKIM records need to be set. SPF needs to pass. Domain verification takes time. What happens before that's done?
Tier 1 email. The customer receives an email from proposals@dealios.app. The sender line reads "Sarah Chen via DealiOS." The reply-to header points to sarah@clientcompany.com, so when the customer hits reply, it goes to Sarah, not to the platform. The email body is identical. The content is the same. The experience is degraded only at the envelope level.
Is Tier 1 email as good as Tier 2? No. Does the customer still receive the proposal? Yes. Does the deal still move forward? Yes. Does Sarah know the system fell back? She was told during setup that emails would send from the platform domain until her company's domain was verified. No surprise. No frustration. Just a reduced experience that's fully transparent.
That transparency matters. Degradation should never be hidden from the user when it affects their professional presentation. The user should know which tier they're operating at and what they can do to upgrade. Hidden degradation breeds mistrust when the user eventually notices.
The "Proposal Works Without Calculator" Principle
This is the hardest design decision in the fallback catalog: deciding that an incomplete feature is better than a blocked workflow.
The calculator is important. It ensures mathematical accuracy (Library Paper #1). It provides the financial analysis that closes deals. It's one of the most valued features in the platform. And the proposal generates without it if the calculator template fails to load.
The numbers section is absent or uses manual inputs. The proposal is less powerful. The deal analysis is less automated. But the proposal exists. The user can share it. The deal can advance. The financial details can be added manually or regenerated once the calculator issue resolves.
This requires a specific kind of product discipline: ranking features by "must-have for any value" versus "important for full value." The proposal's core value is presenting a professional, persuasive document to a buyer. The calculator enhances that value enormously. But the core value survives without it.
Not every feature passes this test. A proposal without content is not a proposal. A workspace without an owner is not a workspace. A contract without terms is not a contract. These are features where "must-have for any value" means degradation isn't possible. The floor is the whole feature.
The discipline is knowing the difference. And designing the degradation plan for every feature that can survive without its dependencies before building the primary experience that depends on them.
Design the Degradation First
Here's the principle that ties all seven fallback chains together:
Before building the primary experience, ask: "What's the version of this that works when the primary dependency is unavailable?" Design that first. Then build upward.
This inverts the normal development sequence. The standard approach: build the feature, test the feature, then figure out what to do when it breaks. The result is error handling that feels bolted on, because it was bolted on. The degraded experience is an afterthought that nobody designed, nobody tested, and nobody wants to maintain.
The inverted approach: design the degraded experience first. Decide what "good enough" looks like when the vector database is slow, when the webhook doesn't arrive, when the email domain isn't verified, when the calculator template fails to load. Build that. Test that. Make sure it actually works. Then build the primary experience on top of it.
This feels slower in the short term. It's faster in the long term, because you never face the panicked 2 AM production incident where a dependency failed and nobody built a fallback. The fallback was the first thing you built. It's been running in production for months. It's battle-tested. It works.
Every one of the seven fallback chains in DealiOS was designed before the primary path was finalized. The RAG retrieval fallbacks existed before the pgvector implementation was complete. The email Tier 1 fallback existed before any workspace had a verified domain. The calculator degradation path existed before the calculator feature was fully built.
The floor was built first. Everything above it is upside.
Testing the Floor
A fallback that has never executed in production is a fallback you're hoping works. Hope is not an engineering strategy.
Each of the seven fallback chains gets tested deliberately. Not just in unit tests. In production conditions. The email Tier 1 fallback runs for every workspace that hasn't yet configured a verified sending domain. That's not a failure scenario. That's the normal state for new workspaces. The fallback is exercised daily.
The RAG fallback chain is tested by artificially introducing timeouts during development. Set the pgvector timeout to 1 millisecond. Does the full-text search fallback activate correctly? Set the full-text search to also fail. Does the recency fallback produce useful results? The answers to these questions should come from testing, not from the first production incident.
The DocuSign polling fallback runs on a schedule regardless of whether webhooks are arriving. It's not dormant, waiting to be activated by failure. It's continuously verifying that signed contracts are in the expected state. If the webhook delivered the notification, the polling check confirms it and moves on. If the webhook was missed, the polling check catches it. The fallback isn't a backup system. It's a parallel verification system that happens to also catch failures.
This continuous exercise of fallback paths means they're maintained, monitored, and trusted. An untested fallback is worse than no fallback, because it gives you false confidence. You believe you have a safety net. When the primary path fails and you land on it, it breaks.
Production systems don't fail in binary. They degrade. The question isn't whether your system will degrade. It's whether you designed the degradation or left it to chance.
The takeaway: Before building any feature, design the degraded version first. The floor you build is the experience your users will find when everything else fails.
References
1. Beyer, B. et al., "Site Reliability Engineering: How Google Runs Production Systems," O'Reilly Media, 2016 2. Hollnagel, E., "Safety-II in Practice: Developing the Resilience Potentials," Routledge, 2018
This paper is part of The Architecture Series by Ed, Founder & CEO of DealiOS. For the weekly series and more technical deep-dives, visit dealios.app/library.
See how these principles come together. Explore the platform →