Most Errors Shouldn't Stop Anything
A formatting glitch shouldn't crash the whole proposal.
The Architecture Series · 8 minute read · By Ed | Founder & CEO, DealiOS
737 try/catch blocks across 250 files.
That's not error handling. That's an error philosophy. And it says something specific: the system has decided, in advance, which failures matter and which ones don't.
Most engineers treat every error as equal. A failed database write and a failed notification email get the same treatment: catch, log, maybe throw. The result: a user sees a 500 error because a CRM sync timed out. The proposal they just built is gone. Not because anything went wrong with the proposal. Because a secondary system had a bad day and took the primary workflow down with it.
That's not a bug. That's a missing philosophy.
The Google SRE team has documented this pattern extensively. In Chapter 22 of their Site Reliability Engineering book, they walk through how cascading failures propagate: a single non-critical dependency stalls, load shifts to remaining resources, timeouts pile up, and suddenly the entire service is down because a component that didn't need to be in the critical path was treated as if it were [1]. The 2025 Uptime Institute Annual Outage Analysis found that IT and networking failures accounted for 23% of impactful data center outages, with many triggered not by primary system failures but by dependency chains that should have been isolated [2].
The pattern repeats across the industry. A secondary system fails. The primary workflow trusts it unconditionally. The user pays the price.
The Three Tiers
The system divides every operation into three tiers. Not by importance to the business. By importance to the user's immediate action.
Tier 1: Must succeed. Authentication. Workspace creation membership step. Document saves. Contract signing. Pipeline state transitions. If these fail, the system stops and tells the user. There is no graceful degradation for a failed authentication check. There is no "try again later" for a document save. These operations are the user's primary workflow. They succeed or they surface an honest error.
Tier 2: Should succeed, never block. RAG indexing. CRM sync. Activity logging. Knowledge pack seeding. Calculator template seeding. Contract clause seeding. Revenue entry creation. Notification sending. Engagement score recalculation. If these fail, the system logs the error, moves on, and retries later. The user never sees the failure. The user's primary action completes successfully regardless.
Tier 3: Might not exist. DocuSign configuration. External CRM connection. Custom calculator templates. Third-party integrations that the workspace may or may not have configured. If these fail, the system degrades gracefully. The feature that depends on the integration simply isn't available. No error. No spinner. Just a reduced feature set that still works.
Treating every error the same is like a hospital running the ER and the gift shop with the same urgency protocol. When the gift shop runs out of teddy bears, nobody should be paging a surgeon.
Treating every error the same is like a hospital running the ER and the gift shop with the same urgency protocol. When the gift shop runs out of teddy bears, nobody should be paging a surgeon.
That's not a joke about prioritization. It's the exact failure mode that takes down production systems. A non-critical operation fails. The failure propagates up the call stack because nobody classified it. The user's critical workflow crashes because the system couldn't distinguish between "the database is down" and "the analytics event didn't fire."
The syncCalculatorToProposal Pattern
There's a function in our codebase called syncCalculatorToProposal. Its job: keep the proposal's financial data aligned with the calculator's current state. It runs every time the calculator updates.
The comment above the function: "Non-blocking. Don't break the caller's flow."
The comment that appears repeatedly in our codebase: "Don't break the caller's flow." That's not error handling. That's a philosophy.
The calculator sync is important. It keeps the proposal's numbers consistent. But if it fails, the proposal still exists. It still saves. It still generates correctly. The numbers in the proposal might be one edit behind the calculator until the next successful sync. That's an acceptable temporary inconsistency. The alternative, crashing the proposal save because the calculator sync timed out, is not acceptable under any definition.
This pattern repeats across every Tier 2 operation. The pattern is always the same: wrap the operation in its own error boundary, execute it asynchronously, log failures, and let the primary workflow continue. The technical implementation varies (Promises, background queues, event-driven handlers) but the philosophy is constant: don't break the caller's flow.
The engineering instinct fights this. Engineers want consistency. If the calculator updated, the proposal should reflect it immediately. Synchronous execution guarantees that. It also guarantees that a calculator service timeout becomes a proposal service timeout. You've traded eventual consistency for coupled failure. That's not a good trade.
The Pre-Stream Save Pattern
This is the pattern born from paranoia. Justified paranoia.
When a user triggers AI generation in DealiOS, the response streams back in real time via Server-Sent Events. The user watches their proposal take shape, sentence by sentence. When the stream completes, the full response needs to be saved to the database.
The obvious implementation: when the stream ends, save the complete message. The problem: serverless functions can be reclaimed after the response stream ends. The runtime doesn't guarantee any post-stream execution time. You might get 100 milliseconds. You might get zero.
The fix: save the assistant message to the database before the SSE stream closes. Not after. The comment in the code: "Serverless function may be reclaimed after response stream ends."
That's not error handling. That's understanding the environment well enough to not trust it. The serverless runtime is not your friend. It's a resource allocator with priorities that don't include "wait for your database write to finish." If you can't complete the save before the stream closes, you need to rethink your stream architecture.
This represents Tier 1 thinking elevated to what I call architectural paranoia. The most critical save in the workflow (the user's generated content) happens before the infrastructure can pull the rug. Not because the infrastructure is unreliable. Because the contract between your code and the runtime doesn't guarantee post-stream execution. And in production, anything that isn't guaranteed will eventually fail.
The Notification Quarantine
Notifications are the classic Tier 2 operation that engineers accidentally promote to Tier 1.
A user publishes a proposal. Eight things need to happen: the document state updates, the published link generates, the engagement tracking initializes, the pipeline stage advances, and notifications go out to the team. If you implement all eight operations in a single synchronous chain, a failed notification email crashes the entire publish flow. The proposal isn't published. The link doesn't generate. The user sees an error. Because an email didn't send.
Our notification system is quarantined. It exists in its own execution context. It receives events. It processes them asynchronously. If it fails, it logs the failure and the message enters a retry queue. The user's publish action completed successfully several hundred milliseconds ago. They're already sharing the link with their client.
The same quarantine applies to CRM sync, activity logging, and engagement score recalculation. Each operates in its own failure domain. Each can fail independently without cascading into the user's workflow. The boundaries aren't just code organization. They're failure isolation.
This mirrors the core insight from Erik Hollnagel's Safety-II framework in resilience engineering: the goal isn't preventing all failures. It's ensuring the system continues to function under varying conditions, including conditions where parts of it are failing [3]. Traditional safety thinking (Safety-I) focuses on eliminating bad outcomes. Safety-II focuses on ensuring good outcomes continue even when components degrade. That's exactly what the non-fatal philosophy does. Side effects can fail. The user's work continues.
The RAG Indexing Example
Here's a Tier 2 operation that feels like it should be Tier 1: indexing a document into the RAG knowledge base.
When a user saves a document, the system needs to embed it into the vector store so the AI can reference it in future generations. This feels critical. Without it, the AI loses access to the most recent version. The knowledge base becomes stale. Future proposals might reference outdated information.
And yet: if the RAG indexing fails, the user's document still saves. The user can still edit it, publish it, share it. The immediate workflow is unaffected. The consequence of a failed index operation is that the AI's knowledge base is temporarily incomplete. The next successful indexing attempt will catch it up. The user never sees the gap because they don't interact with the RAG index directly. They interact with the AI's output, which might be marginally less informed until the index catches up.
The temptation is to make RAG indexing synchronous. The document saves and the index updates atomically. Consistent. Clean. And now a slow embedding model or a vector database hiccup adds 3 seconds to every save. Or worse: a vector database timeout kills the save entirely.
We chose Tier 2. The document saves immediately. The RAG index updates asynchronously. If the index operation fails, it enters a retry queue. The user's save was instant. The AI's knowledge catches up within minutes. The tradeoff is a narrow window of stale knowledge in exchange for a save operation that never fails because of a secondary system.
When Tiers Change
Classifications aren't permanent. An operation's tier can change based on context.
Calculator template seeding during workspace creation is Tier 2. If it fails, the workspace still creates. The user can add calculator templates manually later. But calculator synchronization during a proposal export is closer to Tier 1. If the financial numbers in the exported PDF don't match the calculator's current state, the user sends incorrect financials to their client. Same operation category (calculator sync). Different tier based on the stakes of the moment.
The three-tier classification isn't a static label stamped on each operation at design time. It's a contextual assessment that considers: what is the user doing right now? What would they lose if this operation failed? How immediate is the consequence?
Some operations exist at different tiers depending on the workflow that triggered them. The architecture handles this by evaluating the calling context, not just the operation itself. A CRM sync triggered by a background scheduler is pure Tier 2. A CRM sync triggered by a contract signing event is elevated: still non-blocking, but with aggressive retry behavior and an alert if it fails after three attempts, because the signed contract needs to reach the CRM before the next business morning.
This contextual classification is where the philosophy graduates from a simple framework to a production discipline. The question isn't just "does the user notice if this fails?" It's "does the user notice if this fails right now, in this specific workflow, at this specific moment?"
The Universal Framework
For every operation in the system, one question determines its tier classification:
"If this fails silently, does the user notice?"
If yes: Tier 1. The operation is in the critical path. It must succeed or surface an honest error.
If no: Tier 2. The operation is important but not immediate. It should succeed, it should retry on failure, but it must never block.
If the feature doesn't exist: Tier 3. The integration is optional. Its absence is a feature state, not an error state.
This question, asked during code review, prevents the majority of cascading failures before they reach production. It forces the engineer to think about the user's experience, not the system's consistency. It reframes error handling from "what should the system do?" to "what does the user need right now?"
The 737 try/catch blocks in our codebase aren't evidence of fragile code. They're evidence of intentional tier classification applied to every operation that touches something outside its own boundary. Each one represents a decision: this operation is Tier 2. It should try. If it fails, the world keeps turning.
The number will grow. Every new feature, every new integration, every new side effect adds more operations that need to be classified. The philosophy scales because the question scales. "If this fails silently, does the user notice?" works for 737 operations. It works for 7,000. It works for any system where secondary operations exist alongside a primary workflow.
That's every system. Which means this philosophy isn't optional. It's foundational.
The takeaway: Classify every operation by whether the user notices if it fails. Then never let a secondary system's bad day take down the primary workflow.
References
1. Beyer, B. et al., "Site Reliability Engineering: How Google Runs Production Systems," Chapter 22: Addressing Cascading Failures, O'Reilly Media, 2016 2. Uptime Institute, "Annual Outage Analysis 2025," Uptime Institute, 2025 3. Hollnagel, E., "Safety-I and Safety-II: The Past and Future of Safety Management," Ashgate Publishing, 2014
This paper is part of The Architecture Series by Ed, Founder & CEO of DealiOS. For the weekly series and more technical deep-dives, visit dealios.app/library.
See how these principles come together. Explore the platform →