Never Lose Work
The server crashes. The browser refreshes. Your document is still there.
The Architecture Series · 8 minute read · By Ed | Founder & CEO, DealiOS
"Never lose work" sounds like a feature. It's not. It's a stance.
It means every write path has a fallback. Every stream has recovery. Every draft has a safety net. The system trusts nothing to succeed on the first try. Not the API. Not the database. Not the network. Not the serverless runtime that might reclaim your process while the user is watching their document generate.
Most products add error handling after the happy path is built. The result: errors are second-class citizens. Recovery is an afterthought. The user sees "Something went wrong" and loses 20 minutes of work and all faith in the product.
Build the happy path to include the failure path. They're the same path.
Build the happy path to include the failure path. They're the same path.
That sentence changed how I think about every line of code in DealiOS. Not as an aspiration. As a constraint. Every component, every API call, every write operation starts with the same question: what happens when this doesn't work?
The Silent Failure
The worst failures aren't crashes. They're silent losses.
The API returned a 200 but the data didn't persist. The stream completed but the serverless function was reclaimed before the final save. The auto-save fired but the tab was hidden and the browser throttled the request. These are the failures that users discover 10 minutes later when their work is gone and there's nothing to recover.
A crash is honest. A crash says "I broke." A silent failure says nothing. It lets you keep working, full of hope, while your data dissolves behind the scenes. That's not a software bug. That's a betrayal of trust.
A crash is honest. A silent failure lets you keep working, full of hope, while your data dissolves behind the scenes. The software looked you in the eye and said "everything's fine." It was lying.
Every engineer has been on the receiving end of this. You're working in a tool, the connection hiccups, and when you come back the last 15 minutes of work have vanished into the void. You check the console. No errors. You check the network tab. All 200s. The system reported success at every level while silently dropping your data on the floor.
The standard industry response is to improve uptime. Make the server more reliable. Reduce the error rate. That helps. It doesn't solve the problem. Because the problem isn't that failures happen. The problem is that when failures happen, the user's work is unrecoverable. Reducing failure frequency from 1-in-1,000 to 1-in-10,000 means 10x fewer angry users. It still means angry users. The architectural stance is different: assume the failure will happen. Make sure the work survives it.
Local First, Server Truth
The approach we landed on: localStorage as the primary draft persistence layer. The server as the authoritative sync target.
When a user is editing a proposal in DealiOS, every keystroke update goes to localStorage first. Not as a backup. As the primary store. The local draft has a timestamp. The server copy has a timestamp. When the user returns to an in-progress document, the system compares the two. Newer local draft: offer recovery. Older local draft: silently discard. Seven-day garbage collection cleans up abandoned drafts.
This works because the unit of work has a clear identity (a specific document draft) and a clear owner (the user who created it). The localStorage key maps to that identity. The timestamp resolves conflicts. There's no complex conflict resolution algorithm because there's no multi-user write scenario to resolve. One document, one editor, two persistence layers negotiating through timestamps.
The implication matters: the server going down doesn't mean the user loses work. The network dropping doesn't mean the user loses work. The browser tab crashing doesn't mean the user loses work (localStorage survives tab crashes). The only catastrophic loss scenario is the user clearing their browser data while having unsaved changes. That's a narrow failure window. Architecturally, we've reduced the attack surface for data loss from "anything in the stack fails" to "the user deliberately clears their browser."
The mental model shift is subtle but important. Most engineers think of the server as the primary store and local state as a convenience cache. We inverted it for in-progress work. The local draft is the thing that matters because that's where the user's recent changes live. The server is where truth lives after a successful sync. Two different jobs. Two different persistence layers. Each optimized for its role.
Error Classification for AI Retries
Not all errors are equal. A system that treats every failure the same way will either retry things it shouldn't or give up on things it should keep trying.
We sort errors into two categories: retryable and not retryable. The classification happens at the point of failure, not at the point of display.
Rate limits (429): retryable. The API is telling you it's busy, not broken. Exponential backoff with jitter. Wait, try again, wait longer, try again. These almost always succeed within three attempts.
Overloads (529): retryable with longer backoff. The upstream provider (in our case, often the AI model endpoint) is temporarily overwhelmed. Same pattern, longer intervals.
Timeouts: retryable with state preservation. The critical detail is preserving the user's context so the retry picks up where the failure left off, not from the beginning.
Auth failures (401/403): not retryable. The credentials are wrong or expired. No amount of retrying will fix a permissions problem. Surface immediately.
Invalid requests (400): not retryable. The system sent something the API doesn't understand. Surface with a human-readable translation via our friendlyAIError() function, which converts cryptic API error codes into sentences a non-technical user can act on.
The system doesn't retry everything. It retries the right things and fails fast on the rest. This distinction is the difference between a system that recovers gracefully from transient failures and a system that hammers a broken endpoint for 30 seconds while the user stares at a spinner.
Fire-and-Forget Side Effects
Activity logging. CRM sync. Calculator sync. Engagement score recalculation. RAG indexing. Notification dispatch.
These operations are important. They keep the system consistent, the data current, the analytics accurate. They are also never, under any circumstances, allowed to block the user's primary action.
The comment that appears repeatedly in our codebase: "don't break the caller's flow."
If the CRM sync fails, the proposal still saves. If the activity log times out, the document still generates. If the engagement score recalculation errors, the client still receives the published link. Every side effect is wrapped in its own error boundary. Every side effect fires asynchronously. Every side effect can fail completely and the user never knows.
This is a philosophy: side effects serve the system, not the user's immediate action. They can catch up later. The CRM will sync on the next successful attempt. The activity log will backfill. The engagement score will recalculate on the next event. None of these are urgent enough to justify interrupting the user's workflow.
The temptation for engineers is to make everything synchronous and consistent. If the CRM sync is part of the save operation, the data is always consistent. True. It also means a Salesforce timeout kills your user's proposal save. The consistency you gained is worthless because the user just lost trust in your product. Eventual consistency on secondary operations is a feature, not a compromise.
The Serverless Trust Problem
Here's a failure mode most engineers don't think about until it bites them: serverless function reclamation.
When a user triggers an AI generation in DealiOS, the response streams back via Server-Sent Events (SSE). The serverless function stays alive as long as the stream is open. But the moment the stream closes, the runtime can reclaim that function. Not "will eventually reclaim." Can reclaim immediately. The function might have 50 milliseconds of post-stream life. It might have zero.
If the final database save happens after the stream closes, there's a race condition between your save operation and the runtime pulling the plug. Lose that race and the user watched their entire document generate, saw it appear in real time, and then discovered it was never saved.
The fix is aggressively simple: save the assistant message to the database before the SSE stream closes. Not after. Before. The comment in the code: "Serverless function may be reclaimed after response stream ends." That's not error handling. That's understanding the environment well enough to not trust it.
The same paranoia extends to page visibility. When a browser tab is hidden, the browser can throttle timers, pause requests, and deprioritize network operations. Our auto-save system detects page visibility state changes and adjusts accordingly. Tab hidden: pause auto-save (because the request might get throttled anyway). Tab visible: resume auto-save and force an immediate sync. The user switches back to the tab and the system immediately catches up.
The Page Visibility Problem
Browsers are not passive containers for your application. They're active resource managers that make decisions about your code without asking permission.
When a user switches to a different tab, the browser can throttle setTimeout and setInterval to fire no more than once per second. It can deprioritize or delay network requests. It can suspend JavaScript execution entirely if the device is under memory pressure. Your auto-save that fires every 10 seconds in the foreground might fire once per minute in the background. Or not at all.
Our auto-save system accounts for this. It monitors the Page Visibility API to track whether the tab is active. Tab hidden: pause the auto-save interval entirely. Don't bother sending a request that the browser will throttle or drop. Tab visible again: immediately force a sync. Don't wait for the next interval. The user just came back. Catch up now.
The edge case that prompted this: a user was editing a proposal, switched to their email tab for 5 minutes, switched back, made a few more edits, and closed the browser. The last auto-save had fired while the tab was hidden. The browser throttled the request. The save failed silently. The user's last 5 minutes of work vanished.
After we added visibility-aware auto-save, that failure mode disappeared. Not because the browser stopped throttling requests. Because we stopped sending requests the browser would throttle.
Don't fight the platform. Understand its behavior and design around it.
The broader principle: don't fight the platform. Understand its behavior and design around it. The browser throttles background tabs. Accept that. The serverless runtime reclaims idle functions. Accept that. The network drops connections. Accept that. Every constraint the platform imposes is a failure mode you need to handle, not a bug you need to work around.
The Checklist
Every component, every feature, every write operation in the system gets filtered through these questions:
For every write: "What happens if this fails silently?" If the answer is "the user loses work," the architecture isn't done.
For every stream: "What happens if the runtime reclaims the process?" If the answer is "the last chunk doesn't persist," the save order is wrong.
For every external call: "What happens if this takes 30 seconds?" If the answer is "the user is blocked," the operation needs to be non-blocking.
For every draft: "Where does the user's work live if the server disappears?" If the answer is "nowhere," the local persistence layer is missing.
For every side effect: "Does this need to succeed for the user's action to complete?" If the answer is "no," it should be fire-and-forget.
This checklist doesn't prevent failures. It prevents failures from reaching the user. The system still fails. APIs still time out. Databases still hiccup. Networks still drop. But the user's work survives because every component was designed with the assumption that everything around it would break.
That's the stance. Not "handle errors." Not "add retries." Not "improve uptime." Design every component as if the rest of the system is unreliable. Because it is. Every system is unreliable at sufficient scale and sufficient time. The only question is whether you designed for that truth or pretended it away.
Build the happy path to include the failure path. They're the same path.
The takeaway: "Never lose work" is not a feature: it's a design stance where every component is built assuming the rest of the system is unreliable, because it is.
References
1. Google, "Site Reliability Engineering," Chapter 22: Addressing Cascading Failures, O'Reilly Media, 2016 2. Hollnagel, E., "Safety-I and Safety-II: The Past and Future of Safety Management," Ashgate Publishing, 2014 3. Uptime Institute, "2024 Annual Outage Analysis," Uptime Institute, 2024
This paper is part of The Architecture Series by Ed, Founder & CEO of DealiOS. For the weekly series and more technical deep-dives, visit dealios.app/library.
See how these principles come together. Explore the platform →