Durable Agent Checkpointing with a Key-Value Store

An AI agent that runs for ten seconds and an AI agent that runs for ten minutes are not the same kind of program. The short one is a function call: it either returns or it does not, and if it does not you retry the whole thing. The long one is a process. It calls a model, waits on a tool, calls the model again, pages a human, waits some more, and the longer it runs the more certain it becomes that something will interrupt it before the end. The model provider rate-limits you. The container gets evicted. A tool call times out. The deploy that ships your bug fix also restarts the worker mid-task. In 2026 the polite name for surviving all of that is "durable execution," and the unglamorous machinery underneath it is a checkpoint: a serialized snapshot of where the agent was, written somewhere that outlives the crash.

This is where a lot of teams quietly back into needing a database they did not plan for. LangGraph ships a checkpointer interface; durable-workflow frameworks ship their own persistence; and the common shape underneath every one of them is the same. A checkpoint is a blob of state keyed by a run identifier. You write it after each step, you read the latest one when you resume, and the only hard requirement is that it is still there after the thing that killed your agent. That is a key-value write with a durability guarantee. This article is about treating it as exactly that: what the checkpoint has to capture, how to design the keys, why an in-memory checkpointer is a contradiction in terms, and how the resume flow stays correct when a step runs twice.

What a checkpoint actually has to capture

A checkpoint is not a log line and it is not a transcript. It is the minimum state from which the agent can be reconstructed and continued as if it had never stopped. If you store too little, resume produces a different run than the one that crashed. If you store too much, every step pays a serialization tax and your checkpoints turn into a slow, fat write path. The discipline is in choosing the boundary.

Four things almost always belong in the snapshot:

• The step or node position. Which point in the graph or workflow the agent had reached. On resume this is the program counter. Without it you cannot know whether to re-plan, call a tool, or finalize.
• The accumulated state. For a conversational agent this is the message list; for a structured workflow it is the typed state object (scratchpad, extracted fields, intermediate results, the running plan). This is what makes the next model call produce the right next token rather than starting cold.
• Pending tool calls. If the model has already decided to call search(query) but the tool has not returned, that intent is part of the state. A resume that forgets it will either skip the tool or, worse, re-ask the model and get a different decision.
• External cursors. Anything that tracks progress through the outside world: the offset into a list of files being processed, the last page fetched, the last row written. These are how you avoid redoing work, and they are how the resumed run stays idempotent (more on that below).

What does not belong: large artifacts and long-term memory. A 40 MB generated PDF, a corpus of retrieved documents, the agent's semantic memory of every past conversation - none of that should be inlined into a per-step snapshot you rewrite constantly. The checkpoint stores a reference (a key, a URL, an id) and the heavy thing lives elsewhere. That boundary is the difference between checkpointing (short-term, run-scoped, hot) and memory (long-term, cross-run, cold), and it is worth being deliberate about. We cover the memory side in the agent memory pattern on a KV store, and the framework-level version of the same split - LangGraph's checkpointer versus its store - is laid out well by memnode.

The key design: thread, checkpoint, latest

Once you accept that a checkpoint is a KV value, the interesting work is the key schema, because the key is what lets you find the right snapshot in O(1) instead of scanning. The unit of durable execution is usually a thread or run, identified by a stable id. Everything hangs off that.

A schema that has served well in practice:

thread:{thread_id}:checkpoint:{n}      -> serialized snapshot for step n  (history)
thread:{thread_id}:latest              -> n, or the serialized latest snapshot (fast resume)
thread:{thread_id}:meta                -> status, created_at, parent run, idempotency markers
thread:{thread_id}:lock                -> optional: who is currently driving this thread

There are two distinct write strategies hiding in that list, and you usually want both at once.

The first is append for history. Each step writes a brand new key, checkpoint:1, checkpoint:2, and so on, and never overwrites an earlier one. This gives you time travel: you can replay the run, branch from step 3 into an alternate continuation, or debug exactly what the agent believed at each point. LangGraph's checkpointer model is built around this - every superstep produces an immutable checkpoint, and a thread is the ordered list of them. The cost is storage growth, which you bound with a retention policy (keep the last N, or TTL old threads).

The second is overwrite for latest. Resume does not want to read the whole history; it wants the single most recent good snapshot, now. A thread:{id}:latest pointer (or a directly overwritten latest value) gives you that in one read. The append history and the latest pointer are not redundant - they answer different questions. History answers "how did we get here," latest answers "where do I restart."

The keyspace is naturally hierarchical, and a B+tree-backed store leans into that: every key under thread:{id}: is a contiguous range, so listing a thread's history, or deleting a finished thread, is one ordered scan rather than a scattered set of point deletes. The lexicographic ordering that makes range scans cheap is the same property covered in persistent key-value storage, and it is why prefix-structured keys are not just tidy but fast.

Durability is the whole point, not a feature

Here is the sentence that should govern the entire backend choice: an in-memory checkpointer that evicts under pressure defeats the purpose of checkpointing. The reason you write checkpoints at all is to survive a crash. If the store holding them lives in the same process that crashed, or in a cache that can drop your key when memory gets tight, then the moment you most need the last checkpoint is exactly the moment it is gone.

This sounds obvious and yet it is the most common mistake, because the default checkpointer in most frameworks is in-memory. LangGraph's MemorySaver is documented as a development convenience and explicitly not for production, and the equivalent footgun exists everywhere: a Redis instance configured with maxmemory and an LRU eviction policy will silently delete a checkpoint key to make room, and a Redis instance without persistence loses every checkpoint on restart - which is to say, on exactly the event you were guarding against. A checkpoint store has two non-negotiable properties:

| Property | Why it matters for checkpoints | What breaks without it | |---|---|---| | No eviction | The key must still exist after the crash, however long the agent paused | Resume finds nothing; the run restarts from zero or is lost | | Durable write (survives restart) | The crash often is the process or host restarting | All in-flight runs vanish on deploy/OOM | | Atomic write | A half-written checkpoint is worse than none | Resume loads corrupt state and behaves unpredictably | | Read-your-write consistency | Resume reads what the last step just wrote | Eventual consistency serves a stale or missing snapshot |

The third row deserves emphasis. A checkpoint write must be all-or-nothing. If the process dies in the middle of serializing step 7, you must resume from the intact step 6, not from a snapshot that is half step 6 and half step 7. This is the atomic-claim discipline that shows up across agent infrastructure - the same reasoning behind idempotency keys with a KV store - and it is a property of the store, not something you can bolt on in application code after the fact.

This is the lane BaseKV is built for. It speaks the Redis API, so any checkpointer that talks to Redis points at it unchanged, but it stores every value in a single B+tree file on disk with no eviction policy and atomic writes. There is no maxmemory that can drop your checkpoint and no in-process cache that vanishes on restart. The snapshot you wrote after step 6 is on disk after the OOM kill, which is the entire job.

The resume flow, step by step

Resuming is the read side of the contract, and it is short when the keys are right. When an agent (or its supervisor) starts for a given thread_id, the flow is:

1. Read the latest pointer. GET thread:{id}:latest. If it is empty, this is a fresh run; initialize state and start at step 0.
2. Load the snapshot. Deserialize the latest checkpoint into the agent's state object: messages, scratchpad, cursors, pending tool calls.
3. Reconcile pending work. If the snapshot says a tool call was issued but not completed, decide whether to re-issue it (safe only if the tool is idempotent) or to check whether it actually completed before the crash. This is where external cursors earn their keep.
4. Continue from the recorded position. Hand the state back to the runtime and let it execute the next step, which in turn writes checkpoint:{n+1} and updates latest.

The whole loop is read-latest, do-one-step, write-next, and it is robust precisely because each step is bracketed by a durable write. The granularity of "one step" is a tuning decision: checkpoint after every model call and you can resume almost anywhere but you pay more writes; checkpoint only at coarse milestones and you redo more work on resume but write less. For most agents, per-node (per-superstep) checkpointing is the right grain, which is the default the frameworks chose for good reason.

In a multi-agent setup the resume flow also has to answer "who is allowed to drive this thread right now," because two workers both resuming the same thread from the same checkpoint is a recipe for double execution. That coordination - a short-lived lease on the thread - is its own pattern; see distributed locks with a KV store for the lock, and shared state for multi-agent workflows for the broader blackboard the checkpoints live alongside.

Idempotency: the part that bites on resume

Durable execution has a sharp edge that pure persistence does not solve: a resumed step may run for the second time. Suppose the agent's step was "charge the customer, then write the result." It charged the card, the charge succeeded, and the process died before the checkpoint that recorded "charged" was written. On resume, the last durable checkpoint says "about to charge." So the agent charges again. The customer is billed twice, and your durable workflow has faithfully, reliably, durably done the wrong thing.

The fix is that any step with a side effect must be idempotent, and the checkpoint is what makes that possible. Two complementary techniques:

• Record the cursor before acting, complete after. Write the intent ("processing item 42, idempotency key X") into the checkpoint, perform the side effect tagged with that idempotency key, then advance the cursor. A resume re-reads the same intent and the same key, and the downstream system (or your own KV claim) rejects the duplicate.
• Make the effect itself a conditional KV claim. Before charging, atomically claim effect:{idempotency_key} with SET-if-not-exists. If the claim fails, the effect already happened; skip it. This is the same atomic-claim machinery from idempotency keys with a KV store, now doing duty inside the agent loop.

The point worth internalizing: checkpointing gives you "we will resume," not "we will resume correctly." Correctness on resume is a property you design into each side-effecting step, and the checkpoint store is the durable scratchpad that makes it achievable. Frameworks that market "exactly-once" are really giving you durable checkpoints plus an idempotency convention; understand both halves. This is part of the larger discipline covered in production guardrails for agentic workflows.

The boundary: checkpoints are not memory

It is tempting, once you have a durable store for agent state, to put everything in it. Resist. A checkpoint is short-term and run-scoped: it exists to continue one execution, and once that execution finishes successfully the checkpoints are largely garbage that you TTL away. Long-term memory is the opposite: it is cross-run, semantic, and meant to be queried later ("what did this user tell me about their setup three sessions ago"). Conflating the two produces checkpoints that are bloated, slow to write, and never cleaned up, and a memory layer that is accidentally tied to the lifecycle of individual runs.

The clean split:

| Concern | Checkpoint | Memory / Store | |---|---|---| | Lifetime | One run, then expire | Persists across runs | | Keyed by | Thread / run id + step | Namespace + semantic key or vector | | Access pattern | Read latest, write often | Read by relevance, write occasionally | | Contents | Step, state, cursors, pending calls | Facts, embeddings, large artifacts (by reference) | | What kills it | TTL after success | Explicit forgetting / consolidation |

Both can live in the same durable KV store, in different keyspaces, and there is real operational appeal to that: one backend, one wire protocol, one thing to back up. But they are different access patterns with different retention rules, and you should model them as such rather than smearing one into the other. The memory side, and the framework-level distinction between a checkpointer and a long-term store, is exactly what memnode digs into.

Common questions

Can I really use a plain key-value store as a LangGraph checkpointer backend? Yes. The checkpointer interface is "save this serialized checkpoint for this thread and config, then load the latest for this thread." That maps directly onto KV writes and reads under a thread:{id}:... keyspace. The only hard requirements the store must meet are durability (survives restart), no eviction (the key is still there when you resume), and atomic writes (no half-written snapshot). Because BaseKV speaks the Redis API, any Redis-based checkpointer points at it without code changes, while getting on-disk persistence and no maxmemory eviction by default.

How often should I checkpoint - every step, or at milestones? Per-step (per-node) checkpointing is the safe default: you can resume from almost anywhere and you redo at most one step's work. The cost is one durable write per step, which for most agents is negligible next to a model call. Checkpoint less frequently only if your writes are genuinely expensive or your steps are cheap and numerous, and accept that you will redo more work on resume. Whatever the grain, every side-effecting step still needs its own idempotency guard, because checkpoint frequency reduces redo but never eliminates the possibility of a step running twice.

What happens if the agent crashes while writing the checkpoint itself? This is why atomic writes are non-negotiable. With an atomic write, a crash mid-checkpoint leaves the previous, intact checkpoint as the latest, and resume continues from there - you lose the in-progress step but never load corrupt state. Without atomicity, you can resume into a snapshot that is half old and half new, which is harder to detect and worse than losing the step. Pair this with recording side-effect intent and idempotency keys inside the checkpoint, so that even when a step is redone after a crash, the downstream effect is claimed exactly once.

---

Related: The Agent Memory Pattern on a KV Store, Shared State for Multi-Agent Workflows, Idempotency Keys with a Key-Value Store, Distributed Locks with a KV Store, Production Guardrails for Agentic Workflows.