Why “just add more context” is the wrong fix - and the 9-stage pipeline that actually works.
Here’s a demo that should make you nervous about the agent you shipped last sprint.
Same tools. Same model. Same task. Call it once: “Please cancel my order #9999.”
It comes back - cancelled, confirmed.
Call it again, thirty seconds later, and ask what happened to that same order. It has no idea. Nothing was kept.
No memory, no persistence - just whatever fit inside that one call’s messages.
That’s not a prompt problem. You could rewrite the wording a hundred ways and it wouldn’t matter, because the fact the model needed wasn’t in its context window when it answered. This is the gap context engineering exists to close.
Prompt Engineering vs. Context Engineering
The difference is one sentence: prompt engineering is wording. Context engineering is what information reaches the model, in what order, and for how long.
You can write the perfect instruction - if the right fact isn’t sitting in the window when the model answers, the wording doesn’t matter.
Why More Context Isn’t the Fix
The obvious instinct is to give the model a bigger window. It doesn’t work. Past a point, more context doesn’t make a model smarter it makes it slower to find the one fact that matters, and more likely to act on something stale or contradictory.
Four specific failure modes show up once you know to look for them in a trace:
Context poisoning: a wrong fact enters early, and the agent keeps reasoning from it
Context distraction: accumulated history buries what actually matters right now
Context confusion: irrelevant information competes for attention and the model picks wrong
Context clash: two contradictory facts in the window and the agent gets stuck
Every one of these is a management problem, not a model problem. The job isn’t to maximize what’s in context it’s to curate it, every turn, for the life of the task.
The Pipeline
Here’s the actual architecture, and this is the important part it reruns before every single turn, not once at the start of the conversation.
Query augmentation → retrieval → working memory → long-term memory → compression & budgeting → validation → assembly → the model call → tool output feeding straight back into working memory for the next turn.
One honest note: this build uses a scored keyword retriever over a small policy knowledge base, on purpose - the architecture is the point, not the search algorithm. Swap in a vector DB later and every other stage still works unchanged.
Query Augmentation: Clean the Input First
Real users don’t type clean queries. “can i get a refund on my order it hasnt shipped yet” — no structure, buried intent. Send that straight to retrieval and you get weak matches, and now the model is reasoning from context you polluted at step one.
A small, fast model rewrites the message into a clean query and pulls out anything structured - like an order number - before retrieval ever runs. Cheap model, one job: turn mess into signal.
Retrieval: Chunking Is a Trade-off
Chunk too small and you get a sentence with no surrounding context to make sense of it. Chunk too large and you dilute the match - a query about damaged items pulls back a chunk that’s mostly about something else.
This build chunks on sentence boundaries - sized to stay precise without cutting a fact in half — then scores and ranks each chunk against the query.
Memory Is Two Different Jobs
Working memory is scoped to the task in front of the agent right now: goals, facts, decisions, constraints, open questions. It exists for exactly one task, then it’s gone.
Long-term memory is what survives past that - and it’s not “save everything.” There’s an explicit graduation policy.
Decisions and constraints get promoted, because they’re durable. Open questions carry forward as follow-ups. Raw facts and goals get dropped - they were scratch space for a task that’s now done.
The Loop Learns As It Acts
Context isn’t decided once at the start. Every tool call - checking an order, trying a cancellation — folds its result into working memory before the next turn’s context is even assembled.
And what goes back in isn’t the raw tool response. Payment details, internal IDs, anything the model doesn’t need - stripped before it ever reaches the context window. Extract the signal, not the payload.
The Guardrails on the Pipeline Itself
What stops this pipeline from just recreating the same bloated-context problem it started with? Three things.
Budgeting: every source gets a hard token cap. No source can quietly crowd out the others.
Compression: over the cap, a source gets summarized down to what still matters. Hard truncation is a last resort, not a first move.
Validation: three checks before anything reaches the model: Is it still fresh? Is it permitted for who’s asking (a customer never sees staff-only policy)? Does it contradict something already in the window?
Two real documents from this build’s knowledge base get caught by validation live: a 2025 shipping notice that’s simply expired, and a staff-only escalation policy that’s the wrong role for the asker.
It All Comes Back to One Thing
A context manager runs this entire sequence — augment, retrieve, load memory, budget, compress, validate - fresh, before every single call to the model. Not once per conversation. Every turn.
That’s the concept. Next up: opening the actual code and building it piece by piece, starting with the file that ties all of this together.