anukamithara.com

The Hard Part of an AI Shopping Agent Wasn't the LLM

What Kapuru taught me about memory, tool calls, streaming, and checkout safety

By

When I started building Kapuru for the Kapruka Agent Challenge, I expected the main challenge to be the model.

Could it understand what the user wanted? Could it search products well? Could it respond naturally in multiple languages?

Those questions mattered, but they were not the hardest part.

The hardest part was making the agent behave like a reliable application instead of a clever demo.

A shopping agent carries more state than a normal chatbot

A regular chat can often survive with a list of messages.

A shopping agent quickly accumulates much more:

conversation history
user preferences
retrieved memories
recipient context
current cart
previous carts
order state
product results
UI artifacts
voice session state
tool outputs

At first, adding more context feels like an obvious improvement.

More context should mean a smarter model, right?

In practice, it can make the system slower, more expensive, and less predictable.

The model can receive multiple versions of the same fact. A stale cart summary can conflict with the live cart. A memory extraction from an earlier turn can complete late and affect a later response. Large tool responses can consume a surprising amount of the model context window.

The problem stopped being "How do I give the model more information?"

It became:

What is the minimum correct information this turn actually needs?

Durable memory and conversation history should not be the same thing

One thing I had to clarify early was the difference between chat history and memory.

If a user says:

I usually buy gifts for my mother

that may be useful beyond the current conversation.

But the exact wording of every previous message does not need to stay forever in the active prompt.

I moved toward a model where context can be separated into layers:

Recent conversation
- exact recent turns
 
Durable memory
- selected user preferences and facts
 
Current operational state
- cart, active checkout, current product results
 
Retrieved context
- only what is relevant to this turn

This makes it easier to trim history without losing useful personalization.

It also helps prevent old UI state from competing with current operational state.

Asynchronous memory creates consistency problems

Memory extraction is often moved off the critical response path because waiting for it makes the chat slower.

That creates a new problem.

Suppose turn 10 triggers memory extraction. The user sends turn 11 before that extraction finishes. Then the new memory arrives before turn 12.

The agent's behavior can change between turns even though the user did not provide new information in that moment.

That is not necessarily wrong, but it needs to be understood.

For me, the lesson was that async memory is eventually consistent state.

Once I started thinking about it that way, the design questions became clearer:

  • when does a new memory become visible?
  • can a later turn depend on a memory job that is still running?
  • what happens if memory extraction fails?
  • can a newly extracted memory contradict current session state?

These are application-state questions, not prompt-engineering questions.

Tool calls need time budgets, not only iteration limits

Kapuru could call tools for product search, cart operations, and other actions.

A common safety mechanism for agents is an iteration limit:

maximum 4 tool steps

That is useful, but it does not protect you from one tool call hanging for 40 seconds.

I ran into cases where streaming or upstream reads could remain open too long.

The more useful control is a wall-clock budget.

Conceptually:

Turn starts
    ↓
model reasoning
    ↓
tool call
    ↓
tool result
    ↓
next model step
 
Everything must finish before the turn deadline.

Each network boundary should also have its own timeout.

That means an agent can fail in a controlled way instead of leaving the UI in a permanent "thinking" state.

Tool responses should be designed for models

APIs are normally designed for applications.

An agent changes the consumer.

If a product-search tool returns a massive raw payload, the model has to pay for and reason over all of it.

So I started treating tool output as its own interface design problem.

The useful output is usually not "everything the backend knows".

It is closer to:

{
  "items": [
    {
      "id": "...",
      "name": "...",
      "price": 4200,
      "availability": "in_stock"
    }
  ],
  "has_more": true
}

Pagination, summaries, field selection, and bounded result counts become part of agent reliability.

This is especially important when a ReAct-style flow can call several tools in one turn. A few oversized responses can fill the context window before the model reaches the useful part of the task.

Checkout changed the safety level of the system

Searching for a product is reversible.

Creating an order is not the same kind of action.

That changed how I designed the flow.

The agent should never interpret a vague conversational message as permission to perform a consequential action.

I used explicit confirmation before checkout and kept the final payment flow outside the agent itself. Kapuru prepared the Kapruka click-to-pay flow rather than pretending the model should directly process payment credentials.

The backend also had to protect the action independently of the model.

Important controls included:

  • ownership-scoped Supabase RLS for guest sessions
  • explicit checkout confirmation
  • quotas around checkout behavior
  • preview environments where real checkout is disabled
  • rollback procedures
  • smoke tests for the real production flow

This is an important boundary for me now:

The model can suggest an action. The application still owns authorization and execution safety.

Prompt instructions are not an access-control system.

Voice made the control plane more visible

Kapuru also had live voice interaction.

Voice makes latency much more obvious than text.

A user can tolerate a few seconds of text generation more easily than a silent voice session where nothing seems to happen.

Tool calls from the live session therefore needed predictable serialization and recoverable failures.

If two tool operations race, or a tool result never comes back, the user experiences it as a broken conversation rather than a backend edge case.

That pushed me toward treating the agent as a control plane with strict lifecycle rules instead of a free-form loop around an LLM.

The reliability stack mattered more than model cleverness

By the end, the pieces I cared about most looked less like prompt engineering and more like backend engineering:

bounded context
selective memory
clear operational state
small tool payloads
timeouts
per-turn wall-clock budget
recoverable tool failures
explicit confirmation
backend authorization
rollback
observability

A stronger model can improve product selection or natural language quality.

It cannot compensate for a duplicated checkout, stale cart state, hanging stream, or a tool response that consumes the entire context window.

What I learned

Kapuru changed how I think about agent applications.

The LLM is important, but it is only one component in a stateful distributed workflow.

The engineering work around it decides whether the system feels reliable:

  • separate durable memory from recent chat history
  • retrieve context selectively instead of sending everything
  • treat async memory as eventually consistent state
  • impose network and wall-clock timeouts
  • design compact tool outputs
  • keep consequential actions behind explicit confirmation
  • enforce authorization in the backend, not in prompts
  • make failures recoverable

The most useful shift for me was this:

Do not ask only "Can the model do this?" Ask "Can the whole system do this safely, repeatedly, and under failure?"

That question produces a very different architecture.

+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

© 2026 Anuka Mithara. All rights reserved.

GitHubLinkedInanukamithara.com
Close Menu