A few lessons from building agentic support systems

A few lessons from building agentic support systems:

• Ship early. Don't over-engineer. You won't get it perfectly right from the get-go. What matters is getting real data flowing in, and iterating quickly based on that.
• Review and QA ruthlessly. A tight human-in-the-loop cycle is the biggest unlock.
• Docs =/=truth. They drift. Use them as a foundation, but invest in tools that keep them updated as reality changes.
• Real interactions are gold. Mine as many as possible. They carry practical info other customers need too. Make sure you index carefully and strip PII so you don’t spill someone’s order details to a stranger.
• Chunking + retrieval strategy matters a lot. Even with large context windows, performance is dramatically better when context is precise and concise. Hybrid retrieval (semantic + BM25) feels like table stakes now.
• Invest in data flywheels: every piece of feedback (internal or customer) should improve something (KB content, prompt instructions, SOPs). Use automatic prompt optimization techniques (such as agentic context engineering) to scale these tasks.
• Get the team focused on “agent ops.” Move from answering tickets to managing agents. Prioritize feedback with the highest information value.
• Many complex interactions (refunds, eligibility checks, etc.) don't require deterministic workflow approaches. These can often be replaced by natural-language "playbooks" that reason and act. They are way simpler to set up and maintain, and work great, especially with the newer models like GPT-5 and Grok 4 Fast.

Zoom out and take an iterative approach, and look for the next big chunk of tickets you can automate. That tells you what to improve next: knowledge base, prompts, retrieval, or a new API integration.

Deep dives into ticket data always reveal the next leverage point.

Fewer tickets and more sanity.