How I Would Test an AI Agent Before Letting Customers Use It
Build a small, useful evaluation suite around outcomes, forbidden actions, and realistic failures.
Notes from building
AI engineering,
without the hand-waving.
Practical reads about agents, retrieval, tools, and the things that go wrong between a demo and production.
Follow by RSS ↗The production AI series
Ten worked examples. Realistic constraints. Clear trade-offs.
Your AI Agent Timed Out. Did It Still Send the Refund?
A practical design for agent retries, durable state, and duplicate prevention, using a customer refund workflow.
Your Support Agent Does Not Need the Entire Chat History
How to build a useful context packet for an AI support agent without losing customer commitments or exposing unrelated data.
The RAG Answer Was Correct. The Employee Was Not Allowed to See It.
A worked internal-search design covering permission filters, hybrid retrieval, citations, and stale access.
An Invoice Extractor Should Show Its Working
Design an invoice-processing assistant with source evidence, arithmetic checks, duplicate detection, and review.
Before You Give an AI Agent an MCP Tool, Decide What It Must Never Do
A practical checklist for exposing CRM tools to agents with scoped permissions, previews, and meaningful approval.
The Cheapest Model Can Make Your AI Workflow More Expensive
Compare AI costs per completed task, including retries, review effort, latency, and caching boundaries.
The AI Wrote the Feature. Here Is What I Would Review Before Merging.
Review AI-generated code through authorization, failure handling, dependencies, and meaningful tests.
Do You Need Five AI Agents, or One Clear Workflow?
Use a vendor-research example to decide when multiple agents help and when they add coordination overhead.
When Someone Interrupts Your Voice Agent, Stop More Than the Audio
A practical voice-agent design for barge-in, cancellation, conversation history, and side effects.
Earlier writing
- How I Replaced 50 Hours of Manual Work with a Single AI Agent ↗
- Why Your RAG System Sucks (And How to Fix It) ↗
- System Design: How We Built a Real-Time Analytics Platform at IBM ↗
- Building Scalable AI Applications with RAG and LangChain ↗
- Scaling Tesco's Online Grocery to Millions of Users Across 3 Continents ↗
- Building an AI-Powered Contact Enrichment Platform ↗
- Building a RAG System for 50K+ Regulatory Documents ↗
- Unifying Data from 12 Tools into a Single Dashboard ↗
- Advanced TypeScript Patterns for Enterprise Applications ↗
- Deploying an AI Support Agent That Handles 70% of Queries ↗
- Microservices Architecture with Node.js and Docker ↗
- AI Agent Development with CrewAI and AutoGen ↗
- Event-Driven Architecture with Kafka and Message Queues ↗