Over the past 18 months, we've built and deployed AI agents for six enterprise clients across healthcare, financial services and logistics. Some of these agents are handling tens of thousands of tasks per week with minimal human oversight. Others required significant redesign after failing in production. Here's what we learned.
Agents Are Not Chatbots
The first thing to understand is that an AI agent is fundamentally different from a chatbot. A chatbot responds to questions. An agent takes actions: it queries databases, updates records, sends emails, triggers workflows. This makes agents much more powerful and much more dangerous. When a chatbot gives a wrong answer, a human reads it and decides whether to trust it. When an agent takes a wrong action, it can delete data, send incorrect payments or lock user accounts.
Every agent we've deployed in production has a strict scope of actions it can take, enforced at the code level, not just in the prompt. The LLM can suggest actions, but the system validates each action against a permit list before executing it. This has caught numerous hallucinated actions during development that would have caused real damage in production.
The Reliability Challenge Is Real
LLMs are not reliable in the way that traditional software is reliable. A REST API returns the same response for the same input every time. An LLM might return different responses for the same input five times out of five. This variability is acceptable for content generation, but it's a real problem for autonomous agents that need to make consistent decisions.
We've learned to design agent workflows with human-in-the-loop checkpoints at every decision point that has financial, legal or safety implications. The agent gathers information, presents options and makes recommendations, but a human approves or rejects each action. As the agent demonstrates reliability over time, we gradually increase autonomy for specific, well-understood tasks.
Tool Design Matters More Than Prompt Engineering
Early on, we spent most of our time optimizing prompts. We've since learned that tool design, the quality and structure of the APIs and functions the agent can call, has a much bigger impact on agent performance. A well-designed tool with clear parameters, good documentation and consistent return values makes the agent's job easier. A poorly designed tool, no matter how good the prompt, produces unreliable behavior.
We now follow a simple rule: if a human can use the tool with a one-paragraph description and succeed consistently, the agent can probably use it too. If the tool requires pages of documentation to use correctly, redesign the tool before building the agent.
Where Agents Actually Deliver Value Today
After six production deployments, the use cases where agents consistently deliver value are: data entry and reconciliation (agents can pull data from emails, PDFs and web forms and populate databases with high accuracy), triage and routing (agents can understand incoming requests and route them to the right team or system), monitoring and alerting (agents can correlate signals across multiple systems and determine whether an alert requires human attention), and guided workflows (agents can walk users through complex processes, validating inputs at each step).
The use cases where agents still struggle are: open-ended problem solving that requires multi-step reasoning across unfamiliar domains, situations that require subjective judgment or cultural context, and tasks where the cost of a mistake is very high and the agent cannot explain its reasoning clearly enough for a human reviewer to evaluate.
What We'd Tell Teams Starting Today
Start with a narrow, well-defined task. Give the agent access to one or two tools that you know well. Put a human in the loop from day one. Measure not just accuracy but also reliability: does the agent consistently succeed on the same task? Add autonomy slowly, based on demonstrated performance, not on enthusiasm for the technology.
And most importantly: design your system so that when the agent fails, and it will fail, the failure is contained, detectable and recoverable. Idempotent operations, rollback capabilities and clear audit trails are not optional extras. They're the foundation of any production agent system.