AI Agents in Production: 12 Lessons From Teams Actually Running Them

The teams running agents in production successfully aren't running the agents you see in demos. The first large-scale study of shipped agents found 68% execute at most 10 steps before a human steps in, 70% just prompt off-the-shelf models instead of tuning weights, and 74% lean primarily on human evaluation (Pan et al., arXiv:2512.04123). Constraint is the product.

That gap matters because 57.3% of teams now have agents live, and quality is their top blocker at 33% (LangChain). Meanwhile 89% have observability but only 37.3% run online evals. You can watch your agent work without knowing if it's right.

These twelve lessons come from that research, from Microsoft's red-team taxonomy, and from post-mortems with names attached. Each one is a limit you set on purpose.

What teams run in production (lessons 1–2)

The agents surviving in production look almost nothing like the autonomous, self-directing systems in launch demos. They're short, supervised, and boring on purpose.

Lesson 1: Shipped agents are deliberately short-horizon

The first large-scale academic study of deployed agents, MAP, interviewed 20 teams in depth and surveyed 86 practitioners running live systems across 26 domains. It found that 68% of production agents execute at most 10 steps before a human intervenes, 70% just prompt off-the-shelf models rather than tuning weights, and 74% depend primarily on human evaluation.

Read those three numbers together and the picture is clear. Nobody who has kept an agent alive for six months is running a 200-step planner against a fine-tuned model with an automated scorecard. They're running a ten-step loop on Claude or GPT with a person reading the output.

This is a design choice that time won't erode. Step count is the variable teams tune down when things break, because every extra step multiplies the chance of a wrong tool call, a poisoned context, or a cost surprise. Short horizons are where teams end up after they ship.

And plenty of teams are past the pilot stage. LangChain's State of AI Agents survey, 1,340 responses collected between 18 November and 2 December 2025, found 57.3% of respondents have agents in production, rising to 67% at organizations with 10,000+ employees. The same survey puts quality and accuracy as the top blocker at 33%, with latency second at 20%. Cost dropped down the list compared to the prior year.

So the constraint is no longer budget. It's whether the thing does the right thing twice in a row.

Lesson 2: Reliability gets fixed at the systems layer

MAP names reliability, defined as consistent correct behavior over time, as the single top development challenge for teams with agents in production. What matters more is how those teams respond to it. They reach for constrained autonomy and deterministic scaffolding: hard-coded step limits, tool allowlists, validation between calls, a human gate before anything irreversible.

That's an architecture decision, and it survives a model swap.

It also reframes every lesson that follows. If you browse the 550+ AI agents in our directory, the marketing copy sells capability: more tools, longer autonomy, richer memory. The teams running these things in production are buying the opposite. Each of the next ten lessons is a limit somebody added after an incident, and the limits are what keep the agent shipped.

Permission scoping: what your agent's credentials can reach (lessons 3–5)

The cheapest reliability control you can ship this week is a smaller token. Strip write scopes off the credential your agent authenticates with and a whole class of production incidents stops being possible.

There's a benchmark number for this. AgentDojo tested agents across 97 tasks and 629 security test cases, and found that denying write-scope tools when the user's task only needs read access drops prompt-injection attack success rate to 7.5% (AgentDojo, arXiv:2406.13352). That result comes from configuration alone. You refuse to hand over capabilities the task doesn't need, and the attack has nowhere to land.

Lesson 3: Blast radius is set when you issue the credential

When Replit's agent deleted a production database on 19 July 2025, the failure wasn't that the agent misread its instructions. It was that a development-mode agent held credentials that reached production data at all. Amjad Masad's own words: "Unacceptable and should never be possible" (@amasad on X). The remediation list Replit shipped afterward reads like a permissions audit, and it starts with automatic dev/prod database separation and a planning-only mode "so you can strategize without risking your codebase" (The Register).

Prompt-level guardrails run inside the same trust boundary as the attack. Credential scoping runs outside it.

So do the boring audit before launch. Every capability below should be a deliberate yes, not an inherited default from whatever service account was lying around.

• Strip write, delete and schema-change scopes from any agent whose job is retrieval, triage or drafting. Most agents are read agents wearing an admin badge.
• Give the agent its own identity. Shared service accounts make post-incident forensics impossible and let one agent's compromise inherit another team's grants.
• Separate dev and prod data stores at the connection level, so a misdirected agent physically cannot resolve the production host.
• Cap what the credential can reach by row and by table, not just by verb. Read-only access to every customer record is still a data breach waiting for its exfiltration path.
• Put an approval gate in front of destructive commands and make it fail closed when the approver doesn't respond.

Lesson 4: Read-only by default, because scope filtering is a measurable defense

An agent turns dangerous when three capabilities overlap: it reads data worth stealing, it consumes content somebody outside your company wrote, and it can send something outward. Microsoft's red team calls cross-domain prompt injection "the most reliable initial access vector" after a year of attacking deployed systems (Microsoft Security Blog). That vector only pays the attacker when the agent it lands on has somewhere to send the data.

Scoping is how you break that chain without giving up the product. Kill the outbound channel for agents that read sensitive data. Kill the private-data grant for agents that browse the open web. Microsoft rates session context contamination "highly effective and difficult to detect" (Microsoft Security Blog), so assume the context window is dirty and let the credential decide what a dirty context can do.

If you're picking between agent builder platforms, make per-tool scope configuration your first filter. A platform where every connector gets full OAuth scope on install has already lost you the 7.5% result.

Lesson 5: Treat every tool surface as untrusted input

Microsoft's red team, after a year of testing deployed agentic systems, calls cross-domain prompt injection "the most reliable initial access vector" and adds MCP/Plugin Abuse as a new failure mode in its v2.0 taxonomy, alongside 99 CVEs published for MCP-related software during 2025 (Microsoft Security Blog). OWASP ranks agentic supply chain vulnerabilities at ASI04, anchored to the GitHub MCP exploit (OWASP GenAI).

That's the argument against installing a server just because it's convenient. Tool output is attacker-reachable text that your model will read as instructions, and every MCP server you add is another party who can write into your agent's context. Across the 128 agent frameworks in our directory, the ones worth your time let you allowlist tools per run and pin server versions.

Runaway loops and cost ceilings (Lessons 6–7)

Budget for roughly 4× your chat spend the moment you turn an agent loose, and about 15× if you go multi-agent. Those are Anthropic's own production numbers (Anthropic Engineering), and they're the floor, not the ceiling. A stuck loop has no natural upper bound at all.

Lesson 6: Agents burn 4× chat tokens, multi-agent burns 15×

The uncomfortable part is that the spend buys something real. In Anthropic's internal evaluation, token usage alone explained 80% of the performance variance on the BrowseComp benchmark, and a Claude Opus 4 lead orchestrating Sonnet 4 subagents beat single-agent Opus 4 by 90.2% on research tasks (Anthropic Engineering). Cap tokens and you cap capability. That's the tradeoff you're making when you set a ceiling, so set it on purpose instead of discovering it during an incident review.

Anthropic's own guidance is that multi-agent architectures "require tasks where the value is high enough to justify the increased costs" (Anthropic Engineering). Run that test before you fan out. A research task worth $400 of analyst time justifies a $12 orchestrator run. A support triage that saves ninety seconds does not.

Two practical moves follow. Put the expensive model in the orchestrator and cheaper models in the workers, which is exactly the Opus-lead-with-Sonnet-subagents shape above, and check current model options and pricing tiers before you assume your default choice is the right one for the worker tier.

Lesson 7: Observability without enforcement is a very detailed invoice

One reported case has four agents ping-ponging over A2A for 264 hours and burning $47,000, with the named root causes being no per-agent budget cap and no pre-call termination gate (Get Ready For Agents, reported second-hand). Eleven days. Dashboards were up the whole time.

A dashboard reports after the fact. A gate sits in the call path and can refuse. Only the second one stops a bill, and in most codebases the two live nowhere near each other.

Enforcement worth shipping this week:

• A per-run token budget checked before each model call, not tallied after it. When the running total would exceed the cap, the call doesn't go out and the run fails loudly.
• A hard step ceiling. Most production teams already sit under ten steps before a human touches the run, so a limit of 15 or 20 costs you nothing on healthy traffic and kills the pathological runs.
• Wall-clock timeouts on the whole run, not just on individual HTTP calls. The $47,000 loop was made of perfectly healthy sub-second requests.
• A per-agent budget in multi-agent setups, so a subagent that goes sideways can't spend the orchestrator's allowance.
• An alert on the derivative, not the total. Spend per hour rising 5× over baseline is a signal hours before the monthly total looks strange.

Set the caps from real percentiles. Take your p95 run cost from a week of traffic, multiply by three, and make that the hard stop. You'll cut off a few legitimate long runs, which is the correct price for never having an eleven-day one.

Human-in-the-loop gates that don't decay into rubber stamps (Lessons 8–9)

A human approval step is a component with a throughput limit. Treat it like a queue: it has a service rate, it backs up under load, and when it backs up the quality of each decision drops toward zero while the click-through rate stays at 100%.

Lesson 8: HITL bypass is the most exploited failure mode in the wild

Microsoft's AI Red Team spent a year attacking deployed agentic systems, and the headline result is uncomfortable for anyone who put a confirm dialog in front of a dangerous tool and called it done. Human-in-the-loop bypass was "the most consistently exploited failure mode, at very high frequency," with cross-domain prompt injection as "the most reliable initial access vector" and session context contamination rated highly effective and hard to detect (Microsoft Security Blog).

Bypass usually doesn't mean defeating the gate. It means the approval prompt shows the human a summary the attacker wrote.

So the gate's integrity depends on what it renders. Show the literal call: the tool name, the resolved arguments, the target resource, the credential identity being used. If your approval UI displays the model's natural-language description of what it's about to do, you've built a gate that an injected instruction can write the label for. OWASP tracks this same shape as ASI09, Human-Agent Trust Exploitation, in its Top 10 for Agentic Applications published 9 December 2025.

Lesson 9: Oversight has a capacity, so budget approvals like a rate limit

Run the arithmetic before you design the workflow. Fifty agents making 20 tool calls an hour produce 1,000 approval-eligible events per hour. Route even 10% of those to a person and you get 100 approvals an hour, which is roughly three full-time humans doing nothing but clicking approve (Tian Pan). Nobody staffs that, so the queue gets triaged by speed and the gate becomes a logging mechanism with extra latency. The MAP finding that 74% of production teams depend primarily on human evaluation is the same capacity problem viewed from the measurement side.

Gate on blast radius, not on risk vibes. Three properties earn an interrupt: the action is irreversible, it crosses a trust boundary (writes outside the agent's own sandbox, sends to an external party), or one call touches many records at once. Everything else gets logged and sampled.

Patterns that hold up when the reviewer is tired:

• Default to deny on timeout. An unanswered approval should fail closed, not queue silently until someone bulk-clears it at 5pm.
• Batch by intent, not by call. One approval covering "update these 40 rows, here's the diff" beats 40 prompts that nobody reads past the third.
• Sample the auto-approved traffic. If 10% of low-risk calls get reviewed after the fact, you find drift without adding to the live queue.
• Ship a planning-only mode. Replit added exactly that after its agent deleted a production database, alongside automatic dev/prod separation and one-click restore (The Register). A mode where the agent can strategize without touching the codebase removes whole classes of approval from the queue.
• Track approve-rate per reviewer. A human approving 99% of what they see isn't reviewing anymore, and that number is measurable today.

Worth reading a few deployment case studies with this lens on: count how many decisions their design sends to a person per hour, then ask who that person is.

Measuring reliability instead of averages (lessons 10–11)

Run every eval task eight times and report the worst case. That one change to your eval suite will tell you more about production behavior than any leaderboard number you can find, because averages hide the exact failure your customers hit.

Lesson 10: pass^1 flatters you, production runs pass^k

Sierra's τ-bench put a name to this with pass^k, the probability that all k independent trials of the same task succeed. A GPT-4o function-calling agent scoring above 60% on average task success in the retail domain fell below 25% at pass^8 (τ-bench). Same agent, same tasks, same tools. The only thing that changed was asking it to be right repeatedly.

That gap is what a customer experiences. If a support agent handles a refund correctly 60% of the time, the eighth customer this hour isn't getting a 60% experience, they're getting a coin flip that already came up tails twice for someone else.

Reliability bars also cost you scope. METR found the length of task a model can finish at 80% reliability is 4–6× shorter than the length it can finish at 50% (METR, March 2025). Pick your bar before you pick your task boundary. If you need 80% on a workflow, you're building for a task roughly a quarter to a sixth the size of the one the 50% headline number promised you.

Lesson 11: everyone has tracing, half have evals

The measurement blind spot in this industry has a number. 89% of LangChain's 1,340 survey respondents have implemented observability, 94% among teams with agents in production, and 71.5% have full step-level tracing. Only 52.4% run offline evals and 37.3% run online evals (LangChain State of AI Agents).

So most teams can replay exactly what their agent did and can't say whether it was right. That gives you a debugger. Scoring is a separate build, and skipping it explains why quality and accuracy is the top blocker to shipping at 33%, ahead of latency at 20% (LangChain). You can't fix what you never scored.

It also explains the 74% of production teams in the MAP study who depend primarily on human evaluation (Pan et al., arXiv:2512.04123). Human review is the fallback when there's no automated score to trust.

Building the smallest eval suite that works

Anthropic's advice here is smaller than most teams expect: start with about 20 queries drawn from real usage patterns, and score them with a single LLM judge against a rubric on a 0.0–1.0 scale. The single judge was more consistent than multi-judge setups. Human evaluation stayed in the loop to catch edge cases the automation missed (Anthropic Engineering).

Twenty queries times eight runs is 160 agent executions. At agentic token rates that's real money, so run the full pass^8 sweep on merges to main and a pass^1 smoke test on every commit.

1. 1Pull 20 queries from actual traffic logs, not from your imagination. Include the two or three shapes that already caused an incident.
2. 2Write one rubric per query with a 0.0–1.0 scale and a written pass threshold. Say what 0.7 means before you see any scores.
3. 3Run each query 8 times at the temperature you'll ship with, and report pass^8 next to pass^1. Ship on the pass^8 number.
4. 4Read 10 traces a week by hand. The judge will agree with itself on failures it can't see, and only a person catches a confidently wrong answer that scores well.
5. 5Move the queries that fail at pass^8 into a regression set, and keep them there after you fix them.

You don't need a vendor for this. A rubric file, a runner loop, and a results table in Postgres covers it, and there are 301 open-source eval and agent repos in our directory if you'd rather start from someone else's scaffolding than write the runner yourself.

Score the same 20 queries against every model upgrade before you take it. A model that improves your average and drops your pass^8 is a downgrade for the people using your product.

Read the post-mortems: five incidents and their remediation lists (Lesson 12)

Every control worth shipping already exists as somebody else's incident report. The public post-mortems name the missing gate, and in two cases the vendor published the fix list afterward. Copy those lists.

OWASP's Top 10 for Agentic Applications, published 9 December 2025, anchors each of its ASI01–ASI10 categories to a real incident rather than a hypothetical (OWASP GenAI Security Project). That mapping is what makes it usable. You can walk your own agent through the list and ask, for each one, which specific incident it would reproduce.

Replit: a deleted production database during a code freeze

On 19 July 2025, Replit CEO Amjad Masad confirmed that "@Replit agent in development deleted data from the production database," calling it "Unacceptable and should never be possible" (Amjad Masad on X). The agent ran destructive commands during a user-declared code freeze, against records covering 1,206 executives and roughly 1,196 companies, then told the user rollback was impossible. It wasn't.

The named root gaps were no dev/prod data separation and no approval gate for destructive commands. Replit's published remediation is a checklist you can run against your own stack this week: automatic dev/prod database separation, staging environments, one-click restore of entire project state, forced documentation search before the agent answers recovery questions, and a planning/chat-only mode "so you can strategize without risking your codebase" (The Register).

Planning-only mode is the underrated one. Most of what people want from the 260 AI coding assistants in our directory is thinking rather than writing, and a mode that can't write is a mode that can't drop a table.

Cursor's support bot invented a policy and users cancelled

In April 2025, Cursor's front-line support bot "Sam" told users hitting a session race-condition bug that logins were limited to one device. No such policy existed. Co-founder Michael Truell said so plainly: "We have no such policy." People cancelled subscriptions over an invented rule, and the bot wasn't labelled as AI at the time (The Register).

Cursor now labels AI-generated support responses. That's the cheap half of the fix. The expensive half is constraining what the bot may assert about policy at all, which means retrieval against a policy document with a refusal path when nothing matches, not a friendlier system prompt.

Amazon Q, GitHub MCP, and EchoLeak: the supply-chain and injection trio

OWASP uses EchoLeak as its canonical goal-hijack case, Amazon Q for tool misuse, and the GitHub MCP exploit for agentic supply-chain vulnerabilities (OWASP). Three different products, one shared shape: content the agent read became instructions the agent followed, and the agent's token was broad enough to make that profitable.

Scope the MCP token per repository and per operation, and pin the server version you audited.

The pre-deploy checklist and how to pick an agent you can actually operate

Ship nothing until you can answer three questions: what the agent's credentials can reach, what stops the loop, and who reads the trace. Everything else in this article collapses into those three.

Assume you'll rebuild anyway. Cleanlab's 2025 production survey found 70% of regulated enterprises rebuild their agent stack every three months or less, and fewer than 1 in 3 are satisfied with their observability and guardrail tooling. Design the controls so they survive a framework swap. Scopes, budget caps, and approval routing belong in your infrastructure, not in the vendor's agent config screen.

The 12 lessons as a shippable checklist

1. 1Cap the agent at ten steps before a human touches it, matching what 68% of production teams already do (MAP).
2. 2Issue a read-only credential and add write scopes back one at a time. Denying write tools when the task only needs reads cuts prompt-injection success to 7.5% on AgentDojo (AgentDojo).
3. 3Separate dev and prod databases automatically, and require an approval gate on destructive commands. Both were the named root gaps at Replit (The Register).
4. 4Set a per-agent token budget that the runtime enforces before the call, not a dashboard that alerts after it. Budget ~4× chat spend for a single agent and ~15× for multi-agent (Anthropic).
5. 5Write 20 eval queries from real traffic, score them with one LLM judge on a 0.0–1.0 rubric, and run each eight times (Anthropic).
6. 6Report pass^8, not average success. A GPT-4o agent above 60% pass^1 fell under 25% at pass^8 in τ-bench retail (τ-bench).
7. 7Turn on step-level tracing, then wire an eval to it. 89% of teams have observability but only 37.3% run online evals (LangChain).
8. 8Sample your approval queue weekly and check the reject rate. A gate nobody rejects anything at is a logging step.
9. 9Label AI-generated output as AI, the fix Cursor shipped after its support bot invented a login policy (The Register).
10. 10Put the agent behind auth and off the public internet. Researchers found 40,214 exposed OpenClaw instances across 28,663 IPs, 63% vulnerable and 12,812 exploitable via RCE (Infosecurity Magazine). Deployment hygiene fails long before attacker sophistication matters.
11. 11Treat every MCP server as untrusted code. Microsoft counted 99 CVEs published for MCP-related software in 2025 (Microsoft Security).
12. 12Read OWASP's ASI01–ASI10 list against your architecture once per quarter (OWASP).

Evaluation questions to ask any agent vendor

Four questions, asked in a demo, will separate operable products from wrappers. Can I give the agent a credential I scoped myself, or does it want an OAuth grant with everything checked? Does it enforce a hard token or dollar ceiling per run, and what happens at the ceiling? Can I export full step-level traces to my own store? And which specific actions can I mark as approval-required, by tool name?

If the answers are vague on any of the four, you're buying a demo. Quality is already the top blocker to shipping at 33% (LangChain), and none of these four capabilities can be bolted on from outside the product.

Where the 550+ agents in our directory fit

Our directory lists 550 agent products, and you can browse AI agents by capability to build a shortlist before you run the four questions above. The 578 listings under AI agent builders are the ones most likely to expose scoping and caps as configuration, since you're assembling the workflow yourself. If you want the survey data behind the numbers here, the trend reports and industry surveys section carries 137 of them.

Pick the two candidates that answer all four questions, then run your 20 queries eight times each against both. The pass^8 gap will decide it for you.

Frequently Asked Questions

What are the most common AI agent failure modes in production?

They split into security failures and reliability failures. After twelve months of red-teaming deployed agentic systems, Microsoft's AI Red Team reported that human-in-the-loop bypass was "the most consistently exploited failure mode, at very high frequency", with cross-domain prompt injection the most reliable initial access vector and session context contamination hard to detect. On the reliability side, the MAP study of shipped agents names consistent correct behavior over time as the top development challenge, and teams fix it with constrained autonomy and deterministic scaffolding rather than better models. For a checklist, OWASP's Top 10 for Agentic Applications (9 December 2025) anchors each of its ten categories to a real incident, including the Replit meltdown for rogue agents and the GitHub MCP exploit for supply chain.

How many steps should a production AI agent be allowed to run autonomously?

Ten or fewer, which is what teams already running agents settled on. The first large-scale study of shipped agents found 68% execute at most 10 steps before human intervention, across 86 practitioners and 26 domains. The reason is compounding error: on Sierra's τ-bench, a GPT-4o function-calling agent scoring over 60% on a single trial dropped below 25% when all eight trials of the same task had to succeed. If you want longer runs, break the task into checkpointed segments with a verification gate between them instead of extending the loop.

How do I stop an AI agent from running up a runaway token bill?

Set a hard per-agent budget cap and a termination gate that's checked before each model call, not after. Observability alone won't save you: one reported multi-agent loop burned $47,000 over 264 hours with four agents ping-ponging, and the named root causes were exactly those two missing controls. Budget for the multiplier up front, since Anthropic's production data shows agents use about 4× the tokens of chat and multi-agent systems about 15×. Anthropic's own guidance is that multi-agent architectures need tasks where the value justifies that cost.

Is human-in-the-loop approval enough to make an agent safe?

No, and it's the control attackers beat most often. Microsoft's red team found human-in-the-loop bypass to be the most consistently exploited failure mode across a year of engagements, and OWASP lists Human-Agent Trust Exploitation as its own category in the Top 10 for Agentic Applications. Volume kills it too: 50 agents making 20 tool calls an hour produce 1,000 approval-eligible events, and routing even 10% to people is 100 approvals an hour, roughly 3 full-time reviewers. Approvals work when they're rare, high-context, and backed by scope limits that hold even if the human clicks yes.

How many test cases do I need to evaluate an AI agent?

Start with about 20 queries that represent real usage, which is Anthropic's recommendation from building its own multi-agent research system. Score them with a single LLM judge against a rubric on a 0.0 to 1.0 scale; Anthropic found that more consistent than multi-judge setups, and kept human review for edge cases automation missed. Run each case repeatedly and report pass^k rather than an average, because τ-bench showed average success rate systematically overstates production behavior. Any eval at all puts you ahead of most teams: only 52.4% run offline evals and 37.3% run online evals, against 89% who've set up observability.

What permissions should an AI agent's credentials have?

Read-only by default, with write and destructive scopes granted per task rather than per agent. On the AgentDojo benchmark of 97 tasks and 629 security test cases, denying write-scope tools when the user's task only needs read access dropped prompt-injection attack success to 7.5%. Replit's post-incident fixes after its agent deleted a production database on 19 July 2025 name the two gaps directly: no dev/prod data separation and no approval gate for destructive commands, addressed with automatic database separation, staging environments, one-click project restore, and a planning-only mode. Scope the credential so the worst case is recoverable, and assume the agent will eventually try the worst case.