The blocker
The AI agent went live three months ago. Someone in a steering meeting asks whether it is working. The answers come from different directions. The project team says it handled thousands of conversations. A supervisor says agents still get angry callers who “tried the robot first”. Finance wants to know how many calls it took off the queue. Nobody can say with confidence whether customers got what they came for.
Without that answer, AI projects stall. Expansion to new use cases waits for proof that never arrives, and the first bad anecdote carries more weight than any number.
Why it happens
- Activity is mistaken for success. Counting conversations says nothing about whether the customer’s goal was met.
- Containment is measured crudely. A customer who hangs up in frustration looks the same as one whose issue was solved.
- Technical and business views are separate. Engineers watch errors and latency, while leaders ask about outcomes. Neither sees the other’s numbers.
- Changes are not compared. A new prompt or tool goes live, and nobody checks whether the previous version was better.
How Amazon Connect solves it
Amazon Connect gives you AI agent metrics at several levels: the session, the AI agent, the prompt and each tool. Use them together, from “is it running?” to “did it help?”.
1. Know how much the AI agent handles, and how often it hands off
Start with volume and handoffs. AI involved contacts counts contacts where AI agents took part. AI handoffs counts self-service contacts that escalated to a human agent, and AI handoff rate is that as a percentage. AI response completion rate shows the share of sessions that successfully responded to the customer’s requests. All are listed in the metric definitions with their calculation logic.
A handoff is not always a failure. In agentic self-service, your prompt tells the AI agent when it must escalate. Read the handoff rate next to the reasons, which a custom Escalate tool can capture.
2. Measure outcomes, not just activity
Several metrics use an automated LLM-based evaluation of the conversation:
- Goal success rate scores each session on goal achievement, outcome quality, completeness and appropriateness of the tools and approach used.
- Faithfulness score checks that responses stay faithful to the conversation and tool results.
- Completeness score checks that responses address every part of the request.
- Tool selection, parameter and utilization accuracy show whether the AI agent picked the right tool and passed it the right values.
AWS marks several of these metrics as available as part of Connect Customer AI. Check your pricing plan before you build reports around them.
3. Watch it in one dashboard, version by version
The AI Agent performance dashboard has a self-service summary with AI involved contacts, response completion rate, handoff rate, goal success rate and faithfulness score. Its AI agents table drills down from the use case to each AI agent version, so you can compare a new prompt with the one it replaced. The time range goes back up to three months, with a “compare to” period for trends.
4. Score self-service against your own criteria
The built-in metrics are generic. Your definition of success is not. Performance evaluations of self-service interactions let you build a form with your own questions. AWS’s example asks whether the contact was handled without transfer, whether the customer self-served at least one need, whether they expressed frustration, and whether the AI agent asked them to repeat themselves. Generative AI can fill the answers from the transcript, and a rule can submit evaluations automatically. Set the form’s contact interaction type to Automated interaction. AWS notes this feature is only available as part of Connect Customer with unlimited AI.
To define containment your way, use conversational analytics rules. The self-service evaluations guide describes a containment rule: the contact was handled during the automated interaction and no queue was assigned, optionally with a natural language check that the customer did not ask for a human.
5. Put the numbers where your reports live
The same metrics are available through the GetMetricDataV2 API, for example AI_HANDOFF_RATE and GOAL_SUCCESS_RATE. The analytics data lake holds AI session and AI agent tables, and AWS publishes reference queries, including AI handoff rate. Joining these with repeat contacts or revenue in your own reports is something you build. When a number looks wrong, open AI agent traces on the contacts behind it.
What good looks like
Leaders see one short scorecard: how many contacts the AI agent handled, how many it resolved by your own definition, the handoff rate and the goal success rate, against last month. Engineers see latency and tool success on the same dashboard. Each prompt or tool change is a new version, compared with the last one before it is kept. Weak spots come with transcripts, so the team knows what to fix.
Where to start
- Write down, with the business, what “resolved by the AI agent” means for each use case.
- Give stakeholders access to the AI Agent performance dashboard and agree on four or five metrics to review.
- Build a self-service evaluation form and a containment rule that match your definition.
- Review results every two weeks, and publish each change as a new AI agent version so you can compare.










