Blocker 38 · AI

Is the AI agent actually working? Measuring containment, handoffs and goal success on Amazon Connect

An AI agent that takes calls is easy to launch and hard to judge. Did customers get what they needed, or did they give up and call back? Amazon Connect measures AI agents at several levels, from technical health to whether the customer's goal was met.

Illustration. The blocker: Is the AI agent actually working?. The fix: Containment and handoffs you can see.

The blocker

The AI agent went live three months ago. Someone in a steering meeting asks whether it is working. The answers come from different directions. The project team says it handled thousands of conversations. A supervisor says agents still get angry callers who “tried the robot first”. Finance wants to know how many calls it took off the queue. Nobody can say with confidence whether customers got what they came for.

Without that answer, AI projects stall. Expansion to new use cases waits for proof that never arrives, and the first bad anecdote carries more weight than any number.

Why it happens

  • Activity is mistaken for success. Counting conversations says nothing about whether the customer’s goal was met.
  • Containment is measured crudely. A customer who hangs up in frustration looks the same as one whose issue was solved.
  • Technical and business views are separate. Engineers watch errors and latency, while leaders ask about outcomes. Neither sees the other’s numbers.
  • Changes are not compared. A new prompt or tool goes live, and nobody checks whether the previous version was better.

How Amazon Connect solves it

Amazon Connect gives you AI agent metrics at several levels: the session, the AI agent, the prompt and each tool. Use them together, from “is it running?” to “did it help?”.

1. Know how much the AI agent handles, and how often it hands off

Start with volume and handoffs. AI involved contacts counts contacts where AI agents took part. AI handoffs counts self-service contacts that escalated to a human agent, and AI handoff rate is that as a percentage. AI response completion rate shows the share of sessions that successfully responded to the customer’s requests. All are listed in the metric definitions with their calculation logic.

A handoff is not always a failure. In agentic self-service, your prompt tells the AI agent when it must escalate. Read the handoff rate next to the reasons, which a custom Escalate tool can capture.

2. Measure outcomes, not just activity

Several metrics use an automated LLM-based evaluation of the conversation:

  • Goal success rate scores each session on goal achievement, outcome quality, completeness and appropriateness of the tools and approach used.
  • Faithfulness score checks that responses stay faithful to the conversation and tool results.
  • Completeness score checks that responses address every part of the request.
  • Tool selection, parameter and utilization accuracy show whether the AI agent picked the right tool and passed it the right values.

AWS marks several of these metrics as available as part of Connect Customer AI. Check your pricing plan before you build reports around them.

3. Watch it in one dashboard, version by version

The AI Agent performance dashboard has a self-service summary with AI involved contacts, response completion rate, handoff rate, goal success rate and faithfulness score. Its AI agents table drills down from the use case to each AI agent version, so you can compare a new prompt with the one it replaced. The time range goes back up to three months, with a “compare to” period for trends.

4. Score self-service against your own criteria

The built-in metrics are generic. Your definition of success is not. Performance evaluations of self-service interactions let you build a form with your own questions. AWS’s example asks whether the contact was handled without transfer, whether the customer self-served at least one need, whether they expressed frustration, and whether the AI agent asked them to repeat themselves. Generative AI can fill the answers from the transcript, and a rule can submit evaluations automatically. Set the form’s contact interaction type to Automated interaction. AWS notes this feature is only available as part of Connect Customer with unlimited AI.

To define containment your way, use conversational analytics rules. The self-service evaluations guide describes a containment rule: the contact was handled during the automated interaction and no queue was assigned, optionally with a natural language check that the customer did not ask for a human.

5. Put the numbers where your reports live

The same metrics are available through the GetMetricDataV2 API, for example AI_HANDOFF_RATE and GOAL_SUCCESS_RATE. The analytics data lake holds AI session and AI agent tables, and AWS publishes reference queries, including AI handoff rate. Joining these with repeat contacts or revenue in your own reports is something you build. When a number looks wrong, open AI agent traces on the contacts behind it.

What good looks like

Leaders see one short scorecard: how many contacts the AI agent handled, how many it resolved by your own definition, the handoff rate and the goal success rate, against last month. Engineers see latency and tool success on the same dashboard. Each prompt or tool change is a new version, compared with the last one before it is kept. Weak spots come with transcripts, so the team knows what to fix.

Where to start

  1. Write down, with the business, what “resolved by the AI agent” means for each use case.
  2. Give stakeholders access to the AI Agent performance dashboard and agree on four or five metrics to review.
  3. Build a self-service evaluation form and a containment rule that match your definition.
  4. Review results every two weeks, and publish each change as a new AI agent version so you can compare.
High-level architectureMeasuring AI agents from health to outcome
Customer
  • Voice
  • Chat
Self-service
  • Orchestrator AI agent
  • Tools
  • Escalate
Metrics
  • Handoff rate
  • Goal success rate
  • Tool accuracy
Quality
  • Category rules
  • Self-service evaluations
Reporting
  • AI Agent dashboard
  • GetMetricDataV2
  • Analytics data lake
  • People
  • Amazon Connect
  • AWS services
  1. Customers reach an orchestrator AI agent. It uses tools, and hands off to a person with its Escalate tool when it should.
  2. Amazon Connect records AI agent metrics for every session: invocations, latency, handoffs, and LLM-based scores such as goal success rate and faithfulness.
  3. Conversational analytics rules flag patterns such as contained contacts or customers asking for a human.
  4. Self-service evaluation forms, filled by generative AI, score each automated interaction against your own criteria.
  5. The AI Agent performance dashboard compares versions over time. The same metrics are available through GetMetricDataV2 and the analytics data lake.
References

Official AWS documentation

Every Amazon Connect capability in this post links to AWS’s own documentation, so you can check the details for your Region and account.

  1. AI agent metricsdocs.aws.amazon.com
  2. AI Agent performance dashboarddocs.aws.amazon.com
  3. Metric definitionsdocs.aws.amazon.com
  4. GetMetricDataV2docs.aws.amazon.com
  5. Performance evaluations of self-service interactionsdocs.aws.amazon.com
  6. Automatically categorize contactsdocs.aws.amazon.com
  7. Use agentic self-servicedocs.aws.amazon.com
  8. AI agent traces using Contact search and Contact detailsdocs.aws.amazon.com
  9. Reference queries for the data lakedocs.aws.amazon.com
  10. Analytics data lakedocs.aws.amazon.com
Start here

Tell us what you want to improve.

We will tell you whether a voice agent fits it, what it would take, and what it should return. You will hear back within a week.

  • AWS Advanced Tier Services Partner
  • AWS Migration Services Competency
  • Amazon Connect Delivery Partner
  • AWS AI Services Competency
  • AWS Experience-Based Acceleration (EBA) partner
  • ISO/IEC 27001 certified
  • SOC 2 Type II
Certified supplierMWDBE certifiedTexas HUB certifiedNCTRCA certified
+1 (214) 550-14936275 West Plano Parkway, Suite 500, Plano, Texas 75093