Blocker 50 · DevOps

Something broke and nobody knows what to do: incident response for Amazon Connect

When calls start failing, the first ten minutes decide how many customers are affected. A good incident process answers three questions before anyone asks them: is it us or AWS, who is on call, and what do they check first?

Illustration. The blocker: Nobody knows what to do in an outage. The fix: Alerts, runbooks and a clear owner.

The blocker

At 9:40 a.m. a supervisor messages the IT channel: customers say the main line rings and then drops. Three engineers start looking at once. One checks the flows, one checks the phone numbers, one wonders whether AWS is having a problem. Nobody is sure who is in charge, nobody has the AWS Support login, and nobody tells the contact center floor what to say to customers. Forty minutes later someone finds the cause: a change made the night before.

The monitoring may have been there. What was missing was an incident process: who responds, what they check, how they escalate and what the team learns afterward.

Why it happens

  • Alerts go to a shared inbox. Everyone sees them, so nobody owns them.
  • No runbooks. Each incident starts from a blank page, and the steps live in one person’s head.
  • “Is it AWS?” takes too long. Teams guess instead of checking a known source.
  • Escalation is unclear. Nobody knows the support plan, or what information AWS Support needs.
  • No review. The same incident happens again because nothing changed after the first one.

How Amazon Connect solves it

1. Know whether it is AWS or you

AWS Health gives visibility into the availability of AWS services and the health of your resources. The AWS Health Dashboard is available to every customer at no extra cost and needs no setup. Its public Service health page shows events across Regions. When you sign in, Your account health shows events that can affect your own account.

Do not rely on someone remembering to look. AWS Health sends events to Amazon EventBridge. Create a rule that sends Health events for the services you use to the same channel as your alarms. AWS notes that public events can take up to an hour to start arriving after you create the rule, so set it up before you need it.

2. Send alarms to a person, not an inbox

Our monitoring and alerts post covers which Amazon Connect metrics to alarm on. This step is about where alarms go. CloudWatch alarms can notify through Amazon SNS or AWS User Notifications. User Notifications delivers to email, chat and the AWS Console Mobile Application, and can aggregate notifications during an event so responders are not flooded.

Route each alarm to an on-call rotation with one named person at a time. Put a runbook link in the alarm description, so the person paged knows where to start.

3. Write runbooks for the incidents you will have

The AWS Well-Architected Framework describes a runbook as a documented procedure that produces the same outcome every time, no matter who runs it. Each one should state the desired outcome, the tools and permissions needed, and the error-handling and escalation path. The Amazon Connect operational excellence guidance recommends using the CCP troubleshooting guide to create runbooks and playbooks for agents and supervisors to follow when they hit issues.

A good first set for Amazon Connect covers: calls failing at the start, flow errors rising, a Lambda integration failing, agents unable to log in, and poor audio. Each one starts the same way: check AWS Health, then check CloudTrail for recent changes. AWS’s routing isolation guide shows how to use the contact record and flow logs to decide whether a problem is inside Amazon Connect or outside it, at the carrier or the device.

Include a line for the contact center lead: what to tell agents, and whether to switch to an emergency message.

4. Automate the steps you repeat

AWS Systems Manager Automation runs runbooks as code, with approvals where needed. An automation can be started by an EventBridge event. Start with short, frequent steps, such as collecting the recent CloudTrail events and alarm history into one place for the responder. The automation steps themselves are something you build and test.

5. Escalate to AWS Support with the right details

Know your support plan before an incident. AWS Support case management explains that Basic Support cannot open technical cases, and that you choose a severity when you create a case. Describe the problem in detail and include resource information. For Amazon Connect, that means the instance ARN, the Region, affected contact IDs, times and what changed. AWS’s operational excellence guidance recommends AWS Enterprise Support for business-critical workloads.

6. Review every significant incident

AWS’s guidance on post-incident analysis is to examine the contributing factors behind each customer-impacting event, turn the findings into tracked actions with named owners, and share what you learn across teams. Keep reviews blameless and short. Every review should end with changes: a new alarm, a fixed runbook or a safer deployment step.

What good looks like

An alarm pages one on-call engineer two minutes after calls start failing, with a runbook link. They check AWS Health, see no event, and find a flow change in CloudTrail. They roll it back while the contact center lead tells agents what to say. The review the next day adds a test that would have caught the change.

Where to start

  1. Create an EventBridge rule for AWS Health events and send it to your alert channel.
  2. Route every CloudWatch alarm to a named on-call person, not a shared inbox.
  3. Write five runbooks for your most likely incidents, starting with “calls fail at the start”.
  4. Confirm your AWS Support plan and who can open a case.
  5. Hold a short review after every significant incident, and track its actions to done.
High-level architectureIncident response for an Amazon Connect contact center
Signals
  • Instance metrics
  • Flow logs
  • Contact records
Detect
  • AWS Health events
  • CloudWatch alarms
  • EventBridge rules
Notify
  • Amazon SNS
  • AWS User Notifications
Respond
  • On-call engineer
  • Runbooks
  • Contact center lead
Escalate and learn
  • Systems Manager Automation
  • AWS Support case
  • Post-incident review
  • Amazon Connect
  • AWS services
  • People
  1. CloudWatch alarms on Amazon Connect and Lambda metrics, and AWS Health events through EventBridge, detect a problem.
  2. Alarm and Health notifications go through Amazon SNS or AWS User Notifications to the on-call engineer, with a link to the matching runbook.
  3. The engineer follows the runbook: check AWS Health, check recent changes in CloudTrail, then isolate the problem with contact records and flow logs.
  4. Repeatable fixes run as Systems Manager Automation runbooks. If the cause is inside AWS, the engineer opens an AWS Support case with the contact IDs and instance ARN.
  5. After the incident, a post-incident review records contributing factors and actions, and updates alarms and runbooks.
References

Official AWS documentation

Every Amazon Connect capability in this post links to AWS’s own documentation, so you can check the details for your Region and account.

  1. What is AWS Health?docs.aws.amazon.com
  2. AWS Health Dashboarddocs.aws.amazon.com
  3. Monitoring events in AWS Health with Amazon EventBridgedocs.aws.amazon.com
  4. Using Amazon CloudWatch alarmsdocs.aws.amazon.com
  5. Notifying users on alarm changesdocs.aws.amazon.com
  6. Operational excellence in Amazon Connect workloadsdocs.aws.amazon.com
  7. Troubleshooting issues with the Contact Control Panel (CCP)docs.aws.amazon.com
  8. Isolate routing problems by using the contact record and flow logsdocs.aws.amazon.com
  9. Log Amazon Connect API calls with AWS CloudTraildocs.aws.amazon.com
  10. AWS Systems Manager Automationdocs.aws.amazon.com
  11. Run automations based on EventBridge eventsdocs.aws.amazon.com
  12. AWS Support case managementdocs.aws.amazon.com
  13. OPS05-BP01 Develop and test runbooks to perform proceduresdocs.aws.amazon.com
  14. OPS07-BP01 Perform post-incident analysisdocs.aws.amazon.com
Start here

Tell us what you want to improve.

We will tell you whether a voice agent fits it, what it would take, and what it should return. You will hear back within a week.

  • AWS Advanced Tier Services Partner
  • AWS Migration Services Competency
  • Amazon Connect Delivery Partner
  • AWS AI Services Competency
  • AWS Experience-Based Acceleration (EBA) partner
  • ISO/IEC 27001 certified
  • SOC 2 Type II
Certified supplierMWDBE certifiedTexas HUB certifiedNCTRCA certified
+1 (214) 550-14936275 West Plano Parkway, Suite 500, Plano, Texas 75093