Blocker 47 · DevOps

Last year's calls are gone: keeping and exporting Amazon Connect contact data

Amazon Connect keeps contact records for a fixed window, and the analytics data lake keeps a rolling one. If you need data for longer, or in your own tools, you have to export it. The good news is that every piece of that export is documented.

24 monthsAmazon Connect keeps contact records for 24 months from when the contact started. Stream them to Kinesis to manage retention yourself.Source: AWS, Amazon Connect feature specifications

Illustration. The blocker: Contact data is gone when you need it. The fix: Every record streamed and kept.

The blocker

An auditor asks for every contact with a customer over the past three years. A product team wants to compare this winter’s call reasons with the winter before last. Someone opens contact search, and the older contacts are not there. The recordings exist somewhere, but nobody can match them to a queue, an agent or an outcome.

Contact data is one of the most useful things a contact center produces. But Amazon Connect keeps each kind of data for a set period. If nobody decided where the data goes after that, it is not available when the question is asked.

Why it happens

  • Retention is assumed, not checked. Teams expect the platform to keep everything forever.
  • Streaming was never turned on. Contact records and agent events are not exported by default.
  • Recordings and records live apart. Audio sits in one bucket and the metadata that explains it sits somewhere else, or nowhere.
  • No lifecycle rules. Buckets either grow forever or get cleaned up by hand, with no agreed retention period.

How Amazon Connect solves it

1. Know the retention windows

The feature specifications list them. Contact records for all channels are kept for 24 months from when the contact was initiated, and agent activity for 24 months from when the event occurred. AWS states directly that you can stream contact records to Kinesis to manage retention yourself. The analytics data lake keeps a rolling 25-month window, which you cannot change.

Email messages and attachments are different: their retention follows the S3 Lifecycle configuration of the bucket that stores them.

2. Stream contact records and agent events

Data streaming is a setting on the instance. For contact records you choose an Amazon Data Firehose stream or a Kinesis data stream. For agent events you choose a Kinesis data stream. Agent event streams carry logins, logouts, status changes and contact connections in near real time. AWS recommends separate streams for the two, because they are easier to manage that way.

For long-term storage, Firehose is the simplest route, because it delivers streaming data to Amazon S3 without code. A Kinesis data stream is not storage on its own: it keeps records for 24 hours by default, up to 365 days. If you use one, put a consumer behind it that writes to S3.

Two details from the contact record data model matter for whoever builds the consumer. Records are delivered at least once and can be delivered again when new information arrives, so deduplicate on ContactId and keep the copy with the latest LastUpdateTimestamp. And new fields are added over time, so the consumer should ignore fields it does not recognize.

3. Keep recordings and transcripts in buckets you control

Under Data storage in the instance settings, you choose the S3 bucket and KMS key for call recordings, chat transcripts, exported reports and evaluations. Recording itself is turned on in the flow with the Set recording and analytics behavior block.

Because these buckets are in your account, their retention is your decision. The two kinds of data link up through the contact record: when recording was enabled, its recording fields give the Amazon S3 location of the recording or transcript. Export the records and you can always find the audio for a given queue, agent or outcome.

4. Copy the data lake before it ages out

The analytics data lake holds contact records, conversational analytics, evaluations, forecasts and more, queryable with SQL. To keep data past the 25-month window, AWS’s retention guidance is to run a scheduled query, such as CREATE TABLE AS SELECT or UNLOAD, that writes each table to your own S3 bucket. Each table has a named timestamp column that sets its age, so build the export on that column.

If you only need dashboards, see our post on leadership visibility. This post is about keeping the raw data.

5. Apply S3 Lifecycle rules

S3 Lifecycle rules move objects to lower-cost storage classes after a set time and expire them at the end of a period you set. Use one rule set per kind of data: recordings, transcripts, contact records and data lake exports often have different retention periods. Note that transitions have request costs, and some storage classes have a minimum storage duration charge.

Retention periods are a legal and policy decision, not a technical one. Your legal and compliance teams set them. AWS secures the service; you are responsible for how long you keep data, who can access it and how it is encrypted, under the AWS shared responsibility model.

What good looks like

Every contact record lands in S3 soon after the contact, deduplicated and organized by date. Each record points to its recording or transcript, and both follow the same retention policy. A monthly job copies data lake tables before they age out. Lifecycle rules move old data to cheaper storage and delete it on schedule. When the auditor asks for three years of contacts, the answer is a query, not a project.

Where to start

  1. Write down each kind of contact data you hold, where it lives today and its retention window.
  2. Agree retention periods for each with legal and compliance.
  3. Turn on data streaming for contact records (to Firehose and S3) and agent events.
  4. Schedule an export of the data lake tables you need beyond 25 months.
  5. Add S3 Lifecycle rules to every bucket, and review them yearly.
High-level architectureContact data kept in your own account
Amazon Connect
  • Contact records
  • Agent events
  • Recordings and transcripts
  • Analytics data lake
Streaming
  • Kinesis Data Streams
  • Amazon Data Firehose
Storage
  • Amazon S3
  • S3 Lifecycle
Analysis
  • Amazon Athena
  • Scheduled exports
Your teams
  • Analysts
  • Compliance
  • Amazon Connect
  • AWS services
  • People
  1. Data streaming sends every contact record to Amazon Data Firehose or Kinesis Data Streams, and agent events to a Kinesis data stream.
  2. Firehose delivers the records to an Amazon S3 bucket in your account, with no consumer code to write.
  3. Call recordings and chat transcripts already land in the S3 buckets you chose for the instance.
  4. A scheduled query copies analytics data lake tables into your own bucket before they leave the rolling 25-month window.
  5. S3 Lifecycle rules move older objects to lower-cost storage classes and expire them when your retention period ends.
References

Official AWS documentation

Every Amazon Connect capability in this post links to AWS’s own documentation, so you can check the details for your Region and account.

  1. Amazon Connect feature specificationsdocs.aws.amazon.com
  2. Data model for Amazon Connect contact recordsdocs.aws.amazon.com
  3. Enable data streaming for your instancedocs.aws.amazon.com
  4. Amazon Connect agent event streamsdocs.aws.amazon.com
  5. Enable agent event streamsdocs.aws.amazon.com
  6. Update instance settings (data storage)docs.aws.amazon.com
  7. Enable contact recordingdocs.aws.amazon.com
  8. Amazon Connect data lakedocs.aws.amazon.com
  9. Data retention in the analytics data lakedocs.aws.amazon.com
  10. What is Amazon Data Firehose?docs.aws.amazon.com
  11. Change the data retention period (Kinesis Data Streams)docs.aws.amazon.com
  12. Managing the lifecycle of objects (Amazon S3)docs.aws.amazon.com
Start here

Tell us what you want to improve.

We will tell you whether a voice agent fits it, what it would take, and what it should return. You will hear back within a week.

  • AWS Advanced Tier Services Partner
  • AWS Migration Services Competency
  • Amazon Connect Delivery Partner
  • AWS AI Services Competency
  • AWS Experience-Based Acceleration (EBA) partner
  • ISO/IEC 27001 certified
  • SOC 2 Type II
Certified supplierMWDBE certifiedTexas HUB certifiedNCTRCA certified
+1 (214) 550-14936275 West Plano Parkway, Suite 500, Plano, Texas 75093