Skip to content
Ask AI about this
Help improve this page

Kinesis streams should have an adequate data retention period

Kinesis streams default to a 24-hour retention period. When a downstream consumer fails, a pipeline goes down, or an incident needs investigating, 24 hours is rarely enough time to replay events or reconstruct what happened. A 7-day minimum gives teams a practical window to detect issues, replay failed processing, and do forensic analysis without losing the source records.

Extending retention does increase costs (Kinesis charges for extended data retention beyond 24 hours), but losing event data during an incident typically costs far more than the storage premium.

Retrofit consideration

Increasing retention on an existing stream is a non-disruptive API call, but extended retention pricing applies to every shard in the stream. Review the cost impact before rolling this out across high-shard-count streams at scale.

Implementation

Choose the approach that matches how you manage Terraform.

Use AWS provider resources directly. See docs for the resources involved: aws_kinesis_stream.

resource "aws_kinesis_stream" "this" {
  encryption_type  = "KMS"
  kms_key_id       = "arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012"
  name             = "pofix-abc123"
  retention_period = 168
  shard_count      = 1
}

What this control checks

In Terraform, aws_kinesis_stream exposes retention_period in hours. The AWS API default is 24. Set it to 168 or higher to pass; anything below 168 fails. For example, retention_period = 168 passes, while omitting the argument (which defaults to 24) or setting it to 72 both fail. The maximum allowed value is 8760 (365 days).

Common pitfalls

  • Default retention is 24 hours, not 168

    Omitting retention_period from aws_kinesis_stream results in AWS applying the 24-hour default. Terraform won't show drift if the argument was never set, so streams created without an explicit value silently fail the control.

  • On-demand vs provisioned mode does not change the requirement

    Switching a stream to on-demand capacity mode via stream_mode_details does not affect the retention period. Set retention_period explicitly regardless of which capacity mode the stream uses.

  • Extended retention pricing tiers

    AWS pricing distinguishes retention between 24-168 hours (extended retention) and 168-8760 hours (longer-term retention). Setting retention_period = 168 sits exactly at the tier boundary. Going above 168 hours moves into longer-term pricing, which can surprise teams operating many streams.

  • Decreasing retention deletes data immediately

    Lowering retention_period in Terraform (which calls DecreaseStreamRetentionPeriod) causes data older than the new window to be deleted immediately and irreversibly. Test retention changes in non-production first.

Audit evidence

An auditor expects AWS Config rule results for the Security Hub control (specifically the KINESIS_STREAM_RETENTION_PERIOD_CHECK managed rule or equivalent) showing all Kinesis streams as COMPLIANT. The Kinesis Data Streams detail page in the console, where the "Data retention period" field shows 168 hours or above for each stream, works as supporting evidence. CloudTrail IncreaseStreamRetentionPeriod and DecreaseStreamRetentionPeriod API events show when retention was last changed and by whom, giving auditors a change history for the control.

Framework-specific interpretation

Tool mappings

Use these identifiers to cross-reference this control across tools, reports, and evidence.

  • Compliance.tf Control: kinesis_stream_retention_period_168_hours

  • AWS Config Managed Rule: KINESIS_STREAM_BACKUP_RETENTION_CHECK

  • Powerpipe Control: aws_compliance.control.kinesis_stream_retention_period_168_hours

  • Prowler Check: kinesis_stream_data_retention_period

  • AWS Security Hub Control: Kinesis.3

Last reviewed: 2026-03-09