Skip to content
Ask AI about this
Help improve this page

MSK connectors should have logging enabled

MSK Connect connectors run distributed Kafka Connect workers that process data between systems. Without logging, you have no visibility into connector failures, task restarts, or throughput degradation. A silent connector failure can stall data pipelines for hours before anyone notices.

Worker logs surface authentication errors, serialization exceptions, and offset commit failures that matter most during incident triage. Enabling at least one log destination means you can diagnose issues without redeploying connectors with debug settings after the fact.

Retrofit consideration

Updating log_delivery on an existing aws_mskconnect_connector is typically an in-place connector update, not a replacement. That said, other immutable settings on the same connector can force a replacement. Check the plan carefully and time the change to avoid interrupting active connector tasks.

Implementation

Choose the approach that matches how you manage Terraform.

Use AWS provider resources directly. See docs for the resources involved: aws_mskconnect_connector.

resource "aws_mskconnect_connector" "this" {
  capacity {
    provisioned_capacity {
      mcu_count    = 1
      worker_count = 1
    }
  }
  connector_configuration = { "topic" = "pofix-example-topic" }

  kafka_cluster {
    apache_kafka_cluster {
      bootstrap_servers = "b-1.example.kafka.us-east-1.amazonaws.com:9092"

      vpc {
        security_groups = ["sg-12345678"]
        subnets         = ["subnet-12345678"]
      }
    }
  }

  kafka_cluster_client_authentication {
    authentication_type = "NONE"
  }

  kafka_cluster_encryption_in_transit {
    encryption_type = "PLAINTEXT"
  }

  kafkaconnect_version = "2.7.1"
  name                 = "pofix-abc123"

  plugin {
    custom_plugin {
      arn      = "arn:aws:kafkaconnect:us-east-1:123456789012:custom-plugin/example/1"
      revision = 1
    }
  }

  service_execution_role_arn = "arn:aws:iam::123456789012:role/example-role"

  log_delivery {
    worker_log_delivery {
      cloudwatch_logs {
        enabled = true
      }
    }
  }
}

What this control checks

The policy checks for a log_delivery block with a worker_log_delivery block on the aws_mskconnect_connector resource. Three destinations are available: cloudwatch_logs with enabled = true and a valid log_group ARN, firehose with enabled = true and a valid delivery_stream, or s3 with enabled = true, a bucket name, and an optional prefix. It fails if all three destinations are absent or explicitly set to enabled = false. One destination enabled is enough to pass; all three can be active simultaneously.

Common pitfalls

  • Connector update behavior for log_delivery changes

    The log_delivery block is not universally ForceNew. Changes typically go through in-place connector updates, but other immutable settings on the same resource can still trigger a replacement. Review the Terraform plan before applying, not after.

  • Destination set but not enabled

    Having a cloudwatch_logs, firehose, or s3 block present is not enough. The control looks for enabled = true inside the block. A destination block with no enabled attribute, or with enabled = false, fails the check.

  • Missing IAM permissions for log delivery

    Get the execution role wrong and connector creation fails outright, not silently. For CloudWatch Logs the role needs logs:CreateLogStream and logs:PutLogEvents. For S3, it needs s3:PutObject on the target bucket and prefix. Check the role policy before assuming a failed connector is a configuration problem.

  • Log group must exist before connector creation

    If the CloudWatch Logs log group named in cloudwatch_logs.log_group does not exist when Terraform runs, connector creation fails. Declare an explicit aws_cloudwatch_log_group resource and use depends_on to guarantee it is created first.

Audit evidence

An auditor expects Config rule evaluations showing each connector as COMPLIANT, or equivalent output from a policy scanning tool. The DescribeConnector API response should show the logDelivery.workerLogDelivery object with at least one destination where enabled is true. Actual log data in the target CloudWatch Logs group or S3 bucket strengthens the evidence by proving logs are flowing, not just configured.

For teams using Config conformance packs, the evaluation history for each connector resource provides a timestamped compliance trail for periodic reviews.

Tool mappings

Use these identifiers to cross-reference this control across tools, reports, and evidence.

  • Compliance.tf Control: mskconnect_connector_logging_enabled

  • AWS Config Managed Rule: MSK_CONNECT_CONNECTOR_LOGGING_ENABLED

  • Powerpipe Control: aws_compliance.control.mskconnect_connector_logging_enabled

  • AWS Security Hub Control: MSK.5

Last reviewed: 2026-03-09