Shipping Logs from AWS CloudWatch to ElasticSearch with Fargate and Filebeat

Centralizing logs and efficiently analyzing them is vital for maintaining the health of your system. In this guide, we’ll walk through how to ship logs from AWS CloudWatch to ElasticSearch using Filebeat running on a Fargate cluster.

Requirements

  • A Fargate Cluster.
  • IAM roles for task execution and access control.
  • Filebeat configuration.

Setting Up a Fargate Cluster

Before we proceed, you’ll need to have an operational Fargate cluster ready to run the task.

IAM Roles Configuration

IAM roles define permissions that determine what the task and container can do when running. We need to create two roles:

  1. TaskRole: Allows the task to read secrets from Secrets Manager.

    {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetResourcePolicy",
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret",
        "secretsmanager:ListSecretVersionIds"
      ],
      "Resource": "arn:aws:secretsmanager:*:YOUR_ACCOUND_ID:secret:*"
    }
  ]
}

  2. ExecutionRole: Enables specific permissions including the vital ability for Filebeat to read from CloudWatch.

    {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecr:GetAuthorizationToken",
        "ecr:BatchCheckLayerAvailability",
        "ecr:GetDownloadUrlForLayer",
        "ecr:BatchGetImage",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "*"
    }
  ]
}

Filebeat Configuration

Filebeat is the tool that will be used to read logs from CloudWatch and forward them to ElasticSearch. Here’s a sample configuration for Filebeat:

Includes settings, input for AWS Cloudwatch, and output to Logstash

name: 'filebeat-cloudwatch'
tags: []
fields_under_root: true
fields:
  domain: 'domain.com'
  cluster: 'my-pony-cluster'

logging.level: debug

# This can be used to debug the input by going to
# :5066/inputs/?pretty <-- this will show the status of the cloudwatch input
http:
  enabled: true
  host: 0.0.0.0
  port: 5066

http.pprof.enabled: true

filebeat.inputs:
  - type: aws-cloudwatch
    log_group_arn: arn:aws:logs:YOUR_REGION:YOUR_ACCOUND_ID:log-group:YOUR_LOG_GROUP_TO_SHIP:*
    scan_frequency: 1m
    start_position: end
    region_name: YOUR_REGION
    fields_under_root: true
    fields:
      logzio_codec: json
      token: <REDACTED>
      type: fargate

processors:
  - rename:
      ignore_missing: true
      fields:
        - from: 'agent'
          to: 'beat_agent'
        - from: 'log.file.path'
          to: 'source'

output:
  logstash:
    hosts:
      - 'logstash:5015'
    ssl:
      certificate_authorities:
        - |
          -----BEGIN CERTIFICATE-----
          ...
          ...
          -----END CERTIFICATE-----

Fargate Service Configuration

Define a Fargate service and task in JSON format. These will include the details about the cluster, desired count of running tasks, network configurations, and container definitions.

JSON for Fargate Service

{
  "cluster": "MyTestCluster",
  "serviceName": "filebeat",
  "taskDefinition": "arn:aws:ecs:YOUR_REGION:YOUR_ACCOUND_ID:task-definition/filebeat:1",
  "desiredCount": 1,
  "launchType": "FARGATE",
  "role": "",
  "deploymentConfiguration": {
    "maximumPercent": 200,
    "minimumHealthyPercent": 0
  },
  "networkConfiguration": {
    "awsvpcConfiguration": {
      "subnets": ["subnet-***********"],
      "securityGroups": ["sg-************"],
      "assignPublicIp": "DISABLED"
    }
  }
}

JSON for Fargate Task

{
  "networkMode": "awsvpc",
  "taskRoleArn": "arn:aws:iam::YOUR_ACCOUND_ID:role/ecsTaskRole",
  "executionRoleArn": "arn:aws:iam::YOUR_ACCOUND_ID:role/ecsTaskExecutionRole",
  "cpu": "256",
  "memory": "512",
  "containerDefinitions": [
    {
      "name": "filebeat",
      "essential": true,
      "image": "docker.elastic.co/beats/filebeat:8.8.2-amd64",
      "command": ["-c", "/mnt/share/filebeat/filebeat.yml"],
      "mountPoints": [
        {
          "sourceVolume": "efs-volume",
          "containerPath": "/mnt/share/"
        }
      ],
      "secrets": [
        {
          "name": "AWS_ACCESS_KEY_ID",
          "valueFrom": "arn:aws:secretsmanager:YOUR_REGION:YOUR_ACCOUND_ID:secret:someSecretYouHave:FILEBEAT_AWS__AWS_ACCESS_KEY_ID::"
        },
        {
          "name": "AWS_SECRET_ACCESS_KEY",
          "valueFrom": "arn:aws:secretsmanager:YOUR_REGION:YOUR_ACCOUND_ID:secret:someSecretYouHave:FILEBEAT_AWS__AWS_SECRET_ACCESS_KEY::"
        }
      ],
      "logConfiguration": {   # OPTIONAL
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "YOUR_LOG_GROUP_TO_MONITOR_FILEBEAT",
          "awslogs-region": "YOUR_REGION",
          "awslogs-create-group": "true",
          "awslogs-stream-prefix": "filebeat"
        }
      }
    }
  ],
  "volumes": [
    {
      "name": "efs-volume",
      "efsVolumeConfiguration": {
        "fileSystemId": "fs-****************",
        "transitEncryption": "ENABLED"
      }
    }
  ],
  "requiresCompatibilities": ["FARGATE"],
  "family": "filebeat"
}

Deployment

With the configurations ready, you can deploy the Filebeat on the Fargate cluster:

  1. Deploy the task definition.
  2. Deploy the service definition.
  3. Monitor your logs in ElasticSearch.

Buy Me a Coffee