Sending Fargate Logs to ContainIQ

Create a Firehose stream in Kinesis



• Change delivery stream name to ContainIQ
• Select Direct Put for the source field
• Select HTTP endpoint for the destination
• Put https://api.containiq.com/ingest/logs/fargate into the HTTP endpoint URL
• Put your API key into the access key field
• Add the parameter cluster with the value equal to what was set when you initially deployed the ContainIQ agent

AWS configuration

Create the aws-observability namespace

Run kubectl create ns aws-observability.

Create the FluentBit config

Replace <aws-region> below with the region of your cluster.

cat > fluentbit-config.yaml << EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-logging
  namespace: aws-observability
data:
  parsers.conf: |
    [PARSER]
        Name crio
        Format Regex
        Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>P|F) (?<log>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z
        Time_Keep On

  filters.conf: |
    [FILTER]
        Name parser
        Match kube.*
        Key_name log
        Parser crio

    [FILTER]
        Name                kubernetes
        Match               kube.*
        Merge_Log           On
        Buffer_Size         0
        Kube_Meta_Cache_TTL 300s

  output.conf: |
    [OUTPUT]
        Name kinesis_firehose
        Match kube.*
        region <your-aws-region>
        delivery_stream containiq
EOF

Apply the config

Run kubectl apply -f fluentbit-config.yaml.

Create the role JSON

Replace resource below with the ARN from the Firehose stream you created in the first step.

cat > allow_kinesis_put_permission.json << EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "firehose:PutRecord",
                "firehose:PutRecordBatch"
            ],
            "Resource": "<place firehose arn here>"
        }
    ]
}
EOF

Apply the role JSON

Run aws iam create-policy \ --policy-name FluentBitEKSFargate \ --policy-document file://allow_kinesis_put_permission.json.

Set the pod execution role

Replace <Your-Cluster-Name> below with the name of your EKS cluster. You can find this name by running aws eks list-clusters.

Replace <fargate-profile> below with the Fargate profile you want to get logs from. You can find this by running aws eks list-fargate-profiles --cluster-name <Your-Cluster-Name> .

POD_EXEC_ROLE=$(aws eks describe-fargate-profile \ --cluster-name <Your-Cluster-Name> \ --fargate-profile-name <fargate-profile> | jq -r '.fargateProfile.podExecutionRoleArn' | awk -F"/" '{print (NF>1)? $NF : ""}' )

Set the policy name

POLICY_NAME=$(aws iam list-policies --query 'Policies[?PolicyName==`FluentBitEKSFargate`].Arn' --output text)

Attach the policy

Run aws iam attach-role-policy \ --policy-arn $POLICY_NAME \ --role-name $POD_EXEC_ROLE.