Sending Fargate Logs to ContainIQ

5min
Create a Firehose stream in Kinesis
﻿
Change delivery stream name to ContainIQ
Select Direct Put for the source field
Select HTTP endpoint for the destination
Put https://api.containiq.com/ingest/logs/fargate into the HTTP endpoint URL
Put your API key into the access key field
Add the parameter cluster with the value equal to what was set when you initially deployed the ContainIQ agent
AWS configuration
Create the aws-observability namespace
Run kubectl create ns aws-observability.
Create the FluentBit config
Replace <aws-region> below with the region of your cluster.
Shell
|
cat > fluentbit-config.yaml << EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-logging
  namespace: aws-observability
data:
  parsers.conf: |
    [PARSER]
        Name crio
        Format Regex
        Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>P|F) (?<log>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z
        Time_Keep On

  filters.conf: |
    [FILTER]
        Name parser
        Match kube.*
        Key_name log
        Parser crio

    [FILTER]
        Name                kubernetes
        Match               kube.*
        Merge_Log           On
        Buffer_Size         0
        Kube_Meta_Cache_TTL 300s

  output.conf: |
    [OUTPUT]
        Name kinesis_firehose
        Match kube.*
        region <your-aws-region>
        delivery_stream containiq
EOF
cat > fluentbit-config.yaml << EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-logging
  namespace: aws-observability
data:
  parsers.conf: |
    [PARSER]
        Name crio
        Format Regex
        Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>P|F) (?<log>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z
        Time_Keep On

  filters.conf: |
    [FILTER]
        Name parser
        Match kube.*
        Key_name log
        Parser crio

    [FILTER]
        Name                kubernetes
        Match               kube.*
        Merge_Log           On
        Buffer_Size         0
        Kube_Meta_Cache_TTL 300s

  output.conf: |
    [OUTPUT]
        Name kinesis_firehose
        Match kube.*
        region <your-aws-region>
        delivery_stream containiq
EOF
﻿
Apply the config
Run kubectl apply -f fluentbit-config.yaml.
Create the role JSON
Replace resource below with the ARN from the Firehose stream you created in the first step.
Shell
|
cat > allow_kinesis_put_permission.json << EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "firehose:PutRecord",
                "firehose:PutRecordBatch"
            ],
            "Resource": "<place firehose arn here>"
        }
    ]
}
EOF
cat > allow_kinesis_put_permission.json << EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "firehose:PutRecord",
                "firehose:PutRecordBatch"
            ],
            "Resource": "<place firehose arn here>"
        }
    ]
}
EOF
﻿
Apply the role JSON
Run aws iam create-policy \
--policy-name FluentBitEKSFargate \
--policy-document file://allow_kinesis_put_permission.json.
Set the pod execution role
Replace <Your-Cluster-Name> below with the name of your EKS cluster. You can find this name by running aws eks list-clusters.
Replace  <fargate-profile>  below with the Fargate profile you want to get logs from. You can find this by running aws eks list-fargate-profiles --cluster-name <Your-Cluster-Name> .
POD_EXEC_ROLE=$(aws eks describe-fargate-profile \
--cluster-name <Your-Cluster-Name> \
--fargate-profile-name <fargate-profile> | jq -r '.fargateProfile.podExecutionRoleArn' | awk -F"/" '{print (NF>1)? $NF : ""}' ) 
Set the policy name
Shell
|
POLICY_NAME=$(aws iam list-policies --query 'Policies[?PolicyName==`FluentBitEKSFargate`].Arn' --output text)
POLICY_NAME=$(aws iam list-policies --query 'Policies[?PolicyName==`FluentBitEKSFargate`].Arn' --output text)
﻿
Attach the policy
Run aws iam attach-role-policy \
--policy-arn $POLICY_NAME \
--role-name $POD_EXEC_ROLE.
Updated 03 Mar 2023
Did this page help you?
Installing ContainIQ
Sending Logs from a Sidecar Container
Docs powered by
Archbee
TABLE OF CONTENTS
Create a Firehose stream in Kinesis
AWS configuration