website logo
⌘K
ContainIQ Overview
Installing ContainIQ
Sending Fargate Logs to ContainIQ
Sending Logs from a Sidecar Container
Git Integration
Prometheus Integration
Requirements
Using ContainIQ
Security
Docs powered by archbee 

Sending Logs from a Sidecar Container

5min

In certain situations, it may not be feasible to run the ContainIQ agent as a daemonset. For example, with AWS Fargate you cannot schedule daemonsets onto Fargate nodes, or with GKE autopilot you may run into limitations around Fluentd's host map. Fortunately, ContainIQ also supports collecting logs from a sidecar container.

Setup



Start by creating a ConfigMap with the necessary Fluentd config.

  1. Replace /path-to-log-file with the path where you are logging to
  2. Replace ${record["timestamp"]} with correct record from your log message. See https://docs.fluentd.org/filter/record_transformer for more info on how to set up a record transformer.
  3. If you would like to add additional fields to ContainIQ for improved searchability you will need to add the additional fields to the record section of the configuration. the following are available
    1. pod_name ex: pod_name ${record["app-field-name-in-log"]}
    2. container
    3. stream
YAML
|
      - name: fluentd
          imagePullPolicy: Always
          image: containiq/logging-agent
          env:
            # https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter#environment-variables-for-kubernetes
            - name: K8S_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: INGEST_LOGS_ENDPOINT_URL
              value: https://api.containiq.com/ingest/logs
            - name: NOTIFICATION_FILE_LOCATION
              value: /containiq/notification-config.yaml
            - name: FLUENT_KUBERNETES_METADATA_SKIP_LABELS
              value: 'true'
            - name: FLUENT_KUBERNETES_METADATA_SKIP_CONTAINER_METADATA
              value: 'true'
            - name: FLUENT_KUBERNETES_METADATA_SKIP_NAMESPACE_METADATA
              value: 'true'
            - name: FLUENT_KUBERNETES_METADATA_SKIP_MASTER_URL
              value: 'true'
            - name: FLUENT_CONTAINER_TAIL_EXCLUDE_PATH
              value: /var/log/containers/fluent*
            - name: FLUENTD_SYSTEMD_CONF
              value: 'disable'
            - name: CLUSTER_NAME
              valueFrom:
                configMapKeyRef:
                  name: cluster-config
                  key: cluster-name
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 200Mi
          volumeMounts:
            - name: containiqconfig
              mountPath: /containiq
            - name: sidecarconf
              mountPath: /fluentd/etc/conf.d
            - name: varlog
              mountPath: /path/to/logs
      volumes:
        - name: varlog
          emptyDir: {}
        - name: sidecarconf
          configMap:
            name: sidecarconf
        - name: containiqconfig
          projected:
            sources:
              - secret:
                  name: containiq-notification


Next add the Fluentd container as a sidecar in your deployment and mount the correct volumes.

  1. Replace /path/to/logs with the location of your log file.
  2. Add the varlog volume mount to the container you will be logging from.
YAML
|
      - name: fluentd
          imagePullPolicy: Always
          image: containiq/logging-agent
          env:
            # https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter#environment-variables-for-kubernetes
            - name: K8S_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: INGEST_LOGS_ENDPOINT_URL
              value: https://api.containiq.com/ingest/logs
            - name: NOTIFICATION_FILE_LOCATION
              value: /containiq/notification-config.yaml
            - name: FLUENT_KUBERNETES_METADATA_SKIP_LABELS
              value: 'true'
            - name: FLUENT_KUBERNETES_METADATA_SKIP_CONTAINER_METADATA
              value: 'true'
            - name: FLUENT_KUBERNETES_METADATA_SKIP_NAMESPACE_METADATA
              value: 'true'
            - name: FLUENT_KUBERNETES_METADATA_SKIP_MASTER_URL
              value: 'true'
            - name: FLUENT_CONTAINER_TAIL_EXCLUDE_PATH
              value: /var/log/containers/fluent*
            - name: FLUENTD_SYSTEMD_CONF
              value: 'disable'
            - name: CLUSTER_NAME
              valueFrom:
                configMapKeyRef:
                  name: cluster-config
                  key: cluster-name
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 200Mi
          volumeMounts:
            - name: containiqconfig
              mountPath: /containiq
            - name: sidecarconf
              mountPath: /fluentd/etc/conf.d
            - name: varlog
              mountPath: /path/to/logs
      volumes:
        - name: varlog
          emptyDir: {}
        - name: sidecarconf
          configMap:
            name: sidecarconf
        - name: containiqconfig
          projected:
            sources:
              - secret:
                  name: containiq-notification


Finally, add the Fluentd service account under the spec:

YAML
|
spec:
    serviceAccount: fluentd
    serviceAccountName: fluentd


Pleas reach out to support@containiq.com with any questions.

Updated 03 Mar 2023
Did this page help you?
Yes
No
PREVIOUS
Sending Fargate Logs to ContainIQ
NEXT
Git Integration
Docs powered by archbee 
TABLE OF CONTENTS
Setup