Using fluent-bit to log to Loki on Kubernetes

In this example, I will log to Loki using Fluent-Bit on k3s distribution of Kubernetes on my Raspberry Pi Cluster.

I am referencing the documentation from fluent-bit to get the sources.

I have a Loki instance running on 192.168.0.20 which is listening on port 3100 and will be assigning it to the FLUENT_LOKI_URL environment variable.

Create the namespace:

$ kubectl create namespace logging

Create the service account and role:

$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-service-account.yaml
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role.yaml
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role-binding.yaml

Fluent-bit will be setup as a Daemon set so it will run on every node, and the config will be stored as a config map.

The original config will be available at fluent-bit-configmap.yaml but since we are not logging to Elasticsearch, but to Loki, we will edit the config:

$ wget https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-configmap.yaml

And then add the loki output config:

  output-loki.conf: |
    [OUTPUT]
        Name loki
        Match *
        Url ${FLUENT_LOKI_URL}
        BatchWait 1s
        BatchSize 30720
        Labels {job="fluentbit", stack="pistack"}
        AutoKubernetesLabels true

and also adjust the config to include the file that we are introducing:

...
    @INCLUDE output-loki.conf
...

So my complete config map will look like this:

apiVersion: v1  
kind: ConfigMap  
metadata:  
  name: fluent-bit-config
  namespace: logging
  labels:
    k8s-app: fluent-bit
data:  
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-loki.conf

  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10

  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off

  output-elasticsearch.conf: |
    [OUTPUT]
        Name            es
        Match           *
        Host            ${FLUENT_ELASTICSEARCH_HOST}
        Port            ${FLUENT_ELASTICSEARCH_PORT}
        Index           fluent-pistack-k3s
        Logstash_Format On
        Replace_Dots    On
        Retry_Limit     False
        HTTP_User       user
        HTTP_Passwd     pass
        tls             On

  output-stdout.conf: |
    [OUTPUT]
        Name  stdout
        Match *

  output-loki.conf: |
    [OUTPUT]
        Name loki
        Match *
        Url ${FLUENT_LOKI_URL}
        BatchWait 1s
        BatchSize 30720
        Labels {job="fluentbit", stack="pistack"}
        AutoKubernetesLabels true

  parsers.conf: |
    [PARSER]
        Name   apache
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache2
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache_error
        Format regex
        Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$

    [PARSER]
        Name   nginx
        Format regex
        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

    [PARSER]
        Name        syslog
        Format      regex
        Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key    time
        Time_Format %b %d %H:%M:%S

Create the config map:

$ kubectl apply -f fluent-bit-configmap.yaml

The Daemon set deployment manifest was retrieved from fluent-bit-ds.yaml, but the image used is not based for ARM, so in this post I demonstrated how to build images for ARM.

I have included that image in the manifest below and also informed the environment section about my loki url in FLUENT_LOKI_URL:

$ cat 
apiVersion: apps/v1  
kind: DaemonSet  
metadata:  
  name: fluent-bit
  namespace: logging
  labels:
    k8s-app: fluent-bit-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:  
  selector:
    matchLabels:
      k8s-app: fluent-bit-logging
  template:
    metadata:
      labels:
        k8s-app: fluent-bit-logging
        version: v1
        kubernetes.io/cluster-service: "true"
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "2020"
        prometheus.io/path: /api/v1/metrics/prometheus
    spec:
      containers:
      - name: fluent-bit
        image: pistacks/fluent-bit-plugin-loki:armv7-latest
        #image: fluent/fluent-bit:arm32v7-1.6.1
        imagePullPolicy: Always
        ports:
          - containerPort: 2020
        env:
        - name: FLUENT_LOKI_URL
          value: "http://192.168.0.20:3100/loki/api/v1/push"
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
      terminationGracePeriodSeconds: 10
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config
      serviceAccountName: fluent-bit
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - operator: "Exists"
        effect: "NoExecute"
      - operator: "Exists"
        effect: "NoSchedule"

Go ahead and create the deployment:

$ kubectl apply -f fluent-bit-ds.yaml

Once all the pods has checked in, we can verify:

$ kubectl get pods -n logging -o wide
NAME               READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES  
fluent-bit-sg62c   1/1     Running   0          8h    10.42.4.24   rpi-03   <none>           <none>  
fluent-bit-s4gx4   1/1     Running   0          8h    10.42.2.92   rpi-06   <none>           <none>  
fluent-bit-8p76b   1/1     Running   0          8h    10.42.1.82   rpi-07   <none>           <none>  
fluent-bit-fnvmd   1/1     Running   0          8h    10.42.3.23   rpi-02   <none>           <none>  
fluent-bit-zsp82   1/1     Running   0          8h    10.42.0.77   rpi-05   <none>           <none>  
fluent-bit-89xl5   1/1     Running   0          8h    10.42.5.23   rpi-01   <none>           <none>  

Head over to Grafana and check if we can see logs under the job fluentbit:

image

Let's test it out by running a pod that just pings:

$ kubectl run --rm -it --restart=Never debug --image=busybox -- ping 8.8.8.8

Now let's look for logs with the following labels:

{job="fluentbit", container_name="debug"}

image

Resources: