Sending logs and metrics from a Kubernetes cluster

This guide will show how you can monitor a Kubernetes (k8s) cluster using your Opstrace cluster.

Specifically, the goal is to

  • scrape k8s system metrics, and push them to the Opstrace cluster.
  • collect logs of workloads (containers) running on the k8s cluster, and send them to the Opstrace cluster.

To that end,

  • we will deploy a single Prometheus instance in the k8s cluster (as a k8s deployment): it will scrape the various k8s system metric endpoints, and then push all collected data to the remote Opstrace cluster.
  • we will deploy a Promtail instance on each node in the k8s cluster (as a k8s daemon set): it will collect all local container logs and push them into the remote Opstrace cluster.

Prerequisites

  • An Opstrace cluster.
  • A decision: for which Opstrace tenant would you like to send data?
  • An Opstrace tenant authentication token file (for the tenant of your choice). Also see concepts.

For following this guide step-by-step you will additionally need kind installed on your computer. Kind is a tool for running a local k8s cluster using Docker.

1: Create a Kubernetes cluster

kind create cluster

2: Set up kubectl to point to the kind cluster

kubectl cluster-info --context kind-kind

3: Create a k8s secret with tenant authentication token

Copy the default tenant's data API authentication token. Create a new file and name it secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: tenant-auth-token-default
stringData:
  authToken: <YOUR AUTH TOKEN HERE>

Create the secret:

kubectl apply -f secret.yaml

4: Deploy a Prometheus instance in the k8s cluster for scraping metrics

Create a file named prometheus-config.yaml with the following contents. Replace ${CLUSTER_NAME} with the name of your Opstrace cluster.

Note that the configuration snippets below assume that the Opstrace tenant you would like to send data to is called default.

kind: ConfigMap
apiVersion: v1
metadata:
  name: prometheus-config
data:
  prometheus.yml: |-
    remote_write:
    - url: https://cortex.${TENANT_NAME}.${CLUSTER_NAME}.opstrace.io/api/v1/push
      bearer_token_file: /var/run/${TENANT_NAME}-tenant/authToken

    scrape_configs:
    - job_name: 'kubernetes-pods'
      kubernetes_sd_configs:
        - role: pod

      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

      relabel_configs:

      # Always use HTTPS for the api server
      - source_labels: [__meta_kubernetes_service_label_component]
        regex: apiserver
        action: replace
        target_label: __scheme__
        replacement: https

      # Rename jobs to be <namespace>/<name, from pod name label>
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_label_name]
        action: replace
        separator: /
        target_label: job
        replacement: $1

      # Rename instances to be the pod name
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: instance

      # Include node name as a extra field
      - source_labels: [__meta_kubernetes_pod_node_name]
        target_label: node

    # This scrape config gather all nodes
    - job_name: 'kubernetes-nodes'
      kubernetes_sd_configs:
        - role: node

      tls_config:
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

      relabel_configs:
      - target_label: __scheme__
        replacement: https
      - source_labels: [__meta_kubernetes_node_label_kubernetes_io_hostname]
        target_label: instance

    # This scrape config just pulls in the default/kubernetes service
    - job_name: 'kubernetes-service'
      kubernetes_sd_configs:
        - role: endpoints

      tls_config:
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

      relabel_configs:
      - source_labels: [__meta_kubernetes_service_label_component]
        regex: apiserver
        action: keep

      - target_label: __scheme__
        replacement: https

      - source_labels: []
        target_label: job
        replacement: default/kubernetes

Submit this config map:

kubectl apply -f prometheus-config.yaml

Next up, create a file named prometheus.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
spec:
  replicas: 1
  selector:
    matchLabels:
      name: prometheus
  template:
    metadata:
      labels:
        name: prometheus
    spec:
      serviceAccountName: prometheus
      containers:
      - name: retrieval
        image: prom/prometheus:v2.21.0
        imagePullPolicy: IfNotPresent
        args:
        - --config.file=/etc/prometheus/prometheus.yml
        ports:
        - containerPort: 9090
        volumeMounts:
        - name: config-volume
          mountPath: /etc/prometheus
        - name: tenant-auth-token-default
          mountPath: /var/run/default-tenant
          readOnly: true
      volumes:
        - name: config-volume
          configMap:
            name: prometheus-config
        - name: tenant-auth-token-default
          secret:
            secretName: tenant-auth-token-default

Then start the Prometheus deployment with the following command:

kubectl apply -f prometheus.yaml

5: Deploy Promtail in the k8s cluster for collecting and pushing logs

Create a file named promtail-config.yaml with the following contents. Replace ${CLUSTER_NAME} with the name of your Opstrace cluster.

Note that the configuration snippets below assume that the Opstrace tenant you would like to send data to is called default.

apiVersion: v1
kind: ConfigMap
metadata:
  name: promtail-config
data:
  promtail.yml: |
    client:
      url: https://loki.default.#{CLUSTER_NAME}.opstrace.io/loki/api/v1/push
      bearer_token_file: /var/run/default-tenant/authToken

    scrape_configs:
    - pipeline_stages:
      - docker:
      job_name: kubernetes-pods-name
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels:
        - __meta_kubernetes_pod_label_name
        target_label: __service__
      - source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: __host__
      - action: drop
        regex: ^$
        source_labels:
        - __service__
      - action: replace
        replacement: $1
        separator: /
        source_labels:
        - __meta_kubernetes_namespace
        - __service__
        target_label: k8s_app

      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: k8s_namespace_name

      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: k8s_pod_name

      - action: replace
        source_labels:
        - __meta_kubernetes_pod_container_name
        target_label: k8s_container_name

      - replacement: /var/log/pods/*$1/*.log
        separator: /
        source_labels:
        - __meta_kubernetes_pod_uid
        - __meta_kubernetes_pod_container_name
        target_label: __path__

      - replacement: /var/log/pods/*$1/*.log
        separator: /
        source_labels:
        - __meta_kubernetes_pod_uid
        - __meta_kubernetes_pod_container_name
        target_label: __path__

    - pipeline_stages:
      - docker:
      job_name: kubernetes-pods-static
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: drop
        regex: ^$
        source_labels:
        - __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror

      - action: replace
        source_labels:
        - __meta_kubernetes_pod_label_component
        target_label: __service__

      - source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: __host__

      - action: drop
        regex: ^$
        source_labels:
        - __service__

      - action: replace
        replacement: $1
        separator: /
        source_labels:
        - __meta_kubernetes_namespace
        - __service__
        target_label: k8s_app

      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: k8s_namespace_name

      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: k8s_pod_name

      - action: replace
        source_labels:
        - __meta_kubernetes_pod_container_name
        target_label: k8s_container_name

      - replacement: /var/log/pods/*$1/*.log
        separator: /
        source_labels:
        - __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror
        - __meta_kubernetes_pod_container_name
        target_label: __path__

Submit this config map:

kubectl apply -f promtail-config.yaml

Create a file promtail.yaml with the following contents:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: promtail
spec:
  minReadySeconds: 10
  selector:
    matchLabels:
      name: promtail
  template:
    metadata:
      labels:
        name: promtail
    spec:
      containers:
      - args:
        - -config.file=/etc/promtail/promtail.yml
        env:
        - name: HOSTNAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        image: grafana/promtail:1.6.1
        imagePullPolicy: Always
        name: promtail
        readinessProbe:
          httpGet:
            path: /ready
            port: http-metrics
            scheme: HTTP
          initialDelaySeconds: 10
        ports:
        - containerPort: 80
          name: http-metrics
        securityContext:
          privileged: true
          runAsUser: 0
        volumeMounts:
        - mountPath: /etc/promtail
          name: promtail-config
        - mountPath: /var/log
          name: varlog
        - mountPath: /var/lib/docker/containers
          name: varlibdockercontainers
          readOnly: true
        - mountPath: /var/run/default-tenant
          name: tenant-auth-token-default
          readOnly: true
      serviceAccount: promtail
      tolerations:
      - effect: NoSchedule
        operator: Exists
      volumes:
      - configMap:
          name: promtail-config
        name: promtail-config
      - secret:
          secretName: tenant-auth-token-default
        name: tenant-auth-token-default
      - hostPath:
          path: /var/log
        name: varlog
      - hostPath:
          path: /var/lib/docker/containers
        name: varlibdockercontainers
  updateStrategy:
    type: RollingUpdate
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: promtail
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: promtail
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: promtail
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: promtail
subjects:
- kind: ServiceAccount
  name: promtail
  namespace: default

Start the Promtail deployment with the following command:

kubectl apply -f promtail.yaml

6: Explore

You can explore the logs and metrics sent to your Opstrace cluster using the Grafana explore view:

open https://default.${CLUSTER_NAME}.opstrace.io/grafana/explore