December 10, 2025·4 min read

Over the past few weeks, we have been researching how to use Elastic Cloud's Fleet Agent across Kubernetes clusters to monitor containerized workloads. This post covers how we made container logs visible in Kibana, why some setups silently fail, and what it takes to fix them.

The Goal

We wanted to ship logs from every Kubernetes container into Elastic Cloud using the Fleet-managed Elastic Agent. The goal was simple:

1. Deploy a DaemonSet that runs one Elastic Agent per node
2. Mount /var/log/pods and /var/log/containers
3. Point it to our Fleet Server with an enrollment token
4. Watch logs roll into Kibana

Symptom: No Container Logs

Our deployment looked good on paper: the agent was enrolled, metrics were visible, and /var/log/pods was populated inside the container. But no container logs appeared in Kibana.

The Investigation

1. Reviewed the DaemonSet YAML

Our DaemonSet had mounts for /var/log/pods and /var/lib/containerd, but was missing /var/log/containers — which turned out to be the key input path for the logs.

# Before the fix
volumeMounts:
  - name: pods
    mountPath: /var/log/pods
    readOnly: true
  - name: containerd
    mountPath: /var/lib/containerd
    readOnly: true

volumes:
  - name: pods
    hostPath:
      path: /var/log/pods
  - name: containerd
    hostPath:
      path: /var/lib/containerd

2. Inspected Runtime Mounts

We used this command to check which paths were actually mounted inside the Elastic Agent pod:

kubectl -n kube-system exec -it elastic-agent-XXXX -- mount | grep -E '/var/log|/var/lib/containerd'

The expected /var/log/containers path was not present — meaning the agent had no access to the symlinked container logs.

3. Checked Fleet Integration Configuration

In the Fleet UI (under the "Kubernetes Logs" integration), the log path was set to:

/var/log/containers/*.log

If this path doesn't exist inside the container, logs won't flow. Fleet doesn't surface errors for missing log paths — it just silently fails.

The Fix

We added a hostPath volume for /var/log/containers to the DaemonSet and mounted it inside the container.

# After the fix
volumes:
  - name: containers
    hostPath:
      path: /var/log/containers

volumeMounts:
  - name: containers
    mountPath: /var/log/containers
    readOnly: true

After applying this change, the agent immediately started reading symlinked container logs, and the data appeared in Kibana.

Lessons Learned

You can inspect what the agent is trying to harvest using:

./elastic-agent inspect components --show-config

Bonus: Persisting Agent Identity Across Restarts

We noticed that after restarting Elastic Agent pods, the old agent would show up as "offline" in Fleet, while a new one appeared as "online." This happened because each pod restart generated a new agent.id, leaving duplicates in Fleet.

To solve this, we mounted the Elastic Agent's internal state directory to the node's filesystem using a hostPath. This ensures that when a pod restarts on the same node, it reuses the same identity.

volumes:
  - name: agent-state
    hostPath:
      path: /var/lib/elastic-agent
      type: DirectoryOrCreate

volumeMounts:
  - name: agent-state
    mountPath: /usr/share/elastic-agent/state

This change stopped the agent duplication and made our Fleet inventory much cleaner.

Gotcha: x509 Certificate Errors from Zscaler

One critical issue we hit during enrollment was related to TLS inspection by Zscaler. Elastic Agent failed to enroll with this error:

x509: certificate signed by unknown authority

Although the connection and token looked correct, the underlying HTTPS call to the Fleet Server was being intercepted by a Zscaler proxy. To resolve this, we needed to install the Zscaler root certificate inside the agent pods.

Solution

1. Create a ConfigMap containing the Zscaler certificate:

apiVersion: v1
kind: ConfigMap
metadata:
  name: zscaler-ca-cert
  namespace: kube-system
data:
  zscaler.crt: |
    -----BEGIN CERTIFICATE-----
    [REDACTED CERT CONTENT]
    -----END CERTIFICATE-----

2. Mount the certificate path in the pod:

volumeMounts:
  - name: zscaler-ca
    mountPath: /etc/ssl/certs

volumes:
  - name: zscaler-ca
    configMap:
      name: zscaler-ca-cert

Once this was in place, the agent enrolled successfully. TLS errors went away immediately.

Debugging Tips

# Check for logs inside container
find /var/log/pods -type f -name "*.log"

# Validate filebeat component
ps aux | grep filebeat
./elastic-agent inspect components --show-config | grep -A 20 filestream

# Verify in Kibana
data_stream.dataset : *container*

Final Thoughts

Elastic's Fleet integration is powerful, but sensitive to configuration. To avoid surprises:

1. Triple-check your volume mounts — missing paths fail silently
2. Match the Fleet integration's expected log paths — /var/log/containers, not just /var/log/pods
3. Persist the agent's state so you don't get stale "offline" entries
4. If you're behind Zscaler or a TLS proxy, mount your root cert chain

This setup is now running smoothly in the test lab, and container logs across all nodes are flowing into Kibana as expected.

Reference: Sample Helm Chart (Sanitized)

values.yaml (excerpt)

image:
  repository: docker.elastic.co/beats/elastic-agent
  tag: 8.16.6

zscalerCA:
  enabled: true
  fileName: zscaler.crt
  name: zscaler-ca

daemonset.yaml (excerpt)

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: elastic-agent
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: elastic-agent
  template:
    metadata:
      labels:
        app: elastic-agent
    spec:
      serviceAccountName: elastic-agent
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: elastic-agent
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          env:
            - name: FLEET_ENROLL
              value: "1"
            - name: FLEET_URL
              value: "<redacted>"
            - name: FLEET_ENROLLMENT_TOKEN
              value: "<redacted>"
          volumeMounts:
            - name: pods
              mountPath: /var/log/pods
              readOnly: true
            - name: containerd
              mountPath: /var/lib/containerd
              readOnly: true
            - name: containers
              mountPath: /var/log/containers
              readOnly: true
            - name: agent-data
              mountPath: /usr/share/elastic-agent/state
            - name: zscaler-ca
              mountPath: /etc/ssl/certs
      volumes:
        - name: pods
          hostPath:
            path: /var/log/pods
        - name: containerd
          hostPath:
            path: /var/lib/containerd
        - name: containers
          hostPath:
            path: /var/log/containers
        - name: agent-data
          hostPath:
            path: /var/lib/elastic-agent
            type: DirectoryOrCreate
        - name: zscaler-ca
          configMap:
            name: zscaler-ca-cert

Tags:

Enjoyed this post? Give it a clap!