eBPF for Kubernetes Security: 5 Proven Tests on k3s

eBPF lets you run small programs inside the Linux kernel, attached to events such as system calls, without changing kernel source or loading a module. On a Kubernetes node that means you can watch every process start, file open or connection from every pod in one place. The first version of this article explained the idea with marketing words and one "conceptual" BCC script. This rewrite runs that script for real on a Kubernetes node, shows what it misses, and tests the safety claims (the verifier, root access, overhead) on an actual kernel.

Lab: a Debian 12 KVM virtual machine (kernel 6.1.0-47-amd64, 2 vCPU) with k3s v1.37.2+k3s1, BCC 0.26.0 (python3-bpfcc), bpftrace v0.17.0 and bpftool 7.1.0, run on 7 November 2026. One test pod, shop/web, runs nginx:1.27-alpine. Cilium, Tetragon and Falco were not installed; what this article says about them comes from their documentation.

Check that the node kernel is ready for eBPF

Before any eBPF tool runs on Kubernetes nodes, check the kernel config and BTF type information. BTF in /sys/kernel/btf/vmlinux is what lets CO-RE ("compile once, run everywhere") tools run on many kernel versions without kernel headers:

$ uname -r; grep -E "^CONFIG_(BPF|BPF_SYSCALL|BPF_JIT|DEBUG_INFO_BTF|BPF_LSM)=" /boot/config-$(uname -r); ls -l /sys/kernel/btf/vmlinux
6.1.0-47-amd64
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT=y
CONFIG_BPF_LSM=y
CONFIG_DEBUG_INFO_BTF=y
-r--r--r-- 1 root root 4350808 Nov  7 05:56 /sys/kernel/btf/vmlinux

$ sysctl kernel.unprivileged_bpf_disabled net.core.bpf_jit_enable
kernel.unprivileged_bpf_disabled = 2
net.core.bpf_jit_enable = 1

The JIT is on, so programs run as native code. unprivileged_bpf_disabled = 2 means normal users cannot load eBPF programs at all, which matters for the safety section below. If your nodes lack /sys/kernel/btf/vmlinux, BTF-based tools will not work there; check this on every node image you use.

The old BCC script on a Kubernetes node

This is the script from the original post, unchanged. It attaches a kprobe to execve and prints the current process name with bpf_trace_printk. Run on the idle VM while a shell started id, uname and date:

$ timeout -s INT 10 python3 -u /root/old.py
Tracing execve()... Press Ctrl+C to stop.
b'           <...>-2540    [000] ...21   105.419818: bpf_trace_printk: Process bash is executing a new program'
b''
b'           <...>-2541    [000] ...21   105.422110: bpf_trace_printk: Process bash is executing a new program'
b''
b'           <...>-2542    [000] ...21   105.423535: bpf_trace_printk: Process bash is executing a new program'
b''
Traceback (most recent call last):
  File "/root/old.py", line 16, in <module>
    b.trace_print()

It loads and runs, but three problems show up at once. It reports bash, not id or date: the probe fires before execve replaces the process, so comm is still the caller. It never prints the program being started. And Ctrl+C ends in a Python traceback, because nothing catches KeyboardInterrupt.

On the k3s node it gets worse. During 14 seconds, while someone ran kubectl exec into shop/web, the script printed 425 lines. Counted by name:

425
    258 Process k3s-server is
     43 Process portmap is
     35 Process containerd-shim is
     30 Process containerd is
     20 Process entry is
      7 Process runc:[2:INIT] is
      7 Process runc is
      7 Process (spawn) is
      5 Process flannel is
      5 Process bridge is
      3 Process sh is
      2 Process kubectl is

Most events come from k3s itself and the CNI plugins. The three sh lines are the pod's shell, but nothing says which pod, namespace or container they belong to, and the commands it ran are not shown. This is the gap that Kubernetes-aware eBPF tools fill.

Map eBPF events to pods with the cgroup path

Every container runs in its own cgroup, and the kubelet puts the pod UID into the cgroup name. bpftrace can print the cgroup path of the process next to the program name from the syscall arguments. Filtering the output on the UID of shop/web (dashes replaced by underscores) leaves exactly the kubectl exec session:

$ timeout -s INT 16 bpftrace -q -e "tracepoint:syscalls:sys_enter_execve { printf(\"%-6d %-8s %-14s %s\n\", pid, comm, str(args->filename), cgroup_path(cgroup)); }" | grep "pod3bf4166f_af36_44bf_953b_a0bd31cfb15a"
5671   runc:[2:INIT] /bin/sh        unified:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bf4166f_af36_44bf_953b_a0bd31cfb15a.slice/cri-containerd-1b9ce3905b0d39eedb6b4b7a5af767a5f5b72128d689fa69041e370d30344100.scope
5677   sh       /usr/bin/id    unified:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bf4166f_af36_44bf_953b_a0bd31cfb15a.slice/cri-containerd-1b9ce3905b0d39eedb6b4b7a5af767a5f5b72128d689fa69041e370d30344100.scope
5678   sh       /bin/cat       unified:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bf4166f_af36_44bf_953b_a0bd31cfb15a.slice/cri-containerd-1b9ce3905b0d39eedb6b4b7a5af767a5f5b72128d689fa69041e370d30344100.scope
5679   sh       /usr/bin/wget  unified:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bf4166f_af36_44bf_953b_a0bd31cfb15a.slice/cri-containerd-1b9ce3905b0d39eedb6b4b7a5af767a5f5b72128d689fa69041e370d30344100.scope

$ kubectl -n shop get pod web -o jsonpath="{.metadata.uid}{\"\n\"}{.status.containerStatuses[0].containerID}{\"\n\"}"
3bf4166f-af36-44bf-953b-a0bd31cfb15a
containerd://1b9ce3905b0d39eedb6b4b7a5af767a5f5b72128d689fa69041e370d30344100
eBPF trace of execve calls inside a Kubernetes pod, matched by cgroup path
bpftrace shows the commands of the kubectl exec session and the pod's cgroup.

Now the shell, id, cat and wget appear with the pod UID and the container ID from kubectl. That is the core idea behind runtime security tools. Doing it by hand works for one investigation; in production you want a tool that keeps the pod table up to date. The Tetragon overview says it is "Kubernetes-aware", understanding "namespaces, pods and so-on", so that event detection can be configured per workload, and that it can enforce rules in the kernel as well as observe.

Is eBPF safe? The verifier, tested

The old FAQ said the verifier stops programs that crash the kernel, read forbidden memory or loop forever. The kernel verifier documentation describes a first pass that checks the control flow graph and a second pass that simulates every path. Two deliberately bad programs, loaded with BCC, were both rejected before they ran. An endless loop:

$ python3 /root/bad.py loop 2>&1 | grep -E "infinite|back-edge|loop|Exception|processed" | head -5
infinite loop detected at insn 2
processed 15 insns (limit 1000000) max_states_per_insn 0 total_states 1 peak_states 1 mark_read 1
Exception: Failed to load BPF program b'f': Invalid argument

A stack read with a random index:

$ python3 /root/bad.py oob 2>&1 | tail -6
7: (07) r1 += -16                     ; R1_w=fp-16
8: (0f) r1 += r0
value -2147483648 makes fp pointer be out of bounds

Safe in this sense does not mean harmless. A verified program run by root can still read process data, file names and network contents on the whole node. That is why loading is limited: with unprivileged_bpf_disabled = 2 a normal user gets nothing, and the tools say so:

$ su vagrant -c "bpftrace -e \"BEGIN { exit(); }\""
ERROR: bpftrace currently only supports running as the root user.

In Kubernetes this means eBPF agents, like any eBPF tool with real access, run as privileged DaemonSets. Treat their images and RBAC like any other node-level component.

Overhead of a simple eBPF probe

The old article promised "near-zero overhead". For a small counter on the execve tracepoint, the lab agrees. Starting /bin/true 3000 times took the same time with and without the probe, within the normal run-to-run noise:

$ TIMEFORMAT="no probe: %R s"; for r in 1 2; do time bash /root/loop.sh; done
no probe: 0.994 s
no probe: 0.959 s

$ TIMEFORMAT="with probe: %R s"; for r in 1 2; do time bash /root/loop.sh; done
with probe: 1.079 s
with probe: 0.995 s

The loaded program is tiny; bpftool lists it with its JIT size:

$ bpftool prog show | grep -E "^[0-9]+: tracepoint" -A2 | head -6
202: tracepoint  name sys_enter_execv  tag 6da3b410ac490c74  gpl
	loaded_at 2026-11-07T05:58:33+0000  uid 0
	xlated 216B  jited 129B  memlock 4096B  map_ids 25

The cost of eBPF grows with what you do per event. Printing every event, like the old script, or copying strings for every openat on a busy node is far more work than a counter, so measure your own agent before you roll it out.

eBPF networking on Kubernetes: what the docs say

The old article claimed Cilium "replaces iptables". The Cilium kube-proxy-free guide is more precise: it shows how to provision a cluster without kube-proxy "and to use Cilium to fully replace it", and notes that the kube-proxy replacement "depends on the socket-LB feature". This lab used k3s with its default flannel CNI, so no networking claim was tested here. The CPU savings figure from the old post had no source and was removed.

Practical checklist for eBPF and Kubernetes

  • Check CONFIG_DEBUG_INFO_BTF=y and /sys/kernel/btf/vmlinux on every node image.
  • Use tracepoints and print the filename argument; comm alone shows the parent, as the old script did.
  • Join events to pods by cgroup path or use a Kubernetes-aware tool such as Tetragon.
  • Filter in the eBPF program and aggregate in kernel maps; avoid bpf_trace_printk outside debugging.
  • Keep kernel.unprivileged_bpf_disabled on and limit who can deploy privileged DaemonSets.

For the cluster basics used here, see Kubernetes deep dive on k3s, and for keeping events after pods are gone, Kubernetes history inspector.

Conclusion

eBPF gives a Kubernetes node a single view of what every container does, and the verifier really does reject loops and out-of-bounds reads. The old "conceptual" script shows why tooling matters: on a real node it reports the wrong process name, floods you with k3s events and has no idea which pod did what. Adding the syscall argument and the cgroup path turns the same eBPF probe into a useful trace of one pod. For production, use a maintained Kubernetes-aware agent, run it as root only where needed, and measure its overhead on your own workload.

Comments

Popular posts from this blog

How to Play Minecraft Bedrock Edition on Linux: A Comprehensive Guide for Tech Professionals

Linux desktop frozen? My 5 go-to tricks to try - before forcing a hard reboot

The Ultimate Guide: How to Set Up DXVK in Wine on Linux for Enhanced Gaming Performance