|
| 1 | +# pktvisor on Kubernetes (sidecar) |
| 2 | + |
| 3 | +Run `pktvisord` as a **sidecar** in a pod to observe that pod's network traffic |
| 4 | +(`eth0`) and expose Prometheus metrics on `:10853`. |
| 5 | + |
| 6 | +``` |
| 7 | ++------------------- Pod (shared network namespace) -------------------+ |
| 8 | +| | |
| 9 | +| [ app ] [ traffic-gen (optional) ] [ pktvisord ] | |
| 10 | +| nginx wget loop -> external URL sniffs eth0 | |
| 11 | +| :10853/metrics | |
| 12 | ++----------------------------------------------------------------------+ |
| 13 | + ^ |
| 14 | + | Prometheus scrapes via pod annotations |
| 15 | +``` |
| 16 | + |
| 17 | +Containers in a pod share one network namespace, so the `pktvisord` sidecar sees |
| 18 | +the same `eth0` as your application container. |
| 19 | + |
| 20 | +## Prerequisites |
| 21 | + |
| 22 | +- A Kubernetes cluster and `kubectl`. |
| 23 | +- The target namespace must allow pods to **add the `NET_RAW` and `NET_ADMIN` |
| 24 | + capabilities** (pktvisord captures in promiscuous mode — see the security note). |
| 25 | + Under the Pod Security Standards only the `privileged` level permits adding |
| 26 | + these (or an exemption / custom admission policy that allows them); both |
| 27 | + `baseline` and `restricted` reject them — `baseline` only allows adding |
| 28 | + `NET_BIND_SERVICE`. This is about the namespace PSS *level*; the container |
| 29 | + itself is **not** run as `privileged: true` — it only adds those two |
| 30 | + capabilities pktvisord needs to capture packets. |
| 31 | +- Prometheus (or Grafana Agent) configured to scrape pods by annotation — see |
| 32 | + [Prometheus scrape configuration](#prometheus-scrape-configuration). |
| 33 | +- The example captures `eth0`, the pod's primary interface on most CNIs. If your |
| 34 | + pods use a different name, change `eth0` in the manifest args (or use `auto` to |
| 35 | + let pktvisord pick the busiest interface). Find the name with |
| 36 | + `kubectl exec <pod> -c app -- ls /sys/class/net` (works without `iproute2`, |
| 37 | + which the minimal demo images don't ship). |
| 38 | +- The manifest's args are **IPv4** — the default `status.podIP` family, including |
| 39 | + on dual-stack clusters. On an **IPv6-only or IPv6-primary** cluster, change the |
| 40 | + sidecar args to bind IPv6 and use a `/128` host spec: `-l ::` (in place of |
| 41 | + `-l 0.0.0.0`) and `-H $(POD_IP)/128` (in place of `/32`). |
| 42 | + |
| 43 | +## Deploy |
| 44 | + |
| 45 | +```shell |
| 46 | +# from the repo root (or use the bare filename from inside k8s/) |
| 47 | +kubectl apply -f k8s/pktvisor-sidecar.yaml |
| 48 | +``` |
| 49 | + |
| 50 | +Creates a `pktvisor-demo` Deployment with three containers: your app (`nginx` |
| 51 | +placeholder), an optional traffic generator, and the `pktvisord` sidecar. |
| 52 | + |
| 53 | +## Verify |
| 54 | + |
| 55 | +```shell |
| 56 | +kubectl rollout status deploy/pktvisor-demo |
| 57 | +kubectl port-forward deploy/pktvisor-demo 10853:10853 |
| 58 | +# in another terminal: |
| 59 | +curl -s localhost:10853/metrics | grep -E '^(dns|packets)_' | head |
| 60 | +``` |
| 61 | + |
| 62 | +You should see non-zero `packets_*` and `dns_*` series (the traffic generator |
| 63 | +drives them) — a populated `/metrics` is the "it's working" signal. pktvisord |
| 64 | +aggregates over ~60s windows, so if the first scrape looks sparse, wait a few |
| 65 | +seconds and retry. Note: the |
| 66 | +published image redirects pktvisord's own stdout to a file, so |
| 67 | +`kubectl logs deploy/pktvisor-demo -c pktvisord` is usually **empty** — rely on |
| 68 | +`/metrics`, not logs. (If the sidecar `CrashLoopBackOff`s instead, re-check the |
| 69 | +`NET_RAW`/`NET_ADMIN` capabilities and the `-H .../32` CIDR.) |
| 70 | + |
| 71 | +## Generating observable traffic |
| 72 | + |
| 73 | +pktvisord watches **`eth0`**, so it sees the pod's **ingress/egress** — not |
| 74 | +loopback. The optional `traffic-gen` container produces egress by `wget`-ing an |
| 75 | +external URL (`TARGET_URL`, default `http://example.com`). To generate observable |
| 76 | +**ingress**, drive the app from *another* pod or through a Service so the request |
| 77 | +enters over `eth0` — e.g. |
| 78 | +`kubectl run client --rm -it --image=busybox --restart=Never -- wget -qO- <pod-ip>:80`. |
| 79 | +(`kubectl port-forward` reaches the app over the pod's *loopback*, so that |
| 80 | +traffic does **not** appear on `eth0`.) Repoint with `TARGET_URL`/`INTERVAL`, or |
| 81 | +delete the `traffic-gen` container for real workloads. |
| 82 | + |
| 83 | +No internet egress? If only *external* egress is blocked but in-cluster traffic |
| 84 | +is allowed, the generator's DNS lookups still reach cluster DNS over `eth0`, so |
| 85 | +`dns_*` populates; for richer `packets_*`, point `TARGET_URL` at an in-cluster |
| 86 | +Service. Under a **default-deny `NetworkPolicy`**, egress to kube-dns is blocked |
| 87 | +too — add an egress rule allowing DNS (UDP/TCP `53` to the kube-dns/CoreDNS pods), |
| 88 | +otherwise name resolution itself fails and the metrics stay empty. |
| 89 | + |
| 90 | +## Prometheus scrape configuration |
| 91 | + |
| 92 | +The pod carries the standard annotations: |
| 93 | + |
| 94 | +```yaml |
| 95 | +prometheus.io/scrape: "true" |
| 96 | +prometheus.io/port: "10853" |
| 97 | +prometheus.io/path: "/metrics" |
| 98 | +``` |
| 99 | +
|
| 100 | +A vanilla Prometheus discovers it with a pod job (no Prometheus Operator |
| 101 | +required): |
| 102 | +
|
| 103 | +```yaml |
| 104 | +scrape_configs: |
| 105 | + - job_name: kubernetes-pods |
| 106 | + kubernetes_sd_configs: |
| 107 | + - role: pod |
| 108 | + relabel_configs: |
| 109 | + - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] |
| 110 | + action: keep |
| 111 | + regex: "true" |
| 112 | + - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] |
| 113 | + action: replace |
| 114 | + target_label: __metrics_path__ |
| 115 | + regex: (.+) |
| 116 | + - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] |
| 117 | + action: replace |
| 118 | + # first group matches a bracketed IPv6 literal or an IPv4/host (no colons) |
| 119 | + regex: (\[.+\]|[^:]+)(?::\d+)?;(\d+) |
| 120 | + replacement: $1:$2 |
| 121 | + target_label: __address__ |
| 122 | + - source_labels: [__meta_kubernetes_namespace] |
| 123 | + target_label: namespace |
| 124 | + - source_labels: [__meta_kubernetes_pod_name] |
| 125 | + target_label: pod |
| 126 | +``` |
| 127 | +
|
| 128 | +`role: pod` generates one target per declared container port, so a |
| 129 | +multi-container pod (here `app` + `pktvisord`) is discovered as several targets |
| 130 | +that these rules all rewrite to the same `POD_IP:10853`. Prometheus deduplicates |
| 131 | +targets whose final label sets are identical, so the endpoint is still scraped |
| 132 | +only **once**. Don't add a container-name/port `keep` to this shared job to |
| 133 | +"fix" the duplication — it would drop every other annotated pod; if you want |
| 134 | +explicit selection, do it inside a pktvisor-only scrape job. |
| 135 | + |
| 136 | +> Using the Prometheus Operator (kube-prometheus-stack)? Expose the sidecar with |
| 137 | +> a `Service` on port `10853` and create a `ServiceMonitor` selecting it, instead |
| 138 | +> of using annotations. |
| 139 | + |
| 140 | +## Grafana dashboard |
| 141 | + |
| 142 | +Import the community dashboard **ID 14221**, or the JSON at |
| 143 | +`../centralized_collection/prometheus/grafana-dashboard-prometheus.json`. |
| 144 | + |
| 145 | +## Use with your own workload |
| 146 | + |
| 147 | +Add the sidecar to any existing pod spec: |
| 148 | + |
| 149 | +1. Add the three `prometheus.io/*` pod annotations. For a Deployment/StatefulSet |
| 150 | + they go on the **pod template** (`spec.template.metadata.annotations`), not the |
| 151 | + top-level object metadata; for a bare Pod they go on `metadata.annotations`. |
| 152 | +2. Add the `pktvisord` container — including the `POD_IP` downward-API env, the |
| 153 | + `NET_RAW` and `NET_ADMIN` capabilities, and `args: ["pktvisord", "-l", "0.0.0.0", "-H", "$(POD_IP)/32", "eth0"]`. |
| 154 | +3. Remove the demo `app` and `traffic-gen` containers. |
| 155 | + |
| 156 | +## Security note |
| 157 | + |
| 158 | +The sidecar adds the `NET_RAW` and `NET_ADMIN` capabilities (it is **not** run as |
| 159 | +a `privileged` container): `NET_RAW` opens the AF_PACKET/raw capture socket, and |
| 160 | +`NET_ADMIN` is required because pktvisord opens the interface in **promiscuous |
| 161 | +mode** — matching the project's documented bare-metal `setcap |
| 162 | +cap_net_raw,cap_net_admin`. Under the Pod Security Standards, both the `baseline` |
| 163 | +and `restricted` levels reject adding these capabilities (baseline only allows |
| 164 | +adding `NET_BIND_SERVICE`), so run this in a namespace at the `privileged` level |
| 165 | +or with an exemption / custom policy that permits them. Note `-l 0.0.0.0` exposes |
| 166 | +`/metrics` on the pod IP (no auth) so Prometheus can scrape it — restrict with a |
| 167 | +`NetworkPolicy` if needed. |
0 commit comments