docs(k8s): add pktvisor sidecar deployment example (#743)

k8s/README.md

@@ -0,0 +1,167 @@
+# pktvisor on Kubernetes (sidecar)
+
+Run `pktvisord` as a **sidecar** in a pod to observe that pod's network traffic
+(`eth0`) and expose Prometheus metrics on `:10853`.
+
+```
++------------------- Pod (shared network namespace) -------------------+
+|                                                                      |
+|   [ app ]      [ traffic-gen (optional) ]      [ pktvisord ]         |
+|    nginx        wget loop -> external URL        sniffs eth0         |
+|                                                  :10853/metrics      |
++----------------------------------------------------------------------+
+                                 ^
+                                 |  Prometheus scrapes via pod annotations
+```
+
+Containers in a pod share one network namespace, so the `pktvisord` sidecar sees
+the same `eth0` as your application container.
+
+## Prerequisites
+
+- A Kubernetes cluster and `kubectl`.
+- The target namespace must allow pods to **add the `NET_RAW` and `NET_ADMIN`
+  capabilities** (pktvisord captures in promiscuous mode — see the security note).
+  Under the Pod Security Standards only the `privileged` level permits adding
+  these (or an exemption / custom admission policy that allows them); both
+  `baseline` and `restricted` reject them — `baseline` only allows adding
+  `NET_BIND_SERVICE`. This is about the namespace PSS *level*; the container
+  itself is **not** run as `privileged: true` — it only adds those two
+  capabilities pktvisord needs to capture packets.
+- Prometheus (or Grafana Agent) configured to scrape pods by annotation — see
+  [Prometheus scrape configuration](#prometheus-scrape-configuration).
+- The example captures `eth0`, the pod's primary interface on most CNIs. If your
+  pods use a different name, change `eth0` in the manifest args (or use `auto` to
+  let pktvisord pick the busiest interface). Find the name with
+  `kubectl exec <pod> -c app -- ls /sys/class/net` (works without `iproute2`,
+  which the minimal demo images don't ship).
+- The manifest's args are **IPv4** — the default `status.podIP` family, including
+  on dual-stack clusters. On an **IPv6-only or IPv6-primary** cluster, change the
+  sidecar args to bind IPv6 and use a `/128` host spec: `-l ::` (in place of
+  `-l 0.0.0.0`) and `-H $(POD_IP)/128` (in place of `/32`).
+
+## Deploy
+
+```shell
+# from the repo root (or use the bare filename from inside k8s/)
+kubectl apply -f k8s/pktvisor-sidecar.yaml
+```
+
+Creates a `pktvisor-demo` Deployment with three containers: your app (`nginx`
+placeholder), an optional traffic generator, and the `pktvisord` sidecar.
+
+## Verify
+
+```shell
+kubectl rollout status deploy/pktvisor-demo
+kubectl port-forward deploy/pktvisor-demo 10853:10853
+# in another terminal:
+curl -s localhost:10853/metrics | grep -E '^(dns|packets)_' | head
+```
+
+You should see non-zero `packets_*` and `dns_*` series (the traffic generator
+drives them) — a populated `/metrics` is the "it's working" signal. pktvisord
+aggregates over ~60s windows, so if the first scrape looks sparse, wait a few
+seconds and retry. Note: the
+published image redirects pktvisord's own stdout to a file, so
+`kubectl logs deploy/pktvisor-demo -c pktvisord` is usually **empty** — rely on
+`/metrics`, not logs. (If the sidecar `CrashLoopBackOff`s instead, re-check the
+`NET_RAW`/`NET_ADMIN` capabilities and the `-H .../32` CIDR.)
+
+## Generating observable traffic
+
+pktvisord watches **`eth0`**, so it sees the pod's **ingress/egress** — not
+loopback. The optional `traffic-gen` container produces egress by `wget`-ing an
+external URL (`TARGET_URL`, default `http://example.com`). To generate observable
+**ingress**, drive the app from *another* pod or through a Service so the request
+enters over `eth0` — e.g.
+`kubectl run client --rm -it --image=busybox --restart=Never -- wget -qO- <pod-ip>:80`.
+(`kubectl port-forward` reaches the app over the pod's *loopback*, so that
+traffic does **not** appear on `eth0`.) Repoint with `TARGET_URL`/`INTERVAL`, or
+delete the `traffic-gen` container for real workloads.
+
+No internet egress? If only *external* egress is blocked but in-cluster traffic
+is allowed, the generator's DNS lookups still reach cluster DNS over `eth0`, so
+`dns_*` populates; for richer `packets_*`, point `TARGET_URL` at an in-cluster
+Service. Under a **default-deny `NetworkPolicy`**, egress to kube-dns is blocked
+too — add an egress rule allowing DNS (UDP/TCP `53` to the kube-dns/CoreDNS pods),
+otherwise name resolution itself fails and the metrics stay empty.
+
+## Prometheus scrape configuration
+
+The pod carries the standard annotations:
+
+```yaml
+prometheus.io/scrape: "true"
+prometheus.io/port: "10853"
+prometheus.io/path: "/metrics"
+```
+
+A vanilla Prometheus discovers it with a pod job (no Prometheus Operator
+required):
+
+```yaml
+scrape_configs:
+  - job_name: kubernetes-pods
+    kubernetes_sd_configs:
+      - role: pod
+    relabel_configs:
+      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
+        action: keep
+        regex: "true"
+      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
+        action: replace
+        target_label: __metrics_path__
+        regex: (.+)
+      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
+        action: replace
+        # first group matches a bracketed IPv6 literal or an IPv4/host (no colons)
+        regex: (\[.+\]|[^:]+)(?::\d+)?;(\d+)
+        replacement: $1:$2
+        target_label: __address__
+      - source_labels: [__meta_kubernetes_namespace]
+        target_label: namespace
+      - source_labels: [__meta_kubernetes_pod_name]
+        target_label: pod
+```
+
+`role: pod` generates one target per declared container port, so a
+multi-container pod (here `app` + `pktvisord`) is discovered as several targets
+that these rules all rewrite to the same `POD_IP:10853`. Prometheus deduplicates
+targets whose final label sets are identical, so the endpoint is still scraped
+only **once**. Don't add a container-name/port `keep` to this shared job to
+"fix" the duplication — it would drop every other annotated pod; if you want
+explicit selection, do it inside a pktvisor-only scrape job.
+
+> Using the Prometheus Operator (kube-prometheus-stack)? Expose the sidecar with
+> a `Service` on port `10853` and create a `ServiceMonitor` selecting it, instead
+> of using annotations.
+
+## Grafana dashboard
+
+Import the community dashboard **ID 14221**, or the JSON at
+`../centralized_collection/prometheus/grafana-dashboard-prometheus.json`.
+
+## Use with your own workload
+
+Add the sidecar to any existing pod spec:
+
+1. Add the three `prometheus.io/*` pod annotations. For a Deployment/StatefulSet
+   they go on the **pod template** (`spec.template.metadata.annotations`), not the
+   top-level object metadata; for a bare Pod they go on `metadata.annotations`.
+2. Add the `pktvisord` container — including the `POD_IP` downward-API env, the
+   `NET_RAW` and `NET_ADMIN` capabilities, and `args: ["pktvisord", "-l", "0.0.0.0", "-H", "$(POD_IP)/32", "eth0"]`.
+3. Remove the demo `app` and `traffic-gen` containers.
+
+## Security note
+
+The sidecar adds the `NET_RAW` and `NET_ADMIN` capabilities (it is **not** run as
+a `privileged` container): `NET_RAW` opens the AF_PACKET/raw capture socket, and
+`NET_ADMIN` is required because pktvisord opens the interface in **promiscuous
+mode** — matching the project's documented bare-metal `setcap
+cap_net_raw,cap_net_admin`. Under the Pod Security Standards, both the `baseline`
+and `restricted` levels reject adding these capabilities (baseline only allows
+adding `NET_BIND_SERVICE`), so run this in a namespace at the `privileged` level
+or with an exemption / custom policy that permits them. Note `-l 0.0.0.0` exposes
+`/metrics` on the pod IP (no auth) so Prometheus can scrape it — restrict with a
+`NetworkPolicy` if needed.

k8s/pktvisor-sidecar.yaml

@@ -0,0 +1,80 @@
+# pktvisor as a sidecar: observes the pod's eth0 NIC traffic and exposes
+# Prometheus metrics on :10853. See README.md in this directory for usage,
+# verification, the Prometheus scrape config, and how to adapt to your workload.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: pktvisor-demo
+  labels:
+    app: pktvisor-demo
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: pktvisor-demo
+  template:
+    metadata:
+      labels:
+        app: pktvisor-demo
+      annotations:
+        prometheus.io/scrape: "true"
+        prometheus.io/port: "10853"
+        prometheus.io/path: "/metrics"
+    spec:
+      containers:
+        # ---- your workload: replace this container with your own ----
+        - name: app
+          image: nginx:stable-alpine
+          ports:
+            - containerPort: 80
+
+        # ---- OPTIONAL traffic generator (delete for real workloads) ----
+        # wget's an EXTERNAL URL so the request leaves the pod over eth0, which
+        # pktvisord observes (loopback traffic would NOT be seen). Requires
+        # cluster egress; repoint TARGET_URL or remove this container as needed.
+        - name: traffic-gen
+          image: busybox:stable
+          env:
+            - name: TARGET_URL
+              value: "http://example.com"
+            - name: INTERVAL
+              value: "5"
+          command: ["/bin/sh", "-c"]
+          args:
+            - 'while true; do wget -q -O /dev/null "$TARGET_URL" || true; sleep "$INTERVAL"; done'
+
+        # ---- pktvisord sidecar: capture on eth0, serve /metrics on :10853 ----
+        - name: pktvisord
+          image: netboxlabs/pktvisor
+          # The image ENTRYPOINT (/entry-cp.sh) selects the binary from the first
+          # arg and forwards the rest to pktvisord (adding crashpad/geo defaults).
+          # Keep the leading "pktvisord".
+          #   -l 0.0.0.0   : bind the metrics server on all IPv4 interfaces; the
+          #                  default is localhost, unreachable by Prometheus at POD_IP.
+          #   -H .../32    : single-host CIDR so pktvisord attributes in/out direction
+          #                  (a bare IP throws "invalid CIDR").
+          #   eth0         : run the default net+dhcp+dns+pcap policy on the pod NIC.
+          # These values are IPv4 (the default status.podIP family). On an IPv6-only
+          # or IPv6-primary cluster, use -l :: and -H $(POD_IP)/128 instead.
+          args: ["pktvisord", "-l", "0.0.0.0", "-H", "$(POD_IP)/32", "eth0"]
+          env:
+            - name: POD_IP
+              valueFrom:
+                fieldRef:
+                  fieldPath: status.podIP
+          ports:
+            - name: metrics
+              containerPort: 10853
+          securityContext:
+            capabilities:
+              # NET_RAW opens the AF_PACKET/raw capture socket; NET_ADMIN is needed
+              # because pktvisord captures in promiscuous mode (matches the project's
+              # documented `setcap cap_net_raw,cap_net_admin`). Still NOT privileged.
+              add: ["NET_RAW", "NET_ADMIN"]
+          resources:
+            requests:
+              cpu: "50m"
+              memory: "128Mi"
+            limits:
+              cpu: "500m"
+              memory: "512Mi"