kudurru on Kubernetes with Helm

Two charts (docs/architecture.md §5.1, §5.2, §5.7). Minimum Kubernetes 1.30.

• kudurru-crds: the kudurru.io CRDs, separate so helm uninstall never deletes them (generated by make manifests from api/v1alpha1).
• kudurru: the server. Every value maps onto a settings key of the binary (kudurru config schema), a KUDURRU_* variable or a Kubernetes field; deploy/chart_test.go renders every values permutation, loads each rendered kudurru.yaml through the binary's own config loader and validates the manifests with kubeconform (make helm-check).

Install

kubectl create namespace kudurru-system
kubectl label namespace kudurru-system pod-security.kubernetes.io/enforce=restricted
helm install kudurru-crds oci://ghcr.io/robertherbaugh/charts/kudurru-crds --version 0.1.0
helm install kudurru oci://ghcr.io/robertherbaugh/charts/kudurru --version 0.1.0 \
  --namespace kudurru-system

The default is mode: kubernetes: two replicas of a Deployment storing objects as CRDs (kube driver), a Lease-elected controller, one mixed-protocol LoadBalancer Service on :53 with externalTrafficPolicy: Local (client addresses are the product), a PDB, topology spread, dnsPolicy: Default and the Restricted Pod Security Standard (uid 65532, read-only root, all capabilities dropped except NET_BIND_SERVICE, net.ipv4.ip_unprivileged_port_start=0 so the non-root process binds :53).

First login

The chart generates three Secrets once and keeps them on uninstall (helm.sh/resource-policy: keep): <release>-encryption-key, <release>-admin and, with edge.enabled, <release>-proxy-protocol. The release notes print the commands; in short:

kubectl -n kudurru-system get secret kudurru-admin -o jsonpath='{.data.password}' | base64 -d; echo
kubectl -n kudurru-system port-forward svc/kudurru-api 8080:8080
# open https://localhost:8080/ and sign in as admin

The admin Secret is read only while no user exists; the UI asks for a new password at the first login. Point auth.bootstrapAdminSecret and auth.encryptionKeySecret at your own Secrets (for example from external-secrets) to keep secret material out of Helm entirely.

Shapes

mode=standalone

store.driver: sqlite, a StatefulSet with one replica and a volumeClaimTemplate for /var/lib/kudurru (persistence.standalone). No ClusterRole, no ServiceAccount token, no CRDs: objects come from objects: in the values (written into the node file as file-managed objects) and from the UI.

Persistence in mode=kubernetes

querylog.db and local.db live in /var/lib/kudurru. By default that is an emptyDir (it survives container restarts and node reboots while the pod stays). persistence.querylog.enabled or persistence.local.enabled switch the Deployment to a StatefulSet so every replica gets its own claim (SQLite WAL files must never be shared or put on RWX/NFS). A hostNetwork DaemonSet cannot have per-pod claims; use extraVolumes with a hostPath there.

hostNetwork

hostNetwork.enabled runs a DaemonSet in the host network namespace, where the namespaced net.ipv4.ip_unprivileged_port_start sysctl is not available (Kubernetes rejects net.* sysctls on hostNetwork pods) and Kubernetes gives a non-root process no ambient capabilities. The chart therefore runs ghcr.io/robertherbaugh/kudurru:<version>-hostnetwork (hostNetwork.imageTagSuffix): the same binary with the file capability cap_net_bind_service=+ep (images/hostnetwork/Dockerfile). The container keeps the Restricted security context: uid 65532, all capabilities dropped except NET_BIND_SERVICE, allowPrivilegeEscalation: false. NET_BIND_SERVICE must stay in securityContext.capabilities.add: the kernel refuses to start a binary whose file capability is outside the bounding set, so the chart fails the render without it. Host networking is outside the Baseline level; label the namespace pod-security.kubernetes.io/enforce=privileged. With your own image set hostNetwork.imageTagSuffix: "" and either give the binary the capability or set net.ipv4.ip_unprivileged_port_start=53 on the hosts.

Edge (dnsdist)

With edge.enabled the dns Service selects the edge pods; kudurru listens on edge.backendPort (5353) and accepts PROXYv2 only from edge.proxyProtocolFrom with the TLV 0xE0 secret from <release>-proxy-protocol (D72). The CIDR defaults to the RFC 1918 and ULA ranges; narrow it to your pod CIDR. The supervisor reads the headless Service's EndpointSlices (--backends-from endpointslice:<ns>/<release>-headless) and exposes readiness and /metrics on :8084.

The dnsdist container runs as uid 953 under the Restricted standard, and binds :53 through the pod's ip_unprivileged_port_start sysctl. Two cases need more than Kubernetes gives a non-root process, because added capabilities are only effective for root: edge.ebpf.enabled (creating the BPF maps needs CAP_BPF/CAP_SYS_ADMIN) and dnsdist on the host network (edge.mode=daemonset, vip.provider=keepalived, or the sidecar of a hostNetwork.enabled pod), where the sysctl is not available. There the container starts as root with group 953, allowPrivilegeEscalation: false, a read-only root and every capability dropped except NET_BIND_SERVICE, SETUID, SETGID and KILL (plus BPF and SYS_ADMIN for eBPF); the supervisor passes --dnsdist-user=953:953, so dnsdist binds its sockets and creates the maps as root, then switches to 953 and keeps only CAP_SYS_ADMIN (eBPF) or nothing. make edge-check runs both cases in the image. Such a namespace needs the privileged level (SYS_ADMIN and host networking are outside Baseline).

vip.provider=keepalived runs dnsdist and keepalived in one hostNetwork DaemonSet on the nodes labelled kudurru.io/vip=true. keepalived needs NET_ADMIN, NET_RAW and NET_BROADCAST as root: label the namespace pod-security.kubernetes.io/enforce=privileged.

Encrypted transports

service.dot|doh|doq|doh3.enabled add the ports to the dns Service. Without the edge kudurru terminates TLS with the kubernetes.io/tls Secret named by tls.secretName; with the edge dnsdist terminates it with edge.tlsSecretName.

Configuration

config.node (node keys: listeners, store, log, tsigKeys, notify) and config.settings (cluster keys: upstreams, cache, blocking, querylog, ratelimit) are YAML in the format of kudurru.yaml (docs/architecture.md §6.2):

config:
  node:
    log: { level: info, format: json }
  settings:
    upstreams:
      groups:
        default: { servers: ["tls://1.1.1.1", "tls://9.9.9.9"], strategy: parallel_best, timeout: 2s }
      conditional: { "corp.example.": internal }
    blocking: { defaultResponse: nxdomain }

They render into two ConfigMaps (D25): <release>-node is the file serve --config reads and its checksum annotation rolls the pods; <release>-settings holds the cluster keys and is mounted at /etc/kudurru/settings for hot reload. Today the binary reads only the node file, so the chart also merges the settings into it and a settings change rolls the pods; once the settings file is read and watched (docs/followups.md) the merge goes away and settings changes apply without a restart.

Seed objects with objects: (post-install hook resources in mode=kubernetes: created once, then owned by kubectl and the UI).

Monitoring

The dashboard has the §11.6 rows (tiles, traffic, latency heatmap, blocking, cache, upstreams, zones, pipeline and query log, cluster and store, dnsdist, VIP); every query uses a metric the binary registers today (the deploy tests enforce it), including the serial-divergence panel and KudurruSerialDivergence. The edge and VIP panels and the KudurruEdgeBackendDown and KudurruVIPNoMaster alerts read the supervisors' metrics listeners (:8084, :8085), which the <release>-edge PodMonitor scrapes (a node label names the node of hostNetwork pods).

Network policy

networkPolicy.enabled restricts ingress to the kudurru pods: :5353 only from edge pods (plus edge.proxyProtocolFrom as ipBlocks in the hostNetwork daemonset mode), the internal listener :8082 only from kudurru and edge pods, the API from networkPolicy.apiFrom (default: everyone), metrics from networkPolicy.metricsFrom, probes on :8081, and with blockPage.enabled the block page ports from everyone. Egress is not restricted.

Block page

blockPage.enabled with blockPage.ipv4 (or ipv6) serves the block page on the kudurru pods behind a <release>-blockpage Service (LoadBalancer by default, ports 80 and optionally 443) and sets blockPage.enabled/ipv4/ipv6 in the settings ConfigMap; pick the blockPage response with config.settings.blocking.defaultResponse or per Policy. Setup, the HTTPS caveats and the edge and caching notes: blocking-page.md.

RBAC

The kudurru ServiceAccount gets (D77, §5.3): create/patch/delete/get/list/ watch on the five kudurru.io kinds, update/patch on their status, read-only dnsendpoints, services, namespaces and endpointslices, create on events, tokenreviews and subjectaccessreviews; in the release namespace only, leases (get/create/update), secrets (get/list/watch/create/update/patch/delete: token revocation and user or RoleBinding deletion delete Secrets) and configmaps (get/list/watch/create/update). rbac.aggregateToDefaultRoles adds <release>-viewer|editor|admin aggregated into view|edit|admin and <release>-auditor (aggregated nowhere: it reads the query log). They are generated from internal/rbac/permissions.yaml by make rbac-manifests into templates/rbac-aggregated.yaml.

Values

See charts/kudurru/values.yaml; every value carries a comment. Values that the chart added beyond §5.2: tls.secretName, edge.backendPort, edge.proxyProtocolFrom, edge.tlsSecretName, edge.existingSecret, edge.dnsdistVersion, vip.metallbPool, vip.keepalived.existingSecret, vip.keepalived.keepalivedVersion, auth.encryptionKeyPreviousSecret, auth.bootstrapToken, podMonitor, probes, lameduckSeconds, clusterDomain (default cluster.local), persistence.standalone.enabled, extraEnv, extraVolumes, extraVolumeMounts, blockPage. auth.oidc.providers is a list of OpenID Providers rendered into the settings key auth.identityProviders (the §5.2 outline's single-issuer auth.oidc became a list). webhook from the §5.2 outline is not in the chart: the validating webhook does not exist in the binary.

OIDC usernames: without claimMappings.username.claim a provider's username is the first of preferred_username, upn, email (only when email_verified is true) and sub. Some identity providers let users edit preferred_username, so a role binding with a User subject could be claimed by another account. Bind groups instead, or set claimMappings.username.claim: sub (with a prefix) on such providers.