M3 work package: authentication and authorization wiring

Key auth. This package covers docs/architecture.md §8.1-8.8, §5.5 and decisions D21/D22, D48, D54-D57 and D71. It turns the auth spike (docs/spikes/auth-oidc.md: sealed sessions and the OIDC BFF login) into the identity layer the API server mounts:

• local users and the first-boot admin
• API tokens and service accounts
• Kubernetes ServiceAccount tokens with SubjectAccessReview
• the authenticator chain and the authorizer chain
• the /auth and /iam routes
• the break-glass admin socket
• encryption-key loading

The integrator wires it into cmd/kudurru/serve.go as described in Serve wiring.

It closes these follow-up rows:

• "roles_snapshot is frozen at login": rbac.BindingSource honours a sealed grant only while a live binding still provides it and no binding change removed it after the session's iat.
• "Export an rbac subject-matching helper" (rbac.SubjectMatches, SubjectsMatch, GrantsFor; oidc.BindingResolver uses them, and an empty name never matches).
• "Wire Registry.Register, session.Authenticator, ValidAfter, keys": done by stack.
• "Serve internal kinds (Secrets labelled kudurru.io/kind) in the kube driver".
• The InternalKinds part of the store-sqlite row.

Narrower rows replace them; see Follow-ups.

What was built

The chains

Authenticate (auth.Middleware, in this order; the first step that recognises its credential decides):

1. bearer kdr_ (tokens.Tokens): a personal token acts as its owner, and a service-account token as serviceaccount:<ns>:<name>.
2. bearer JWT routed by iss (auth.BearerJWT):
   • an exact issuer match goes to that OIDC provider (VerifyJWT);
   • the cluster's service-account issuer goes to k8s.Authenticator (JWKS);
   • any other issuer goes to k8s.Fallback(), which asks TokenReview only while discovery is failing and otherwise rejects.
3. session cookie (session.Authenticator): CrossOriginProtection and X-Kudurru-CSRF on unsafe methods; the idle timeout slides; Grants = roles snapshot and GrantsIssuedAt = iat. On a CSRF failure the session is dropped, not failed: the failure is recorded with auth.WithCSRFFailure, and the chain goes on. Routes that need a principal then answer 403 csrf (Authorize, RequireAuthenticated), logout refuses, and public routes such as login run anonymous.
4. unix peer (auth.UnixPeer): connections that carry SO_PEERCRED from uid 0, the server uid or the configured gid become LocalRoot, the builtin admin.
5. internal token (auth.InternalToken, internalapi.Credentials.ValidToken): a Node principal with the implicit grants viewer and auditor.

The chain has three outcomes:

• A request that no step recognises passes through anonymous (public routes serve it; Authorize answers 401).
• A bearer credential that no step claims is 401, never anonymous.
• A credential that a step rejects is answered by the middleware.

LoadPrincipal (users.LoadPrincipal): local users are re-read from the cache on every request. A deleted or disabled account is 401. Live groups, builtin-admin and mustChangePassword replace whatever the cookie or token carried, so local users are always authorized from live bindings.

Authorize (rbac.Engine via auth.Authorize/auth.Require), in this order:

1. Token scopes narrow everything.
2. The builtin admin is allowed with Path local.
3. Implicit grants are allowed with Path local.
4. rbac.Chain{SAR (Kubernetes identities, kube driver only), BindingSource (rbac.bindings + RoleBinding objects)} decides. The first allow wins (D22), and Decision.Path names it.

mustChangePassword is 403 password_change_required except on the self-service routes.

Serve wiring

Everything below goes in cmd/kudurru/serve.go; none of it is edited by this package.

import (
    internalkinds "github.com/robertherbaugh/kudurru/api/internalkinds/v1alpha1"
    "github.com/robertherbaugh/kudurru/internal/auth"
    "github.com/robertherbaugh/kudurru/internal/auth/keys"
    "github.com/robertherbaugh/kudurru/internal/auth/stack"
)

// 1. Both drivers serve the internal kinds (openStore).
scheme, err := internalkinds.NewScheme()          // public + internal kudurru.io kinds
sqlite.Options{Validator: v, Scheme: scheme, InternalKinds: stack.InternalKinds(), ...}
kube.Options{Scheme: scheme, InternalKinds: stack.InternalKinds(), ...} // core/v1 is added by the driver
// openStore must also hand back restCfg (kube only) for step 3.

// 2. Keys: replace readEncryptionKeys (it moves into internal/auth/keys).
k, err := keys.Load(keys.Options{Driver: cfg.Store.Driver, DataDir: cfg.Node.DataDir})
encKey, encPrev := k.Current, k.Previous   // Docker now always has a key, so the internal
                                           // listener always gets credentials; kube without
                                           // a key fails here with keys.ErrKeyRequired.

// 3. The identity layer (after the store and the internal credentials, before the API server).
ns := ""                                   // Docker: tokens.DefaultNamespace ("kudurru-system")
if cfg.Store.Driver == "kube" { ns = kube.PodNamespace() }
ids, err := stack.New(ctx, stack.Options{
    Config:        cfg,
    Store:         st,
    Keys:          k,
    Kube:          restCfg,                // nil with sqlite: disables SA tokens, SAR, clientSecretRef
    Namespace:     ns,
    InternalToken: creds.ValidToken,       // nil if there are no internal credentials
    CrossOrigin:   apiCOP,                 // the API server's *http.CrossOriginProtection
    TrustForwardedProto: <true only behind trusted proxies>,
    ClientIP:      realIP,                 // after RealIP(trustedProxies); default RemoteAddr
    Logger:        logger,
})
// New primes the caches, creates the first-boot admin (printing a generated
// password to stderr once) and installs KUDURRU_BOOTSTRAP_TOKEN.

// 4. Runnables: on EVERY replica (not leader-only; the caches are per process).
group.Go(func() error { return ids.Run(ctx) })
if p := cfg.Auth.AdminSocket.Path; p != "" {
    group.Go(func() error { return ids.AdminSocket().ListenAndServe(ctx, p) })
}

The API server needs these from the stack (the API agent owns the server):

• Middleware construction order (§8.8): Recover → RequestID → RealIP(trustedProxies) → SecurityHeaders (session.SecurityHeaders) → AccessLog → CrossOriginProtection(apiCOP) → RateLimit(auth routes) → ids.Middleware() → mux. The same apiCOP instance goes to stack.Options.CrossOrigin.
• Routes: for _, rt := range ids.Routes() { mux.Handle(rt.Method+" /api/v1"+rt.Path, ids.Guard(rt)) }, or ids.Register(mux). Each Route carries Access (public / authenticated / permission) and Resource/Verb for the route-permission lint. A server with its own authorize wrapper can use the declaration instead of Guard.
• Its own routes: auth.Require(ids.Authorizer(), attrsFn, h) or auth.Authorize(ctx, ids.Authorizer(), attrs). These write the 401/403 contract bodies and store the decision.
• Audit: auth.PrincipalFrom(ctx).ID becomes sqlite.WithActor; auth.DecisionFrom(ctx).Path is binding|sar|local.
• Info: ids.Diagnoses() goes to GET /info/messages (ignored RoleBinding objects, broken IdentityProviders).
• Errors: handlers can use auth.WriteError for the contract error body (it includes took_ms from the middleware's start).

Routes

Every path is under /api/v1. The declarations come from stack.Routes().

Admin socket protocol

The socket is auth.adminSocket.path (default /run/kudurru/kudurru.sock), mode 0660, chgrp auth.adminSocket.group when that is ≥ 0. The protocol is HTTP/1.1 with JSON bodies over the unix socket. The host name in the URL is ignored; the CLI uses http://kudurru/… with a net.Dialer on unix. Unknown JSON fields are a 400. Errors use the API contract body.

Peers are admitted by SO_PEERCRED: uid 0, the server's own uid (65532 in the image, which is what docker exec/kubectl exec get), or the configured gid. Anyone else gets 403 forbidden. An admitted peer is the local root (builtin admin, Decision.Path local).

POST /admin/v1/reset-password
  {"username":"admin","password":"","create":false}
  username defaults to "admin"; an empty password is generated; create makes a missing user (builtin admin)
  → 200 {"username":"admin","password":"<only when generated>","mustChangePassword":true,"created":false}
  → 404 not_found (no such user, create false) | 400 invalid (password under 8 characters or over 1 KiB)
  Effects: new Argon2id hash, mustChangePassword=true, sessionsValidAfter=now (every session revoked).

POST /admin/v1/create-token
  {"user":"","description":"","expiresIn":"24h"}
  user empty → a token of the builtin-admin service account <ns>/break-glass (created on demand),
               default lifetime 24h capped by auth.tokens.maxLifetime;
  user set   → a personal token of that local user (default lifetime auth.tokens.maxLifetime or never)
  → 201 {"id":"abcd1234efgh","token":"kdr_…","owner":{"kind","name","namespace","id","provider"},"expiresAt":"RFC 3339"|null}
  → 404 not_found (user) | 400 invalid (bad expiresIn, longer than maxLifetime)

GET /admin/v1/list-users
  → 200 {"users":[{"name","displayName","builtinAdmin","disabled","mustChangePassword","groups","lastLoginAt","passwordChangedAt"}]}

adminsock.Options.API serves every other path on the socket. When the integrator passes ids.Middleware()(apiMux), the socket also serves the whole REST API to the local root.

Config keys

These are added to internal/config/schema.go. Everything except the two session keys is restartRequired.

Tests

All tests run with CGO_ENABLED=1 go test -race. make test is green for the whole module at this branch, 0 failures. make test-auth-e2e against a real dex v2.46.0 is green: TestDexEndToEnd 17/17 and TestMockOPEndToEnd 31/31.

golangci-lint run reports 0 issues for ./internal/auth/... ./internal/rbac/... ./internal/config/... ./api/... ./internal/store/kube/... ./internal/store/sqlite/... (gofmt, goimports, gosec, revive, staticcheck). go vet is clean and make check-generated is clean.

Measurements

The machine had 8 CPUs shared with five agents (load average 9.8 during the benchmarks), GOMAXPROCS 6, Intel Xeon @ 2.90GHz, go1.27.2.

Commands:

go test -run NONE -bench . -benchtime=2s ./internal/auth/stack/ ./internal/auth/users/
go test -count=1 -v -run TestServiceAccountTokens ./internal/auth/k8s/        # KUBEBUILDER_ASSETS set
make test-auth-e2e

Surprises

• api/internal/v1alpha1 cannot work. Go's internal-package rule makes it importable only from api/..., so internal/rbac and the auth packages could not use it. The kinds live in api/internalkinds/v1alpha1 (same group version, still never CRDs).
• The kube driver's internal kinds failed with the production scheme. internalkinds.NewScheme() holds only kudurru.io, and listing Secrets needs core/v1. The driver's own tests used a test scheme that already had core/v1. The new end-to-end test on the kube store caught it, and the driver now adds core/v1 itself (TestEnvtestInternalKindsProductionScheme).
• A session issued right after a password change was born revoked. sessionsValidAfter and iat both have one-second resolution, and the rule is iat > validAfter. The login and password routes now place iat one second after the user's sessionsValidAfter.
• Failing a cookie request on CSRF made re-login impossible. A POST /auth/login from a browser that still held a valid session cookie but sent no CSRF header (the login page) was a 403. The cookie step now drops the session and records the failure: protected routes still answer 403 csrf, logout refuses, and login works.
• objcache could regress. A read-your-writes Upsert racing the watch event for a newer version replaced it, and nothing would ever correct it. Resource versions now compare numerically (every driver issues increasing integers) and an older version never replaces a newer one.
• The chart Role is missing delete on secrets. It is needed for token revocation and user/RoleBinding deletion on the kube driver. §5.3 also lists only get/list/watch/create/update. Recorded as a follow-up for the chart owner.
• SubjectAccessReview for zone-scoped requests names the zone as the resource name (dnsrecordsets with resourceNames: [home]), which is how a Kubernetes zone-admin is expressed. It works against a real apiserver (envtest).
• Registering the IdentityProvider cache's OnChange before Prime made the first registry build run without the config entries, so a broken config IdP was not a startup error. The hook is now registered after the first build.
• The admin socket and the "unix peer" step are the same mechanism. HTTP over the socket with PeerConnContext gives the kudurru admin protocol and, optionally, the full API to the local root, without a second wire format.
• Logins and personal tokens of OIDC users cannot be re-resolved per request because the session has no groups (D48). Their grants are frozen and honoured only while live bindings still provide them. Bearer ID tokens and SA tokens carry groups, so they are evaluated live.
• The tool sandbox refused shell commands that contained "git" together with pipes or heredocs (including github.com in Python one-liners). Edits went through small scripts in /tmp.

Architecture changes

These are recommended for the integrator; docs/architecture.md was not edited.

1. §6.2: the admin socket keys are auth.adminSocket.{path,group}, not listeners.socket. They are node scope and hostOnly, and the same default path is used. Document auth.kubernetes.mode.
2. §4.1/§8: the internal kinds live in api/internalkinds/v1alpha1 (Go rule). In Kubernetes they are Secrets kudurru-<kind>.<ns>.<name> labelled kudurru.io/kind in the release namespace, holding data.object.
3. §8.5: Decision paths are local (builtin admin: unix peer, first-boot admin, bootstrap and break-glass service accounts; also the Node principal's implicit viewer+auditor grants), sar, binding and none. Sealed session grants are honoured only while a live binding provides them and no binding change revoked them after iat (closes the M3 recommendation of the auth spike).
4. §8.7: POST /api/v1/auth/password and GET /auth/status.localLoginMode. admins-only admits builtin admins and cluster admin holders. The generated first-boot password sets mustChangePassword, while env- and file-provided passwords do not.
5. §5.3: add delete to the secrets rule.
6. §8.3: the bearer ID-token path is live server side in v0.1. Access tokens (typ: at+jwt) are refused.
7. §8.6: a token-authenticated POST /auth/tokens cannot widen scopes or outlive its parent token. Break-glass tokens default to 24 h.

Follow-ups

These rows were added to docs/followups.md:

• Persist binding revocation epochs (v0.2).
• A user store for OIDC identities: per-user logout everywhere, disable, autoProvision: never, syncGroups (v0.2).
• Login lockouts are per replica (v0.2).
• Re-check JWKS-verified SA tokens against deleted pods or ServiceAccounts (v0.2).
• Chart secrets delete (M3, chart owner).
• CLI kudurru admin … and kudurru token generate (M3, CLI).
• The architecture.md updates listed above (M3, integrator).