Greentic AI · Phase D · Milestone M2

Deploying a worker on Kubernetes:
the distributor-pull path, explained

A walk-through of my_demos/k8s-deploy-demo — what it proves, how a Greentic worker actually boots and serves itself inside Kubernetes, what every command does, and why the whole thing is built around a worker that ships empty and fetches its own packs at boot.

Mechanism: bundle-less boot-time pull Runtime: greentic-start start --env Cluster: kind (K8s-in-Docker) Integrity: sha256 digest-gate, fail-closed

01The whole idea

One sentence, then the consequence.

M2 — distributor-pull.

A worker pod ships with only an environment.json (delivered in a Kubernetes ConfigMap). At boot it pulls its .gtbundle from a distributor over the network, digest-verifies the bytes, materializes the packs to disk, activates the routed revision, and serves it over HTTP.

That is the difference between a toy and a cloud runtime. The image baked into the pod knows nothing about which packs it runs — that's decided by control-plane state and resolved at runtime. The same immutable image can serve any tenant, any revision, any bundle. Nothing about the workload is welded into the container.

Greentic's layering makes this clean. Each layer is a sealed artifact the one above composes:

Component Flow Pack (.gtpack, signed) Bundle (.gtbundle) Operator / Runner

The demo exercises the bottom-right of that chain: the Operator authors desired state, the Runner (a greentic-start pod) pulls the Bundle and brings it to life.

02Why pull at boot, instead of baking the bundle in?

The constraint that shapes the entire design.

❌ Bake the bundle into the image

  • A new image build for every revision of every tenant
  • Registry bloat; slow rollouts
  • The runtime image is no longer generic
  • Secrets / packs entangled with the binary

✅ Pull the bundle at boot (M2)

  • One immutable runtime image, forever
  • Revisions change = ConfigMap changes, no rebuild
  • Pod is configured by control-plane state, not by its image
  • Integrity guaranteed by a pinned sha256, not by trust in the registry

The cold state a pod boots from is deliberately minimal. The deployer materializes packs and a runtime-config.json on the authoring host — but a cloud pod never receives those. Its ConfigMap carries environment.json and nothing else. The demo reproduces this exactly by deleting runtime-config.json and revisions/ before boot — byte-for-byte the state a real pod wakes up in. That emptiness is what forces the pull.

03What happens when the worker boots

The core mechanism — greentic-start start --env local, step by step.

The pod's container command is just greentic-start start --env <id>. It reads the env store, finds an empty runtime-config, and the boot seam takes over:

1
Load env store → find runtime-config empty
The ConfigMap shipped only environment.json. There are no materialized packs on disk. The boot seam detects rc.revisions.is_empty().
2
Find the routed revision carrying a bundle_source_uri
Only traffic-routed revisions are pulled. The revision's bundle_source_uri says where to fetch the bundle from (HTTP / OCI / file).
3
Fetch the bundle bytesrevision_pull::fetch_bundle_to_file
A raw GET of the .gtbundle from the distributor. No early trust — just bytes on disk.
4
🔒 Digest-gate the pulled bytes against bundle_digest
Fail-closed. The sha256 pinned at stage time is the integrity backstop. If the pulled bytes don't match, boot aborts — a tampered or wrong bundle can never activate.
5
Materializematerialize_revision_from_bundle
Unpack the bundle, lay out packs + pack-configs on disk, rewrite runtime-config.json. Failure-atomic: an existing live revision is moved aside and rolled back on error, never half-bricked.
6
Re-load & activate
The now-populated runtime-config shadows the empty one; the runtime activates the revision (a pure projection from runtime-config.json).
7
✅ Serve/healthz → 200
Boot banner reads serving 1 revision(s) for env `local`not serving probes only. That banner is the exact assertion the K8s E2E makes.

Why the digest matters more than the source. bundle_source_uri is an opaque, caller-asserted reference — the worker will fetch whatever it points at. Trust does not come from the URL; it comes from the bundle_digest pinned on the revision at stage time. Breadth of the source ≠ injection risk, because the bytes are gated before anything is unpacked.

04The commands — authoring the deployment

Every op verb the operator runs, in order, and what it records.

All authoring happens on the control-plane host with the deployer's op verbs. Each verb writes to the env store (the thing that becomes the in-cluster ConfigMap). Nothing touches a cluster until reconcile.

CommandWhat it does
env create Writes environment.json — the seed of the whole env. Becomes the gtc-env-store ConfigMap in-cluster.
trust-root bootstrap Seeds the operator's Ed25519 signing key. bundles add refuses to run without it — packs are signed artifacts.
bundles add Registers a deployment — a routing target. route_binding.path_prefixes:["/"] routes every request path to it. Returns a deployment_id.
revisions stage M2 key step Records two things on the revision:
bundle_digest = sha256 of the local fixture → the integrity backstop
bundle_source_uri = where a remote worker will pull the bundle from (the M2 field). Returns a revision_id.
revisions warm Flips the revision lifecycle staged → ready. In-cluster this waits for the rollout to actually become healthy before proceeding.
traffic set Routes traffic (weight_bps: 10000 = 100%). Only routed revisions get pulled, so this is what arms the boot-time fetch. Idempotency-keyed.
env-packs add K8s only Binds the K8s deployer (greentic.deployer.k8s@1.0.0). This is what makes the env a Kubernetes env. Pins the runtime image for the rendered pods.
env reconcile K8s only The only verb that touches the cluster. Server-side-applies all desired objects against the real API server (idempotent, converging).

Authoring a pullable revision (Track 1 — loopback HTTP distributor)

# 1. env + trust root
op env create            # writes environment.json
op trust-root bootstrap local

# 2. a routing target
op bundles add            # route_binding.path_prefixes = ["/"]  → deployment_id

# 3. THE M2 STEP — stage a PULLABLE revision
op revisions stage \
   bundle_path=perf-smoke-bundle.gtbundle     # → pins its sha256 as bundle_digest \
   bundle_source_uri=http://127.0.0.1:8090/bundle.gtbundle   # where to pull

# 4. ready it, then send 100% of traffic (only routed revs are pulled)
op revisions warm         # staged → ready
op traffic set            # weight_bps = 10000 → revision_id

The fixture: perf-smoke-bundle.gtbundle (4 KiB, sha256:c5199a71…). It's the one ready-made valid bundle in the workspace, and its pack carries no .wasm — which is perfect here. It activates and serves /healthz, proving the pull → materialize → activate chain, without needing a component runtime. (A real request can't run a flow with it — that's the honest edge.)

05Two tracks — and what each one proves

The demo can run the same pull code in two places. ./demo.sh <cmd>.

TrackWhat it runsReal K8s?Local status
1 · host-pull The M2 pull/serve code on the host, no cluster. Reproduces a pod's cold boot exactly — same pull → digest-gate → materialize → activate → serve code. No (simulates the pod) ✅ fully verified
2 · kind-serve A real worker POD boots start --env, reaches Ready, serves /healthz. Readiness probe = kubelet → pod. Yes ✅ green locally
2 · kind-pull The full in-cluster pull: worker pod pulls (pod → pod) from an in-cluster busybox-httpd serving the bundle, logs the real-revision banner. Yes ⚠️ CI-only here

The whole demo, end to end

cd /home/vampik/greenticai/my_demos/k8s-deploy-demo

./demo.sh host-pull     # Track 1 — start here. ~10s, no Kubernetes needed.
./demo.sh host-stop     # stop the worker it leaves running

./demo.sh kind-up       # Track 2 — create a kind cluster + load the :develop runtime image
./demo.sh kind-serve    #          prove a worker POD boots + serves /healthz   (local-green)
./demo.sh kind-pull     #          full in-cluster pull                          (CI-green)
./demo.sh kind-down     # delete the cluster

Track 1 proves the code is correct

It runs the identical pull/materialize/activate/serve path, just outside a pod. The only thing it skips is the pod→pod transport. 100% reproducible on any machine.

Track 2 proves it works inside Kubernetes

Same objects, same pod boot command, same reconcile path as cloud (EKS/GKE/AKS). kind is just K8s running as Docker containers on your machine.

Why kind-pull is CI-only on this machine. The final hop is the worker pod pulling pod → pod from an in-cluster httpd pod. On this host (CachyOS 7.0.11 + docker 29.5.3), same-node pod-to-pod networking has 100% packet loss — a local kindnet/kernel defect, not a code or test defect (kubelet → pod is fine, which is why kind-serve works). The exact same test is green on CI's ubuntu-latest. Track 1's host-pull is the local stand-in for precisely this path.

06Inside Kubernetes — control-plane vs data-plane

The split that holds true on real cloud too.

WhatWhere it runsVerb / command
Author the env
create · env-packs add · bundles add · stage · warm · traffic
Host / control-plane. You build the env store, which becomes a ConfigMap in-cluster. Nothing touches the cluster. op …
Converge onto the cluster Against the cluster. Server-side apply of every desired object onto the real API server. op env reconcile
Boot, serve, pull Inside the cluster. Worker / router pods boot start --env, serve /healthz, pull their bundles. the pod's container command

Visually — what lives where during a K8s deploy:

CONTROL PLANE · your host

  • env store on disk (environment.json + trust root)
  • op verbs author desired state
  • op env reconcile talks to the API server
  • Holds the bundle sha256 + bundle_source_uri
DATA PLANE · the cluster
  • gtc-env-store ConfigMap — only environment.json
  • worker pod boots greentic-start start --env
  • Pulls bundle from bundle_source_uri → digest-gate → serve
  • Readiness probe hits the /healthz it serves

The K8s-only delta vs the host track

# Bind the K8s deployer — THIS makes the env a Kubernetes env
op env-packs add \
   slot=deployer kind=greentic.deployer.k8s@1.0.0 pack_ref=builtin \
   answers_ref=deployer-answers.json     # pins runtime_image for the rendered pods

# …author the revision exactly as Track 1…

# The ONLY verb that touches the cluster:
op env reconcile local       # server-side apply → 14 objects

# Watch a real Greentic worker come up inside Kubernetes:
kubectl rollout status deployment/gtc-worker-<revid> -n gtc-local
kubectl logs deployment/gtc-worker-<revid> -n gtc-local | grep "revision(s) for env"
# SUCCESS:  serving 1 revision(s) for env `local`     (NOT "serving probes only")

The worker Deployment is named after the revision ULID: gtc-worker-<lowercased revision_id>. Both router and worker pods boot the same start --env command.

07What reconcile applies — the 14 objects

12 env-level objects + 2 per warmed revision.

12 env-level objects

  • Namespace gtc-local
  • ConfigMap gtc-env-store — carries environment.json (with the routed revision + its bundle_source_uri)
  • ConfigMap gtc-runtime-config
  • Deployment gtc-router
  • Service + PodDisruptionBudget for the router
  • 6× NetworkPolicy — default-deny + scoped allows (incl. router-egress so the router can pull its bundle at boot)

+2 per warmed revision

  • Deployment gtc-worker-<revid> — boots start --env
  • Service for the worker

Archive the revision and re-reconcile → the worker pair is pruned back to the 12 env-level objects. Reconcile converges both directions; it's not append-only.

NetworkPolicy is load-bearing. The env's gtc-default-deny denies ingress to every pod, and kindnet v1.31+ enforces it. That's why pulling needs an explicit allow — the router gets a router-egress policy, and a worker pulling at boot needs a worker-egress allow. In real cloud the bundle source is an external registry (outside the namespace, unaffected); in-cluster, the allow is the fixture's concern.

08How the local demo maps to real cloud

Every local stand-in, and its production counterpart.

Demo (local)Real cloud (EKS / GKE / AKS)
kind create cluster The managed cluster you provisioned
state/.greentic/…/environment.json The gtc-env-store ConfigMap mounted into the pod
Stripping runtime-config.json + revisions/ The pod simply never receives them — the ConfigMap carries only environment.json
python3 -m http.server on :8090  ·  in-cluster busybox-httpd The pack distributor / OCI registry the bundle_source_uri points at
env-packs add binding greentic.deployer.k8s@1.0.0 The env declaring it deploys to Kubernetes
op env reconcile local The operator / control-plane converging desired state onto the cluster
Worker pod booting start --env Identical — the same container command

The single production transport this machine can't exercise — a worker pod pulling over the cluster network — is exactly what CI covers, and what Track 1's host-pull substitutes for locally. Everything else in the chain runs unchanged from laptop to cloud.

09The honest edge — what a 200-with-an-error means

Where the fixture stops, and what comes next.

While the worker runs you can poke it. /status confirms it activated a real revision:

Probe the running worker

curl -s http://127.0.0.1:8080/status
{"bundles_active":1,"deployments_routed":1,"env_id":"local","revisions_active":1,...}

curl -s -X POST http://127.0.0.1:8080/ -H 'content-type: application/json' -d '{"text":"hi"}'
# HTTP 200, body:
[{"kind":{"kind":"custom","action":"response"},
  "payload":{"metadata":{"error_kind":"flow_execution_failed",
                         "error_message":"flow main has no start node",...}}}]

That POST is the honest edge of the fixture. The request reaches the pulled-and-activated revision and is dispatched into its flow — the whole chain works — but perf-smoke ships no real flow graph, so the flow itself fails with flow main has no start node.

A useful 200 reply needs a bundle whose pack ships components (.wasm). That is M3 — the next milestone. M2 proves the delivery; M3 proves the work.

The milestone ladder

M1
A real worker pod boots and serves /healthz
Pod runs start --env, binds 0.0.0.0, reaches Ready. Probes-only. ✅ shipped.
M2
The worker pulls its bundle at boot and activates the revision
The subject of this demo: distributor-pull → digest-gate → materialize → activate → serve. ✅ shipped (host + CI green).
M3
Serve a real request
Needs a bundle whose pack carries real .wasm components, so a flow can actually run and return a useful reply. ⏳ next.