Greentic AI · Phase D · Milestones M1 → M4

Deploying a worker on Kubernetes:
the distributor-pull path, explained

A walk-through of my_demos/k8s-deploy-demo — what it proves, how a Greentic worker actually boots and serves itself inside Kubernetes, what every command does, and why the whole thing is built around a worker that ships empty and fetches its own packs at boot.

Mechanism: bundle-less boot-time pull Runtime: greentic-start start --env Transports: http:// · file:// · oci:// Cluster: kind (K8s-in-Docker) Integrity: sha256 digest-gate, fail-closed

01The whole idea

One sentence, then the consequence.

Updated this week (week of 2026-06-15 → 22).

The deployment line that this demo proves now runs end to end: M2 (pull a bundle at boot) was joined by M3 (a real flow executes on it) and M4 (an external request reaches a real component through the router). The worker now also pulls over oci:// (not just http:///file://), gets a bound ServiceAccount identity, and a new real-bundle demo adds Telegram + a Webchat /chat GUI. Jump to the real demo or the milestone ladder for the new parts.

M2 — distributor-pull.

A worker pod ships with only an environment.json (delivered in a Kubernetes ConfigMap). At boot it pulls its .gtbundle from a distributor over the network, digest-verifies the bytes, materializes the packs to disk, activates the routed revision, and serves it over HTTP.

That is the difference between a toy and a cloud runtime. The image baked into the pod knows nothing about which packs it runs — that's decided by control-plane state and resolved at runtime. The same immutable image can serve any tenant, any revision, any bundle. Nothing about the workload is welded into the container.

Greentic's layering makes this clean. Each layer is a sealed artifact the one above composes:

Component Flow Pack (.gtpack, signed) Bundle (.gtbundle) Operator / Runner

The demo exercises the bottom-right of that chain: the Operator authors desired state, the Runner (a greentic-start pod) pulls the Bundle and brings it to life.

02Why pull at boot, instead of baking the bundle in?

The constraint that shapes the entire design.

❌ Bake the bundle into the image

  • A new image build for every revision of every tenant
  • Registry bloat; slow rollouts
  • The runtime image is no longer generic
  • Secrets / packs entangled with the binary

✅ Pull the bundle at boot (M2)

  • One immutable runtime image, forever
  • Revisions change = ConfigMap changes, no rebuild
  • Pod is configured by control-plane state, not by its image
  • Integrity guaranteed by a pinned sha256, not by trust in the registry

The cold state a pod boots from is deliberately minimal. The deployer materializes packs and a runtime-config.json on the authoring host — but a cloud pod never receives those. Its ConfigMap carries environment.json and nothing else. The demo reproduces this exactly by deleting runtime-config.json and revisions/ before boot — byte-for-byte the state a real pod wakes up in. That emptiness is what forces the pull.

03What happens when the worker boots

The core mechanism — greentic-start start --env local, step by step.

The pod's container command is just greentic-start start --env <id>. It reads the env store, finds an empty runtime-config, and the boot seam takes over:

1
Load env store → find runtime-config empty
The ConfigMap shipped only environment.json. There are no materialized packs on disk. The boot seam detects rc.revisions.is_empty().
2
Find the routed revision carrying a bundle_source_uri
Only traffic-routed revisions are pulled. The revision's bundle_source_uri says where to fetch the bundle from (HTTP / OCI / file).
3
Fetch the bundle bytesrevision_pull::fetch_bundle_to_file
A raw GET of the .gtbundle from the distributor. No early trust — just bytes on disk.
4
🔒 Digest-gate the pulled bytes against bundle_digest
Fail-closed. The sha256 pinned at stage time is the integrity backstop. If the pulled bytes don't match, boot aborts — a tampered or wrong bundle can never activate.
5
Materializematerialize_revision_from_bundle
Unpack the bundle, lay out packs + pack-configs on disk, rewrite runtime-config.json. Failure-atomic: an existing live revision is moved aside and rolled back on error, never half-bricked.
6
Re-load & activate
The now-populated runtime-config shadows the empty one; the runtime activates the revision (a pure projection from runtime-config.json).
7
✅ Serve/healthz → 200
Boot banner reads serving 1 revision(s) for env `local`not serving probes only. That banner is the exact assertion the K8s E2E makes.

Why the digest matters more than the source. bundle_source_uri is an opaque, caller-asserted reference — the worker will fetch whatever it points at. Trust does not come from the URL; it comes from the bundle_digest pinned on the revision at stage time. Breadth of the source ≠ injection risk, because the bytes are gated before anything is unpacked.

04The commands — authoring the deployment

Every op verb the operator runs, in order, and what it records.

All authoring happens on the control-plane host with the deployer's op verbs. Each verb writes to the env store (the thing that becomes the in-cluster ConfigMap). Nothing touches a cluster until reconcile.

CommandWhat it does
env create Writes environment.json — the seed of the whole env. Becomes the gtc-env-store ConfigMap in-cluster.
trust-root bootstrap Seeds the operator's Ed25519 signing key. bundles add refuses to run without it — packs are signed artifacts.
bundles add Registers a deployment — a routing target. route_binding.path_prefixes:["/"] routes every request path to it. Returns a deployment_id.
revisions stage M2 key step Records two things on the revision:
bundle_digest = sha256 of the local fixture → the integrity backstop
bundle_source_uri = where a remote worker will pull the bundle from (the M2 field). Returns a revision_id.
revisions warm Flips the revision lifecycle staged → ready. In-cluster this waits for the rollout to actually become healthy before proceeding.
traffic set Routes traffic (weight_bps: 10000 = 100%). Only routed revisions get pulled, so this is what arms the boot-time fetch. Idempotency-keyed.
env-packs add K8s only Binds the K8s deployer (greentic.deployer.k8s@1.0.0). This is what makes the env a Kubernetes env. Pins the runtime image for the rendered pods.
env reconcile K8s only The only verb that touches the cluster. Server-side-applies all desired objects against the real API server (idempotent, converging).

Authoring a pullable revision (Track 1 — loopback HTTP distributor)

# 1. env + trust root
op env create            # writes environment.json
op trust-root bootstrap local

# 2. a routing target
op bundles add            # route_binding.path_prefixes = ["/"]  → deployment_id

# 3. THE M2 STEP — stage a PULLABLE revision
op revisions stage \
   bundle_path=perf-smoke-bundle.gtbundle     # → pins its sha256 as bundle_digest \
   bundle_source_uri=http://127.0.0.1:8090/bundle.gtbundle   # where to pull

# 4. ready it, then send 100% of traffic (only routed revs are pulled)
op revisions warm         # staged → ready
op traffic set            # weight_bps = 10000 → revision_id

Three transports for bundle_source_uri

The worker fetches the bundle from whatever scheme the URI uses — the digest-gate is the same for all three:

SchemeWhere the bundle livesUsed by
http:// A plain HTTP distributor (e.g. python3 -m http.server, or an in-cluster busybox-httpd) Track 1 host-pull · kind-pull
file:// A local file — no server needed at all Running on a PC with only gtc dev
oci:// new this week An OCI registry (ghcr, or an in-cluster registry:2). For a plain-HTTP registry the worker needs GREENTIC_OCI_INSECURE_REGISTRIES set — a hatch injected post-reconcile so the production render never carries it. The real demo (pulls from public ghcr)

Extra verbs for the real demo

When the bundle is real (with a flow, a messaging endpoint, and secrets), a few more op verbs come into play — see section 09:

CommandWhat it does
env-packs add (secrets)Binds greentic.secrets.dev-store@1.0.0 — required before secrets put.
secrets putStores a value (a webhook secret, a Telegram bot token) in the env's dev-store.
messaging endpoint addRegisters a messaging endpoint (e.g. messaging.telegram.bot) → endpoint_id.
messaging endpoint link-bundleWires the endpoint to a bundle, so its webhook routes into that revision's flow.

The fixture: perf-smoke-bundle.gtbundle (4 KiB, sha256:c5199a71…). It's the one ready-made valid bundle in the workspace, and its pack carries no .wasm — which is perfect here. It activates and serves /healthz, proving the pull → materialize → activate chain, without needing a component runtime. (A real request can't run a flow with this fixture — for that you need a bundle whose pack ships .wasm components, which is exactly what the real demo does.)

05Two tracks — and what each one proves

The demo can run the same pull code in two places. ./demo.sh <cmd>.

TrackWhat it runsReal K8s?Local status
1 · host-pull The M2 pull/serve code on the host, no cluster. Reproduces a pod's cold boot exactly — same pull → digest-gate → materialize → activate → serve code. No (simulates the pod) ✅ fully verified
2 · kind-serve A real worker POD boots start --env, reaches Ready, serves /healthz. Readiness probe = kubelet → pod. Yes ✅ green locally
2 · kind-pull The full in-cluster pull: worker pod pulls (pod → pod) from an in-cluster busybox-httpd serving the bundle, logs the real-revision banner. Yes ⚠️ CI-only here

The whole demo, end to end

cd /home/vampik/greenticai/my_demos/k8s-deploy-demo

./demo.sh host-pull     # Track 1 — start here. ~10s, no Kubernetes needed.
./demo.sh host-stop     # stop the worker it leaves running

./demo.sh kind-up       # Track 2 — create a kind cluster + load the :develop runtime image
./demo.sh kind-serve    #          prove a worker POD boots + serves /healthz   (local-green)
./demo.sh kind-pull     #          full in-cluster pull                          (CI-green)
./demo.sh kind-down     # delete the cluster

Track 1 proves the code is correct

It runs the identical pull/materialize/activate/serve path, just outside a pod. The only thing it skips is the pod→pod transport. 100% reproducible on any machine.

Track 2 proves it works inside Kubernetes

Same objects, same pod boot command, same reconcile path as cloud (EKS/GKE/AKS). kind is just K8s running as Docker containers on your machine.

Why kind-pull is CI-only on this machine. The final hop is the worker pod pulling pod → pod from an in-cluster httpd pod. On this host (CachyOS 7.0.11 + docker 29.5.3), same-node pod-to-pod networking has 100% packet loss — a local kindnet/kernel defect, not a code or test defect (kubelet → pod is fine, which is why kind-serve works). The exact same test is green on CI's ubuntu-latest. Track 1's host-pull is the local stand-in for precisely this path.

06Inside Kubernetes — control-plane vs data-plane

The split that holds true on real cloud too.

WhatWhere it runsVerb / command
Author the env
create · env-packs add · bundles add · stage · warm · traffic
Host / control-plane. You build the env store, which becomes a ConfigMap in-cluster. Nothing touches the cluster. op …
Converge onto the cluster Against the cluster. Server-side apply of every desired object onto the real API server. op env reconcile
Boot, serve, pull Inside the cluster. Worker / router pods boot start --env, serve /healthz, pull their bundles. the pod's container command

Visually — what lives where during a K8s deploy:

CONTROL PLANE · your host

  • env store on disk (environment.json + trust root)
  • op verbs author desired state
  • op env reconcile talks to the API server
  • Holds the bundle sha256 + bundle_source_uri
DATA PLANE · the cluster
  • gtc-env-store ConfigMap — only environment.json
  • worker pod boots greentic-start start --env
  • Pulls bundle from bundle_source_uri → digest-gate → serve
  • Readiness probe hits the /healthz it serves

The K8s-only delta vs the host track

# Bind the K8s deployer — THIS makes the env a Kubernetes env
op env-packs add \
   slot=deployer kind=greentic.deployer.k8s@1.0.0 pack_ref=builtin \
   answers_ref=deployer-answers.json     # pins runtime_image for the rendered pods

# …author the revision exactly as Track 1…

# The ONLY verb that touches the cluster:
op env reconcile local       # server-side apply → 14 objects

# Watch a real Greentic worker come up inside Kubernetes:
kubectl rollout status deployment/gtc-worker-<revid> -n gtc-local
kubectl logs deployment/gtc-worker-<revid> -n gtc-local | grep "revision(s) for env"
# SUCCESS:  serving 1 revision(s) for env `local`     (NOT "serving probes only")

The worker Deployment is named after the revision ULID: gtc-worker-<lowercased revision_id>. Both router and worker pods boot the same start --env command.

07What reconcile applies — the 14 objects

12 env-level objects + 2 per warmed revision.

12 env-level objects

  • Namespace gtc-local
  • ConfigMap gtc-env-store — carries environment.json (with the routed revision + its bundle_source_uri)
  • ConfigMap gtc-runtime-config
  • Deployment gtc-router
  • Service + PodDisruptionBudget for the router
  • 6× NetworkPolicy — default-deny + scoped allows (incl. router-egress so the router can pull its bundle at boot)

+2 per warmed revision

  • Deployment gtc-worker-<revid> — boots start --env
  • Service for the worker

Archive the revision and re-reconcile → the worker pair is pruned back to the 12 env-level objects. Reconcile converges both directions; it's not append-only.

NetworkPolicy is load-bearing. The env's gtc-default-deny denies ingress to every pod, and kindnet v1.31+ enforces it. That's why pulling needs an explicit allow — the router gets a router-egress policy, and a worker pulling at boot needs a worker-egress allow. In real cloud the bundle source is an external registry (outside the namespace, unaffected); in-cluster, the allow is the fixture's concern.

08How the local demo maps to real cloud

Every local stand-in, and its production counterpart.

Demo (local)Real cloud (EKS / GKE / AKS)
kind create cluster The managed cluster you provisioned
state/.greentic/…/environment.json The gtc-env-store ConfigMap mounted into the pod
Stripping runtime-config.json + revisions/ The pod simply never receives them — the ConfigMap carries only environment.json
python3 -m http.server on :8090  ·  in-cluster busybox-httpd The pack distributor / OCI registry the bundle_source_uri points at
env-packs add binding greentic.deployer.k8s@1.0.0 The env declaring it deploys to Kubernetes
op env reconcile local The operator / control-plane converging desired state onto the cluster
Worker pod booting start --env Identical — the same container command

The single production transport this machine can't exercise — a worker pod pulling over the cluster network — is exactly what CI covers, and what Track 1's host-pull substitutes for locally. Everything else in the chain runs unchanged from laptop to cloud.

09The real demo — a bundle + Telegram + Webchat new this week

Same K8s flow, upgraded from the empty fixture to a real worker that talks back. See RUN-REAL-FISH.md.

Everything so far proved delivery with the empty perf-smoke bundle. The new runbook swaps in a real bundlewebchat-bot.gtbundle (~4.9 MB, a real flow + WASM components) — pulled oci:// from public ghcr, and wires up two ways to actually talk to it: a Telegram messaging endpoint and the built-in Webchat /chat GUI.

StepProbes-only (earlier)Real (this week)
Bundlenone — probes-only revisionwebchat-bot.gtbundle pulled oci:// from ghcr
Routepath_prefixes:["/"]+ tenant_selector tenant-default so /chat routes
Secretsnot bounddev-store pack bound → secrets put for the bot token + webhook secret
MessagingnoneTelegram endpoint + link-bundle + a known webhook secret
Serves/healthz, /status+ /chat GUI, /workers/invoke flow, /webhook/telegram

Talking to the worker

Webchat, headless flow, and the Telegram webhook gate

# Webchat GUI — the welcome card renders on load
xdg-open http://127.0.0.1:8080/chat

# Headless — run the flow directly
curl -s -X POST http://127.0.0.1:8080/workers/invoke -d '{...HostWorkerRequest...}' | jq .

# Telegram webhook: correct secret-token -> 200, wrong -> 401
curl -s -o /dev/null -w '%{http_code}\n' -X POST http://127.0.0.1:8080/webhook/telegram \
  -H "x-telegram-bot-api-secret-token: $WHVAL" -d '{"update_id":1,"message":{...}}'   # 200

For a live bot, a cloudflared tunnel exposes the worker to the internet and setWebhook points Telegram at <tunnel>/webhook/telegram — then messaging the bot returns the welcome card.

The one wrinkle: bridging secrets into the pod

The K8s deployer renders only environment.json into the pod — not runtime secrets (workload-identity / secret-store is deferred to Phase E). A real bundle needs its bot token and webhook secret inside the pod, so the demo bridges the env's dev-store in:

  • a Secret gtc-dev-secrets built from the dev-store file
  • an init container (bridge-patch.yaml) that copies it into the worker's writable HOME at the path greentic-start resolves

On real cloud, a proper secret-store / workload identity replaces this bridge.

Three things the pod-to-pod defect forces — and why they don't weaken the proof:

  • the bundle is pulled oci:// from ghcr over the internet (pod→internet, healthy), not from an in-cluster httpd (pod→pod, dead here)
  • you reach the worker via kubectl port-forward (kubelet→pod), not through the router (router→worker is pod→pod)
  • runtime secrets are bridged in (above) rather than rendered

On a machine with healthy pod-to-pod networking the in-cluster pull and the router path work too — and CI exercises both.

A fully-scripted bash equivalent lives next door at ../k8s-telegram-webchat-demo/ (demo.sh + its own RUN-IN-K8S.md). RUN-REAL-FISH.md is the copy-paste fish version.

10The milestone ladder — now through M4

What a 200-with-a-flow-error meant, and how M3 + M4 closed it.

While the worker runs you can poke it. /status confirms it activated a real revision:

Probe the running worker

curl -s http://127.0.0.1:8080/status
{"bundles_active":1,"deployments_routed":1,"env_id":"local","revisions_active":1,...}

curl -s -X POST http://127.0.0.1:8080/ -H 'content-type: application/json' -d '{"text":"hi"}'
# HTTP 200, body:
[{"kind":{"kind":"custom","action":"response"},
  "payload":{"metadata":{"error_kind":"flow_execution_failed",
                         "error_message":"flow main has no start node",...}}}]

That POST is the honest edge of the fixture. The request reaches the pulled-and-activated revision and is dispatched into its flow — the whole chain works — but perf-smoke ships no real flow graph, so the flow itself fails with flow main has no start node.

A useful 200 reply needs a bundle whose pack ships components (.wasm) — and a flow that invokes them. That was M3, and it has now shipped (with M4 right behind it). M2 proved the delivery; M3 + M4 prove the work — see the ladder below.

The ladder — all four shipped

M1
A real worker pod boots and serves /healthz
Pod runs start --env, binds 0.0.0.0, reaches Ready. Probes-only. ✅ shipped (#332).
M2
The worker pulls its bundle at boot and activates the revision
The core of this demo: distributor-pull → digest-gate → materialize → activate → serve. Now over http://, file://, and oci://. ✅ shipped (host + CI green; oci #346).
M3
A real flow executes on the pulled revision
A templates-bundle fixture ships a real component-templates@0.6.0 WASM component + a one-node flow. POST /workers/invoke → 200, body contains component-templates::handle_message => … (the component ran). ✅ shipped 2026-06-19 (#345).
M4
An external request reaches a real component through the router
The production front door: a genuinely external request (an in-cluster client Job in a separate gtc-ext-ingress namespace, modelling the real Gateway) goes through the env router to the component. ✅ shipped 2026-06-21 (#354).

Also new this week — a bound ServiceAccount identity. --bind now mints and binds a deployer ServiceAccount at bootstrap, so a bound SA drives reconcile (dropping the cluster-scoped Namespace permission) with a durable in-cluster home for the identity (#347–#349, #353). That's the cloud-shaped auth model: the operator acts as a scoped identity, not cluster-admin.