01The whole idea
One sentence, then the consequence.
Updated this week (week of 2026-06-15 → 22).
The deployment line that this demo proves now runs end to end: M2 (pull a bundle at boot)
was joined by M3 (a real flow executes on it) and M4 (an external request reaches a real
component through the router). The worker now also pulls over oci:// (not just
http:///file://), gets a bound ServiceAccount identity, and a new
real-bundle demo adds Telegram + a Webchat /chat GUI. Jump to
the real demo or the milestone ladder for the new parts.
M2 — distributor-pull.
A worker pod ships with only an environment.json (delivered in a Kubernetes ConfigMap).
At boot it pulls its .gtbundle from a distributor over the network,
digest-verifies the bytes, materializes the packs to disk,
activates the routed revision, and serves it over HTTP.
That is the difference between a toy and a cloud runtime. The image baked into the pod knows nothing about which packs it runs — that's decided by control-plane state and resolved at runtime. The same immutable image can serve any tenant, any revision, any bundle. Nothing about the workload is welded into the container.
Greentic's layering makes this clean. Each layer is a sealed artifact the one above composes:
The demo exercises the bottom-right of that chain: the Operator authors desired state,
the Runner (a greentic-start pod) pulls the Bundle and brings it to life.
02Why pull at boot, instead of baking the bundle in?
The constraint that shapes the entire design.
❌ Bake the bundle into the image
- A new image build for every revision of every tenant
- Registry bloat; slow rollouts
- The runtime image is no longer generic
- Secrets / packs entangled with the binary
✅ Pull the bundle at boot (M2)
- One immutable runtime image, forever
- Revisions change = ConfigMap changes, no rebuild
- Pod is configured by control-plane state, not by its image
- Integrity guaranteed by a pinned
sha256, not by trust in the registry
The cold state a pod boots from is deliberately minimal. The deployer materializes packs and
a runtime-config.json on the authoring host — but a cloud pod never receives those.
Its ConfigMap carries environment.json and nothing else. The demo reproduces this exactly
by deleting runtime-config.json and revisions/ before boot — byte-for-byte
the state a real pod wakes up in. That emptiness is what forces the pull.
03What happens when the worker boots
The core mechanism — greentic-start start --env local, step by step.
The pod's container command is just greentic-start start --env <id>. It reads the env store,
finds an empty runtime-config, and the boot seam takes over:
environment.json. There are no materialized packs on disk. The boot seam detects rc.revisions.is_empty().bundle_source_uri
bundle_source_uri says where to fetch the bundle from (HTTP / OCI / file).revision_pull::fetch_bundle_to_file
.gtbundle from the distributor. No early trust — just bytes on disk.bundle_digest
sha256 pinned at stage time is the integrity backstop. If the pulled bytes don't match, boot aborts — a tampered or wrong bundle can never activate.materialize_revision_from_bundle
runtime-config.json. Failure-atomic: an existing live revision is moved aside and rolled back on error, never half-bricked.runtime-config.json)./healthz → 200
serving probes only. That banner is the exact assertion the K8s E2E makes.
Why the digest matters more than the source. bundle_source_uri is an
opaque, caller-asserted reference — the worker will fetch whatever it points at.
Trust does not come from the URL; it comes from the bundle_digest pinned on the
revision at stage time. Breadth of the source ≠ injection risk, because the bytes are gated before
anything is unpacked.
04The commands — authoring the deployment
Every op verb the operator runs, in order, and what it records.
All authoring happens on the control-plane host with the deployer's op verbs.
Each verb writes to the env store (the thing that becomes the in-cluster ConfigMap). Nothing
touches a cluster until reconcile.
| Command | What it does |
|---|---|
env create |
Writes environment.json — the seed of the whole env. Becomes the gtc-env-store ConfigMap in-cluster. |
trust-root bootstrap |
Seeds the operator's Ed25519 signing key. bundles add refuses to run without it — packs are signed artifacts. |
bundles add |
Registers a deployment — a routing target. route_binding.path_prefixes:["/"] routes every request path to it. Returns a deployment_id. |
revisions stage M2 key step |
Records two things on the revision: • bundle_digest = sha256 of the local fixture → the integrity backstop• bundle_source_uri = where a remote worker will pull the bundle from (the M2 field). Returns a revision_id.
|
revisions warm |
Flips the revision lifecycle staged → ready. In-cluster this waits for the rollout to actually become healthy before proceeding. |
traffic set |
Routes traffic (weight_bps: 10000 = 100%). Only routed revisions get pulled, so this is what arms the boot-time fetch. Idempotency-keyed. |
env-packs add K8s only |
Binds the K8s deployer (greentic.deployer.k8s@1.0.0). This is what makes the env a Kubernetes env. Pins the runtime image for the rendered pods. |
env reconcile K8s only |
The only verb that touches the cluster. Server-side-applies all desired objects against the real API server (idempotent, converging). |
Authoring a pullable revision (Track 1 — loopback HTTP distributor)
# 1. env + trust root op env create # writes environment.json op trust-root bootstrap local # 2. a routing target op bundles add # route_binding.path_prefixes = ["/"] → deployment_id # 3. THE M2 STEP — stage a PULLABLE revision op revisions stage \ bundle_path=perf-smoke-bundle.gtbundle # → pins its sha256 as bundle_digest \ bundle_source_uri=http://127.0.0.1:8090/bundle.gtbundle # where to pull # 4. ready it, then send 100% of traffic (only routed revs are pulled) op revisions warm # staged → ready op traffic set # weight_bps = 10000 → revision_id
Three transports for bundle_source_uri
The worker fetches the bundle from whatever scheme the URI uses — the digest-gate is the same for all three:
| Scheme | Where the bundle lives | Used by |
|---|---|---|
http:// |
A plain HTTP distributor (e.g. python3 -m http.server, or an in-cluster busybox-httpd) |
Track 1 host-pull · kind-pull |
file:// |
A local file — no server needed at all | Running on a PC with only gtc dev |
oci:// new this week |
An OCI registry (ghcr, or an in-cluster registry:2). For a plain-HTTP registry the worker needs GREENTIC_OCI_INSECURE_REGISTRIES set — a hatch injected post-reconcile so the production render never carries it. |
The real demo (pulls from public ghcr) |
Extra verbs for the real demo
When the bundle is real (with a flow, a messaging endpoint, and secrets), a few more op verbs come into play — see section 09:
| Command | What it does |
|---|---|
env-packs add (secrets) | Binds greentic.secrets.dev-store@1.0.0 — required before secrets put. |
secrets put | Stores a value (a webhook secret, a Telegram bot token) in the env's dev-store. |
messaging endpoint add | Registers a messaging endpoint (e.g. messaging.telegram.bot) → endpoint_id. |
messaging endpoint link-bundle | Wires the endpoint to a bundle, so its webhook routes into that revision's flow. |
The fixture: perf-smoke-bundle.gtbundle (4 KiB, sha256:c5199a71…).
It's the one ready-made valid bundle in the workspace, and its pack carries no .wasm —
which is perfect here. It activates and serves /healthz, proving the
pull → materialize → activate chain, without needing a component runtime.
(A real request can't run a flow with this fixture — for that you need a bundle whose
pack ships .wasm components, which is exactly what the real demo does.)
05Two tracks — and what each one proves
The demo can run the same pull code in two places. ./demo.sh <cmd>.
| Track | What it runs | Real K8s? | Local status |
|---|---|---|---|
| 1 · host-pull | The M2 pull/serve code on the host, no cluster. Reproduces a pod's cold boot exactly — same pull → digest-gate → materialize → activate → serve code. | No (simulates the pod) | ✅ fully verified |
| 2 · kind-serve | A real worker POD boots start --env, reaches Ready, serves /healthz. Readiness probe = kubelet → pod. |
Yes | ✅ green locally |
| 2 · kind-pull | The full in-cluster pull: worker pod pulls (pod → pod) from an in-cluster busybox-httpd serving the bundle, logs the real-revision banner. | Yes | ⚠️ CI-only here |
The whole demo, end to end
cd /home/vampik/greenticai/my_demos/k8s-deploy-demo ./demo.sh host-pull # Track 1 — start here. ~10s, no Kubernetes needed. ./demo.sh host-stop # stop the worker it leaves running ./demo.sh kind-up # Track 2 — create a kind cluster + load the :develop runtime image ./demo.sh kind-serve # prove a worker POD boots + serves /healthz (local-green) ./demo.sh kind-pull # full in-cluster pull (CI-green) ./demo.sh kind-down # delete the cluster
Track 1 proves the code is correct
It runs the identical pull/materialize/activate/serve path, just outside a pod. The only thing it skips is the pod→pod transport. 100% reproducible on any machine.
Track 2 proves it works inside Kubernetes
Same objects, same pod boot command, same reconcile path as cloud (EKS/GKE/AKS). kind is just K8s running as Docker containers on your machine.
Why kind-pull is CI-only on this machine. The final hop is the worker pod pulling
pod → pod from an in-cluster httpd pod. On this host (CachyOS 7.0.11 + docker 29.5.3),
same-node pod-to-pod networking has 100% packet loss — a local kindnet/kernel defect, not a
code or test defect (kubelet → pod is fine, which is why kind-serve works).
The exact same test is green on CI's ubuntu-latest. Track 1's host-pull is the
local stand-in for precisely this path.
06Inside Kubernetes — control-plane vs data-plane
The split that holds true on real cloud too.
| What | Where it runs | Verb / command |
|---|---|---|
| Author the env create · env-packs add · bundles add · stage · warm · traffic |
Host / control-plane. You build the env store, which becomes a ConfigMap in-cluster. Nothing touches the cluster. | op … |
| Converge onto the cluster | Against the cluster. Server-side apply of every desired object onto the real API server. | op env reconcile |
| Boot, serve, pull | Inside the cluster. Worker / router pods boot start --env, serve /healthz, pull their bundles. |
the pod's container command |
Visually — what lives where during a K8s deploy:
CONTROL PLANE · your host
- env store on disk (
environment.json+ trust root) opverbs author desired stateop env reconciletalks to the API server- Holds the bundle
sha256+bundle_source_uri
- gtc-env-store ConfigMap — only
environment.json - worker pod boots
greentic-start start --env - Pulls bundle from
bundle_source_uri→ digest-gate → serve - Readiness probe hits the
/healthzit serves
The K8s-only delta vs the host track
# Bind the K8s deployer — THIS makes the env a Kubernetes env op env-packs add \ slot=deployer kind=greentic.deployer.k8s@1.0.0 pack_ref=builtin \ answers_ref=deployer-answers.json # pins runtime_image for the rendered pods # …author the revision exactly as Track 1… # The ONLY verb that touches the cluster: op env reconcile local # server-side apply → 14 objects # Watch a real Greentic worker come up inside Kubernetes: kubectl rollout status deployment/gtc-worker-<revid> -n gtc-local kubectl logs deployment/gtc-worker-<revid> -n gtc-local | grep "revision(s) for env" # SUCCESS: serving 1 revision(s) for env `local` (NOT "serving probes only")
The worker Deployment is named after the revision ULID: gtc-worker-<lowercased revision_id>.
Both router and worker pods boot the same start --env command.
07What reconcile applies — the 14 objects
12 env-level objects + 2 per warmed revision.
12 env-level objects
- Namespace
gtc-local - ConfigMap
gtc-env-store— carriesenvironment.json(with the routed revision + itsbundle_source_uri) - ConfigMap
gtc-runtime-config - Deployment
gtc-router - Service + PodDisruptionBudget for the router
- 6× NetworkPolicy — default-deny + scoped allows (incl. router-egress so the router can pull its bundle at boot)
+2 per warmed revision
- Deployment
gtc-worker-<revid>— bootsstart --env - Service for the worker
Archive the revision and re-reconcile → the worker pair is pruned back to the 12 env-level objects. Reconcile converges both directions; it's not append-only.
NetworkPolicy is load-bearing. The env's gtc-default-deny denies ingress to every pod,
and kindnet v1.31+ enforces it. That's why pulling needs an explicit allow — the router gets a
router-egress policy, and a worker pulling at boot needs a worker-egress allow. In real cloud the bundle
source is an external registry (outside the namespace, unaffected); in-cluster, the allow is the fixture's concern.
08How the local demo maps to real cloud
Every local stand-in, and its production counterpart.
| Demo (local) | Real cloud (EKS / GKE / AKS) |
|---|---|
kind create cluster |
The managed cluster you provisioned |
state/.greentic/…/environment.json |
The gtc-env-store ConfigMap mounted into the pod |
Stripping runtime-config.json + revisions/ |
The pod simply never receives them — the ConfigMap carries only environment.json |
python3 -m http.server on :8090 · in-cluster busybox-httpd |
The pack distributor / OCI registry the bundle_source_uri points at |
env-packs add binding greentic.deployer.k8s@1.0.0 |
The env declaring it deploys to Kubernetes |
op env reconcile local |
The operator / control-plane converging desired state onto the cluster |
Worker pod booting start --env |
Identical — the same container command |
The single production transport this machine can't exercise — a worker pod pulling over the cluster network — is exactly what CI covers, and what Track 1's host-pull substitutes for locally. Everything else in the chain runs unchanged from laptop to cloud.
09The real demo — a bundle + Telegram + Webchat new this week
Same K8s flow, upgraded from the empty fixture to a real worker that talks back. See RUN-REAL-FISH.md.
Everything so far proved delivery with the empty perf-smoke bundle. The new runbook
swaps in a real bundle — webchat-bot.gtbundle (~4.9 MB, a real flow + WASM
components) — pulled oci:// from public ghcr, and wires up two ways to actually talk to it:
a Telegram messaging endpoint and the built-in Webchat /chat GUI.
| Step | Probes-only (earlier) | Real (this week) |
|---|---|---|
| Bundle | none — probes-only revision | webchat-bot.gtbundle pulled oci:// from ghcr |
| Route | path_prefixes:["/"] | + tenant_selector tenant-default so /chat routes |
| Secrets | not bound | dev-store pack bound → secrets put for the bot token + webhook secret |
| Messaging | none | Telegram endpoint + link-bundle + a known webhook secret |
| Serves | /healthz, /status | + /chat GUI, /workers/invoke flow, /webhook/telegram |
Talking to the worker
Webchat, headless flow, and the Telegram webhook gate
# Webchat GUI — the welcome card renders on load xdg-open http://127.0.0.1:8080/chat # Headless — run the flow directly curl -s -X POST http://127.0.0.1:8080/workers/invoke -d '{...HostWorkerRequest...}' | jq . # Telegram webhook: correct secret-token -> 200, wrong -> 401 curl -s -o /dev/null -w '%{http_code}\n' -X POST http://127.0.0.1:8080/webhook/telegram \ -H "x-telegram-bot-api-secret-token: $WHVAL" -d '{"update_id":1,"message":{...}}' # 200
For a live bot, a cloudflared tunnel exposes the worker to the internet and
setWebhook points Telegram at <tunnel>/webhook/telegram — then messaging the bot
returns the welcome card.
The one wrinkle: bridging secrets into the pod
The K8s deployer renders only environment.json into the pod — not runtime secrets
(workload-identity / secret-store is deferred to Phase E). A real bundle needs its bot token and
webhook secret inside the pod, so the demo bridges the env's dev-store in:
- a Secret
gtc-dev-secretsbuilt from the dev-store file - an init container (
bridge-patch.yaml) that copies it into the worker's writableHOMEat the pathgreentic-startresolves
On real cloud, a proper secret-store / workload identity replaces this bridge.
Three things the pod-to-pod defect forces — and why they don't weaken the proof:
- the bundle is pulled
oci://from ghcr over the internet (pod→internet, healthy), not from an in-cluster httpd (pod→pod, dead here) - you reach the worker via
kubectl port-forward(kubelet→pod), not through the router (router→workerispod→pod) - runtime secrets are bridged in (above) rather than rendered
On a machine with healthy pod-to-pod networking the in-cluster pull and the router path work too — and CI exercises both.
A fully-scripted bash equivalent lives next door at ../k8s-telegram-webchat-demo/
(demo.sh + its own RUN-IN-K8S.md). RUN-REAL-FISH.md is the copy-paste fish version.
10The milestone ladder — now through M4
What a 200-with-a-flow-error meant, and how M3 + M4 closed it.
While the worker runs you can poke it. /status confirms it activated a real revision:
Probe the running worker
curl -s http://127.0.0.1:8080/status {"bundles_active":1,"deployments_routed":1,"env_id":"local","revisions_active":1,...} curl -s -X POST http://127.0.0.1:8080/ -H 'content-type: application/json' -d '{"text":"hi"}' # HTTP 200, body: [{"kind":{"kind":"custom","action":"response"}, "payload":{"metadata":{"error_kind":"flow_execution_failed", "error_message":"flow main has no start node",...}}}]
That POST is the honest edge of the fixture. The request reaches the
pulled-and-activated revision and is dispatched into its flow — the whole chain works —
but perf-smoke ships no real flow graph, so the flow itself fails with
flow main has no start node.
A useful 200 reply needs a bundle whose pack ships components (.wasm) — and a flow that
invokes them. That was M3, and it has now shipped (with M4 right behind it).
M2 proved the delivery; M3 + M4 prove the work — see the ladder below.
The ladder — all four shipped
/healthzstart --env, binds 0.0.0.0, reaches Ready. Probes-only. ✅ shipped (#332).http://, file://, and oci://. ✅ shipped (host + CI green; oci #346).templates-bundle fixture ships a real component-templates@0.6.0 WASM component + a one-node flow. POST /workers/invoke → 200, body contains component-templates::handle_message => … (the component ran). ✅ shipped 2026-06-19 (#345).gtc-ext-ingress namespace, modelling the real Gateway) goes through the env router to the component. ✅ shipped 2026-06-21 (#354).
Also new this week — a bound ServiceAccount identity. --bind now mints and binds a
deployer ServiceAccount at bootstrap, so a bound SA drives reconcile (dropping the
cluster-scoped Namespace permission) with a durable in-cluster home for the identity (#347–#349, #353).
That's the cloud-shaped auth model: the operator acts as a scoped identity, not cluster-admin.