One small cluster for every app I ship

Every side project used to get its own VPS. Its own nginx, its own certs, its own “how did I deploy this again”. Five projects in, I had five slightly different setups and no idea which ones were still healthy.

So now everything runs on one small Kubernetes cluster. It costs about €47 a month, has zero open ports, and adding an app is one block of Terraform. Here’s how it fits together, with the real config.

The shape

Here’s the whole thing. Requests go left to right along the top; everything below is how I see what’s happening.

Grafana sits behind Access too, at its own hostname. Same route as any app.

The cluster is one Terraform module

I didn’t hand-roll k3s. kube-hetzner is a Terraform module that builds the servers, the network, the firewall and the cluster, on an OS that patches itself. My config is mostly a list of nodes:

module "cluster" {
  source  = "kube-hetzner/kube-hetzner/hcloud"
  version = "x.y.z"   # pin it

  control_plane_nodepools = [
    { name = "control", server_type = "cx33", location = "fsn1", count = 1, labels = [], taints = [] }
  ]
  agent_nodepools = [
    { name = "agent", server_type = "cx43", location = "fsn1", count = 2, labels = [], taints = [] }
  ]

  ingress_controller = "traefik"
  enable_cert_manager = false   # TLS ends at Cloudflare

  # Never let a Hetzner load balancer appear (see below).
  enable_klipper_metal_lb = true

  # The only traffic the firewall needs to allow is cloudflared dialling out.
  extra_firewall_rules = [
    {
      description     = "cloudflared -> Cloudflare edge (QUIC)"
      direction       = "out"
      protocol        = "udp"
      port            = "7844"
      destination_ips = ["0.0.0.0/0", "::/0"]
    },
  ]
}

Going to three control-plane nodes later means changing count = 1 to 3 and running terraform apply.

No open ports

Journey one: a request.

Normally you’d point DNS at a server, open 443 and manage certificates. Here, a small program called cloudflared runs inside the cluster and opens an outbound connection to Cloudflare. Requests for my hostnames travel back down that connection. Nothing on the internet can start a connection to the node, because nothing is listening.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cloudflared
  namespace: cloudflare
spec:
  replicas: 2   # one restarting never drops every route
  template:
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                topologyKey: kubernetes.io/hostname
                labelSelector: { matchLabels: { app.kubernetes.io/name: cloudflared } }
      containers:
        - name: cloudflared
          image: cloudflare/cloudflared:<pinned version>
          args: [tunnel, --no-autoupdate, run]   # the tunnel token comes from a Secret

The routes aren’t in this file. The tunnel keeps its config in Cloudflare, so adding a hostname is one API call and cloudflared keeps running untouched. Anything without a specific route falls through to Traefik, which reads ordinary Kubernetes Ingress objects. So an app just ships a normal Ingress.

Two Cloudflare products, two jobs. Tunnel is how a request gets in. Access is who’s allowed to: it puts a login (an emailed code) in front of a hostname, so a designer can see a preview without me creating an account anywhere.

Adding an app is one block

Journey two: shipping something.

An app lives in its own repo with its own Dockerfile, manifests and CI. The infra repo only decides the things that were never the app’s call: its namespace, what its CI may touch, its hostname and who can open it. That’s one module block per app:

module "portfolio" {
  source = "./modules/app-onboarding"

  name = "portfolio"
  repo = "gthanasis/portfolio"
  tier = "app"

  # Public on purpose: it's a portfolio. Leave access_emails empty only when that's true.
  hostname = "gthanasis.com"
  zone_id  = local.zones["gthanasis.com"]

  cloudflare_account_id = local.cloudflare_account_id
  tunnel_id             = local.tunnel_id
}

terraform apply turns that into:

• a namespace, labelled so the network policies and security scans know it’s an app
• a CI identity that can deploy into that namespace and nothing else. It can’t read secrets, and it can’t see other apps.
• a Cloudflare service token that lets that app’s CI, and only that app’s CI, through to the cluster API
• a DNS record pointing the hostname at the tunnel
• an Access application with the allowed emails, when the app isn’t public

A script then pipes the credentials straight into the app repo’s GitHub secrets. Nothing is printed, because anything echoed to a terminal lives on in scrollback and history.

Deploying from CI without exposing the API

The cluster API isn’t on the internet either. That’s a problem for GitHub Actions: runners have no fixed IP to allow through a firewall.

The fix is the same trick as the front door. The runner uses cloudflared to open an authenticated tunnel to the API, using its service token, and talks to it on localhost:

- name: Open a tunnel to the cluster API
  env:
    TUNNEL_SERVICE_TOKEN_ID: ${{ secrets.CF_ACCESS_CLIENT_ID }}
    TUNNEL_SERVICE_TOKEN_SECRET: ${{ secrets.CF_ACCESS_CLIENT_SECRET }}
  run: |
    ./cloudflared access tcp --hostname "$CLUSTER_HOSTNAME" --url 127.0.0.1:6443 \
      --service-token-id "$TUNNEL_SERVICE_TOKEN_ID" \
      --service-token-secret "$TUNNEL_SERVICE_TOKEN_SECRET" &
    for i in $(seq 1 30); do nc -z 127.0.0.1 6443 && exit 0; sleep 1; done
    exit 1

- name: Deploy
  run: |
    kubectl config set-cluster pilot --server=https://127.0.0.1:6443 --certificate-authority=ca.crt
    kubectl config set-credentials ci --token="$KUBE_TOKEN"
    kubectl apply -k deploy/k8s
    kubectl rollout status deploy/portfolio --timeout=180s

TLS still checks out end to end: the tunnel forwards raw TCP, and 127.0.0.1 is in the API certificate. The rollout status line matters too. If the new pods never become ready, the build goes red. Otherwise a broken deploy could sit in a crash loop while CI reports success.

Seeing everything without registering anything

Nothing in the infra repo holds a list of apps. Prometheus watches every namespace. Alloy tails every container on every node. Grafana loads dashboards from every namespace. An app becomes visible by existing.

What an app does to get good data out is mostly labels:

template:
  metadata:
    labels:
      app.kubernetes.io/name: portfolio   # logs now say app="portfolio"

If it exposes metrics, a ServiceMonitor in its own namespace is picked up automatically. If it ships a dashboard, a ConfigMap labelled grafana_dashboard: "1" is loaded. Logs need nothing at all: the moment a pod starts, {namespace="portfolio"} works in Grafana.

Severity gets pulled out of each line once, when it’s collected, and stored as a label. Grafana can then colour rows and filter by level without searching text. Any of these formats works:

A line that merely contains the word “error” is labelled unknown, which is the right answer. An earlier version read the level out of a startup banner that said Log level: info.

Knowing when it’s down

Journey three: something breaks.

Alert rules run in Prometheus and Alertmanager sends them to Telegram. Pods crash-looping, a disk filling up, a backup that hasn’t succeeded in a day, a new critical CVE with a fix in a public app’s image.

There’s a hole in that, though. Prometheus runs on the same cluster it’s watching. If the cluster goes down, the thing that would tell me goes down with it, and silence looks exactly like everything being fine. On 28 August that silence lasted 6 hours 41 minutes.

So the last check lives outside it. Alertmanager has an alert that’s always firing, on purpose, and sends it every minute to a small Cloudflare Worker. The Worker remembers when the last one arrived and checks every five minutes:

async function heartbeat(env) {
  const now = nowSeconds()
  const staleAfter = Number(env.STALE_AFTER_SECONDS ?? 300)
  const last = Number((await env.HEARTBEAT.get('last-ping')) ?? 0)
  if (!last) return // never pinged: a fresh deploy hasn't lost anything yet

  const outageSince = Number((await env.HEARTBEAT.get('outage-since')) ?? 0)
  const age = now - last

  if (age > staleAfter && !outageSince) {
    const sent = await notify(env, `Cluster heartbeat lost. No ping for ${duration(age)}.`)
    // Only remember the outage once the message actually went out.
    if (sent) await env.HEARTBEAT.put('outage-since', String(last))
  }
}

All it knows is that the whole chain stopped answering. That’s also the one thing the in-cluster alerts can never report about themselves.

The same Worker runs a second check every hour: is what’s merged on main actually running? Between 29 August and 21 September the deploy workflow couldn’t reach the cluster, and nothing complained. The cluster kept serving its last good config, so every alert stayed quiet. Three weeks of merged changes simply never arrived. Now the Worker asks GitHub, from outside both, and pings me when they disagree.

What it doesn’t do

It’s a small setup and it has real gaps. I’d rather know them than find them.

For what I run on it, that trade is worth it. One place to deploy, one place to look, and nothing listening on the internet.

Want something like this for your team? Let’s talk.