Chapters on Jōkamachi Systems

Why Kubernetes?

Mon, 01 Jan 0001 00:00:00 +0000

Jōkamachi Systems has roots in functional programming and classic Linux systems administration. The natural gravitational pull of that combination is towards prescriptive tools — Terraform, NixOS, terranix. You declare how the world ought to be, run the apply, and the tool drives reality towards your specification. Apply once. Done.

This works beautifully when reality cooperates by holding its shape.

The polite fiction of a snapshot

In What functional programmers get wrong about systems (2026), Ian Duncan observes that

Overview

Mon, 01 Jan 0001 00:00:00 +0000

This booklet covers how a Jōkamachi cluster is laid out: control plane, worker pools, multi-tenancy, isolation between customers, and the deliberately boring networking choices that hold the whole thing together.

The booklet you read before you trust us with your workloads.

Overview

Mon, 01 Jan 0001 00:00:00 +0000

This booklet introduces the technologies that make a Jōkamachi cluster run: Kubernetes, the container runtime, the network plane, storage, ingress, and the supply-chain pieces around them.

Written for someone who has heard of Kubernetes but has never had to operate one.

Overview

Mon, 01 Jan 0001 00:00:00 +0000

The booklet covers, with numbers:

Hetzner Load Balancer vs. custom NAT gateway — when each is the right choice for your traffic mix, with throughput and latency measurements under sustained load.
Shared, dedicated, and performance CPU cores — what each costs, what each delivers, and where the boundaries actually live under contention from a noisy neighbour.
Storage and network throughput between Hetzner cloud VMs, dedicated servers, and the connection between them in the same datacentre.
End-to-end SLO numbers for a reference workload running on each configuration we sell.

Overview

Mon, 01 Jan 0001 00:00:00 +0000

This booklet walks you from a fresh Hetzner account to a working Jōkamachi Kubernetes cluster you operate yourself. No prior Kubernetes experience assumed.

If you’re evaluating whether Kubernetes is the right shape for your team at all, start with the introduction booklet first — it covers the why before this booklet covers the how.

Why Kubenix?

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes runs whatever spec you hand it; the question this chapter answers is what kind of artefact the spec should be. The short answer is: not a pile of Helm charts.

Helm charts are the Kubernetes ecosystem’s standard way to ship a service. Each chart, in isolation, works fine. But a real cluster is not one service; it’s many services, often inter-dependent, often sharing config — DNS names, certificates, secrets, network policies, RBAC bindings — that no individual chart owns.

You don't need KAOS!

Mon, 01 Jan 0001 00:00:00 +0000

The motivation behind KAOS is to derive and automate collective operating experience. But “collective experience” is built one incident at a time, and every individual contribution starts the same way: someone — a human, or a robot — runs kubectl against the cluster, or receives an alert notification, notices something is off, and fixes it.

Real experience is earned through trial. You can run those kubectl commands yourself, or you can hand them to an agentic CLI with access to the cluster — either path produces experience. Each path leaves a different gap, though. Humans produce organisational knowledge, but bring it with them when they leave the organisation. Standalone agentic CLIs don’t retain knowledge across sessions without extensive prompting. And human operators of agentic CLIs tend to overlook the solution entirely — because the agent did the fixing, and you only learn through pain.

How far one prompt takes you

Mon, 01 Jan 0001 00:00:00 +0000

If you’ve never tried this: open a terminal against a cluster you have admin on, start whichever agentic CLI your team already trusts (Claude Code, Aider with a shell tool, an in-house equivalent), and type the following prompt — exactly the following prompt, with no system message, no playbook, no pre-defined toolbelt:

❯ Are there any incidents in the Kubernetes cluster? Use kubectl, it's already authenticated.

A typical frontier model, in a typical agentic CLI, will respond with something close to this:

How an agentic skill is formed

Mon, 01 Jan 0001 00:00:00 +0000

Frontier models, given access to a cluster, will read the event log, walk every namespace, and recognise an error pattern when they see one. They do this without much prompting — the ability is built into the model. What they don’t have, out of the box, is incident- management context. They don’t track timelines, they don’t know whether an issue is ongoing or resolved, and they don’t remember whether the same pattern was seen and fixed last Tuesday.

Property testing Kubenix

Mon, 01 Jan 0001 00:00:00 +0000

The chapter covers the property-testing techniques we apply to Kubenix-defined clusters: invariants that span services, the test runner setup, examples of properties we ship by default, and how to add tenant-specific properties.

A property test for a cluster looks much like a property test for any other piece of typed code: state an invariant in the type system or in code, run a generator that produces representative cluster states, and check that the invariant holds for every generated state. The difference is the substrate — the “code” being tested is the desired state of a cluster, not a function — and the value of the technique scales with how much your cluster’s correctness depends on relationships you can’t see by reading any one chart.

Guardrails and incident maturity

Mon, 01 Jan 0001 00:00:00 +0000

KAOS is never granted blanket autonomy. Its authority is scoped per class of incident, and each class is promoted through three states as the playbook matures:

Propose — KAOS describes what it would do; a human reviews and takes the action manually.
Approve — KAOS proposes an action with the exact change prepared; a human signs off and KAOS applies it.
Act — KAOS handles the incident end-to-end, logging the decision; a human reviews after the fact.

Promotion happens incident-class by incident-class, only after the class has been resolved enough times the same way that “same way” is a fact, not a guess.

What KAOS sees, and what it doesn't

Mon, 01 Jan 0001 00:00:00 +0000

Training on incidents

KAOS gets better the more incidents it has seen. We train on incidents we resolve across all customers to improve the substrate for every customer; the alternative is that each customer’s KAOS starts from scratch, which would be a worse product. We don’t pretend otherwise.