nixcluster

• What you get
• Start from a scenario template
• How converge orders itself
• Cluster modules
• Writing a cluster by hand
• CLI
• Checks
• Architecture
• Known limitations
• License

Each template is a complete, evaluating cluster with a README naming exactly which values to change.

mkdir my-cluster && cd my-cluster
nix flake init --template github:kitsunoff/nixcluster#k3s-ha

# Edit modules/nodes/*.nix (addresses) and modules/base.nix (disk, SSH key):
nix run .#cluster-k3s-ha -- converge --dry-run   # plan only, nothing touched
nix run .#cluster-k3s-ha -- converge

Roles in the node files are the only input. Nothing in the cluster definition says "three servers":

$ nix eval .#clusterConfigurations.k3s-ha.convergeOrder
["member-server-1","member-server-2","member-server-3","k3s.bootstrap","k3s.kubeconfig","member-agent-1","member-agent-2","k3s.prune"]

$ nix eval .#clusterConfigurations.k3s-ha.config.converge.steps \
    --apply 'steps: builtins.mapAttrs (_: s: s.deps) steps'
  k3s.bootstrap          <- member-server-1, member-server-2, member-server-3
  k3s.kubeconfig         <- member-server-1, member-server-2, member-server-3
  k3s.prune              <- member-server-1, member-server-2, member-server-3, member-agent-1, member-agent-2, k3s.kubeconfig
  member-agent-1         <- member-server-1, member-server-2, member-server-3
  member-agent-2         <- member-server-1, member-server-2, member-server-3
  member-server-1        <- (nothing)
  member-server-2        <- member-server-1
  member-server-3        <- member-server-1

Read that as: server-1 bootstraps etcd, so server-2 and server-3 wait for it and not for each other; each agent waits for all three servers, because there is no control plane to join until they are up; the kubeconfig needs a server and does not care about agents; and pruning happens last, once the desired members have converged.

A step whose dependency did not succeed is skipped, not run — a failed bootstrap does not drag the agents through a pointless join attempt.

preSteps / postSteps still exist as sugar over the same attrset, carrying the dependencies their names implied.

Nothing is auto-applied: a cluster imports what it wants.

nixcluster.prod =
  { clusterModules, self, ... }:
  {
    imports = [ clusterModules.k3s clusterModules.sops ];

    k3s.enable = true;
    sops.enable = true;

    # Members with no nixosConfiguration of their own inherit this — which is what
    # lets a member be supplied as pure data (an address and nothing else).
    defaultNixosConfiguration = self.nixosConfigurations.base;

    members.node-1 = {
      install.ip = "10.0.0.11";
      install.disk = "/dev/vda";
      k3s.role = "server";
      networking.hostName = "prod-node-1";   # any NixOS option is a patch
    };
  };

nix run .#cluster-prod -- --help
nix run .#cluster-prod -- converge [--dry-run] [--json] [--age-key-file path]
nix run .#cluster-prod -- apply node-1          # switch one member
nix run .#cluster-prod -- install node-1        # first-time provisioning
nix run .#cluster-prod -- gen-config 10.0.0.12  # SSH-discover a node, emit a member file
nix run .#cluster-prod -- k3s nodes             # module command groups

converge writes human output to stderr and exactly one machine-readable line to stdout, so a consumer gets a single marker to grep:

::nixcluster:result:: {"cluster":"prod","result":"success","members":[
  {"name":"node-1","ip":"10.0.0.11","action":"switch","status":"Applied","durationSeconds":47},
  {"name":"old-node","ip":"","action":"removed","status":"Applied","message":"removed from the k3s registry (unreachable)"}],
  "steps":[{"name":"sops.gen","phase":"pre","status":"ok"},{"name":"k3s.prune","phase":"post","status":"ok"}]}

Both are evaluation-only and run in CI on every push and pull request.

$ scripts/check-templates.sh
24 check(s) passed, 0 failed

$ scripts/check-prune.sh
38 check(s) passed, 0 failed

check-templates.sh locks every registered template against the checkout and asserts the expected members, a NixOS configuration per member, the per-cluster CLI app, the resolved converge order, and the converge steps with their dependencies — so a template whose plan drifts from its documented contract fails loudly. It also evaluates every member's system.build.toplevel, which catches option and type errors across the whole configuration, while building nothing.

check-prune.sh covers the only part of converge that deletes things: the engine's branches against an injected registry (no-op, one stale member, unreachable host → force path, empty desired set → loud refusal with nothing removed, quorum-breaking removal → refused) and the real k3s step against stubbed kubectl and ssh.

nixcluster.<name>                     one definition (merged across ./modules by import-tree)
  │
  ├─ core.nix                         members, converge DAG, CLI plumbing
  ├─ clusterModules.*                 explicitly imported; each contributes
  │                                   NixOS modules, CLI commands, converge steps
  ▼
clusterConfigurations.<name>
  ├─ nixosConfigurations.<name>-<member>     one per member
  ├─ convergeOrder                           resolved topological plan
  └─ cli  →  apps.<system>.cluster-<name>    the per-cluster CLI

Every unit of converge work — a module step and each member's install-or-switch — is a node in converge.steps. lib/convergePlan.nix topologically sorts them and fails evaluation on an unknown dependency, a cycle, or a step that is neither a run nor a member. lib/prune.nix holds the registry-diff engine all pruning modules share, so the dangerous logic exists exactly once.

• Pruning has never run against a live cluster. The engine and the k3s step are covered by stubbed tests; the Incus path is verified only by evaluation.
• nebula and keepalived do not prune. Nebula revocation is only enforced through pki.blocklist, which is deliberately not distributed by lighthouses, so it needs a persisted blocklist wired through the NixOS module.
• Per-member converge status is coarse. The result JSON reports one action per member, not a step-by-step trace.
• converge builds where it runs. There is no --build-host, so driving it from a machine that cannot build the members' system needs a configured remote builder.

MIT. See LICENSE.