Kurrent Deployment Guide

Covered roles: controller server + storage node — co-located on one machine for self-testing, or split across servers and scaled out (one Agent per storage server). Steps below follow the two roles; for co-located self-testing, run them in order on the same machine.

This guide follows Kurrent's declarative-configuration model: the control-plane declaration control_plane/kurrent.yaml is generated as a template by kurrent config print init-defaults, edited, then validated and converged by kurrent init; the storage-node declaration storager/kurrent.yaml is generated by kurrent config print node-defaults, edited, then applied by kurrent join. The yml is the only input — the CLI validates and converges, and container orchestration stays an internal detail.

Topology and Port Planning

┌─────────────────────┐         ┌─────────────────────┐
│  Controller server  │         │  Storage node       │
│  kurrent-dnsmasq    │   LAN   │  storager-agent     │
│   (DHCP/TFTP 67/69) │◄───────►│   (HTTPS 4840)      │
│  kurrent-control-   │         │  kurrent-nvmet-host │
│   plane (loopback)  │         │   (NVMe/TCP 4421)   │
│  kurrent-webui      │         └──────────┬──────────┘
│   (443, only entry) │                    │ storage btrfs/xfs
└─────────┬───────────┘         ┌──────────▼──────────┐
          │ PXE network         │  Bare-metal clients │
          └────────────────────►│  (Devices)          │
                               └─────────────────────┘

Example network plan (replace with your own):

Item	Value	Note
PXE subnet	192.168.80.0/24	The serving subnet the controller binds (DHCP/TFTP/boot traffic)
DHCP pool	192.168.80.50 – .100	Handed out to bare-metal clients
Gateway / DNS	192.168.80.2 / 223.5.5.5	DHCP options 3 / 6
Controller IP	192.168.1.10	LAN management address (WebUI / API / storage-node callback)
Storage node IP	192.168.1.20	How the controller reaches the Agent (4840)
Storage hostname	storage-01	Any name (node identity is auto-generated by join, decoupled from the hostname)

Ports:

Port	Protocol	Service	Where
67 / 69	UDP	DHCP / TFTP (host network)	Controller server
80	TCP	HTTP → 443 automatic 301 upgrade (carries no content or credentials)	Controller server
443	TCP	The only entry: HTTPS (WebUI + /api/cp reverse proxy to the control plane + component enroll/renew + boot chain)	Controller server
4840	TCP	Storage Agent HTTPS (controller → Agent)	Storage node
4421	TCP	NVMe/TCP data plane, single port with TLS 1.3 (shared by the firmware and kernel boot stages)	Storage node

nvmet-host's management plane has no TCP port: the Agent and nvmet-host talk over the shared-volume unix socket /run/kurrent/nvmet/nvmet-host.sock (mTLS, directory mode 0700), unreachable from the LAN.

The boot chain behind 443 only proxies three read-only endpoints: /boot-vars, /devices/report and /devices/challenge. The remaining /devices/* management endpoints are not exposed. The control-plane management plane publishes no TCP port; it listens on an in-container unix socket (/run/kurrent/cp.sock) that nginx reaches through a shared volume, and the container health check uses the same socket. nginx clears the identity headers by default for every proxied location and injects DN/issuer/serial/fingerprint only when mTLS verification succeeds on /api/cp/. The interactive control-plane API docs (/openapi.json, /docs) are off by default and can be enabled for debugging with KURRENT_CP_OPENAPI=1.

Boot network boundary (L1 trust profile)

The first hop of the boot chain (DHCP → TFTP snponly.efi → TFTP boot.ipxe) has no cryptographic anchor available: the device firmware does not verify it, and iPXE has not yet obtained the controller certificate. The current trust profile is L1: treat the boot segment itself as the trust boundary, keeping device-identity TOFU (NVRAM key + registration window + binding) and server TOFU on top. A deployment must satisfy:

• Physical/logical isolation: the PXE segment is a dedicated VLAN or a dedicated physical port, not shared with the office network or the storage data plane; the controller is the only host serving DHCP and TFTP on it.
• Layer-2 controls: enable DHCP snooping (only the controller uplink is trusted), Dynamic ARP Inspection, RA guard, and port security/MAC limits on access switches; use 802.1X or MAB to restrict which ports can attach where available.
• Admission control: open the registration window only when needed and set a quota; once closed, unknown MACs cannot enter the pool.
• Revocation closed loop: DELETE /devices/{mac} writes MAC,ignore into the dnsmasq binding file, so a revoked device gets no lease; re-registration (/devices/report) or purging (DELETE /devices/revoked) removes that line.
• Minimal TFTP surface: the TFTP root mounts only snponly.efi and boot.ipxe; configuration and menu are fetched over HTTPS.

Residual risk (explicitly out of scope for L1): an attacker on the same Layer 2 (or on a host in the same VLAN as a compromised port) can still forge DHCP/ARP, answer ahead of the controller, and replace the first-stage binary or pointer, then use the device's NVRAM key as a signing oracle to fetch the DHHC-1/PSK from /boot-vars. The same holds if the controller or TFTP service itself is compromised. The device key is a bearer credential: the controller authenticates "a request holding this key", not "the code that runs with this key". Eliminating this requires L2 (Secure Boot + self-signed iPXE + a pinned trust anchor) or L3 (TPM measured boot/remote attestation), neither of which is in the current implementation.

Prerequisites

Linux servers (controller and storage-node hosts, x86_64; verified on Debian 12/13 and Ubuntu 24.04/26.04; co-located for self-testing):

# 1. Docker with Compose v2
docker --version && docker compose version

# 2. Storage node: nvmet host prerequisites (kernel modules + configfs + persistence; idempotent)
sudo bash storager/nvmeof/host-prepare.sh
#    Read-only diagnosis (no root, for pre-deployment checks): bash storager/nvmeof/host-prepare.sh --check

# 3. Storage node: NVMe/TCP TLS prerequisites — the kernel needs CONFIG_NVME_TCP_TLS /
#    CONFIG_NVME_TARGET_TCP_TLS / CONFIG_NVME_KEYRING, and tlshd must be running
grep -E 'CONFIG_NVME_(TCP_TLS|TARGET_TCP_TLS|KEYRING)' /boot/config-$(uname -r)
sudo apt install -y ktls-utils        # Debian/Ubuntu; use the equivalent package elsewhere
sudo systemctl enable --now tlshd

# 4. Storage node: btrfs or xfs is recommended for the data disk (both support
#    reflink/FICLONE, so golden-image clones are instant; ext4 has no reflink and
#    falls back to full copies, clone time growing linearly with image size)
#    Mount example: sudo mkfs.btrfs /dev/sdb && sudo mount /dev/sdb /pool1

host-prepare.sh only checks the nvmet modules and configfs; the TLS kernel options and tlshd from step 3 must currently be verified by hand. If the 4421 port cannot be created, the node reports the port as not ready and the control plane refuses the boot — there is no plaintext fallback.

Host baseline (a precondition for the auth model): read-only by default — checks public TCP listeners, Docker exposure and credential-file permissions; the script does not modify system state.

bash control_plane/host-baseline.sh

• A controller host should expose only 22 (ops), 53/67/69 (dnsmasq) and 80/443 (nginx); storage nodes add 4840/4421
• nvmet-host's management plane uses a shared-volume UDS (directory 0700) and creates no TCP listener; --fix-4841 is no longer needed
• ufw does not filter Docker-published ports: constrain them in the DOCKER-USER chain (the script warns when ufw is active but that chain is empty)
• The controller container does not mount docker.sock (dnsmasq watches dhcp-hostsfile; nginx reloads itself via an in-container watcher)
• CA private keys, users.yml, sessions.yml, admin.conf and audit.jsonl should be mode 600
• Treat FAIL items as blockers: host root or a compromised nginx process is equivalent to a compromised management plane

Step 1: Controller Server

1.1 Get the repo and install the kurrent CLI

git clone https://github.com/dutyc/kurrent && cd kurrent

# CLI without a local build: grab the release binary (shipped since v0.3.0; Linux amd64/arm64, Windows)
curl -sL -o cli/kurrent https://github.com/dutyc/kurrent/releases/download/v0.3.0/kurrent-linux-amd64
chmod +x cli/kurrent
sudo install -m 0755 cli/kurrent /usr/local/bin/kurrent   # global install (the rest of this guide uses plain `kurrent`)
kurrent version   # verify

Versions move with the Releases page (https://github.com/dutyc/kurrent/releases; Windows: kurrent-windows-amd64.exe). Alternatively build from source instead of downloading (cd cli && go build -o kurrent ., needs Go 1.27+).

1.2 Generate the control-plane declaration

The declaration file is control_plane/kurrent.yaml (the yml is the only input — the CLI only validates and starts). Generate the annotated template, then edit it for your environment:

kurrent config print init-defaults > control_plane/kurrent.yaml   # generate template
vim control_plane/kurrent.yaml        # fill spec.networking (mandatory five keys: NIC/subnet/DHCP pool/gateway/DNS)

The five spec.networking keys are deployment facts and must be filled in explicitly (the template ships example values and comments); the remaining blocks (component PKI policy, TOFU cert, boot behavior, …) carry defaults — leave them as-is.

1.3 Initialize and bring up the control plane

One command validates and converges to the running state (declaration-driven config, CLI as tooling; container orchestration is an internal detail):

kurrent init        # validate kurrent.yaml → start/restart the control plane → wait /healthz → restart dnsmasq to load the new conf

On startup the controller automatically: renders dnsmasq/dnsmasq.conf from spec.networking (yml is authoritative, conf is derived — do not hand-edit it), and generates the TOFU server cert and component PKI. kurrent init is idempotent — re-run it after editing the declaration and the change takes effect.

1.4 Verify

curl -k https://127.0.0.1/healthz           # {"status":"ok"} (via 443; the management plane has no TCP port)
head -5 dnsmasq/dnsmasq.conf                # interface=ens33 ... (rendered from yml)

Open https://<controller-ip>/ in a browser (WebUI; the only entry on 443 — plain HTTP is 301-redirected).

CLI credentials: setup has already bootstrapped control_plane/state/admin.conf (kurrent-native flat YAML: entry URL plus the admin client certificate and its trust anchor, 0600). The CLI discovers it in the order --credentials > KURRENT_CREDENTIALS > auto-discovery (walk up from the current directory to control_plane/state/admin.conf → the repo layout next to the executable → the user-level snapshot ~/.kurrent/admin.conf) — works out of the box inside the repo (admin cert over 443 mTLS). Outside the repo, run kurrent credentials install once to create the snapshot (re-run after credential rotation); --server overrides the entry URL in the file. See API docs 1.3 for the authenticator chain / sessions / RBAC.

WebUI sign-in (either one):

1. Management token (login page): issue with kurrent token create --usage admin, paste it on the login page → an HttpOnly session cookie is issued (12-hour TTL, sliding renewal only in the second half; survives refresh and browser restarts; sign out from the sidebar footer).

2. Admin certificate (automatic): convert the admin certificate to PKCS#12 and import it into the system trust store — the browser then authenticates automatically over mTLS, no login needed:

   openssl pkcs12 -export -out kurrent-admin.p12 \
     -inkey control_plane/state/pki/users/admin/client.key \
     -in control_plane/state/pki/users/admin/client.crt
   # Double-click kurrent-admin.p12 to import (Windows/macOS system store;
   # Firefox keeps its own store and needs a separate import)

Firewall (controller server):

sudo ufw allow 67,69/udp && sudo ufw allow 80,443/tcp && sudo ufw reload

1.5 Issue the storage-node bootstrap credentials

kurrent token create --agent-id node-01 --cp-url https://192.168.1.10
# omitting --agent-id generates a random 12-hex node identity; the command carries it either way

--cp-url must be the HTTPS entry of the control plane as reachable from the storage node (port 443 /api/cp reverse proxy — Agents enroll/renew through it; derived from the --server host otherwise). --agent-id is the node identity (= the declaration metadata.name), and the token is bound to it. Output resembles (the join command carries the address and node id, and is ready to paste):

bootstrap token: a1b2c3.d4e5f6a7b8c9d0e1（expires ...，绑定节点 node-01，0/1）

# Run on the bound node (--agent-id is written into the declaration metadata.name):
kurrent join https://192.168.1.10 --token a1b2c3.d4e5f6a7b8c9d0e1 --agent-id node-01
# After joining, verify: kurrent agents list (or WebUI → Agents, health=ok)

The token is bound to the node identity and single-use by default: enroll only accepts that agent_id; an allowance is reserved atomically before issuance and released if issuance fails. TTL defaults to 60 minutes with a 24-hour cap (spec.pki.bootstrapTokenTtlMinutes); for bulk onboarding use kurrent token create --agent-id ... --uses N (max 100). Re-issue after expiry or revocation. No nvmet-host credential needs issuing: it is derived by the control plane when the agent enrolls with backend=nvmet.

1.6 Component certificate revocation (F-11)

Component channels (443 enroll, 4840 inbound, control-plane outbound, agent → nvmet-host over the UDS) perform application-level revocation checks on top of TLS chain validation:

• The control plane generates two component CRLs: state/pki/nginx/nodes.crl (nodes CA, for peer components) and control.crl (control CA, for the control-plane client); nginx separately uses clients.crl for management handshakes.
• Components fetch these with their own client cert from /api/cp/enroll/crl and refresh every 60 seconds by default (KURRENT_COMPONENT_CRL_REFRESH_SECONDS); control-plane outbound and agent → nvmet-host load nodes.crl on every connection and check nextUpdate.
• Fail-closed: a missing or stale CRL refuses the connection outright — no degraded allow. nodes.crl / control.crl carry a 30-day nextUpdate to tolerate a long control-plane outage.
• Revocation entry points: DELETE /api/cp/pki/components/{agent_id} (nodes domain) and DELETE /api/cp/pki/control/{cn} (control domain), both accepting serial to target a single certificate.
• Upgrade order: upgrade and restart the control plane first (it generates the CRLs and opens the distribution endpoint), then rebuild the component containers. Older components that do not know /enroll/crl keep their previous chain-only behavior and do not refuse to start.
• Revocation takes effect at the component's next refresh after the control plane writes the CRL (≤60 s), with no component restart.
• Startup self-check: a revoked certificate cannot complete the /enroll/crl mTLS handshake, so components fetch the public read-only GET /crl/nodes.crl (unauthenticated, GET/HEAD only) at startup, verify its signature against the local nodes CA, and compare the client / serving serials; a hit re-enrolls with the bootstrap credential (KURRENT_PKI_SELF_CHECK=0 disables the check).
• Management UI: the WebUI "Certificates" page shows expiry and status across the users / nodes / control domains and can revoke user and node certificates; an abnormal node row also offers "Recover".

Verification:

bash tests/deploy/probe_edge.sh      # includes component CRL distribution, public nodes CRL, control CA ledger, and human-cert 403 assertions
docker logs storager-agent 2>&1 | grep 'crl:'          # initial fetch ok (revoked counts / next_update)
docker exec kurrent-nvmet-host python /app/healthcheck.py; echo $?   # 0

1.7 Recovery after component certificate revocation

Revocation also deletes the node's bootstrap tokens, so recovery requires a newly issued credential. The old local certificates do not need manual deletion.

1. Control plane: WebUI "Certificates" → "Storage nodes" → "Recover", or kurrent token create --agent-id <id> --cp-url https://<cp-host>. The WebUI pre-fills the control-plane URL and shows a single-use token plus the join command after issuance (plaintext visible once).
2. Node: run the printed kurrent join <cp-url> --token <token> --agent-id <id>.
3. Startup self-check: the agent finds the local client / serving serials in the nodes CRL, re-enrolls with the new token, and overwrites the local certificates; join then waits for the derived credential and restarts nvmet-host automatically.

Verification:

kurrent agents list                       # node health=ok
docker logs storager-agent 2>&1 | grep 'pki:'   # re-enroll / self-check records
docker logs kurrent-nvmet-host 2>&1 | grep 'pki:'

Step 2: Storage Node

2.1 Prepare

git clone https://github.com/dutyc/kurrent
cd kurrent
mkdir -p storager_img        # disk image dir (kurrent join default; custom path in 2.3)
curl -sL -o cli/kurrent https://github.com/dutyc/kurrent/releases/download/v0.3.0/kurrent-linux-amd64 && chmod +x cli/kurrent
sudo install -m 0755 cli/kurrent /usr/local/bin/kurrent   # install the CLI as in 1.1 (or copy the binary from the controller)

2.2 Declare the node config and join the cluster

The node declaration file is storager/kurrent.yaml (the yml is the declaration). No file preparation on the node: the issued command already carries the control-plane URL and the node id — paste and run:

cd /path/to/kurrent
kurrent join https://192.168.1.10 --token a1b2c3.d4e5f6a7b8c9d0e1 --agent-id node-01

kurrent join does three things (idempotent, safe to re-run):

1. Generates/updates storager/kurrent.yaml (when missing, generated from defaults: metadata.name = --agent-id, the command address synced into spec.controlPlane.url; when present, read in and merged — non-forbid, hand edits are kept; a declared identity that differs from --agent-id is rejected. The only hand-written field is metadata.remark — a human label registered into the control-plane ledger on first enroll, shown on the WebUI card and editable in the edit form)
2. Writes the node-bound bootstrap credential storager/bootstrap/agent.token (given via --token; otherwise the file must already be in place. nvmet-host.token is derived automatically by the agent's enroll — nothing to provide by hand)
3. Syncs the .env interpolation keys (component PKI host paths such as KURRENT_AGENT_PKI_HOST) and starts/restarts the agent container (the backend picks the orchestration: nvmet → storager/nvmeof, stgt/lio → storager/iscsi)

Pre-declaring (optional): to customize business keys such as backend/diskDir/advertiseUrl, first generate the annotated template with kurrent config print node-defaults > storager/kurrent.yaml and edit it, then run join (the declaration is read in and merged; hand edits are kept).

The key step for split deployment: the Agent's advertised URL defaults to https://<cp-host>:4840 (co-located form). When the controller and the storage node are on separate machines, the controller must reach the Agent via the storage node's LAN IP — override it in the declaration (the override lives at the declaration layer):

# Edit storager/kurrent.yaml: spec.agent.advertiseUrl → https://192.168.1.20:4840
# Then re-run: kurrent join <cp-url> --token <token> (converges: restarts the agent so the new declaration takes effect)

2.3 Bring up the storage data plane

kurrent join already converged-started the agent (the backend picks the orchestration: nvmet → storager/nvmeof, stgt/lio → storager/iscsi):

docker ps    # expect: storager-agent / kurrent-nvmet-host (or the stgt/lio equivalents)

Custom disk dir: edit spec.agent.diskDir in storager/kurrent.yaml (host path, e.g. /pool1/iscsi_img), then re-run kurrent join --token <token> (idempotently syncs KURRENT_DISK_DIR into .env and restarts the agent to apply it).

Firewall (storage node): sudo ufw allow 4840,4421/tcp && sudo ufw reload

nvmet-host's management plane uses a shared-volume UDS (/run/kurrent/nvmet/nvmet-host.sock, directory 0700): no TCP port, unreachable from the LAN. 4421 is the only data-plane port — restrict its sources as needed.

2.4 Verify

docker logs storager-agent 2>&1 | grep pki     # pki: client cert ok (cn=agent-0d26b6f33a89)
docker logs kurrent-nvmet-host 2>&1 | grep -i "数据面端口就绪"   # 4421 tsas=tls1.3
docker exec kurrent-nvmet-host python /app/healthcheck.py; echo $?   # 0 (UDS + mTLS + configfs)
ss -H -lnt | grep ':4841 ' || echo "no TCP listener on 4841 (expected)"
# back on the controller server:
kurrent agents list
# ID             REMARK       HEALTH  BASE_URL                       ROLE  ENABLED  TAGS
# 0d26b6f33a89   storage-01   ok      https://192.168.1.20:4840      disk  true     auto,nvmet,storage

Step 3: Minimal End-to-End Check

Plug a bare-metal client into the PXE network with network boot enabled:

# on the controller server: watch the DHCP/boot flow
docker logs -f kurrent-dnsmasq 2>&1 | grep -E "DHCPACK|pxe|ipxe"

Expected chain: the client gets an address via DHCP → pulls iPXE and the boot.ipxe pointer over TFTP → iPXE fetches boot.ipxe.cfg and the remaining scripts over HTTPS → device identity report and challenge-response verification → the device auto-registers into the pool (visible under WebUI → Devices). Binding devices, creating system disks and delivering Workers are covered by the Usage Guide.

The TFTP root mounts only snponly.efi and boot.ipxe; boot.ipxe.cfg / menu.ipxe are served over HTTPS only. Recreate or restart the kurrent-dnsmasq container after replacing snponly.efi (file-level mounts pin the inode).

Troubleshooting

Symptom	Check
agents list HEALTH not ok	Can the controller reach spec.agent.advertiseUrl (storage 4840 / firewall)? Storage docker logs storager-agent (cert-renewal failures usually relate to PKI dir permissions)
join: token invalid / expired	Token expired (60 minutes by default), already used (single-use by default), never issued, or the node id does not match the token binding — re-run kurrent token create --agent-id <node id>, then kurrent join --token <new-token> --agent-id <node id>
No DHCP answers	Does dnsmasq.conf interface= match the real NIC (ip a)? Firewall 67/69 UDP
Agent container restarting	docker logs storager-agent; usually the controller is unreachable (controlPlane.url 443 blocked) or the bootstrap credential file is missing (kurrent join --token <token> rewrites agent.token; delete nvmet-host.token and restart the agent to trigger re-derivation)
Controller 503 / enroll failures	Is nginx 443 with the /api/cp reverse proxy healthy? curl -k https://<cp-ip>/api/cp/healthz
Data-plane port not ready / boot refused	Does the storage host carry the NVMe/TCP TLS kernel options, and is tlshd running? Check the 4421 port-creation log in docker logs kurrent-nvmet-host; /capabilities should report port.ready=true

Appendix: Declaration Files at a Glance

control_plane/kurrent.yaml (template generated by kurrent config print init-defaults, edited, then validated/started by kurrent init): spec.networking five keys (interface/subnet/dhcpRange/gateway/dns) plus default blocks pki/serverCert/boot/agentTimeoutSec — the authoritative controller file; dnsmasq.conf is derived from it. The controller container does not mount docker.sock: dnsmasq watches the DHCP binding file itself, and nginx reloads its own CRL via an in-container watcher (confirmed by the controller through a shared-volume sentinel).

storager/kurrent.yaml (template generated by kurrent config print node-defaults, edited, then validated/converged by kurrent join): spec.agent (backend/advertiseUrl/diskDir/nqnBase) + spec.controlPlane.url — the authoritative node file. The NQN namespace is owned solely by its nqnBase (the controller discovers it via the capabilities report, so Host NQNs and disk NQNs stay in one domain automatically).

Both are runtime files (not committed); templates come from the matching kurrent.yaml.example or kurrent config print.