Open-source distributed block storage · MIT
Spindle turns the hard drives already in your servers into replicated block volumes. A client knows where every chunk lives and talks straight to the drive that holds it, so throughput adds up across every disk and node. It is one small Rust binary, built to share its hosts with the workloads it serves.
# on any client host
$ sudo spindle attach pgdata
/dev/spindle/pgdata
$ sudo mkfs.xfs /dev/spindle/pgdataMost open-source block storage is designed around flash, or around being everything at once. Spindle is designed around one job: make a rack of hard drives fast, safe block storage without taking the rack's CPU and memory away from the workloads on it.
On HDD clusters the limits are seeks and the network, not CPU. Spindle spends its design budget there: reads go to either copy to double random-read IOPS, queues stay deep so drives can reorder, a data I/O never adds a metadata I/O, and unwritten space reads back as a flag instead of zeros on the wire.
The client computes where every chunk lives from a table it already holds, the model ScaleIO proved on spinning disks. No lookup, no gateway, no extra hop. Each request carries one 32-40 byte header; payloads land straight in their final buffer.
Storage that runs beside your VMs and pods has to stay out of their way. Spindle is Rust with no async runtime: its event loop is written directly on io_uring. A placement row costs 16 bytes of memory. Raft runs on the control plane only, so the consensus log never sees data I/O.
Losing data is never an acceptable trade for speed. Writes acknowledge only after every copy succeeds, checksums are on by default, a write-intent log keeps copies identical across a whole-rack power cut, and a failed drive's missed chunks are recorded before the error is returned. The details.
The same binary runs as systemd units on bare metal (tarball, install.sh or .deb) or from a Helm chart with a CSI driver, managers as a StatefulSet and servers and clients as DaemonSets. Mix them: Kubernetes nodes and plain Linux hosts can be clients of one cluster.
You do not need five nodes to start. Two storage nodes plus a storageless witness, a cloud VM or a small box running only the manager, keep quorum without split brain. Each node picks which network carries client, replication and control traffic.
ublk, the fastest path from user space to a block device. The client holds the table and sends each request to the drive that owns the chunk.The full design explains each choice, the performance rule behind it, and the measurement that decided it.
The project's first rule is that data loss is unacceptable; its second is that data stays available. Every mechanism below is built and covered by tests or a recorded lab run.
A write is acknowledged only when every copy succeeded. When a drive fails mid-write, the chunks it may have missed are fsynced to a record before the error reply is sent, and reads of those chunks go to the good copy.
Designed for a rack that loses power with no UPS. Flushed data survives, and a write-intent log makes every copy of an in-flight chunk converge on the same contents, so a filesystem replaying its journal never sees reads flip between copies.
Client devices use ublk user recovery: the kernel keeps the device and holds its I/O while the daemon restarts. In a lab run, clientd was SIGKILLed during fio write+verify and restarted 5 s later; both runs finished with 0 errors.
Per-request deadlines catch a stalled server whose kernel still ACKs TCP. Reads move to the other copy; writes to a down disk's rows fail fast before any copy changes. A 30 s network partition recovers within 1 s of healing.
Drives carry a superblock tied to disk, node and cluster. A drive reformatted in place is treated as empty, so reads never return zeros where data should be. Stable /dev/disk/by-id paths and a global LVM filter keep the host from grabbing spindle drives.
Leases and fencing come from the Raft managers. A storageless witness supplies the third vote for two-node sites. Manager state has scheduled backups for recovering from a lost majority.
There is excellent open-source storage already. Here is where Spindle fits, and where you should pick something else.
| Spindle | Ceph RBD (and Rook) | Longhorn | LINSTOR / DRBD (Piraeus) | OpenEBS Mayastor | |
|---|---|---|---|---|---|
| Interfaces | Block | Block, file, S3 object | Block | Block | Block |
| Where it runs | systemd hosts, Kubernetes, or both | Bare metal (cephadm) or Kubernetes (Rook) | Kubernetes only | Bare metal or Kubernetes | Kubernetes only |
| Data path | Client computes placement, talks to drives directly (ublk) | Client computes placement with CRUSH (librbd / krbd) | Per-volume engine fronting replicas | In-kernel replication of a whole device to N nodes | SPDK engine, NVMe-oF to the app node |
| One volume spreads over | Every drive in the pool | Every OSD in the pool | Its replica nodes' disks | Its replica nodes' disks | Its replica nodes' disks |
| Designed around | HDDs, with an optional SSD read cache | Any media, any scale | Simplicity on Kubernetes | Mature, local-read replication | NVMe at low latency |
| Host footprint | Small: no async runtime, 16 B per placement row | Larger: about 4 GiB memory target per OSD by default, plus MONs and MGRs | Moderate: engine and replica processes per volume | Small: kernel module plus controller | Dedicated polling cores and hugepages per engine |
| Smallest HA cluster | 2 storage nodes + storageless witness | 3 MONs recommended | 3 nodes recommended | 2 nodes + diskless tiebreaker | 3 nodes recommended |
| Maturity | Young: running in one lab, no outside production users yet | Very mature, huge production base | Mature; CNCF incubating project | Very mature; DRBD has run in production for two decades | Production releases; part of OpenEBS (CNCF Sandbox) |
Peer figures are from each project's public documentation and defaults, which are tunable. Spindle's own numbers come from its repository's lab runs. We have not published a head-to-head benchmark yet; when we do, the method and raw results will be in the repo.
Spindle is young and moving fast. This is what exists today, what is landing, and what is only designed. When in doubt, the code and the design documents in the repo are the source of truth.
/dev/spindle/NAME namesspindle install; Helm chart and CSI driver (csi-sanity: 68 passed, 0 failed)Pick the shape that matches your hosts. Every path uses the same spindle binary.
Install the binary from a release, the .deb or install.sh, set up three managers (or two and a witness), format drives and start servers and clients.
Install on systemd →Deploy the Helm chart: managers as a StatefulSet, servers on labelled storage nodes, clients and the CSI driver everywhere volumes mount.
Install on Kubernetes →Run managers and storage on plain hosts and attach volumes from Kubernetes, or the other way round. One cluster, both worlds.
Mixed installs →# every host: binary, units, udev rules, LVM filter, kernel modules
$ sudo spindle install
# storage hosts, once: format the data drives
$ sudo spindle format -c /etc/spindle/cluster.conf -n 1 --dry-run
# client hosts: a volume is a block device
$ sudo spindle attach VOLUMEAlready have a cluster? Quickstart: add a client or storage host · Manager API · Alert runbook · Releases