ellie@princess ~ $ bat ~/blog/why-i-built-a-bare-metal-cluster.md
cd ../blogAt some point, every application became a landlord
At some point, every application became a landlord.
Your photos live in one company’s building. Your email lives in another. Your documents, passwords, conversations, calendar, finances, recipes, artwork and half-finished thoughts each occupy a furnished little room in somebody else’s data centre. The rent looks harmless because it arrives in twelve separate subscriptions, each priced somewhere between “a coffee” and “surely I cancelled this.”
Then the landlord changes the terms.
The free tier shrinks. The export button develops strong religious objections to portability. A feature you relied on moves into the Business Pro Max Enterprise Friendship Edition. The monthly price rises because the product now contains an AI assistant you did not request and cannot fully disable. A new paragraph appears in the terms of service granting the company broader rights to process what you upload. You receive an email titled We’re updating our terms, written in the warm, bloodless dialect corporations use when the update is mandatory and the explanation would frighten the horses.
Your options are to accept or leave.
Leaving means discovering that five years of your life have been stored in a format best described as “technically JSON.”
This is why I built a bare-metal cluster.
I built it because convenience had slowly turned into dependency, dependency had turned into exposure, and exposure had started to feel reckless.
I wanted a boundary I controlled.
The decision was less philosophical than the rest of this essay may make it sound. I looked at where the authoritative copies of my life actually lived and realised that almost none of them were under my control. Photographs existed primarily in somebody else’s library. Documents depended on accounts that could be repriced or closed. Conversations and creative work were scattered across companies whose future owners I could not predict and whose terms I could not negotiate. There was no single cinematic betrayal; just more irreplaceable data and fewer credible exit routes.
So I rented two dedicated machines. Each has a six-core Ryzen 5 3600 and 64 GiB of memory. One contributes two 2 TB spinning disks; the other has two 512 GB NVMe drives. Talos runs the nodes, Kubernetes schedules the applications, Cilium moves the packets, Rook Ceph turns four disks into storage, and OpenTofu records enough of the arrangement that future me has at least a fighting chance.
This is not the architecture I would sell to a bank. It is the architecture I chose for one person who wants custody, useful capacity and the ability to replace parts without rebuilding an entire digital life from memory.
We asked for the cloud, and for good reasons
SaaS did not conquer the world solely through deceit and decorative gradients. It solved real problems.
Software used to arrive in a box. You bought a version, installed it, discovered that the installer needed three floppy disks in an order known only to the Oracle at Delphi, and eventually owned a working copy. Businesses ran servers in cupboards where the air conditioning was theoretical and the backup tape had last been tested during the Blair administration. If the disk died, your software did not float gracefully into another availability zone. It became a learning opportunity.
Salesforce made the subscription model culturally legible in 1999: business software delivered through a browser, without every customer maintaining the whole stack. Then Amazon launched S3 and EC2 in 2006, turning storage and compute into APIs instead of a procurement process involving six signatures and a Dell sales representative. The cloud made infrastructure previously reserved for large organisations available to anyone with a payment card and insufficient supervision.
That change was genuinely liberating. Small teams could deploy globally. Managed databases removed a category of 03:17 phone calls. PaaS products went further: push code, let somebody else care about the operating system, routing, certificates, autoscaling and the endless parade of CVEs produced by software continuing to exist.
Each layer removed work. Each layer also removed control.
At first that exchange was obviously worthwhile. Usually it still is. I do not want every hospital writing its own video-conferencing stack or every bakery operating an email server because somebody on the internet shouted “digital sovereignty” through a mouthful of ethernet cable.
The problem was not renting software.
The problem was allowing rented software to become the only place our lives existed.
Cloud software is different. What you buy is not an object. It is an ongoing relationship with an organisation whose incentives can change.
The service can be excellent and the people building it can care deeply. None of that suspends economics. The company may be acquired; investors may demand growth; the cheap product that attracted millions may now be expected to extract more money from them. Previously included features migrate upward through pricing tiers like frightened woodland animals escaping a forest fire.
And the company has three assets it can monetise:
- The product.
- Your attention.
- Your data.
The first is respectable but apparently insufficient for the infinite-growth machine. The second gave us the advertising internet. The third has become extremely interesting now that every executive presentation contains the phrase AI strategy and at least one arrow pointing toward a cylinder labelled DATA.
I do not need to believe everyone involved is malicious. Incentives are more scalable than malice.
AI made the ownership problem impossible to ignore
Generative AI did not invent extractive data practices. Advertising platforms, data brokers and recommendation systems had already industrialised those. AI changed the perceived value of the material.
Photos became potential training data. Posts became a language corpus. Voice recordings became speech data. Documents, illustrations and source code became examples from which models might learn.
The response across the industry has not been uniform, and accuracy matters here. The policies below were checked in July 2026; terms mutate like bacteria under a heat lamp. “Every company trains on all your private data” is a satisfying sentence and a bad argument.
Adobe prompted users to re-accept terms in 2024 and caused an entirely predictable panic among creators. Adobe subsequently clarified that it had not trained generative AI on customer content and promised to put that commitment into its legally binding terms. Its own explanation acknowledged that companies hosting creative work must be precise about what rights they claim and why.
That was a better outcome than people initially feared. It also demonstrated the power imbalance: each creator had to determine whether a broad licence was operational boilerplate or a combine harvester approaching their life’s work.
Other companies are explicit about using content. Meta announced plans to train its models on public posts, comments, photos and captions shared by adults in Europe, with a process through which users could object. Meta’s own description says exactly that: it wants to train on public content from European users.
Public posts are not private photo libraries, product improvement is not always generative-model training, and an opt-out is not an opt-in. We should criticise what companies actually do, not flatten everything into a single sinister blob wearing a conference lanyard.
But the pattern still bothers me. The default relationship is increasingly that a service receives broad access, the policy evolves, and the user is responsible for noticing, understanding and finding the correct toggle. Consent becomes a scavenger hunt conducted inside an interface the company can redesign tomorrow.
I am not against AI. I patched Mealie to use a language model for recipe imports, an act that combined artificial intelligence, Kubernetes and dinner with a level of restraint rarely seen outside defence procurement.
Machine learning is useful. Generative systems can translate, summarise, classify, help people write code and make interfaces more accessible.
The unresolved question is what we are entitled to feed them.
Training useful models requires a great deal of data. That does not make every accessible work ethically ownerless. “It was on the internet” is a description of network reachability, not a theory of consent. A forum post written to help one stranger debug a kernel panic was not necessarily offered as free industrial feedstock forever.
My position is not “stop developing models.” Tell people what is collected. Distinguish operating a service from training a new product. Ask before using private material, make refusal as easy as acceptance, respect creators’ reservations, and document sources well enough that accountability is possible. Do not call extraction innovation merely because it ends in matrix multiplication.
I will send some data to an AI service when the purpose is clear and the material appropriate. That is different from storing the canonical copy of my private life there and hoping its next business model remains charming. Agency is the point.
Privacy is not a confession
Whenever privacy comes up, somebody announces that they have nothing to hide.
This sounds brave until you ask them to publish their passwords, medical history, private messages, bank statements, precise location, family photographs, search history and every unfinished thought they typed at 01:40 and wisely deleted.
Nobody actually has nothing to hide. What they mean is that they do not currently expect the information collected about them to be used by somebody hostile.
That is a prediction about power, not a fact about data.
Privacy scholar Daniel Solove has spent years dismantling the “nothing to hide” argument. Its central failure is treating privacy only as secrecy about wrongdoing. Privacy is also protection against aggregation, exclusion, distortion, secondary use, loss of context and systems making decisions about us that we cannot inspect or challenge. His essay, “I’ve Got Nothing to Hide” and Other Misunderstandings of Privacy, remains annoyingly relevant.
One isolated fact may be harmless. A collection becomes a map.
Your photographs reveal relationships, locations, homes, routines and health. Email reveals who you know and which institutions you depend on. Calendar entries reveal where you will be. Search history reveals fears before you have words for them. Metadata can reveal that two people communicate regularly even when the messages themselves are encrypted.
The danger also changes over time. Data collected under one government, one management team or one social norm remains available when those conditions change.
This matters to everyone. It matters especially to people whose safety has historically depended on controlling context.
I am queer. With the brown wave rising again across Europe and elsewhere, I do not consider it paranoid to ask how identity, relationships, location and private communication could be used under a more hostile political order. Rights that look settled can become campaign material. Healthcare can become evidence. A social graph can become an investigative lead. A list created for advertising can become useful to an authority with a different mandate.
The Electronic Frontier Foundation and Access Now have warned that data concerning sexual orientation and gender identity can enable surveillance and violence against LGBTQ+ people, and have argued for stronger protection of that data in a submission to the UN Independent Expert on sexual orientation and gender identity.
You do not build privacy only for the government you trust.
You build it for the next government, the next acquisition, the next database breach, the next policy update and the next executive who looks at a table of user content and sees an unmonetised asset.
Encryption changes who is capable of betraying you
A privacy policy is a promise. End-to-end encryption is an architectural constraint.
Those are not equivalent.
If a service can read your data, it may do so for legitimate operational reasons. It can also be breached, compelled, acquired, misconfigured or persuaded by a future policy. Access may be carefully audited and restricted, but the capability exists.
With correctly implemented end-to-end encryption, the service does not possess the keys required to read the content. It can still expose metadata, clients can still be compromised, participants can still take screenshots, and backups can quietly ruin the whole arrangement if designed by a committee. E2EE does not solve every problem. It does remove the service operator from one extremely important trust boundary.
Self-hosting and end-to-end encryption solve different problems. Self-hosting changes who operates the service. E2EE limits what even the operator can read. Where possible, I want both.
Signal describes the principle plainly: its service is designed so it does not have access to message contents, and it also works to minimise retained metadata. The expensive part of Signal is not drawing a blue send button; it is maintaining a system deliberately built to know less about its users. Its explanation of the costs and architecture is worth reading: privacy is priceless, but Signal is expensive.
European data-protection authorities likewise describe encryption as necessary for privacy and free expression. The European Data Protection Supervisor warns that weakening or circumventing E2EE through backdoors or key escrow destroys its effective protection. Its encryption overview is much less ambiguous than most political proposals involving the phrase “lawful access.”
The recurring demand for systems that are private except when the correct authority asks is cryptographic astrology. A backdoor available only to good people is not a security property. It is a staffing assumption.
For queer people, journalists, activists, abuse survivors, migrants, healthcare workers and anyone living under a government with authoritarian ambitions, secure communication is not a decorative civil liberty. It is infrastructure.
Why bare metal, and why a cluster?
The argument so far would justify a NAS, a VPS running Docker Compose, a managed Nextcloud provider, or three Raspberry Pis committing distributed-computing offences behind the television.
All of those can be good answers. Mine was dedicated servers because the workload stopped being one application and started becoming a small platform.
Photos are the largest part. Immich wants substantial storage, background processing, PostgreSQL with vector search, a machine-learning worker and room for the library to grow. Matrix, Vaultwarden, Mealie, Actual Budget, Outline, Grist, SearXNG, monitoring, CI and the rest add databases, caches, object storage and persistent volumes. Individually they are modest. Together they want predictable memory, direct disks and enough spare capacity that an update does not turn the scheduler into a Victorian workhouse.
Bare metal gives me dedicated RAM and CPU, direct access to the storage devices, predictable monthly capacity and no metered egress between my own workloads. Ceph can use the real disks rather than virtual volumes rented from the same abstraction it is trying to replace. Large photo and backup transfers do not produce a bill that reads like a ransom note.
Why two machines? One server would be simpler, but compute, storage and the control plane would all share one machine and one failure domain. Because both nodes are schedulable, Kubernetes can move stateless workloads and give me somewhere to move work while maintaining a node. Some databases can also place replicas apart. The second machine added fast NVMe storage alongside the larger spinning disks.
Why Kubernetes? Because it is the operational model I know and enjoy. It gives me declarative deployments, health checks, scheduling, identity integration, storage primitives and enough YAML to prevent dangerous levels of free time. Most of these applications could run under Docker Compose. I am not claiming a Deployment object makes the grocery list taste better.
Why Ceph? Applications need several kinds of persistence. RBD provides block volumes, CephFS provides shared filesystems, and RGW provides S3-compatible object storage. One storage system can serve PostgreSQL volumes, application PVCs, media and backup objects while allowing a disk to be replaced without teaching every application a new ritual.
Now for the sentence storage people have been holding their breath for: two nodes do not make a production-grade Ceph cluster.
There are four OSDs across the two machines and pools use two replicas with the CRUSH failure domain set to osd, not host. Ceph guarantees different OSDs, but both copies may still land on the same machine. The layout protects against an individual OSD failure; losing a host can mean restoration, not merely an outage. The cluster also has one Kubernetes control-plane node and one Ceph monitor.
This design is distributed. It is not highly available.
That is a deliberate cost and complexity boundary, not a hidden triumph. A third node would allow proper etcd and Ceph monitor quorum and stronger host-level replica placement. Until then, the second node improves maintenance options and some failure handling; it does not transform two computers into three.
The answer to “what failure can this survive?” is therefore specific: individual pods, processes and OSDs can fail without losing the whole service. A node or control-plane failure becomes a recovery event rather than a transparent shrug. Loss of the entire cluster requires restoration from somewhere that is not the cluster.
Distributed systems remain stubbornly opposed to inspirational arithmetic.
The cheapest invoice is not always the lowest cost
This is where self-hosting evangelists traditionally produce a spreadsheet proving they save €14 per month, provided their labour has no value and hard drives emerge spontaneously from woodland soil.
Together, including their IP addresses, the two servers cost €78.80 per month before VAT, or €98.50 with Swedish VAT. They replace several subscriptions and provide terabytes of raw storage, 128 GiB of memory and twelve physical CPU cores across the cluster. Buying that capacity as separate managed databases, object storage, photo hosting, CI, monitoring and application plans would be expensive. Buying only the few consumer subscriptions I strictly need might be cheaper.
Then there is labour:
1Systems administration: financially catastrophic if invoiced honestly
I do enjoy this work. Debugging Cilium at two in the morning is not cheaper if accounted for honestly, but it is at least my preferred genre of regrettable decision.
The financial case works for my collection of storage-heavy services. The larger case includes control, portability and privacy. Sometimes the cheapest service has a low invoice because you are paying with lock-in, attention, exposure or the future difficulty of leaving.
A boundary, not a fortress
Moving data onto my own infrastructure does not automatically make it safe.
Google has security teams, redundant data centres, hardware supply-chain programmes, abuse detection and people who understand email delivery at a depth normally associated with ocean trenches. I have strong opinions, two servers and a TODO file containing the phrase do these first.
The present architecture makes those limits concrete. Database backups written into the same Ceph cluster are useful for logical recovery, but they are not disaster recovery from loss of that cluster. Until encrypted off-site copies, external etcd snapshots and a tested restore path exist, this remains a migration in progress rather than a finished risk-management system. A green dashboard does not negotiate with a failed disk.
Backups are a process. Merely possessing backup-shaped objects is storage-themed optimism.
There is also concentration risk. Earlier I said that a collection becomes a map; this cluster deliberately assembles much of that map inside one administrative domain. A compromised administrator, identity provider or recovery account could reach several services at once.
Some mitigations already exist: applications have separate credentials and namespaces, while sensitive interfaces use restricted network paths. Independent recovery credentials, off-site copies and regular restore drills are still work to complete. One login button across everything is convenient. It is also a blast radius wearing a friendly logo.
Self-hosting changes the threat model. It reduces exposure to platform policy changes and mass commercial data use while increasing exposure to my mistakes, targeted attacks and hardware failures. A personal server is a small target with an administrator who occasionally deploys after midnight.
The correct questions are not “cloud or self-hosted: which is secure?” They are: which failures am I preparing for, who can access the data, who can change the rules, how do I leave, where are the backups, and what happens when I am wrong?
If those questions have no answers, the deployment model is branding.
The cloud gave us extraordinary capabilities. I do not want to reverse that history; I want its next part to be less extractive. Convenience should not require surrendering control over our memories and creative work. AI development should not pretend consent is implicit wherever a crawler can reach. Privacy should not be a luxury feature, and encryption should not be weakened because inaccessible data is administratively inconvenient.
Digital sovereignty is not a binary state. It is the practice of reducing unnecessary dependence and making the remaining dependencies explicit.
You do not need a bare-metal Kubernetes cluster to move in that direction. Export your photos. Keep local copies of documents. Use a password manager and E2EE. Choose services with credible business models and usable exports. Use a NAS if that solves the problem. Learn where your data goes.
Sovereignty is a direction, not a rack size.
My cluster is not a fortress. It is a boundary. I have not eliminated landlords; I have made them replaceable. A provider may still evict the machines, but it does not get to evict me from my own data. The SaaS vendors I chose not to use never receive the authoritative copy in the first place.
That matters more to me now than it did ten years ago. The industry is consolidating, generative AI has made data newly valuable, and the political weather is getting worse. Under those conditions, keeping intimate information inside systems I can inspect, move and shut down feels less like a hobby and more like basic risk management.
I still love technology. I love it enough to want better from it.
So I built the cluster: two servers, too many databases, several thousand lines of OpenTofu, and one very expensive way to say that my life is not an untapped dataset.
back to all field notes