Fieldnotes by Ellie Augustini

Fieldnotes by Ellie Augustinihttps://augustini.wtf/fieldnotes/Technical fieldnotes about infrastructure, self-hosting, firmware, terminal interfaces, and engineering incidents that escalated into essays.Hugoen-USTue, 14 Jul 2026 14:00:00 +0200At some point, every application became a landlordhttps://augustini.wtf/fieldnotes/why-i-built-a-bare-metal-cluster/Tue, 14 Jul 2026 14:00:00 +0200https://augustini.wtf/fieldnotes/why-i-built-a-bare-metal-cluster/Ellie Augustinibare-metalself-hostingprivacysaasaiencryptiondigital-sovereigntykubernetesAt some point, every application became a landlord.
Your photos live in one company’s building. Your email lives in another. Your documents, passwords, conversations, calendar, finances, recipes, artwork and half-finished thoughts each occupy a furnished little room in somebody else’s data centre. The rent looks harmless because it arrives in twelve separate subscriptions, each priced somewhere between “a coffee” and “surely I cancelled this.”At some point, every application became a landlord.
Your photos live in one company’s building. Your email lives in another. Your documents, passwords, conversations, calendar, finances, recipes, artwork and half-finished thoughts each occupy a furnished little room in somebody else’s data centre. The rent looks harmless because it arrives in twelve separate subscriptions, each priced somewhere between “a coffee” and “surely I cancelled this.”
Then the landlord changes the terms.
The free tier shrinks. The export button develops strong religious objections to portability. A feature you relied on moves into the Business Pro Max Enterprise Friendship Edition. The monthly price rises because the product now contains an AI assistant you did not request and cannot fully disable. A new paragraph appears in the terms of service granting the company broader rights to process what you upload. You receive an email titled We’re updating our terms, written in the warm, bloodless dialect corporations use when the update is mandatory and the explanation would frighten the horses.
Your options are to accept or leave.
Leaving means discovering that five years of your life have been stored in a format best described as “technically JSON.”
This is why I built a bare-metal cluster.
I built it because convenience had slowly turned into dependency, dependency had turned into exposure, and exposure had started to feel reckless.
I wanted a boundary I controlled.
The decision was less philosophical than the rest of this essay may make it sound. I looked at where the authoritative copies of my life actually lived and realised that almost none of them were under my control. Photographs existed primarily in somebody else’s library. Documents depended on accounts that could be repriced or closed. Conversations and creative work were scattered across companies whose future owners I could not predict and whose terms I could not negotiate. There was no single cinematic betrayal; just more irreplaceable data and fewer credible exit routes.
So I rented two dedicated machines. Each has a six-core Ryzen 5 3600 and 64 GiB of memory. One contributes two 2 TB spinning disks; the other has two 512 GB NVMe drives. Talos runs the nodes, Kubernetes schedules the applications, Cilium moves the packets, Rook Ceph turns four disks into storage, and OpenTofu records enough of the arrangement that future me has at least a fighting chance.
This is not the architecture I would sell to a bank. It is the architecture I chose for one person who wants custody, useful capacity and the ability to replace parts without rebuilding an entire digital life from memory.
<h2 id="we-asked-for-the-cloud-and-for-good-reasons" class="heading-with-permalink">We asked for the cloud, and for good reasons<a
 class="heading-permalink"
 href="#we-asked-for-the-cloud-and-for-good-reasons"
 aria-label="Copy link to section: We asked for the cloud, and for good reasons"
 data-heading-permalink
 >#</a>
</h2>
SaaS did not conquer the world solely through deceit and decorative gradients. It solved real problems.
Software used to arrive in a box. You bought a version, installed it, discovered that the installer needed three floppy disks in an order known only to the Oracle at Delphi, and eventually owned a working copy. Businesses ran servers in cupboards where the air conditioning was theoretical and the backup tape had last been tested during the Blair administration. If the disk died, your software did not float gracefully into another availability zone. It became a learning opportunity.
Salesforce made the subscription model culturally legible in 1999: business software delivered through a browser, without every customer maintaining the whole stack. Then Amazon launched S3 and EC2 in 2006, turning storage and compute into APIs instead of a procurement process involving six signatures and a Dell sales representative. The cloud made infrastructure previously reserved for large organisations available to anyone with a payment card and insufficient supervision.
That change was genuinely liberating. Small teams could deploy globally. Managed databases removed a category of 03:17 phone calls. PaaS products went further: push code, let somebody else care about the operating system, routing, certificates, autoscaling and the endless parade of CVEs produced by software continuing to exist.
Each layer removed work. Each layer also removed control.
At first that exchange was obviously worthwhile. Usually it still is. I do not want every hospital writing its own video-conferencing stack or every bakery operating an email server because somebody on the internet shouted “digital sovereignty” through a mouthful of ethernet cable.
The problem was not renting software.
The problem was allowing rented software to become the only place our lives existed.
Cloud software is different. What you buy is not an object. It is an ongoing relationship with an organisation whose incentives can change.
The service can be excellent and the people building it can care deeply. None of that suspends economics. The company may be acquired; investors may demand growth; the cheap product that attracted millions may now be expected to extract more money from them. Previously included features migrate upward through pricing tiers like frightened woodland animals escaping a forest fire.
And the company has three assets it can monetise:
<ol>
<li>The product.</li>
<li>Your attention.</li>
<li>Your data.</li>
</ol>
The first is respectable but apparently insufficient for the infinite-growth machine. The second gave us the advertising internet. The third has become extremely interesting now that every executive presentation contains the phrase AI strategy and at least one arrow pointing toward a cylinder labelled <code>DATA</code>.
I do not need to believe everyone involved is malicious. Incentives are more scalable than malice.
<h2 id="ai-made-the-ownership-problem-impossible-to-ignore" class="heading-with-permalink">AI made the ownership problem impossible to ignore<a
 class="heading-permalink"
 href="#ai-made-the-ownership-problem-impossible-to-ignore"
 aria-label="Copy link to section: AI made the ownership problem impossible to ignore"
 data-heading-permalink
 >#</a>
</h2>
Generative AI did not invent extractive data practices. Advertising platforms, data brokers and recommendation systems had already industrialised those. AI changed the perceived value of the material.
Photos became potential training data. Posts became a language corpus. Voice recordings became speech data. Documents, illustrations and source code became examples from which models might learn.
The response across the industry has not been uniform, and accuracy matters here. The policies below were checked in July 2026; terms mutate like bacteria under a heat lamp. “Every company trains on all your private data” is a satisfying sentence and a bad argument.
Adobe prompted users to re-accept terms in 2024 and caused an entirely predictable panic among creators. Adobe subsequently clarified that it had not trained generative AI on customer content and promised to put that commitment into its legally binding terms. Its own explanation acknowledged that companies hosting creative work must be precise about <a href="https://blog.adobe.com/en/publish/2024/06/10/updating-adobes-terms-of-use">what rights they claim and why</a>.
That was a better outcome than people initially feared. It also demonstrated the power imbalance: each creator had to determine whether a broad licence was operational boilerplate or a combine harvester approaching their life’s work.
Other companies are explicit about using content. Meta announced plans to train its models on public posts, comments, photos and captions shared by adults in Europe, with a process through which users could object. Meta’s own description says exactly that: it wants to train on <a href="https://about.fb.com/news/2025/04/making-ai-work-harder-for-europeans/">public content from European users</a>.
Public posts are not private photo libraries, product improvement is not always generative-model training, and an opt-out is not an opt-in. We should criticise what companies actually do, not flatten everything into a single sinister blob wearing a conference lanyard.
But the pattern still bothers me. The default relationship is increasingly that a service receives broad access, the policy evolves, and the user is responsible for noticing, understanding and finding the correct toggle. Consent becomes a scavenger hunt conducted inside an interface the company can redesign tomorrow.
I am not against AI. I <a href="/fieldnotes/hacking-mealie-for-fun-and-dinner/">patched Mealie to use a language model for recipe imports</a>, an act that combined artificial intelligence, Kubernetes and dinner with a level of restraint rarely seen outside defence procurement.
Machine learning is useful. Generative systems can translate, summarise, classify, help people write code and make interfaces more accessible.
The unresolved question is what we are entitled to feed them.
Training useful models requires a great deal of data. That does not make every accessible work ethically ownerless. “It was on the internet” is a description of network reachability, not a theory of consent. A forum post written to help one stranger debug a kernel panic was not necessarily offered as free industrial feedstock forever.
My position is not “stop developing models.” Tell people what is collected. Distinguish operating a service from training a new product. Ask before using private material, make refusal as easy as acceptance, respect creators’ reservations, and document sources well enough that accountability is possible. Do not call extraction innovation merely because it ends in matrix multiplication.
I will send some data to an AI service when the purpose is clear and the material appropriate. That is different from storing the canonical copy of my private life there and hoping its next business model remains charming. Agency is the point.
<h2 id="privacy-is-not-a-confession" class="heading-with-permalink">Privacy is not a confession<a
 class="heading-permalink"
 href="#privacy-is-not-a-confession"
 aria-label="Copy link to section: Privacy is not a confession"
 data-heading-permalink
 >#</a>
</h2>
Whenever privacy comes up, somebody announces that they have nothing to hide.
This sounds brave until you ask them to publish their passwords, medical history, private messages, bank statements, precise location, family photographs, search history and every unfinished thought they typed at 01:40 and wisely deleted.
Nobody actually has nothing to hide. What they mean is that they do not currently expect the information collected about them to be used by somebody hostile.
That is a prediction about power, not a fact about data.
Privacy scholar Daniel Solove has spent years dismantling the “nothing to hide” argument. Its central failure is treating privacy only as secrecy about wrongdoing. Privacy is also protection against aggregation, exclusion, distortion, secondary use, loss of context and systems making decisions about us that we cannot inspect or challenge. His essay, <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=998565">“I’ve Got Nothing to Hide” and Other Misunderstandings of Privacy</a>, remains annoyingly relevant.
One isolated fact may be harmless. A collection becomes a map.
Your photographs reveal relationships, locations, homes, routines and health. Email reveals who you know and which institutions you depend on. Calendar entries reveal where you will be. Search history reveals fears before you have words for them. Metadata can reveal that two people communicate regularly even when the messages themselves are encrypted.
The danger also changes over time. Data collected under one government, one management team or one social norm remains available when those conditions change.
This matters to everyone. It matters especially to people whose safety has historically depended on controlling context.
I am queer. With the brown wave rising again across Europe and elsewhere, I do not consider it paranoid to ask how identity, relationships, location and private communication could be used under a more hostile political order. Rights that look settled can become campaign material. Healthcare can become evidence. A social graph can become an investigative lead. A list created for advertising can become useful to an authority with a different mandate.
The Electronic Frontier Foundation and Access Now have warned that data concerning sexual orientation and gender identity can enable surveillance and violence against LGBTQ+ people, and have argued for stronger protection of that data in a <a href="https://www.eff.org/files/2024/01/31/access_now_eff_written_submission_un_ie_on_sogie_-_jan_2024_0.pdf">submission to the UN Independent Expert on sexual orientation and gender identity</a>.
You do not build privacy only for the government you trust.
You build it for the next government, the next acquisition, the next database breach, the next policy update and the next executive who looks at a table of user content and sees an unmonetised asset.
<h2 id="encryption-changes-who-is-capable-of-betraying-you" class="heading-with-permalink">Encryption changes who is capable of betraying you<a
 class="heading-permalink"
 href="#encryption-changes-who-is-capable-of-betraying-you"
 aria-label="Copy link to section: Encryption changes who is capable of betraying you"
 data-heading-permalink
 >#</a>
</h2>
A privacy policy is a promise. End-to-end encryption is an architectural constraint.
Those are not equivalent.
If a service can read your data, it may do so for legitimate operational reasons. It can also be breached, compelled, acquired, misconfigured or persuaded by a future policy. Access may be carefully audited and restricted, but the capability exists.
With correctly implemented end-to-end encryption, the service does not possess the keys required to read the content. It can still expose metadata, clients can still be compromised, participants can still take screenshots, and backups can quietly ruin the whole arrangement if designed by a committee. E2EE does not solve every problem. It does remove the service operator from one extremely important trust boundary.
Self-hosting and end-to-end encryption solve different problems. Self-hosting changes who operates the service. E2EE limits what even the operator can read. Where possible, I want both.
Signal describes the principle plainly: its service is designed so it does not have access to message contents, and it also works to minimise retained metadata. The expensive part of Signal is not drawing a blue send button; it is maintaining a system deliberately built to know less about its users. Its explanation of the costs and architecture is worth reading: <a href="https://signal.org/blog/signal-is-expensive/">privacy is priceless, but Signal is expensive</a>.
European data-protection authorities likewise describe encryption as necessary for privacy and free expression. The European Data Protection Supervisor warns that weakening or circumventing E2EE through backdoors or key escrow destroys its effective protection. Its <a href="https://www.edps.europa.eu/data-protection/our-work/subjects/encryption_en">encryption overview</a> is much less ambiguous than most political proposals involving the phrase “lawful access.”
The recurring demand for systems that are private except when the correct authority asks is cryptographic astrology. A backdoor available only to good people is not a security property. It is a staffing assumption.
For queer people, journalists, activists, abuse survivors, migrants, healthcare workers and anyone living under a government with authoritarian ambitions, secure communication is not a decorative civil liberty. It is infrastructure.
<h2 id="why-bare-metal-and-why-a-cluster" class="heading-with-permalink">Why bare metal, and why a cluster?<a
 class="heading-permalink"
 href="#why-bare-metal-and-why-a-cluster"
 aria-label="Copy link to section: Why bare metal, and why a cluster?"
 data-heading-permalink
 >#</a>
</h2>
The argument so far would justify a NAS, a VPS running Docker Compose, a managed Nextcloud provider, or three Raspberry Pis committing distributed-computing offences behind the television.
All of those can be good answers. Mine was dedicated servers because the workload stopped being one application and started becoming a small platform.
Photos are the largest part. Immich wants substantial storage, background processing, PostgreSQL with vector search, a machine-learning worker and room for the library to grow. Matrix, Vaultwarden, Mealie, Actual Budget, Outline, Grist, SearXNG, monitoring, CI and the rest add databases, caches, object storage and persistent volumes. Individually they are modest. Together they want predictable memory, direct disks and enough spare capacity that an update does not turn the scheduler into a Victorian workhouse.
Bare metal gives me dedicated RAM and CPU, direct access to the storage devices, predictable monthly capacity and no metered egress between my own workloads. Ceph can use the real disks rather than virtual volumes rented from the same abstraction it is trying to replace. Large photo and backup transfers do not produce a bill that reads like a ransom note.
Why two machines? One server would be simpler, but compute, storage and the control plane would all share one machine and one failure domain. Because both nodes are schedulable, Kubernetes can move stateless workloads and give me somewhere to move work while maintaining a node. Some databases can also place replicas apart. The second machine added fast NVMe storage alongside the larger spinning disks.
Why Kubernetes? Because it is the operational model I know and enjoy. It gives me declarative deployments, health checks, scheduling, identity integration, storage primitives and enough YAML to prevent dangerous levels of free time. Most of these applications could run under Docker Compose. I am not claiming a Deployment object makes the grocery list taste better.
Why Ceph? Applications need several kinds of persistence. RBD provides block volumes, CephFS provides shared filesystems, and RGW provides S3-compatible object storage. One storage system can serve PostgreSQL volumes, application PVCs, media and backup objects while allowing a disk to be replaced without teaching every application a new ritual.
Now for the sentence storage people have been holding their breath for: two nodes do not make a production-grade Ceph cluster.
There are four OSDs across the two machines and pools use two replicas with the CRUSH failure domain set to <code>osd</code>, not <code>host</code>. Ceph guarantees different OSDs, but both copies may still land on the same machine. The layout protects against an individual OSD failure; losing a host can mean restoration, not merely an outage. The cluster also has one Kubernetes control-plane node and one Ceph monitor.
This design is distributed. It is not highly available.
That is a deliberate cost and complexity boundary, not a hidden triumph. A third node would allow proper etcd and Ceph monitor quorum and stronger host-level replica placement. Until then, the second node improves maintenance options and some failure handling; it does not transform two computers into three.
The answer to “what failure can this survive?” is therefore specific: individual pods, processes and OSDs can fail without losing the whole service. A node or control-plane failure becomes a recovery event rather than a transparent shrug. Loss of the entire cluster requires restoration from somewhere that is not the cluster.
Distributed systems remain stubbornly opposed to inspirational arithmetic.
<h2 id="the-cheapest-invoice-is-not-always-the-lowest-cost" class="heading-with-permalink">The cheapest invoice is not always the lowest cost<a
 class="heading-permalink"
 href="#the-cheapest-invoice-is-not-always-the-lowest-cost"
 aria-label="Copy link to section: The cheapest invoice is not always the lowest cost"
 data-heading-permalink
 >#</a>
</h2>
This is where self-hosting evangelists traditionally produce a spreadsheet proving they save €14 per month, provided their labour has no value and hard drives emerge spontaneously from woodland soil.
Together, including their IP addresses, the two servers cost €78.80 per month before VAT, or €98.50 with Swedish VAT. They replace several subscriptions and provide terabytes of raw storage, 128 GiB of memory and twelve physical CPU cores across the cluster. Buying that capacity as separate managed databases, object storage, photo hosting, CI, monitoring and application plans would be expensive. Buying only the few consumer subscriptions I strictly need might be cheaper.
Then there is labour:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text">1Systems administration: financially catastrophic if invoiced honestly
</code></pre></div>I do enjoy this work. Debugging Cilium at two in the morning is not cheaper if accounted for honestly, but it is at least my preferred genre of regrettable decision.
The financial case works for my collection of storage-heavy services. The larger case includes control, portability and privacy. Sometimes the cheapest service has a low invoice because you are paying with lock-in, attention, exposure or the future difficulty of leaving.
<h2 id="a-boundary-not-a-fortress" class="heading-with-permalink">A boundary, not a fortress<a
 class="heading-permalink"
 href="#a-boundary-not-a-fortress"
 aria-label="Copy link to section: A boundary, not a fortress"
 data-heading-permalink
 >#</a>
</h2>
Moving data onto my own infrastructure does not automatically make it safe.
Google has security teams, redundant data centres, hardware supply-chain programmes, abuse detection and people who understand email delivery at a depth normally associated with ocean trenches. I have strong opinions, two servers and a TODO file containing the phrase do these first.
The present architecture makes those limits concrete. Database backups written into the same Ceph cluster are useful for logical recovery, but they are not disaster recovery from loss of that cluster. Until encrypted off-site copies, external etcd snapshots and a tested restore path exist, this remains a migration in progress rather than a finished risk-management system. A green dashboard does not negotiate with a failed disk.
Backups are a process. Merely possessing backup-shaped objects is storage-themed optimism.
There is also concentration risk. Earlier I said that a collection becomes a map; this cluster deliberately assembles much of that map inside one administrative domain. A compromised administrator, identity provider or recovery account could reach several services at once.
Some mitigations already exist: applications have separate credentials and namespaces, while sensitive interfaces use restricted network paths. Independent recovery credentials, off-site copies and regular restore drills are still work to complete. One login button across everything is convenient. It is also a blast radius wearing a friendly logo.
Self-hosting changes the threat model. It reduces exposure to platform policy changes and mass commercial data use while increasing exposure to my mistakes, targeted attacks and hardware failures. A personal server is a small target with an administrator who occasionally deploys after midnight.
The correct questions are not “cloud or self-hosted: which is secure?” They are: which failures am I preparing for, who can access the data, who can change the rules, how do I leave, where are the backups, and what happens when I am wrong?
If those questions have no answers, the deployment model is branding.
The cloud gave us extraordinary capabilities. I do not want to reverse that history; I want its next part to be less extractive. Convenience should not require surrendering control over our memories and creative work. AI development should not pretend consent is implicit wherever a crawler can reach. Privacy should not be a luxury feature, and encryption should not be weakened because inaccessible data is administratively inconvenient.
Digital sovereignty is not a binary state. It is the practice of reducing unnecessary dependence and making the remaining dependencies explicit.
You do not need a bare-metal Kubernetes cluster to move in that direction. Export your photos. Keep local copies of documents. Use a password manager and E2EE. Choose services with credible business models and usable exports. Use a NAS if that solves the problem. Learn where your data goes.
Sovereignty is a direction, not a rack size.
My cluster is not a fortress. It is a boundary. I have not eliminated landlords; I have made them replaceable. A provider may still evict the machines, but it does not get to evict me from my own data. The SaaS vendors I chose not to use never receive the authoritative copy in the first place.
That matters more to me now than it did ten years ago. The industry is consolidating, generative AI has made data newly valuable, and the political weather is getting worse. Under those conditions, keeping intimate information inside systems I can inspect, move and shut down feels less like a hobby and more like basic risk management.
I still love technology. I love it enough to want better from it.
So I built the cluster: two servers, too many databases, several thousand lines of OpenTofu, and one very expensive way to say that my life is not an untapped dataset.
Hacking Mealie for fun and dinnerhttps://augustini.wtf/fieldnotes/hacking-mealie-for-fun-and-dinner/Tue, 14 Jul 2026 10:00:00 +0200https://augustini.wtf/fieldnotes/hacking-mealie-for-fun-and-dinner/Ellie Augustinimealieopenaigpt-5.6pythonkubernetesopentofuself-hosting<img src="https://augustini.wtf/fieldnotes/hacking-mealie-for-fun-and-dinner/mealie-recipes_hu_4fa5d2ce769adad4.webp" alt="My Mealie recipe library, now considerably more international and significantly less allergic to metric units." width="1024" height="566" loading="lazy" decoding="async">
There is a very specific kind of optimism involved in clicking Import recipe and expecting a food blog to become structured data.<img src="https://augustini.wtf/fieldnotes/hacking-mealie-for-fun-and-dinner/mealie-recipes_hu_4fa5d2ce769adad4.webp" alt="My Mealie recipe library, now considerably more international and significantly less allergic to metric units." width="1024" height="566" loading="lazy" decoding="async">
There is a very specific kind of optimism involved in clicking Import recipe and expecting a food blog to become structured data.
The page has a 900-word childhood memory, three newsletter pop-ups, an autoplaying video, twelve affiliate links, two incompatible versions of the recipe, and—somewhere beneath the GDPR consent manager—a list containing <code>1 cup of flour</code>.
Computers love this. It is basically archaeology, except every pottery fragment has a Pinterest button attached.
I only wanted somewhere nice to keep recipes.
More specifically, I wanted to cook food from more cultures and stop cycling through the same handful of meals like a cron job with low self-esteem. Swedish comfort food is lovely. So are Sichuan pork, Malaysian mushroom korma, Serbian meat rolls, Polish cabbage rolls, Italian ragù, British cakes, and whatever delicious thing I discover at 23:40 while browsing recipes instead of sleeping.
That is how I arrived at <a href="https://mealie.io/">Mealie</a>: an open-source, self-hosted recipe manager. It stores recipes, images, categories and tags; builds meal plans and shopping lists; scales quantities; organizes cookbooks; and imports recipes from the web. In other words, it is the sensible domestic application I needed, so naturally I deployed it onto Kubernetes with PostgreSQL, OIDC, persistent Ceph storage, Gateway API routing, and backups.
Some people keep recipes in a binder. I keep mine behind a CloudNativePG cluster. We all have coping mechanisms.
<h2 id="the-incident-report" class="heading-with-permalink">The incident report<a
 class="heading-permalink"
 href="#the-incident-report"
 aria-label="Copy link to section: The incident report"
 data-heading-permalink
 >#</a>
</h2>
Mealie&rsquo;s URL importer is genuinely useful. Paste a recipe URL and, in the happy path, a complete recipe appears with its title, image, ingredients, instructions, times and servings. That is the kind of feature that feels like magic until you import recipes from several countries and discover that the magic has regional settings.
One recipe says <code>2 cups flour</code>. Another says <code>200 millilitres whipping cream</code>. A third says <code>1 EL Öl</code>. Temperatures alternate between Celsius and Fahrenheit depending on which side of the Atlantic currently has custody of the oven. Instructions arrive with decorative entries such as <code>Make the sauce</code> occupying a whole step, followed by the actual instruction in the next step. Ingredient parsing occasionally decides that preparation notes are foods, units are philosophical suggestions, and fractions are an attack on its family.
What I wanted was less “preserve whatever the page happened to publish” and more this:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"> 1Source:
 21 cup water
 3Bake at 375°F.
 4Make the sauce:
 5Add the garlic, lemon juice and stock to the pan.
 6
 7Normalized:
 8240 ml water
 9Bake at 190°C.
10To make the sauce, add the garlic, lemon juice and stock to the pan.
</code></pre></div>Nothing revolutionary. Just consistent measurements, an oven setting compatible with the continent where my oven lives, and instructions that do not mistake typography for cooking.
Translation adds another layer. I do not want culturally specific dish names flattened into bland English product copy, but I do want instructions in one language I can reliably follow while holding a hot pan. <code>Hui Guo Rou</code> should remain <code>Hui Guo Rou</code>; the paragraph explaining when to add the doubanjiang can be English. This is a semantic decision, not a string replacement.
Under the hood, Mealie&rsquo;s first URL strategy uses <a href="https://github.com/hhursev/recipe-scrapers">hhursev/recipe-scrapers</a>, an impressive Python library with parsers for hundreds of recipe sites plus schema.org/JSON-LD support. For extraction, this is excellent engineering. Most competent recipe sites already publish machine-readable <code>Recipe</code> data because search engines reward them for doing so. The scraper can collect that data deterministically, quickly, locally, and without paying an API bill every time somebody wants soup.
The limitation is not that <code>recipe-scrapers</code> fails at its job. The limitation is that its job is scraping.
It can tell me the source says <code>1 cup all-purpose flour</code>. It should not casually decide whether that means 120 g, 125 g, or 140 g, because volume-to-weight conversion depends on the ingredient and sometimes on how enthusiastically somebody packed the cup. It should not rewrite an entire Polish recipe into natural English, decide that a visual section heading is not a cooking step, preserve a culturally meaningful name, and keep the converted temperature consistent everywhere it occurs.
That is no longer extraction. That is editorial normalization with consequences. A deterministic scraper is a very good postal worker; I was becoming annoyed that it would not also translate the letter, convert the measurements, remove the advertising, and check whether the instructions made sense. A wildly unfair performance review.
<h2 id="the-fallback-that-never-fell-back" class="heading-with-permalink">The fallback that never fell back<a
 class="heading-permalink"
 href="#the-fallback-that-never-fell-back"
 aria-label="Copy link to section: The fallback that never fell back"
 data-heading-permalink
 >#</a>
</h2>
Mealie already has OpenAI integration. It also supports custom prompt files through <code>OPENAI_CUSTOM_PROMPT_DIR</code>. Lovely. I wrote custom prompts, mounted them into the container, enabled an AI provider, and imported a normal recipe URL.
Nothing changed.
This was not an AI problem. It was an ordered list problem, the natural predator of engineering confidence.
At the time of this deployment, Mealie&rsquo;s default scraper order was:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">1DEFAULT_SCRAPER_STRATEGIES = [
2 RecipeScraperPackage,
3 RecipeScraperOpenAITranscription,
4 RecipeScraperOpenAI,
5 RecipeScraperOpenGraph,
6]
</code></pre></div>Mealie tries each strategy and returns the first usable result. <code>RecipeScraperPackage</code> is the wrapper around <code>recipe-scrapers</code>, and on most recipe pages it succeeds. That means the loop returns before <code>RecipeScraperOpenAI</code> ever gets a turn. My magnificent custom scrape prompt was mounted correctly and being ignored with flawless reliability.
Fallbacks only happen when the first choice fails. A result can be technically successful and still not be the result you wanted. Distributed systems engineers know this as “Tuesday.”
So I applied a tiny, deeply impolite patch:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">1DEFAULT_SCRAPER_STRATEGIES = [
2 RecipeScraperOpenAI,
3 RecipeScraperPackage,
4 RecipeScraperOpenAITranscription,
5 RecipeScraperOpenGraph,
6]
</code></pre></div>That is the whole behavioral change. OpenAI gets first refusal; the regular scraper remains as a fallback.
I call it monkey-patching because that communicates the correct level of shame, although technically it is a containerized file overlay. My OpenTofu creates a ConfigMap containing the patched <code>recipe_scraper.py</code> and mounts it over the installed module inside Mealie&rsquo;s container:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-hcl" data-lang="hcl">1volume_mount {
2 name = &#34;scraper-patch&#34;
3 mount_path = &#34;/opt/mealie/lib/python3.12/site-packages/mealie/services/scraper/recipe_scraper.py&#34;
4 sub_path = &#34;recipe_scraper.py&#34;
5 read_only = true
6}
</code></pre></div>The custom prompts are mounted separately under <code>/app/custom-prompts</code>, and Mealie is pointed there with:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text">1OPENAI_CUSTOM_PROMPT_DIR=/app/custom-prompts
</code></pre></div>Checksums of the environment, Secret, prompt data and Python patch live in the Deployment&rsquo;s pod-template annotations, so changing any of them causes Kubernetes to roll the pod. Infrastructure as code is wonderful because even my crimes against package ownership are reproducible.
There is an obvious maintenance bill: the patched file is copied from Mealie&rsquo;s installed version. Every Mealie upgrade now requires comparing that upstream module and verifying the path, imports and strategy interface still match. If upstream changes the scraper pipeline, my ConfigMap could replace new code with an old assumption and produce cuisine-flavoured sadness.
Do not cargo-cult this patch into production and then blame Python when an upgrade explodes. The repository README contains a large warning because future me deserves at least one witness.
<h2 id="what-the-ai-path-actually-does" class="heading-with-permalink">What the AI path actually does<a
 class="heading-permalink"
 href="#what-the-ai-path-actually-does"
 aria-label="Copy link to section: What the AI path actually does"
 data-heading-permalink
 >#</a>
</h2>
The clever part is that Mealie does not ask the model to emit its internal database objects directly.
<code>RecipeScraperOpenAI</code> strips the page into text, appends its JSON-LD blocks, finds a likely recipe image, and sends that material to the model with the <code>recipes.scrape-recipe</code> prompt. The model returns a JSON representation of a schema.org <code>Recipe</code>. Mealie then wraps that JSON in a tiny fake HTML document as an <code>application/ld+json</code> script and passes it back through the existing package scraper and cleaner.
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text">1web page
2 -&gt; visible text + JSON-LD + likely image
3 -&gt; GPT-5.6 Luna
4 -&gt; JSON representation of a schema.org Recipe
5 -&gt; synthetic &lt;script type=&#34;application/ld+json&#34;&gt;
6 -&gt; existing recipe-scrapers/Mealie cleaning path
7 -&gt; stored recipe
</code></pre></div>It is an adapter wearing a fake moustache so the mature import pipeline accepts the model&rsquo;s output. I mean that affectionately. Reusing the existing validation and cleaning path is much safer than creating a second route into the database just because the words “AI integration” have entered the meeting.
<h2 id="choosing-luna-instead-of-hiring-the-sun" class="heading-with-permalink">Choosing Luna instead of hiring the Sun<a
 class="heading-permalink"
 href="#choosing-luna-instead-of-hiring-the-sun"
 aria-label="Copy link to section: Choosing Luna instead of hiring the Sun"
 data-heading-permalink
 >#</a>
</h2>
I used <a href="https://developers.openai.com/api/docs/models/gpt-5.6-luna">GPT-5.6 Luna</a>, the fastest and least expensive member of the GPT-5.6 family.
The family has three tiers:
<ul>
<li>Sol is the flagship: the most capable option for hard coding, scientific, cyber and long-running agentic work.</li>
<li>Terra balances capability and cost for everyday professional work.</li>
<li>Luna is the fastest, lowest-cost tier.</li>
</ul>
At launch, <a href="https://openai.com/index/gpt-5-6/">OpenAI priced GPT-5.6</a> per million tokens at $5 input / $30 output for Sol, $2.50 / $15 for Terra, and $1 / $6 for Luna. The ratios are wonderfully easy: for the same token profile, Terra costs 2.5 times Luna and Sol costs five times Luna.
Sol would be spectacular overkill here. I am not asking the model to prove a theorem, coordinate agents across a week-long migration, or discover a novel compiler vulnerability. I am asking it to look at <code>8 oz cream cheese</code> and return roughly <code>225 g cream cheese</code> without turning the author&rsquo;s life story into Step 4.
Terra would also do the job, but the workload does not justify paying 2.5 times more unless testing shows a meaningful quality difference. It did not. Recipe extraction is bounded, repetitive, strongly instructed, and produces structured output. Luna already has much more linguistic and reasoning ability than this task needs.
Using the biggest available model for every workload is not architecture. It is standing at the kitchen counter slicing chives with a rescue helicopter.
<h2 id="the-prompts-became-the-product" class="heading-with-permalink">The prompts became the product<a
 class="heading-permalink"
 href="#the-prompts-became-the-product"
 aria-label="Copy link to section: The prompts became the product"
 data-heading-permalink
 >#</a>
</h2>
The <a href="https://github.com/mealie-recipes/mealie/tree/mealie-next/mealie/services/openai/prompts">default Mealie prompts</a> are intentionally compact. Its stock <code>scrape-recipe.txt</code> is essentially: extract recipe data from webpage content as schema.org <code>Recipe</code>, do not invent information, and return <code>{}</code> when there is not enough data. The stock ingredient prompt adds sensible basics about order, ambiguity, ranges, multilingual grammar and common units.
Those defaults have to work for everyone. Mine only has to work for my kitchen, where I want English recipe text, metric measurements, Celsius ovens, exact unit abbreviations and no instruction step whose sole contribution is <code>Make the sauce</code>.
I could not get that behavior from the default prompts because they never ask for it. Models are capable, not psychic. “Extract this recipe” does not secretly mean “apply Ellie&rsquo;s house style, but preserve the epistemological integrity of dumpling nomenclature.”
The full custom prompts live with the deployment, and I have also published <a href="https://gist.augustini.xyz/myceliatrix/e89a90923f31466ea3e6ee66ca3b6ad8/">both prompts in their full, gloriously over-specified form</a> for anyone who wants to read the actual incantations instead of my civilized summary. Their jobs break down like this.
<h3 id="prompt-one-scrape-and-normalize-the-whole-recipe" class="heading-with-permalink">Prompt one: scrape and normalize the whole recipe<a
 class="heading-permalink"
 href="#prompt-one-scrape-and-normalize-the-whole-recipe"
 aria-label="Copy link to section: Prompt one: scrape and normalize the whole recipe"
 data-heading-permalink
 >#</a>
</h3>
<code>scrape-recipe.txt</code> is the large one. It tells Luna to:
<ol>
<li>Find the actual recipe. Prefer the author&rsquo;s content over navigation, SEO text, advertising, comments, subscription prompts and related recipes. Deduplicate responsive layouts, print views and repeated metadata.</li>
<li>Return one schema.org Recipe object. Unsupported fields are omitted, insufficient pages become <code>{}</code>, and invented data is forbidden.</li>
<li>Normalize measurements sensibly. US customary units become practical metric values, Fahrenheit becomes Celsius, but teaspoons, tablespoons, counts and package units remain when conversion would make the recipe worse.</li>
<li>Avoid laboratory cosplay. <code>1 cup water</code> can become about <code>240 ml</code>; it must not become <code>236.588 ml</code>, because my measuring jug has not completed a metrology doctorate.</li>
<li>Use a unit house style. Metric units become <code>mg</code>, <code>g</code>, <code>kg</code>, <code>ml</code> and <code>l</code>; kitchen units become <code>tsp</code> and <code>tbsp</code>. No <code>grams</code> in one ingredient, <code>g</code> in the next, and <code>grammes</code> arriving later with a fake passport.</li>
<li>Keep countable things countable. <code>4 cloves garlic</code> remains natural language. It does not become <code>4 units garlic cloves</code>, a phrase only a procurement database could love.</li>
<li>Clean the instructions. Every step needs a concrete cooking action. Heading-only steps are removed or merged into the following action. Temperatures and repeated quantities must agree with the normalized ingredient list.</li>
<li>Translate into natural English. Titles, descriptions, ingredients and instructions are translated, while proper nouns, brands, place names and culturally specific dish names survive when translation would reduce precision.</li>
<li>Choose the food image. Prefer the main finished-dish image, reject logos, portraits, ads, pixels and decorative nonsense, and return one absolute URL.</li>
<li>Audit itself. The prompt ends with a validation checklist covering ingredient order, unit forms, practical quantities, action-only instructions, image URLs and unsupported claims.</li>
</ol>
That final checklist mattered more than I expected. Long prompts are not automatically good prompts; often they are merely documentation that bills by the token. But explicit invariants gave Luna something concrete to verify before returning the JSON.
<h3 id="prompt-two-parse-ingredients-without-eating-any" class="heading-with-permalink">Prompt two: parse ingredients without eating any<a
 class="heading-permalink"
 href="#prompt-two-parse-ingredients-without-eating-any"
 aria-label="Copy link to section: Prompt two: parse ingredients without eating any"
 data-heading-permalink
 >#</a>
</h3>
After importing, Mealie can parse ingredient strings into structured quantity, unit, food and note fields. My <code>parse-recipe-ingredients.txt</code> prompt treats that as a lossless transformation:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text">1&#34;2 tbsp olive oil, plus more for frying&#34;
2
3quantity: 2
4unit: tbsp
5food: olive oil
6note: plus more for frying
</code></pre></div>Its most important rule is one input ingredient in, one structured ingredient out, in exactly the same order. Never merge, split, remove, reorder or invent ingredients. If a fragment is uncertain, preserve it in the note instead of confidently throwing it into the void.
The rest handles vulgar fractions, mixed numbers, written numbers, ranges, grouped quantities such as <code>2 dozen</code>, canonical units, preparation notes, multilingual grammar and ambiguous tokens. It explicitly separates recognition from output: the model may understand <code>tablespoons</code>, <code>tbs</code> and <code>tbsp</code>, but the stored unit must be exactly <code>tbsp</code>.
This distinction sounds fussy until shopping-list aggregation meets <code>tablespoon</code>, <code>tablespoons</code>, <code>tbsp</code> and <code>T</code>. Normalization is how four aliases stop pretending to be four pantry items.
The two prompts deliberately have different language policies. Whole-recipe scraping translates non-English recipes into English. Ingredient parsing preserves its source language unless translation was explicitly requested, because it may be invoked independently on an existing recipe. Context matters. Apparently even dinner needs an API contract.
<h2 id="eight-commits-one-afternoon-several-unit-related-opinions" class="heading-with-permalink">Eight commits, one afternoon, several unit-related opinions<a
 class="heading-permalink"
 href="#eight-commits-one-afternoon-several-unit-related-opinions"
 aria-label="Copy link to section: Eight commits, one afternoon, several unit-related opinions"
 data-heading-permalink
 >#</a>
</h2>
The prompts did not work perfectly on the first attempt because nothing works perfectly on the first attempt except <code>rm -rf</code>, and that works much too well.
I tested against recipes from different sites, languages and formatting traditions. Then I inspected what Mealie actually stored, found a new species of nonsense, tightened an invariant, and tried again. The deployment history records eight prompt-and-scraper refinements on July 13 alone.
Early versions said “convert to metric” but did not constrain the final vocabulary, so long unit names still leaked through. Then I required abbreviations. The model produced heading-only instruction steps, so the prompt gained action tests and merge rules. Images needed stricter selection. Counts needed protection from over-normalization. Translation needed an explicit boundary around culturally meaningful names. Ingredient parsing needed stronger lossless guarantees.
Prompt engineering, at least the useful kind, looks less like wizardry and more like property-based testing conducted by an increasingly opinionated cook:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text">1invariant: ingredient_count(output) == ingredient_count(input)
2invariant: every instruction contains a cooking action
3invariant: Fahrenheit does not survive normalization
4invariant: supported metric units belong to {mg, g, kg, ml, l}
5invariant: culturally meaningful names are not translated into oatmeal
</code></pre></div>The goal was not to describe every possible recipe on Earth. The goal was to identify failures I actually observed and convert them into rules precise enough to test on the next import.
<h2 id="the-bill-featuring-an-anticlimactic-amount-of-capitalism" class="heading-with-permalink">The bill, featuring an anticlimactic amount of capitalism<a
 class="heading-permalink"
 href="#the-bill-featuring-an-anticlimactic-amount-of-capitalism"
 aria-label="Copy link to section: The bill, featuring an anticlimactic amount of capitalism"
 data-heading-permalink
 >#</a>
</h2>
<img src="https://augustini.wtf/fieldnotes/hacking-mealie-for-fun-and-dinner/openai-usage_hu_30a3bec08f2f8a6e.webp" alt="OpenAI platform usage after prompt testing and importing roughly thirty recipes: 360,724 tokens, 73 requests and $0.73 spent." width="1024" height="136" loading="lazy" decoding="async">
After all the prompt testing and importing roughly thirty recipes, the OpenAI platform reported:
<ul>
<li>73 requests</li>
<li>360,724 total tokens</li>
<li>$0.73 spent</li>
<li>0 blocked requests</li>
</ul>
That is about one US cent per request across the entire messy test session, or roughly two and a half cents per finished recipe if I unfairly charge all prompt-development traffic to the thirty recipes. Future imports should be cheaper because I am no longer repeatedly asking the model whether <code>millilitres</code> means <code>ml</code> and then updating a text file when it gets creative.
With the same token mix, Terra would have cost about $1.83 and Sol about $3.65. None of those totals would endanger the household economy, but cost discipline is a habit. The correct question is not “can I afford the flagship?” It is “does the flagship improve this workload enough to justify five times the price?”
For parsing recipes, Luna&rsquo;s answer was already good. Sol cannot make 190°C more Celsius.
<h2 id="the-aha-moment" class="heading-with-permalink">The aha moment<a
 class="heading-permalink"
 href="#the-aha-moment"
 aria-label="Copy link to section: The aha moment"
 data-heading-permalink
 >#</a>
</h2>
The original scraper and the model are not really competitors. They solve different layers of the problem.
<code>recipe-scrapers</code> extracts what a page says. Luna applies a house policy to what it means. Mealie&rsquo;s existing cleaner then converts the result into application data. The small Python patch merely changes which specialist speaks first.
That layering is why the result feels much better than replacing everything with “AI.” Deterministic code still fetches pages, controls the strategy pipeline, validates schema-shaped output, cleans fields and stores data. The model handles the fuzzy boundary where language, culture and inconsistent measurements make rigid parsers miserable.
The lesson is not that every scraper needs an LLM. Most do not. If you need faithful extraction from known sites, <code>recipe-scrapers</code> is faster, cheaper and more predictable. My requirement was different: translate, normalize, preserve meaning, reconcile repeated values, and enforce a personal editorial standard across many sources. That is precisely where a language model earns its tiny invoice.
<h2 id="the-post-mortem" class="heading-with-permalink">The post-mortem<a
 class="heading-permalink"
 href="#the-post-mortem"
 aria-label="Copy link to section: The post-mortem"
 data-heading-permalink
 >#</a>
</h2>
My Mealie library now contains food from the places I wanted to explore, with English instructions, sensible metric quantities, Celsius temperatures, useful images and ingredients that can participate in shopping lists without developing multiple identities.
The final system is wonderfully disproportionate:
<ul>
<li>Mealie runs as a single restricted Kubernetes pod.</li>
<li>CloudNativePG stores the database.</li>
<li>Authentik handles OIDC.</li>
<li>OpenTofu mounts custom prompts and one patched Python module through ConfigMaps.</li>
<li>GPT-5.6 Luna turns hostile food-blog archaeology into a normalized JSON representation of schema.org <code>Recipe</code> data.</li>
<li>The regular scraper remains available when the AI path cannot help.</li>
</ul>
Would a notebook have been simpler? Yes. A notebook also refuses to translate a Serbian recipe, convert the oven temperature, build a shopping list and roll itself after a prompt checksum changes. Checkmate, paper.
The patch is small. The prompts are not. That is the real shape of this project: six rearranged lines of Python opened the route, but most of the engineering went into defining what “a clean recipe” actually means.
If you try something similar, start with the cheapest capable model, test on hostile real-world inputs, write down invariants instead of vibes, and keep deterministic validation on both sides of the model. Also leave yourself a very loud upgrade warning when you mount a ConfigMap over somebody else&rsquo;s Python package.
Future you will already be maintaining Kubernetes for a recipe manager. Be kind to her.
The pink keyboard firmware incidenthttps://augustini.wtf/fieldnotes/the-pink-keyboard-firmware-incident/Sun, 21 Jun 2026 02:32:21 +0200https://augustini.wtf/fieldnotes/the-pink-keyboard-firmware-incident/Ellie AugustiniqmkfirmwarekeyboardembeddeddebuggingisoBuying a keyboard because it is pink is a perfectly reasonable engineering decision. Anyone who disagrees has never spent eight hours a day communicating with a computer through a beige rectangle designed by a committee that thought &ldquo;texture&rdquo; meant &ldquo;slightly different plastic sadness.&rdquo;Buying a keyboard because it is pink is a perfectly reasonable engineering decision. Anyone who disagrees has never spent eight hours a day communicating with a computer through a beige rectangle designed by a committee that thought &ldquo;texture&rdquo; meant &ldquo;slightly different plastic sadness.&rdquo;
So there I was at Gliched, near Elgiganten, doing the responsible adult thing: impulse-purchasing a NuPhy Halo75 V2 because it was pink, cute, and I needed a new keyboard. That is not a supply-chain event. That is self-care with USB-C.
Then I did what any normal person would do after buying a keyboard: I went to the manufacturer&rsquo;s website, downloaded the firmware for the model I owned, flashed it, and expected my keys to continue being keys.
This was my first mistake. My second mistake was assuming &ldquo;firmware for this model&rdquo; included the ISO version of this model. My third mistake was having hope, which remains the most dangerous undefined behavior in consumer electronics.
The firmware flashed fine. The keyboard booted. The RGB did its little capitalism rainbow. And then the ISO keys were dead.
Not metaphorically dead. Actually dead. The useful Scandinavian/European bits of the board, the keys that make an ISO keyboard an ISO keyboard instead of an ANSI board wearing a taller Enter key as a disguise, simply stopped participating in society. The <code>&lt; &gt;</code> key, the section/sign key, the extra punctuation choreography, all gone sideways.
I had taken a working keyboard and converted it into a very expensive pink argument.
<h2 id="the-incident-report" class="heading-with-permalink">The incident report<a
 class="heading-permalink"
 href="#the-incident-report"
 aria-label="Copy link to section: The incident report"
 data-heading-permalink
 >#</a>
</h2>
NuPhy published firmware for the Halo75 V2, but the downloadable build was effectively for ANSI. Flash that onto an ISO board and the matrix does exactly what firmware always does: precisely the wrong thing, with confidence.
I asked NuPhy for the ISO firmware. They could not provide it.
This is the part where the story should end with a support link, a zip file, and maybe a mild amount of shame about flashing vendor firmware like it came from a reliable civilization. Instead, the trail led to an abandoned-looking GitHub repository containing NuPhy&rsquo;s QMK source for the ANSI version.
The good news: source code existed.
The bad news: it was old QMK source, vendor-shaped, copied across variants, and written in the dialect of embedded C where every file has three jobs, six globals, and at least one function name that looks like it was translated through a fax machine.
The worse news: I was now emotionally invested.
On June 16, 2026, the first commit landed on the <code>nuphy-keyboards</code> branch:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text">13064e8c506f Start working on ISO version of the halo75 v2 and add woodpecker
</code></pre></div>That commit was not a delicate edit. It added a full ISO variant: keyboard metadata, keymaps, LED tables, RF code, side LED code, sleep handling, VIA support, and CI. About 4,800 lines of &ldquo;I bought this keyboard yesterday and now apparently I maintain it.&rdquo;
At this point, the problem looked simple in the same way a leaking pipe looks simple before you learn the wall is load-bearing.
The ISO board needed:
<ul>
<li>the right matrix positions,</li>
<li>the right ISO keycodes,</li>
<li>the right VIA definition,</li>
<li>the right LED map,</li>
<li>and the firmware to stop treating Nordic punctuation like optional DLC.</li>
</ul>
So I fixed keymaps. Then fixed keymaps again. Then fixed RGB. Then fixed an invalid matrix position. Then did some more layout fixes. The commit log from June 16 is basically firmware debugging as a cardio exercise:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text">1Fix invalid matrix position
2Maybe keyfixes for iso?
3Fix rgb
4Fix keymaps
5Fix keymaps
6Fix keymaps
7Minior layout fixes
</code></pre></div>Yes, &ldquo;Minior.&rdquo; The typo is part of the archaeological record. Leave it in the museum.
<h2 id="the-rabbit-hole" class="heading-with-permalink">The rabbit hole<a
 class="heading-permalink"
 href="#the-rabbit-hole"
 aria-label="Copy link to section: The rabbit hole"
 data-heading-permalink
 >#</a>
</h2>
The original NuPhy source was based around QMK 0.25.7-ish. Current upstream QMK had moved on to 0.33.7. That does not sound like a huge jump if you think version numbers are decorative. In firmware land, it means APIs moved, config schemas changed, old aliases disappeared, and whatever worked before now has the social stability of a JavaScript build tool after a minor release.
On June 18, the branch became <code>nuphy-on-current</code>, and this happened:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text">1977cf375c51 Port nuphy keyboards onto QMK 0.33.7; strip non-nuphy keyboards
</code></pre></div>That was the forklift upgrade. The old NuPhy code got dragged onto modern QMK, and the repo was stripped down so it was no longer carrying the entire keyboard directory like a cursed shipping container.
The migration broke in all the ways you would expect and several ways you would not unless you have personally offended an STM32.
QMK had renamed metadata expectations. Some keyboards had <code>keyboard.json</code>; newer QMK wanted <code>info.json</code> in the right places. RGB matrix coordinates needed to validate as integers. Old RGB keycodes had been deprecated, so the firmware needed modern <code>RM_*</code> names instead of legacy <code>RGB_*</code> aliases. EEPROM datablock functions had changed shape. GPIO helpers had changed names. ChibiOS configuration needed attention.
None of this is glamorous. This is the work where a compiler looks at you and says, &ldquo;I see you brought me 2024 code. Unfortunately, this is 2026, and I have developed standards.&rdquo;
Then came the proper embedded bugs, the ones that do not politely fail at compile time.
<h2 id="the-keyboard-that-typed-from-the-wrong-pins" class="heading-with-permalink">The keyboard that typed from the wrong pins<a
 class="heading-permalink"
 href="#the-keyboard-that-typed-from-the-wrong-pins"
 aria-label="Copy link to section: The keyboard that typed from the wrong pins"
 data-heading-permalink
 >#</a>
</h2>
The best bug was the UART pin conflict.
The Halo75 V2 uses an STM32F072 talking to an nRF wireless module. The keyboard matrix also uses MCU pins. This is normal until the firmware configures the wrong pins for UART and accidentally turns matrix columns into a serial peripheral.
That is not &ldquo;a bug.&rdquo; That is a séance with scan codes.
The QMK UART driver defaulted to pins that were fine in some universe, just not this one. On this board, USART1 needed PB6/PB7. Without overriding the UART TX/RX pins, the driver defaulted to A9/A10, which were matrix columns 13 and 14. So when <code>rf_uart_init()</code> ran, it reconfigured keyboard matrix pins as USART pins, and the keyboard started inventing input like it had discovered improv.
The fix landed at 05:15 on June 19:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text">1dea005ec792 Fix Nuphy Halo75 V2 ISO: LED matrix, typing, and UART pin conflict
</code></pre></div>That commit brought the board back toward reality: restore the IS31FL3733 compatibility macros, fix I2C addresses, re-enable the RF housekeeping path, bring the RGB matrix power rail up at the right point, restore DMA config, add missing LED positions, and force the UART pins to the board&rsquo;s actual hardware.
This is the part of firmware work where the abstraction stack politely excuses itself and you are left reading datasheets like a medieval monk with a logic analyzer.
Then Bluetooth still did not work.
The keyboard would show the pairing blink locally, but nothing appeared in the Bluetooth device list. That is a special kind of insult: the UI says &ldquo;pairing,&rdquo; the radio says nothing, and your laptop says &ldquo;new phone who dis.&rdquo;
The next commit found it:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text">149352a2a3ac Fix RF/Bluetooth: set UART_TX/RX_PAL_MODE = 0 for STM32F072
</code></pre></div>On STM32F072, USART1 on PB6/PB7 uses alternate function 0. QMK&rsquo;s default UART alternate-function mode was 7, appropriate for STM32F4-style expectations, but wrong here. AF7 on those pins was not USART1. The nRF module was never receiving the command to advertise. The keyboard was blinking into the void.
Firmware debugging is mostly asking, &ldquo;is the hardware lying, is the code lying, or did I accidentally configure a comparator output and call it Bluetooth?&rdquo;
The answer, as usual, was yes.
<h2 id="the-led-side-quest-because-of-course-there-was-one" class="heading-with-permalink">The LED side quest, because of course there was one<a
 class="heading-permalink"
 href="#the-led-side-quest-because-of-course-there-was-one"
 aria-label="Copy link to section: The LED side quest, because of course there was one"
 data-heading-permalink
 >#</a>
</h2>
Once the keyboard typed, the LEDs demanded legal representation.
The Halo75 V2 has 128 LEDs driven by IS31FL3733 chips. The main keys, status indicators, battery/system lights, and side/rim LEDs all share the stage. This is fine until RGB animations decide the entire LED array belongs to them and overwrite status indicators like a startup founder overwriting a database with a Notion export.
The commit log from late June 18 into early June 19 reads like someone negotiating with a disco:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text">1Fix side LED bugs so it leaves status leds alone
2Fix rim animations overwriting status LEDs
3Fix custom RGB matrix effects overwriting status LEDs
4Fix startup animation power_play_index bound
5Slow down startup animation timing for smoother effect
6Skip status LED indices in startup animation to prevent interference
7Prevent animations from turning off status LEDs
8Add rgb_matrix_indicators_user hook to restore status LEDs after built-in effects
</code></pre></div>The architectural fix was to separate responsibility. QMK RGB matrix effects can draw the main keyboard lighting, but status and side LEDs need to win at the end of the frame. Otherwise the keyboard loses the ability to tell you useful things like battery, mode, and &ldquo;please stop turning my indicators into vaporwave.&rdquo;
Later, that turned into a cleaner shared implementation:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text">1776478e78ab Major refactor, share the code between ansi and iso as much as possible
</code></pre></div>Before that refactor, ANSI and ISO carried duplicate copies of RF code, side LED code, sleep code, headers, and config. That is how you grow two firmware variants that look related but fail in different accents. The refactor pulled shared behavior up into <code>keyboards/nuphy/halo75_v2/</code>, leaving the variants to describe what is actually different: physical layout, keymaps, and LED zoning.
The result was smaller, less duplicated, and less likely to require fixing the same UART bug twice. A low bar, yes, but embedded firmware is built on respecting low bars. They are usually connected to ground.
<h2 id="the-performance-pass" class="heading-with-permalink">The performance pass<a
 class="heading-permalink"
 href="#the-performance-pass"
 aria-label="Copy link to section: The performance pass"
 data-heading-permalink
 >#</a>
</h2>
After the board typed and lit up, it still felt sluggish.
There is a very specific humiliation in making a boutique mechanical keyboard feel worse than a three-dollar office keyboard that has seen things in a municipal procurement closet. The Halo75 V2 had side LED work, RF housekeeping, UART command handling, sleep handling, and RGB animation all sharing a tiny real-time budget. Blocking waits in hot-ish paths are how you turn &ldquo;firmware&rdquo; into &ldquo;firm maybe later.&rdquo;
So June 19 turned into a latency cleanup:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text">1cf1f86e9329 Remove blocking waits from hot-ish paths
28f1c1a0b4d7 Cap rgb animations at approx. 60 fps
31b5728b8298 Coalesce/drop low-priority deferred UART commands
40314ae48b22 Harden RF parser and make device reset non-blocking
51b9638856b3 Remove dead ack-wait from uart_send_cmd, convert sleep callers to deferred
</code></pre></div>The idea was straightforward: keep the main loop moving. Do not block the scan path waiting for decorative work. Do not make a factory reset freeze the firmware for over a second. Do not let low-priority UART chatter pile up while key reports are trying to leave the building.
This is the point where &ldquo;keyboard firmware&rdquo; stops being a keymap and becomes a tiny cooperative scheduler with lighting and emotional baggage.
The RF parser also got stricter. Instead of trusting arbitrary packet lengths and setting ACK/sync state too early, it started validating command-specific lengths and resetting RX state on parse exits. In C, optimism is not a strategy. It is how you spend Saturday night discovering that one stale byte can become a lifestyle.
<h2 id="the-hid-console-plot-twist" class="heading-with-permalink">The HID console plot twist<a
 class="heading-permalink"
 href="#the-hid-console-plot-twist"
 aria-label="Copy link to section: The HID console plot twist"
 data-heading-permalink
 >#</a>
</h2>
Then came the weird one: Cmd+V behaved strangely.
At this stage, it was reasonable to suspect ghosting. The keyboard had already had a real UART/matrix pin conflict. The firmware had just been ported across years of QMK changes. There were RF paths, NKRO reports, modifier state, layers, debounce, and a physical matrix all involved. Plenty of suspects. Nobody had an alibi.
So I added a debug console:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text">10de414bd565 debug: add matrix/report debug console for input diagnostics
</code></pre></div>It can be enabled like this:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh">1qmk flash -kb nuphy/halo75_v2/iso -km via -e CONSOLE_ENABLE=yes
</code></pre></div>The console logs raw matrix rows, debounced rows, per-column masks, key events, layers, mods, weak mods, oneshot mods, and the row state at the moment each event fires. It is intentionally cheap: compare <code>MATRIX_ROWS</code>, skip output if nothing changed, allocate nothing, and print only when useful.
This is exactly the kind of tool you write when you are convinced the machine is gaslighting you and you would like timestamps for the deposition.
And then the console proved the keyboard was innocent.
The &ldquo;ghosting&rdquo; was me.
Because of nerve damage in my left hand, I was accidentally bumping Ctrl without feeling it. So Cmd+V was sometimes not Cmd+V in the clean little mental model where fingers are reliable peripherals. The firmware was fine. The matrix was fine. The report was fine. The bug was an accessibility mismatch between switch resistance and sensory feedback.
That is the kind of debugging revelation that makes you stare at the ceiling for a moment.
Hours of firmware investigation. A custom HID console. Matrix masks. Modifier reports. RF paranoia. And the fix was not a bitmask.
The fix was replacing the Ctrl switch with one that has higher resistance, so I can feel when I bump it.
Honestly? That is beautiful. Annoying, but beautiful.
Software people love pretending all bugs live in software. Embedded work cures you of that disease. The system includes the PCB, the MCU, the wireless module, the firmware, the operating system, the desk, the key switch, and the human hand attached to the debugging session. Ignore any layer and it will eventually file a bug report in production.
<h2 id="the-post-mortem" class="heading-with-permalink">The post-mortem<a
 class="heading-permalink"
 href="#the-post-mortem"
 aria-label="Copy link to section: The post-mortem"
 data-heading-permalink
 >#</a>
</h2>
By June 20, the branch had reached:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text">1d1fe40f3bcc Minior bugfix
</code></pre></div>The repository is public, if you want to inspect the suffering in its natural habitat: <a href="https://code.augustini.xyz/myceliatrix/qmk_firmware">myceliatrix/qmk_firmware</a>.
From June 16 to June 20, the firmware went from &ldquo;I would like my ISO keys back&rdquo; to a modern QMK 0.33.7 port with:
<ul>
<li>ISO and ANSI variants,</li>
<li>shared Halo75 V2 firmware code,</li>
<li>VIA-compatible keymaps,</li>
<li>fixed ISO matrix/key positions,</li>
<li>working LED matrix and side LED zoning,</li>
<li>IS31FL3733 compatibility fixes,</li>
<li>UART pin and alternate-function fixes for STM32F072,</li>
<li>RF/Bluetooth housekeeping restored,</li>
<li>USB suspend behavior fixed for wireless mode,</li>
<li>fewer blocking waits,</li>
<li>a stricter RF parser,</li>
<li>deferred UART command handling,</li>
<li>and a HID console that found a human factors bug wearing a firmware costume.</li>
</ul>
This is why &ldquo;just flash the firmware&rdquo; is a cursed phrase. Firmware is not an app update. Firmware is a tiny treaty between silicon, timing, voltage, physical switches, protocol state, and somebody&rsquo;s hand at 01:07 in the morning.
The lesson is not &ldquo;never buy a pink keyboard impulsively.&rdquo; That would be cowardice, and also aesthetically wrong.
The lesson is: if a device ships with QMK, source matters. If the vendor only publishes one layout&rsquo;s firmware, layout matters. If an old firmware tree works, the exact QMK version matters. If wireless goes quiet, alternate functions matter. If keys appear haunted, matrix logs matter. And if your body is part of the input system, switch feel matters too.
Also, before assuming your firmware is broken, maybe check whether your Ctrl key is being lovingly shoulder-checked by a hand with partial sensation.
But still write the debug console.
Future you deserves receipts.
How this site works: a tiny static site in a surprisingly serious trenchcoathttps://augustini.wtf/fieldnotes/how-this-site-works/Sun, 07 Jun 2026 14:00:00 +0200https://augustini.wtf/fieldnotes/how-this-site-works/Ellie Augustinihugoinfrastructuretalosopentofuci-cdsecuritykubernetesself-hostingThis website looks like a small terminal window that wandered into a pastel cabinet. That is the fun part. The less visible part is that it is built like a production service: reproducible static build, minimal container, private registry, pinned deployments, restricted Kubernetes namespace, Gateway API routes, and OpenTofu state living in a Rook Ceph object store.This website looks like a small terminal window that wandered into a pastel cabinet. That is the fun part. The less visible part is that it is built like a production service: reproducible static build, minimal container, private registry, pinned deployments, restricted Kubernetes namespace, Gateway API routes, and OpenTofu state living in a Rook Ceph object store.
This is perhaps excessive for a personal website. That is also the point.
A static site is one of the nicest places to practice infrastructure discipline because the application itself is brutally simple. There is no database migration trying to eat your evening. No websocket fleet. No &ldquo;just one background worker&rdquo; that secretly wants to become a platform. The whole runtime requirement is: please serve these files over HTTP without doing anything cursed.
So this post is a tour of the whole thing. Not just the Hugo bits in this repository, but also the deployment estate in <code>/Users/ellie/talos/opentofu-apps/augustini-wtf</code>, because that repo is the other half of the creature. One repo makes the artifact. The other repo tells the cluster how to run it.
<h2 id="the-shape-of-the-system" class="heading-with-permalink">The shape of the system<a
 class="heading-permalink"
 href="#the-shape-of-the-system"
 aria-label="Copy link to section: The shape of the system"
 data-heading-permalink
 >#</a>
</h2>
There are two repositories involved:
<ul>
<li><code>augustini.wtf</code>: the website source, Hugo theme, Tailwind source CSS, Woodpecker pipeline, and scratch-image Dockerfile.</li>
<li><code>talos/opentofu-apps/augustini-wtf</code>: the OpenTofu workspace that creates Kubernetes resources and Gateway API routes for the site.</li>
</ul>
The deployment path looks like this:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"> 1git push
 2 -&gt; Woodpecker builds Hugo site
 3 -&gt; Tailwind emits the final theme CSS
 4 -&gt; Hugo writes public/
 5 -&gt; Docker Buildx packages public/ with static-web-server
 6 -&gt; Harbor receives latest and a timestamp tag
 7 -&gt; Woodpecker updates Terrakube variable image_tag
 8 -&gt; Terrakube runs the OpenTofu workspace
 9 -&gt; Kubernetes Deployment rolls to the pinned image
10 -&gt; Gateway API sends augustini.wtf traffic to the Service
</code></pre></div>The design goal is boring in the good way: the website repo should not need cluster credentials, my laptop should not be the deployment authority, and Kubernetes should not run an image named <code>latest</code> while everyone politely pretends that is a version.
<h2 id="hugo-the-part-that-makes-pages" class="heading-with-permalink">Hugo: the part that makes pages<a
 class="heading-permalink"
 href="#hugo-the-part-that-makes-pages"
 aria-label="Copy link to section: Hugo: the part that makes pages"
 data-heading-permalink
 >#</a>
</h2>
The site is Hugo with a local theme named <code>techprincess</code>. The root <code>hugo.toml</code> is intentionally conventional:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml">1baseURL = &#39;https://augustini.wtf/&#39;
2title = &#39;Ellie Augustini&#39;
3theme = &#39;techprincess&#39;
4enableRobotsTXT = true
5summaryLength = 24
</code></pre></div>Content lives under <code>content/</code>: home, about, fieldnotes, projects, and individual project/fieldnote pages. Hugo&rsquo;s content model does most of the routing without ceremony. A post under <code>content/fieldnotes/foo.md</code> becomes a fieldnote. A project under <code>content/projects/foo.md</code> becomes a project page. The theme layouts decide how those sections feel.
The theme is very much not a generic &ldquo;minimal fieldnotes&rdquo; theme. It is a terminal UI cosplay machine, but with the useful parts kept and the fake-terminal nonsense mostly avoided. The base layout wraps the page in a bordered panel with a title bar, three colored dots, a scanline overlay, and then regular semantic HTML inside it:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-html" data-lang="html"> 1&lt;body class=&#34;min-h-screen bg-background text-foreground font-mono antialiased&#34;&gt;
 2 &lt;a class=&#34;skip-link&#34; href=&#34;#main&#34;&gt;...&lt;/a&gt;
 3 &lt;div class=&#34;...&#34;&gt;
 4 &lt;div class=&#34;... border border-border bg-card ...&#34;&gt;
 5 &lt;div class=&#34;... border-b border-border bg-secondary/60 ...&#34;&gt;
 6 &lt;span class=&#34;... bg-pink&#34;&gt;&lt;/span&gt;
 7 &lt;span class=&#34;... bg-gold&#34;&gt;&lt;/span&gt;
 8 &lt;span class=&#34;... bg-mint&#34;&gt;&lt;/span&gt;
 9 &lt;span&gt;ellie@princess: ~&lt;/span&gt;
10 &lt;/div&gt;
11 &lt;div class=&#34;scanlines&#34;&gt;
12 {{ partial &#34;header.html&#34; . }}
13 &lt;main id=&#34;main&#34; tabindex=&#34;-1&#34;&gt;
14 {{ block &#34;main&#34; . }}{{ end }}
15 &lt;/main&gt;
16 &lt;/div&gt;
17 &lt;/div&gt;
18 {{ partial &#34;footer.html&#34; . }}
19 &lt;/div&gt;
20&lt;/body&gt;
</code></pre></div>There is a real skip link. The main region is focusable. The terminal aesthetic is allowed to be cute, but it is not allowed to eat the accessibility basics.
<h2 id="the-theme-one-font-many-little-constraints" class="heading-with-permalink">The theme: one font, many little constraints<a
 class="heading-permalink"
 href="#the-theme-one-font-many-little-constraints"
 aria-label="Copy link to section: The theme: one font, many little constraints"
 data-heading-permalink
 >#</a>
</h2>
The visual system is all Iosevka, all the time. The font files are self-hosted from <code>themes/techprincess/static/fonts/</code>, so the finished site does not ask Google Fonts, Bunny, Adobe, or some random CDN for permission to render text.
The full Iosevka latin woff2 files are about a megabyte each, which is silly for a site whose pages are a few hundred kilobytes. After Hugo writes <code>public/</code>, a small Python step (<code>scripts/subset-fonts.py</code>, run via <code>npm run subset-fonts</code>) walks every rendered <code>.html</code>, collects the set of characters actually used, and runs <code>pyftsubset</code> over each source font to produce a subset covering exactly those glyphs (plus a small safety set for the footer clock and CSS <code>content</code> bullets). The source fonts stay untouched; only <code>public/fonts/</code> is rewritten. The result is roughly 4 MB of fonts shrinking to about 195 KB total, with no visible difference.
Tailwind is used as a compiler and constraint system, not as a browser dependency. <code>package.json</code> has the important scripts:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json">1{
2 &#34;scripts&#34;: {
3 &#34;build:css&#34;: &#34;tailwindcss -i themes/techprincess/assets/css/source.css -o themes/techprincess/assets/css/main.css --minify&#34;,
4 &#34;subset-fonts&#34;: &#34;python3 scripts/subset-fonts.py&#34;,
5 &#34;build&#34;: &#34;npm run build:css &amp;&amp; hugo --minify &amp;&amp; npm run subset-fonts&#34;,
6 &#34;serve&#34;: &#34;npm run build:css &amp;&amp; hugo serve --disableFastRender&#34;
7 }
8}
</code></pre></div>The Tailwind config scans Hugo content and layouts, the theme JS, and both Hugo config files. It also enables Iconify&rsquo;s dynamic selectors so the templates can use classes like <code>icon-[lucide--heart]</code> and <code>icon-[simple-icons--forgejo]</code> without manually vendoring SVGs into every partial.
The palette is expressed as OKLCH tokens in <code>source.css</code>:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-css" data-lang="css"> 1:root {
 2 --background: 0.16 0.012 300;
 3 --foreground: 0.92 0.015 330;
 4 --card: 0.2 0.015 305;
 5 --primary: 0.8 0.16 350;
 6 --accent: 0.86 0.16 168;
 7 --pink: 0.82 0.17 352;
 8 --mint: 0.86 0.16 168;
 9 --gold: 0.87 0.15 92;
10 --lilac: 0.8 0.12 300;
11}
</code></pre></div>Tailwind maps those into named colors:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-js" data-lang="js">1colors: {
2 background: &#39;oklch(var(--background) / &lt;alpha-value&gt;)&#39;,
3 foreground: &#39;oklch(var(--foreground) / &lt;alpha-value&gt;)&#39;,
4 pink: &#39;oklch(var(--pink) / &lt;alpha-value&gt;)&#39;,
5 mint: &#39;oklch(var(--mint) / &lt;alpha-value&gt;)&#39;,
6 gold: &#39;oklch(var(--gold) / &lt;alpha-value&gt;)&#39;,
7 lilac: &#39;oklch(var(--lilac) / &lt;alpha-value&gt;)&#39;,
8}
</code></pre></div>That means the layouts can stay readable: <code>border-pink/60</code>, <code>text-muted-foreground</code>, <code>bg-secondary/50</code>, and so on. No component library. No design token runtime. Just CSS variables, Tailwind&rsquo;s build step, and a strong preference for rectangles with manners.
<h2 id="hugo-pipes-cache-busted-assets-without-a-bundler-shaped-monster" class="heading-with-permalink">Hugo pipes: cache-busted assets without a bundler-shaped monster<a
 class="heading-permalink"
 href="#hugo-pipes-cache-busted-assets-without-a-bundler-shaped-monster"
 aria-label="Copy link to section: Hugo pipes: cache-busted assets without a bundler-shaped monster"
 data-heading-permalink
 >#</a>
</h2>
The theme uses Hugo&rsquo;s asset pipeline for the final CSS and JS includes.
In development, CSS is linked directly:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go-html-template" data-lang="go-html-template">1{{- with resources.Get &#34;css/main.css&#34; }}
2 {{- if hugo.IsDevelopment }}
3 &lt;link rel=&#34;stylesheet&#34; href=&#34;{{ .RelPermalink }}&#34;&gt;
4 {{- else }}
5 {{- with . | fingerprint }}
6 &lt;link rel=&#34;stylesheet&#34; href=&#34;{{ .RelPermalink }}&#34; integrity=&#34;{{ .Data.Integrity }}&#34; crossorigin=&#34;anonymous&#34;&gt;
7 {{- end }}
8 {{- end }}
9{{- end }}
</code></pre></div>In production, Hugo fingerprints it and adds an integrity attribute. The JavaScript gets the same treatment after passing through <code>js.Build</code> with minification enabled outside development.
The browser JS is intentionally tiny. It does two things:
<ol>
<li>Adds the <code>dark</code> class to the document element.</li>
<li>Adds copy buttons to Chroma code blocks, stripping line number spans before writing text to the clipboard.</li>
</ol>
There is also a small inline footer clock and a privacy-aware loader for the site&rsquo;s self-hosted Umami analytics. The loader respects Do Not Track, Global Privacy Control and the local opt-out, while removing query strings and fragments before a page view leaves the browser. This means the live browser behavior is not zero JavaScript, but it is measured in teaspoons. There is no client router, hydration pass, state manager, session recorder, or marketing attribution ritual.
<h2 id="syntax-highlighting-chroma-but-dressed-for-the-room" class="heading-with-permalink">Syntax highlighting: Chroma, but dressed for the room<a
 class="heading-permalink"
 href="#syntax-highlighting-chroma-but-dressed-for-the-room"
 aria-label="Copy link to section: Syntax highlighting: Chroma, but dressed for the room"
 data-heading-permalink
 >#</a>
</h2>
Hugo uses Chroma for syntax highlighting. The config keeps classes enabled:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml">1[markup.highlight]
2 noClasses = false
3 style = &#39;monokai&#39;
4 lineNos = true
5 lineNumbersInTable = false
6 tabWidth = 2
</code></pre></div>The <code>style = 'monokai'</code> line mostly stops being the story once <code>noClasses = false</code> is set. Hugo emits semantic token classes, and the theme CSS decides what those classes mean.
Keywords become mint. Types and tag names become pink. Functions become gold. Strings become lilac. Comments get shoved into the quiet corner at muted 60 percent opacity, as comments deserve. Line numbers are non-interactive, unselectable spans, which matters because copy buttons clone the code block and remove <code>.lnt</code> and <code>.ln</code> before copying.
The result is syntax highlighting that belongs to the same visual world as the rest of the site. It does not look like somebody taped a Dracula code block onto a different website and fled the scene.
<h2 id="woodpecker-from-source-tree-to-artifact" class="heading-with-permalink">Woodpecker: from source tree to artifact<a
 class="heading-permalink"
 href="#woodpecker-from-source-tree-to-artifact"
 aria-label="Copy link to section: Woodpecker: from source tree to artifact"
 data-heading-permalink
 >#</a>
</h2>
The CI pipeline is <code>.woodpecker/publish.yml</code>. It runs on pushes and manual events.
The first step uses <code>node:22-bookworm-slim</code>, because the build needs npm for Tailwind and Hugo for the static site. Hugo is installed explicitly from GitHub releases:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh">1hugo_version=&#39;0.162.1&#39;
2case &#34;$(uname -m)&#34; in
3 x86_64) hugo_arch=&#39;linux-amd64&#39; ;;
4 aarch64|arm64) hugo_arch=&#39;linux-arm64&#39; ;;
5 *) echo &#34;Unsupported CI architecture: $(uname -m)&#34; &gt;&amp;2; exit 1 ;;
6esac
7curl -fsSL &#34;https://github.com/gohugoio/hugo/releases/download/v${hugo_version}/hugo_extended_${hugo_version}_${hugo_arch}.tar.gz&#34; |
8 tar -xz -C /usr/local/bin hugo
</code></pre></div>Then:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh">1npm ci
2npm run build
3test -f public/index.html
4date -u +%Y%m%d%H%M%S &gt; .release_tag
5printf &#39;latest,%s\n&#39; &#34;$(cat .release_tag)&#34; &gt; .tags
</code></pre></div>That <code>.tags</code> file is for the Docker plugin. Every build gets <code>latest</code> for convenience and a UTC timestamp tag for deployment. The timestamp is the important one. <code>latest</code> is a bookmark. The timestamp is a version.
<h2 id="the-container-static-web-server-inside-scratch" class="heading-with-permalink">The container: static-web-server inside scratch<a
 class="heading-permalink"
 href="#the-container-static-web-server-inside-scratch"
 aria-label="Copy link to section: The container: static-web-server inside scratch"
 data-heading-permalink
 >#</a>
</h2>
The image build is wonderfully rude:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-dockerfile" data-lang="dockerfile"> 1FROM ghcr.io/static-web-server/static-web-server:latest AS sws-binary
 2
 3FROM scratch
 4
 5COPY --from=sws-binary /static-web-server /sws
 6COPY ./public /public
 7
 8ENV SERVER_PORT=8080
 9ENV SERVER_ROOT=/public
10
11EXPOSE 8080
12
13ENTRYPOINT [&#34;/sws&#34;]
</code></pre></div>It copies the <code>static-web-server</code> binary out of the upstream image, copies Hugo&rsquo;s <code>public/</code> directory, and then runs from <code>scratch</code>.
There is no shell. No package manager. No CA bundle unless the binary and app need one, which this site does not for serving local files. No busybox escape hatch. If someone gets arbitrary command execution inside this container, their prize is a filesystem containing a web server binary and some HTML. It is the infrastructure equivalent of opening a safe and finding a polite note that says &ldquo;no.&rdquo;
Static Web Server listens on <code>8080</code> and serves <code>/public</code>. The site is pure static output, so it does not need an application runtime. The container is not where logic lives. The container is where files wait to be sent.
The <code>publish-image</code> Woodpecker step uses <code>woodpeckerci/plugin-docker-buildx</code>, pushes to <code>registry.augustini.xyz/sites/augustiniwtf</code>, and targets <code>linux/amd64</code>.
<h2 id="the-handoff-woodpecker-pokes-terrakube-not-kubernetes" class="heading-with-permalink">The handoff: Woodpecker pokes Terrakube, not Kubernetes<a
 class="heading-permalink"
 href="#the-handoff-woodpecker-pokes-terrakube-not-kubernetes"
 aria-label="Copy link to section: The handoff: Woodpecker pokes Terrakube, not Kubernetes"
 data-heading-permalink
 >#</a>
</h2>
The deploy step is where the two repos shake hands.
Woodpecker does not run <code>kubectl apply</code>. It does not carry a kubeconfig. It does not patch a Deployment directly. Instead, it calls the Terrakube API.
The script:
<ol>
<li>Reads <code>.release_tag</code>.</li>
<li>Finds organization <code>v0cloud</code>.</li>
<li>Finds workspace <code>augustini-wtf</code>.</li>
<li>Finds Terraform variable <code>image_tag</code>.</li>
<li>Patches that variable to the new timestamp.</li>
<li>Finds the template named <code>Plan and apply</code>.</li>
<li>Creates a Terrakube job for that workspace/template pair.</li>
</ol>
This is the most important boundary in the deployment. Woodpecker is allowed to produce an artifact and request a deployment. Terrakube is the thing allowed to evaluate and apply infrastructure.
That gives a cleaner audit trail: source build in Woodpecker, stateful infrastructure action in Terrakube, actual reconciliation in Kubernetes.
After deployment, Woodpecker also prunes Harbor artifacts. It keeps five non-<code>latest</code> artifacts with timestamp tags matching <code>^[0-9]{14}$</code>, deletes older artifacts by digest, and sweeps any already-untagged artifacts left behind by previous cleanup runs. This is housekeeping, not heroism, but housekeeping is how registries avoid becoming museums of forgotten builds.
<h2 id="opentofu-the-other-repo" class="heading-with-permalink">OpenTofu: the other repo<a
 class="heading-permalink"
 href="#opentofu-the-other-repo"
 aria-label="Copy link to section: OpenTofu: the other repo"
 data-heading-permalink
 >#</a>
</h2>
The OpenTofu workspace in <code>/Users/ellie/talos/opentofu-apps/augustini-wtf</code> is small, which is exactly what you want for an app layer.
It uses OpenTofu <code>&gt;= 1.12.0, &lt; 1.13.0</code> and the HashiCorp Kubernetes provider <code>~&gt; 3.1.0</code>. The lock file pins provider checksums. The provider config supports both local validation and Terrakube execution:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-hcl" data-lang="hcl">1provider &#34;kubernetes&#34; {
2 config_path = var.kubeconfig_path != &#34;&#34; ? var.kubeconfig_path : null
3}
</code></pre></div>Locally, set <code>kubeconfig_path</code>. In Terrakube, leave it empty and let the runner pod authenticate in-cluster through its ServiceAccount.
State is not local, and Terrakube is not pretending to be the state backend. The backend is S3-compatible storage served by Rook Ceph RGW inside the cluster:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-hcl" data-lang="hcl"> 1terraform {
 2 backend &#34;s3&#34; {
 3 bucket = &#34;tofu-state&#34;
 4 key = &#34;augustini-wtf/terraform.tfstate&#34;
 5 region = &#34;us-east-1&#34;
 6
 7 endpoints = {
 8 s3 = &#34;http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc:80&#34;
 9 }
10
11 use_path_style = true
12 skip_credentials_validation = true
13 skip_region_validation = true
14 skip_metadata_api_check = true
15 skip_requesting_account_id = true
16 }
17}
</code></pre></div>That tells you quite a lot about the platform around the app: this is a Kubernetes-native setup, running on Talos, with Terrakube runners inside the cluster and Rook Ceph providing object storage for state. The app workspace does not define Talos itself, Rook itself, Terrakube itself, or the public Gateway. It attaches to those platform primitives.
That separation is healthy. App repo says &ldquo;run this site.&rdquo; Platform repos say &ldquo;here is the cluster, object storage, runner identity, gateway, DNS automation, and so on.&rdquo;
<h2 id="variables-small-escape-hatches-tight-defaults" class="heading-with-permalink">Variables: small escape hatches, tight defaults<a
 class="heading-permalink"
 href="#variables-small-escape-hatches-tight-defaults"
 aria-label="Copy link to section: Variables: small escape hatches, tight defaults"
 data-heading-permalink
 >#</a>
</h2>
The OpenTofu variables are boring in the pleasing sense.
The resource base name and namespace default to <code>augustini-wtf</code>. The image repository defaults to <code>registry.augustini.xyz/sites/augustiniwtf</code>. The image tag has a validation rule:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-hcl" data-lang="hcl"> 1variable &#34;image_tag&#34; {
 2 description = &#34;Pinned container image tag.&#34;
 3 type = string
 4 default = &#34;20260607072706&#34;
 5
 6 validation {
 7 condition = var.image_tag != &#34;latest&#34;
 8 error_message = &#34;image_tag must be pinned to a concrete tag instead of latest.&#34;
 9 }
10}
</code></pre></div>That single validation block prevents the classic personal-site incident where <code>latest</code> works until the registry, node cache, rollout controller, and your assumptions all form a small committee and vote against you.
The Harbor credentials are sensitive variables. The registry host defaults to <code>registry.augustini.xyz</code>. Replica count defaults to <code>2</code>, with validation requiring at least one. CPU and memory defaults are intentionally tiny:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-hcl" data-lang="hcl">1requests = {
2 cpu = &#34;10m&#34;
3 memory = &#34;32Mi&#34;
4}
5limits = {
6 cpu = &#34;250m&#34;
7 memory = &#34;128Mi&#34;
8}
</code></pre></div>This is a static web server. If it needs more than that under normal personal-site traffic, something is either famous or on fire.
<h2 id="kubernetes-the-runtime-contract" class="heading-with-permalink">Kubernetes: the runtime contract<a
 class="heading-permalink"
 href="#kubernetes-the-runtime-contract"
 aria-label="Copy link to section: Kubernetes: the runtime contract"
 data-heading-permalink
 >#</a>
</h2>
The OpenTofu module creates a namespace with common labels:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-hcl" data-lang="hcl">1&#34;app.kubernetes.io/name&#34; = var.name
2&#34;app.kubernetes.io/part-of&#34; = &#34;talos&#34;
3&#34;app.kubernetes.io/managed-by&#34; = &#34;terrakube&#34;
</code></pre></div>Then it merges in Pod Security Admission labels:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-hcl" data-lang="hcl">1&#34;pod-security.kubernetes.io/enforce&#34; = &#34;restricted&#34;
2&#34;pod-security.kubernetes.io/audit&#34; = &#34;restricted&#34;
3&#34;pod-security.kubernetes.io/warn&#34; = &#34;restricted&#34;
</code></pre></div>That means the namespace is not merely &ldquo;please be secure&rdquo; flavored. It asks Kubernetes to enforce the restricted profile. The pod spec then matches that intent:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-hcl" data-lang="hcl"> 1automount_service_account_token = false
 2
 3security_context {
 4 run_as_non_root = true
 5 run_as_user = 10001
 6 run_as_group = 10001
 7 fs_group = 10001
 8
 9 seccomp_profile {
10 type = &#34;RuntimeDefault&#34;
11 }
12}
</code></pre></div>The container security context tightens the rest:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-hcl" data-lang="hcl">1allow_privilege_escalation = false
2read_only_root_filesystem = true
3run_as_non_root = true
4
5capabilities {
6 drop = [&#34;ALL&#34;]
7}
</code></pre></div>This lines up nicely with the scratch image. The image has nothing interesting to mutate, and Kubernetes makes the root filesystem read-only anyway. The pod gets no service account token by default. Linux capabilities are gone. Privilege escalation is off. Seccomp uses the runtime default.
It is a very small box with a very small job.
The Deployment exposes one named container port:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-hcl" data-lang="hcl">1port {
2 name = &#34;http&#34;
3 container_port = 8080
4 protocol = &#34;TCP&#34;
5}
</code></pre></div>Readiness and liveness probes both hit <code>/</code> through that named port. The Service maps port <code>80</code> to target port <code>http</code>, giving the Gateway a stable backend target without making the container pretend it is allowed to bind privileged ports.
<h2 id="registry-auth-dockerconfigjson-as-infrastructure" class="heading-with-permalink">Registry auth: dockerconfigjson as infrastructure<a
 class="heading-permalink"
 href="#registry-auth-dockerconfigjson-as-infrastructure"
 aria-label="Copy link to section: Registry auth: dockerconfigjson as infrastructure"
 data-heading-permalink
 >#</a>
</h2>
Because the image lives in private Harbor, the module creates a <code>kubernetes.io/dockerconfigjson</code> secret:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-hcl" data-lang="hcl"> 1data = {
 2 &#34;.dockerconfigjson&#34; = jsonencode({
 3 auths = {
 4 (var.harbor_url) = {
 5 username = var.harbor_username
 6 password = var.harbor_password
 7 auth = base64encode(&#34;${var.harbor_username}:${var.harbor_password}&#34;)
 8 }
 9 }
10 })
11}
</code></pre></div>Then the Deployment references it via <code>image_pull_secrets</code>.
This is a good example of infrastructure code being allowed to be unglamorous. The cluster needs credentials to pull the image. Terrakube owns the sensitive inputs. OpenTofu renders the exact Kubernetes secret. The pod uses it. Nobody needs to click around in a registry UI and hope the namespace has the right secret forever.
<h2 id="gateway-api-routing-as-an-app-owned-resource" class="heading-with-permalink">Gateway API: routing as an app-owned resource<a
 class="heading-permalink"
 href="#gateway-api-routing-as-an-app-owned-resource"
 aria-label="Copy link to section: Gateway API: routing as an app-owned resource"
 data-heading-permalink
 >#</a>
</h2>
The app does not define a load balancer. It attaches routes to an existing Gateway named <code>main-gateway</code> in the <code>default</code> namespace.
For the apex hostname, the route attaches to section <code>https-wtf-apex</code> and sends <code>/</code> to the Service:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-hcl" data-lang="hcl"> 1hostnames = [var.hostname]
 2
 3parentRefs = [
 4 {
 5 group = &#34;gateway.networking.k8s.io&#34;
 6 kind = &#34;Gateway&#34;
 7 name = var.gateway_name
 8 namespace = var.gateway_namespace
 9 sectionName = var.gateway_https_section
10 }
11]
</code></pre></div>The route also sets HSTS:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-hcl" data-lang="hcl"> 1filters = [
 2 {
 3 type = &#34;ResponseHeaderModifier&#34;
 4 responseHeaderModifier = {
 5 set = [
 6 {
 7 name = &#34;Strict-Transport-Security&#34;
 8 value = &#34;max-age=63072000; includeSubDomains; preload&#34;
 9 }
10 ]
11 }
12 }
13]
</code></pre></div>There is a separate <code>www</code> route attached to <code>https-wtf</code> that returns a <code>301</code> redirect to the apex host. There is also an optional HTTP route, enabled by default, that attaches to the Gateway&rsquo;s <code>http</code> section and redirects both <code>augustini.wtf</code> and <code>www.augustini.wtf</code> to HTTPS on the apex hostname.
This is where Gateway API feels cleaner than the old Ingress pile. The platform owns the Gateway and listeners. The app owns its hostnames, redirects, headers, and backend references. The contract between them is explicit: route attaches to named listener section, listener accepts it, traffic moves.
ExternalDNS is controlled through route annotations. If publishing is enabled, the route gets:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-hcl" data-lang="hcl">1&#34;external-dns.alpha.kubernetes.io/include&#34; = &#34;true&#34;
</code></pre></div>If disabled, it gets a controller annotation pointing at an ignore value. That gives the app workspace a switch for &ldquo;publish records from these routes&rdquo; without making DNS a manual side quest.
<h2 id="the-talos-part-and-what-this-repo-does-not-own" class="heading-with-permalink">The Talos part, and what this repo does not own<a
 class="heading-permalink"
 href="#the-talos-part-and-what-this-repo-does-not-own"
 aria-label="Copy link to section: The Talos part, and what this repo does not own"
 data-heading-permalink
 >#</a>
</h2>
The OpenTofu labels say <code>part-of = talos</code>, and the path says the same thing: this app belongs to a Talos-based cluster estate. Talos matters here less because of any one line in the app module and more because of the operational model it implies.
Talos is not a general-purpose Linux host that also happens to run Kubernetes. It is an immutable Kubernetes appliance OS. No SSH as the normal management interface. No &ldquo;just apt install htop on the node.&rdquo; No artisanal node drift. Cluster operations go through APIs and declared configuration.
That philosophy matches the site deployment:
<ul>
<li>The app container is scratch.</li>
<li>The pod has a read-only root filesystem.</li>
<li>The namespace enforces restricted Pod Security.</li>
<li>The deployment is driven by OpenTofu from Terrakube.</li>
<li>State lives in object storage, not on a laptop.</li>
<li>Routes attach to a pre-existing platform Gateway.</li>
</ul>
The website is tiny, but the operational shape is the same one you want for larger services: artifacts are built once, deployed by tag, reconciled by controllers, and exposed through platform-owned ingress.
<h2 id="what-actually-changes-when-i-publish-a-post" class="heading-with-permalink">What actually changes when I publish a post<a
 class="heading-permalink"
 href="#what-actually-changes-when-i-publish-a-post"
 aria-label="Copy link to section: What actually changes when I publish a post"
 data-heading-permalink
 >#</a>
</h2>
When this post gets merged and pushed, the pipeline does not mutate the cluster directly. It creates a new immutable-ish artifact and moves one pointer.
The only deployment variable Woodpecker changes is <code>image_tag</code>. That variable goes from one timestamp to another. Terrakube runs the plan and apply. Kubernetes sees the Deployment pod template image change from:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text">1registry.augustini.xyz/sites/augustiniwtf:20260607072706
</code></pre></div>to something like:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text">1registry.augustini.xyz/sites/augustiniwtf:20260607123456
</code></pre></div>That pod template change triggers a rollout. New pods pull the new image using the Harbor secret. Readiness probes pass. The Service selector keeps pointing at the same labels. The Gateway route keeps pointing at the same Service. Traffic gradually finds the new pods.
There is no &ldquo;deploy content&rdquo; step. The content is inside the image. That makes rollback boring too: set <code>image_tag</code> back to an older timestamp, apply, and let Kubernetes roll back to pods serving that older <code>public/</code> tree.
<h2 id="why-this-is-funny-and-why-it-is-not" class="heading-with-permalink">Why this is funny and why it is not<a
 class="heading-permalink"
 href="#why-this-is-funny-and-why-it-is-not"
 aria-label="Copy link to section: Why this is funny and why it is not"
 data-heading-permalink
 >#</a>
</h2>
Yes, this is a lot of machinery for personal fieldnotes with a heart icon in the header.
But the machinery is not there because the website needs compute. It is there because I want the deployment path to teach the right habits:
<ul>
<li>Build static output deterministically.</li>
<li>Package the runtime as a small container.</li>
<li>Push to a registry.</li>
<li>Deploy pinned tags.</li>
<li>Keep infrastructure state remote.</li>
<li>Let CI request deployment, not impersonate the cluster admin.</li>
<li>Run with fewer privileges than the app could ever need.</li>
<li>Put routing, redirects, and headers in declarative resources.</li>
</ul>
The actual app is almost aggressively simple: HTML, CSS, a tiny amount of JavaScript, and a Rust static file server. The platform around it is the part that makes the simplicity durable.
That is the nice thing about static sites. You can make the runtime so small that all the interesting engineering moves to the edges: how assets are built, how images are tagged, who is allowed to apply infrastructure, where state lives, how traffic enters, and what the pod is forbidden from doing.
The result is a terminal princess website that deploys like a grown-up service and runs like a locked filing cabinet with a web server taped to it. It is pastel. It is strict. It contains almost no moving parts. It sparks joy, and then immediately drops all Linux capabilities.