#10: My Git forge issued 2.1 million challenges. Only 20 were solved.

Self-hosting used to mean worrying about disks, backups and whether the tiny computer under the stairs had developed opinions about thermodynamics.

Now it also means watching automated clients come through apparent residential addresses and request every source tree, blame page, commit, comparison and archive on your Git forge as quickly as the application can regret serving them.

On 28 August, that happened to my Forgejo instance. It became nearly unusable, particularly when I was logged in. The traffic was not merely visiting public repositories. It was repeatedly selecting the Git-backed views that made Forgejo do the most work, and it came close enough to flattening the service that I was watching my Talos cluster get mugged in real time.

To quote Ylvis, concisely: “On a scale from 1 to 10 my friend: you’re fucked!”

I run the cluster on bare metal, so the immediate cost was my own capacity and sleep. On usage-priced infrastructure, somebody else’s abusive traffic can become your cloud bill before you have finished working out which fashionable new company is ignoring robots.txt this week.

I cannot attribute this incident to a specific company. The request pattern was consistent with aggressive indexing or dataset collection, but headers are self-reported and motives do not appear in Prometheus. My broader, firmly held opinion is that some of the worst crawlers I encounter belong to small AI companies with too much venture capital and no apparent adult supervision. That is an opinion earned elsewhere. The evidence here establishes automation, distributed source addresses, expensive route selection and almost no completed proofs. It does not establish the operator.

This distinction matters because the larger problem is real even when the culprit remains anonymous. Independent operators are expected to absorb the infrastructure cost of somebody else’s collection programme. Some cannot. They stop publishing services, remove public access or move everything behind a giant intermediary.

I could have put the forge behind Cloudflare. It would have been quick and effective. It would also have routed every visitor through a large American company and solved a problem facing the independent web by making my part of it less independent. I happen to care about privacy and centralisation even when those principles create paperwork.

So I deployed Anubis, a proof-of-work gate I could run, inspect and swear at locally.

The bandwidth was innocent#

This looked like a denial-of-service attack, but raw bandwidth was not the interesting resource. Application work was.

A static asset is cheap. A repository tree, blame page, commit graph, comparison, search result, raw file or newly generated archive is not. Forgejo may need to walk Git objects, inspect history, render a template or construct a bundle. One tiny HTTP request can purchase a surprising amount of server-side labour, billed to the server.

Two early policy snapshots made the pattern visible. I classified /src, /blame, /commits, /commit, /raw, /compare, /graph, /find and /archive as expensive repository views because each invokes Git-backed work or archive generation. Those routes accounted for 662 of 694 challenges in one snapshot and 1,837 of 1,915 in the next: roughly 95–96%.

My large public QMK firmware fork from the pink keyboard incident was especially attractive. It contains plenty of files, history and commits, and one archive cache miss for that repository produced a bundle of roughly 294 MiB. Asking for an archive is a small request in the same sense that pulling a fire alarm is a small hand movement.

The requests rotated through many addresses that appeared to belong to residential allocations. I did not prove that they came from one residential-proxy provider, or even one coordinated swarm, so I am not going to turn an address pattern into a detective novel. It was enough to make per-IP rate limiting the wrong primary defence. A distributed client can change addresses faster than I can add increasingly theatrical CIDR blocks, while legitimate users behind shared NAT receive the shrapnel.

I did not need to know who operated it. Incident response does not require identifying the clown if you can stop the tiny car at the gate.

Nuclear bouncer, bedtime, architecture tomorrow#

It was late. Forgejo was being murdered by HTTP requests. I wanted to sleep.

My first emergency policy therefore issued everybody a level-64 proof-of-work challenge. This was not nuanced traffic engineering. It was the infrastructure equivalent of lowering a portcullis onto the entire road, confirming the screaming had stopped and going to bed.

Emergency containment and durable policy are different jobs. The level-64 rule bought me sleep. The configuration committed at deeafe3 late on 28 August began making the rules sane.

Anubis reverses the cost asymmetry. Before an unknown browser reaches Forgejo, it must complete client-side work. A human pays occasionally and receives an authorisation cookie. A crawler requesting millions of pages must keep paying.

1before: cheap request -> expensive Forgejo work -> useful HTML
2after:  cheap request -> client-side work -> maybe reach Forgejo

The final policy on Anubis 1.27.0 uses its fast SHA-256 challenge at difficulty 4 for an ordinary unknown browser. Self-identified bots, crawlers and headless automation receive my maximum configured fast challenge difficulty of 16, which Anubis treats as effectively impossible. The suspicion thresholds use metarefresh at 1 and fast proofs at 2, 4 or 6, although the explicit catch-all keeps unrecognised Forgejo web traffic at difficulty 4. A successful proof produces a SameSite=Lax authorisation cookie valid for 24 hours and bound to the rule that issued it.

It is not currently bound to the client address. Anubis 1.27.0 can restrict its signed JWT to a request header, but this deployment does not set JWT_RESTRICTION_HEADER. Someone who obtained a valid cookie could therefore replay it from another address until it expires, provided the request still matches the same policy rule. That is a limitation, and a considerably more useful sentence than pretending the configuration comment was a security boundary.

I have not benchmarked solve time across phones and low-power laptops, so I will not invent a reassuring median. Anubis defines difficulty as the number of leading zeroes required in the result hash; its human cost depends on the device and browser. That cost is not imaginary. Visitors need JavaScript and cookies. Restrictive browser settings, text browsers and constrained hardware may fail or pay disproportionately. Search engines and archival crawlers may stop indexing the forge unless I give them a narrow verified exception.

The challenge state and authorisation cookie stay under my control, but that does not make the protection layer invisible. Anubis still observes client addresses and request metadata because it needs them to decide what to do. The privacy advantage over Cloudflare is not that no component sees the traffic. It is that I operate the component that does.

Proof of work moves cost to the client device, which is not always the same machine or person controlling abusive traffic. Anubis calls itself a nuclear response for a reason. Mine was deployed because the alternative was letting anonymous HTML requests continue to consume the forge.

A forge is not just a website#

Putting a browser challenge in front of a blog is comparatively simple. A forge has rather more client species.

Git smart HTTP, Git LFS, API clients, OAuth, webhooks, package managers, feeds, ActivityPub and CI workers cannot execute JavaScript. Blocking them would make the forge impressively safe from both bots and usefulness.

The durable policy added narrow machine exceptions before one browser catch-all. Git discovery and RPC rules match exact paths plus the expected method, service query, media types and Git client shape. Git LFS has its own scoped paths. The OAuth exception permits only a form-encoded POST to /login/oauth/access_token. Immutable /assets/ files are globally allowed because they are cheap and never invoke Git; without them, a human who had passed the challenge received HTML where Forgejo’s module loader expected JavaScript.

Requests under /api/ with a non-empty Authorization header also pass Anubis, but calling them “authenticated requests” would be a lie told one proxy too early. Anubis recognises an authentication attempt. Forgejo validates the credential and rejects a fake token cheaply. The exception still exposes Forgejo’s authentication path to traffic and deserves its own rate alert; it is an accepted cost, not a magical authenticated force field. The unauthenticated OAuth exchange and version probe are similarly narrow, deliberately cheap openings.

Git smart HTTP is the largest exception and the most important trade-off. Anyone can still clone a public repository. That is intentional. Git’s native transfer protocol is efficient and bounded compared with repeatedly rendering source trees, blame views and archives as HTML. I am protecting the application from work amplification, not pretending published source code is secret.

The opposite rule—trust anything with git in its user agent—would be security by string appreciation. Headers are client-supplied text. A fake Git user agent still cannot reopen arbitrary web pages. It can clone what I deliberately publish.

There is one final browser challenge rule rather than separate rules for archives, source trees and generic pages. Anubis binds authorisation to the rule that issued it. My first fragmented policy forced human visitors through multiple proofs and could return challenge HTML to background JSON requests. Commit e234c9e replaced those overlapping rules with one authorisation context after the machine exceptions.

Do not attack yourself on behalf of the bot#

Anubis can fetch upstream Open Graph metadata so a challenged page still produces a handsome social preview. That becomes a problem when the protected URL is an expensive Forgejo page.

If Anubis fetches that page for metadata, the scraper no longer needs to reach Forgejo. The protection layer does it for them. I had briefly built a distributed denial-of-service proxy against myself, but at least the unfurled link would have looked lovely.

I disabled Open Graph passthrough. A request that has not passed policy must not cause Forgejo to render the target route. Rejected work remains rejected all the way to the expensive application.

I also set request logging to warnings. The Anubis processes expose cumulative counters; Prometheus samples those counters every 30 seconds and derives rates or increases from the stored series. Recording millions of routine challenges as individual log events would merely translate a Forgejo incident into an Alloy and Loki incident.

Then I broke Woodpecker#

The morning after the nuclear bedtime rule, Anubis revealed every machine path I had forgotten.

Static assets received challenge pages. The bundled Git rule did not cover all smart-HTTP RPC shapes. Woodpecker’s OAuth token exchange and periodic Forgejo version probe could not perform browser proofs. Testing only the homepage remains a sophisticated method for proving that the homepage works.

My first attempt to rescue Woodpecker, in commit fbc8efb, set WOODPECKER_FORGEJO_URL to Forgejo’s internal Kubernetes address and used WOODPECKER_EXPERT_FORGE_OAUTH_HOST for the public OAuth host. This seemed tidy: server-to-server traffic stays inside the cluster, browser traffic stays public.

It was the wrong boundary.

The forge identity appeared in browser-visible Woodpecker state. Browsers, displaying a disappointing lack of Kubernetes integration, cannot resolve *.svc.cluster.local. The internal URL leaked into the login flow, OAuth failed, and the workaround also walked around the single enforcement point I had just built.

I reverted it. Woodpecker now uses the public protected hostname for version checks, OAuth exchange, API traffic and Git operations, with exact protocol exceptions. One canonical URL is boring. Boring is what you want from OAuth.

That exposed a failure below the policy rules. Cilium’s Gateway source-IP handling has Envoy set X-Envoy-External-Address from the trusted client address for external requests. With the Gateway’s default zero trusted X-Forwarded-For hops, a public client cannot make its own forwarded value authoritative; Envoy uses the connection source and sanitises its internal headers. Hairpin traffic from inside the cluster did not always receive X-Envoy-External-Address, however. Anubis 1.27.0 could discard private or link-local forwarded addresses, derive an empty X-Real-IP and reject Woodpecker before its allow rule ran.

Commit 5fdd8c5 added a tiny nginx sidecar immediately before each Anubis container. It uses the Envoy-derived address when present and the connection peer for in-cluster traffic, then overwrites X-Real-IP before proxying locally. The client-supplied X-Original-URI header has no routing authority in this chain; because an older Anubis bug used it as a policy bypass, I still send any request carrying it to the difficulty-16 rule. If a future proxy ever gives that header routing meaning, it must be stripped at ingress rather than trusted after a solved proof.

Naturally, the first sidecar rollout entered CrashLoopBackOff.

The container had a read-only root filesystem, as it should. I redirected nginx’s client-body and proxy temporary paths to /tmp. nginx also initialises FastCGI, uWSGI and SCGI temporary directories despite this proxy using none of them, because nginx arrived prepared for three other meetings. Those paths still pointed into /var/cache/nginx, so the process exited.

Commit 254c000, six minutes after the sidecar commit, moved all five temporary paths to the writable volume. Woodpecker login and pipelines then completed through the public route, exercising OAuth, API calls, cloning and status updates end to end.

The correct smoke test was not GET /.

1login -> assets -> background API -> OAuth -> clone -> build -> status update

Each arrow is a place where two healthy components can combine into something completely fucked.

The final path, with the dents left visible#

After those failures, the public HTTP path looked like this:

 1browser or automated client
 2            |
 3            v
 4      Cilium Gateway
 5            |
 6            v
 7   real-IP normaliser -> Anubis replica 1 or 2
 8                              |       |
 9                              +-- Valkey
10                              |
11                              v
12                           Forgejo HTTP
13
14Git over SSH ----------------> Forgejo SSH
15Woodpecker CI ---------------> public HTTP path above

The Cilium Gateway sends every Forgejo HTTP route to Anubis. A NetworkPolicy permits Forgejo’s HTTP port only from pods carrying the Anubis labels, so another workload cannot discover the old Service and wander around the bouncer. SSH remains direct because key authentication is its boundary and asking an SSH client to run JavaScript would be innovative only in the LinkedIn sense.

Two Anubis replicas use a shared persistent Ed25519 signing key so either can verify an authorisation cookie issued by the other. They also use a dedicated Valkey instance for temporary challenge state. Preferred pod anti-affinity asks the scheduler to place the replicas on different nodes; at the incident snapshot, it had done so. A PodDisruptionBudget limits voluntary disruptions to one at a time.

None of that guarantees high availability. Preferred affinity is a request, not a blood oath. A PodDisruptionBudget does nothing against involuntary node failure or a broken rollout. The single Valkey replica is still a state-layer failure point, although losing its ephemeral challenge state is less exciting than losing Git data. This design removes the Anubis pod as an obvious single point of failure. It does not repeal distributed-systems law.

What 2.1 million challenges proved#

The headline numbers have one specific scope. They are the live counter values summed across the two Anubis replicas running after the final rollout on 29 August:

Anubis names the second metric anubis_challenges_validated; I use “solved” colloquially in the title. It counts accepted proof validations, while neither headline number measures unique humans, browsers or bots.

A manually captured earlier generation recorded 56,926 challenges, 121 upstream requests and two validations. Combining non-overlapping snapshots gives an observed incident lower bound above 2.17 million issuances and 22 validations. I did not reconstruct a complete total from Prometheus: early deployment coverage was incomplete, and the headline uses current live counters plus the retained snapshot. Counter resets themselves are not the problem—rate() and increase() handle them when the sampled series survives. Missing samples do not become history because the query is optimistic.

The 18,095 proxied requests included 7,583 /api/ requests carrying an Authorization header, 7,578 version probes, 1,520 Git discovery requests, 764 Git RPC requests and 126 OAuth exchanges. The remaining 524 included 376 well-known metadata requests, 78 static-asset requests, several smaller explicit categories and authorised browser traffic. The categories show compatibility, not purity: some are intentionally cheap unauthenticated openings, and the API count proves only that a header was present.

The rate history came from Prometheus counter increases divided by each stated window in Europe/Stockholm time. The partial 29 August window begins with the corrected rollout at 16:33 CEST; the partial 3 September window ends at the 21:07 CEST report snapshot.

The rolling 24-hour window ending 3 September at 21:07 CEST contained approximately 3,342 challenge issuances. That is not the same window as the 2 September calendar day: the rolling window includes only its final 2 hours and 53 minutes, followed by most of the much quieter 3 September. The arithmetic was not haunted. The labels were previously inadequate.

The retained 15-minute peak occurred on 31 August at 19:00 CEST: 16.61 challenges per second, or about 997 per minute. Across the same days, the proxied-request rate remained near 0.041 per second while challenge traffic moved by orders of magnitude. That separation—not the pleasingly large counter—is the result. Hostile ingress stopped becoming Forgejo work.

At 21:07 CEST on 3 September, an instantaneous kubectl top snapshot showed Forgejo using 2 millicores and the two Anubis-plus-sidecar pods using 16–18 millicores each. Prometheus showed Forgejo averaging roughly 4 millicores since the final rollout, with a highest retained 15-minute average of 13.8 millicores. A search of the preceding 24 hours of Forgejo logs found no error-level entries. These are explicitly bounded observations, not a warranty.

The smallest useful Prometheus view includes both rates and totals over a declared period:

1sum(rate(anubis_challenges_issued[15m]))
2sum(rate(anubis_proxied_requests_total[15m]))
3sum(increase(anubis_challenges_issued[24h]))
4sum(increase(anubis_challenges_validated[24h]))

Put those beside Forgejo CPU and latency. If challenges surge while proxied traffic and application work stay flat, the bouncer is doing its job. If requests carrying fake credentials begin flooding the allowed API route, the bouncer’s guest list needs another revision. There is always another revision.

The post-mortem#

This incident did not end with a triumphant attribution or an IP blocklist. It ended with a boundary around expensive work.

Anonymous clients can still clone my public repositories. Humans with JavaScript disabled may not reach their web pages. A low-powered device pays more for the challenge than my desktop does. A copied authorisation cookie remains replayable until its 24 hours expire. The Valkey instance can fail. A new Forgejo endpoint or Woodpecker flow can invalidate today’s neat policy. Proof of work is an economic throttle, not absolution.

But abusive web traffic no longer gets to turn a tiny request into a large amount of anonymous Git-backed rendering. Normal browser sessions, Git, OAuth and CI work. The forge is responsive. I did not have to send every visitor through Cloudflare, and the infrastructure extracting value from independent sites no longer gets to make me pay quite so much for the privilege.

That is the actual argument for this design. Self-hosting cannot survive if every abusive crawler is allowed to outsource its compute bill to the smallest machine on the route. Defending the independent web sometimes means making the visiting machine perform its share of the work, then documenting all the protocols you broke while learning where to put the bouncer.

On the Ylvis scale, the situation is no longer fundamentally fucked.