IPv6 has been the future of the internet since 1998, which makes it one of our most successful long-running science-fiction franchises.

I like IPv6. My Talos cluster is dual-stack. The nodes understand it, Kubernetes allocates it, Cilium routes it, the Gateways advertise it, and DNS publishes it. There are enough colons in the network configuration to make a C++ programmer feel emotionally supported.

So when I deployed Vaultwarden, I gave its Kubernetes Service both address families. This felt correct. Modern, even. The configuration explicitly said RequireDualStack, which is the infrastructure equivalent of checking the box marked Yes, I understand the future.

Vaultwarden started. PostgreSQL started. The pod was ready. The Service existed. The HTTPRoute was accepted. The Gateway was programmed. TLS worked.

And requests occasionally died with:

1upstream connect error or disconnect/reset before headers

The cluster supported IPv6.

The Service supported IPv6.

The pod had IPv6.

The application was listening on 0.0.0.0.

Ah.

Vaultwarden sits behind a Cilium Gateway API Gateway in my Kubernetes cluster. Once DNS has found the front door, the HTTP traffic follows a fairly ordinary path:

1browser
2  -> Cilium Gateway / Envoy
3  -> Vaultwarden pod:8080

The HTTPRoute points to a ClusterIP Service, and Cilium uses the Service’s backend endpoints to configure where Envoy sends requests. The route and Service describe the journey; neither is another HTTP server along the way.

“Fairly ordinary” is infrastructure terminology for “the diagram is small because we have hidden most of the machinery.”

The original OpenTofu variables for the Vaultwarden Service looked like this:

 1variable "service_ip_family_policy" {
 2  description = "IP family policy for the Vaultwarden ClusterIP Service."
 3  type        = string
 4  default     = "RequireDualStack"
 5}
 6
 7variable "service_ip_families" {
 8  description = "IP families for the Vaultwarden ClusterIP Service."
 9  type        = list(string)
10  default     = ["IPv4", "IPv6"]
11}

The Service used those values directly:

 1resource "kubernetes_service_v1" "vaultwarden" {
 2  spec {
 3    selector = local.app_labels
 4
 5    ip_family_policy = var.service_ip_family_policy
 6    ip_families      = var.service_ip_families
 7
 8    port {
 9      name        = "http"
10      port        = 80
11      target_port = "http"
12      protocol    = "TCP"
13    }
14
15    type = "ClusterIP"
16  }
17}

This was not an accidental default inherited from a Helm chart written during a lunar eclipse. I had deliberately requested both families.

The cluster itself was built for them. Talos had IPv4 and IPv6 pod and Service networks:

1cluster:
2  network:
3    podSubnets:
4      - 10.244.0.0/16
5      - fd00:10:244::/48
6    serviceSubnets:
7      - 10.96.0.0/12
8      - fd00:10:96::/108

Cilium had IPv6 pod networking enabled, and its Gateway layer could accept traffic over either family. As far as Kubernetes was concerned, the application had been invited to the dual-stack party and issued two name badges.

Inside the pod, however, Vaultwarden was configured like this:

1ROCKET_ADDRESS=0.0.0.0
2ROCKET_PORT=8080

0.0.0.0 means “all IPv4 interfaces.” It does not mean “all interfaces across every address family currently known to networking standards committees.” The IPv6 wildcard address is ::.

The process had opened an IPv4 listening socket. Kubernetes had then advertised an IPv6 path to it with the confidence of a restaurant adding rooftop seating because the lift has an extra button.

The phrase “the application is dual-stack” is dangerously compact. It can describe several different layers, and they do not share a group chat.

The cluster and CNI can do their jobs perfectly, allocate both families and deliver an IPv6 packet all the way into the pod’s network namespace. The Service can declare both families, EndpointSlices can publish both pod addresses, and Envoy can learn those addresses and choose one for an upstream connection. All of that gets the packet to the right place.

Then something has to answer. The process inside that namespace still has to open a socket for the address family Envoy selected.

My configuration had arranged everything up to that last handoff and treated the actual listener as a lifestyle choice.

That was the assumption I had smuggled into the deployment: the infrastructure supported IPv6, so putting the application inside it felt like finishing the job. Kubernetes was not lying. The Service really was dual-stack. Cilium was not lying. It really could route IPv6. Envoy was not lying when it reported a connection failure. Vaultwarden was not lying either; I had explicitly told it to listen on IPv4.

Every component was correct in isolation. Together they formed a very standards-compliant outage.

A Service is not the application. It is not even a network socket. It is a stable virtual address plus a selection mechanism for reaching endpoints.

For a dual-stack Service, Kubernetes can assign two cluster IPs:

1spec:
2  clusterIP: 10.96.121.226
3  clusterIPs:
4    - 10.96.121.226
5    - fd00:10:96::79e2
6  ipFamilyPolicy: RequireDualStack
7  ipFamilies:
8    - IPv4
9    - IPv6

The exact addresses are not interesting. The important part is that the Service now claims both routes are intentional.

EndpointSlices carry the actual backend addresses. They are separated by address type, so a dual-stack workload may produce an IPv4 slice and an IPv6 slice:

1NAME                 ADDRESSTYPE   PORTS   ENDPOINTS
2vaultwarden-a1b2c    IPv4          8080    10.244.1.127
3vaultwarden-d3e4f    IPv6          8080    fd00:10:244:1::7f

That says the pod has addresses and the Service selector found it. It does not prove a userspace process is accepting TCP connections on both.

The containerPort declaration does not save us either:

1port {
2  name           = "http"
3  container_port = 8080
4}

Container ports are metadata. They help name ports and communicate intent. They do not reach into the process, call bind(2), and correct its networking decisions while murmuring reassuring things about platform engineering.

The only authoritative answer to “what is listening?” lives in the network namespace containing the process.

On a normal Linux system, that means looking at the sockets:

1ss -lntp

The relevant distinction looks roughly like this:

1LISTEN 0 128 0.0.0.0:8080 0.0.0.0:*

versus:

1LISTEN 0 128 [::]:8080 [::]:*

The first is IPv4. The second is an IPv6 wildcard socket. Whether that IPv6 socket also accepts IPv4-mapped connections depends on the application and the IPV6_V6ONLY socket option, because apparently even “listen everywhere” needed an asterisk and several paragraphs of kernel documentation.

The least surprising approach is to verify the exact behavior rather than assume :: is a magical universal solvent. If an application supports separate IPv4 and IPv6 listeners, configuring both explicitly is clearer still.

The most useful part of this incident was not that IPv6 failed. Networks fail all the time. It is how many healthy signals survived around the failure.

Vaultwarden had startup, readiness and liveness probes against /alive on the named HTTP port:

 1readiness_probe {
 2  http_get {
 3    path = "/alive"
 4    port = "http"
 5  }
 6
 7  period_seconds    = 10
 8  timeout_seconds   = 3
 9  failure_threshold = 6
10}

The probe reached the working listener and passed. A successful HTTP readiness probe proves that the kubelet can reach one pod address on one port using the path it selected. It is not an exhaustive certification of every family advertised elsewhere.

Readiness is a gate, not a doctoral thesis. I had asked whether the application answered at the address the kubelet used, and then mentally promoted the answer into a much broader claim about the deployment.

That readiness result also fed endpoint discovery. The selector matched the pod, so Kubernetes generated EndpointSlices for its addresses. The IPv4 and IPv6 endpoints inherited the same pod readiness; the IPv6 entry did not have to pass a separate IPv6 application check. One working listener had effectively supplied a reference for the address where nobody was answering.

The HTTPRoute’s accepted status was reassuring for a different reason. Gateway API distinguishes several status conditions: Accepted reports that the controller accepted the route for that Gateway; ResolvedRefs separately reports whether its references, including backends, are valid and permitted. These are control-plane checks. Neither tests whether the application will complete a TCP handshake at every published address.

You can submit a perfectly valid address for a restaurant that is currently on fire. The postal system has still done its job.

Cilium had also translated the Gateway and route into Envoy configuration. The Gateway’s Programmed condition reports that configuration has been sent to the data plane. It does not certify the application at the far end. Envoy cannot make Vaultwarden open an IPv6 socket through force of personality.

I had a collection of quite specific answers and was reading them as one large everything works. The missing test lived in the space between those answers.

And then there was the especially helpful part: IPv4 worked. Requests routed to the IPv4 backend completed normally. Requests routed through the non-listening IPv6 path failed before headers. The same application could look healthy or broken depending on which address Envoy selected.

Intermittent success is excellent fuel for an investigation with no useful boundaries. Perhaps TLS session resumption is broken. Perhaps the OIDC provider is timing out or PostgreSQL has run out of connections. Give it another ten minutes and DNS caching is haunted, Cilium has offended the BPF verifier, and Envoy has developed preferences.

The database, authentication flow and HTTP application were innocent. The TCP connection was dying before any of them got an opportunity to disappoint me.

One easy trap is assuming the client’s address family determines the backend’s address family.

It does not have to.

The browser may connect to the Gateway over IPv4. Envoy terminates that downstream connection, processes the HTTP request, selects an upstream endpoint, and opens a new connection to the pod. That upstream connection can use IPv6 if the proxy’s endpoint configuration contains an IPv6 address.

1browser --IPv4--> Envoy --IPv6--> pod

Or the reverse:

1browser --IPv6--> Envoy --IPv4--> pod

Proxies are connection laundromats. The protocol that enters the building does not need to be the protocol that leaves it.

This is why testing curl -4 and curl -6 against the public hostname is useful but incomplete. Those flags control the browser-to-Gateway side. They do not necessarily constrain the Gateway-to-pod side.

To debug the complete path, I needed to treat it as two separate connections:

1downstream: client  <-> Gateway
2upstream:   Gateway <-> application pod

The error message contained the clue. upstream connect error meant the request had reached Envoy and the failure concerned its backend connection. It did not diagnose IPv6 by itself, but it told me which side of the proxy needed attention. That cuts an enormous amount of fashionable speculation out of the incident response.

There were two legitimate fixes.

I could configure and verify Vaultwarden’s HTTP server for IPv6, then keep the Service dual-stack. That would mean checking the listener’s behaviour over both families, including what happened to IPv4 when binding to ::.

Or I could narrow the Service to the protocol the process was already serving. The application had a working IPv4 listener. Envoy could reach it. The broken part was the additional backend address I had advertised without a listener behind it.

I chose the smaller change:

1 variable "service_ip_family_policy" {
2-  default = "RequireDualStack"
3+  default = "SingleStack"
4 }
5
6 variable "service_ip_families" {
7-  default = ["IPv4", "IPv6"]
8+  default = ["IPv4"]
9 }

The fix landed at 04:53 on May 24 with this majestic archaeological record:

1d56cbcb Fix vw service

Future historians will have to infer that vw means Vaultwarden and not Volkswagen. Fortunately, I also added a note to the deployment README explaining the actual failure:

1The Vaultwarden Service defaults to IPv4-only. The container listens on
20.0.0.0; a dual-stack Service can make Gateway/Envoy select an IPv6 pod
3endpoint and return upstream connect error.

After the change, the Service published only the IPv4 backends Envoy could actually use. I also put the reason in the variable descriptions: keep this Service IPv4-only unless Vaultwarden is configured to listen on IPv6 too. Otherwise, future me could easily find the apparently conservative default and helpfully reintroduce the outage under the heading IPv6 support.

The Gateway could still accept IPv6 connections from clients and reach Vaultwarden over IPv4. IPv6 remained enabled on the nodes, in DNS, in the Gateways and for other workloads. I had narrowed one Service to match one application’s real capability.

I like building infrastructure that can do more. I also want to be able to leave it alone once it works. Those ambitions get along better when each deployment declares what it can actually serve. A system is not more modern because it advertises paths that terminate in a connection refusal.

Two weeks later, I switched the analytics deployment to Umami. Its server also bound to IPv4:

1umami_env = {
2  HOSTNAME = "0.0.0.0"
3  PORT     = "3000"
4}

This time the Service started with SingleStack and ["IPv4"], and the variable description explained why: Umami bound its HTTP listener to IPv4 in this deployment. Enabling the other family would be a deliberate change to the listener and the Service together.

This is one of the nicer properties of infrastructure as code: trauma can be converted into defaults.

The first deployment teaches you the failure mode at 04:53. The next deployment contains a variable description written by someone who has seen the ghost and would prefer not to host it again.

The useful part of this incident fits into a few commands. I want to compare what the Service advertises with what the process actually serves, following the backend addresses until the two accounts either agree or become interesting.

First, look at the Service’s declared families and assigned cluster IPs:

1kubectl -n vaultwarden get svc vaultwarden \
2  -o jsonpath='{.spec.ipFamilyPolicy}{"\n"}{.spec.ipFamilies}{"\n"}{.spec.clusterIPs}{"\n"}'

Then inspect the backend address families Kubernetes has published:

1kubectl -n vaultwarden get endpointslice \
2  -l kubernetes.io/service-name=vaultwarden \
3  -o custom-columns='NAME:.metadata.name,FAMILY:.addressType,PORT:.ports[*].port,ENDPOINTS:.endpoints[*].addresses[*]'

Check the pod addresses rather than assuming the wide output tells the whole story:

1kubectl -n vaultwarden get pod \
2  -l app.kubernetes.io/name=vaultwarden \
3  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .status.podIPs[*]}{.ip}{" "}{end}{"\n"}{end}'

The HTTPRoute’s status helps check whether the route was accepted and its backend references resolved. It is still worth reading, provided I remember what question it answers:

1kubectl -n vaultwarden get httproute vaultwarden-internal -o yaml

Then check the other end of the claim: what the application actually opened. If the image contains ss:

1kubectl -n vaultwarden exec deploy/vaultwarden -- ss -lntp

Minimal images often omit diagnostic tools, because security and debugging have agreed to share custody of your evening. An ephemeral debug container in the same pod network namespace lets me inspect the sockets without installing packages into the running application container and creating a bespoke production snowflake.

To check whether each address answers, test the pod addresses directly from a debug pod allowed to reach them. Using the example addresses above, that looks like this; substitute the real addresses from the EndpointSlices:

1curl --noproxy '*' --connect-timeout 3 'http://10.244.1.127:8080/alive'
2curl --noproxy '*' --connect-timeout 3 'http://[fd00:10:244:1::7f]:8080/alive'

This bypasses the Gateway and tests each backend family explicitly. If the listener and direct tests look right, the next question is which upstream address Envoy actually selected, visible in its access logs when the upstream host is included. A request through the public hostname cannot answer all of that by itself.

The important habit is to carry the address through the investigation. An address assigned to a pod, published in an EndpointSlice and selected by a proxy should eventually meet a listening socket. Do not stop after the first green answer.

The post-mortem

The mistake was not enabling IPv6. The mistake was treating IPv6 support as an inherited property.

The cluster supported it, therefore the pod must support it. The pod had an IPv6 address, therefore the application must be listening on it. The Service declared both families, therefore both families must work. Every arrow felt reasonable. The conclusion was wrong.

I had read a series of individually reassuring facts as an end-to-end guarantee. The application never made that guarantee. It opened the socket I asked for and left the rest of the platform to discover the consequences with exceptional speed and observability.

The solution was not another controller, sidecar or service mesh. It was making the Service tell the truth about the application behind it.

IPv6 remains enabled across the cluster. It still handles public addresses, private Gateway VIPs, pod networking and the slow work of dragging the internet out of 1981. Vaultwarden simply stays on IPv4 internally until its listener is deliberately configured and tested for both.

The science-fiction franchise can continue. This particular episode ended with a smaller Service configuration and a comment explaining why.

Advertise what works, verify the entire path, and never assume 0.0.0.0 contains more colons than it visibly does.