IPv4/IPv6 Split-Stack Incompatibility with Serf Gossip over NAT64 – Protocols or Workarounds?

[Kamal Srangal]

Hey! 

I’m evaluating a cluster topology involving an IPv4-only Consul server control plane and IPv6-only application worker nodes (e.g., EKS IPv6-only pods) communicating via a NAT64/DNS64 gateway.

During our testing, we observed that while standard HTTP API operations and initial TCP push/pull joins succeed over the translated NAT64 prefix (e.g., 64:ff9b::/96), the LAN Serf/memberlist gossip protocol breaks down immediately after:

1. Memberlist IP Storage: The server and agents exchange raw IP byte arrays during state sync. The agent receives plain IPv4 literals (10.0.0.x) without the NAT64 prefix, while the server receives pure IPv6 literals (fd00::x).

2. Asymmetric Failure:

   • The agent cannot send UDP gossip probes to raw IPv4 addresses because its network stack is IPv6-only.

   • The server cannot send UDP gossip probes to the agent’s IPv6 address because it lacks an IPv6 route/stack, and NAT64 does not support arbitrary IPv4 \rightarrow IPv6 inbound initiation without static 1:1 mappings.

3. Cluster Ejection: Serf suspicion timers fire on both sides, causing agents to be marked dead and ejected shortly after joining.

We are aware that flags like advertise_addr only support a single IP per interface and that features like translate_wan_addrs target WAN/service discovery queries rather than local LAN gossip payloads.

Questions for the group:

1. Gossip Protocol Roadmap: Are there any proposals or workarounds within Serf/memberlist to support dual-address advertising or context-aware address rewriting for local LAN gossip payloads?

2. Alternative Solutions: Has anyone successfully run traditional Consul Client Agents in a NAT64 split-stack environment without full dual-stacking on either side?

Thanks in advance for your insights!