[openssl/openssl] bff0a6: DTLS listener: bound the per-connection receive queue

[Mounir IDRASSI]

Branch: refs/heads/openssl-4.1
Home: https://github.com/openssl/openssl
Commit: bff0a6356e31c3897eff8572977384dc1f23bea4
https://github.com/openssl/openssl/commit/bff0a6356e31c3897eff8572977384dc1f23bea4
Author: Mounir IDRASSI <mounir....@idrix.fr>
Date: 2026-10-14 (Wed, 14 Oct 2026)

Changed paths:
M include/internal/dtls_record_rx.h
M ssl/dtls_record_rx.c
M test/dtlsssllistenertest.c

Log Message:
-----------
DTLS listener: bound the per-connection receive queue

The DTLS listener queues datagrams from a known peer into a per-connection
list (DTLS_RX::urxe_pending) with no limit on count or bytes. A reachable
peer that completes the ordinary cookie exchange can grow one parked
connection's queue without bound as long as the application keeps
the shared demux pumped, moving datagrams into userspace past the bound
the kernel socket buffer would otherwise provide.

Cap the queue at OSSL_DTLS_RX_DEFAULT_MAX_QUEUED (128) datagrams
per connection - a datagram count, not a byte cap. Every queued URXE
holds a buffer of at least the listener's maximum datagram size whatever
the datagram's length, so a full queue costs about 295 KB per connection
at the default 2000-byte size, roughly one default kernel socket receive
buffer; across a full 256-connection listener budget that is about 75 MB.
The queue is consumed FIFO, so on overflow the oldest datagram is evicted
rather than the newest: a DTLS flight can span several datagrams,
and under overload keeping the newest biases the queue toward the flight
the peer just sent instead of stale fragments of an older one.
The evicted URXE is unhooked under rx->mutex and released to the demux
after dropping it, so demux->mutex is never taken while rx->mutex
is held.

References: https://github.com/openssl/openssl/issues/33173
Assisted-by: Claude Code:moonshotai/kimi-k3
Reviewed-by: Tomas Mraz <to...@openssl.foundation>
Reviewed-by: Ryan Hooper <ry...@openssl.foundation>
Reviewed-by: Eugene Syromiatnikov <es...@openssl.org>
Merge-date: Wed Oct 14 10:54:54 2026
Merged-from: https://github.com/openssl/openssl/pull/33197
(cherry picked from commit 8a57e59db7b51258e3c9fec8359c90b85b4d5f2c)


Commit: a329a5d88062490c824c837219ea71c74ceff4d8
https://github.com/openssl/openssl/commit/a329a5d88062490c824c837219ea71c74ceff4d8
Author: Mounir IDRASSI <mounir....@idrix.fr>
Date: 2026-10-14 (Wed, 14 Oct 2026)

Changed paths:
M doc/man3/SSL_get_value_uint.pod
M doc/man3/SSL_new_listener.pod
M doc/man7/ossl-guide-dtlsv13.pod
M ssl/d1_lib.c
M ssl/ssl_local.h
M test/dtlsssllistenertest.c

Log Message:
-----------
DTLS listener: make max_pending_conns a combined admission budget

The pending-connection cap only applied to pending_conns, the table
of connections still doing their handshake: the moment a connection passed
cookie validation it was moved onto the accept queue (incoming_connections)
with no length check and no expiry. A reachable peer completing
the ordinary cookie exchange from many source endpoints could accumulate
an unbounded, unexpiring queue of listener-owned SSL objects.
This is the same missing admission bound as the QUIC listener admission
check fixed in [1]; the DTLS listener is new in 4.1 and not in any final
release.

Mirror the QUIC fix: enforce the bound at admission, in the packet
handler, before any per-connection state is allocated.
SSL_VALUE_DTLS_LISTENER_MAX_PENDING_CONNS is now a single admission budget
covering both listener-owned tables, pending plus accept queue,
so listener-owned connections are bounded at N in total, including
under SSL_LISTENER_FLAG_NO_VALIDATE. Moving a connection from one table
to the other leaves the total unchanged, so a validated connection
can always be queued: none is discarded, and no handshake work (cookie
exchange, server flight) is spent on an attempt that cannot be admitted.
A peer whose datagram is dropped while the budget is exhausted retransmits
its ClientHello and is admitted once the application drains the queue;
measured on real UDP, such a client completes its handshake about one
retransmit interval after the drain.

This also aligns the DTLS knob's meaning with the QUIC one ("not yet
accepted"), which it was modelled on in [2].

Accept-queue entries still do not expire; a full queue stays full until
the application drains it. With the budget, an attacker can pin at most N
accept-queue slots, and only after proving return routability - spoofed
ClientHellos can still occupy pending slots until the pending timeout
reaps them, as before this change - so what remains is a bounded
availability concern, not memory exhaustion.

[1] https://github.com/openssl/openssl/pull/32052
[2] https://github.com/openssl/openssl/pull/31980

Resolves: https://github.com/openssl/openssl/issues/33172
Assisted-by: Claude Code:moonshotai/kimi-k3
Reviewed-by: Tomas Mraz <to...@openssl.foundation>
Reviewed-by: Ryan Hooper <ry...@openssl.foundation>
Reviewed-by: Eugene Syromiatnikov <es...@openssl.org>
Merge-date: Wed Oct 14 10:54:55 2026
Merged-from: https://github.com/openssl/openssl/pull/33197
(cherry picked from commit 8628bb9f21365f3c448b3e08cef337992027587c)


Commit: 6198fb889602a7476dad1d652041153081a87b17
https://github.com/openssl/openssl/commit/6198fb889602a7476dad1d652041153081a87b17
Author: Mounir IDRASSI <mounir....@idrix.fr>
Date: 2026-10-14 (Wed, 14 Oct 2026)

Changed paths:
M include/internal/dgram_demux.h
M ssl/d1_lib.c
M ssl/dgram_demux.c
M test/dtlsssllistenertest.c

Log Message:
-----------
DTLS listener: bound the demux free list

The demux free list caches released URXEs so a receive burst can be
served without allocating, but nothing bounds it: a transient flood
raising the per-connection receive queues also raises the free list's
retained high-water mark for the remaining lifetime of the listener.

Add a per-demux limit, ossl_dgram_demux_set_max_free_urxe(), enforced
in ossl_dgram_demux_release_urxe(): a URXE released while the free list
is at its limit is freed instead of cached, so the bound is strict
and no later trim pass is needed. Freeing there honours the documented
contract that no reference may be made to a URXE or its buffer after
it is released, and no release caller touches the entry afterwards.
The DTLS listener sets the limit to DEMUX_MAX_FREE_URXE, two receive
batches, so one full batch can always be cached; QUIC keeps the default
unbounded behaviour.

Resolves: https://github.com/openssl/openssl/issues/33173
Assisted-by: Claude Code:moonshotai/kimi-k3
Reviewed-by: Tomas Mraz <to...@openssl.foundation>
Reviewed-by: Ryan Hooper <ry...@openssl.foundation>
Reviewed-by: Eugene Syromiatnikov <es...@openssl.org>
Merge-date: Wed Oct 14 10:54:56 2026
Merged-from: https://github.com/openssl/openssl/pull/33197
(cherry picked from commit 497bdda39a8bbd293d44f0454c0be620c11cc8c2)


Compare: https://github.com/openssl/openssl/compare/ad6335398b8a...6198fb889602

To unsubscribe from these emails, change your notification settings at https://github.com/openssl/openssl/settings/notifications