{Proposal} Control Protocol: Extended Control Protocol 1

[Sharvil Nanavati]

Summary:

This proposal consists of two control packets that could reduce the
connection overhead. These packets comprise Extended Control Protocol
1 (ECP1) and should be used after successful negotiation of the
protocol through the proposed Protocol Upgrade extension.

Rationale:

The first packet, "Acknowledge All", aims to reduce the overhead of
acknowledging packets. Currently, the "Acknowledge Reliable" packet
only operates on a single packet. Most UDP packets arrive in order so
reliable packets usually get acknowledged in-order. By using a single
"Acknowledge All" packet, all previous reliable packets implicitly get
acknowledged, reducing the connection overhead.

The second packet, "Set Window Size", can reduce the number of
reliable packets thrown away when received. Reliable packets beyond
the window of the SubSpace protocol stack are dropped as the receiver
might not have enough memory (or limits the amount of memory) for
reliable packets. The sender, if unaware of the receiver's window
size, would resend those dropped reliable packets and send more
reliable packets outside the window, leading to additional waste. This
packet allows both parties to specify their window size.

Packets:

Packet Name: Acknowledge All
Packet: 0x00 0x92 <reliableID (4)>
reliableID: the largest sequence number of the reliable packet being
acknowledged - all reliable packets before this ID are implicitly
acknowledged by the receiver.

Packet Name: Set Window Size
Packet: 0x00 0x93 <windowSize (4)>
windowSize: the number of reliable packets beyond the last reliable
packet that can be accepted.

Usage:

Both packets can be used at any time by both parties after ECP1 or
higher has been negotiated.

[Chris "Cerium" Rog]

> The first packet, "Acknowledge All", aims to reduce the overhead of
> acknowledging packets. Currently, the "Acknowledge Reliable" packet
> only operates on a single packet. Most UDP packets arrive in order so
> reliable packets usually get acknowledged in-order. By using a single
> "Acknowledge All" packet, all previous reliable packets implicitly get
> acknowledged, reducing the connection overhead.

The problem I can see here is that if you have 10 packets to
acknowledge, but only received 9, you'll never get the missing packet.
For some packets this wouldn't matter, but I can think of several
cases where reliable packets are used strictly to ensure that they are
processed in a set order. This has the potential to severely break
that.

What we could do as an alternative is have a list of packet ids,
rather than just the single we're sending now. This way we can cut
down on overhead and bandwidth a bit and still not improperly ack
packets we haven't actually received.
Technically, we could already do this with packet chunking... but that
typically gets sent reliably itself, which requires another ack (which
is kinda silly to be ack'ing the acks... ack, ack ack ack).

> The second packet, "Set Window Size", can reduce the number of
> reliable packets thrown away when received. Reliable packets beyond
> the window of the SubSpace protocol stack are dropped as the receiver
> might not have enough memory (or limits the amount of memory) for
> reliable packets. The sender, if unaware of the receiver's window
> size, would resend those dropped reliable packets and send more
> reliable packets outside the window, leading to additional waste. This
> packet allows both parties to specify their window size.

Are you planning for embedded systems or something? Unless there are
some severe packetloss issues involved, I don't think I've ever seen
reliables get thrown out.

[Sharvil Nanavati]

> > The first packet, "Acknowledge All", aims to reduce the overhead of
> > acknowledging packets. Currently, the "Acknowledge Reliable" packet
> > only operates on a single packet. Most UDP packets arrive in order so
> > reliable packets usually get acknowledged in-order. By using a single
> > "Acknowledge All" packet, all previous reliable packets implicitly get
> > acknowledged, reducing the connection overhead.
>
> The problem I can see here is that if you have 10 packets to
> acknowledge, but only received 9, you'll never get the missing packet.
> For some packets this wouldn't matter, but I can think of several
> cases where reliable packets are used strictly to ensure that they are
> processed in a set order. This has the potential to severely break
> that.
>

I'm not sure I follow the reasoning here. How do you have 10 packets
to acknowledge after receiving only 9 reliable packets? You would only
generate an ACK for a reliable once you've processed it (or buffered
it for deferred, in-order processing).

> What we could do as an alternative is have a list of packet ids,
> rather than just the single we're sending now. This way we can cut
> down on overhead and bandwidth a bit and still not improperly ack
> packets we haven't actually received.
> Technically, we could already do this with packet chunking... but that
> typically gets sent reliably itself, which requires another ack (which
> is kinda silly to be ack'ing the acks... ack, ack ack ack).

I would love to extend the 0x00 0x04 packet so that every 32-bit
integer after the packet identifier is a reliable ID that's being
acknowledged. It would be particularly useful when one reliable goes
missing in a series of reliables. In that case, the ACK-all wouldn't
work and the single-ACK would have a lot of overhead (as you
mentioned).

> > The second packet, "Set Window Size", can reduce the number of
> > reliable packets thrown away when received. Reliable packets beyond
> > the window of the SubSpace protocol stack are dropped as the receiver
> > might not have enough memory (or limits the amount of memory) for
> > reliable packets. The sender, if unaware of the receiver's window
> > size, would resend those dropped reliable packets and send more
> > reliable packets outside the window, leading to additional waste. This
> > packet allows both parties to specify their window size.
>
> Are you planning for embedded systems or something? Unless there are
> some severe packetloss issues involved, I don't think I've ever seen
> reliables get thrown out.

I certainly do want to see embedded devices running SubSpace-related
software. But there's more to it than just that.

Reliables *should* be discarded if they're outside a window or else
the reliable sender could mount a nasty memory attack. If your
protocol stack accepts all reliables and keeps them around until
they're ready to be processed, I could keep sending you reliables in
the 0x10000000-0xFFFFFFFF range and you'd keep allocating more and
more memory. Eventually, you'd run out of memory (assuming 32-bit
address space) and your application dies. Even if I don't manage to
take it down, you'll have enough memory pressure that your machine
will start swapping and will be running with degraded performance. You
could drop the connection entirely if the reliable happens to be too
far out but that would yield an intolerant system. I feel that the
control protocol should be resilient to failures and errors where
possible. Instead of dropping the connection, I think dropping the
packet would be a more appropriate action.

Given my argument for discarding reliables, I would say that knowing
the remote host's window size would help build a more efficient
connection.

[Chris "Cerium" Rog]

> I'm not sure I follow the reasoning here. How do you have 10 packets
> to acknowledge after receiving only 9 reliable packets? You would only
> generate an ACK for a reliable once you've processed it (or buffered
> it for deferred, in-order processing).

I meant the situation you described below: The server is waiting for
10 acks, the client receives only 9 packets but sends an ack-all. The
server thinks all 10 have been received when in fact only 9 have.

> I would love to extend the 0x00 0x04 packet so that every 32-bit
> integer after the packet identifier is a reliable ID that's being
> acknowledged. It would be particularly useful when one reliable goes
> missing in a series of reliables. In that case, the ACK-all wouldn't
> work and the single-ACK would have a lot of overhead (as you
> mentioned).

I still see a generic ack-all as have very few uses in itself. If
you're to add a new packet for doing multiple acks in a single packet,
why not have a ranged-ack; a packet which has two 32bit values for the
packet ids which define the range. Everything in between (and
counting) those ids will be ack'd. This way you could send two packets
to cover the missing packet in the situation we've both described or
you could just do 0x00 to 0xFFFFFFFF as a way of sending an ack-all.

>
> Reliables *should* be discarded if they're outside a window or else
> the reliable sender could mount a nasty memory attack. If your
> protocol stack accepts all reliables and keeps them around until
> they're ready to be processed, I could keep sending you reliables in
> the 0x10000000-0xFFFFFFFF range and you'd keep allocating more and
> more memory. Eventually, you'd run out of memory (assuming 32-bit
> address space) and your application dies. Even if I don't manage to
> take it down, you'll have enough memory pressure that your machine
> will start swapping and will be running with degraded performance. You
> could drop the connection entirely if the reliable happens to be too
> far out but that would yield an intolerant system. I feel that the
> control protocol should be resilient to failures and errors where
> possible. Instead of dropping the connection, I think dropping the
> packet would be a more appropriate action.
>
> Given my argument for discarding reliables, I would say that knowing
> the remote host's window size would help build a more efficient
> connection.

I see. But would the window size define how any packets the server/
client will keep in memory, or how far away from it's current id it
will accept?

[Sharvil Nanavati]

> I still see a generic ack-all as have very few uses in itself. If
> you're to add a new packet for doing multiple acks in a single packet,
> why not have a ranged-ack; a packet which has two 32bit values for the
> packet ids which define the range. Everything in between (and
> counting) those ids will be ack'd. This way you could send two packets
> to cover the missing packet in the situation we've both described or
> you could just do 0x00 to 0xFFFFFFFF as a way of sending an ack-all.
>

The ack-all packet acknowledges everything <= the ID specified in the
packet. Most connections these days have < 1% packetloss so the
probability of a reliable being lost is close to 0. In 6 bytes, we can
handle the most common case so I still think it's a valuable addition
and I would argue that we'll see this packet being used more often
than any other type of ack.

You're right in saying that if there's a hole, we can't use an ack-all
and we would have to fall back to either ack-single or some sort of
ack-range. Let's say we extend the ack-all packet to support ack-range
as well. If the packet is 6 bytes, it's an ack-all and if it's 10
bytes, it's an ack-range following:

0x00 0x92 <rangeStart (4)> <rangeEnd (4)>

where the start and end are inclusive.

> I see. But would the window size define how any packets the server/
> client will keep in memory, or how far away from it's current id it
> will accept?

The window size tells the other side how many unacknowledged reliables
it can send. For example, if I specify a window size of 3 and you have
10 reliables to send, you can first send reliables 0-2 (inclusive),
then wait for at least one ack before sending the next reliable. At
any given time, you can have at most 3 reliables sent that have not
been acknowledged.

[Chris "Cerium" Rog]

So the window size is basically just to tell the other end the maximum
number of packets it can have in transit before bad things (tm) happen?

[Sharvil Nanavati]

That's just it, where Bad Things = packets dropped. This is just a
crude version of how TCP does flow control and I think it applies to
SubSpace reliables just as well as it does to a TCP stream.

[Sharvil Nanavati]

There haven't been any arguments against this proposal. Can we approve
this by end-of-day Wednesday (with the addition that if the
"Acknowledge All" packet is 9 bytes long it's an "Acknowledge Range")?

On Jul 7, 12:20 pm, Sharvil Nanavati <Sharvil.Nanav...@gmail.com>
wrote:

[Cerium]

Sounds good.

[D1st0rt]

Agreed

[Sharvil Nanavati]

I've just finished implementing ECP1 in ASSS 1.4.4. I tested it
against my own implementation and everything looks correct. The patch
can be found in the "Files" section of this site. Before applying the
patch for ECP1, make sure you apply the patch for the upgrade
protocol. In the asss/src directory, run:

patch -p0 < asss-1.4.4-ecp1.patch