FutureValue idiom to solve length/offset/checksum fields

[Saar Raz]

To get a feel for Construct (Construct 3, specifically), I attempted to represent TCP using it (according to the wikipedia page on TCP).

I came across the "data offset" field, which is a nibble whose value should be the total length of the TCP header, including the variable sized options at the end, and had no way of really representing this kind of relationship.

During unpacking, I could place a validator at the end of the options which would assert that the options combined with the constant length fields do not exceed the value specified, and it would work, but!

Using this method, during packing, I would have to know the correct value of the data_offset field in advance, and put it in the Container passed in to pack().

This is kind of a burden, and the person constructing the packet should not need to calculate such a value in advance.

I hereby propose a solution to this problem, in a way that makes both packing and unpacking more pleasant, the FutureValue idiom.

A FutureValue is, similar to Value, a field whose value is a computed value at pack/unpack time. Its value is completely dependent upon the context and the user does not have to or should not provide it himself.

The difference is - the value of a FutureValue is not known at the time the parser reaches it, but will only be known later in the parsing process.

It looks like this:

packet = (
    'foo' / Const(byte, 0xF0)
    >> 'total_size' / FutureValue(uint16b),
    >> 'flags' / Bitwise(
        'a_present' / bit
        >> Padding(7)
    )
    >> Embedded(If(this.flags.a_present, Struct('a' / uint32b), noop))
    >> '_total_size' / anchor
    >> Embedded(BindValue(this.total_size, this._total_size))
)

>>> packet.unpack('\xf0\x00\x04\x00')

Container:

  total_size = 4

  foo = 240

  flags = Container:

    a_present = 0

>>> packet.unpack('\xf0\x00\x05\x00')
ValidationError: Expected value 4 but got 5

>>> packet.pack(Container(flags=Container(a_present=1), a=0xaaaaaaaa))

'\xf0\x00\x08\x80\xaa\xaa\xaa\xaa'

>>> packet.unpack('\xf0\x00\x08\x80\xaa\xaa\xaa\xaa')

Container:

  total_size = 8

  a = 2863311530L

  foo = 240

  flags = Container:

    a_present = 1

The BindValue packer will be put whenever the value of a past FutureValue is able to be calculated, and its packing operation will replace the placeholder value put in by the FutureValue packer, with the actual, calculated value.

If BindValue sees a value has already been put in FutureValue (either it was parsed or provided in the Container passed in to pack), it will only assert its value matches the calculated value.

What do you guys think of this idiom? Do you have a more elegant solution?

Here is, by the way, my end result TCP representation (note that ~SomePacker is a shorthand I introduced that evaluates to Embedded(SomePacker))

TCP = (
    'source_port' / uint16b
    >> 'destination_port' / uint16b
    >> 'sequence_number' / uint32b
    >> 'acknowledgement_number' / uint32b
    >> ~Bitwise(
        'data_offset' / FutureValue(Adapter(nibble, decode=lambda off, ctx: off * 4, encode=lambda bytes, ctx: bytes / 4))
        >> 'reserved' / Const(Bits(3), 0)
        >> 'flags' / (
            'ns' / bit
            >> 'cwr' / bit
            >> 'ece' / bit
            >> 'urg' / bit
            >> 'ack' / bit
            >> 'psh' / bit
            >> 'rst' / bit
            >> 'syn' / bit
            >> 'fin' / bit
        )
    )
    >> 'window_size' / uint16b
    >> 'checksum' / uint16b
    >> 'urgent_pointer' / uint16b
    >> '_header_length' / anchor
    >> 'options' / Range(
        0,
        None,
        If(
            lambda ctx: any(option['option_kind'] == 'EOO' for key, option in ctx.iteritems() if key != '_'),
            error('Option past an END_OF_OPTIONS marker.'),
            '_option_start' / anchor
            >> 'option_kind' / Enum(uint8, EOO=0, NOP=1, MSS=2, WS=3, SACKP=4, SACK=5, TIMESTAMP=8)
            >> ~Switch(
                this.option_kind,
                {
                    'EOO': noop,
                    'NOP': noop
                },
                default=(
                    'option_length' / FutureValue(byte)
                    >> ~Switch(
                        this.option_kind,
                        {
                            'MSS': 'maximum_segment_size' / uint16b,
                            'WS': 'window_scale' / uint32b,
                            'SACKP': noop,
                            'SACK': 'blocks' / Range(1, None, 'left_edge' / uint32b >> 'right_edge' / uint32b)
                        }
                    )
                    >> '_option_end' / anchor
                    >> ~BindValue(this.option_length, this._option_end - this._option_start)
                )
            )
        )
    )
    >> '_options_end' / anchor
    >> Padding(lambda ctx: ctx['data_offset'] - ctx['_options_end'] if ctx['data_offset'] else 0 if ctx['_options_end'] % 4 == 0 else 4 - ctx['_options_end'] % 4)
    >> '_tcp_length' / anchor
    >> ~BindValue(this.data_offset, this._tcp_length)
)

Note that the idiom is also used in individual options as well as in the complete packet with the aforementioned data offset field.

I do have this all implemented in a pretty ugly way but it works.