Array of "as many as will fit" - how to implement?

[Paul Moore]

I'm new to Construct, but I'm trying to understand the RepeatUntil type of object (at least, I think that's what I want to use!). What I have is a data structure (TLV = Type, Length, Value). Type is an unsigned byte, length is the length (in bytes) of the value, and value is a string of bytes of the given length, which is interpreted according to the type.

Type 0 is a simple byte string. So I have

TLV = Struct("TLV",

    UBInt8("type"),

    UBInt16("length"),

    Switch("value", lambda ctx: ctx.type, {

        # Type 0 - raw bytes data

        0 : Bytes("data", lambda ctx: ctx.length),

    }

    )

)

Type 1 is where things get messy. A type 1 entry contains as its value, a series of TLV objects. Each subobject has its own type and value, and the combined length of the subobjects (including their type and length fields) must equal the length specified in the containing type=1 object.

So this is valid: b"\1\0\9\0\0\1a\0\0\2bc". It's Container(type=1, length=9, value=[Container(type=0, length=1, value=b"a"), Container(type=0, length=2, value=b"bc")]).

How would I define a structure like this?

One other thing, I want length values to be checked to ensure there are no "extra" bytes. TVP.parse(data) by default ignores trailing bytes that aren't needed. Can that be turned into an error? Also, I don't want type 1 values to be allowed to have unused trailing bytes.

Thanks for any help. I'm sorry if this is documented somewhere, I have spent some time looking through the docs but couldn't find anything. I *think* that I should be using RepeatUntil, but I'm not entirely sure how to do so (I'd need to keep track of "number of bytes remaining unused" somehow, and I'm not clear how that would work).

Thanks,

Paul

[Frank Carney]

Is there a max depth of recursion on this data structure?  If not you may have to parse to bytes, then parse again.  I cannot find a way to define a structure recursively in the library.  There may be a way through adapters or something, but I am not seeing it.

[Tomer Filiba]

for recursive stuff there's LazyBound (http://construct.readthedocs.org/en/latest/misc.html#lazybound)

for parse-as-much-as-possible there's GreedyRange (http://construct.readthedocs.org/en/latest/api/macros.html#construct.macros.GreedyRange)

-----------------------------------------------------------------

Tomer Filiba  tomerfiliba.com

--
You received this message because you are subscribed to the Google Groups "Construct" group.
To unsubscribe from this group and stop receiving emails from it, send an email to construct3+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Paul Moore]

On Monday, 18 May 2015 18:50:40 UTC+1, Tomer Filiba wrote:

for recursive stuff there's LazyBound (http://construct.readthedocs.org/en/latest/misc.html#lazybound)

for parse-as-much-as-possible there's GreedyRange (http://construct.readthedocs.org/en/latest/api/macros.html#construct.macros.GreedyRange) 

Nice! So the following then works:

from construct import *

TLV = Struct("TLV",

    UBInt8("type"),

    UBInt16("length"),

    Switch("value", lambda ctx: ctx.type, {

        # Type 0 - raw bytes data

        0 : Bytes("data", lambda ctx: ctx.length),

        1 : GreedyRange(LazyBound("data", lambda: TLV)),

    }

    )

)

print(TLV.parse(b"\1\0\x09\0\0\1a\0\0\2bc"))

That's brilliant - many thanks.

Paul

[Frank Carney]

Yeah, i just looked through the posts and realized that:

# testing construct recursive parsing

from construct import *

tlv = Struct("tlv",

    UBInt8("type"),
    UBInt16("length"),
    Switch("value", lambda ctx: ctx.type,
        {
        # Type 0 - raw bytes data
        0 : Bytes("data", lambda ctx: ctx.length),

        1 : GreedyRange(LazyBound("next", lambda: tlv)),
        }
    ),
)

def main():
    # fixed byte data to match parse, still not checking for data left on stream.
    data = bytearray([0x1,0x0,0x9,0x0,0x0,0x1,0xa,0x0,0x0,0x2,0xbc,0xa])

    print tlv.parse(data)

if __name__ == '__main__':
    main()

Took a bit to get my head around it.  Learned something new about construct today.

[Arek Bulski]

And then there is Terminator singleton.