The Context¶

Meta constructs are the key to the declarative power of Construct. Meta constructs are constructs which are affected by the context of the construction (during parsing and building). The context is a dictionary that is created during the parsing and building process by Structs and Sequences, and is “propagated” down and up to all constructs along the way, so that other members can access other members parsing or building intermediate results. It basically represents a mirror image of the construction tree, as it is altered by the different constructs. Nested structs create nested contexts, just as they create nested containers.

In order to see the context, let’s try this snippet:

>>> d = Struct(
...     "a" / Byte,
...     Probe(),
...     "b" / Byte,
...     Probe(),
... )
>>> d.parse(b"\x01\x02")
--------------------------------------------------
Probe, path is (parsing), into is None
Container:
    a = 1
--------------------------------------------------
--------------------------------------------------
Probe, path is (parsing), into is None
Container:
    a = 1
    b = 2
--------------------------------------------------
Container(a=1, b=2)

As you can see, the context looks different at different points of the construction.

You may wonder what does the little underscore (_) that is found in the context means. It basically represents the parent node, like the .. in unix pathnames (../foo.txt). We’ll use it only when we refer to the context of upper layers.

Using the context is easy. All meta constructs take a function as a parameter, which is usually passed as a lambda function, although “big” named functions are just as good. This function, unless otherwise stated, takes a single parameter called ctx (short for context), and returns a result calculated from that context.

>>> st = Struct(
...     "count" / Byte,
...     "data" / Bytes(lambda ctx: ctx.count),
... )
>>> st.parse(b"\x05abcde")
Container(count=5, data=b'abcde')

Of course a function can return anything (it does not need to depend on the context):

>>> Computed(lambda ctx: 7)
>>> Computed(lambda ctx: os.urandom(16))

Nesting¶

And here’s how we use the special _ name to get to the upper container in a nested containers situation (which happens when parsing nested Struct). Notice that length1 is on different (upper) level than length2, therefore it exists within a different up-level containter.

>>> d = Struct(
...     "length1" / Byte,
...     "inner" / Struct(
...         "length2" / Byte,
...         "sum" / Computed(lambda ctx: ctx._.length1 + ctx.length2),
...     ),
... )
>>> d.parse(b"12")
Container(length1=49, inner=Container(length2=50, sum=99))

Context entries can also be passed directly through parse and build methods. However, one should take into account that some classes are nesting context (like Struct, Sequence, Union, FocusedSeq or LazyStruct), so entries passed to these end up on upper level. Compare examples:

>>> d = Bytes(lambda ctx: ctx.n)
>>> d.parse(bytes(100), n=4)
b'\x00\x00\x00\x00'

>>> d = Struct(
...     "data" / Bytes(lambda ctx: ctx._.n),
... )
>>> d.parse(bytes(100), n=4)
Container(data=b'\x00\x00\x00\x00')

Refering to inlined constructs¶

If you need to refer to a subcon like Enum, that was inlined in the struct (and therefore wasnt assigned to any variable in the namespace), you can access it as Struct attribute under same name. This feature is particularly handy when using Enum and EnumFlag classes.

>>> d = Struct(
...     "animal" / Enum(Byte, giraffe=1),
... )
>>> d.animal.giraffe
EnumIntegerString.new(1, 'giraffe')

If you need to refer to the size of a field, that was inlined in the same struct (and therefore wasnt assigned to any variable in the namespace), you can use a special _subcons context entry that contains all struct members. Note that you need to use a lambda (because this expression is not supported).

>>> d = Struct(
...     "count" / Byte,
...     "data" / Bytes(lambda ctx: ctx.count - ctx._subcons.count.sizeof()),
... )
>>> d.parse(b"\x05four")
Container(count=5, data=b'four')

>>> d = Union(None,
...     "chars" / Byte[4],
...     "data" / Bytes(lambda ctx: ctx._subcons.chars.sizeof()),
... )
>>> d.parse(b"\x01\x02\x03\x04")
Container(chars=ListContainer([1, 2, 3, 4]), data=b'\x01\x02\x03\x04')

Using `this` expression¶

Certain classes take a number of parameters and allow a callable to be provided instead of constants. This callable is called at parsing and building, and is provided the current context dictionary. Context is always a Container, not a dict, so it supports attribute access as well as key access. Amazingly, this can get even more fancy. Tomer Filiba provided an even better syntax. The this singleton object can be used to build a lambda expression. All four examples below are equivalent, but first is recommended:

>>> this._.field
>>> lambda this: this._.field
>>> this["_"]["field"]
>>> lambda this: this["_"]["field"]

Of course, this expression can be mixed with other calculations. When evaluating, each instance of this is replaced by context Container which supports attribute access to keys.

>>> this.width * this.height - this.offset

When creating an Array (the "items" field in the following example), rather than specifying a constant count, you can use a previous field value as count.

>>> d = Struct(
...     "count" / Rebuild(Byte, lambda ctx: len(ctx.items)),
...     "items" / Byte[this.count],
... )
>>> d.build(dict(items=[1,2,3,4,5]))
b'\x05\x01\x02\x03\x04\x05'

Switch can branch the construction path based on previously parsed value.

>>> d = Struct(
...     "type" / Enum(Byte, INT1=1, INT2=2, INT4=3, STRING=4),
...     "data" / Switch(this.type,
...     {
...         "INT1" : Int8ub,
...         "INT2" : Int16ub,
...         "INT4" : Int32ub,
...         "STRING" : CString('ascii'),
...     }),
... )
>>> d.parse(b"\x02\x00\xff")
Container(type=EnumIntegerString.new(2, 'INT2'), data=255)
>>> d.parse(b"\x04abcdef\x00")
Container(type=EnumIntegerString.new(4, 'STRING'), data='abcdef')

Using len_` expression¶

There used to be a bit of a hassle when you used built-in functions like len, sum, min, max or abs on context items. Built-in len takes a list and returns an integer but len_ analog takes a lambda and returns a lambda. This allows you to use this kind of shorthand:

>>> len_(this.items)
>>> lambda ctx: len(ctx.items)

These can be used in Rebuild wrappers that compute count/length fields from another list-alike field:

>>> d = Struct(
...     "count" / Rebuild(Byte, len_(this.items)),
...     "items" / Byte[this.count],
... )
>>> d.build(dict(items=[1,2,3,4,5]))
b'\x05\x01\x02\x03\x04\x05'

Using `obj_` expression¶

There is also an analog that takes (obj, context) or (obj, list, context) unlike this singleton which only takes a context as a single parameter:

>>> obj_ > 0
>>> lambda obj,ctx: obj > 0

These can be used in at least one construct:

>>> RepeatUntil(obj_ == 0, Byte).parse(b"aioweqnjkscs\x00")
ListContainer([97, 105, 111, 119, 101, 113, 110, 106, 107, 115, 99, 115, 0])

Using `list_` expression¶

Warning

The list_ expression is implemented but buggy, using it is not recommended at present time.

There is also a third expression that takes (obj, list, context) and computes on the second parameter (the list). In constructs that use lambdas with all 3 parameters, those constructs usually process lists of elements and the 2nd parameter is a list of elements processed so far.

These can be used in at least one construct:

>>> RepeatUntil(list_[-1] == 0, Byte).parse(b"aioweqnjkscs\x00")
[97, 105, 111, 119, 101, 113, 110, 106, 107, 115, 99, 115, 0]

In that example, list_ gets substituted with following, at each iteration. Index -1 means last element:

list_ <- [97]
list_ <- [97, 105]
list_ <- [97, 105, 111]
list_ <- [97, 105, 111, 119]
...

Known deficiencies¶

Logical and, or, not operators cannot be used in this expressions. You have to either use a lambda or equivalent bitwise operators:

>>> ~this.flag1 | this.flag2 & this.flag3
>>> lambda this: not this.flag1 or this.flag2 and this.flag3

Contains operator in cannot be used in this expressions, you have to use a lambda:

>>> lambda this: this.value in (1, 2, 3)

Indexing (square brackets) do not work in this expressions. Use a lambda:

>>> lambda this: this.list[this.index]

Sizeof method does not work in this expressions. Use a lambda:

>>> lambda this: this._subcons.<member>.sizeof()

Lambdas (unlike this expressions) are not compilable.

The Context¶

Nesting¶

Refering to inlined constructs¶

Using this expression¶

Using len_` expression¶

Using obj_ expression¶

Using list_ expression¶

Known deficiencies¶

Using `this` expression¶

Using `obj_` expression¶

Using `list_` expression¶