Streaming tactics

Note

Certain constructs are available only for seekable streams (in-memory and files) and some require tellable streams (which in fact is a subset of seekability). Sockets and pipes do not support seeking, so you’ll have to first read the data from the stream, and parse it in-memory, or use the Rebuffered() wrapper.

Wrappers

Pointer

Pointer allows for non-sequential construction. The pointer first changes the stream position, does the construction, and restores the original stream position.

>>> Pointer(8, Bytes(1)).parse(b"abcdefghijkl")
b'i'
>>> Pointer(8, Bytes(1)).build(b"x")
b'\x00\x00\x00\x00\x00\x00\x00\x00x'

Peek

Parses the subconstruct but restores the stream position afterwards (it does peeking).

>>> Sequence(Peek(Byte), Peek(Int16ub)).parse(b"\x01\x02")
[1, 258]

Pure side effects

Seek makes a jump within the stream and leaves it at that point. It does not read or write anything to the stream by itself.

class construct.core.Seek(at, whence=0)

Sets a new stream position when parsing or building. Seeks are useful when many other fields follow the jump. Pointer works when there is only one field to look at, but when there is more to be done, Seek may come useful.

See also

Analog Pointer() wrapper that has same side effect but also processed a subcon.

Parameters:
  • at – where to jump to, can ne an int or a context lambda
  • whence – is the offset from beginning (0) or from current position (1) or from ending (2), can be an int or a context lambda, default is 0

Example:

>>> (Seek(5) >> Byte).parse(b"01234x")
[5, 120]
>>> (Bytes(10) >> Seek(5) >> Byte).build([b"0123456789", None, 255])
b'01234\xff6789'

Tell checks the current stream position and returns it, also putting it into the context. It does not read or write anything to the stream by itself.

construct.core.Tell()

Gets the stream position when parsing or building.

Tells are useful for adjusting relative offsets to absolute positions, or to measure sizes of Constructs. To get an absolute pointer, use a Tell plus a relative offset. To get a size, place two Tells and measure their difference using a Compute.

See also

Better to use CopyRaw() wrapper in almost any case.

Example:

>>> Struct("num"/VarInt, "offset"/Tell).build(dict(num=88))
b'X'
>>> Struct("num"/VarInt, "offset"/Tell).parse(_)
Container(num=88)(offset=1)

Stream manipulation

construct.core.Rebuffered(subcon, tailcutoff=None)

Caches bytes from the underlying stream, so it becomes seekable and tellable. Also makes the stream blocking, in case it came from a socket or a pipe. Optionally, stream can forget bytes that went a certain amount of bytes beyond the current offset, allowing only a limited seeking capability while allowing to process an endless stream.

Warning

Experimental implementation. May not be mature enough.

Parameters:
  • subcon – the subcon which will operate on the buffered stream
  • tailcutoff – optional, amount of bytes kept in buffer, by default buffers everything

Example:

Rebuffered(RepeatUntil(lambda obj,ctx: ?,Byte), tailcutoff=1024).parse_stream(endless_nonblocking_stream)
construct.core.Restreamed(subcon, encoder, encoderunit, decoder, decoderunit, sizecomputer)

Transforms bytes between the underlying stream and the subcon.

When the parsing or building is done, the wrapper stream is closed. If read buffer or write buffer is not empty, error is raised.

See also

Both Bitwise() and Bytewise() are implemented using Restreamed.

Warning

Remember that subcon must consume or produce an amount of bytes that is a multiple of encoding or decoding units. For example, in a Bitwise context you should process a multiple of 8 bits or the stream will fail after parsing/building. Also do NOT use pointers inside.

Parameters:
  • subcon – the subcon which will operate on the buffer
  • encoder – a function that takes a b-string and returns a b-string (used when building)
  • encoderunit – ratio as int, encoder takes that many bytes at once
  • decoder – a function that takes a b-string and returns a b-string (used when parsing)
  • decoderunit – ratio as int, decoder takes that many bytes at once
  • sizecomputer – a function that computes amount of bytes outputed by some bytes

Example:

Bitwise  <--> Restreamed(subcon, bits2bytes, 8, bytes2bits, 1, lambda n: n//8)
Bytewise <--> Restreamed(subcon, bytes2bits, 1, bits2bytes, 8, lambda n: n*8)