Core API: Streaming

construct.Pointer(offset, subcon)

Changes the stream position to a given offset, where the construction should take place, and restores the stream position when finished.

See also

Analog OnDemandPointer() field, which also seeks to a given offset.

Parameters:
  • offset – an int or a function that takes context and returns absolute stream position, where the construction would take place, can return negative integer as position from the end backwards
  • subcon – the subcon to use at the offset

Example:

>>> Pointer(8, Bytes(1)).parse(b"abcdefghijkl")
b'i'
>>> Pointer(8, Bytes(1)).build(b"x")
b'\x00\x00\x00\x00\x00\x00\x00\x00x'
>>> Pointer(8, Bytes(1)).sizeof()
0
construct.Peek(subcon)

Peeks at the stream. Parses without changing the stream position. If the end of the stream is reached when peeking, returns None. Sizeof returns 0 by design because build does not put anything into the stream. Building is no-op.

See also

The Union() class.

Parameters:subcon – the subcon to peek at

Example:

>>> Sequence(Peek(Byte), Peek(Int16ub)).parse(b"\x01\x02")
[1, 258]
>>> Sequence(Peek(Byte), Peek(Int16ub)).sizeof()
0
construct.Tell()

Gets the stream position when parsing or building.

Tells are useful for adjusting relative offsets to absolute positions, or to measure sizes of Constructs. To get an absolute pointer, use a Tell plus a relative offset. To get a size, place two Tells and measure their difference using a Compute.

See also

Better use RawCopy() instead of manually extracting bytes.

Example:

>>> Struct("num"/VarInt, "offset"/Tell).build(dict(num=88))
b'X'
>>> Struct("num"/VarInt, "offset"/Tell).parse(_)
Container(num=88)(offset=1)
construct.Seek(at, whence=0)

Sets a new stream position when parsing or building. Seeks are useful when many other fields follow the jump. Pointer works when there is only one field to look at, but when there is more to be done, Seek may come useful.

See also

Analog Pointer() wrapper that has same side effect but also processed a subcon.

Parameters:
  • at – where to jump to, can be an int or a context lambda
  • whence – is the offset from beginning (0) or from current position (1) or from ending (2), can be an int or a context lambda, default is 0

Example:

>>> (Seek(5) >> Byte).parse(b"01234x")
[5, 120]
>>> (Bytes(10) >> Seek(5) >> Byte).build([b"0123456789", None, 255])
b'01234\xff6789'
construct.Pass()

A do-nothing construct, useful as the default case for Switch. Returns None on parsing, puts nothing on building.

Example:

>>> Pass.parse(b"")
None
>>> Pass.build(None)
b''
>>> Pass.sizeof()
0
construct.Terminated()

Asserts the end of the stream has been reached at the point it was placed. You can use this to ensure no more unparsed data follows.

This construct is only meaningful for parsing. For building, it’s a no-op.

Example:

>>> Terminated.parse(b"")
None
>>> Terminated.parse(b"remaining")
construct.core.TerminatedError: expected end of stream
construct.Restreamed(subcon, encoder, encoderunit, decoder, decoderunit, decoderunitname, sizecomputer)

Transforms bytes between the underlying stream and the subcon.

When the parsing or building is done, the wrapper stream is closed. If read buffer or write buffer is not empty, error is raised.

See also

Both Bitwise() and Bytewise() are implemented using Restreamed.

Warning

Remember that subcon must consume or produce an amount of bytes that is a multiple of encoding or decoding units. For example, in a Bitwise context you should process a multiple of 8 bits or the stream will fail after parsing/building. Also do NOT use pointers inside.

Parameters:
  • subcon – the subcon which will operate on the buffer
  • encoder – a function that takes a b-string and returns a b-string (used when building)
  • encoderunit – ratio as int, encoder takes that many bytes at once
  • decoder – a function that takes a b-string and returns a b-string (used when parsing)
  • decoderunit – ratio as int, decoder takes that many bytes at once
  • decoderunitname – English string that describes the units (plural) returned by the decoder. Used for error messages.
  • sizecomputer – a function that computes amount of bytes outputed by some bytes

Example:

Bitwise  <--> Restreamed(subcon, bits2bytes, 8, bytes2bits, 1, lambda n: n//8)
Bytewise <--> Restreamed(subcon, bytes2bits, 1, bits2bytes, 8, lambda n: n*8)
construct.Rebuffered(subcon, tailcutoff=None)

Caches bytes from the underlying stream, so it becomes seekable and tellable. Also makes the stream blocking, in case it came from a socket or a pipe. Optionally, stream can forget bytes that went a certain amount of bytes beyond the current offset, allowing only a limited seeking capability while allowing to process an endless stream.

Warning

Experimental implementation. May not be mature enough.

Parameters:
  • subcon – the subcon which will operate on the buffered stream
  • tailcutoff – optional, amount of bytes kept in buffer, by default buffers everything

Example:

Rebuffered(RepeatUntil(lambda obj,ctx: ?,Byte), tailcutoff=1024).parse_stream(endless_nonblocking_stream)