Streaming tactics **¶
Certain constructs are available only for seekable streams (in-memory and files) and some require tellable streams (which in fact is a subset of seekability). Sockets and pipes do not support seeking, so you’ll have to first read the data from the stream, and parse it in-memory, or use the
Pointer allows for non-sequential construction. The pointer first changes the stream position, does the construction, and restores the original stream position.
>>> Pointer(8, Bytes(1)).parse(b"abcdefghijkl") b'i' >>> Pointer(8, Bytes(1)).build(b"x") b'\x00\x00\x00\x00\x00\x00\x00\x00x'
Parses the subconstruct but restores the stream position afterwards (it does peeking).
>>> Sequence(Peek(Byte), Peek(Int16ub)).parse(b"\x01\x02") [1, 258]
Pure side effects¶
Seek makes a jump within the stream and leaves it at that point. It does not read or write anything to the stream by itself.
Sets a new stream position when parsing or building. Seeks are useful when many other fields follow the jump. Pointer works when there is only one field to look at, but when there is more to be done, Seek may come useful.
Pointer()wrapper that has same side effect but also processed a subcon.
- at – where to jump to, can ne an int or a context lambda
- whence – is the offset from beginning (0) or from current position (1) or from ending (2), can be an int or a context lambda, default is 0
>>> (Seek(5) >> Byte).parse(b"01234x") [5, 120] >>> (Bytes(10) >> Seek(5) >> Byte).build([b"0123456789", None, 255]) b'01234\xff6789'
Tell checks the current stream position and returns it, also putting it into the context. It does not read or write anything to the stream by itself.
Gets the stream position when parsing or building.
Tells are useful for adjusting relative offsets to absolute positions, or to measure sizes of Constructs. To get an absolute pointer, use a Tell plus a relative offset. To get a size, place two Tells and measure their difference using a Compute.
Better to use
CopyRaw()wrapper in almost any case.
>>> Struct("num"/VarInt, "offset"/Tell).build(dict(num=88)) b'X' >>> Struct("num"/VarInt, "offset"/Tell).parse(_) Container(num=88)(offset=1)
Caches bytes from the underlying stream, so it becomes seekable and tellable. Also makes the stream blocking, in case it came from a socket or a pipe. Optionally, stream can forget bytes that went a certain amount of bytes beyond the current offset, allowing only a limited seeking capability while allowing to process an endless stream.
Experimental implementation. May not be mature enough.
Parameters: subcon – the subcon which will operate on the buffered stream
Rebuffered(RepeatUntil(lambda obj,ctx: ?,Byte), tailcutoff=1024).parse_stream(endless_nonblocking_stream)
Restreamed(subcon, encoder, encoderunit, decoder, decoderunit, sizecomputer)¶
Transforms bytes between the underlying stream and the subcon.
When the parsing or building is done, the wrapper stream is closed. If read buffer or write buffer is not empty, error is raised.
Remember that subcon must consume or produce an amount of bytes that is a multiple of encoding or decoding units. For example, in a Bitwise context you should process a multiple of 8 bits or the stream will fail after parsing/building. Also do NOT use pointers inside.
- subcon – the subcon which will operate on the buffer
- encoder – a function that takes a b-string and returns a b-string (used when building)
- encoderunit – ratio as int, encoder takes that many bytes at once
- decoder – a function that takes a b-string and returns a b-string (used when parsing)
- decoderunit – ratio as int, decoder takes that many bytes at once
Bitwise is implemented as Restreamed(subcon, bits2bytes, 8, bytes2bits, 1, lambda n: n//8) Bytewise is implemented as Restreamed(subcon, bytes2bits, 1, bits2bytes, 8, lambda n: n*8)