Skip to content

Commit 9287c4e

Browse files
authored
feat: Struct-level Alignment (AlignTo)
Merge pull request #69 from MatrixEditor/feat/struct-alignment
2 parents 7623d4c + 96e8d52 commit 9287c4e

12 files changed

Lines changed: 720 additions & 121 deletions

File tree

‎docs/sphinx/source/library/fields/common.rst‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -331,6 +331,18 @@ Special Structs
331331
332332
.. versionadded:: 2.4.0
333333
334+
.. autoclass:: caterpillar.fields.AlignTo
335+
:members:
336+
337+
A trailing, self-contained alignment spec used via the ``align_to=``
338+
keyword on :func:`~caterpillar.py.struct`, :func:`~caterpillar.py.union`
339+
and :func:`~caterpillar.py.bitfield`. Unlike :class:`Aligned`, which pads
340+
relative to the *absolute* stream position, ``align_to=`` pads the model
341+
itself so its own packed size is always a multiple of the alignment -
342+
regardless of where it is embedded. See :ref:`tutorial-align_to`.
343+
344+
.. versionadded:: 2.10.0
345+
334346
.. autoclass:: caterpillar.fields.Computed
335347
:members:
336348

‎docs/sphinx/source/library/model/bitfield.rst‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,11 @@ Main Interface
8989
.. versionchanged:: 2.5.0
9090
Added the ``alignment`` parameter.
9191

92+
.. versionchanged:: 2.10.0
93+
Added the ``align_to`` parameter for whole-bitfield trailing byte
94+
alignment (distinct from the bit-group ``alignment`` parameter
95+
above). See :ref:`tutorial-align_to`.
96+
9297
Default Factory Classes
9398
-----------------------
9499

‎docs/sphinx/source/library/model/struct.rst‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,12 @@ Standard Interface
2929

3030
.. autofunction:: caterpillar.model.struct
3131

32+
.. versionchanged:: 2.10.0
33+
Added the ``align_to`` parameter (also accepted by :func:`union`
34+
above) for whole-model trailing byte alignment. See
35+
:ref:`tutorial-align_to`.
36+
37+
3238
.. autofunction:: caterpillar.model.pack
3339

3440
.. autofunction:: caterpillar.model.pack_into

‎docs/sphinx/source/tutorial/basics/padding.rst‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,3 +16,71 @@ length using the `padding` keyword. This is useful when you need to ensure
1616
that certain fields are aligned or when the structure requires reserved spaces.
1717

1818
>>> field = padding[10] # greedy or dynamic size
19+
20+
21+
.. _tutorial-align_to:
22+
23+
Whole-model alignment
24+
----------------------
25+
26+
.. versionadded:: 2.10.0
27+
28+
The :code:`padding` field above reserves space explicitly, as an extra field
29+
of its own. Sometimes what you actually want is different: a whole
30+
:code:`@struct`, :code:`@union` or :code:`@bitfield` class whose *total*
31+
packed size is always rounded up to a multiple of some alignment.
32+
Pass :code:`align_to=` to the decorator instead:
33+
34+
>>> @struct(align_to=4)
35+
... class Format:
36+
... a: uint8
37+
...
38+
>>> sizeof(Format)
39+
4
40+
>>> unpack(Format, b"\x01\x00\x00\x00")
41+
Format(a=1)
42+
43+
This is deliberately different from :class:`~caterpillar.fields.Aligned` and
44+
:func:`~caterpillar.fields.align`, which both pad relative to the *absolute*
45+
stream position. :code:`align_to=` instead measures *this model's own* start
46+
and end, so the padding only ever depends on the model's own content - not on
47+
where it happens to be embedded:
48+
49+
>>> @struct
50+
... class Outer:
51+
... b: uint8
52+
... inner: Format
53+
... c: uint8
54+
...
55+
>>> pack(Outer(b=1, inner=Format(a=2), c=3))
56+
b'\x01\x02\x00\x00\x00\x03'
57+
58+
A bare :code:`int` or context lambda is accepted directly, as shown above.
59+
Use :class:`~caterpillar.fields.AlignTo` explicitly to customize the fill
60+
byte (or a multi-byte fill pattern) and/or relax verification:
61+
62+
>>> @struct(align_to=AlignTo(4, fill=0xFF, strict=False))
63+
... class Lenient:
64+
... a: uint8
65+
66+
With the default :code:`strict=True`, unpacking verifies that the padding
67+
bytes actually match :code:`fill` and raises :code:`ValueError` if they
68+
don't. With :code:`strict=False`, padding bytes are consumed without
69+
verification.
70+
71+
:code:`align_to=` also works on :code:`@union` (padding after the largest
72+
member) and :code:`@bitfield` (padding after all bit-groups have been
73+
finalized to whole bytes), and applies per-element when the model is used
74+
inside an array, so each element keeps its own alignment:
75+
76+
>>> @struct
77+
... class Many:
78+
... items: Format[2]
79+
...
80+
>>> pack(Many(items=[Format(a=1), Format(a=2)]))
81+
b'\x01\x00\x00\x00\x02\x00\x00\x00'
82+
83+
A dynamic (context-dependent) alignment value is supported too, but makes the
84+
model's size undeterminable ahead of time - :func:`~caterpillar.model.sizeof`
85+
raises :class:`~caterpillar.exception.DynamicSizeError` in that case.
86+

‎src/caterpillar/fields/__init__.py‎

Lines changed: 74 additions & 60 deletions
Original file line numberDiff line numberDiff line change
@@ -12,11 +12,19 @@
1212
#
1313
# You should have received a copy of the GNU General Public License
1414
# along with this program. If not, see <https://www.gnu.org/licenses/>.
15-
from ._base import DEFAULT_OPTION, INVALID_DEFAULT, Field, singleton
15+
from ._base import (
16+
DEFAULT_OPTION,
17+
IGNORED_DEFAULT,
18+
INVALID_DEFAULT,
19+
Field,
20+
has_default,
21+
singleton,
22+
)
1623
from ._mixin import Chain, FieldMixin, FieldStruct, Operator, get_args, get_kwargs
1724
from .common import (
1825
ENUM_STRICT,
1926
Aligned,
27+
AlignTo,
2028
AsLengthRef,
2129
Bytes,
2230
Computed,
@@ -144,147 +152,153 @@
144152
uintptr,
145153
uintptr_fn,
146154
)
155+
from .unnamed import Unnamed, sized
147156
from .varint import VARINT_LSB, VarInt, vint
148157

149158
__all__ = [
159+
"CTX_DIGEST",
160+
"CTX_DIGEST_ALGO",
161+
"CTX_DIGEST_HOOK",
162+
"CTX_DIGEST_OBJ",
163+
"DEFAULT_OPTION",
164+
"ENUM_STRICT",
165+
"HMAC",
166+
"IGNORED_DEFAULT",
167+
"INVALID_DEFAULT",
168+
"MAC",
169+
"PTR_STRICT",
170+
"VARINT_LSB",
171+
"Adler",
150172
"Adler_Algo",
151173
"Adler_Field",
152-
"Adler",
153174
"Algorithm",
154-
"align",
175+
"AlignTo",
155176
"Aligned",
156177
"And",
157178
"AsLengthRef",
158-
"boolean",
159179
"Branch",
160180
"Bytes",
161181
"Bz2Compressed",
182+
"CString",
162183
"Chain",
163-
"char",
164184
"Compressed",
165185
"Computed",
166186
"ConditionalChain",
167187
"Const",
168188
"ConstBytes",
169189
"ConstString",
190+
"Crc32",
170191
"Crc32_Algo",
171192
"Crc32_Field",
172-
"Crc32",
173-
"CString",
174-
"CTX_DIGEST_ALGO",
175-
"CTX_DIGEST_HOOK",
176-
"CTX_DIGEST_OBJ",
177-
"CTX_DIGEST",
178-
"DEFAULT_OPTION",
179193
"Digest",
180194
"DigestField",
181195
"DigestFieldAction",
182-
"double",
183196
"Else",
184197
"ElseIf",
185198
"Encrypted",
186199
"End",
187-
"ENUM_STRICT",
188200
"Enum",
189201
"Field",
190202
"FieldMixin",
191203
"FieldStruct",
192-
"float16",
193-
"float32",
194-
"float64",
195-
"get_args",
196-
"get_kwargs",
197-
"HMAC",
198204
"HMACAlgorithm",
199-
"If",
200-
"Int",
201-
"int16",
202-
"int24",
203-
"int32",
204-
"int64",
205-
"int8",
206-
"intptr_fn",
207-
"intptr",
208-
"INVALID_DEFAULT",
209205
"IOHook",
210206
"IPv4Address",
211207
"IPv6Address",
208+
"If",
209+
"Int",
212210
"KeyCipher",
213-
"Lazy",
214211
"LZMACompressed",
215212
"LZOCompressed",
216-
"MAC",
213+
"Lazy",
217214
"MACAddress",
215+
"Md5",
218216
"Md5_Algo",
219217
"Md5_Field",
220-
"Md5",
221218
"Memory",
222-
"offintptr",
223-
"offuintptr",
224219
"Operator",
225220
"Or",
226221
"Otherwise",
227222
"Padded",
228-
"padding",
229223
"Padding",
230224
"Pass",
231-
"pointer",
232225
"Pointer",
233226
"PostPad",
234-
"Prefixed",
235227
"PrePad",
236-
"psize",
237-
"pssize",
238-
"PTR_STRICT",
228+
"Prefixed",
239229
"PyStructFormattedField",
240-
"relative_pointer",
241230
"RelativePointer",
231+
"Sha1",
242232
"Sha1_Algo",
243233
"Sha1_Field",
244-
"Sha1",
234+
"Sha2_224",
245235
"Sha2_224_Algo",
246236
"Sha2_224_Field",
247-
"Sha2_224",
237+
"Sha2_256",
248238
"Sha2_256_Algo",
249239
"Sha2_256_Field",
250-
"Sha2_256",
240+
"Sha2_384",
251241
"Sha2_384_Algo",
252242
"Sha2_384_Field",
253-
"Sha2_384",
243+
"Sha2_512",
254244
"Sha2_512_Algo",
255245
"Sha2_512_Field",
256-
"Sha2_512",
246+
"Sha3_224",
257247
"Sha3_224_Algo",
258248
"Sha3_224_Field",
259-
"Sha3_224",
249+
"Sha3_256",
260250
"Sha3_256_Algo",
261251
"Sha3_256_Field",
262-
"Sha3_256",
252+
"Sha3_384",
263253
"Sha3_384_Algo",
264254
"Sha3_384_Field",
265-
"Sha3_384",
255+
"Sha3_512",
266256
"Sha3_512_Algo",
267257
"Sha3_512_Field",
268-
"Sha3_512",
269-
"singleton",
270258
"Start",
271259
"String",
272260
"Timestamp",
273261
"Transformer",
274262
"UInt",
263+
"Unnamed",
264+
"Uuid",
265+
"VarInt",
266+
"When",
267+
"Xor",
268+
"ZLibCompressed",
269+
"align",
270+
"boolean",
271+
"char",
272+
"double",
273+
"float16",
274+
"float32",
275+
"float64",
276+
"get_args",
277+
"get_kwargs",
278+
"has_default",
279+
"int8",
280+
"int16",
281+
"int24",
282+
"int32",
283+
"int64",
284+
"intptr",
285+
"intptr_fn",
286+
"offintptr",
287+
"offuintptr",
288+
"padding",
289+
"pointer",
290+
"psize",
291+
"pssize",
292+
"relative_pointer",
293+
"singleton",
294+
"sized",
295+
"uint8",
275296
"uint16",
276297
"uint24",
277298
"uint32",
278299
"uint64",
279-
"uint8",
280-
"uintptr_fn",
281300
"uintptr",
282-
"Uuid",
283-
"VARINT_LSB",
284-
"VarInt",
301+
"uintptr_fn",
285302
"vint",
286303
"void_ptr",
287-
"When",
288-
"Xor",
289-
"ZLibCompressed",
290304
]

0 commit comments

Comments
 (0)