Unpack the texture archive in pieces when drawing the button prompts.

pad_prompts.py gave each packed game file to the decompressor in one
call, which returns one bytes object grown by copying. On made-up data
of the size of the interface texture archive, 69 MB packed and 244 MB
unpacked, that takes 21 s on a fast PC; players' builds waited 44 to 52 s
before the step's first folder, most of its minute. In 64 KiB pieces
joined once the same bytes take 0.3 s.

The prompts step's key held the digest of the three scripts' bytes, so
this edit would have drawn every install's prompts again, as two of the
last four releases did. It now holds a constant changed by hand, whose
value is the scripts' digest in 0.7.0 and 0.7.1, and a test fails when
the scripts' logic changes, to ask whether the constant must change.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
chrg6
2026-11-10 15:59:45 -05:00
parent e678e1f38e
commit 62a02da381
4 changed files with 150 additions and 9 deletions

View File

@@ -205,7 +205,7 @@ class Romfs:
def dictionary(self, dict_id):
if self.dicts is None:
self.dicts = {}
pack = zstandard.ZstdDecompressor().decompressobj().decompress(self.raw(DICTIONARIES))
pack = unpack(self.raw(DICTIONARIES))
for lo, hi in sarc_members(pack).values():
d = zstandard.ZstdCompressionDict(pack[lo:hi])
self.dicts[d.dict_id()] = d
@@ -218,7 +218,27 @@ class Romfs:
return raw, None
dict_id = zstandard.get_frame_parameters(raw).dict_id
d = self.dictionary(dict_id) if dict_id else None
return zstandard.ZstdDecompressor(dict_data=d).decompressobj().decompress(raw), d
return unpack(raw, d), d
# How much of a packed file unpack gives the decompressor at a time.
PIECE = 1 << 16
def unpack(raw, dictionary=None):
"""The contents of a zstd frame, with the dictionary it was packed with; bytes after the frame are left out.
The file goes to the decompressor in pieces of PIECE bytes and the pieces' outputs are joined once. Given the
whole file in one call, the library returns one bytes object that it grows by copying as the output comes: made-up
data of the size of the interface texture archive, 69 MB packed and 244 MB unpacked, took 21 s that way on a
fast PC against 0.3 s in pieces. Players' builds waited 44 to 52 s before this tool's first folder, most of
its minute."""
out, parts = zstandard.ZstdDecompressor(dict_data=dictionary).decompressobj(), []
for at in range(0, len(raw), PIECE):
parts.append(out.decompress(raw[at:at + PIECE]))
if out.eof:
break
return b''.join(parts)
def sarc_members(data):