mirror of
https://github.com/tangerineturtle1/NX2navi.git
synced 2026-11-18 13:09:57 -04:00
Read the game files in 64 KiB pieces when checking them, and keep the check's key.
romfs_check.py gave the decompressor 4 MiB of a file at a time. Each call returns its output as one bytes object grown by copying, under the interpreter's lock: on 32 MiB frames that ran at about 120 MB/s of packed data on twelve threads, and at about 2,500 MB/s with 64 KiB pieces; on a 1 MiB file at 490 against 3,000. Files smaller than a piece are as fast as before. The check now stops at the frame's end, so what it says of a file does not depend on the piece size; a test compares both sizes and the check as it was on damaged, cut and whole frames. The scan step's key held the sha256 of the script's bytes, so this edit would have made every install check its 26 GB again. The key now holds a constant changed by hand, whose value is the sha256 the script had in 0.7.0 and 0.7.1: the key of every install that passed stays what it is. A test fails when the script's logic changes, to ask whether the constant must change. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
@@ -35,7 +35,12 @@ import zstandard
|
||||
STANDIN_MOD = 'Damaged file stand-ins'
|
||||
DICTIONARIES = 'Pack/ZsDic.pack.zs'
|
||||
ZSTD_MAGIC = b'\x28\xb5\x2f\xfd'
|
||||
PIECE = 1 << 22
|
||||
# How much of a file the decompressor is given at a time. Each call returns its output as one bytes object, which
|
||||
# the library grows as the output comes: with 4 MiB of input a call's output is tens of MiB, grown by copying, and
|
||||
# the copying holds the interpreter's lock, so the worker threads wait on each other. Measured on 32 MiB frames
|
||||
# with 12 threads: about 120 MB/s of packed data with 4 MiB pieces, about 2,500 MB/s with 64 KiB pieces; files
|
||||
# smaller than a piece are the same either way.
|
||||
PIECE = 1 << 16
|
||||
TEXTURE = re.compile(r'Tex/([^/]+)\.bntx\.zs')
|
||||
|
||||
|
||||
@@ -110,6 +115,8 @@ def problem(image, entry):
|
||||
size = 0
|
||||
for at in range(0, len(data), PIECE):
|
||||
size += len(out.decompress(data[at:at + PIECE]))
|
||||
if out.eof: # the frame is whole: bytes after it are not the frame's, whichever piece they fall in
|
||||
break
|
||||
except zstandard.ZstdError as e:
|
||||
return f'does not decompress ({e})'
|
||||
if params.content_size and size != params.content_size:
|
||||
|
||||
Reference in New Issue
Block a user