Make every packaging script refuse a translator not built from the checkout's sources, early.

build_xlate.py --id-only wrote an nx2navi_xlate.id for whatever exe sat in the folder, from the sources as they are,
so make_launcher's guard (check_xlate, which compares only the .id) passed an old exe beside a fresh .id. Nothing
calls the option; it is gone, and only a build writes an .id. make_release.check_translator compared build/xlate's .id
with build_xlate.identity(), which hashes the tree the last build prepared there, so a patch or vendor change since
that build passed unseen; make_appimage.py and make_kit.py checked nothing. The fresh-prepare identity moves from
make_launcher into build_xlate as source_identity(), and build_xlate.check_built(folder, exe) refuses a missing exe or
.id and an .id that differs from it. make_launcher, make_release, make_appimage and make_kit stop on its reason, each
before its downloads, builds or deletions: make_launcher's check moves from build_app to the top of main, before the
--out folder is deleted and the toolchain lock is resolved. A translator folder built from the current sources, as the
release scripts pass it (work\xlate_6866a13e), passes as before.

Not reproduced before the change beyond the audit's probes. tests/xlate_package_check.py (new) checks a matching
folder, missing files, another identity and a stale build (refused although identity() over its own tree equals its
.id), that --id-only is refused, and that make_launcher, make_release and make_kit stop on a foreign translator, the
kit before any download; check_built passes on work\xlate_6866a13e.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
chrg6
2026-11-13 17:12:31 -05:00
parent 72047d12af
commit f3d8f08689
7 changed files with 213 additions and 41 deletions

View File

@@ -813,6 +813,12 @@ A change to the SM86 tensor lowering (`tools/xlate/AmpereLowering.cs` Hmma, Ldsm
`tests\maxwell_warp_emu.py`, a model of what the shader translator makes of Maxwell code (the docstring names the `tests\maxwell_warp_emu.py`, a model of what the shader translator makes of Maxwell code (the docstring names the
vector file and what is checked). `tests\maxwell_warp_emu_selftest.py` checks the emulator itself. No game files. vector file and what is checked). `tests\maxwell_warp_emu_selftest.py` checks the emulator itself. No game files.
A change to how the packaging scripts check the translator they ship (`tools/build_xlate.py` `source_identity`,
`check_built`; `make_launcher.py`, `make_release.py`, `make_appimage.py`, `make_kit.py`) passes
`python tests\xlate_package_check.py`: a folder whose `nx2navi_xlate.id` names this checkout's sources passes, a
missing file, another identity and a folder built before a patch changed are refused, `build_xlate.py` writes no `.id`
without a build, and each script stops before its slow steps. It needs git on PATH; no game files.
## Reverse engineering (optional) ## Reverse engineering (optional)
- Ghidra 12.1.2 with the Ghidra-Switch-Loader extension in `toolchain\ghidra\` (JDK 21) opens the game modules (NSO). - Ghidra 12.1.2 with the Ghidra-Switch-Loader extension in `toolchain\ghidra\` (JDK 21) opens the game modules (NSO).

View File

@@ -0,0 +1,143 @@
"""Check the shader translator guard of the packaging scripts (tools/build_xlate.py source_identity and check_built,
used by tools/make_launcher.py check_xlate, tools/make_release.py check_translator, tools/make_kit.py and
tools/make_appimage.py):
- a folder whose nx2navi_xlate.id names this checkout's sources passes; a missing exe or .id is refused;
- a folder built before a patch changed is refused, although identity() over its own prepared tree (what
make_release.py compared before) equals its .id;
- build_xlate.py has no option that writes an .id beside an exe it did not build (--id-only is gone);
- make_launcher.check_xlate, make_release.check_translator and make_kit stop on a foreign translator, make_kit before
it downloads or deletes anything; make_launcher and make_appimage run the check before their slow steps.
Needs git on PATH (prepare_sources applies the patches with it) and tools/vendor. No game files; any Python 3.10+.
Usage: xlate_package_check.py
"""
import contextlib
import inspect
import io
import shutil
import subprocess
import sys
import tempfile
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT))
sys.path.insert(0, str(ROOT / 'tools'))
import build_xlate # noqa: E402
EXE = 'nx2navi_xlate.exe'
def folder(base, name, ident, exe=True):
d = base / name
d.mkdir(parents=True)
if exe:
(d / EXE).write_bytes(b'MZ stand-in, not a translator')
if ident is not None:
(d / 'nx2navi_xlate.id').write_text(ident + '\n', encoding='ascii')
return d
def refused(fn, *args):
try:
with contextlib.redirect_stdout(io.StringIO()):
fn(*args)
except SystemExit as e:
return str(e.code)
return None
def main():
if not shutil.which('git'):
print('xlate package check: git is not on PATH; skipped')
return 0
with contextlib.redirect_stdout(io.StringIO()):
want = build_xlate.source_identity()
assert len(want) == 64 and build_xlate.BUILD == ROOT / 'build' / 'xlate', 'source_identity restores BUILD'
with tempfile.TemporaryDirectory(prefix='xlate-pkg-') as t:
base = Path(t)
good = folder(base, 'good', want)
no_exe = folder(base, 'no_exe', want, exe=False)
no_id = folder(base, 'no_id', None)
other = folder(base, 'other', '4d051299931d390e' + '0' * 48)
# A folder built before a patch changed: its prepared tree differs from a fresh one by one file, and its .id
# is identity() over that tree, as build_xlate.main wrote it then.
stale = base / 'stale'
old = build_xlate.BUILD
build_xlate.BUILD = stale
try:
with contextlib.redirect_stdout(io.StringIO()):
tree = build_xlate.prepare_sources()
victim = sorted(p for p in (tree / 'src').rglob('*.cs'))[0]
victim.unlink()
stale_id = build_xlate.identity()
finally:
build_xlate.BUILD = old
(stale / EXE).write_bytes(b'MZ stand-in, not a translator')
(stale / 'nx2navi_xlate.id').write_text(stale_id + '\n', encoding='ascii')
assert stale_id != want
def check(d):
with contextlib.redirect_stdout(io.StringIO()):
return build_xlate.check_built(d, EXE)
assert check(good) is None
assert 'missing' in check(no_exe)
assert 'missing' in check(no_id)
assert 'other sources' in check(other)
assert 'other sources' in check(stale)
print('check_built: this checkout\'s translator passes; missing files, another identity and a stale build '
'are refused')
# --id-only is gone: the option is refused and writes nothing.
bare = folder(base, 'bare', None)
r = subprocess.run([sys.executable, str(ROOT / 'tools' / 'build_xlate.py'), '--id-only', '--out', str(bare)],
capture_output=True, text=True)
assert r.returncode != 0 and not (bare / 'nx2navi_xlate.id').exists(), r.stdout + r.stderr
print('build_xlate.py: --id-only refused, no .id written')
# The packaging scripts, with the fresh identity computed once.
build_xlate.source_identity = lambda: want
import make_launcher
import make_release
import make_kit
assert refused(make_launcher.check_xlate, good) is None
for bad in (other, stale, no_exe):
assert refused(make_launcher.check_xlate, bad)
main_root = make_release.MAIN_ROOT
try:
for d, ok in ((good, True), (stale, False), (other, False)):
rel = base / f'rel_{d.name}'
shutil.copytree(d, rel / 'build' / 'xlate')
make_release.MAIN_ROOT = rel
assert (refused(make_release.check_translator) is None) == ok, d.name
finally:
make_release.MAIN_ROOT = main_root
out = base / 'kit_out'
argv = sys.argv
try:
sys.argv = ['make_kit.py', '--xlate', str(stale), '--out', str(out)]
why = refused(make_kit.main)
finally:
sys.argv = argv
assert why and 'other sources' in why and not out.exists()
print('make_launcher, make_release and make_kit stop on a foreign translator (make_kit before any download)')
# The check comes before the slow steps.
src = inspect.getsource(make_launcher.main)
assert src.index('check_xlate(a.xlate)') < src.index('shutil.rmtree(out') < src.index('toolchain.make_lock')
import make_appimage
src = inspect.getsource(make_appimage.main)
assert src.index('build_xlate.check_built(') < src.index('dlss_pin.resolve(') < src.index('shutil.rmtree(app')
src = inspect.getsource(make_release.main)
assert src.index('check_translator()') < src.index('shutil.rmtree(out)')
print('xlate package check: passed')
return 0
if __name__ == '__main__':
sys.exit(main())

View File

@@ -7,10 +7,11 @@ NativeAOT links with the system's clang (no MSVC); NativeAOT builds for the OS i
Next to the exe it writes nx2navi_xlate.id: the translator identity, a sha256 over everything the build reads (this script, Next to the exe it writes nx2navi_xlate.id: the translator identity, a sha256 over everything the build reads (this script,
Program.cs, the csproj, the patches, the vendor tree). runtime/gfx_shader.c puts it into every shader cache key, so a Program.cs, the csproj, the patches, the vendor tree). runtime/gfx_shader.c puts it into every shader cache key, so a
translator change re-keys the cache without a hand-made KEY_VERSION bump. --id-only writes it for the exe already built translator change re-keys the cache without a hand-made KEY_VERSION bump. Only a build writes it: the packaging scripts
(the caller vouches that the exe matches the sources). (make_launcher.py, make_release.py, make_appimage.py, make_kit.py) take a translator whose .id names this checkout's
sources for one built from them (check_built), so no option writes an .id beside an exe it did not build.
python tools/build_xlate.py [--id-only] [--out DIR] python tools/build_xlate.py [--out DIR]
--out builds into DIR instead of build/xlate (a task build dir: nx2navi.exe takes the translator next to it). --out builds into DIR instead of build/xlate (a task build dir: nx2navi.exe takes the translator next to it).
""" """
@@ -202,9 +203,43 @@ def write_identity(exe: Path) -> None:
print(f"translator identity {ident[:16]} -> {exe.with_suffix('.id')}") print(f"translator identity {ident[:16]} -> {exe.with_suffix('.id')}")
def source_identity() -> str:
"""The identity a translator built from this checkout's sources gets: the vendor tree copied and patched in a
temporary folder (prepare_sources) and hashed as the build hashes it (identity). identity() over a build folder
hashes the tree its last build prepared, so a patch or vendor change made since then would not show. Nothing is
compiled; it takes a few seconds and needs git on PATH (prepare_sources applies the patches with it)."""
import tempfile
global BUILD
old = BUILD
with tempfile.TemporaryDirectory(prefix="xlate-id-") as d:
BUILD = Path(d) / "xlate"
try:
prepare_sources()
return identity()
finally:
BUILD = old
def check_built(folder: Path, exe: str = EXE) -> str | None:
"""None when folder holds the translator exe and an nx2navi_xlate.id naming this checkout's sources
(source_identity); otherwise why not. The packaging scripts stop on the reason: 0.10.0 and 0.10.1 shipped the
0.9.2 translator (identity 4d051299) after tools/xlate/Program.cs had raised the texture pool to 40960 slots, so
their translations still clamped every pool handle to slot 32767 and the ground went see-through after some area
loads. exe: the file name on the release's OS (EXE is this machine's)."""
folder = Path(folder)
for name in (exe, "nx2navi_xlate.id"):
if not (folder / name).is_file():
return f"{folder / name} is missing: build the shader translator with tools/build_xlate.py --out {folder}"
have = (folder / "nx2navi_xlate.id").read_text(encoding="ascii", errors="replace").strip()
want = source_identity()
if have != want:
return (f"{folder}: its translator was built from other sources (identity {have[:16]}) than this checkout's "
f"({want[:16]}): build it again with tools/build_xlate.py --out {folder}")
return None
def main() -> None: def main() -> None:
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument("--id-only", action="store_true", help="write nx2navi_xlate.id for the exe already built")
parser.add_argument("--out", type=Path, help="build directory instead of build/xlate") parser.add_argument("--out", type=Path, help="build directory instead of build/xlate")
parser.add_argument("--llvm", type=Path, help="Linux: an LLVM directory whose bin/clang and lld link the exe") parser.add_argument("--llvm", type=Path, help="Linux: an LLVM directory whose bin/clang and lld link the exe")
parser.add_argument("--sysroot", type=Path, help="Linux: link against this sysroot's glibc instead of the host's " parser.add_argument("--sysroot", type=Path, help="Linux: link against this sysroot's glibc instead of the host's "
@@ -213,9 +248,6 @@ def main() -> None:
if a.out: if a.out:
global BUILD global BUILD
BUILD = a.out.resolve() BUILD = a.out.resolve()
if a.id_only:
write_identity(BUILD / EXE)
return
started = time.time() started = time.time()
dotnet = shutil.which("dotnet") dotnet = shutil.which("dotnet")
if not dotnet: if not dotnet:

View File

@@ -7,7 +7,7 @@ with --events and reads its progress lines).
[--dlss libnvidia-ngx-dlss.so.<version> | --no-dlss] [--dlss libnvidia-ngx-dlss.so.<version> | --no-dlss]
The AppImage holds no game files: the kit's tools/, recomp/, runtime/ and docs, the shader translator built by The AppImage holds no game files: the kit's tools/, recomp/, runtime/ and docs, the shader translator built by
tools/build_xlate.py (bin/nx2navi_xlate), NVIDIA's DLSS library with its notice (bin/, tools/dlss_pin.py: downloaded tools/build_xlate.py from this checkout's sources (bin/nx2navi_xlate; the script stops first on another, check_built), NVIDIA's DLSS library with its notice (bin/, tools/dlss_pin.py: downloaded
from NVIDIA's DLSS SDK repository and checked, or --dlss; not with --no-dlss: an AppImage made without the library from NVIDIA's DLSS SDK repository and checked, or --dlss; not with --no-dlss: an AppImage made without the library
downloads it after a build on machines with NVIDIA's driver, AppRun's get_dlss), the pinned Vulkan Memory Allocator downloads it after a build on machines with NVIDIA's driver, AppRun's get_dlss), the pinned Vulkan Memory Allocator
header, the pinned Vulkan headers (toolchain/vulkan-headers, tools/vulkan_pin.py: the runtime names Vulkan extensions header, the pinned Vulkan headers (toolchain/vulkan-headers, tools/vulkan_pin.py: the runtime names Vulkan extensions
@@ -201,6 +201,10 @@ def main():
if not need.exists(): if not need.exists():
sys.exit(f'{need} is missing (tools/build_xlate.py builds the translator; tools/build_game.py fetches VMA)') sys.exit(f'{need} is missing (tools/build_xlate.py builds the translator; tools/build_game.py fetches VMA)')
sys.path.insert(0, str(ROOT / 'tools')) sys.path.insert(0, str(ROOT / 'tools'))
import build_xlate
reason = build_xlate.check_built(xlate, 'nx2navi_xlate') # the translator of this checkout's sources, or none
if reason:
sys.exit(reason)
import dlss_pin import dlss_pin
dlss = None if a.no_dlss else dlss_pin.resolve(a.dlss, ROOT / 'build' / 'dl', linux=True) dlss = None if a.no_dlss else dlss_pin.resolve(a.dlss, ROOT / 'build' / 'dl', linux=True)

View File

@@ -13,7 +13,8 @@ or --dlss; not with --no-dlss), DLSS Frame Generation in bin/dlssg/ with --dlssg
tools/build_dlssg.py beside NVIDIA Streamline's runtime, tools/streamline_pin.py: downloaded from NVIDIA's Streamline tools/build_dlssg.py beside NVIDIA Streamline's runtime, tools/streamline_pin.py: downloaded from NVIDIA's Streamline
SDK release and checked, or --streamline), AMD's FSR 3.1 library with --ffx, its two FSR 4 libraries with --ffx4, the SDK release and checked, or --streamline), AMD's FSR 3.1 library with --ffx, its two FSR 4 libraries with --ffx4, the
button art with --prompt-art, and README-BuildKit.md; no game code, data or anything generated from the game. button art with --prompt-art, and README-BuildKit.md; no game code, data or anything generated from the game.
check_content runs over the folder before it is zipped. check_content runs over the folder before it is zipped. The --xlate folder must hold a translator built from this
checkout's sources (tools/build_xlate.py check_built); the script stops first if not.
--prompt-art names folders that hold XGamepad, P5Gamepad and Keyboard_Mouse, or those three folders themselves: the --prompt-art names folders that hold XGamepad, P5Gamepad and Keyboard_Mouse, or those three folders themselves: the
pictures tools/pad_prompts.py reads go into the kit's prompt-art folder (tools/prompt_art.py art_files), and pictures tools/pad_prompts.py reads go into the kit's prompt-art folder (tools/prompt_art.py art_files), and
@@ -73,11 +74,15 @@ def main():
a = ap.parse_args() a = ap.parse_args()
sys.path.insert(0, str(ROOT / 'tools')) sys.path.insert(0, str(ROOT / 'tools'))
import build_shaders import build_shaders
import build_xlate
import dlss_pin import dlss_pin
import prompt_art import prompt_art
import streamline_pin import streamline_pin
if a.streamline and not a.dlssg: if a.streamline and not a.dlssg:
sys.exit('--streamline goes with --dlssg') sys.exit('--streamline goes with --dlssg')
reason = build_xlate.check_built(a.xlate, 'nx2navi_xlate.exe') # the translator of this checkout's sources
if reason:
sys.exit(reason)
if a.prompt_art: if a.prompt_art:
prompt_art.art_files(a.prompt_art) # the art is all there before anything is downloaded prompt_art.art_files(a.prompt_art) # the art is all there before anything is downloaded
dlss = None if a.no_dlss else dlss_pin.resolve(a.dlss, ROOT / 'build' / 'dl') dlss = None if a.no_dlss else dlss_pin.resolve(a.dlss, ROOT / 'build' / 'dl')

View File

@@ -188,32 +188,17 @@ def build_python(py):
# ---------------------------------------------------------------- app/<version>/ # ---------------------------------------------------------------- app/<version>/
def xlate_source_identity():
"""The identity tools/build_xlate.py writes for a translator built from this checkout: the vendor tree copied and
patched in a folder of its own (build_xlate.prepare_sources) and hashed as the build hashes it (build_xlate.identity).
Nothing is compiled; it takes a few seconds."""
import tempfile
import build_xlate
old = build_xlate.BUILD
with tempfile.TemporaryDirectory(prefix='xlate-id-') as d:
build_xlate.BUILD = Path(d) / 'xlate'
try:
build_xlate.prepare_sources()
return build_xlate.identity()
finally:
build_xlate.BUILD = old
def check_xlate(xlate): def check_xlate(xlate):
"""A release carries the translator of its own sources. 0.10.0 and 0.10.1 shipped the 0.9.2 translator (identity """A release carries the translator of its own sources. 0.10.0 and 0.10.1 shipped the 0.9.2 translator (identity
4d051299) after tools/xlate/Program.cs had raised the texture pool to 40960 slots, so their translations still 4d051299) after tools/xlate/Program.cs had raised the texture pool to 40960 slots, so their translations still
clamped every pool handle to slot 32767 and the ground went see-through after some area loads (issues #569 and clamped every pool handle to slot 32767 and the ground went see-through after some area loads (issues #569 and
#824). A translator whose nx2navi_xlate.id differs from this checkout's sources stops the build.""" #824). A missing translator, or one whose nx2navi_xlate.id differs from this checkout's sources
(build_xlate.check_built), stops the build; main runs this before anything is resolved, downloaded or deleted."""
import build_xlate
reason = build_xlate.check_built(xlate, 'nx2navi_xlate.exe')
if reason:
sys.exit(reason)
have = (xlate / 'nx2navi_xlate.id').read_text(encoding='ascii').strip() have = (xlate / 'nx2navi_xlate.id').read_text(encoding='ascii').strip()
want = xlate_source_identity()
if have != want:
sys.exit(f'{xlate}: its translator was built from other sources (identity {have[:16]}) than this checkout\'s '
f'({want[:16]}): build it again with tools/build_xlate.py --out {xlate}')
print(f'translator identity {have[:16]}: built from this checkout\'s sources') print(f'translator identity {have[:16]}: built from this checkout\'s sources')
@@ -233,11 +218,7 @@ def build_app(app, ver, xlate, feed_repo, lock, ffx=None, dlss=None, art=None, d
shutil.copy2(ROOT / t, app / t) shutil.copy2(ROOT / t, app / t)
(app / 'VERSION').write_text(ver + '\n', encoding='utf-8') (app / 'VERSION').write_text(ver + '\n', encoding='utf-8')
(app / 'bin').mkdir() (app / 'bin').mkdir()
for n in ('nx2navi_xlate.exe', 'nx2navi_xlate.id'): for n in ('nx2navi_xlate.exe', 'nx2navi_xlate.id'): # main checked them first (check_xlate)
if not (xlate / n).is_file():
sys.exit(f'{xlate / n} is missing: build the shader translator first (tools/build_xlate.py)')
check_xlate(xlate)
for n in ('nx2navi_xlate.exe', 'nx2navi_xlate.id'):
shutil.copy2(xlate / n, app / 'bin' / n) shutil.copy2(xlate / n, app / 'bin' / n)
if ffx: # FSR 3.1 loads it from beside the game (runtime/vk_ffx.cpp); make_game.py copies it there if ffx: # FSR 3.1 loads it from beside the game (runtime/vk_ffx.cpp); make_game.py copies it there
shutil.copy2(ffx, app / 'bin' / 'amd_fidelityfx_vk.dll') shutil.copy2(ffx, app / 'bin' / 'amd_fidelityfx_vk.dll')
@@ -415,6 +396,7 @@ def main():
prompt_art.art_files(a.prompt_art) # the art is all there before anything is downloaded or built prompt_art.art_files(a.prompt_art) # the art is all there before anything is downloaded or built
if a.pgo: if a.pgo:
pgo.fitting(a.pgo) # the profile fits this checkout's recomp/ before anything is downloaded or built pgo.fitting(a.pgo) # the profile fits this checkout's recomp/ before anything is downloaded or built
check_xlate(a.xlate) # the translator is this checkout's before anything is downloaded, built or deleted
out = a.out out = a.out
shutil.rmtree(out, ignore_errors=True) shutil.rmtree(out, ignore_errors=True)
folder = out / 'NX2navi' folder = out / 'NX2navi'

View File

@@ -306,12 +306,12 @@ def build_llvm(out):
def check_translator(): def check_translator():
"""Stop on a translator older than the sources. build/xlate is built by hand and only copied here (the Linux release """Stop on a translator older than the sources. build/xlate is built by hand and only copied here (the Linux release
builds its own), so without a rebuild a translator change reaches no Windows release, and nothing fails: the runtime builds its own), so without a rebuild a translator change reaches no Windows release, and nothing fails: the runtime
sends job fields the old exe ignores, as happened to gather_bias.""" sends job fields the old exe ignores, as happened to gather_bias. The identity it must have is that of this
idf = MAIN_ROOT / 'build' / 'xlate' / 'nx2navi_xlate.id' checkout's sources prepared afresh (build_xlate.check_built), not build_xlate.identity() over build/xlate, which
have = idf.read_text(encoding='ascii').strip() if idf.is_file() else 'none' hashes the tree its last build prepared and so matches a translator built before a patch changed."""
want = build_xlate.identity() reason = build_xlate.check_built(MAIN_ROOT / 'build' / 'xlate', 'nx2navi_xlate.exe')
if have != want: if reason:
sys.exit(f'build/xlate has translator {have[:16]}, the sources are {want[:16]}: run tools/build_xlate.py') sys.exit(reason)
def build_bin(out): def build_bin(out):