Add a Mods screen to the launcher and build the mods' code patches into the launcher's game.

Mods, on the home screen, opens the game's mods folder, adds a downloaded mod from its .zip, .7z or .rar file or its
folder, switches each mod on or off, and sets the load order (mods_disabled and mods_order in nx2navi.ini, written as
the game writes them). Each mod's description says what it changes, which other mod changes the same files and which
of the two wins, and why the game leaves it off when it does.

Before the game starts, Play puts the mods folder in order (launcher/mods.py prepare): an archive in it is unpacked into
a folder of its name and kept, a mod whose files sit a folder or more down has them moved up, and a romfs, exefs or
cheats folder put straight into the mods folder becomes a mod of its own. An archive that would write outside its
folder is refused, and nothing is deleted.

The launcher's build (tools/make_game.py) left the mods' code cheats and ExeFS patch files out, so those mods stayed
off. The lift now turns them into patch sets first, as the setup does, with the files in the lift's key and
build-cache/mod-code.json recording what it took up. Play builds the game again before it starts when an enabled mod
brings code no build has seen. The shaders' key names the lift and the compile without the mods' code, so such a build
compiles only what the patches change and leaves the shaders prepared. tools/mod_files.py is the one place the tools,
the build and the launcher find a mod's parts.

Tested: a zipped mod holding its own folder, put into a mods folder, was unpacked and moved up by the launcher's code,
and the game served its file in place of the game's. The lifter took a code cheat and a pchtxt patch made for the game's
main as patch sets. The launcher's and the setup's tests pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
epicmeme546
2026-11-12 08:11:57 -05:00
parent 432b4e4b1c
commit 89c453d7bb
15 changed files with 1234 additions and 85 deletions

View File

@@ -30,7 +30,7 @@ hash: the same over the cheat's words, so the runtime sees an edited cheat that
A code cheat that calls or jumps into a code cave it writes (words where main has zeros) gets the cave's body inlined
at the call site; a conditional skip of one store in the body (cbz wN, +8; str wN, ...) is dropped.
Standard library and recomp/ only.
Standard library, recomp/ and tools/mod_files.py only.
"""
import argparse
import struct
@@ -38,9 +38,11 @@ import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'recomp'))
sys.path.insert(0, str(Path(__file__).resolve().parent))
import flags # noqa: E402
from lifter import Lifter, Unsupported # noqa: E402
from nso import Nso # noqa: E402
from mod_files import cheat_files # noqa: E402
BASE = 0x7000000000 # main's base in the layout (recomp/layout.py); only pc-relative decoding sees it
NOP = 0xD503201F
@@ -90,25 +92,6 @@ def parse(text, mod):
return out
TITLE = '0400486024DEC000' # the game's title id: a mod's Atmosphere folders are named after it
def cheat_files(mods, bid):
"""[(mod folder name, cheat file)] of the folders of mods, by name: <mod>/cheats/<bid>.txt or its Atmosphere places
(runtime/mods.c find_root), the file name in any case."""
out = []
for mod in sorted((d for d in mods.iterdir() if d.is_dir()), key=lambda d: d.name) if mods.is_dir() else []:
for where in ('cheats', f'atmosphere/contents/{TITLE}/cheats', f'contents/{TITLE}/cheats', f'{TITLE}/cheats'):
folder = mod / where
if folder.is_dir():
f = next((p for p in sorted(folder.iterdir()) if p.is_file() and p.name.lower() == f'{bid}.txt'.lower()),
None)
if f:
out.append((mod.name, f))
break
return out
def ops(words):
"""[(type, [words])] of a cheat; type is the opcode nibble, 0xC0.. for the extended ones. ValueError when the last
opcode is cut short."""

View File

@@ -29,7 +29,7 @@ comma separated, or "-": the runtime switches them off while the mod is on, so t
A file the rules cannot express is unsupported as a whole: half of a patch is worse than none.
Standard library, recomp/ and tools/cheats.py only.
Standard library, recomp/, tools/cheats.py and tools/mod_files.py only.
"""
import argparse
import re
@@ -38,29 +38,10 @@ import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
from cheats import TITLE, Main, convert, fnv # noqa: E402
from cheats import Main, convert, fnv # noqa: E402
from mod_files import patch_files # noqa: E402
NSO_HEADER = 0x100
EXEFS_PLACES = ('exefs', f'atmosphere/contents/{TITLE}/exefs', f'contents/{TITLE}/exefs', f'{TITLE}/exefs')
def patch_files(mods):
"""[(mod folder name, path in the mod, file)] of the folders of mods, by name: the .ips and .pchtxt files of the
first exefs place that exists (runtime/mods.c find_root) and of atmosphere/exefs_patches/<name>/."""
out = []
for mod in sorted((d for d in mods.iterdir() if d.is_dir()), key=lambda d: d.name) if mods.is_dir() else []:
folders = []
exefs = next((mod / w for w in EXEFS_PLACES if (mod / w).is_dir()), None)
if exefs:
folders.append(exefs)
patches = mod / 'atmosphere' / 'exefs_patches'
if patches.is_dir():
folders += sorted(d for d in patches.iterdir() if d.is_dir())
for folder in folders:
for f in sorted(folder.iterdir()):
if f.is_file() and f.suffix.lower() in ('.ips', '.pchtxt'):
out.append((mod.name, f.relative_to(mod).as_posix(), f))
return out
def same_build(name, build_id):

View File

@@ -12,7 +12,11 @@ Steps, each skipped when a run before finished it for the same inputs (state in
3. romfs every RomFS file, checked against the container's sha256, into one image -> <out>/romfs/romfs.bin and
index.json (16-byte aligned, in the byte order of the paths). When the lift and the compile follow, it
runs on a thread of its own beside them: only the RomFS index and romfs.c wait for the image
4. lift recomp/gen.py with recomp/patches-v111.txt -> <out>/build-cache/lift, plus the two SDK bring-up fixes
4. lift recomp/gen.py with recomp/patches-v111.txt -> <out>/build-cache/lift, plus the two SDK bring-up fixes.
The code cheats and ExeFS patch files of the mods in <out>/mods (tools/mod_files.py) become patch sets
first (tools/cheats.py, tools/exefs_patches.py), which the game switches on with their mod; the step's
key names those files, and build-cache/mod-code.json records the ones it took up (the launcher
rebuilds the game before it starts when a mod brings others, launcher/mods.py)
5. compile the lifted C and runtime/ with MSYS2's clang, linked into <out>/ocarina.exe; the MSYS2 DLLs it loads,
the shader translator and romfs-index.bin beside it
6. shaders every shader in the game files (tools/prepare_shaders.py, default bind state) translated, and each one's
@@ -84,6 +88,7 @@ from pathlib import Path
KIT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(KIT / 'tools'))
import nup # noqa: E402
import mod_files # noqa: E402
import prompt_art # noqa: E402
TITLE_ID = 0x0400486024DEC000
@@ -99,7 +104,10 @@ OWN_KEY_STEPS = ('prompts', 'scan')
# (a step that runs, a later step that reads nothing it writes): the later step is not made to run again. The lift
# reads only the unpacked ExeFS (step_key), so the RomFS step leaves it alone: an install that was cut short while the
# image was still being packed, the lift beside it already done, packs the image again but does not lift again.
UNAFFECTED = {('romfs', 'lift')}
# The shaders step's key names the lift and the compile it depends on without the mods' code (step_key): a lift and a
# compile that run for a mod's code cheats or ExeFS patches leave the shaders prepared, as they leave the game's
# shaders as they were, and one that runs for anything else (another kit) changes that key and prepares them again.
UNAFFECTED = {('romfs', 'lift'), ('lift', 'shaders'), ('compile', 'shaders')}
# The Python packages tools/pad_prompts.py imports (pip name: import name), checked before a build with button art.
PROMPT_PACKAGES = {'zstandard': 'zstandard', 'fonttools': 'fontTools', 'pillow': 'PIL'}
# The line tools/pad_prompts.py prints for each overlay folder it wrote.
@@ -517,12 +525,35 @@ def lift_digest():
return h.hexdigest()
def step_lift(out, cache):
"""gen.py into build-cache/lift. The folder is kept: gen.py rewrites only the chunks whose text changed and
removes the ones it no longer writes."""
lift = cache / 'lift'
cmd = py_cmd(KIT / 'recomp' / 'gen.py', '--exefs', out / 'exefs', '--out', lift,
'--patches', KIT / 'recomp' / 'patches-v111.txt')
# The mods' code (tools/mod_files.py): the tools that make patch sets of it, the patch file each writes, the status
# file the runtime reads in build-cache next to the exe (runtime/cheats.c, runtime/mods.c) and what the log calls it
MOD_STAGES = (('cheats.py', 'patches-cheats.txt', 'cheats.status', 'code cheats'),
('exefs_patches.py', 'patches-exefs.txt', 'exefs.status', 'ExeFS patches'))
# build-cache/: {"<mod>/<path>": sha256} of the files the last lift took up, built in or refused (the status files say
# which): the launcher builds the game again before Play only for files no build has seen (launcher/mods.py)
MOD_RECORD = 'mod-code.json'
def mod_code(out):
"""{"<mod>/<path in the mod>": sha256} of the code cheats and ExeFS patch files of <out>/mods for main's build."""
return mod_files.code_record(out / 'mods', mod_files.main_build_id(out / 'exefs' / 'main'))
def mod_digest(out):
"""The mods' part of the lift's key: '' without mod code (the key of a game without mods stays what it was), else
the files by name and content and the tools that turn them into patch sets."""
code = mod_code(out)
if not code:
return ''
h = hashlib.sha256(json.dumps(code, sort_keys=True).encode())
for tool in ('cheats.py', 'exefs_patches.py', 'mod_files.py'):
h.update(sha256_file(KIT / 'tools' / tool).encode() if (KIT / 'tools' / tool).is_file() else b'no tool')
return h.hexdigest()
def run_lifter(out, lift, patches):
"""gen.py with the patch file into lift; its exit code."""
cmd = py_cmd(KIT / 'recomp' / 'gen.py', '--exefs', out / 'exefs', '--out', lift, '--patches', patches)
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, encoding='utf-8',
errors='replace', bufsize=1)
for line in p.stdout:
@@ -531,8 +562,52 @@ def step_lift(out, cache):
m = EMITTED.search(line)
if m:
emit('progress', step='lift', done=int(m.group(1)), total=int(m.group(2)), unit='chunks')
if p.wait():
fail('the code generator failed (see above)')
return p.wait()
def mod_patches(out, cache, plain):
"""The patch file with the mods' code cheats and ExeFS patches added (plain when there are none), the status files
written, and the mods' code. Each status file goes first: without it the runtime takes every such file as not
built. The files are the player's downloads, so one the tools refuse costs its stage, never the build."""
patches, done = plain, []
code = mod_code(out)
for tool, written, status, what in MOD_STAGES:
(cache / status).unlink(missing_ok=True)
if not code:
continue
cmd = py_cmd(KIT / 'tools' / tool, '--exefs', out / 'exefs', '--mods', out / 'mods', '--patches', patches,
'--out', cache / written, '--status', cache / status)
r = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, encoding='utf-8',
errors='replace')
for line in r.stdout.splitlines():
say(f' {line}')
if r.returncode:
(cache / status).unlink(missing_ok=True)
say(f' the mods\' {what} could not be built in (see above): the game is built without them')
else:
patches = cache / written
done.append(status)
return patches, done, code
def step_lift(out, cache):
"""gen.py into build-cache/lift. The folder is kept: gen.py rewrites only the chunks whose text changed and
removes the ones it no longer writes. The mods' code cheats and ExeFS patches go in as patch sets (mod_patches);
should the lifter refuse those, the game is lifted without them."""
lift = cache / 'lift'
plain = KIT / 'recomp' / 'patches-v111.txt'
(cache / MOD_RECORD).unlink(missing_ok=True)
patches, done, code = mod_patches(out, cache, plain)
if run_lifter(out, lift, patches):
if patches == plain:
fail('the code generator failed (see above)')
say(' the code generator refused the mods\' code: the game is lifted without it')
for status in done:
(cache / status).unlink(missing_ok=True)
if run_lifter(out, lift, plain):
fail('the code generator failed (see above)')
if code:
(cache / MOD_RECORD).write_text(json.dumps(code, indent=1, sort_keys=True), encoding='utf-8')
for mod, old, new in LIFT_FIXES:
hits = [p for p in sorted((lift / mod).glob('*.c')) if old in p.read_text(encoding='utf-8')]
if len(hits) != 1 or hits[0].read_text(encoding='utf-8').count(old) != 1:
@@ -1559,6 +1634,15 @@ def xlate_source():
fail(f'{XLATE_NAME} is missing: build it with tools/build_xlate.py')
def xlate_id():
"""The shader translator's id file's sha256 for the shaders step's key, 'missing' without a translator: the key is
worked out before the steps run (steps_to_run), and the shaders step itself says what is missing when it runs."""
for d in (KIT / 'bin', KIT / 'build' / 'xlate'):
if (d / XLATE_NAME).is_file():
return sha256_file(d / 'nx2navi_xlate.id')
return 'missing'
# The x86-64 level the lifted code was compiled for, beside the objects (lifted_level).
LIFTED_BUILD = 'lifted-build.json'
LIFTED_OBJECTS = ('main_', 'sdk_', 'subsdk0_') # the objects of the lifted code, by the module their names start with
@@ -2208,10 +2292,12 @@ def step_scan(out):
def step_key(step, identity, out=None):
"""What a step's output depends on. exefs and romfs: the game files only; lift: the unpacked ExeFS and the
lifter; compile: the lift, the runtime sources, this script and the processor (cpu_tag); prompts: the game files,
the kit's button art, the version of the tools that draw it (PROMPT_TOOLS), the bind_ lines of nx2navi.ini, and
whether every folder it writes is still there; scan: the game files, the check's version (SCAN_TOOL) and whether its report is still there."""
"""What a step's output depends on. exefs and romfs: the game files only; lift: the unpacked ExeFS, the lifter and
the mods' code cheats and ExeFS patches (mod_digest); compile: the lift, the runtime sources, this script and the
processor (cpu_tag); shaders: the compile without the mods' code, the preparing script and the translator;
prompts: the game files, the kit's button art, the version of the tools that draw it (PROMPT_TOOLS), the bind_
lines of nx2navi.ini, and whether every folder it writes is still there; scan: the game files, the check's version
(SCAN_TOOL) and whether its report is still there."""
if step in ('exefs', 'romfs'):
return f'{step}-1|{identity}'
if step == 'scan':
@@ -2223,17 +2309,21 @@ def step_key(step, identity, out=None):
written = all((out / f / 'manifest.tsv').is_file() for f in prompt_art.OVERLAYS)
return (f'prompts-1|{identity}|{tree_digest(prompt_art_dir(), ("**/*.png",))}|{PROMPT_TOOLS}|{binds}'
f'|{"written" if written else "missing"}')
lift = f'lift|{exefs_digest(out)}|{lift_digest()}'
base = f'lift|{exefs_digest(out)}|{lift_digest()}'
mods = mod_digest(out) if step in ('lift', 'compile') else ''
lift = f'{base}|mods {mods}' if mods else base
if step == 'lift':
return lift
compile_key = (f'compile|{lift}|{tree_digest(KIT / "runtime", ("**/*",))}|{sha256_file(Path(__file__))}|{kit_version()}'
f'|cpu {cpu_tag()}')
def compile_of(lift):
return (f'compile|{lift}|{tree_digest(KIT / "runtime", ("**/*",))}|{sha256_file(Path(__file__))}'
f'|{kit_version()}|cpu {cpu_tag()}')
if step == 'compile':
return compile_key
# shaders: the built game and the translator that prepared them, the one xlate_source finds (the kit's bin/, or a
# source checkout's build/xlate/)
return (f'shaders-1|{compile_key}|{sha256_file(KIT / "tools" / "prepare_shaders.py")}'
f'|{sha256_file(xlate_source() / "nx2navi_xlate.id")}')
return compile_of(lift)
# shaders: the built game without the mods' code (UNAFFECTED) and the translator that prepared them, the one
# xlate_source finds (the kit's bin/, or a source checkout's build/xlate/)
return (f'shaders-1|{compile_of(base)}|{sha256_file(KIT / "tools" / "prepare_shaders.py")}'
f'|{xlate_id()}')
def out_path(arg):

View File

@@ -31,9 +31,11 @@ NAME = 'NX2navi-BuildKit'
TOOLS = ('make_game.py', 'nup.py', 'romfs_index.py', 'check_content.py', 'hostmem.py', 'prepare_shaders.py',
'shader_census.py', 'shader_list.py', 'pipeline_list.py', 'zstd_compat.py', 'pad_prompts.py', 'prompt_art.py', 'button_icons.py', 'romfs_check.py',
'build_game.py', 'check_isa.py', 'build_shaders.py', 'make_dlaa_mask.py', 'make_mvobj_fs.py',
'pack_shader_cache.py', 'vulkan_pin.py') # make_game.py imports build_game's vma_include and vulkan_include on
# Linux for the VMA header and for Vulkan headers newer than the system's (which build_game takes from vulkan_pin), and
# build_game.py imports check_isa and build_shaders, which reads the two make_ scripts' shaders
'pack_shader_cache.py', 'vulkan_pin.py', 'mod_files.py', 'cheats.py', 'exefs_patches.py')
# make_game.py imports build_game's vma_include and vulkan_include on Linux for the VMA header and for Vulkan headers
# newer than the system's (which build_game takes from vulkan_pin), and build_game.py imports check_isa and
# build_shaders, which reads the two make_ scripts' shaders; the lift runs cheats.py and exefs_patches.py for the mods'
# code, and both import mod_files.py
TOP = ('LICENSE', 'LICENSE-ADDITIONS', 'THIRD_PARTY_NOTICES.md', 'VERSION', 'nx2navi.ini.example')
SKIP_DIRS = {'__pycache__', '.pytest_cache'}
SKIP_SUFFIXES = {'.o', '.obj', '.pyc', '.exe', '.dll', '.a', '.lib'}

View File

@@ -80,7 +80,7 @@ PINS = {
}
TOOLS = ('make_game.py', 'nup.py', 'romfs_index.py', 'check_content.py', 'hostmem.py', 'prepare_shaders.py',
'shader_census.py', 'shader_list.py', 'pipeline_list.py', 'zstd_compat.py', 'pad_prompts.py', 'prompt_art.py', 'button_icons.py', 'romfs_check.py',
'pack_shader_cache.py')
'pack_shader_cache.py', 'mod_files.py', 'cheats.py', 'exefs_patches.py')
TOP = ('LICENSE', 'LICENSE-ADDITIONS', 'THIRD_PARTY_NOTICES.md', 'nx2navi.ini.example')
SKIP_DIRS = {'__pycache__', '.pytest_cache', 'native'}
SKIP_SUFFIXES = {'.o', '.obj', '.pyc', '.exe', '.dll', '.a', '.lib'}

View File

@@ -114,7 +114,7 @@ LLVM_SKIP_SUFFIXES = {'.idl', '.tlb'}
TOOLS_FILES = ['build_game.py', 'build_shaders.py', 'button_icons.py', 'check_content.py', 'check_isa.py', 'extract.py',
'hostmem.py', 'make_romfs_patches.py', 'ncz.py', 'pack_shader_cache.py', 'prepare_shaders.py',
'romfs_index.py', 'shader_census.py', 'shader_list.py', 'pipeline_list.py', 'ultrawide_layout.py', 'cheats.py',
'exefs_patches.py']
'exefs_patches.py', 'mod_files.py']
TOOLS_DIRS = ['vendor/macnativegames', 'data'] # data/: expected hashes
RUNTIME_SUFFIXES = {'.h', '.inc', '.ico'} # .ico: the installer window icon

99
tools/mod_files.py Normal file
View File

@@ -0,0 +1,99 @@
"""The player's mods folder as the game reads it (runtime/mods.c): where each mod keeps its parts, and the files of the
mods that change the game's code.
One folder per mod (a symbolic link or junction to a folder counts). A part (romfs, romfslite, exefs, cheats) is the
first of <mod>/<part>, <mod>/atmosphere/contents/<title>/<part>, <mod>/contents/<title>/<part> and
<mod>/<title>/<part> that is a folder (runtime/mods.c find_root).
The files that change the game's code are its Atmosphere cheat file for main's build (<cheats>/<build id>.txt, the
name in any case) and the ExeFS patch files (.ips, .pchtxt) of its exefs part and of
<mod>/atmosphere/exefs_patches/<name>/. The lifted game cannot be patched while it runs, so the build turns them into
patch sets (tools/cheats.py, tools/exefs_patches.py) and the runtime switches those on with their mod; a file the
build did not see waits for the next one. tools/make_game.py builds them in and records what it built
(build-cache/mod-code.json, code_record); the launcher compares that record with the folder (launcher/mods.py) to
rebuild the game before it starts when a mod brought new ones.
Standard library only: the launcher imports it from the kit's tools folder.
"""
import hashlib
from pathlib import Path
TITLE = '0400486024DEC000' # the game's title id: a mod's Atmosphere folders are named after it
PLACES = ('{}', f'atmosphere/contents/{TITLE}/{{}}', f'contents/{TITLE}/{{}}', f'{TITLE}/{{}}')
PATCH_SUFFIXES = ('.ips', '.pchtxt')
def mod_folders(mods):
"""The mod folders of the mods folder, by name (as the setup's tools list them: Python's order of the names)."""
mods = Path(mods)
if not mods.is_dir():
return []
return sorted((d for d in mods.iterdir() if d.is_dir()), key=lambda d: d.name)
def part(mod, name):
"""The folder of a mod's part (romfs, romfslite, exefs, cheats): the first of its places that is a folder, or
None."""
return next((Path(mod) / w.format(name) for w in PLACES if (Path(mod) / w.format(name)).is_dir()), None)
def cheat_files(mods, bid):
"""[(mod folder name, cheat file)] of the folders of mods, by name: <cheats>/<bid>.txt, the name in any case. bid is
the first 8 bytes of main's build id in hex, as cheat files name themselves (tools/cheats.py build_id)."""
out = []
for mod in mod_folders(mods):
folder = part(mod, 'cheats')
if folder:
f = next((p for p in sorted(folder.iterdir()) if p.is_file() and p.name.lower() == f'{bid}.txt'.lower()),
None)
if f:
out.append((mod.name, f))
return out
def patch_files(mods):
"""[(mod folder name, path in the mod, file)] of the folders of mods, by name: the .ips and .pchtxt files of the
exefs part and of atmosphere/exefs_patches/<name>/."""
out = []
for mod in mod_folders(mods):
folders = []
exefs = part(mod, 'exefs')
if exefs:
folders.append(exefs)
patches = mod / 'atmosphere' / 'exefs_patches'
if patches.is_dir():
folders += sorted(d for d in patches.iterdir() if d.is_dir())
for folder in folders:
for f in sorted(folder.iterdir()):
if f.is_file() and f.suffix.lower() in PATCH_SUFFIXES:
out.append((mod.name, f.relative_to(mod).as_posix(), f))
return out
def main_build_id(main):
"""The first 8 bytes of main's build id in upper-case hex (the cheat files' name), '' when main cannot be read."""
try:
with open(main, 'rb') as f:
head = f.read(0x48)
except OSError:
return ''
return head[0x40:0x48].hex().upper() if len(head) == 0x48 and head[:4] == b'NSO0' else ''
def code_files(mods, bid):
"""[(mod folder name, path in the mod, file)] of every file of the mods folder that changes the game's code: the
cheat files for bid, then the ExeFS patch files."""
return ([(m, f.relative_to(Path(mods) / m).as_posix(), f) for m, f in cheat_files(mods, bid)]
+ patch_files(mods))
def code_record(mods, bid):
"""{"<mod>/<path in the mod>": sha256 of its bytes} of code_files: what a build that lifts them in takes. A file
that cannot be read is left out (the tools would fail on it too, and the next build tries again)."""
out = {}
for mod, rel, f in code_files(mods, bid):
try:
out[f'{mod}/{rel}'] = hashlib.sha256(f.read_bytes()).hexdigest()
except OSError:
pass
return out