Redraw the in-game menu with the game's own interface textures, and bring in the launcher's save import and faster builds.

In-game menu: the settings window, its tabs, rows, buttons and cursor, the performance overlay and the Preparing
shaders box are drawn with the game's interface textures where the RomFS holds them (runtime/ui/menu_skin.cpp,
perf_panel.cpp; plain shapes until they are decoded). Restart joins Continue and Quit, each behind a confirmation;
VSync is an on/off switch; DLSS is listed only on NVIDIA GPUs; the menu sounds are quieter; Picture shape is gone and
the setting descriptions are shorter; edge cases of input, layout and fonts are fixed. tools/menu_preview.py draws the
menu without the game.

Launcher: Import Saves takes a save from the game folder's saves/ folder or one the player picks; downloads fall back
to Windows' own certificate check when Python's refuses a server; updates are faster; the title bar is drawn in the
window and Navi is every icon. tools/make_game.py builds with CLANG64's clang where the launcher installed it, packs
the RomFS beside the lift and the compile, takes a prebuilt runtime, and compiles the game's resources without cmd.exe,
so a user folder with a space in its name builds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
chrg6
2026-11-08 03:57:01 -05:00
parent 5b28da3a06
commit 8daedbd806
42 changed files with 4138 additions and 972 deletions

View File

@@ -10,7 +10,8 @@ Steps, each skipped when a run before finished it for the same inputs (state in
2. exefs the five ExeFS files, every module's segments unpacked (ZBIC through the player's own `zbic` package)
and stored plainly, as the runtime's loader reads them -> <out>/exefs
3. romfs every RomFS file, checked against the container's sha256, into one image -> <out>/romfs/romfs.bin and
index.json (16-byte aligned, in the byte order of the paths)
index.json (16-byte aligned, in the byte order of the paths). When the lift and the compile follow, it
runs on a thread of its own beside them: only the RomFS index and romfs.c wait for the image
4. lift recomp/gen.py with recomp/patches-v111.txt -> <out>/build-cache/lift, plus the two SDK bring-up fixes
5. compile the lifted C and runtime/ with MSYS2's clang, linked into <out>/ocarina.exe; the MSYS2 DLLs it loads,
the shader translator and romfs-index.bin beside it
@@ -20,8 +21,16 @@ Steps, each skipped when a run before finished it for the same inputs (state in
The exefs and romfs steps depend only on the game files; lift and compile depend on this kit's code too. A newer kit
therefore lifts and compiles again but keeps the unpacked game files, and --rebuild needs no game files at all. The
compile step keeps every object with the dependency list clang wrote for it (build-cache/obj/objects.json) and
compiles a file again only when it, a header it includes, the flags or the compiler changed. The new ocarina.exe
replaces the old one only once it links, so a failed build leaves the last good one in place.
compiles a file again only when it, a header it includes, the flags or the compiler changed; the kit's own folder
is not part of that key, so a kit in another folder (the launcher keeps each version in its own) reuses the objects.
A kit with lib/libnx2navirt.a built by --make-runtime-lib for the same runtime/ and the same compiler links the
runtime from it and compiles only the runtime files that include headers written per install. The new ocarina.exe
replaces the old one only once it links (with lld when the toolchain has it), so a failed build leaves the last good
one in place. Compiles run on every logical processor, the largest files first. The lifted code is built for this
PC's x86-64 level (cpu_level), with MSYS2 CLANG64's clang when it sits beside the UCRT64 folder and reports the same
version (fast_clang: the same objects, about a fifth faster).
python tools/make_game.py --make-runtime-lib <kit>/lib --msys <ucrt64> # the release's prebuilt runtime
Nothing is written outside --out. The game folder then holds only what this PC built from the player's files: never
share it. Needs Python 3.10+ with numpy, lz4, zstandard and zbic (pip), and MSYS2 UCRT64's clang and libraries
@@ -43,6 +52,7 @@ import shutil
import struct
import subprocess
import sys
import threading
import time
from pathlib import Path
@@ -76,11 +86,14 @@ LIFT_FIXES = [
LINK_LIBS = ['-lvulkan-1', '-lSDL3', '-lopus', '-lvpx', '-l:libdav1d.a', '-lzstd', '-l:libatomic.a', '-lbcrypt',
'-limm32', '-luuid', '-lole32', '-luser32', '-lshell32', '-lgdi32', '-ladvapi32', '-lwinmm']
SYSTEM_DLLS_SKIP = {'vulkan-1.dll'} # the graphics driver's loader, never a copy
# GNU windres's preprocessor, named explicitly: left to find it, windres puts the full path of the gcc beside it on a
# cmd.exe command line, and once that path and the .rc's both hold a space, cmd.exe drops the outer quotes and splits
# the command at the first space. gcc is then found on PATH (the toolchain's bin first); the arguments are windres's own.
WINDRES_PP = ['--preprocessor=gcc', '--preprocessor-arg=-E', '--preprocessor-arg=-xc', '--preprocessor-arg=-DRC_INVOKED']
EXE_NAME = 'ocarina.exe'
# Headers this build writes per install (gen.py, romfs_index.py): a runtime source that includes one, directly or
# through a runtime header, is compiled on the player's PC; every other runtime source can come prebuilt.
GENERATED_HEADERS = ('layout.h', 'patches.h', 'romfs_config.h')
RUNTIME_LIB = 'libnx2navirt.a'
RUNTIME_PATTERNS = ('*.c', '*.cpp', 'platform/*.c', 'ui/*.cpp', 'third_party/imgui/*.cpp',
'third_party/imgui/backends/*.cpp')
INCLUDE = re.compile(r'^[ \t]*#[ \t]*include[ \t]+"([^"]+)"', re.M)
# Runs a script the way `python script.py` does, with the script's folder first on sys.path. The launcher's
# embeddable Python leaves the script folder off sys.path (its ._pth file), so every script this build starts goes
@@ -89,24 +102,53 @@ RUN_SCRIPT = ('import os, runpy, sys; s = sys.argv[1]; sys.argv = sys.argv[1:];
'sys.path.insert(0, os.path.dirname(os.path.abspath(s))); runpy.run_path(s, run_name="__main__")')
EVENTS = False
OUT_LOCK = threading.Lock() # the RomFS step prints from its own thread (main)
def say(msg):
print(msg, flush=True)
with OUT_LOCK:
print(msg, flush=True)
def emit(ev, **fields):
"""One "@@{json}" line for the launcher (--events); nothing otherwise."""
if EVENTS:
print('@@' + json.dumps({'ev': ev, **fields}, ensure_ascii=False), flush=True)
line = '@@' + json.dumps({'ev': ev, **fields}, ensure_ascii=False)
with OUT_LOCK:
print(line, flush=True)
def fail(msg):
emit('error', message=msg)
print(f'\nerror: {msg}', file=sys.stderr, flush=True)
with OUT_LOCK:
print(f'\nerror: {msg}', file=sys.stderr, flush=True)
sys.exit(1)
class Background:
"""One step on a thread of its own. join() waits for it and raises its failure (fail's SystemExit included) in
the caller; failed() tells the caller early, so it can stop work that would be thrown away."""
def __init__(self, fn):
self.exc = None
self.thread = threading.Thread(target=self._run, args=(fn,), name='romfs', daemon=True)
self.thread.start()
def _run(self, fn):
try:
fn()
except BaseException as e: # noqa: B036 - handed to join()
self.exc = e
def failed(self):
return self.exc is not None
def join(self):
self.thread.join()
if self.exc is not None:
raise self.exc
def py_cmd(script, *args):
"""The command line that runs a Python script of this kit with this interpreter (see RUN_SCRIPT)."""
flags = ['-I'] if sys.flags.isolated else []
@@ -317,11 +359,11 @@ def step_romfs(src, out, jobs):
done += size
emit('progress', step='romfs', done=done, total=total, unit='bytes')
if start // batch % 40 == 0:
say(f' {done / 1e9:5.1f} of {total / 1e9:.1f} GB, {time.time() - t0:.0f} s')
say(f' RomFS {done / 1e9:5.1f} of {total / 1e9:.1f} GB, {time.time() - t0:.0f} s')
part.replace(dst / 'romfs.bin')
index = {'status': 'pass', 'romfs_size': pos, 'romfs_sha256': h.hexdigest(), 'files': entries}
(dst / 'index.json').write_text(json.dumps(index), encoding='utf-8')
say(f' image {pos / 1e9:.2f} GB in {time.time() - t0:.0f} s')
say(f' RomFS image {pos / 1e9:.2f} GB in {time.time() - t0:.0f} s')
# ---------------------------------------------------------------- lift
@@ -491,17 +533,33 @@ def parse_deps(text):
class ObjectCache:
"""Which objects are current: per object the key of its command and source, and the sha256 of every file its
dependency list names (build-cache/obj/objects.json)."""
"""Which objects are current: per object the key of its command and source, and the sha256 (with size and
modification time, to skip hashing an untouched file) of every file its dependency list names
(build-cache/obj/objects.json). Paths under a root are kept relative to it ({'<kit>': KIT}), so the same files
in a kit at another place still count as the same."""
def __init__(self, path):
def __init__(self, path, roots=None):
self.path = path
self.roots = {k: os.path.normcase(os.path.abspath(v)) for k, v in (roots or {}).items()}
try:
self.entries = json.loads(path.read_text(encoding='utf-8')) if path.is_file() else {}
except (OSError, ValueError):
self.entries = {}
self._sha = {}
def norm(self, p):
a = os.path.normcase(os.path.abspath(p))
for name, root in self.roots.items():
if a == root or a.startswith(root + os.sep):
return name + a[len(root):].replace(os.sep, '/')
return a
def real(self, p):
for name, root in self.roots.items():
if p.startswith(name):
return root + p[len(name):].replace('/', os.sep)
return p
def file_sha(self, p):
p = os.path.normcase(os.path.abspath(p))
if p not in self._sha:
@@ -511,18 +569,37 @@ class ObjectCache:
self._sha[p] = None
return self._sha[p]
def _same(self, p, rec):
path = self.real(p)
if isinstance(rec, str): # a cache without sizes and times
return self.file_sha(path) == rec
digest, size, mtime = rec
try:
st = os.stat(path)
except OSError:
return False
if st.st_size == size and st.st_mtime_ns == mtime:
return True
return st.st_size == size and self.file_sha(path) == digest
def current(self, obj, key):
e = self.entries.get(obj.name)
return (e is not None and e.get('key') == key and obj.is_file()
and all(self.file_sha(p) == h for p, h in e.get('deps', {}).items()))
and all(self._same(p, rec) for p, rec in e.get('deps', {}).items()))
def record(self, obj, key, dep_file, src):
try:
deps = parse_deps(Path(dep_file).read_text(encoding='utf-8', errors='replace'))
except OSError:
deps = []
deps = deps or [str(src)]
self.entries[obj.name] = {'key': key, 'deps': {p: self.file_sha(p) for p in deps}}
out = {}
for p in deps or [str(src)]:
try:
st = os.stat(p)
out[self.norm(p)] = [self.file_sha(p), st.st_size, st.st_mtime_ns]
except OSError:
out[self.norm(p)] = [None, -1, -1]
self.entries[obj.name] = {'key': key, 'deps': out}
def forget(self, obj):
self.entries.pop(obj.name, None)
@@ -539,22 +616,31 @@ def compiler_id(clang, env):
return sha256_bytes(f'{p.stdout}\n{st.st_size}\n{int(st.st_mtime)}'.encode())
def compile_all(srcs, cmd_for, cache, jobs, env, progress=None, salt=''):
def compile_all(srcs, cmd_for, cache, jobs, env, progress=None, salt='', stop=None):
"""Compile each (source, object) whose object is not current in cache. cmd_for(src, dst) -> argv without the
dependency-file options; salt identifies the compiler. Returns (compiled, reused, failures) with failures as
(src, stderr tail)."""
(src, stderr tail). The cache's roots stand for their paths in the flags too. The largest sources start first:
compile time grows with size, and a big file that started last would run on alone after the others finished.
stop() true makes the compiles not yet started end at once, unrecorded (a step beside this one failed)."""
todo, keys = [], {}
for src, dst in srcs:
cmd = cmd_for(src, dst)
flags = [c for c in cmd if c not in (str(src), str(dst))]
flags = [c for c in cmd[1:] if c not in (str(src), str(dst))]
for name, root in cache.roots.items():
flags = [name + os.path.normcase(f)[len(root):].replace(os.sep, '/') if os.path.normcase(f).startswith(root)
else f for f in flags]
flags = [os.path.basename(cmd[0])] + flags
keys[dst] = sha256_bytes(json.dumps([salt, flags, cache.file_sha(src)]).encode())
if not cache.current(dst, keys[dst]):
todo.append((src, dst))
reused = len(srcs) - len(todo)
todo.sort(key=lambda job: job[0].stat().st_size, reverse=True)
def cc(job):
src, dst = job
dep = dst.with_suffix('.d')
if stop and stop():
return src, dst, dep, None, ''
cmd = cmd_for(src, dst)
cmd[1:1] = ['-MMD', '-MF', str(dep)]
p = subprocess.run(cmd, capture_output=True, text=True, env=env, errors='replace')
@@ -566,6 +652,8 @@ def compile_all(srcs, cmd_for, cache, jobs, env, progress=None, salt=''):
with concurrent.futures.ThreadPoolExecutor(max(1, jobs)) as ex:
for src, dst, dep, rc, err in ex.map(cc, todo):
done += 1
if rc is None: # not started (stop)
continue
if rc:
bad.append((src, err))
cache.forget(dst)
@@ -577,33 +665,236 @@ def compile_all(srcs, cmd_for, cache, jobs, env, progress=None, salt=''):
return len(todo), reused, bad
def step_compile(out, cache_dir, msys, jobs):
def runtime_sources():
"""(source, object name) of every runtime file the Windows build compiles."""
rt = KIT / 'runtime'
out = []
for pat in RUNTIME_PATTERNS:
for p in sorted(rt.glob(pat)):
if p.stem.endswith('_sdl') or p.name == 'linux.c':
continue
out.append((p, 'rt_' + p.relative_to(rt).as_posix().replace('/', '_').rsplit('.', 1)[0] + '.o'))
return out
def per_install_names(generated=GENERATED_HEADERS):
"""The basenames of the generated headers and of every runtime header that includes one, at any depth."""
rt = KIT / 'runtime'
headers = {}
for p in list(rt.rglob('*.h')) + list(rt.rglob('*.inc')):
headers[p.name] = {os.path.basename(i) for i in INCLUDE.findall(p.read_text(encoding='utf-8', errors='replace'))}
tainted = set(generated)
grew = True
while grew:
grew = False
for name, incs in headers.items():
if name not in tainted and incs & tainted:
tainted.add(name)
grew = True
return tainted
def per_install(src, tainted):
incs = {os.path.basename(i) for i in INCLUDE.findall(Path(src).read_text(encoding='utf-8', errors='replace'))}
return bool(incs & tainted)
def clang_version(clang, env):
p = subprocess.run([str(clang), '--version'], capture_output=True, text=True, env=env)
return (p.stdout.splitlines() or [''])[0].strip()
def runtime_digest():
return tree_digest(KIT / 'runtime', ('**/*',))
# The x86-64 levels the lifted code is built and tested for (docs/building.md, "Checks"), as the macros clang defines
# for a CPU that has them.
CPU_V2 = {'__SSE3__', '__SSSE3__', '__SSE4_1__', '__SSE4_2__', '__POPCNT__', '__LAHF_SAHF__',
'__GCC_HAVE_SYNC_COMPARE_AND_SWAP_16'}
CPU_V3 = CPU_V2 | {'__AVX__', '__AVX2__', '__BMI__', '__BMI2__', '__F16C__', '__FMA__', '__LZCNT__', '__MOVBE__',
'__XSAVE__'}
def cpu_level(clang, env):
"""This PC's x86-64 level for the lifted code: 3 (AVX2, FMA, BMI2), 2 (SSE4.2, POPCNT) or 0 when clang cannot
tell. The game runs on the PC that builds it, so it takes this CPU's level, as tools/build_game.py --cpu does by
hand. Without one the lifted code would be generic x86-64: every fused multiply-add of the game a call to
runtime/fma.c's software fma, and a level the differential tests do not cover."""
p = subprocess.run([str(clang), '-march=native', '-E', '-dM', '-x', 'c', os.devnull], capture_output=True,
text=True, env=env, errors='replace')
have = set(re.findall(r'^#define (__\w+) ', p.stdout, re.M)) if p.returncode == 0 else set()
return 3 if CPU_V3 <= have else 2 if CPU_V2 <= have else 0
def clang_ident(clang, env):
"""(version, target) a clang reports, or None when it does not run."""
try:
p = subprocess.run([str(clang), '--version'], capture_output=True, text=True, env=env, errors='replace')
except OSError:
return None
v, t = re.search(r'clang version (\S+)', p.stdout), re.search(r'^Target: (\S+)', p.stdout, re.M)
return (v.group(1), t.group(1)) if p.returncode == 0 and v and t else None
def fast_clang(msys, clang, env):
"""MSYS2's CLANG64 clang beside the UCRT64 one (msys64/clang64, which the launcher installs): the same LLVM release
built with profile-guided optimisation, about a fifth faster on the lifted code. With UCRT64's headers
(--sysroot) it writes the same objects as UCRT64's clang, byte for byte but the COFF timestamp; it is used only
when it reports the same version and target. None otherwise."""
fast = Path(msys).parent / 'clang64' / 'bin' / 'clang.exe'
if not fast.is_file():
return None
mine = clang_ident(clang, env)
return fast if mine and clang_ident(fast, env) == mine else None
def runtime_cmd(clang, clangxx, inc, level=0, fast=None, msys=None):
"""fast: fast_clang's compiler for the lifted code, with msys as its sysroot."""
def cmd_for(src, dst):
cpp = src.suffix in ('.cpp', '.cxx')
# the lifted code at -O1 (its size makes -O2 slow to build), the runtime at -O2; neither may assume strict
# aliasing: the lifted code reaches guest memory through casts of one address to several types
# (tools/build_game.py's flags). The lifted code is built for this PC's level (cpu_level); the runtime
# stays generic, so the prebuilt one serves every PC (its vectors are 16 bytes, passed alike at every level).
guest = dst.name.startswith(('main_', 'sdk_', 'subsdk0_'))
march = [f'-march=x86-64-v{level}'] if guest and level else []
cc = [str(fast), f'--sysroot={msys}'] if guest and fast else [str(clangxx if cpp else clang)]
return [*cc, '-c', '-std=gnu++17' if cpp else '-std=gnu11', '-O1' if guest else '-O2',
*march, '-fno-strict-aliasing', '-w', *inc, '-o', str(dst), str(src)]
return cmd_for
def make_runtime_lib(dest, msys, jobs, obj=None):
"""--make-runtime-lib: compile every runtime source that does not include a per-install header into
<dest>/libnx2navirt.a, with runtime-lib.json naming the runtime/ tree and the compiler it was built from. The
objects stay in obj (default <dest>/obj), which a release keeps out of what it ships."""
dest = Path(dest).resolve()
dest.mkdir(parents=True, exist_ok=True)
obj = Path(obj).resolve() if obj else dest / 'obj'
obj.mkdir(parents=True, exist_ok=True)
clang, clangxx = msys / 'bin' / 'clang.exe', msys / 'bin' / 'clang++.exe'
env = dict(os.environ, PATH=str(msys / 'bin') + os.pathsep + os.environ.get('PATH', ''))
rt = KIT / 'runtime'
tainted = per_install_names()
srcs = [(p, obj / name) for p, name in runtime_sources() if not per_install(p, tainted)]
inc = ['-I', str(rt), '-I', str(rt / 'third_party' / 'imgui'), '-I', str(msys / 'include' / 'vma')]
cache = ObjectCache(obj / 'objects.json', {'<kit>': KIT})
t0 = time.time()
compiled, reused, bad = compile_all(srcs, runtime_cmd(clang, clangxx, inc), cache, jobs, env,
lambda d, t: (d % 20 == 0 or d == t) and say(f' {d} of {t}'),
salt=compiler_id(clang, env))
if bad:
for src, err in bad[:3]:
print(f'--- {src}\n{err}', file=sys.stderr)
fail(f'{len(bad)} runtime files did not compile')
lib = dest / RUNTIME_LIB
lib.unlink(missing_ok=True)
rsp = dest / 'ar.rsp'
rsp.write_text('\n'.join('"%s"' % o.as_posix() for _, o in srcs), encoding='utf-8')
ar = os.environ.get('NX2NAVI_AR') or str(msys / 'bin' / 'ar.exe')
if subprocess.run([ar, 'rcs', str(lib), '@' + str(rsp)], env=env).returncode:
fail('the runtime library could not be archived')
meta = {'runtime': runtime_digest(), 'compiler': clang_version(clang, env),
'objects': sorted(o.name for _, o in srcs)}
(dest / 'runtime-lib.json').write_text(json.dumps(meta, indent=1), encoding='utf-8')
rsp.unlink(missing_ok=True)
say(f'{lib}: {len(srcs)} runtime files ({compiled} compiled, {reused} kept), {lib.stat().st_size / 1e6:.1f} MB, '
f'{time.time() - t0:.0f} s')
def prebuilt_runtime(clang, env):
"""(library, object names it holds) when the kit's lib/ matches this runtime/ and this compiler, else None."""
lib, meta = KIT / 'lib' / RUNTIME_LIB, KIT / 'lib' / 'runtime-lib.json'
try:
m = json.loads(meta.read_text(encoding='utf-8'))
except (OSError, ValueError):
return None
if not lib.is_file() or m.get('runtime') != runtime_digest() or m.get('compiler') != clang_version(clang, env):
say(' the prebuilt runtime does not match this kit or compiler: compiling runtime/')
return None
return lib, set(m.get('objects', []))
def romfs_index(out, gen):
if subprocess.run(py_cmd(KIT / 'tools' / 'romfs_index.py', '--romfs', out / 'romfs', '--out', gen)).returncode:
fail('the RomFS index could not be written')
def windres_command(msys, rco):
"""MSYS2's windres on nx2navi.rc, run with runtime/ as the working directory (compile_resources says why)."""
pp = ('-E', '-xc', '-DRC_INVOKED') # windres's own preprocessor flags, which --preprocessor replaces
return [str(msys / 'bin' / 'windres.exe'), '--use-temp-file', f'--preprocessor={msys / "bin" / "gcc.exe"}',
*(f'--preprocessor-arg={a}' for a in pp), '-O', 'coff', '-i', 'nx2navi.rc', '-o', str(rco)]
def compile_resources(msys, rt, obj, env):
"""runtime/nx2navi.rc into obj/nx2navi_rc.o: the manifest of the game's exe (per-monitor DPI awareness, so the
picture keeps the monitor's real size rather than one Windows stretches) and its icon.
windres preprocesses the .rc with gcc through one command line it writes itself, quoting each path that holds a
space. By default it hands that line to cmd.exe /c, which drops the first and the last double quote of a line
holding more than two: gcc's quoted path loses its opening quote, and under a user folder with a space in its
name (C:\\Users\\Ali Sayed) cmd.exe stops at that space ("'C:\\Users\\Ali' is not recognized").
With --use-temp-file windres starts gcc itself instead, no shell involved: it splits the line into arguments and
passes each one to gcc intact. Its splitter reads a quoted first word right, but it cuts a quoted -I path at its
spaces and reads past the end of the line after a quoted last word, so the line holds no path but gcc's, its
first word: the .rc goes by its bare name from runtime/ as the working directory, where windres also finds the
manifest and the icon, and there is no -I. windres's temporary files go to a folder of the build cache, deleted
afterwards (it leaves two of them behind on every run).
The object is written under another name and replaces the old one only when its bytes differ, so an unchanged
resource keeps its time and does not make the link key change (a rebuild with nothing new does not link again)."""
rco, new = obj / 'nx2navi_rc.o', obj / 'nx2navi_rc.o.new'
tmp = obj / 'windres-tmp'
shutil.rmtree(tmp, ignore_errors=True)
tmp.mkdir()
try:
r = subprocess.run(windres_command(msys, new), cwd=rt, env=dict(env, TMP=str(tmp), TEMP=str(tmp)),
capture_output=True, text=True, errors='replace')
finally:
shutil.rmtree(tmp, ignore_errors=True)
if r.returncode:
new.unlink(missing_ok=True)
fail(f'windres failed on runtime/nx2navi.rc: {r.stderr.strip()[:400]}')
if rco.is_file() and rco.read_bytes() == new.read_bytes():
new.unlink()
else:
os.replace(new, rco)
return rco
def step_compile(out, cache_dir, msys, jobs, romfs_job=None):
"""romfs_job: the RomFS step still running beside this one (a first build). Everything that does not include
romfs_config.h compiles meanwhile; the RomFS index and the few files that include it follow once it is done."""
lift, gen, obj = cache_dir / 'lift', cache_dir / 'gen', cache_dir / 'obj'
gen.mkdir(parents=True, exist_ok=True)
obj.mkdir(parents=True, exist_ok=True)
rt = KIT / 'runtime'
clang, clangxx = msys / 'bin' / 'clang.exe', msys / 'bin' / 'clang++.exe'
env = dict(os.environ, PATH=str(msys / 'bin') + os.pathsep + os.environ.get('PATH', ''))
if subprocess.run(py_cmd(KIT / 'tools' / 'romfs_index.py', '--romfs', out / 'romfs', '--out', gen)).returncode:
fail('the RomFS index could not be written')
if not romfs_job:
romfs_index(out, gen)
srcs = [(p, obj / f'{mod}_{p.stem}.o') for mod in ('main', 'sdk', 'subsdk0') for p in sorted((lift / mod).glob('*.c'))]
for pat in ('*.c', '*.cpp', 'platform/*.c', 'ui/*.cpp', 'third_party/imgui/*.cpp', 'third_party/imgui/backends/*.cpp'):
for p in sorted(rt.glob(pat)):
if p.stem.endswith('_sdl') or p.name == 'linux.c':
continue
srcs.append((p, obj / ('rt_' + p.relative_to(rt).as_posix().replace('/', '_').rsplit('.', 1)[0] + '.o')))
prebuilt = prebuilt_runtime(clang, env)
runtime = [(p, obj / name) for p, name in runtime_sources()]
if prebuilt:
runtime = [(p, o) for p, o in runtime if o.name not in prebuilt[1]]
say(f' the runtime comes prebuilt; {len(runtime)} runtime files are compiled here')
srcs += runtime
srcs += [(modules_c(lift, out, gen), obj / 'modules.o'), (gen / 'shim.c', obj / 'shim.o')]
inc = ['-I', str(rt), '-I', str(lift), '-I', str(gen), '-I', str(rt / 'third_party' / 'imgui'),
'-I', str(msys / 'include' / 'vma')]
cid = compiler_id(clang, env)
def cmd_for(src, dst):
cpp = src.suffix in ('.cpp', '.cxx')
# the lifted code at -O1 (its size makes -O2 slow to build), the runtime at -O2; neither may assume strict aliasing:
# the lifted code reaches guest memory through casts of one address to several types (tools/build_game.py's flags)
guest = dst.name.startswith(('main_', 'sdk_', 'subsdk0_'))
return [str(clangxx if cpp else clang), '-c', '-std=gnu++17' if cpp else '-std=gnu11', '-O1' if guest else '-O2',
'-fno-strict-aliasing', '-w', *inc, '-o', str(dst), str(src)]
level = cpu_level(clang, env)
say(f' the lifted code is built for x86-64-v{level}, this processor\'s level' if level else
' the lifted code is built for generic x86-64: the processor\'s level could not be read')
fast = fast_clang(msys, clang, env)
if fast:
cid += compiler_id(fast, env)
say(f' the lifted code compiles with {fast}')
cmd_for = runtime_cmd(clang, clangxx, inc, level, fast, msys)
t0 = time.time()
last = [0.0]
@@ -617,29 +908,42 @@ def step_compile(out, cache_dir, msys, jobs):
say(f' compiled {done} of {total}, {now - t0:.0f} s')
say(f' {len(srcs)} files on {jobs} threads')
cache = ObjectCache(obj / 'objects.json')
compiled, reused, bad = compile_all(srcs, cmd_for, cache, jobs, env, progress, salt=cid)
cache = ObjectCache(obj / 'objects.json', {'<kit>': KIT})
late = []
if romfs_job: # the lifted chunks include only cpu.h and simd.h: only the runtime and generated files are read
tainted = per_install_names(('romfs_config.h',))
late = [(s, o) for s, o in runtime + srcs[-2:] if per_install(s, tainted)]
say(f' {len(late)} files wait for the RomFS index: {", ".join(s.name for s, _ in late)}')
early = [j for j in srcs if j not in late]
compiled, reused, bad = compile_all(early, cmd_for, cache, jobs, env, lambda d, t: progress(d, len(srcs)),
salt=cid, stop=romfs_job.failed if romfs_job else None)
if romfs_job:
if romfs_job.thread.is_alive():
say(' waiting for the RomFS image')
romfs_job.join() # raises the RomFS step's failure; a failed compile still keeps the finished image
if not bad:
romfs_index(out, gen)
c2, r2, bad = compile_all(late, cmd_for, cache, jobs, env,
lambda d, t: progress(len(early) + d, len(srcs)), salt=cid)
compiled, reused = compiled + c2, reused + r2
say(f' {compiled} compiled, {reused} unchanged since the last build')
if bad:
for src, err in bad[:3]:
print(f'--- {src}\n{err}', file=sys.stderr)
fail(f'{len(bad)} files did not compile')
# the manifest (per-monitor DPI awareness: the picture at the monitor's real size, not one Windows stretches) and the
# icon, runtime/nx2navi.rc through MSYS2's windres. windres runs its preprocessor through cmd.exe: run from the
# runtime folder with relative names and gcc by bare name (WINDRES_PP), that command line holds no path, so a space
# in the install folder (a user name) cannot split it.
rco = (obj / 'nx2navi_rc.o').resolve()
r = subprocess.run([str(msys / 'bin' / 'windres.exe'), *WINDRES_PP, '-I', '.', '-O', 'coff', '-i', 'nx2navi.rc',
'-o', str(rco)], capture_output=True, text=True, env=env, errors='replace', cwd=rt)
if r.returncode:
fail(f'windres failed on runtime/nx2navi.rc: {r.stderr.strip()[:400]}')
rco = compile_resources(msys, rt, obj, env)
objs = [o for _, o in srcs] + [rco]
exe = out / EXE_NAME
rsp = cache_dir / 'link.rsp'
rsp.write_text('\n'.join('"%s"' % o.as_posix() for o in objs), encoding='utf-8')
link_key = sha256_bytes(json.dumps([LINK_LIBS, cid] + [[o.name, o.stat().st_size, o.stat().st_mtime_ns]
for o in objs]).encode())
# lld links several times faster than GNU ld; the stack matches tools/build_game.py's build
ld = ['-fuse-ld=lld', '-Wl,--stack,16777216'] if (msys / 'bin' / 'ld.lld.exe').is_file() else []
lib = ['-Wl,--whole-archive', str(prebuilt[0]), '-Wl,--no-whole-archive'] if prebuilt else []
stamp = [[o.name, o.stat().st_size, o.stat().st_mtime_ns] for o in objs]
if prebuilt:
stamp.append([prebuilt[0].name, prebuilt[0].stat().st_size, sha256_file(prebuilt[0])])
link_key = sha256_bytes(json.dumps([LINK_LIBS, ld, cid] + stamp).encode())
state_link = cache_dir / 'link.key'
if exe.is_file() and state_link.is_file() and state_link.read_text(encoding='utf-8') == link_key:
say(' ocarina.exe is current, not linked again')
@@ -647,8 +951,8 @@ def step_compile(out, cache_dir, msys, jobs):
emit('phase', step='compile', phase='link')
say(' linking')
tmp = out / (EXE_NAME + '.new')
p = subprocess.run([str(clangxx), '-o', str(tmp), '@' + str(rsp), '-L', str(msys / 'lib'), *LINK_LIBS],
capture_output=True, text=True, env=env, errors='replace')
p = subprocess.run([str(clangxx), *ld, '-o', str(tmp), '@' + str(rsp), *lib, '-L', str(msys / 'lib'),
*LINK_LIBS], capture_output=True, text=True, env=env, errors='replace')
if p.returncode:
print(p.stderr[-4000:], file=sys.stderr)
tmp.unlink(missing_ok=True)
@@ -736,9 +1040,14 @@ def main():
ap.add_argument('--msys', default=r'C:\msys64\ucrt64', help='MSYS2 UCRT64 folder (default C:\\msys64\\ucrt64)')
ap.add_argument('--jobs', type=int, default=os.cpu_count() or 4, help='parallel compiles')
ap.add_argument('--events', action='store_true', help='machine-readable progress lines on stdout ("@@" + JSON)')
ap.add_argument('--make-runtime-lib', metavar='DIR', help='build the prebuilt runtime library of a release into DIR')
ap.add_argument('--obj', help='with --make-runtime-lib: the folder for its objects')
ap.add_argument('--until', choices=STEPS, help=argparse.SUPPRESS) # tests: stop after this step
a = ap.parse_args()
EVENTS = a.events
if a.make_runtime_lib:
make_runtime_lib(a.make_runtime_lib, Path(a.msys), a.jobs, a.obj)
return
given = bool(a.nup) + bool(a.exefs or a.romfs) + bool(a.rebuild)
if given != 1 or (not a.nup and not a.rebuild and not (a.exefs and a.romfs)):
ap.error('give --nup, both --exefs and --romfs, or --rebuild')
@@ -774,29 +1083,60 @@ def main():
state['kit'] = kit_version()
keep_awake()
t0 = time.time()
romfs_job = None
runs = {'exefs': lambda: step_exefs(src, out), 'romfs': lambda: step_romfs(src, out, min(a.jobs, 8)),
'lift': lambda: step_lift(out, cache), 'compile': lambda: step_compile(out, cache, msys, a.jobs),
'lift': lambda: step_lift(out, cache), 'compile': lambda: step_compile(out, cache, msys, a.jobs, romfs_job),
'shaders': lambda: step_shaders(out)}
for name in STEPS:
key = step_key(name, src.identity, out)
if state['steps'].get(name) == key:
say(f'{name}: done before')
emit('step', step=name, state='skip')
else:
# a step that runs makes every step after it run too: their keys hash its output, not its key
for later in STEPS[STEPS.index(name):]:
state['steps'].pop(later, None)
say(f'{name}:')
emit('step', step=name, state='start')
t = time.time()
runs[name]()
state['steps'][name] = step_key(name, src.identity, out)
state_lock = threading.Lock() # the RomFS step records itself from its own thread
def save():
with state_lock:
state_path.write_text(json.dumps(state, indent=1), encoding='utf-8')
say(f'{name}: {time.time() - t:.0f} s')
emit('step', step=name, state='done', seconds=round(time.time() - t, 1))
if name == a.until:
break
state_path.write_text(json.dumps(state, indent=1), encoding='utf-8')
def run(name):
t = time.time()
runs[name]()
with state_lock:
state['steps'][name] = step_key(name, src.identity, out)
save()
say(f'{name}: {time.time() - t:.0f} s')
emit('step', step=name, state='done', seconds=round(time.time() - t, 1))
try:
for name in STEPS:
key = step_key(name, src.identity, out)
if state['steps'].get(name) == key:
say(f'{name}: done before')
emit('step', step=name, state='skip')
else:
# a step that runs makes every step after it run too: their keys hash its output, not its key
with state_lock:
for later in STEPS[STEPS.index(name):]:
state['steps'].pop(later, None)
say(f'{name}:')
emit('step', step=name, state='start')
if name == 'shaders' and romfs_job: # the shaders come out of the RomFS image
romfs_job.join()
if name == 'romfs' and a.until != 'romfs':
# The RomFS image is mostly disk work, the lift and the compile are processor work: they run side
# by side, and only the RomFS index and the files that include it wait for the image
# (step_compile). A first build takes about as long as the longer of the two, not their sum.
romfs_job = Background(lambda: run('romfs'))
else:
run(name)
if romfs_job and romfs_job.failed():
romfs_job.join()
if name == a.until:
break
if romfs_job:
romfs_job.join()
except SystemExit:
# a failed lift or compile: the RomFS image still finishes and is kept, so the next run starts after it
if romfs_job and romfs_job.thread.is_alive():
say('waiting for the RomFS image before stopping')
romfs_job.thread.join()
raise
save()
say(f'\nfinished in {(time.time() - t0) / 60:.1f} min: {out / EXE_NAME}')
emit('end', exe=str(out / EXE_NAME), seconds=round(time.time() - t0, 1), kit=kit_version())