mirror of
https://github.com/tangerineturtle1/NX2navi.git
synced 2026-11-18 17:29:57 -04:00
Redraw the in-game menu with the game's own interface textures, and bring in the launcher's save import and faster builds.
In-game menu: the settings window, its tabs, rows, buttons and cursor, the performance overlay and the Preparing shaders box are drawn with the game's interface textures where the RomFS holds them (runtime/ui/menu_skin.cpp, perf_panel.cpp; plain shapes until they are decoded). Restart joins Continue and Quit, each behind a confirmation; VSync is an on/off switch; DLSS is listed only on NVIDIA GPUs; the menu sounds are quieter; Picture shape is gone and the setting descriptions are shorter; edge cases of input, layout and fonts are fixed. tools/menu_preview.py draws the menu without the game. Launcher: Import Saves takes a save from the game folder's saves/ folder or one the player picks; downloads fall back to Windows' own certificate check when Python's refuses a server; updates are faster; the title bar is drawn in the window and Navi is every icon. tools/make_game.py builds with CLANG64's clang where the launcher installed it, packs the RomFS beside the lift and the compile, takes a prebuilt runtime, and compiles the game's resources without cmd.exe, so a user folder with a space in its name builds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -10,7 +10,8 @@ Steps, each skipped when a run before finished it for the same inputs (state in
|
||||
2. exefs the five ExeFS files, every module's segments unpacked (ZBIC through the player's own `zbic` package)
|
||||
and stored plainly, as the runtime's loader reads them -> <out>/exefs
|
||||
3. romfs every RomFS file, checked against the container's sha256, into one image -> <out>/romfs/romfs.bin and
|
||||
index.json (16-byte aligned, in the byte order of the paths)
|
||||
index.json (16-byte aligned, in the byte order of the paths). When the lift and the compile follow, it
|
||||
runs on a thread of its own beside them: only the RomFS index and romfs.c wait for the image
|
||||
4. lift recomp/gen.py with recomp/patches-v111.txt -> <out>/build-cache/lift, plus the two SDK bring-up fixes
|
||||
5. compile the lifted C and runtime/ with MSYS2's clang, linked into <out>/ocarina.exe; the MSYS2 DLLs it loads,
|
||||
the shader translator and romfs-index.bin beside it
|
||||
@@ -20,8 +21,16 @@ Steps, each skipped when a run before finished it for the same inputs (state in
|
||||
The exefs and romfs steps depend only on the game files; lift and compile depend on this kit's code too. A newer kit
|
||||
therefore lifts and compiles again but keeps the unpacked game files, and --rebuild needs no game files at all. The
|
||||
compile step keeps every object with the dependency list clang wrote for it (build-cache/obj/objects.json) and
|
||||
compiles a file again only when it, a header it includes, the flags or the compiler changed. The new ocarina.exe
|
||||
replaces the old one only once it links, so a failed build leaves the last good one in place.
|
||||
compiles a file again only when it, a header it includes, the flags or the compiler changed; the kit's own folder
|
||||
is not part of that key, so a kit in another folder (the launcher keeps each version in its own) reuses the objects.
|
||||
A kit with lib/libnx2navirt.a built by --make-runtime-lib for the same runtime/ and the same compiler links the
|
||||
runtime from it and compiles only the runtime files that include headers written per install. The new ocarina.exe
|
||||
replaces the old one only once it links (with lld when the toolchain has it), so a failed build leaves the last good
|
||||
one in place. Compiles run on every logical processor, the largest files first. The lifted code is built for this
|
||||
PC's x86-64 level (cpu_level), with MSYS2 CLANG64's clang when it sits beside the UCRT64 folder and reports the same
|
||||
version (fast_clang: the same objects, about a fifth faster).
|
||||
|
||||
python tools/make_game.py --make-runtime-lib <kit>/lib --msys <ucrt64> # the release's prebuilt runtime
|
||||
|
||||
Nothing is written outside --out. The game folder then holds only what this PC built from the player's files: never
|
||||
share it. Needs Python 3.10+ with numpy, lz4, zstandard and zbic (pip), and MSYS2 UCRT64's clang and libraries
|
||||
@@ -43,6 +52,7 @@ import shutil
|
||||
import struct
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
@@ -76,11 +86,14 @@ LIFT_FIXES = [
|
||||
LINK_LIBS = ['-lvulkan-1', '-lSDL3', '-lopus', '-lvpx', '-l:libdav1d.a', '-lzstd', '-l:libatomic.a', '-lbcrypt',
|
||||
'-limm32', '-luuid', '-lole32', '-luser32', '-lshell32', '-lgdi32', '-ladvapi32', '-lwinmm']
|
||||
SYSTEM_DLLS_SKIP = {'vulkan-1.dll'} # the graphics driver's loader, never a copy
|
||||
# GNU windres's preprocessor, named explicitly: left to find it, windres puts the full path of the gcc beside it on a
|
||||
# cmd.exe command line, and once that path and the .rc's both hold a space, cmd.exe drops the outer quotes and splits
|
||||
# the command at the first space. gcc is then found on PATH (the toolchain's bin first); the arguments are windres's own.
|
||||
WINDRES_PP = ['--preprocessor=gcc', '--preprocessor-arg=-E', '--preprocessor-arg=-xc', '--preprocessor-arg=-DRC_INVOKED']
|
||||
EXE_NAME = 'ocarina.exe'
|
||||
# Headers this build writes per install (gen.py, romfs_index.py): a runtime source that includes one, directly or
|
||||
# through a runtime header, is compiled on the player's PC; every other runtime source can come prebuilt.
|
||||
GENERATED_HEADERS = ('layout.h', 'patches.h', 'romfs_config.h')
|
||||
RUNTIME_LIB = 'libnx2navirt.a'
|
||||
RUNTIME_PATTERNS = ('*.c', '*.cpp', 'platform/*.c', 'ui/*.cpp', 'third_party/imgui/*.cpp',
|
||||
'third_party/imgui/backends/*.cpp')
|
||||
INCLUDE = re.compile(r'^[ \t]*#[ \t]*include[ \t]+"([^"]+)"', re.M)
|
||||
|
||||
# Runs a script the way `python script.py` does, with the script's folder first on sys.path. The launcher's
|
||||
# embeddable Python leaves the script folder off sys.path (its ._pth file), so every script this build starts goes
|
||||
@@ -89,24 +102,53 @@ RUN_SCRIPT = ('import os, runpy, sys; s = sys.argv[1]; sys.argv = sys.argv[1:];
|
||||
'sys.path.insert(0, os.path.dirname(os.path.abspath(s))); runpy.run_path(s, run_name="__main__")')
|
||||
|
||||
EVENTS = False
|
||||
OUT_LOCK = threading.Lock() # the RomFS step prints from its own thread (main)
|
||||
|
||||
|
||||
def say(msg):
|
||||
print(msg, flush=True)
|
||||
with OUT_LOCK:
|
||||
print(msg, flush=True)
|
||||
|
||||
|
||||
def emit(ev, **fields):
|
||||
"""One "@@{json}" line for the launcher (--events); nothing otherwise."""
|
||||
if EVENTS:
|
||||
print('@@' + json.dumps({'ev': ev, **fields}, ensure_ascii=False), flush=True)
|
||||
line = '@@' + json.dumps({'ev': ev, **fields}, ensure_ascii=False)
|
||||
with OUT_LOCK:
|
||||
print(line, flush=True)
|
||||
|
||||
|
||||
def fail(msg):
|
||||
emit('error', message=msg)
|
||||
print(f'\nerror: {msg}', file=sys.stderr, flush=True)
|
||||
with OUT_LOCK:
|
||||
print(f'\nerror: {msg}', file=sys.stderr, flush=True)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
class Background:
|
||||
"""One step on a thread of its own. join() waits for it and raises its failure (fail's SystemExit included) in
|
||||
the caller; failed() tells the caller early, so it can stop work that would be thrown away."""
|
||||
|
||||
def __init__(self, fn):
|
||||
self.exc = None
|
||||
self.thread = threading.Thread(target=self._run, args=(fn,), name='romfs', daemon=True)
|
||||
self.thread.start()
|
||||
|
||||
def _run(self, fn):
|
||||
try:
|
||||
fn()
|
||||
except BaseException as e: # noqa: B036 - handed to join()
|
||||
self.exc = e
|
||||
|
||||
def failed(self):
|
||||
return self.exc is not None
|
||||
|
||||
def join(self):
|
||||
self.thread.join()
|
||||
if self.exc is not None:
|
||||
raise self.exc
|
||||
|
||||
|
||||
def py_cmd(script, *args):
|
||||
"""The command line that runs a Python script of this kit with this interpreter (see RUN_SCRIPT)."""
|
||||
flags = ['-I'] if sys.flags.isolated else []
|
||||
@@ -317,11 +359,11 @@ def step_romfs(src, out, jobs):
|
||||
done += size
|
||||
emit('progress', step='romfs', done=done, total=total, unit='bytes')
|
||||
if start // batch % 40 == 0:
|
||||
say(f' {done / 1e9:5.1f} of {total / 1e9:.1f} GB, {time.time() - t0:.0f} s')
|
||||
say(f' RomFS {done / 1e9:5.1f} of {total / 1e9:.1f} GB, {time.time() - t0:.0f} s')
|
||||
part.replace(dst / 'romfs.bin')
|
||||
index = {'status': 'pass', 'romfs_size': pos, 'romfs_sha256': h.hexdigest(), 'files': entries}
|
||||
(dst / 'index.json').write_text(json.dumps(index), encoding='utf-8')
|
||||
say(f' image {pos / 1e9:.2f} GB in {time.time() - t0:.0f} s')
|
||||
say(f' RomFS image {pos / 1e9:.2f} GB in {time.time() - t0:.0f} s')
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- lift
|
||||
@@ -491,17 +533,33 @@ def parse_deps(text):
|
||||
|
||||
|
||||
class ObjectCache:
|
||||
"""Which objects are current: per object the key of its command and source, and the sha256 of every file its
|
||||
dependency list names (build-cache/obj/objects.json)."""
|
||||
"""Which objects are current: per object the key of its command and source, and the sha256 (with size and
|
||||
modification time, to skip hashing an untouched file) of every file its dependency list names
|
||||
(build-cache/obj/objects.json). Paths under a root are kept relative to it ({'<kit>': KIT}), so the same files
|
||||
in a kit at another place still count as the same."""
|
||||
|
||||
def __init__(self, path):
|
||||
def __init__(self, path, roots=None):
|
||||
self.path = path
|
||||
self.roots = {k: os.path.normcase(os.path.abspath(v)) for k, v in (roots or {}).items()}
|
||||
try:
|
||||
self.entries = json.loads(path.read_text(encoding='utf-8')) if path.is_file() else {}
|
||||
except (OSError, ValueError):
|
||||
self.entries = {}
|
||||
self._sha = {}
|
||||
|
||||
def norm(self, p):
|
||||
a = os.path.normcase(os.path.abspath(p))
|
||||
for name, root in self.roots.items():
|
||||
if a == root or a.startswith(root + os.sep):
|
||||
return name + a[len(root):].replace(os.sep, '/')
|
||||
return a
|
||||
|
||||
def real(self, p):
|
||||
for name, root in self.roots.items():
|
||||
if p.startswith(name):
|
||||
return root + p[len(name):].replace('/', os.sep)
|
||||
return p
|
||||
|
||||
def file_sha(self, p):
|
||||
p = os.path.normcase(os.path.abspath(p))
|
||||
if p not in self._sha:
|
||||
@@ -511,18 +569,37 @@ class ObjectCache:
|
||||
self._sha[p] = None
|
||||
return self._sha[p]
|
||||
|
||||
def _same(self, p, rec):
|
||||
path = self.real(p)
|
||||
if isinstance(rec, str): # a cache without sizes and times
|
||||
return self.file_sha(path) == rec
|
||||
digest, size, mtime = rec
|
||||
try:
|
||||
st = os.stat(path)
|
||||
except OSError:
|
||||
return False
|
||||
if st.st_size == size and st.st_mtime_ns == mtime:
|
||||
return True
|
||||
return st.st_size == size and self.file_sha(path) == digest
|
||||
|
||||
def current(self, obj, key):
|
||||
e = self.entries.get(obj.name)
|
||||
return (e is not None and e.get('key') == key and obj.is_file()
|
||||
and all(self.file_sha(p) == h for p, h in e.get('deps', {}).items()))
|
||||
and all(self._same(p, rec) for p, rec in e.get('deps', {}).items()))
|
||||
|
||||
def record(self, obj, key, dep_file, src):
|
||||
try:
|
||||
deps = parse_deps(Path(dep_file).read_text(encoding='utf-8', errors='replace'))
|
||||
except OSError:
|
||||
deps = []
|
||||
deps = deps or [str(src)]
|
||||
self.entries[obj.name] = {'key': key, 'deps': {p: self.file_sha(p) for p in deps}}
|
||||
out = {}
|
||||
for p in deps or [str(src)]:
|
||||
try:
|
||||
st = os.stat(p)
|
||||
out[self.norm(p)] = [self.file_sha(p), st.st_size, st.st_mtime_ns]
|
||||
except OSError:
|
||||
out[self.norm(p)] = [None, -1, -1]
|
||||
self.entries[obj.name] = {'key': key, 'deps': out}
|
||||
|
||||
def forget(self, obj):
|
||||
self.entries.pop(obj.name, None)
|
||||
@@ -539,22 +616,31 @@ def compiler_id(clang, env):
|
||||
return sha256_bytes(f'{p.stdout}\n{st.st_size}\n{int(st.st_mtime)}'.encode())
|
||||
|
||||
|
||||
def compile_all(srcs, cmd_for, cache, jobs, env, progress=None, salt=''):
|
||||
def compile_all(srcs, cmd_for, cache, jobs, env, progress=None, salt='', stop=None):
|
||||
"""Compile each (source, object) whose object is not current in cache. cmd_for(src, dst) -> argv without the
|
||||
dependency-file options; salt identifies the compiler. Returns (compiled, reused, failures) with failures as
|
||||
(src, stderr tail)."""
|
||||
(src, stderr tail). The cache's roots stand for their paths in the flags too. The largest sources start first:
|
||||
compile time grows with size, and a big file that started last would run on alone after the others finished.
|
||||
stop() true makes the compiles not yet started end at once, unrecorded (a step beside this one failed)."""
|
||||
todo, keys = [], {}
|
||||
for src, dst in srcs:
|
||||
cmd = cmd_for(src, dst)
|
||||
flags = [c for c in cmd if c not in (str(src), str(dst))]
|
||||
flags = [c for c in cmd[1:] if c not in (str(src), str(dst))]
|
||||
for name, root in cache.roots.items():
|
||||
flags = [name + os.path.normcase(f)[len(root):].replace(os.sep, '/') if os.path.normcase(f).startswith(root)
|
||||
else f for f in flags]
|
||||
flags = [os.path.basename(cmd[0])] + flags
|
||||
keys[dst] = sha256_bytes(json.dumps([salt, flags, cache.file_sha(src)]).encode())
|
||||
if not cache.current(dst, keys[dst]):
|
||||
todo.append((src, dst))
|
||||
reused = len(srcs) - len(todo)
|
||||
todo.sort(key=lambda job: job[0].stat().st_size, reverse=True)
|
||||
|
||||
def cc(job):
|
||||
src, dst = job
|
||||
dep = dst.with_suffix('.d')
|
||||
if stop and stop():
|
||||
return src, dst, dep, None, ''
|
||||
cmd = cmd_for(src, dst)
|
||||
cmd[1:1] = ['-MMD', '-MF', str(dep)]
|
||||
p = subprocess.run(cmd, capture_output=True, text=True, env=env, errors='replace')
|
||||
@@ -566,6 +652,8 @@ def compile_all(srcs, cmd_for, cache, jobs, env, progress=None, salt=''):
|
||||
with concurrent.futures.ThreadPoolExecutor(max(1, jobs)) as ex:
|
||||
for src, dst, dep, rc, err in ex.map(cc, todo):
|
||||
done += 1
|
||||
if rc is None: # not started (stop)
|
||||
continue
|
||||
if rc:
|
||||
bad.append((src, err))
|
||||
cache.forget(dst)
|
||||
@@ -577,33 +665,236 @@ def compile_all(srcs, cmd_for, cache, jobs, env, progress=None, salt=''):
|
||||
return len(todo), reused, bad
|
||||
|
||||
|
||||
def step_compile(out, cache_dir, msys, jobs):
|
||||
def runtime_sources():
|
||||
"""(source, object name) of every runtime file the Windows build compiles."""
|
||||
rt = KIT / 'runtime'
|
||||
out = []
|
||||
for pat in RUNTIME_PATTERNS:
|
||||
for p in sorted(rt.glob(pat)):
|
||||
if p.stem.endswith('_sdl') or p.name == 'linux.c':
|
||||
continue
|
||||
out.append((p, 'rt_' + p.relative_to(rt).as_posix().replace('/', '_').rsplit('.', 1)[0] + '.o'))
|
||||
return out
|
||||
|
||||
|
||||
def per_install_names(generated=GENERATED_HEADERS):
|
||||
"""The basenames of the generated headers and of every runtime header that includes one, at any depth."""
|
||||
rt = KIT / 'runtime'
|
||||
headers = {}
|
||||
for p in list(rt.rglob('*.h')) + list(rt.rglob('*.inc')):
|
||||
headers[p.name] = {os.path.basename(i) for i in INCLUDE.findall(p.read_text(encoding='utf-8', errors='replace'))}
|
||||
tainted = set(generated)
|
||||
grew = True
|
||||
while grew:
|
||||
grew = False
|
||||
for name, incs in headers.items():
|
||||
if name not in tainted and incs & tainted:
|
||||
tainted.add(name)
|
||||
grew = True
|
||||
return tainted
|
||||
|
||||
|
||||
def per_install(src, tainted):
|
||||
incs = {os.path.basename(i) for i in INCLUDE.findall(Path(src).read_text(encoding='utf-8', errors='replace'))}
|
||||
return bool(incs & tainted)
|
||||
|
||||
|
||||
def clang_version(clang, env):
|
||||
p = subprocess.run([str(clang), '--version'], capture_output=True, text=True, env=env)
|
||||
return (p.stdout.splitlines() or [''])[0].strip()
|
||||
|
||||
|
||||
def runtime_digest():
|
||||
return tree_digest(KIT / 'runtime', ('**/*',))
|
||||
|
||||
|
||||
# The x86-64 levels the lifted code is built and tested for (docs/building.md, "Checks"), as the macros clang defines
|
||||
# for a CPU that has them.
|
||||
CPU_V2 = {'__SSE3__', '__SSSE3__', '__SSE4_1__', '__SSE4_2__', '__POPCNT__', '__LAHF_SAHF__',
|
||||
'__GCC_HAVE_SYNC_COMPARE_AND_SWAP_16'}
|
||||
CPU_V3 = CPU_V2 | {'__AVX__', '__AVX2__', '__BMI__', '__BMI2__', '__F16C__', '__FMA__', '__LZCNT__', '__MOVBE__',
|
||||
'__XSAVE__'}
|
||||
|
||||
|
||||
def cpu_level(clang, env):
|
||||
"""This PC's x86-64 level for the lifted code: 3 (AVX2, FMA, BMI2), 2 (SSE4.2, POPCNT) or 0 when clang cannot
|
||||
tell. The game runs on the PC that builds it, so it takes this CPU's level, as tools/build_game.py --cpu does by
|
||||
hand. Without one the lifted code would be generic x86-64: every fused multiply-add of the game a call to
|
||||
runtime/fma.c's software fma, and a level the differential tests do not cover."""
|
||||
p = subprocess.run([str(clang), '-march=native', '-E', '-dM', '-x', 'c', os.devnull], capture_output=True,
|
||||
text=True, env=env, errors='replace')
|
||||
have = set(re.findall(r'^#define (__\w+) ', p.stdout, re.M)) if p.returncode == 0 else set()
|
||||
return 3 if CPU_V3 <= have else 2 if CPU_V2 <= have else 0
|
||||
|
||||
|
||||
def clang_ident(clang, env):
|
||||
"""(version, target) a clang reports, or None when it does not run."""
|
||||
try:
|
||||
p = subprocess.run([str(clang), '--version'], capture_output=True, text=True, env=env, errors='replace')
|
||||
except OSError:
|
||||
return None
|
||||
v, t = re.search(r'clang version (\S+)', p.stdout), re.search(r'^Target: (\S+)', p.stdout, re.M)
|
||||
return (v.group(1), t.group(1)) if p.returncode == 0 and v and t else None
|
||||
|
||||
|
||||
def fast_clang(msys, clang, env):
|
||||
"""MSYS2's CLANG64 clang beside the UCRT64 one (msys64/clang64, which the launcher installs): the same LLVM release
|
||||
built with profile-guided optimisation, about a fifth faster on the lifted code. With UCRT64's headers
|
||||
(--sysroot) it writes the same objects as UCRT64's clang, byte for byte but the COFF timestamp; it is used only
|
||||
when it reports the same version and target. None otherwise."""
|
||||
fast = Path(msys).parent / 'clang64' / 'bin' / 'clang.exe'
|
||||
if not fast.is_file():
|
||||
return None
|
||||
mine = clang_ident(clang, env)
|
||||
return fast if mine and clang_ident(fast, env) == mine else None
|
||||
|
||||
|
||||
def runtime_cmd(clang, clangxx, inc, level=0, fast=None, msys=None):
|
||||
"""fast: fast_clang's compiler for the lifted code, with msys as its sysroot."""
|
||||
def cmd_for(src, dst):
|
||||
cpp = src.suffix in ('.cpp', '.cxx')
|
||||
# the lifted code at -O1 (its size makes -O2 slow to build), the runtime at -O2; neither may assume strict
|
||||
# aliasing: the lifted code reaches guest memory through casts of one address to several types
|
||||
# (tools/build_game.py's flags). The lifted code is built for this PC's level (cpu_level); the runtime
|
||||
# stays generic, so the prebuilt one serves every PC (its vectors are 16 bytes, passed alike at every level).
|
||||
guest = dst.name.startswith(('main_', 'sdk_', 'subsdk0_'))
|
||||
march = [f'-march=x86-64-v{level}'] if guest and level else []
|
||||
cc = [str(fast), f'--sysroot={msys}'] if guest and fast else [str(clangxx if cpp else clang)]
|
||||
return [*cc, '-c', '-std=gnu++17' if cpp else '-std=gnu11', '-O1' if guest else '-O2',
|
||||
*march, '-fno-strict-aliasing', '-w', *inc, '-o', str(dst), str(src)]
|
||||
return cmd_for
|
||||
|
||||
|
||||
def make_runtime_lib(dest, msys, jobs, obj=None):
|
||||
"""--make-runtime-lib: compile every runtime source that does not include a per-install header into
|
||||
<dest>/libnx2navirt.a, with runtime-lib.json naming the runtime/ tree and the compiler it was built from. The
|
||||
objects stay in obj (default <dest>/obj), which a release keeps out of what it ships."""
|
||||
dest = Path(dest).resolve()
|
||||
dest.mkdir(parents=True, exist_ok=True)
|
||||
obj = Path(obj).resolve() if obj else dest / 'obj'
|
||||
obj.mkdir(parents=True, exist_ok=True)
|
||||
clang, clangxx = msys / 'bin' / 'clang.exe', msys / 'bin' / 'clang++.exe'
|
||||
env = dict(os.environ, PATH=str(msys / 'bin') + os.pathsep + os.environ.get('PATH', ''))
|
||||
rt = KIT / 'runtime'
|
||||
tainted = per_install_names()
|
||||
srcs = [(p, obj / name) for p, name in runtime_sources() if not per_install(p, tainted)]
|
||||
inc = ['-I', str(rt), '-I', str(rt / 'third_party' / 'imgui'), '-I', str(msys / 'include' / 'vma')]
|
||||
cache = ObjectCache(obj / 'objects.json', {'<kit>': KIT})
|
||||
t0 = time.time()
|
||||
compiled, reused, bad = compile_all(srcs, runtime_cmd(clang, clangxx, inc), cache, jobs, env,
|
||||
lambda d, t: (d % 20 == 0 or d == t) and say(f' {d} of {t}'),
|
||||
salt=compiler_id(clang, env))
|
||||
if bad:
|
||||
for src, err in bad[:3]:
|
||||
print(f'--- {src}\n{err}', file=sys.stderr)
|
||||
fail(f'{len(bad)} runtime files did not compile')
|
||||
lib = dest / RUNTIME_LIB
|
||||
lib.unlink(missing_ok=True)
|
||||
rsp = dest / 'ar.rsp'
|
||||
rsp.write_text('\n'.join('"%s"' % o.as_posix() for _, o in srcs), encoding='utf-8')
|
||||
ar = os.environ.get('NX2NAVI_AR') or str(msys / 'bin' / 'ar.exe')
|
||||
if subprocess.run([ar, 'rcs', str(lib), '@' + str(rsp)], env=env).returncode:
|
||||
fail('the runtime library could not be archived')
|
||||
meta = {'runtime': runtime_digest(), 'compiler': clang_version(clang, env),
|
||||
'objects': sorted(o.name for _, o in srcs)}
|
||||
(dest / 'runtime-lib.json').write_text(json.dumps(meta, indent=1), encoding='utf-8')
|
||||
rsp.unlink(missing_ok=True)
|
||||
say(f'{lib}: {len(srcs)} runtime files ({compiled} compiled, {reused} kept), {lib.stat().st_size / 1e6:.1f} MB, '
|
||||
f'{time.time() - t0:.0f} s')
|
||||
|
||||
|
||||
def prebuilt_runtime(clang, env):
|
||||
"""(library, object names it holds) when the kit's lib/ matches this runtime/ and this compiler, else None."""
|
||||
lib, meta = KIT / 'lib' / RUNTIME_LIB, KIT / 'lib' / 'runtime-lib.json'
|
||||
try:
|
||||
m = json.loads(meta.read_text(encoding='utf-8'))
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
if not lib.is_file() or m.get('runtime') != runtime_digest() or m.get('compiler') != clang_version(clang, env):
|
||||
say(' the prebuilt runtime does not match this kit or compiler: compiling runtime/')
|
||||
return None
|
||||
return lib, set(m.get('objects', []))
|
||||
|
||||
|
||||
def romfs_index(out, gen):
|
||||
if subprocess.run(py_cmd(KIT / 'tools' / 'romfs_index.py', '--romfs', out / 'romfs', '--out', gen)).returncode:
|
||||
fail('the RomFS index could not be written')
|
||||
|
||||
|
||||
def windres_command(msys, rco):
|
||||
"""MSYS2's windres on nx2navi.rc, run with runtime/ as the working directory (compile_resources says why)."""
|
||||
pp = ('-E', '-xc', '-DRC_INVOKED') # windres's own preprocessor flags, which --preprocessor replaces
|
||||
return [str(msys / 'bin' / 'windres.exe'), '--use-temp-file', f'--preprocessor={msys / "bin" / "gcc.exe"}',
|
||||
*(f'--preprocessor-arg={a}' for a in pp), '-O', 'coff', '-i', 'nx2navi.rc', '-o', str(rco)]
|
||||
|
||||
|
||||
def compile_resources(msys, rt, obj, env):
|
||||
"""runtime/nx2navi.rc into obj/nx2navi_rc.o: the manifest of the game's exe (per-monitor DPI awareness, so the
|
||||
picture keeps the monitor's real size rather than one Windows stretches) and its icon.
|
||||
|
||||
windres preprocesses the .rc with gcc through one command line it writes itself, quoting each path that holds a
|
||||
space. By default it hands that line to cmd.exe /c, which drops the first and the last double quote of a line
|
||||
holding more than two: gcc's quoted path loses its opening quote, and under a user folder with a space in its
|
||||
name (C:\\Users\\Ali Sayed) cmd.exe stops at that space ("'C:\\Users\\Ali' is not recognized").
|
||||
|
||||
With --use-temp-file windres starts gcc itself instead, no shell involved: it splits the line into arguments and
|
||||
passes each one to gcc intact. Its splitter reads a quoted first word right, but it cuts a quoted -I path at its
|
||||
spaces and reads past the end of the line after a quoted last word, so the line holds no path but gcc's, its
|
||||
first word: the .rc goes by its bare name from runtime/ as the working directory, where windres also finds the
|
||||
manifest and the icon, and there is no -I. windres's temporary files go to a folder of the build cache, deleted
|
||||
afterwards (it leaves two of them behind on every run).
|
||||
|
||||
The object is written under another name and replaces the old one only when its bytes differ, so an unchanged
|
||||
resource keeps its time and does not make the link key change (a rebuild with nothing new does not link again)."""
|
||||
rco, new = obj / 'nx2navi_rc.o', obj / 'nx2navi_rc.o.new'
|
||||
tmp = obj / 'windres-tmp'
|
||||
shutil.rmtree(tmp, ignore_errors=True)
|
||||
tmp.mkdir()
|
||||
try:
|
||||
r = subprocess.run(windres_command(msys, new), cwd=rt, env=dict(env, TMP=str(tmp), TEMP=str(tmp)),
|
||||
capture_output=True, text=True, errors='replace')
|
||||
finally:
|
||||
shutil.rmtree(tmp, ignore_errors=True)
|
||||
if r.returncode:
|
||||
new.unlink(missing_ok=True)
|
||||
fail(f'windres failed on runtime/nx2navi.rc: {r.stderr.strip()[:400]}')
|
||||
if rco.is_file() and rco.read_bytes() == new.read_bytes():
|
||||
new.unlink()
|
||||
else:
|
||||
os.replace(new, rco)
|
||||
return rco
|
||||
|
||||
|
||||
def step_compile(out, cache_dir, msys, jobs, romfs_job=None):
|
||||
"""romfs_job: the RomFS step still running beside this one (a first build). Everything that does not include
|
||||
romfs_config.h compiles meanwhile; the RomFS index and the few files that include it follow once it is done."""
|
||||
lift, gen, obj = cache_dir / 'lift', cache_dir / 'gen', cache_dir / 'obj'
|
||||
gen.mkdir(parents=True, exist_ok=True)
|
||||
obj.mkdir(parents=True, exist_ok=True)
|
||||
rt = KIT / 'runtime'
|
||||
clang, clangxx = msys / 'bin' / 'clang.exe', msys / 'bin' / 'clang++.exe'
|
||||
env = dict(os.environ, PATH=str(msys / 'bin') + os.pathsep + os.environ.get('PATH', ''))
|
||||
if subprocess.run(py_cmd(KIT / 'tools' / 'romfs_index.py', '--romfs', out / 'romfs', '--out', gen)).returncode:
|
||||
fail('the RomFS index could not be written')
|
||||
if not romfs_job:
|
||||
romfs_index(out, gen)
|
||||
srcs = [(p, obj / f'{mod}_{p.stem}.o') for mod in ('main', 'sdk', 'subsdk0') for p in sorted((lift / mod).glob('*.c'))]
|
||||
for pat in ('*.c', '*.cpp', 'platform/*.c', 'ui/*.cpp', 'third_party/imgui/*.cpp', 'third_party/imgui/backends/*.cpp'):
|
||||
for p in sorted(rt.glob(pat)):
|
||||
if p.stem.endswith('_sdl') or p.name == 'linux.c':
|
||||
continue
|
||||
srcs.append((p, obj / ('rt_' + p.relative_to(rt).as_posix().replace('/', '_').rsplit('.', 1)[0] + '.o')))
|
||||
prebuilt = prebuilt_runtime(clang, env)
|
||||
runtime = [(p, obj / name) for p, name in runtime_sources()]
|
||||
if prebuilt:
|
||||
runtime = [(p, o) for p, o in runtime if o.name not in prebuilt[1]]
|
||||
say(f' the runtime comes prebuilt; {len(runtime)} runtime files are compiled here')
|
||||
srcs += runtime
|
||||
srcs += [(modules_c(lift, out, gen), obj / 'modules.o'), (gen / 'shim.c', obj / 'shim.o')]
|
||||
inc = ['-I', str(rt), '-I', str(lift), '-I', str(gen), '-I', str(rt / 'third_party' / 'imgui'),
|
||||
'-I', str(msys / 'include' / 'vma')]
|
||||
cid = compiler_id(clang, env)
|
||||
|
||||
def cmd_for(src, dst):
|
||||
cpp = src.suffix in ('.cpp', '.cxx')
|
||||
# the lifted code at -O1 (its size makes -O2 slow to build), the runtime at -O2; neither may assume strict aliasing:
|
||||
# the lifted code reaches guest memory through casts of one address to several types (tools/build_game.py's flags)
|
||||
guest = dst.name.startswith(('main_', 'sdk_', 'subsdk0_'))
|
||||
return [str(clangxx if cpp else clang), '-c', '-std=gnu++17' if cpp else '-std=gnu11', '-O1' if guest else '-O2',
|
||||
'-fno-strict-aliasing', '-w', *inc, '-o', str(dst), str(src)]
|
||||
level = cpu_level(clang, env)
|
||||
say(f' the lifted code is built for x86-64-v{level}, this processor\'s level' if level else
|
||||
' the lifted code is built for generic x86-64: the processor\'s level could not be read')
|
||||
fast = fast_clang(msys, clang, env)
|
||||
if fast:
|
||||
cid += compiler_id(fast, env)
|
||||
say(f' the lifted code compiles with {fast}')
|
||||
cmd_for = runtime_cmd(clang, clangxx, inc, level, fast, msys)
|
||||
|
||||
t0 = time.time()
|
||||
last = [0.0]
|
||||
@@ -617,29 +908,42 @@ def step_compile(out, cache_dir, msys, jobs):
|
||||
say(f' compiled {done} of {total}, {now - t0:.0f} s')
|
||||
|
||||
say(f' {len(srcs)} files on {jobs} threads')
|
||||
cache = ObjectCache(obj / 'objects.json')
|
||||
compiled, reused, bad = compile_all(srcs, cmd_for, cache, jobs, env, progress, salt=cid)
|
||||
cache = ObjectCache(obj / 'objects.json', {'<kit>': KIT})
|
||||
late = []
|
||||
if romfs_job: # the lifted chunks include only cpu.h and simd.h: only the runtime and generated files are read
|
||||
tainted = per_install_names(('romfs_config.h',))
|
||||
late = [(s, o) for s, o in runtime + srcs[-2:] if per_install(s, tainted)]
|
||||
say(f' {len(late)} files wait for the RomFS index: {", ".join(s.name for s, _ in late)}')
|
||||
early = [j for j in srcs if j not in late]
|
||||
compiled, reused, bad = compile_all(early, cmd_for, cache, jobs, env, lambda d, t: progress(d, len(srcs)),
|
||||
salt=cid, stop=romfs_job.failed if romfs_job else None)
|
||||
if romfs_job:
|
||||
if romfs_job.thread.is_alive():
|
||||
say(' waiting for the RomFS image')
|
||||
romfs_job.join() # raises the RomFS step's failure; a failed compile still keeps the finished image
|
||||
if not bad:
|
||||
romfs_index(out, gen)
|
||||
c2, r2, bad = compile_all(late, cmd_for, cache, jobs, env,
|
||||
lambda d, t: progress(len(early) + d, len(srcs)), salt=cid)
|
||||
compiled, reused = compiled + c2, reused + r2
|
||||
say(f' {compiled} compiled, {reused} unchanged since the last build')
|
||||
if bad:
|
||||
for src, err in bad[:3]:
|
||||
print(f'--- {src}\n{err}', file=sys.stderr)
|
||||
fail(f'{len(bad)} files did not compile')
|
||||
|
||||
# the manifest (per-monitor DPI awareness: the picture at the monitor's real size, not one Windows stretches) and the
|
||||
# icon, runtime/nx2navi.rc through MSYS2's windres. windres runs its preprocessor through cmd.exe: run from the
|
||||
# runtime folder with relative names and gcc by bare name (WINDRES_PP), that command line holds no path, so a space
|
||||
# in the install folder (a user name) cannot split it.
|
||||
rco = (obj / 'nx2navi_rc.o').resolve()
|
||||
r = subprocess.run([str(msys / 'bin' / 'windres.exe'), *WINDRES_PP, '-I', '.', '-O', 'coff', '-i', 'nx2navi.rc',
|
||||
'-o', str(rco)], capture_output=True, text=True, env=env, errors='replace', cwd=rt)
|
||||
if r.returncode:
|
||||
fail(f'windres failed on runtime/nx2navi.rc: {r.stderr.strip()[:400]}')
|
||||
rco = compile_resources(msys, rt, obj, env)
|
||||
objs = [o for _, o in srcs] + [rco]
|
||||
exe = out / EXE_NAME
|
||||
rsp = cache_dir / 'link.rsp'
|
||||
rsp.write_text('\n'.join('"%s"' % o.as_posix() for o in objs), encoding='utf-8')
|
||||
link_key = sha256_bytes(json.dumps([LINK_LIBS, cid] + [[o.name, o.stat().st_size, o.stat().st_mtime_ns]
|
||||
for o in objs]).encode())
|
||||
# lld links several times faster than GNU ld; the stack matches tools/build_game.py's build
|
||||
ld = ['-fuse-ld=lld', '-Wl,--stack,16777216'] if (msys / 'bin' / 'ld.lld.exe').is_file() else []
|
||||
lib = ['-Wl,--whole-archive', str(prebuilt[0]), '-Wl,--no-whole-archive'] if prebuilt else []
|
||||
stamp = [[o.name, o.stat().st_size, o.stat().st_mtime_ns] for o in objs]
|
||||
if prebuilt:
|
||||
stamp.append([prebuilt[0].name, prebuilt[0].stat().st_size, sha256_file(prebuilt[0])])
|
||||
link_key = sha256_bytes(json.dumps([LINK_LIBS, ld, cid] + stamp).encode())
|
||||
state_link = cache_dir / 'link.key'
|
||||
if exe.is_file() and state_link.is_file() and state_link.read_text(encoding='utf-8') == link_key:
|
||||
say(' ocarina.exe is current, not linked again')
|
||||
@@ -647,8 +951,8 @@ def step_compile(out, cache_dir, msys, jobs):
|
||||
emit('phase', step='compile', phase='link')
|
||||
say(' linking')
|
||||
tmp = out / (EXE_NAME + '.new')
|
||||
p = subprocess.run([str(clangxx), '-o', str(tmp), '@' + str(rsp), '-L', str(msys / 'lib'), *LINK_LIBS],
|
||||
capture_output=True, text=True, env=env, errors='replace')
|
||||
p = subprocess.run([str(clangxx), *ld, '-o', str(tmp), '@' + str(rsp), *lib, '-L', str(msys / 'lib'),
|
||||
*LINK_LIBS], capture_output=True, text=True, env=env, errors='replace')
|
||||
if p.returncode:
|
||||
print(p.stderr[-4000:], file=sys.stderr)
|
||||
tmp.unlink(missing_ok=True)
|
||||
@@ -736,9 +1040,14 @@ def main():
|
||||
ap.add_argument('--msys', default=r'C:\msys64\ucrt64', help='MSYS2 UCRT64 folder (default C:\\msys64\\ucrt64)')
|
||||
ap.add_argument('--jobs', type=int, default=os.cpu_count() or 4, help='parallel compiles')
|
||||
ap.add_argument('--events', action='store_true', help='machine-readable progress lines on stdout ("@@" + JSON)')
|
||||
ap.add_argument('--make-runtime-lib', metavar='DIR', help='build the prebuilt runtime library of a release into DIR')
|
||||
ap.add_argument('--obj', help='with --make-runtime-lib: the folder for its objects')
|
||||
ap.add_argument('--until', choices=STEPS, help=argparse.SUPPRESS) # tests: stop after this step
|
||||
a = ap.parse_args()
|
||||
EVENTS = a.events
|
||||
if a.make_runtime_lib:
|
||||
make_runtime_lib(a.make_runtime_lib, Path(a.msys), a.jobs, a.obj)
|
||||
return
|
||||
given = bool(a.nup) + bool(a.exefs or a.romfs) + bool(a.rebuild)
|
||||
if given != 1 or (not a.nup and not a.rebuild and not (a.exefs and a.romfs)):
|
||||
ap.error('give --nup, both --exefs and --romfs, or --rebuild')
|
||||
@@ -774,29 +1083,60 @@ def main():
|
||||
state['kit'] = kit_version()
|
||||
keep_awake()
|
||||
t0 = time.time()
|
||||
romfs_job = None
|
||||
runs = {'exefs': lambda: step_exefs(src, out), 'romfs': lambda: step_romfs(src, out, min(a.jobs, 8)),
|
||||
'lift': lambda: step_lift(out, cache), 'compile': lambda: step_compile(out, cache, msys, a.jobs),
|
||||
'lift': lambda: step_lift(out, cache), 'compile': lambda: step_compile(out, cache, msys, a.jobs, romfs_job),
|
||||
'shaders': lambda: step_shaders(out)}
|
||||
for name in STEPS:
|
||||
key = step_key(name, src.identity, out)
|
||||
if state['steps'].get(name) == key:
|
||||
say(f'{name}: done before')
|
||||
emit('step', step=name, state='skip')
|
||||
else:
|
||||
# a step that runs makes every step after it run too: their keys hash its output, not its key
|
||||
for later in STEPS[STEPS.index(name):]:
|
||||
state['steps'].pop(later, None)
|
||||
say(f'{name}:')
|
||||
emit('step', step=name, state='start')
|
||||
t = time.time()
|
||||
runs[name]()
|
||||
state['steps'][name] = step_key(name, src.identity, out)
|
||||
state_lock = threading.Lock() # the RomFS step records itself from its own thread
|
||||
|
||||
def save():
|
||||
with state_lock:
|
||||
state_path.write_text(json.dumps(state, indent=1), encoding='utf-8')
|
||||
say(f'{name}: {time.time() - t:.0f} s')
|
||||
emit('step', step=name, state='done', seconds=round(time.time() - t, 1))
|
||||
if name == a.until:
|
||||
break
|
||||
state_path.write_text(json.dumps(state, indent=1), encoding='utf-8')
|
||||
|
||||
def run(name):
|
||||
t = time.time()
|
||||
runs[name]()
|
||||
with state_lock:
|
||||
state['steps'][name] = step_key(name, src.identity, out)
|
||||
save()
|
||||
say(f'{name}: {time.time() - t:.0f} s')
|
||||
emit('step', step=name, state='done', seconds=round(time.time() - t, 1))
|
||||
|
||||
try:
|
||||
for name in STEPS:
|
||||
key = step_key(name, src.identity, out)
|
||||
if state['steps'].get(name) == key:
|
||||
say(f'{name}: done before')
|
||||
emit('step', step=name, state='skip')
|
||||
else:
|
||||
# a step that runs makes every step after it run too: their keys hash its output, not its key
|
||||
with state_lock:
|
||||
for later in STEPS[STEPS.index(name):]:
|
||||
state['steps'].pop(later, None)
|
||||
say(f'{name}:')
|
||||
emit('step', step=name, state='start')
|
||||
if name == 'shaders' and romfs_job: # the shaders come out of the RomFS image
|
||||
romfs_job.join()
|
||||
if name == 'romfs' and a.until != 'romfs':
|
||||
# The RomFS image is mostly disk work, the lift and the compile are processor work: they run side
|
||||
# by side, and only the RomFS index and the files that include it wait for the image
|
||||
# (step_compile). A first build takes about as long as the longer of the two, not their sum.
|
||||
romfs_job = Background(lambda: run('romfs'))
|
||||
else:
|
||||
run(name)
|
||||
if romfs_job and romfs_job.failed():
|
||||
romfs_job.join()
|
||||
if name == a.until:
|
||||
break
|
||||
if romfs_job:
|
||||
romfs_job.join()
|
||||
except SystemExit:
|
||||
# a failed lift or compile: the RomFS image still finishes and is kept, so the next run starts after it
|
||||
if romfs_job and romfs_job.thread.is_alive():
|
||||
say('waiting for the RomFS image before stopping')
|
||||
romfs_job.thread.join()
|
||||
raise
|
||||
save()
|
||||
say(f'\nfinished in {(time.time() - t0) / 60:.1f} min: {out / EXE_NAME}')
|
||||
emit('end', exe=str(out / EXE_NAME), seconds=round(time.time() - t0, 1), kit=kit_version())
|
||||
|
||||
|
||||
Reference in New Issue
Block a user