mirror of
https://github.com/tangerineturtle1/NX2navi.git
synced 2026-11-15 19:19:58 -04:00
f57a327a86
Import Saves: the saves folder picked in the dialog imports what waits in it (it had made the folder's README the save); the save backed up is the one the game would load, in runtime/fs_host.c save_recover's order, so an interrupted commit's only copy is kept, and the rest of the commit journal moves into the backup instead of being deleted; the imported save goes into place before the old one is deleted, which comes back if that fails; another NX2navi install's finished commit counts as its save. Tests cover each. Quit waits while the game writes a save, as Restart does: the game runs on under the menu, and closing the window finishes only a commit that has begun. The VSync row says "in use: Immediate" when VSync On falls back to tearing, and its default follows the device profile (cfg_default). Without the game's textures the menu's divider is a plain line and diamond of its own, not a tracing of the game's ornament. make_game.py: the compile key holds the processor's features (cpu_tag), so a game folder moved to a PC without AVX2 compiles again instead of crashing; the main thread gets 16 MiB of stack with GNU ld too; any error waits for the RomFS image, so its .part file is not cut short. The launcher tests use a neutral title id; tools/menu_preview.cpp has cfg_default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1165 lines
56 KiB
Python
1165 lines
56 KiB
Python
"""Build Ocarina of Time (Switch 2, game 1.1.1) for Windows on the player's PC, from the player's own game files.
|
|
|
|
python -I tools/make_game.py --nup "C:\\path\\game.nup" --out "C:\\Games\\Ocarina"
|
|
python -I tools/make_game.py --exefs "C:\\path\\exefs" --romfs "C:\\path\\romfs" --out "C:\\Games\\Ocarina"
|
|
python -I tools/make_game.py --nup "C:\\path\\game.nup" --check # only checks the game's version
|
|
python -I tools/make_game.py --rebuild --out "C:\\Games\\Ocarina" # a newer kit, the files of the last build
|
|
|
|
Steps, each skipped when a run before finished it for the same inputs (state in <out>/build-cache/state.json):
|
|
1. check the title (0400486024DEC000) and main's build id (game 1.1.1)
|
|
2. exefs the five ExeFS files, every module's segments unpacked (ZBIC through the player's own `zbic` package)
|
|
and stored plainly, as the runtime's loader reads them -> <out>/exefs
|
|
3. romfs every RomFS file, checked against the container's sha256, into one image -> <out>/romfs/romfs.bin and
|
|
index.json (16-byte aligned, in the byte order of the paths). When the lift and the compile follow, it
|
|
runs on a thread of its own beside them: only the RomFS index and romfs.c wait for the image
|
|
4. lift recomp/gen.py with recomp/patches-v111.txt -> <out>/build-cache/lift, plus the two SDK bring-up fixes
|
|
5. compile the lifted C and runtime/ with MSYS2's clang, linked into <out>/ocarina.exe; the MSYS2 DLLs it loads,
|
|
the shader translator and romfs-index.bin beside it
|
|
6. shaders every shader in the game files (tools/prepare_shaders.py, default bind state) translated, and each one's
|
|
GPU driver build done once, so play does not stop to compile them (`ocarina.exe --prepare-shaders`)
|
|
|
|
The exefs and romfs steps depend only on the game files; lift and compile depend on this kit's code too. A newer kit
|
|
therefore lifts and compiles again but keeps the unpacked game files, and --rebuild needs no game files at all. The
|
|
compile step keeps every object with the dependency list clang wrote for it (build-cache/obj/objects.json) and
|
|
compiles a file again only when it, a header it includes, the flags or the compiler changed; the kit's own folder
|
|
is not part of that key, so a kit in another folder (the launcher keeps each version in its own) reuses the objects.
|
|
A kit with lib/libnx2navirt.a built by --make-runtime-lib for the same runtime/ and the same compiler links the
|
|
runtime from it and compiles only the runtime files that include headers written per install. The new ocarina.exe
|
|
replaces the old one only once it links (with lld when the toolchain has it), so a failed build leaves the last good
|
|
one in place. Compiles run on every logical processor, the largest files first. The lifted code is built for this
|
|
PC's x86-64 level (cpu_level), with MSYS2 CLANG64's clang when it sits beside the UCRT64 folder and reports the same
|
|
version (fast_clang: the same objects, about a fifth faster).
|
|
|
|
python tools/make_game.py --make-runtime-lib <kit>/lib --msys <ucrt64> # the release's prebuilt runtime
|
|
|
|
Nothing is written outside --out (but for --make-runtime-lib DIR, which writes DIR). The game folder then holds
|
|
only what this PC built from the player's files: never
|
|
share it. Needs Python 3.10+ with numpy, lz4, zstandard and zbic (pip), and MSYS2 UCRT64's clang and libraries
|
|
(README-BuildKit.md lists the commands). Delete <out>/build-cache afterwards to free about 5 GB; the next build then
|
|
compiles everything again.
|
|
|
|
--events adds one machine-readable line per step and progress change to stdout, each starting with "@@" and followed
|
|
by a JSON object (launcher/build.py reads them): {"ev": "step", "step": ..., "state": "start" | "done" | "skip"},
|
|
{"ev": "progress", "step": ..., "done": n, "total": n, "unit": ...}, {"ev": "phase", ...}, {"ev": "error", "message":
|
|
...} and {"ev": "end", ...}.
|
|
"""
|
|
import argparse
|
|
import concurrent.futures
|
|
import hashlib
|
|
import json
|
|
import os
|
|
import re
|
|
import shutil
|
|
import struct
|
|
import subprocess
|
|
import sys
|
|
import threading
|
|
import time
|
|
from pathlib import Path
|
|
|
|
KIT = Path(__file__).resolve().parents[1]
|
|
sys.path.insert(0, str(KIT / 'tools'))
|
|
import nup # noqa: E402
|
|
|
|
TITLE_ID = 0x0400486024DEC000
|
|
MAIN_BUILD_ID = 'e0f3ccfaf50065ae4ea01eb5aeb13b1a' # main of game 1.1.1
|
|
EXEFS_FILES = ('main', 'main.npdm', 'sdk', 'subsdk0', 'rtld')
|
|
MODULES = ('main', 'sdk', 'subsdk0', 'rtld')
|
|
ALIGN = 16
|
|
STATE_VERSION = 2
|
|
STEPS = ('exefs', 'romfs', 'lift', 'compile', 'shaders')
|
|
|
|
# Bring-up fixes in the lifted SDK, each anchored on exact generated text that must occur once.
|
|
LIFT_FIXES = [
|
|
# nn::oe::SetupGpuErrorHandler (sdk+0x59a430) registers a system worker the host does not provide: GPU error
|
|
# reporting is not needed, so it returns at once.
|
|
('sdk', 'void f_700579a430(Cpu *c) {\n FN_ENTER(0x700579a430ull);\n',
|
|
'void f_700579a430(Cpu *c) {\n c->x[0] = 0; return; /* bring-up: GpuErrorHandler (make_game.py) */\n'
|
|
' FN_ENTER(0x700579a430ull);\n'),
|
|
# The thread-start path loads TPIDR_EL0 from TLR+0x1F0 before the SDK stored its TLS there for a host-created
|
|
# thread: a zero slot keeps the TLS pointer the host seeded.
|
|
('sdk', ' { uint64_t ea = c->x[19] + 0x1b8ull; ST64(ea, x8); }\n x8 = (uint64_t)(c->tpidrro);\n'
|
|
' { uint64_t ea = x8 + 0x1f0ull; uint64_t v = LD64(ea); x8 = (uint64_t)(v); }\n c->tpidr = x8;\n',
|
|
' { uint64_t ea = c->x[19] + 0x1b8ull; ST64(ea, x8); }\n x8 = (uint64_t)(c->tpidrro);\n'
|
|
' { uint64_t ea = x8 + 0x1f0ull; uint64_t v = LD64(ea); x8 = (uint64_t)(v); }\n'
|
|
' if (x8) c->tpidr = x8; /* zero slot: keep the host\'s TLS (make_game.py) */\n'),
|
|
]
|
|
LINK_LIBS = ['-lvulkan-1', '-lSDL3', '-lopus', '-lvpx', '-l:libdav1d.a', '-lzstd', '-l:libatomic.a', '-lbcrypt',
|
|
'-limm32', '-luuid', '-lole32', '-luser32', '-lshell32', '-lgdi32', '-ladvapi32', '-lwinmm']
|
|
SYSTEM_DLLS_SKIP = {'vulkan-1.dll'} # the graphics driver's loader, never a copy
|
|
EXE_NAME = 'ocarina.exe'
|
|
# Headers this build writes per install (gen.py, romfs_index.py): a runtime source that includes one, directly or
|
|
# through a runtime header, is compiled on the player's PC; every other runtime source can come prebuilt.
|
|
GENERATED_HEADERS = ('layout.h', 'patches.h', 'romfs_config.h')
|
|
RUNTIME_LIB = 'libnx2navirt.a'
|
|
RUNTIME_PATTERNS = ('*.c', '*.cpp', 'platform/*.c', 'ui/*.cpp', 'third_party/imgui/*.cpp',
|
|
'third_party/imgui/backends/*.cpp')
|
|
INCLUDE = re.compile(r'^[ \t]*#[ \t]*include[ \t]+"([^"]+)"', re.M)
|
|
|
|
# Runs a script the way `python script.py` does, with the script's folder first on sys.path. The launcher's
|
|
# embeddable Python leaves the script folder off sys.path (its ._pth file), so every script this build starts goes
|
|
# through this.
|
|
RUN_SCRIPT = ('import os, runpy, sys; s = sys.argv[1]; sys.argv = sys.argv[1:]; '
|
|
'sys.path.insert(0, os.path.dirname(os.path.abspath(s))); runpy.run_path(s, run_name="__main__")')
|
|
|
|
EVENTS = False
|
|
OUT_LOCK = threading.Lock() # the RomFS step prints from its own thread (main)
|
|
|
|
|
|
def say(msg):
|
|
with OUT_LOCK:
|
|
print(msg, flush=True)
|
|
|
|
|
|
def emit(ev, **fields):
|
|
"""One "@@{json}" line for the launcher (--events); nothing otherwise."""
|
|
if EVENTS:
|
|
line = '@@' + json.dumps({'ev': ev, **fields}, ensure_ascii=False)
|
|
with OUT_LOCK:
|
|
print(line, flush=True)
|
|
|
|
|
|
def fail(msg):
|
|
emit('error', message=msg)
|
|
with OUT_LOCK:
|
|
print(f'\nerror: {msg}', file=sys.stderr, flush=True)
|
|
sys.exit(1)
|
|
|
|
|
|
class Background:
|
|
"""One step on a thread of its own. join() waits for it and raises its failure (fail's SystemExit included) in
|
|
the caller; failed() tells the caller early, so it can stop work that would be thrown away."""
|
|
|
|
def __init__(self, fn):
|
|
self.exc = None
|
|
self.thread = threading.Thread(target=self._run, args=(fn,), name='romfs', daemon=True)
|
|
self.thread.start()
|
|
|
|
def _run(self, fn):
|
|
try:
|
|
fn()
|
|
except BaseException as e: # noqa: B036 - handed to join()
|
|
self.exc = e
|
|
|
|
def failed(self):
|
|
return self.exc is not None
|
|
|
|
def join(self):
|
|
self.thread.join()
|
|
if self.exc is not None:
|
|
raise self.exc
|
|
|
|
|
|
def py_cmd(script, *args):
|
|
"""The command line that runs a Python script of this kit with this interpreter (see RUN_SCRIPT)."""
|
|
flags = ['-I'] if sys.flags.isolated else []
|
|
return [sys.executable, *flags, '-c', RUN_SCRIPT, str(script), *map(str, args)]
|
|
|
|
|
|
def sha256_bytes(data):
|
|
return hashlib.sha256(data).hexdigest()
|
|
|
|
|
|
def sha256_file(path):
|
|
h = hashlib.sha256()
|
|
with open(path, 'rb') as f:
|
|
while b := f.read(8 << 20):
|
|
h.update(b)
|
|
return h.hexdigest()
|
|
|
|
|
|
def tree_digest(root, patterns):
|
|
"""sha256 over the relative names and contents of the files under root matching the glob patterns."""
|
|
h = hashlib.sha256()
|
|
files = sorted({p for pat in patterns for p in Path(root).glob(pat) if p.is_file() and '__pycache__' not in p.parts})
|
|
for p in files:
|
|
h.update(p.relative_to(root).as_posix().encode() + b'\0')
|
|
h.update(p.read_bytes())
|
|
h.update(b'\0')
|
|
return h.hexdigest()
|
|
|
|
|
|
# ---------------------------------------------------------------- input
|
|
|
|
class Source:
|
|
"""The game files, from a .nup container or from extracted exefs/romfs folders."""
|
|
|
|
def __init__(self, a):
|
|
self.n = None
|
|
if a.nup:
|
|
try:
|
|
self.n = nup.Nup(a.nup)
|
|
except (OSError, nup.NupError) as e:
|
|
fail(f'cannot read {a.nup}: {e}')
|
|
# The base title and its update (title id + 0x800), the update first and the newest of each kind first.
|
|
# A module comes from the first of them that holds it whole: an update's ExeFS may leave a module out or
|
|
# hold something else under its name.
|
|
pkgs = [p for p in self.n.packages if p['title_id'] in (TITLE_ID, TITLE_ID | 0x800)]
|
|
if not pkgs:
|
|
ids = ', '.join(f"{p['title_id']:016X}" for p in self.n.packages)
|
|
fail(f'this .nup holds {ids}, not Ocarina of Time (0400486024DEC000)')
|
|
pkgs.sort(key=lambda p: (p['kind'] != 1, -p['version']))
|
|
self.exefs = {}
|
|
for name in EXEFS_FILES:
|
|
for p in pkgs:
|
|
fi = dict(self.n.tree(p['exefs'])).get(name) if p['exefs'] < self.n.ndir else None
|
|
magic = b'META' if name.endswith('.npdm') else b'NSO0'
|
|
if fi is not None and self.n.size(fi) >= 0x100 and next(self.n.chunks(fi))[:4] == magic:
|
|
self.exefs[name] = fi
|
|
break
|
|
self.romfs = []
|
|
for p in pkgs:
|
|
if p['romfs'] < self.n.ndir:
|
|
self.romfs = list(self.n.tree(p['romfs']))
|
|
if self.romfs:
|
|
break
|
|
st = os.stat(a.nup)
|
|
self.identity = f'nup {os.path.abspath(a.nup)} {st.st_size} {int(st.st_mtime)}'
|
|
else:
|
|
self.exefs_dir, self.romfs_dir = Path(a.exefs), Path(a.romfs)
|
|
for d in (self.exefs_dir, self.romfs_dir):
|
|
if not d.is_dir():
|
|
fail(f'{d} is not a folder')
|
|
self.exefs = {name: self.exefs_dir / name for name in EXEFS_FILES if (self.exefs_dir / name).is_file()}
|
|
self.romfs = [(p.relative_to(self.romfs_dir).as_posix(), p) for p in self.romfs_dir.rglob('*') if p.is_file()]
|
|
self.identity = f'folders {self.exefs_dir.resolve()} {self.romfs_dir.resolve()} {len(self.romfs)}'
|
|
missing = [x for x in EXEFS_FILES if x not in self.exefs]
|
|
if missing:
|
|
fail(f'the ExeFS lacks {", ".join(missing)} (or holds something else under that name)')
|
|
if not self.romfs:
|
|
fail('the RomFS is empty')
|
|
|
|
def read_exefs(self, name):
|
|
return self.n.read(self.exefs[name]) if self.n else self.exefs[name].read_bytes()
|
|
|
|
def romfs_size(self, ref):
|
|
return self.n.size(ref) if self.n else ref.stat().st_size
|
|
|
|
def romfs_chunks(self, ref):
|
|
if self.n:
|
|
yield from self.n.chunks(ref)
|
|
return
|
|
with open(ref, 'rb') as f:
|
|
while b := f.read(nup.CHUNK):
|
|
yield b
|
|
|
|
|
|
class BuiltSource:
|
|
"""--rebuild: the ExeFS and RomFS an earlier run unpacked into the game folder, under that run's input identity.
|
|
The ExeFS modules are stored plainly there but keep their headers, so check() reads the same build id."""
|
|
|
|
def __init__(self, out, state):
|
|
self.out = out
|
|
if state.get('version') != STATE_VERSION or not state.get('input'):
|
|
fail(f'{out} holds no finished build of this kit to rebuild from: give --nup, or --exefs and --romfs')
|
|
for step in ('exefs', 'romfs'):
|
|
if step not in state.get('steps', {}):
|
|
fail(f'the last build in {out} did not finish its {step} step: give the game files again')
|
|
missing = [n for n in EXEFS_FILES if not (out / 'exefs' / n).is_file()]
|
|
missing += [f'romfs/{n}' for n in ('romfs.bin', 'index.json') if not (out / 'romfs' / n).is_file()]
|
|
if missing:
|
|
fail(f'{out} lacks {", ".join(missing)}: give the game files again')
|
|
self.identity = state['input']
|
|
|
|
def read_exefs(self, name):
|
|
return (self.out / 'exefs' / name).read_bytes()
|
|
|
|
|
|
def check(src):
|
|
main = src.read_exefs('main')
|
|
if main[:4] != b'NSO0':
|
|
fail('main is not an NSO module')
|
|
build = main[0x40:0x50].hex()
|
|
if build != MAIN_BUILD_ID:
|
|
fail(f'this is not game version 1.1.1: main has build {build[:8]}, the recomp supports build {MAIN_BUILD_ID[:8]}. '
|
|
'Install the 1.1.1 update and dump the game again.')
|
|
return main
|
|
|
|
|
|
# ---------------------------------------------------------------- exefs
|
|
|
|
def flat_nso(raw, name):
|
|
"""An NSO with its segments stored uncompressed (the runtime's loader reads only those); segment hashes kept."""
|
|
flags = struct.unpack_from('<I', raw, 0xC)[0]
|
|
out = bytearray(raw[:0x100])
|
|
for i, seg in enumerate(('text', 'ro', 'data')):
|
|
file_off, mem_off, size = struct.unpack_from('<III', raw, 0x10 + i * 0x10)
|
|
stored = struct.unpack_from('<I', raw, 0x60 + i * 4)[0]
|
|
data = raw[file_off:file_off + (stored if flags >> i & 1 else size)]
|
|
if flags >> i & 1:
|
|
if flags & 0x80:
|
|
try:
|
|
import zbic
|
|
except ImportError:
|
|
fail('the game\'s modules need the zbic package: pip install zbic')
|
|
data = zbic.decompress(data)
|
|
else:
|
|
import lz4.block
|
|
data = lz4.block.decompress(data, uncompressed_size=size)
|
|
if len(data) != size:
|
|
fail(f'{name}: {seg} unpacks to {len(data):#x} bytes, its header says {size:#x}')
|
|
if flags >> (3 + i) & 1 and hashlib.sha256(data).digest() != raw[0xA0 + 0x20 * i:0xC0 + 0x20 * i]:
|
|
fail(f'{name}: the {seg} segment is damaged (sha256 mismatch)')
|
|
struct.pack_into('<III', out, 0x10 + i * 0x10, len(out), mem_off, size)
|
|
struct.pack_into('<I', out, 0x60 + i * 4, size)
|
|
out += data
|
|
struct.pack_into('<I', out, 0xC, flags & ~0x87) # segments stored plainly, no ZBIC
|
|
return bytes(out)
|
|
|
|
|
|
def step_exefs(src, out):
|
|
dst = out / 'exefs'
|
|
dst.mkdir(parents=True, exist_ok=True)
|
|
for i, name in enumerate(EXEFS_FILES):
|
|
raw = src.read_exefs(name)
|
|
data = flat_nso(raw, name) if name in MODULES else raw
|
|
(dst / name).write_bytes(data)
|
|
say(f' {name:9} {len(raw) / 1e6:6.1f} MB -> {len(data) / 1e6:6.1f} MB')
|
|
emit('progress', step='exefs', done=i + 1, total=len(EXEFS_FILES), unit='files')
|
|
|
|
|
|
def exefs_digest(out):
|
|
return tree_digest(out / 'exefs', EXEFS_FILES)
|
|
|
|
|
|
# ---------------------------------------------------------------- romfs
|
|
|
|
def step_romfs(src, out, jobs):
|
|
dst = out / 'romfs'
|
|
dst.mkdir(parents=True, exist_ok=True)
|
|
files = sorted(src.romfs, key=lambda e: e[0].encode())
|
|
total = sum(src.romfs_size(ref) for _, ref in files)
|
|
free = shutil.disk_usage(dst).free
|
|
if free < total + (1 << 30):
|
|
fail(f'{dst} needs {total / 1e9:.1f} GB free for the RomFS image, it has {free / 1e9:.1f} GB')
|
|
say(f' {len(files)} files, {total / 1e9:.1f} GB')
|
|
entries, h, pos, done, t0 = [], hashlib.sha256(), 0, 0, time.time()
|
|
part = dst / 'romfs.bin.part'
|
|
emit('progress', step='romfs', done=0, total=total, unit='bytes')
|
|
|
|
def load(e): # small files are read whole on a worker; big ones are streamed by the writer
|
|
rel, ref = e
|
|
return None if src.romfs_size(ref) > (64 << 20) else b''.join(src.romfs_chunks(ref))
|
|
|
|
with open(part, 'wb', buffering=16 << 20) as img, concurrent.futures.ThreadPoolExecutor(jobs) as ex:
|
|
batch = 256
|
|
for start in range(0, len(files), batch):
|
|
group = files[start:start + batch]
|
|
for (rel, ref), data in zip(group, ex.map(load, group)):
|
|
pad = -pos % ALIGN
|
|
if pad:
|
|
img.write(b'\0' * pad)
|
|
h.update(b'\0' * pad)
|
|
pos += pad
|
|
size = src.romfs_size(ref)
|
|
entries.append({'path': rel, 'offset': pos, 'size': size})
|
|
for chunk in ((data,) if data is not None else src.romfs_chunks(ref)):
|
|
img.write(chunk)
|
|
h.update(chunk)
|
|
pos += size
|
|
done += size
|
|
emit('progress', step='romfs', done=done, total=total, unit='bytes')
|
|
if start // batch % 40 == 0:
|
|
say(f' RomFS {done / 1e9:5.1f} of {total / 1e9:.1f} GB, {time.time() - t0:.0f} s')
|
|
part.replace(dst / 'romfs.bin')
|
|
index = {'status': 'pass', 'romfs_size': pos, 'romfs_sha256': h.hexdigest(), 'files': entries}
|
|
(dst / 'index.json').write_text(json.dumps(index), encoding='utf-8')
|
|
say(f' RomFS image {pos / 1e9:.2f} GB in {time.time() - t0:.0f} s')
|
|
|
|
|
|
# ---------------------------------------------------------------- lift
|
|
|
|
EMITTED = re.compile(r'emitted (\d+)/(\d+) chunks')
|
|
|
|
|
|
def lift_digest():
|
|
"""The lifter's code, the patch table and the bring-up fixes: what the lifted C depends on besides the ExeFS."""
|
|
h = hashlib.sha256(tree_digest(KIT / 'recomp', ('*.py', 'patches-v111.txt')).encode())
|
|
h.update(repr(LIFT_FIXES).encode())
|
|
return h.hexdigest()
|
|
|
|
|
|
def step_lift(out, cache):
|
|
"""gen.py into build-cache/lift. The folder is kept: gen.py rewrites only the chunks whose text changed and
|
|
removes the ones it no longer writes."""
|
|
lift = cache / 'lift'
|
|
cmd = py_cmd(KIT / 'recomp' / 'gen.py', '--exefs', out / 'exefs', '--out', lift,
|
|
'--patches', KIT / 'recomp' / 'patches-v111.txt')
|
|
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, encoding='utf-8',
|
|
errors='replace', bufsize=1)
|
|
for line in p.stdout:
|
|
line = line.rstrip('\n')
|
|
say(f' {line}')
|
|
m = EMITTED.search(line)
|
|
if m:
|
|
emit('progress', step='lift', done=int(m.group(1)), total=int(m.group(2)), unit='chunks')
|
|
if p.wait():
|
|
fail('the code generator failed (see above)')
|
|
for mod, old, new in LIFT_FIXES:
|
|
hits = [p for p in sorted((lift / mod).glob('*.c')) if old in p.read_text(encoding='utf-8')]
|
|
if len(hits) != 1 or hits[0].read_text(encoding='utf-8').count(old) != 1:
|
|
fail(f'a bring-up fix found its anchor {len(hits)} times in the lifted {mod}, not once')
|
|
p = hits[0]
|
|
p.write_text(p.read_text(encoding='utf-8').replace(old, new), encoding='utf-8', newline='\n')
|
|
|
|
|
|
# ---------------------------------------------------------------- compile
|
|
|
|
def modules_c(lift, out, gen):
|
|
layout = json.loads((lift / 'layout.json').read_text(encoding='utf-8'))
|
|
L = ['#include "dispatch.h"', '#include "host.h"', '#include "layout.h"', '']
|
|
for m in layout:
|
|
n = m['name']
|
|
L += [f'extern const uint64_t {n}_fn_addrs[];', f'extern const GuestFn {n}_fn_ptrs[];',
|
|
f'extern const uint64_t {n}_fn_count;']
|
|
L += ['', 'const RtLayoutEntry rt_layout[] = {']
|
|
L += [f' {{"{m["name"]}", MOD_{m["name"].upper()}_BASE, MOD_{m["name"].upper()}_SIZE}},' for m in layout]
|
|
L += ['};', f'const int rt_layout_count = {len(layout)};', '', 'int rt_linked_modules(RtModule *out, int max) {',
|
|
' int n = 0;']
|
|
for m in layout:
|
|
n, u = m['name'], m['name'].upper()
|
|
L.append(f' if (n < max) out[n++] = (RtModule){{"{n}", MOD_{u}_BASE, MOD_{u}_SIZE, {n}_fn_addrs, {n}_fn_ptrs, '
|
|
f'{n}_fn_count}};')
|
|
L += [' return n;', '}', '', 'const RtFileHash rt_exefs_sha256[] = {']
|
|
L += [f' {{"{name}", "{sha256_file(out / "exefs" / name)}"}},' for name in EXEFS_FILES]
|
|
L += ['};', f'const int rt_exefs_sha256_count = {len(EXEFS_FILES)};', '',
|
|
f'const char rt_default_exefs[] = {json.dumps(str(out / "exefs"))};',
|
|
f'const char rt_default_romfs[] = {json.dumps(str(out / "romfs"))};',
|
|
f'const char rt_default_profile[] = {json.dumps(str(out / "profile"))};',
|
|
'const int rt_game_version = 111;',
|
|
'const unsigned long long rt_app_pool_bytes = 9809ull << 20; /* the Switch 2 application pool */',
|
|
'const int rt_nvn_api_version[2] = {231, 40}; /* NVN2: main asks for major 231, minor 40 or later */',
|
|
f'const char rt_port_version[] = {json.dumps(kit_version())};']
|
|
p = gen / 'modules.c'
|
|
write_if_changed(p, '\n'.join(L) + '\n')
|
|
write_if_changed(gen / 'shim.c', '/* Defined by the SDL window path only; this build uses the Win32 one. */\n'
|
|
'void menu_sdl_event(void *e) { (void)e; }\n')
|
|
return p
|
|
|
|
|
|
def write_if_changed(path, text):
|
|
if not path.is_file() or path.read_text(encoding='utf-8') != text:
|
|
path.write_text(text, encoding='utf-8')
|
|
|
|
|
|
def kit_version():
|
|
v = KIT / 'VERSION'
|
|
return v.read_text(encoding='utf-8').strip() if v.is_file() else '0.0.1-ocarina'
|
|
|
|
|
|
def pe_imports(path):
|
|
"""The DLL names a PE file imports (its import directory), lower case."""
|
|
d = Path(path).read_bytes()
|
|
pe = struct.unpack_from('<I', d, 0x3C)[0]
|
|
nsec, opt = struct.unpack_from('<H', d, pe + 6)[0], struct.unpack_from('<H', d, pe + 20)[0]
|
|
sec = pe + 24 + opt
|
|
rva = struct.unpack_from('<I', d, pe + 24 + 120)[0] # PE32+ data directory 1: imports
|
|
|
|
def off(r):
|
|
for i in range(nsec):
|
|
va, vs, raw = (struct.unpack_from('<I', d, sec + 40 * i + k)[0] for k in (12, 8, 20))
|
|
if va <= r < va + max(vs, 1):
|
|
return r - va + raw
|
|
return None
|
|
|
|
names, at = [], off(rva) if rva else None
|
|
while at is not None:
|
|
name_rva = struct.unpack_from('<I', d, at + 12)[0]
|
|
if not name_rva:
|
|
break
|
|
o = off(name_rva)
|
|
names.append(d[o:d.index(b'\0', o)].decode('ascii').lower())
|
|
at += 20
|
|
return names
|
|
|
|
|
|
def copy_if_changed(src, dst):
|
|
"""Copy src over dst unless dst already has its size and modification time. A file the running game holds open
|
|
cannot be replaced on Windows: that stops the build with a message instead of a traceback."""
|
|
s = os.stat(src)
|
|
try:
|
|
d = os.stat(dst)
|
|
if d.st_size == s.st_size and int(d.st_mtime) == int(s.st_mtime):
|
|
return False
|
|
except FileNotFoundError:
|
|
pass
|
|
try:
|
|
shutil.copy2(src, dst)
|
|
except PermissionError:
|
|
fail(f'{dst} is in use: close the game and try again')
|
|
return True
|
|
|
|
|
|
def copy_dlls(exe, msys_bin, out):
|
|
seen, todo, copied = set(), [exe], []
|
|
while todo:
|
|
for name in pe_imports(todo.pop()):
|
|
src = msys_bin / name
|
|
if name in seen or name in SYSTEM_DLLS_SKIP or not src.is_file():
|
|
continue
|
|
seen.add(name)
|
|
copy_if_changed(src, out / src.name)
|
|
copied.append(src.name)
|
|
todo.append(src)
|
|
return copied
|
|
|
|
|
|
def parse_deps(text):
|
|
"""The prerequisites of a make-style dependency file as clang -MMD writes it (the target dropped)."""
|
|
text = text.replace('\\\r\n', ' ').replace('\\\n', ' ')
|
|
tokens, cur, i = [], '', 0
|
|
while i < len(text):
|
|
c = text[i]
|
|
if c == '\\' and i + 1 < len(text) and text[i + 1] in ' #':
|
|
cur += text[i + 1]
|
|
i += 2
|
|
continue
|
|
if c == '$' and text[i + 1:i + 2] == '$':
|
|
cur += '$'
|
|
i += 2
|
|
continue
|
|
if c.isspace():
|
|
if cur:
|
|
tokens.append(cur)
|
|
cur = ''
|
|
else:
|
|
cur += c
|
|
i += 1
|
|
if cur:
|
|
tokens.append(cur)
|
|
for k, t in enumerate(tokens): # the target ends with ':' (a drive letter's colon is followed by a separator)
|
|
if t.endswith(':'):
|
|
return tokens[k + 1:]
|
|
return tokens
|
|
|
|
|
|
class ObjectCache:
|
|
"""Which objects are current: per object the key of its command and source, and the sha256 (with size and
|
|
modification time, to skip hashing an untouched file) of every file its dependency list names
|
|
(build-cache/obj/objects.json). Paths under a root are kept relative to it ({'<kit>': KIT}), so the same files
|
|
in a kit at another place still count as the same."""
|
|
|
|
def __init__(self, path, roots=None):
|
|
self.path = path
|
|
self.roots = {k: os.path.normcase(os.path.abspath(v)) for k, v in (roots or {}).items()}
|
|
try:
|
|
self.entries = json.loads(path.read_text(encoding='utf-8')) if path.is_file() else {}
|
|
except (OSError, ValueError):
|
|
self.entries = {}
|
|
self._sha = {}
|
|
|
|
def norm(self, p):
|
|
a = os.path.normcase(os.path.abspath(p))
|
|
for name, root in self.roots.items():
|
|
if a == root or a.startswith(root + os.sep):
|
|
return name + a[len(root):].replace(os.sep, '/')
|
|
return a
|
|
|
|
def real(self, p):
|
|
for name, root in self.roots.items():
|
|
if p.startswith(name):
|
|
return root + p[len(name):].replace('/', os.sep)
|
|
return p
|
|
|
|
def file_sha(self, p):
|
|
p = os.path.normcase(os.path.abspath(p))
|
|
if p not in self._sha:
|
|
try:
|
|
self._sha[p] = sha256_file(p)
|
|
except OSError:
|
|
self._sha[p] = None
|
|
return self._sha[p]
|
|
|
|
def _same(self, p, rec):
|
|
path = self.real(p)
|
|
if isinstance(rec, str): # a cache without sizes and times
|
|
return self.file_sha(path) == rec
|
|
digest, size, mtime = rec
|
|
try:
|
|
st = os.stat(path)
|
|
except OSError:
|
|
return False
|
|
if st.st_size == size and st.st_mtime_ns == mtime:
|
|
return True
|
|
return st.st_size == size and self.file_sha(path) == digest
|
|
|
|
def current(self, obj, key):
|
|
e = self.entries.get(obj.name)
|
|
return (e is not None and e.get('key') == key and obj.is_file()
|
|
and all(self._same(p, rec) for p, rec in e.get('deps', {}).items()))
|
|
|
|
def record(self, obj, key, dep_file, src):
|
|
try:
|
|
deps = parse_deps(Path(dep_file).read_text(encoding='utf-8', errors='replace'))
|
|
except OSError:
|
|
deps = []
|
|
out = {}
|
|
for p in deps or [str(src)]:
|
|
try:
|
|
st = os.stat(p)
|
|
out[self.norm(p)] = [self.file_sha(p), st.st_size, st.st_mtime_ns]
|
|
except OSError:
|
|
out[self.norm(p)] = [None, -1, -1]
|
|
self.entries[obj.name] = {'key': key, 'deps': out}
|
|
|
|
def forget(self, obj):
|
|
self.entries.pop(obj.name, None)
|
|
|
|
def save(self):
|
|
tmp = self.path.with_suffix('.tmp')
|
|
tmp.write_text(json.dumps(self.entries), encoding='utf-8')
|
|
tmp.replace(self.path)
|
|
|
|
|
|
def compiler_id(clang, env):
|
|
p = subprocess.run([str(clang), '--version'], capture_output=True, text=True, env=env)
|
|
st = os.stat(clang)
|
|
return sha256_bytes(f'{p.stdout}\n{st.st_size}\n{int(st.st_mtime)}'.encode())
|
|
|
|
|
|
def compile_all(srcs, cmd_for, cache, jobs, env, progress=None, salt='', stop=None):
|
|
"""Compile each (source, object) whose object is not current in cache. cmd_for(src, dst) -> argv without the
|
|
dependency-file options; salt identifies the compiler. Returns (compiled, reused, failures) with failures as
|
|
(src, stderr tail). The cache's roots stand for their paths in the flags too. The largest sources start first:
|
|
compile time grows with size, and a big file that started last would run on alone after the others finished.
|
|
stop() true makes the compiles not yet started end at once, unrecorded (a step beside this one failed)."""
|
|
todo, keys = [], {}
|
|
for src, dst in srcs:
|
|
cmd = cmd_for(src, dst)
|
|
flags = [c for c in cmd[1:] if c not in (str(src), str(dst))]
|
|
for name, root in cache.roots.items():
|
|
flags = [name + os.path.normcase(f)[len(root):].replace(os.sep, '/') if os.path.normcase(f).startswith(root)
|
|
else f for f in flags]
|
|
flags = [os.path.basename(cmd[0])] + flags
|
|
keys[dst] = sha256_bytes(json.dumps([salt, flags, cache.file_sha(src)]).encode())
|
|
if not cache.current(dst, keys[dst]):
|
|
todo.append((src, dst))
|
|
reused = len(srcs) - len(todo)
|
|
todo.sort(key=lambda job: job[0].stat().st_size, reverse=True)
|
|
|
|
def cc(job):
|
|
src, dst = job
|
|
dep = dst.with_suffix('.d')
|
|
if stop and stop():
|
|
return src, dst, dep, None, ''
|
|
cmd = cmd_for(src, dst)
|
|
cmd[1:1] = ['-MMD', '-MF', str(dep)]
|
|
p = subprocess.run(cmd, capture_output=True, text=True, env=env, errors='replace')
|
|
return src, dst, dep, p.returncode, p.stderr[-1500:]
|
|
|
|
done, bad = reused, []
|
|
if progress:
|
|
progress(done, len(srcs))
|
|
with concurrent.futures.ThreadPoolExecutor(max(1, jobs)) as ex:
|
|
for src, dst, dep, rc, err in ex.map(cc, todo):
|
|
done += 1
|
|
if rc is None: # not started (stop)
|
|
continue
|
|
if rc:
|
|
bad.append((src, err))
|
|
cache.forget(dst)
|
|
else:
|
|
cache.record(dst, keys[dst], dep, src)
|
|
if progress:
|
|
progress(done, len(srcs))
|
|
cache.save()
|
|
return len(todo), reused, bad
|
|
|
|
|
|
def runtime_sources():
|
|
"""(source, object name) of every runtime file the Windows build compiles."""
|
|
rt = KIT / 'runtime'
|
|
out = []
|
|
for pat in RUNTIME_PATTERNS:
|
|
for p in sorted(rt.glob(pat)):
|
|
if p.stem.endswith('_sdl') or p.name == 'linux.c':
|
|
continue
|
|
out.append((p, 'rt_' + p.relative_to(rt).as_posix().replace('/', '_').rsplit('.', 1)[0] + '.o'))
|
|
return out
|
|
|
|
|
|
def per_install_names(generated=GENERATED_HEADERS):
|
|
"""The basenames of the generated headers and of every runtime header that includes one, at any depth."""
|
|
rt = KIT / 'runtime'
|
|
headers = {}
|
|
for p in list(rt.rglob('*.h')) + list(rt.rglob('*.inc')):
|
|
headers[p.name] = {os.path.basename(i) for i in INCLUDE.findall(p.read_text(encoding='utf-8', errors='replace'))}
|
|
tainted = set(generated)
|
|
grew = True
|
|
while grew:
|
|
grew = False
|
|
for name, incs in headers.items():
|
|
if name not in tainted and incs & tainted:
|
|
tainted.add(name)
|
|
grew = True
|
|
return tainted
|
|
|
|
|
|
def per_install(src, tainted):
|
|
incs = {os.path.basename(i) for i in INCLUDE.findall(Path(src).read_text(encoding='utf-8', errors='replace'))}
|
|
return bool(incs & tainted)
|
|
|
|
|
|
def clang_version(clang, env):
|
|
p = subprocess.run([str(clang), '--version'], capture_output=True, text=True, env=env)
|
|
return (p.stdout.splitlines() or [''])[0].strip()
|
|
|
|
|
|
def runtime_digest():
|
|
return tree_digest(KIT / 'runtime', ('**/*',))
|
|
|
|
|
|
# The x86-64 levels the lifted code is built and tested for (docs/building.md, "Checks"), as the macros clang defines
|
|
# for a CPU that has them.
|
|
CPU_V2 = {'__SSE3__', '__SSSE3__', '__SSE4_1__', '__SSE4_2__', '__POPCNT__', '__LAHF_SAHF__',
|
|
'__GCC_HAVE_SYNC_COMPARE_AND_SWAP_16'}
|
|
CPU_V3 = CPU_V2 | {'__AVX__', '__AVX2__', '__BMI__', '__BMI2__', '__F16C__', '__FMA__', '__LZCNT__', '__MOVBE__',
|
|
'__XSAVE__'}
|
|
|
|
|
|
def cpu_level(clang, env):
|
|
"""This PC's x86-64 level for the lifted code: 3 (AVX2, FMA, BMI2), 2 (SSE4.2, POPCNT) or 0 when clang cannot
|
|
tell. The game runs on the PC that builds it, so it takes this CPU's level, as tools/build_game.py --cpu does by
|
|
hand. Without one the lifted code would be generic x86-64: every fused multiply-add of the game a call to
|
|
runtime/fma.c's software fma, and a level the differential tests do not cover."""
|
|
p = subprocess.run([str(clang), '-march=native', '-E', '-dM', '-x', 'c', os.devnull], capture_output=True,
|
|
text=True, env=env, errors='replace')
|
|
have = set(re.findall(r'^#define (__\w+) ', p.stdout, re.M)) if p.returncode == 0 else set()
|
|
return 3 if CPU_V3 <= have else 2 if CPU_V2 <= have else 0
|
|
|
|
|
|
def clang_ident(clang, env):
|
|
"""(version, target) a clang reports, or None when it does not run."""
|
|
try:
|
|
p = subprocess.run([str(clang), '--version'], capture_output=True, text=True, env=env, errors='replace')
|
|
except OSError:
|
|
return None
|
|
v, t = re.search(r'clang version (\S+)', p.stdout), re.search(r'^Target: (\S+)', p.stdout, re.M)
|
|
return (v.group(1), t.group(1)) if p.returncode == 0 and v and t else None
|
|
|
|
|
|
def fast_clang(msys, clang, env):
|
|
"""MSYS2's CLANG64 clang beside the UCRT64 one (msys64/clang64, which the launcher installs): the same LLVM release
|
|
built with profile-guided optimisation, about a fifth faster on the lifted code. With UCRT64's headers
|
|
(--sysroot) it writes the same objects as UCRT64's clang, byte for byte but the COFF timestamp; it is used only
|
|
when it reports the same version and target. None otherwise."""
|
|
fast = Path(msys).parent / 'clang64' / 'bin' / 'clang.exe'
|
|
if not fast.is_file():
|
|
return None
|
|
mine = clang_ident(clang, env)
|
|
return fast if mine and clang_ident(fast, env) == mine else None
|
|
|
|
|
|
def runtime_cmd(clang, clangxx, inc, level=0, fast=None, msys=None):
|
|
"""fast: fast_clang's compiler for the lifted code, with msys as its sysroot."""
|
|
def cmd_for(src, dst):
|
|
cpp = src.suffix in ('.cpp', '.cxx')
|
|
# the lifted code at -O1 (its size makes -O2 slow to build), the runtime at -O2; neither may assume strict
|
|
# aliasing: the lifted code reaches guest memory through casts of one address to several types
|
|
# (tools/build_game.py's flags). The lifted code is built for this PC's level (cpu_level); the runtime
|
|
# stays generic, so the prebuilt one serves every PC (its vectors are 16 bytes, passed alike at every level).
|
|
guest = dst.name.startswith(('main_', 'sdk_', 'subsdk0_'))
|
|
march = [f'-march=x86-64-v{level}'] if guest and level else []
|
|
cc = [str(fast), f'--sysroot={msys}'] if guest and fast else [str(clangxx if cpp else clang)]
|
|
return [*cc, '-c', '-std=gnu++17' if cpp else '-std=gnu11', '-O1' if guest else '-O2',
|
|
*march, '-fno-strict-aliasing', '-w', *inc, '-o', str(dst), str(src)]
|
|
return cmd_for
|
|
|
|
|
|
def make_runtime_lib(dest, msys, jobs, obj=None):
|
|
"""--make-runtime-lib: compile every runtime source that does not include a per-install header into
|
|
<dest>/libnx2navirt.a, with runtime-lib.json naming the runtime/ tree and the compiler it was built from. The
|
|
objects stay in obj (default <dest>/obj), which a release keeps out of what it ships."""
|
|
dest = Path(dest).resolve()
|
|
dest.mkdir(parents=True, exist_ok=True)
|
|
obj = Path(obj).resolve() if obj else dest / 'obj'
|
|
obj.mkdir(parents=True, exist_ok=True)
|
|
clang, clangxx = msys / 'bin' / 'clang.exe', msys / 'bin' / 'clang++.exe'
|
|
env = dict(os.environ, PATH=str(msys / 'bin') + os.pathsep + os.environ.get('PATH', ''))
|
|
rt = KIT / 'runtime'
|
|
tainted = per_install_names()
|
|
srcs = [(p, obj / name) for p, name in runtime_sources() if not per_install(p, tainted)]
|
|
inc = ['-I', str(rt), '-I', str(rt / 'third_party' / 'imgui'), '-I', str(msys / 'include' / 'vma')]
|
|
cache = ObjectCache(obj / 'objects.json', {'<kit>': KIT})
|
|
t0 = time.time()
|
|
compiled, reused, bad = compile_all(srcs, runtime_cmd(clang, clangxx, inc), cache, jobs, env,
|
|
lambda d, t: (d % 20 == 0 or d == t) and say(f' {d} of {t}'),
|
|
salt=compiler_id(clang, env))
|
|
if bad:
|
|
for src, err in bad[:3]:
|
|
print(f'--- {src}\n{err}', file=sys.stderr)
|
|
fail(f'{len(bad)} runtime files did not compile')
|
|
lib = dest / RUNTIME_LIB
|
|
lib.unlink(missing_ok=True)
|
|
rsp = dest / 'ar.rsp'
|
|
rsp.write_text('\n'.join('"%s"' % o.as_posix() for _, o in srcs), encoding='utf-8')
|
|
ar = os.environ.get('NX2NAVI_AR') or str(msys / 'bin' / 'ar.exe')
|
|
if subprocess.run([ar, 'rcs', str(lib), '@' + str(rsp)], env=env).returncode:
|
|
fail('the runtime library could not be archived')
|
|
meta = {'runtime': runtime_digest(), 'compiler': clang_version(clang, env),
|
|
'objects': sorted(o.name for _, o in srcs)}
|
|
(dest / 'runtime-lib.json').write_text(json.dumps(meta, indent=1), encoding='utf-8')
|
|
rsp.unlink(missing_ok=True)
|
|
say(f'{lib}: {len(srcs)} runtime files ({compiled} compiled, {reused} kept), {lib.stat().st_size / 1e6:.1f} MB, '
|
|
f'{time.time() - t0:.0f} s')
|
|
|
|
|
|
def prebuilt_runtime(clang, env):
|
|
"""(library, object names it holds) when the kit's lib/ matches this runtime/ and this compiler, else None."""
|
|
lib, meta = KIT / 'lib' / RUNTIME_LIB, KIT / 'lib' / 'runtime-lib.json'
|
|
try:
|
|
m = json.loads(meta.read_text(encoding='utf-8'))
|
|
except (OSError, ValueError):
|
|
return None
|
|
if not lib.is_file() or m.get('runtime') != runtime_digest() or m.get('compiler') != clang_version(clang, env):
|
|
say(' the prebuilt runtime does not match this kit or compiler: compiling runtime/')
|
|
return None
|
|
return lib, set(m.get('objects', []))
|
|
|
|
|
|
def romfs_index(out, gen):
|
|
if subprocess.run(py_cmd(KIT / 'tools' / 'romfs_index.py', '--romfs', out / 'romfs', '--out', gen)).returncode:
|
|
fail('the RomFS index could not be written')
|
|
|
|
|
|
def windres_command(msys, rco):
|
|
"""MSYS2's windres on nx2navi.rc, run with runtime/ as the working directory (compile_resources says why)."""
|
|
pp = ('-E', '-xc', '-DRC_INVOKED') # windres's own preprocessor flags, which --preprocessor replaces
|
|
return [str(msys / 'bin' / 'windres.exe'), '--use-temp-file', f'--preprocessor={msys / "bin" / "gcc.exe"}',
|
|
*(f'--preprocessor-arg={a}' for a in pp), '-O', 'coff', '-i', 'nx2navi.rc', '-o', str(rco)]
|
|
|
|
|
|
def compile_resources(msys, rt, obj, env):
|
|
"""runtime/nx2navi.rc into obj/nx2navi_rc.o: the manifest of the game's exe (per-monitor DPI awareness, so the
|
|
picture keeps the monitor's real size rather than one Windows stretches) and its icon.
|
|
|
|
windres preprocesses the .rc with gcc through one command line it writes itself, quoting each path that holds a
|
|
space. By default it hands that line to cmd.exe /c, which drops the first and the last double quote of a line
|
|
holding more than two: gcc's quoted path loses its opening quote, and under a user folder with a space in its
|
|
name (C:\\Users\\Ali Sayed) cmd.exe stops at that space ("'C:\\Users\\Ali' is not recognized").
|
|
|
|
With --use-temp-file windres starts gcc itself instead, no shell involved: it splits the line into arguments and
|
|
passes each one to gcc intact. Its splitter reads a quoted first word right, but it cuts a quoted -I path at its
|
|
spaces and reads past the end of the line after a quoted last word, so the line holds no path but gcc's, its
|
|
first word: the .rc goes by its bare name from runtime/ as the working directory, where windres also finds the
|
|
manifest and the icon, and there is no -I. windres's temporary files go to a folder of the build cache, deleted
|
|
afterwards (it leaves two of them behind on every run).
|
|
|
|
The object is written under another name and replaces the old one only when its bytes differ, so an unchanged
|
|
resource keeps its time and does not make the link key change (a rebuild with nothing new does not link again)."""
|
|
rco, new = obj / 'nx2navi_rc.o', obj / 'nx2navi_rc.o.new'
|
|
tmp = obj / 'windres-tmp'
|
|
shutil.rmtree(tmp, ignore_errors=True)
|
|
tmp.mkdir()
|
|
try:
|
|
r = subprocess.run(windres_command(msys, new), cwd=rt, env=dict(env, TMP=str(tmp), TEMP=str(tmp)),
|
|
capture_output=True, text=True, errors='replace')
|
|
finally:
|
|
shutil.rmtree(tmp, ignore_errors=True)
|
|
if r.returncode:
|
|
new.unlink(missing_ok=True)
|
|
fail(f'windres failed on runtime/nx2navi.rc: {r.stderr.strip()[:400]}')
|
|
if rco.is_file() and rco.read_bytes() == new.read_bytes():
|
|
new.unlink()
|
|
else:
|
|
os.replace(new, rco)
|
|
return rco
|
|
|
|
|
|
def step_compile(out, cache_dir, msys, jobs, romfs_job=None):
|
|
"""romfs_job: the RomFS step still running beside this one (a first build). Everything that does not include
|
|
romfs_config.h compiles meanwhile; the RomFS index and the few files that include it follow once it is done."""
|
|
lift, gen, obj = cache_dir / 'lift', cache_dir / 'gen', cache_dir / 'obj'
|
|
gen.mkdir(parents=True, exist_ok=True)
|
|
obj.mkdir(parents=True, exist_ok=True)
|
|
rt = KIT / 'runtime'
|
|
clang, clangxx = msys / 'bin' / 'clang.exe', msys / 'bin' / 'clang++.exe'
|
|
env = dict(os.environ, PATH=str(msys / 'bin') + os.pathsep + os.environ.get('PATH', ''))
|
|
if not romfs_job:
|
|
romfs_index(out, gen)
|
|
srcs = [(p, obj / f'{mod}_{p.stem}.o') for mod in ('main', 'sdk', 'subsdk0') for p in sorted((lift / mod).glob('*.c'))]
|
|
prebuilt = prebuilt_runtime(clang, env)
|
|
runtime = [(p, obj / name) for p, name in runtime_sources()]
|
|
if prebuilt:
|
|
runtime = [(p, o) for p, o in runtime if o.name not in prebuilt[1]]
|
|
say(f' the runtime comes prebuilt; {len(runtime)} runtime files are compiled here')
|
|
srcs += runtime
|
|
srcs += [(modules_c(lift, out, gen), obj / 'modules.o'), (gen / 'shim.c', obj / 'shim.o')]
|
|
inc = ['-I', str(rt), '-I', str(lift), '-I', str(gen), '-I', str(rt / 'third_party' / 'imgui'),
|
|
'-I', str(msys / 'include' / 'vma')]
|
|
cid = compiler_id(clang, env)
|
|
level = cpu_level(clang, env)
|
|
say(f' the lifted code is built for x86-64-v{level}, this processor\'s level' if level else
|
|
' the lifted code is built for generic x86-64: the processor\'s level could not be read')
|
|
fast = fast_clang(msys, clang, env)
|
|
if fast:
|
|
cid += compiler_id(fast, env)
|
|
say(f' the lifted code compiles with {fast}')
|
|
cmd_for = runtime_cmd(clang, clangxx, inc, level, fast, msys)
|
|
|
|
t0 = time.time()
|
|
last = [0.0]
|
|
|
|
def progress(done, total):
|
|
now = time.time()
|
|
if done == total or now - last[0] >= 0.25:
|
|
last[0] = now
|
|
emit('progress', step='compile', done=done, total=total, unit='files')
|
|
if done % 50 == 0 or done == total:
|
|
say(f' compiled {done} of {total}, {now - t0:.0f} s')
|
|
|
|
say(f' {len(srcs)} files on {jobs} threads')
|
|
cache = ObjectCache(obj / 'objects.json', {'<kit>': KIT})
|
|
late = []
|
|
if romfs_job: # the lifted chunks include only cpu.h and simd.h: only the runtime and generated files are read
|
|
tainted = per_install_names(('romfs_config.h',))
|
|
late = [(s, o) for s, o in runtime + srcs[-2:] if per_install(s, tainted)]
|
|
say(f' {len(late)} files wait for the RomFS index: {", ".join(s.name for s, _ in late)}')
|
|
early = [j for j in srcs if j not in late]
|
|
compiled, reused, bad = compile_all(early, cmd_for, cache, jobs, env, lambda d, t: progress(d, len(srcs)),
|
|
salt=cid, stop=romfs_job.failed if romfs_job else None)
|
|
if romfs_job:
|
|
if romfs_job.thread.is_alive():
|
|
say(' waiting for the RomFS image')
|
|
romfs_job.join() # raises the RomFS step's failure; a failed compile still keeps the finished image
|
|
if not bad:
|
|
romfs_index(out, gen)
|
|
c2, r2, bad = compile_all(late, cmd_for, cache, jobs, env,
|
|
lambda d, t: progress(len(early) + d, len(srcs)), salt=cid)
|
|
compiled, reused = compiled + c2, reused + r2
|
|
say(f' {compiled} compiled, {reused} unchanged since the last build')
|
|
if bad:
|
|
for src, err in bad[:3]:
|
|
print(f'--- {src}\n{err}', file=sys.stderr)
|
|
fail(f'{len(bad)} files did not compile')
|
|
|
|
rco = compile_resources(msys, rt, obj, env)
|
|
objs = [o for _, o in srcs] + [rco]
|
|
exe = out / EXE_NAME
|
|
rsp = cache_dir / 'link.rsp'
|
|
rsp.write_text('\n'.join('"%s"' % o.as_posix() for o in objs), encoding='utf-8')
|
|
# lld links several times faster than GNU ld; the stack matches tools/build_game.py's build
|
|
# 16 MiB for the main thread with either linker (GNU ld takes the same flag): the default 2 MiB is too little
|
|
ld = (['-fuse-ld=lld'] if (msys / 'bin' / 'ld.lld.exe').is_file() else []) + ['-Wl,--stack,16777216']
|
|
lib = ['-Wl,--whole-archive', str(prebuilt[0]), '-Wl,--no-whole-archive'] if prebuilt else []
|
|
stamp = [[o.name, o.stat().st_size, o.stat().st_mtime_ns] for o in objs]
|
|
if prebuilt:
|
|
stamp.append([prebuilt[0].name, prebuilt[0].stat().st_size, sha256_file(prebuilt[0])])
|
|
link_key = sha256_bytes(json.dumps([LINK_LIBS, ld, cid] + stamp).encode())
|
|
state_link = cache_dir / 'link.key'
|
|
if exe.is_file() and state_link.is_file() and state_link.read_text(encoding='utf-8') == link_key:
|
|
say(' ocarina.exe is current, not linked again')
|
|
else:
|
|
emit('phase', step='compile', phase='link')
|
|
say(' linking')
|
|
tmp = out / (EXE_NAME + '.new')
|
|
p = subprocess.run([str(clangxx), *ld, '-o', str(tmp), '@' + str(rsp), *lib, '-L', str(msys / 'lib'),
|
|
*LINK_LIBS], capture_output=True, text=True, env=env, errors='replace')
|
|
if p.returncode:
|
|
print(p.stderr[-4000:], file=sys.stderr)
|
|
tmp.unlink(missing_ok=True)
|
|
fail('the link failed')
|
|
try:
|
|
os.replace(tmp, exe)
|
|
except PermissionError:
|
|
tmp.unlink(missing_ok=True)
|
|
fail(f'{exe} is in use: close the game and try again')
|
|
state_link.write_text(link_key, encoding='utf-8')
|
|
copy_if_changed(gen / 'romfs-index.bin', out / 'romfs-index.bin')
|
|
for name in ('nx2navi_xlate.exe', 'nx2navi_xlate.id'):
|
|
copy_if_changed(KIT / 'bin' / name, out / name)
|
|
if (KIT / 'bin' / 'amd_fidelityfx_vk.dll').is_file(): # FSR 3.1 (runtime/vk_ffx.cpp)
|
|
copy_if_changed(KIT / 'bin' / 'amd_fidelityfx_vk.dll', out / 'amd_fidelityfx_vk.dll')
|
|
say(f' {EXE_NAME} {exe.stat().st_size / 1e6:.0f} MB; beside it: {", ".join(copy_dlls(exe, msys / "bin", out))}')
|
|
(out / 'profile').mkdir(exist_ok=True)
|
|
|
|
|
|
# ---------------------------------------------------------------- main
|
|
|
|
def step_shaders(out):
|
|
"""Every shader of the game files into the game's shader cache and the driver's own cache, before the first start."""
|
|
cmd = [sys.executable, str(KIT / 'tools' / 'prepare_shaders.py'), str(out), '--romfs', str(out / 'romfs'),
|
|
'--exefs', str(out / 'exefs'), '--no-list', '--exe', 'ocarina.exe']
|
|
if subprocess.run(cmd).returncode:
|
|
fail('preparing the shaders failed (the lines above say why); the game still runs, compiling shaders as it '
|
|
'meets them. Run the build again to retry this step')
|
|
for seed in ('seed-programs.bin', 'seed-answers.bin'): # translated now: the shader cache holds them
|
|
(out / seed).unlink(missing_ok=True)
|
|
|
|
|
|
def keep_awake():
|
|
"""Hold off sleep for this process's life (Windows): the build runs for many minutes with no input, and a PC that
|
|
sleeps in the middle stops the compiles. The hold ends with the process."""
|
|
if os.name != 'nt':
|
|
return
|
|
try:
|
|
import ctypes
|
|
ES_CONTINUOUS, ES_SYSTEM_REQUIRED = 0x80000000, 0x00000001
|
|
ctypes.windll.kernel32.SetThreadExecutionState(ES_CONTINUOUS | ES_SYSTEM_REQUIRED)
|
|
except (OSError, AttributeError):
|
|
pass
|
|
|
|
|
|
def load_state(path):
|
|
try:
|
|
state = json.loads(path.read_text(encoding='utf-8')) if path.is_file() else {}
|
|
except (OSError, ValueError):
|
|
state = {}
|
|
if state.get('version') == 1: # the first kit: a list of finished steps, all tied to one kit version
|
|
done = state.get('done', [])
|
|
state = {'version': STATE_VERSION, 'input': state.get('input'), 'steps': {}}
|
|
for step in ('exefs', 'romfs'):
|
|
if step in done:
|
|
state['steps'][step] = step_key(step, state['input'])
|
|
return state
|
|
|
|
|
|
def cpu_tag():
|
|
"""This processor's features the lifted code's level follows (cpu_level), as Windows reports them; part of the
|
|
compile key, so a game folder moved to a PC without them (or a new processor) is compiled again rather than kept
|
|
with instructions the processor lacks. 'any' where it cannot be read."""
|
|
if os.name != 'nt':
|
|
return 'any'
|
|
try:
|
|
import ctypes
|
|
present = ctypes.windll.kernel32.IsProcessorFeaturePresent
|
|
# PF_SSE4_2_INSTRUCTIONS_AVAILABLE 38, PF_AVX2_INSTRUCTIONS_AVAILABLE 40, PF_AVX512F_INSTRUCTIONS_AVAILABLE 41
|
|
return ''.join('1' if present(f) else '0' for f in (38, 40, 41))
|
|
except (OSError, AttributeError):
|
|
return 'any'
|
|
|
|
|
|
def step_key(step, identity, out=None):
|
|
"""What a step's output depends on. exefs and romfs: the game files only; lift: the unpacked ExeFS and the
|
|
lifter; compile: the lift, the runtime sources, this script and the processor (cpu_tag)."""
|
|
if step in ('exefs', 'romfs'):
|
|
return f'{step}-1|{identity}'
|
|
lift = f'lift|{exefs_digest(out)}|{lift_digest()}'
|
|
if step == 'lift':
|
|
return lift
|
|
compile_key = (f'compile|{lift}|{tree_digest(KIT / "runtime", ("**/*",))}|{sha256_file(Path(__file__))}|{kit_version()}'
|
|
f'|cpu {cpu_tag()}')
|
|
if step == 'compile':
|
|
return compile_key
|
|
# shaders: the built game and the translator that prepared them
|
|
return f'shaders-1|{compile_key}|{sha256_file(KIT / "tools" / "prepare_shaders.py")}|{sha256_file(KIT / "bin" / "nx2navi_xlate.id")}'
|
|
|
|
|
|
def main():
|
|
global EVENTS
|
|
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
|
ap.add_argument('--nup', help='the game as one .nup file')
|
|
ap.add_argument('--exefs', help='the game\'s exefs folder (with --romfs, instead of --nup)')
|
|
ap.add_argument('--romfs', help='the game\'s romfs folder')
|
|
ap.add_argument('--rebuild', action='store_true',
|
|
help='build again from the game files the last run unpacked into --out (after a kit update)')
|
|
ap.add_argument('--out', help='the folder for the finished game')
|
|
ap.add_argument('--check', action='store_true', help='only check the game\'s version; builds nothing')
|
|
ap.add_argument('--msys', default=r'C:\msys64\ucrt64', help='MSYS2 UCRT64 folder (default C:\\msys64\\ucrt64)')
|
|
ap.add_argument('--jobs', type=int, default=os.cpu_count() or 4, help='parallel compiles')
|
|
ap.add_argument('--events', action='store_true', help='machine-readable progress lines on stdout ("@@" + JSON)')
|
|
ap.add_argument('--make-runtime-lib', metavar='DIR', help='build the prebuilt runtime library of a release into DIR')
|
|
ap.add_argument('--obj', help='with --make-runtime-lib: the folder for its objects')
|
|
ap.add_argument('--until', choices=STEPS, help=argparse.SUPPRESS) # tests: stop after this step
|
|
a = ap.parse_args()
|
|
EVENTS = a.events
|
|
if a.make_runtime_lib:
|
|
make_runtime_lib(a.make_runtime_lib, Path(a.msys), a.jobs, a.obj)
|
|
return
|
|
given = bool(a.nup) + bool(a.exefs or a.romfs) + bool(a.rebuild)
|
|
if given != 1 or (not a.nup and not a.rebuild and not (a.exefs and a.romfs)):
|
|
ap.error('give --nup, both --exefs and --romfs, or --rebuild')
|
|
if a.rebuild and not a.out:
|
|
ap.error('--rebuild needs --out')
|
|
out = Path(a.out).resolve() if a.out else None
|
|
cache = out / 'build-cache' if out else None
|
|
state_path = cache / 'state.json' if cache else None
|
|
state = load_state(state_path) if state_path else {}
|
|
|
|
emit('step', step='check', state='start')
|
|
src = BuiltSource(out, state) if a.rebuild else Source(a)
|
|
check(src)
|
|
say('check: Ocarina of Time 1.1.1 (build e0f3ccfa)')
|
|
emit('check', version='1.1.1', build=MAIN_BUILD_ID[:8])
|
|
emit('step', step='check', state='done')
|
|
if a.check:
|
|
return
|
|
if not out:
|
|
ap.error('--out is needed to build')
|
|
msys = Path(a.msys)
|
|
for need in ('bin/clang.exe', 'bin/clang++.exe', 'bin/SDL3.dll', 'lib/libSDL3.dll.a', 'include/vma'):
|
|
if not (msys / need).exists():
|
|
fail(f'{msys / need} is missing: install MSYS2 and the packages README-BuildKit.md lists')
|
|
for mod in ('numpy', 'lz4') + (() if a.rebuild else ('zbic',)):
|
|
try:
|
|
__import__(mod)
|
|
except ImportError:
|
|
fail(f'the Python package {mod} is missing: pip install numpy lz4 zstandard zbic')
|
|
cache.mkdir(parents=True, exist_ok=True)
|
|
if state.get('version') != STATE_VERSION or state.get('input') != src.identity:
|
|
state = {'version': STATE_VERSION, 'input': src.identity, 'steps': {}}
|
|
state['kit'] = kit_version()
|
|
keep_awake()
|
|
t0 = time.time()
|
|
romfs_job = None
|
|
runs = {'exefs': lambda: step_exefs(src, out), 'romfs': lambda: step_romfs(src, out, min(a.jobs, 8)),
|
|
'lift': lambda: step_lift(out, cache), 'compile': lambda: step_compile(out, cache, msys, a.jobs, romfs_job),
|
|
'shaders': lambda: step_shaders(out)}
|
|
state_lock = threading.Lock() # the RomFS step records itself from its own thread
|
|
|
|
def save():
|
|
with state_lock:
|
|
state_path.write_text(json.dumps(state, indent=1), encoding='utf-8')
|
|
|
|
def run(name):
|
|
t = time.time()
|
|
runs[name]()
|
|
with state_lock:
|
|
state['steps'][name] = step_key(name, src.identity, out)
|
|
save()
|
|
say(f'{name}: {time.time() - t:.0f} s')
|
|
emit('step', step=name, state='done', seconds=round(time.time() - t, 1))
|
|
|
|
try:
|
|
for name in STEPS:
|
|
key = step_key(name, src.identity, out)
|
|
if state['steps'].get(name) == key:
|
|
say(f'{name}: done before')
|
|
emit('step', step=name, state='skip')
|
|
else:
|
|
# a step that runs makes every step after it run too: their keys hash its output, not its key
|
|
with state_lock:
|
|
for later in STEPS[STEPS.index(name):]:
|
|
state['steps'].pop(later, None)
|
|
say(f'{name}:')
|
|
emit('step', step=name, state='start')
|
|
if name == 'shaders' and romfs_job: # the shaders come out of the RomFS image
|
|
romfs_job.join()
|
|
if name == 'romfs' and a.until != 'romfs':
|
|
# The RomFS image is mostly disk work, the lift and the compile are processor work: they run side
|
|
# by side, and only the RomFS index and the files that include it wait for the image
|
|
# (step_compile). A first build takes about as long as the longer of the two, not their sum.
|
|
romfs_job = Background(lambda: run('romfs'))
|
|
else:
|
|
run(name)
|
|
if romfs_job and romfs_job.failed():
|
|
romfs_job.join()
|
|
if name == a.until:
|
|
break
|
|
if romfs_job:
|
|
romfs_job.join()
|
|
except (SystemExit, Exception):
|
|
# a failed lift or compile, or an error of this script: the RomFS image still finishes and is kept, so the
|
|
# next run starts after it (the thread is a daemon, and the process ending would cut its .part file short)
|
|
if romfs_job and romfs_job.thread.is_alive():
|
|
say('waiting for the RomFS image before stopping')
|
|
romfs_job.thread.join()
|
|
raise
|
|
save()
|
|
say(f'\nfinished in {(time.time() - t0) / 60:.1f} min: {out / EXE_NAME}')
|
|
emit('end', exe=str(out / EXE_NAME), seconds=round(time.time() - t0, 1), kit=kit_version())
|
|
|
|
|
|
if __name__ == '__main__':
|
|
main()
|