MacKwnoX/NX2navi
1
0
Fork 0
mirror of https://github.com/tangerineturtle1/NX2navi.git synced 2026-11-15 19:19:58 -04:00
Code Issues Packages Projects Releases Wiki Activity

Merge the launcher fixes: updates that stop keep the game, honest progress and errors, and a faster file check.

Browse Source 
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
chrg6
2026-11-10 16:08:15 -05:00
parent f17f4f9bcc 918ceab1bb
commit 92e3cc0adb
12 changed files with 2016 additions and 102 deletions
Download Patch File Download Diff File Expand all files Collapse all files

59
docs/launcher.md
View File

@@ -125,11 +125,17 @@ A path the Windows ANSI code page cannot hold breaks the MSYS2 tools, so the too
complete. No MSYS2 shell or pacman runs. Then zbic: the wheel for the bundled Python from PyPI, checked against
PyPI's sha256, into `pydeps/`.
2. **Inspect Your Game** to **Compile**: `tools/make_game.py --nup <dump> --out <game> --msys <toolchain> --events`,
whose `@@` progress lines drive the rows (check, exefs, romfs, lift, compile). How it keeps the build short:
whose `@@` progress lines drive the rows (check, exefs, romfs, lift, compile). Before the first long step it
looks at what can be known to fail later: the disk space of every step that will run (the RomFS image and 1 GiB,
2 GiB for the build cache, 4 GiB for the prepared shaders; `check_space` names each part and how much is
missing), and that the build tools compile and link an empty program (`probe_toolchain`). How it keeps the build
short:
- The RomFS image (26 GB, mostly disk work) is packed beside the lift and the compile (processor work), so a first
build takes about as long as the longer of the two. Only `romfs.c`, which includes the RomFS index's
`romfs_config.h`, waits for the image.
- Every logical processor compiles, the largest files first, so no big file starts last and runs on alone.
- Every logical processor compiles, the largest files first, so no big file starts last and runs on alone; fewer
compiles run when memory is short, as many as 80 % of the free memory holds at 768 MiB each (`make_game.py`
`default_jobs`).
- The lifted code compiles with MSYS2 CLANG64's clang when the toolchain has it: the same LLVM release as
UCRT64's, built with profile-guided optimisation, about a fifth faster on this code and writing the same objects
(it reads UCRT64's headers; `make_game.py` `fast_clang` checks that both report the same version and target).
@@ -148,14 +154,20 @@ A path the Windows ANSI code page cannot hold breaks the MSYS2 tools, so the too
and writes the Xbox, PlayStation, Steam Deck and keyboard and mouse prompts into `romfs-patches-*` beside the game (about
30 s), which the game's Controls > Button prompts loads. The keyboard and mouse prompts show the keys of
`nx2navi.ini`'s `bind_` lines, so Rebuild the Game after rebinding keys draws them again; the step's key names
all it reads (the RomFS, the art, the tools, those lines), so it runs again only when one changed or a folder is
gone, and runs no other step again. Without art the row is not shown and the game keeps its own prompts.
all it reads (the RomFS, the art, those lines) and the version of the tools (`make_game.py` `PROMPT_TOOLS`, changed
by hand when a change to them changes what they write: an edit of a script alone does not draw the prompts
again), so it runs again only when one changed or a folder is gone, and runs no other step again. Without art
the row is not shown and the game keeps its own prompts.
5. **Check the Game Files**: make_game.py's last step runs `tools/romfs_check.py`, once for each copy of the game data
(about 4 minutes, in worker threads, never while the game runs). It decompresses every file, writes
(about 4 minutes on players' PCs up to 0.7.1, in worker threads, never while the game runs; its key holds the
check's version, `SCAN_TOOL`, in the same way). It decompresses every file, writes
`romfs-check.json` beside the game, and for each damaged texture puts a stand-in, another texture of the same kind
under the damaged one's name, in `mods/Damaged file stand-ins`, so the game can load what needs it. A damaged file
of any other kind is only reported: only a clean dump restores it. In play the game logs a file it keeps opening
and cannot load (runtime/romfs.c `open_note`).
and cannot load (runtime/romfs.c `open_note`). The launcher shows what the check found (`jobs.Job.note_scan`):
the count in the step's row ("4 Flagged, 3 Stand-Ins"), and after a job in which the check ran a sentence on the
home screen and in `launcher.log`. The four files the check flags on every copy of the game (`jobs.SCAN_EXPECTED`)
are told as that, with nothing for the player to do; other files are counted and `romfs-check.json` named.
6. The game's Cetra Display is decoded for the window.
## Updating
@@ -169,6 +181,10 @@ directly. Every address is https (plain http only to this PC), a redirect to htt
sent on when a redirect leaves the host it was meant for (`launcher/net.py`). A manifest that does not give a file's
size and sha256 is refused.
GitHub allows an address 60 such requests an hour. The Updates row and F5 therefore start no check within a minute
of the last one (`FORCED_CHECK_PAUSE`), and a check GitHub refuses says when the allowance is whole again, from the
answer's `X-RateLimit-Reset` (`feed.retry_text`). A failed check is one line in `launcher.log`.
### In the background
The player never waits for a download and play is never interrupted. A check that finds a newer release (the check
@@ -240,17 +256,28 @@ The stub swaps `python.next` in while no Python runs, then starts the new versio
objects of 0.5.0 and earlier); Rebuild the Game compiles it for this processor's (`--retune`), and runs the
compile step for that even when nothing else changed (`retune_due`).
6. **Swap In the New Build**: lld links `ocarina.exe.new`, which replaces `ocarina.exe` only once it is complete. A
failed rebuild leaves the old build playable; Rebuild the Game tries again.
failed rebuild leaves the old build playable; Rebuild the Game tries again. A rebuild that lifted again and then
stopped has no finished compile on record (`build.built_version` is empty), but the old exe and the unpacked game
files are there: the launcher still counts the game as installed (`build.installed`), opens on home, where
Rebuild the Game reads "Not Finished", and never asks for the dump again. Begin Installation into a folder whose
game files are already unpacked builds on from them as well (`jobs.InstallJob.args`).
7. **Prepare the Shaders** is skipped once shaders were prepared before (`shader-prepared.bin`): the game keeps using
them and translates them again with the new translator in the background (`NX2NAVI_SHADER_CARRY`). With
`shader_carry = 0`, or nothing prepared yet, the step prepares them all as at the install.
`shader_carry = 0`, or nothing prepared yet, the step prepares them all as at the install. The step still builds
the prepared shaders into the graphics driver's cache again (`make_game.py` `warm_driver_cache`): seconds while
the driver has them, 20 minutes on a player's PC when the update changed what they are built with. Its bar
follows that count from 0 to 100 %, and the Updates row says an update takes from a few minutes to half an hour
or more (`launcher/ui/app.py` `UPDATE_TAKES`).
8. **Draw the Button Prompts**, with button art in the new version: only when the art, the prompt tools or the key
bindings changed, or a prompt folder is missing (a first version with art draws them all).
9. Every version but the current and the previous one is deleted, except a staged release newer than the current
one.
When the new version fails to start (exit code 86 from `start.pyw`, or any error within six seconds), the stub sets
`current.txt` back to `previous.txt`'s version and starts it with `--rolled-back`. The game is never rebuilt while it
`current.txt` back to `previous.txt`'s version and starts it with `--rolled-back`. `start.pyw` writes the failure to
`data/logs/crash.log` and shows it in a message box only when there is no version to go back to, when Steam started
it, or when the six seconds are over (`quiet_exit`): a box holds the exit code back until it is closed, and the
broken version then stayed the current one. The game is never rebuilt while it
runs. Keep Build Cache off deletes the lifted C and the objects (about 5 GB) after each build but keeps
`build-cache/state.json`, so updates still rebuild without the dump (compiling everything).
@@ -407,6 +434,20 @@ refuses: no manifest, another format or game, a newer format, paths outside the
job with a fake
`make_game.py` (with and without the prompts row), and the window's selftest when a display is there.
What goes wrong around a build is tested the same way, on made-up folders and stand-in processes: an update that
stops in its compile (the game stays installed, home says how to finish, the install goes on from the unpacked
files), `start.pyw` on a version that cannot be imported (exit code 86 at once, no message box, with a version to go
back to), a cut RomFS image and the free space, the object cache written while the compile runs and kept when it is
cut, a RomFS step that does not repeat the lift, the update's duration text and the warm-up's bar, a damaged dump
named by its file and an unexpected error by its message, the file check's result in its row and on home, the
game's FATAL line after exit codes 70 and 71, the job count by free memory and the build's priority class, a line
handler that raises and a log that cannot be written, one forced update check a minute and GitHub's reset time, a
game folder with characters outside the Basic Multilingual Plane (its `modules.c` compiled when a C compiler is
there), the file check at both piece sizes against the check as it was, the disk space of every step and the build
tools' test compile, and the prompt tool's unpacking. Two tests record the logic of `tools/romfs_check.py` and of
the three prompt scripts (their code without comments and docstrings) beside `SCAN_TOOL` and `PROMPT_TOOLS`: they
fail when that logic changes, to ask whether the step must run again on every install.
## Requirements and limits
- Windows 10 or 11, x86-64; the build's own requirements (README-BuildKit.md): 16 GB of RAM, about 30 GB free for

65
launcher/build.py
View File

@@ -14,6 +14,11 @@ RUN = ('import os, runpy, sys; p = sys.argv[1]; s = sys.argv[2]; sys.argv = sys.
'runpy.run_path(s, run_name="__main__")')
CREATE_NO_WINDOW = 0x08000000
CREATE_NEW_PROCESS_GROUP = 0x00000200
# The build runs below normal priority: a compile on every processor for ten minutes and more otherwise takes the
# PC from whatever the player does meanwhile, which the job screen invites ("Keep using your PC if you like"). On an
# idle PC it costs nothing. Windows hands a below-normal process's class on to the processes it starts, so the
# compilers, the lifter's workers and the shader preparation that make_game.py starts run that way too.
BELOW_NORMAL_PRIORITY_CLASS = 0x00004000
def command(layout, args):
@@ -61,33 +66,47 @@ class Run:
self.error = None
kw = {}
if os.name == 'nt':
kw['creationflags'] = CREATE_NO_WINDOW | CREATE_NEW_PROCESS_GROUP
kw['creationflags'] = CREATE_NO_WINDOW | CREATE_NEW_PROCESS_GROUP | BELOW_NORMAL_PRIORITY_CLASS
else:
kw['start_new_session'] = True
env = dict(os.environ, PYTHONUNBUFFERED='1', PYTHONIOENCODING='utf-8')
self.proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL,
cwd=cwd, env=env, text=True, encoding='utf-8', errors='replace', bufsize=1, **kw)
self.cancelled = False
self.dropped = 0 # lines and events whose handler raised (_read)
self._reader = threading.Thread(target=self._read, args=(on_event, on_line), daemon=True)
self._reader.start()
def _read(self, on_event, on_line):
"""Reads the build's output to its end, whatever the handlers do. A handler that raises (the launcher's log
on a full disk, a drawing error) loses its line, counted in dropped, and the reading goes on: when this
thread ended on the exception nobody read the pipe any more, the build stopped at its next line once the
pipe was full, and the job waited for it for good."""
def call(handler, arg):
try:
handler(arg)
except Exception: # noqa: BLE001 - any failure of a handler must leave the pipe read
self.dropped += 1
for line in self.proc.stdout:
line = line.rstrip('\r\n')
if line.startswith('@@'):
try:
ev = json.loads(line[2:])
except ValueError:
on_line(line)
call(on_line, line)
continue
if ev.get('ev') == 'error':
self.error = ev.get('message')
on_event(ev)
if isinstance(ev, dict):
if ev.get('ev') == 'error':
self.error = ev.get('message')
call(on_event, ev)
else: # "@@" before something that is no event
call(on_line, line)
elif line.startswith('error: ') and not self.error:
self.error = line[7:]
on_line(line)
call(on_line, line)
else:
on_line(line)
call(on_line, line)
def wait(self):
code = self.proc.wait()
@@ -148,6 +167,38 @@ def built_version(game_dir):
return ''
def scan_report(game_dir):
"""What the check of the game files found (tools/romfs_check.py writes <game>/romfs-check.json): (the paths of
the damaged files, how many of them got a stand-in), or None when the folder has no readable report."""
try:
with open(os.path.join(str(game_dir), 'romfs-check.json'), encoding='utf-8') as f:
report = json.load(f)
return sorted(report['damaged']), len(report.get('repaired') or ())
except (OSError, ValueError, KeyError, TypeError):
return None
def can_rebuild(game_dir):
st = build_state(game_dir)
return st.get('version') == 2 and all(s in st.get('steps', {}) for s in ('exefs', 'romfs'))
def unpacked(game_dir):
"""True when the game files a rebuild reads are in the game folder (make_game.py BuiltSource names the same
ones): the unpacked ExeFS, the RomFS image and its index."""
gd = str(game_dir)
return all(os.path.isfile(os.path.join(gd, *parts)) for parts in
(('exefs', 'main'), ('romfs', 'romfs.bin'), ('romfs', 'index.json')))
def installed(game_dir):
"""True when the game folder holds a game the player can start: a finished build, or the exe of the last
finished build beside a build that stopped later. make_game.py drops a step's record when a step before it
runs again, so an update that lifted again and then stopped in the compile (an error, Cancel, a power cut)
leaves state.json without `compile` and built_version says '': the old exe is still there and playable, and
a rebuild continues from the unpacked game files without the dump (can_rebuild)."""
if not game_dir:
return False
if built_version(game_dir):
return True
return os.path.isfile(os.path.join(str(game_dir), 'ocarina.exe')) and can_rebuild(game_dir)

2
launcher/cli.py
View File

@@ -98,7 +98,7 @@ def main(argv=None):
if a.cmd == 'continue-update':
gd = s.get('game_dir')
return follow(jobs.ContinueUpdateJob(ctx, gd if gd and build.built_version(gd) else '', a.old,
return follow(jobs.ContinueUpdateJob(ctx, gd if build.installed(gd) else '', a.old,
s.get('dump')))
if a.cmd == 'tools':

12
launcher/context.py
View File

@@ -34,12 +34,18 @@ class Context:
'it was is kept as launcher.json.bad')
def log(self, line):
"""One line (or several) into launcher.log. A log that cannot be written (a full disk, a file closed under
it) loses the line and nothing else: the callers log from the build's reader thread and from the window's
timers, where an exception would stop the build's output from being read or the timer from running."""
stamp = time.strftime('%H:%M:%S')
with self._lock:
if self._file:
for part in str(line).splitlines() or ['']:
self._file.write(f'{stamp} {part}\n')
self._file.flush()
try:
for part in str(line).splitlines() or ['']:
self._file.write(f'{stamp} {part}\n')
self._file.flush()
except (OSError, ValueError): # ValueError: the file was closed
pass
if self.echo:
print(line, flush=True)

30
launcher/feed.py
View File

@@ -14,6 +14,7 @@ The feed (app/<version>/feed.json, or the player's override in settings) is eith
"""
import json
import re
import time
import urllib.parse
from pathlib import Path
@@ -78,6 +79,30 @@ def _check_manifest(m, where):
return m
def retry_text(headers, now=None):
"""When GitHub takes requests again, for the message of a refused one: "after 14:32 (in 25 minutes)". From
X-RateLimit-Reset, the time the hourly allowance is whole again, when the response says the allowance is used
up (X-RateLimit-Remaining: 0), else from Retry-After, the seconds to wait. '' when the response says neither
or says something that cannot be right (more than a day)."""
now = time.time() if now is None else now
try:
if headers.get('X-RateLimit-Remaining') == '0' and headers.get('X-RateLimit-Reset'):
wait = int(headers.get('X-RateLimit-Reset')) - now
elif headers.get('Retry-After'):
wait = int(headers.get('Retry-After'))
else:
return ''
except (AttributeError, TypeError, ValueError):
return ''
if wait > 86400:
return ''
if wait <= 0: # the allowance is whole again by this PC's clock
return 'in a minute'
minutes = int(-(-wait // 60))
return (f'after {time.strftime("%H:%M", time.localtime(now + wait))} '
f'(in {minutes} minute{"" if minutes == 1 else "s"})')
def _github(feed, channel, token, version):
api = feed.get('api', GITHUB_API).rstrip('/')
headers = {'Accept': 'application/vnd.github+json', 'X-GitHub-Api-Version': '2022-11-28'}
@@ -89,8 +114,9 @@ def _github(feed, channel, token, version):
code = getattr(e, 'code', None)
if code == 404:
raise FeedError(f'no releases found for {feed["repo"]} (a private repository needs a token)') from e
if code == 403:
raise FeedError('GitHub refused the request (rate limit); try again in an hour') from e
if code in (403, 429): # GitHub answers either when the allowance of requests is used up
when = retry_text(getattr(e, 'headers', None))
raise FeedError(f'GitHub refused the request (rate limit); try again {when or "in an hour"}') from e
raise FeedError(f'GitHub could not be reached: {e}') from e
# The API lists the newest release first: the first one the channel takes that has a manifest is the answer.
for r in releases:

76
launcher/jobs.py
View File

@@ -13,6 +13,16 @@ RECHECK = 3600 # seconds between update checks while the launcher stays open,
PENDING, RUNNING, DONE, SKIPPED, FAILED = 'pending', 'running', 'done', 'skipped', 'failed'
# Failures whose message says all there is to say: the log gets one line for them (failure_line), not a traceback.
# A refused update check wrote 22 lines of two tracebacks into launcher.log each time.
PLAIN_FAILURES = (feed.FeedError,)
def failure_line(e):
"""A failure in one line for the log: its kind, its message and what it came from."""
cause = e.__cause__
return f'{type(e).__name__}: {e}' + (f' ({type(cause).__name__}: {cause})' if cause else '')
STEPS = {
'tools': ('Gather Build Tools', 'Downloading the compiler and libraries the build uses. This happens once and '
'takes about 170 MB.'),
@@ -40,6 +50,36 @@ WEIGHTS = {'tools': 12, 'tools_update': 4, 'check': 1, 'exefs': 2, 'romfs': 14,
'compile_changed': 55, 'shaders': 18, 'prompts': 4, 'scan': 5, 'ready': 0, 'download': 8, 'verify': 1, 'swap': 6}
# The game files tools/romfs_check.py reports as damaged on every copy of the game looked at so far: they are that
# way in the game's own data, not through a player's dump, so a new dump does not change them. The three textures
# get a stand-in; the game does without the fourth file.
SCAN_EXPECTED = frozenset((
'Tex/CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs',
'Tex/RBake_Scene_GanonStage_G_2_1_d0_t0.bntx.zs',
'Tex/Rock_WaterTemple_Gate_M_01_Nrm.bntx.zs',
'gsys/EnvObjSetMgrParam/Artwork.gsys__EnvObjSetMgrParam.bgyml',
))
def scan_summary(damaged, stood_in):
"""(the scan row's value, a sentence for the home screen) for what the check of the game files found
(build.scan_report); ('', '') when nothing is damaged. The files of SCAN_EXPECTED are told as what they are, the
same ones as on every copy: the player learns what the check did and that nothing needs doing. Other files are
counted and pointed to, without advice the check cannot back."""
n = len(damaged)
if not n:
return '', ''
files = '1 game file' if n == 1 else f'{n} game files'
stand = ('none got a stand-in' if not stood_in else '1 texture got a stand-in' if stood_in == 1 else
f'{stood_in} textures got a stand-in')
value = f'{n} Flagged, {stood_in} Stand-In' + ('' if stood_in == 1 else 's')
if set(damaged) <= SCAN_EXPECTED:
return value, (f'The file check flagged {files}, the same as in every copy of the game: {stand}, and nothing '
'needs doing.')
return value, (f'The file check flagged {files} as damaged: {stand}, so what uses them still loads. '
'romfs-check.json in the game folder lists them.')
def prompt_rows(ctx):
"""The prompts row, for a version with button art (make_game.py draws the prompts after the shaders), or none."""
return ['prompts'] if build.has_prompt_art(ctx.layout) else []
@@ -75,6 +115,7 @@ class Job:
self.started = time.time()
self.finished = None
self.after = None # what the window does next ('restart', ...)
self.note = '' # a sentence for the home screen after the job (note_scan)
self._lock = threading.Lock()
# ---- state the window reads
@@ -199,7 +240,7 @@ class Job:
self.result = 'cancelled'
else:
self.result, self.error = 'failed', str(e) or type(e).__name__
self.ctx.log(traceback.format_exc())
self.ctx.log(failure_line(e) if isinstance(e, PLAIN_FAILURES) else traceback.format_exc())
finally:
for s in self.steps:
if s.state == RUNNING:
@@ -212,6 +253,26 @@ class Job:
raise NotImplementedError
# ---- shared pieces
def note_scan(self, game_dir):
"""After a build: what the check of the game files found goes into its row, and when the check ran in this
job, into the note the home screen shows and into the log. The check itself only printed its counts into
the build's lines, which scroll away, so a player never learned that files were flagged or that stand-ins
are in use. A check kept from an earlier build shows its count in the row only."""
s = self.step('scan')
found = build.scan_report(game_dir)
if not s or s.state not in (DONE, SKIPPED) or not found:
return
value, note = scan_summary(*found)
if not value:
return
if s.state == DONE:
s.value = value
self.note = note
self.ctx.log(f'file check: {note}')
elif s.value == 'Unchanged':
s.value = f'Unchanged · {value}'
self._touch()
def ensure_tools(self, sid):
layout = self.ctx.layout
self.begin(sid)
@@ -301,13 +362,23 @@ class InstallJob(Job):
'scan', 'ready'])
self.dump, self.game_dir = str(dump), str(game_dir)
def args(self):
"""make_game.py's arguments. A game folder whose game files an earlier run already unpacked and checked
(build.can_rebuild, build.unpacked) is built on from those, for this processor's level as a first build is:
the dump is not read again, so an install that stopped after the RomFS image goes on although the dump
was moved or deleted since, and the 26 GB image is not packed a second time."""
if build.can_rebuild(self.game_dir) and build.unpacked(self.game_dir):
return build.rebuild_args(self.ctx.layout, self.game_dir, retune=True)
return build.install_args(self.ctx.layout, self.dump, self.game_dir)
def run(self):
self.ensure_tools('tools')
self.make_game(build.install_args(self.ctx.layout, self.dump, self.game_dir),
self.make_game(self.args(),
{s: s for s in ('check', 'exefs', 'romfs', 'lift', 'compile', 'shaders', 'prompts', 'scan')})
for sid in ('shaders', 'prompts', 'scan'):
if self.step(sid) and self.step(sid).state == PENDING:
self.skip(sid)
self.note_scan(self.game_dir)
self.finish('ready', 'Ready')
try:
gamefiles.extract_fonts(self.ctx.layout, self.game_dir)
@@ -340,6 +411,7 @@ class RebuildJob(Job):
for sid in ('lift', 'compile_changed', 'swap', 'shaders', 'prompts', 'scan'):
if self.step(sid) and self.step(sid).state == PENDING:
self.skip(sid)
self.note_scan(self.game_dir)
class UpdateDownloadJob(Job):

95
launcher/start.pyw
View File

@@ -1,27 +1,79 @@
"""What NX2navi.exe runs: python\\pythonw.exe app\\<version>\\launcher\\start.pyw [arguments].
The embeddable Python puts only its ._pth entries on sys.path, so the release folder (the parent of launcher/) goes
first here. A failure before the window is up is written to data/logs/crash.log, shown in a message box and ends the
process with exit code 86, which tells the stub to go back to the previous version (launcher/native/nx2navi.c).
first here. A failure before the window is up is written to data/logs/crash.log and ends the process with exit code
86, which tells the stub to go back to the previous version (launcher/native/nx2navi.c wWinMain). The stub waits
six seconds for that code (QUICK_MS), so the failure is shown in a message box only when the stub has no version to
go back to or no longer waits (quiet_exit): a box holds the exit code back until the player closes it, and a new
version that could not start then stayed the current one.
"""
import os
import sys
import traceback
APP = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, APP)
STARTUP_FAILED = 86
# The stub's wait for this process's exit code is QUICK_MS, 6 s (launcher/native/nx2navi.c start_release). An exit
# this long after the process was made still reaches it, with a second to spare for the exit itself.
STUB_WAITS = 5.0
def crash(text):
root = os.environ.get('NX2NAVI_LAUNCHER_ROOT') or os.path.dirname(os.path.dirname(APP))
def launcher_root():
return os.environ.get('NX2NAVI_LAUNCHER_ROOT') or os.path.dirname(os.path.dirname(APP))
def rollback_target(root):
"""The version the stub goes back to when this one ends with STARTUP_FAILED (launcher/native/nx2navi.c
wWinMain): the one app/previous.txt names, when it is another than this one and its folder holds a launcher.
'' when there is none."""
try:
logs = os.path.join(root, 'data', 'logs')
with open(os.path.join(root, 'app', 'previous.txt'), encoding='utf-8-sig') as f:
prev = (f.read().splitlines() or [''])[0].strip()
except (OSError, UnicodeDecodeError):
return ''
if not prev or prev == os.path.basename(APP):
return ''
return prev if os.path.isfile(os.path.join(root, 'app', prev, 'launcher', 'start.pyw')) else ''
def process_age():
"""Seconds since Windows made this process, which is when the stub began to wait for it. Infinite where that
cannot be read, so that the failure is then shown rather than left to a stub that may have stopped waiting."""
try:
import ctypes
from ctypes import wintypes
k = ctypes.windll.kernel32
k.GetCurrentProcess.restype = wintypes.HANDLE
k.GetProcessTimes.argtypes = [wintypes.HANDLE] + [ctypes.POINTER(wintypes.FILETIME)] * 4
made, ended, kernel, user, now = (wintypes.FILETIME() for _ in range(5))
if not k.GetProcessTimes(k.GetCurrentProcess(), *(ctypes.byref(t) for t in (made, ended, kernel, user))):
return float('inf')
k.GetSystemTimeAsFileTime(ctypes.byref(now))
ticks = [t.dwHighDateTime << 32 | t.dwLowDateTime for t in (made, now)] # 100 ns each
return max(0.0, (ticks[1] - ticks[0]) / 1e7)
except Exception:
return float('inf')
def quiet_exit(root, argv, age):
"""True when a failed start ends without a message box: the stub goes back to the previous version on the
exit code (rollback_target), and that version's home screen says which version did not start. Not when Steam
started this version without the stub (--play, launcher/steam.py PLAY_TEXT), and not once the stub has stopped
waiting (age, process_age): nobody acts on the exit code then, and without the box nothing would be seen."""
return bool(rollback_target(root)) and '--play' not in argv and age < STUB_WAITS
def crash(text, box=True):
"""The failure into data/logs/crash.log and, with box, into a message box that waits for the player."""
try:
logs = os.path.join(launcher_root(), 'data', 'logs')
os.makedirs(logs, exist_ok=True)
with open(os.path.join(logs, 'crash.log'), 'a', encoding='utf-8') as f:
f.write(text + '\n')
except OSError:
pass
if not box:
return
try:
import ctypes
ctypes.windll.user32.MessageBoxW(None, 'NX2navi could not start.\n\n' + text[-1500:], 'NX2navi', 0x10)
@@ -29,17 +81,22 @@ def crash(text):
print(text, file=sys.stderr)
try:
from launcher.ui.app import main
except Exception:
crash(traceback.format_exc())
sys.exit(STARTUP_FAILED)
def run():
sys.path.insert(0, APP)
try:
from launcher.ui.app import main
except Exception:
crash(traceback.format_exc(), box=not quiet_exit(launcher_root(), sys.argv[1:], process_age()))
sys.exit(STARTUP_FAILED)
try:
code = main()
except SystemExit:
raise
except Exception:
crash(traceback.format_exc())
sys.exit(1)
sys.exit(code or 0)
try:
code = main()
except SystemExit:
raise
except Exception:
crash(traceback.format_exc())
sys.exit(1)
sys.exit(code or 0)
if __name__ == '__main__':
run()

102
launcher/ui/app.py
View File

@@ -127,6 +127,31 @@ def exit_code_text(code):
return f'{shown} ({meaning})' if meaning else shown
FATAL_TAIL = 256 << 10 # bytes of the game's log last_fatal reads from its end
FATAL_SHOWN = 240 # characters of a FATAL line the home screen shows; the launcher's log has all of it
def last_fatal(game_dir):
"""The last FATAL line of the game's latest run, without its "FATAL: ", or ''. The game writes one before it
exits with 70 (runtime/dispatch.c rt_fatal) or 71 (runtime/main.c crash_handler) into logs/nx2navi.log, which
each run starts anew (runtime/main.c log_open). A line this launcher added for an earlier run's end
(note_exit_in_log) ends the search: what stands before it is not this run's."""
if not game_dir:
return ''
try:
with open(Path(game_dir) / 'logs' / 'nx2navi.log', 'rb') as f:
f.seek(max(0, f.seek(0, 2) - FATAL_TAIL))
lines = f.read().decode('utf-8', 'replace').splitlines()
except OSError:
return ''
for line in reversed(lines):
if line.startswith('[launcher] the game ended'):
return ''
if line.startswith('FATAL: '):
return line[len('FATAL: '):].strip()
return ''
def note_exit_in_log(game_dir, text):
"""Adds the exit code to the end of the game's log, so a log a player sends shows how the run ended even when the
process died before it could write a line itself."""
@@ -140,6 +165,39 @@ def note_exit_in_log(game_dir, text):
pass
def partial_image_gb(game_dir):
"""The size of what an install that was cut short left of the RomFS image (romfs/romfs.bin.part, up to 26 GB).
The next install writes the image anew (tools/make_game.py step_romfs), so that space is the install's to use:
the first-run screen counts it as free."""
try:
return (Path(game_dir) / 'romfs' / 'romfs.bin.part').stat().st_size / 1e9
except OSError:
return 0.0
def drop_partial_image(game_dir):
"""Removes that file (partial_image_gb), so that it does not count against the free space an install asks
for."""
try:
(Path(game_dir) / 'romfs' / 'romfs.bin.part').unlink(missing_ok=True)
except OSError:
pass
# Seconds after an update check before the Updates row or F5 starts another. Each check is a request to GitHub,
# which allows 60 an hour for an address: a player who selected the row 25 times in 90 seconds used the allowance
# up, and every check of the next hour was refused.
FORCED_CHECK_PAUSE = 60
# How long an update's rebuild takes, for the Updates row. No single figure holds: a version that changed little
# compiles a few files and links, minutes; one that changed the lifter translates and compiles the game's code again,
# and one that changed what the shaders are built with builds all of them into the graphics driver's cache again
# (tools/make_game.py warm_driver_cache). Players' updates took 24, 34 and 45 minutes where the text said "a few
# minutes".
UPDATE_TAKES = ('That takes a few minutes when little changed, and half an hour or more when the game\'s code or '
'its shaders are built again.')
def drive_of(path):
return Path(path).anchor.rstrip('\\/') or str(Path(path).parent)
@@ -653,9 +711,11 @@ class App:
elif a.continue_update is not None and not a.rolled_back:
# after a rollback the stub passes the failed version's --continue-update on to this one: the game was
# never rebuilt for the new version, so there is nothing to finish, and home tells why the version is back
self.run_job(jobs.ContinueUpdateJob(self.ctx, gd if gd and build.built_version(gd) else '',
self.run_job(jobs.ContinueUpdateJob(self.ctx, gd if build.installed(gd) else '',
a.continue_update, self.settings.get('dump')), 'home')
elif gd and build.built_version(gd):
elif build.installed(gd):
# also a game whose last update stopped in the compile (build.installed): home, where Play starts the
# build from before the update and Rebuild the Game finishes it, not the first-run screen and its dump
self.show('home')
self.after_update_notice()
if self.install_staged('a staged update waited for the launcher to start'):
@@ -729,7 +789,7 @@ class App:
'found in Downloads, Desktop or Documents.',)
gd = self.game_dir()
need = 30
free = free_gb(gd)
free = free_gb(gd) + partial_image_gb(gd) # Begin Installation removes the partial image (begin_install)
folder_desc = (f'The game is built into this folder. {free:.0f} GB free on {drive_of(gd)}; the game needs '
f'about {need} GB. Select this row to choose another folder.',)
ready = bool(dump)
@@ -826,6 +886,7 @@ class App:
self.choose_dump()
return
gd = self.game_dir()
drop_partial_image(gd) # before the free space is measured
if free_gb(gd) < 31 and not (Path(gd) / 'romfs' / 'romfs.bin').is_file():
self.message = f'{drive_of(gd)} has {free_gb(gd):.0f} GB free; the game needs about 30 GB.'
self.render()
@@ -870,11 +931,17 @@ class App:
upd = Row('value', 'Updates', f'Version {mine}', 'normal', big=True, icon='refresh',
on=lambda: self.check_now(force=True),
desc=(f'You have NX2navi {mine}. Select this row to look for a new version.',))
rebuild_value = f'Built with {versions.display(built)}' if built else 'Not built'
# an update that stopped in the compile left no finished build on record, but the exe of the build before it
unfinished = not built and build.installed(gd)
rebuild_value = (f'Built with {versions.display(built)}' if built else
'Not Finished' if unfinished else 'Not built')
rebuild_style = 'normal' if versions.display(built) == mine else 'salmon'
play_desc = ('Start Ocarina of Time. The launcher steps aside while you play and comes back when you quit.',)
if built and versions.display(built) != mine:
play_desc += (f'The game was built with {versions.display(built)}. Rebuild the Game brings it to {mine}.',)
elif unfinished:
play_desc += (f'The last rebuild of the game did not finish, so Play starts the build from before it. '
f'Rebuild the Game finishes it and brings the game to {mine}; your dump is not needed.',)
if staged and self.settings.get('auto_install'):
play_desc += (f'NX2navi {staged["version"]} is downloaded and installs after you quit the game.',)
if self.message:
@@ -1211,8 +1278,8 @@ class App:
nv, mine = versions.display(r.version), versions.display(self.layout.version)
notes = feed.plain_notes(r.notes, 6)
return {'title': f'Version {nv} Is Ready',
'text': [f'You have {mine}. Updating downloads the new version and rebuilds the game in a few '
f'minutes. Your saves and settings stay as they are.'],
'text': [f'You have {mine}. Updating downloads the new version and rebuilds the game. {UPDATE_TAKES} '
f'Your saves and settings stay as they are.'],
'divider': bool(notes), 'subtitle': "What's New" if notes else '', 'bullets': notes}
def staged_desc(self, m):
@@ -1226,7 +1293,7 @@ class App:
text = f'You have {mine}. Version {nv} is downloaded. Select this row to install it.'
return {'title': f'Version {nv} Is Ready', 'text': [text],
'divider': bool(notes), 'subtitle': "What's New" if notes else '', 'bullets': notes,
'note': 'Installing restarts the launcher and rebuilds the game in a few minutes. Your saves and '
'note': f'Installing restarts the launcher and rebuilds the game. {UPDATE_TAKES} Your saves and '
'settings stay as they are.'}
def staging_desc(self):
@@ -1254,6 +1321,11 @@ class App:
return
if not force and time.time() - self.settings.get('last_check', 0) < 600 and self.release:
return
if force and time.time() - self.check_started < FORCED_CHECK_PAUSE:
self.message = 'NX2navi looked for updates less than a minute ago.'
if self.screen == 'home':
self.show('home', self.focus)
return
self.check_error = ''
self.check_started = time.time()
self.check = jobs.CheckJob(self.ctx).start()
@@ -1314,8 +1386,17 @@ class App:
self.root.lift()
if code not in (0, None):
text = exit_code_text(code)
self.ctx.log(f'the game ended with code {text}')
self.message = f'The game ended with code {text}. Its log is logs\\nx2navi.log in the game folder.'
# 70 and 71 follow a FATAL line of the game's own, which names the error: the player gets that line
# and the launcher's log too, where the exit code alone said only that there was one
fatal = last_fatal(self.game_dir()) if code & 0xFFFFFFFF in (70, 71) else ''
if fatal:
shown = fatal if len(fatal) <= FATAL_SHOWN else fatal[:FATAL_SHOWN].rstrip() + '…'
self.ctx.log(f'the game ended with code {code & 0xFFFFFFFF}; its log says: FATAL: {fatal}')
self.message = (f'The game ended with a fatal error (code {code & 0xFFFFFFFF}): {shown}. Its log is '
'logs\\nx2navi.log in the game folder.')
else:
self.ctx.log(f'the game ended with code {text}')
self.message = f'The game ended with code {text}. Its log is logs\\nx2navi.log in the game folder.'
note_exit_in_log(self.game_dir(), text)
if self.install_staged('the game exited'):
return
@@ -1592,6 +1673,9 @@ class App:
self.message = 'The game is rebuilt and up to date.' if j.kind == 'rebuild' else \
f'NX2navi is now {versions.display(self.layout.version)}.'
self.release = None
note = getattr(j, 'note', '') # what the file check found (jobs.Job.note_scan)
if note and j.after is None:
self.message = f'{self.message} {note}'.strip()
self.reload_fonts_if_new()
self.show(self.job_return)
if j.kind == 'install' and self.settings.get('auto_check'):

1287
tests/launcher_test.py
View File

File diff suppressed because it is too large Load Diff

357
tools/make_game.py
View File

@@ -26,7 +26,8 @@ Steps, each skipped when a run before finished it for the same inputs (state in
tools, those lines), so it runs again only when one of them changed or a folder is gone, and running
it makes no other step run again. A failure here leaves the game and its shaders complete.
8. scan tools/romfs_check.py decompresses every file of the game data once, writes <out>/romfs-check.json and
makes stand-ins for damaged textures in <out>/mods. Its key names the game files and the tool, and it
makes stand-ins for damaged textures in <out>/mods. Its key names the game files and the check's
version (SCAN_TOOL, changed by hand: an edit of the tool alone does not run it again), and it
is last: nothing of it runs while the game does.
Without art the step is skipped and the game shows its own prompts, as before
@@ -35,10 +36,12 @@ therefore lifts and compiles again but keeps the unpacked game files, and --rebu
compile step keeps every object with the dependency list clang wrote for it (build-cache/obj/objects.json) and
compiles a file again only when it, a header it includes, the flags or the compiler changed; the kit's own folder
is not part of that key, so a kit in another folder (the launcher keeps each version in its own) reuses the objects.
The list goes to disk every 25 objects while the compile runs, so a compile that is cut short keeps what it finished.
A kit with lib/libnx2navirt.a built by --make-runtime-lib for the same runtime/ and the same compiler links the
runtime from it and compiles only the runtime files that include headers written per install. The new ocarina.exe
replaces the old one only once it links (with lld when the toolchain has it), so a failed build leaves the last good
one in place. Compiles run on every logical processor, the largest files first. The lifted code is built for this
one in place. Compiles run on every logical processor, or on as many as the free memory allows (default_jobs; --jobs
sets the number), the largest files first. The lifted code is built for this
PC's x86-64 level (cpu_level), with MSYS2 CLANG64's clang when it sits beside the UCRT64 folder and reports the same
version (fast_clang: the same objects, about a fifth faster).
@@ -63,6 +66,7 @@ by a JSON object (launcher/build.py reads them): {"ev": "step", "step": ..., "st
"""
import argparse
import concurrent.futures
import errno
import hashlib
import json
import os
@@ -74,6 +78,7 @@ import sys
import tempfile
import threading
import time
import traceback
from pathlib import Path
KIT = Path(__file__).resolve().parents[1]
@@ -91,11 +96,28 @@ STEPS = ('exefs', 'romfs', 'lift', 'compile', 'shaders', 'prompts', 'scan')
# Steps whose key names every input they read rather than the step before them: a step before them that runs does
# not make them run again, and they make no step after them run again.
OWN_KEY_STEPS = ('prompts', 'scan')
# (a step that runs, a later step that reads nothing it writes): the later step is not made to run again. The lift
# reads only the unpacked ExeFS (step_key), so the RomFS step leaves it alone: an install that was cut short while the
# image was still being packed, the lift beside it already done, packs the image again but does not lift again.
UNAFFECTED = {('romfs', 'lift')}
# The Python packages tools/pad_prompts.py imports (pip name: import name), checked before a build with button art.
PROMPT_PACKAGES = {'zstandard': 'zstandard', 'fonttools': 'fontTools', 'pillow': 'PIL'}
# The line tools/pad_prompts.py prints for each overlay folder it wrote.
PROMPTS_WRITTEN = re.compile(r'^.+: \d+ files, the font grew by -?\d+ bytes$')
SCAN_CHECKED = re.compile(r'^checked (\d+) of (\d+)$') # tools/romfs_check.py's progress lines
# What stands for tools/romfs_check.py in the scan step's key (step_key). The key held the sha256 of the script's
# bytes, so any edit of the script, a comment's too, made every install check its 26 GB of game files again, four
# minutes. It is this constant now, changed by hand when a change to the script can change what the check finds
# or writes, and only then. Its value is the sha256 the script had in 0.7.0 and 0.7.1, the releases that brought the
# check, so the key of every install that passed it stays what it is. tests/launcher_test.py
# test_the_file_checks_logic_is_the_recorded_one fails when the script's logic changes, to ask this question.
SCAN_TOOL = '55ea0c150aaf9dac93a2e6fd3b4ca11e1c572b5eee8b6bd74960535964384634'
# The same for the tools that draw the button prompts (tools/pad_prompts.py, prompt_art.py, button_icons.py) in the
# prompts step's key, which held the digest of the three scripts' bytes: two of four releases changed a script, and
# each time every install drew its prompts again, a minute. Changed by hand when a change to the scripts can change
# what they write. Its value is their digest in 0.7.0 and 0.7.1, so the key of an install made by those stays what it
# is. tests/launcher_test.py test_the_prompt_tools_logic_is_the_recorded_one asks the question when the logic changes.
PROMPT_TOOLS = 'ad59f2c4533f6ea2630b916ef5f3695f72daffe9b2b7663eea4168a4f08e4249'
# Bring-up fixes in the lifted SDK, each anchored on exact generated text that must occur once.
LIFT_FIXES = [
@@ -160,6 +182,16 @@ def fail(msg):
sys.exit(1)
def error_text(e):
"""An exception in a line a player can read: what the system said and about which file for a file error, the
full disk named as such, else the exception's kind and message."""
if isinstance(e, OSError) and e.strerror:
if e.errno == errno.ENOSPC:
return f'the disk is full ({e.filename})' if e.filename else 'the disk is full'
return e.strerror + (f': {e.filename}' if e.filename else '')
return f'{type(e).__name__}: {e}' if str(e) else type(e).__name__
class Background:
"""One step on a thread of its own. join() waits for it and raises its failure (fail's SystemExit included) in
the caller; failed() tells the caller early, so it can stop work that would be thrown away."""
@@ -368,6 +400,19 @@ def exefs_digest(out):
ROMFS_READ_AHEAD = 64 << 20 # total file bytes requested from workers in one group
def unreadable(rel, err):
"""Ends the build on a game file that could not be read while the RomFS image was packed, naming the file. A
file that does not match the dump's own checksum (nup.NupError) means the dump is damaged: a copy that went
wrong or a dump that was cut short, which only a new copy or a new dump mends."""
if isinstance(err, nup.NupError):
why = str(err).split(': ', 1)[-1] # without nup.py's "file <number>: ", which says nothing to a player
if why.startswith('damaged (') and why.endswith(')'):
why = why[len('damaged ('):-1]
fail(f'the dump is damaged: {rel}: {why}. Copy the dump to this PC again, or dump the game again, and then '
'try again')
fail(f'cannot read {rel} from the game files: {error_text(err)}')
def romfs_read_groups(files):
group, buffered = [], 0
for entry in files:
@@ -387,38 +432,71 @@ def step_romfs(src, out, jobs):
dst.mkdir(parents=True, exist_ok=True)
files = sorted(((rel, ref, src.romfs_size(ref)) for rel, ref in src.romfs), key=lambda e: e[0].encode())
total = sum(size for _, _, size in files)
# What a run that was cut short (Cancel, a full disk, a power cut) left of the image goes first: this run
# writes the image anew, and counted as used space its up to 26 GB made a retry on a nearly full disk fail the
# check below although the retry needs no more room than the first try.
part = dst / 'romfs.bin.part'
try:
part.unlink(missing_ok=True)
except OSError as e:
fail(f'cannot remove {part}, left by an earlier run ({e.strerror or e}): close whatever has it open')
free = shutil.disk_usage(dst).free
if free < total + (1 << 30):
fail(f'{dst} needs {total / 1e9:.1f} GB free for the RomFS image, it has {free / 1e9:.1f} GB')
say(f' {len(files)} files, {total / 1e9:.1f} GB')
entries, h, pos, done, t0 = [], hashlib.sha256(), 0, 0, time.time()
part = dst / 'romfs.bin.part'
emit('progress', step='romfs', done=0, total=total, unit='bytes')
def load(e): # small files are read whole on a worker; big ones are streamed by the writer
_, ref, size = e
return None if size > ROMFS_READ_AHEAD else b''.join(src.romfs_chunks(ref))
if size > ROMFS_READ_AHEAD:
return None
try:
return b''.join(src.romfs_chunks(ref))
except (nup.NupError, OSError) as err: # handed to the writer, which knows the file's name
return err
with open(part, 'wb', buffering=16 << 20) as img, concurrent.futures.ThreadPoolExecutor(jobs) as ex:
for batch, group in enumerate(romfs_read_groups(files)):
for (rel, ref, size), data in zip(group, ex.map(load, group)):
pad = -pos % ALIGN
if pad:
img.write(b'\0' * pad)
h.update(b'\0' * pad)
pos += pad
offset = pos
for chunk in ((data,) if data is not None else src.romfs_chunks(ref)):
img.write(chunk)
h.update(chunk)
pos += len(chunk)
if pos - offset != size:
fail(f'{rel} changed size while packing the RomFS: expected {size} bytes, read {pos - offset}')
entries.append({'path': rel, 'offset': offset, 'size': size})
done += size
emit('progress', step='romfs', done=done, total=total, unit='bytes')
if batch % 40 == 0:
say(f' RomFS {done / 1e9:5.1f} of {total / 1e9:.1f} GB, {time.time() - t0:.0f} s')
def pieces(rel, ref, data):
"""The file's bytes; a file of the game files that cannot be read, or does not match its checksum in the
dump, ends the step with its name (unreadable)."""
if isinstance(data, Exception):
unreadable(rel, data)
if data is not None:
yield data
return
try:
yield from src.romfs_chunks(ref)
except (nup.NupError, OSError) as err:
unreadable(rel, err)
try:
with open(part, 'wb', buffering=16 << 20) as img, concurrent.futures.ThreadPoolExecutor(jobs) as ex:
for batch, group in enumerate(romfs_read_groups(files)):
for (rel, ref, size), data in zip(group, ex.map(load, group)):
pad = -pos % ALIGN
if pad:
img.write(b'\0' * pad)
h.update(b'\0' * pad)
pos += pad
offset = pos
for chunk in pieces(rel, ref, data):
img.write(chunk)
h.update(chunk)
pos += len(chunk)
if pos - offset != size:
fail(f'{rel} changed size while packing the RomFS: expected {size} bytes, read {pos - offset}')
entries.append({'path': rel, 'offset': offset, 'size': size})
done += size
emit('progress', step='romfs', done=done, total=total, unit='bytes')
if batch % 40 == 0:
say(f' RomFS {done / 1e9:5.1f} of {total / 1e9:.1f} GB, {time.time() - t0:.0f} s')
except BaseException:
# a failed image is of no use, the next run writes it anew: up to 26 GB of it would only fill the disk
try:
part.unlink(missing_ok=True)
except OSError:
pass
raise
part.replace(dst / 'romfs.bin')
index = {'status': 'pass', 'romfs_size': pos, 'romfs_sha256': h.hexdigest(), 'files': entries}
(dst / 'index.json').write_text(json.dumps(index), encoding='utf-8')
@@ -463,6 +541,15 @@ def step_lift(out, cache):
# ---------------------------------------------------------------- compile
def c_string(text):
"""text as a C string literal for a source file in UTF-8, which modules.c is: JSON's escapes for the quote, the
backslash and the control characters are C's too, and every other character stands as itself. With JSON's
default, which writes each character outside ASCII as \\uXXXX, a character outside the Basic Multilingual Plane
came out as two surrogate halves (\\ud83e\\uddda), which C refuses ("invalid universal character"): a game
folder with an emoji or a rare Han character in its path failed at the end of the compile."""
return json.dumps(text, ensure_ascii=False)
def modules_c(lift, out, gen):
layout = json.loads((lift / 'layout.json').read_text(encoding='utf-8'))
L = ['#include "dispatch.h"', '#include "host.h"', '#include "layout.h"', '']
@@ -481,13 +568,13 @@ def modules_c(lift, out, gen):
L += [' return n;', '}', '', 'const RtFileHash rt_exefs_sha256[] = {']
L += [f' {{"{name}", "{sha256_file(out / "exefs" / name)}"}},' for name in EXEFS_FILES]
L += ['};', f'const int rt_exefs_sha256_count = {len(EXEFS_FILES)};', '',
f'const char rt_default_exefs[] = {json.dumps(str(out / "exefs"))};',
f'const char rt_default_romfs[] = {json.dumps(str(out / "romfs"))};',
f'const char rt_default_profile[] = {json.dumps(str(out / "profile"))};',
f'const char rt_default_exefs[] = {c_string(str(out / "exefs"))};',
f'const char rt_default_romfs[] = {c_string(str(out / "romfs"))};',
f'const char rt_default_profile[] = {c_string(str(out / "profile"))};',
'const int rt_game_version = 111;',
'const unsigned long long rt_app_pool_bytes = 9809ull << 20; /* the Switch 2 application pool */',
'const int rt_nvn_api_version[2] = {231, 40}; /* NVN2: main asks for major 231, minor 40 or later */',
f'const char rt_port_version[] = {json.dumps(kit_version())};']
f'const char rt_port_version[] = {c_string(kit_version())};']
p = gen / 'modules.c'
write_if_changed(p, '\n'.join(L) + '\n')
if WIN: # on Linux ui/menu_sdl.cpp defines it
@@ -781,6 +868,8 @@ class ObjectCache:
return 1
def save(self):
"""Whole or not at all: written beside the file and moved over it, so a process that ends in the middle
(Cancel) leaves the last complete objects.json."""
tmp = self.path.with_suffix('.tmp')
tmp.write_text(json.dumps(self.entries), encoding='utf-8')
tmp.replace(self.path)
@@ -842,6 +931,32 @@ def earlier_kits():
return sorted(kits)
# Peak memory of one compile (tools/build_game.py JOB_MEM: 120 to 180 MiB for a chunk of lifted code, 568 MiB for the
# largest file), and the share of the free memory the compiles may take in all; the rest stays with the system, the
# RomFS step beside them and whatever else the player runs.
JOB_MEM = 768 << 20
JOB_MEM_SHARE = 0.8
def default_jobs(cpus=None, free=None):
"""(compiles at a time, why) without --jobs: one per logical processor, but no more than JOB_MEM_SHARE of the
free memory holds at JOB_MEM each, and at least one. One per processor alone took no account of memory: on a
PC with many threads and little free memory the compiles filled it, the largest files first as they start
together. cpus, free: the processor count and the free bytes instead of this machine's (tests); without a
reading of the free memory (tools/hostmem.py) the processor count alone decides, as before."""
cpus = cpus or os.cpu_count() or 4
if free is None:
try:
import hostmem
free = hostmem.available_ram()
except (ImportError, OSError, AttributeError):
free = None
if free is None:
return cpus, f'{cpus} processors, free memory unknown'
by_mem = max(1, int(free * JOB_MEM_SHARE) // JOB_MEM)
return min(cpus, by_mem), f'{cpus} processors, {free / 2 ** 30:.1f} GiB of free memory allow {by_mem}'
SOURCE_WAIT, SOURCE_PAUSE = 60, 2 # seconds source_size waits for a drive in all, and between two tries
@@ -870,6 +985,11 @@ def source_size(src):
time.sleep(SOURCE_PAUSE)
# compile_all writes the object cache to disk after this many more objects, and at most this often (seconds): the
# cache of a whole game is some megabytes, and writing it after every object would take longer than the compiles.
CACHE_SAVE_EVERY, CACHE_SAVE_PAUSE = 25, 2.0
def compile_all(srcs, cmd_for, cache, jobs, env, progress=None, salt='', stop=None, alts=None):
"""Compile each (source, object) whose object is not current in cache. cmd_for(src, dst) -> argv without the
dependency-file options; salt identifies the compiler. Returns (compiled, reused, failures) with failures as
@@ -877,7 +997,10 @@ def compile_all(srcs, cmd_for, cache, jobs, env, progress=None, salt='', stop=No
compile time grows with size, and a big file that started last would run on alone after the others finished.
stop() true makes the compiles not yet started end at once, unrecorded (a step beside this one failed).
alts(src, dst) -> (keys, kits): the keys earlier versions gave the same compile, and the folders of their kits; an
object recorded under one of them is taken over (ObjectCache.adopt) instead of compiled again."""
object recorded under one of them is taken over (ObjectCache.adopt) instead of compiled again.
Each object is recorded when its compile ends, whichever started first, and the cache goes to disk every
CACHE_SAVE_EVERY objects: a compile that is cut short (Cancel ends the process, a crash, a power cut) keeps the
objects finished until then, where it used to keep none and the next run compiled the whole game again."""
todo, keys = [], {}
adopted = 0
for src, dst in srcs:
@@ -904,10 +1027,14 @@ def compile_all(srcs, cmd_for, cache, jobs, env, progress=None, salt='', stop=No
return src, dst, dep, p.returncode, p.stderr[-1500:]
done, bad = reused, []
unsaved, saved_at = 0, time.monotonic()
if progress:
progress(done, len(srcs))
with concurrent.futures.ThreadPoolExecutor(max(1, jobs)) as ex:
for src, dst, dep, rc, err in ex.map(cc, todo):
# submitted largest first, taken as they end: in submission order every result would wait for the largest
# file's, and nothing would be recorded during the minutes that one compiles
for result in concurrent.futures.as_completed([ex.submit(cc, job) for job in todo]):
src, dst, dep, rc, err = result.result()
done += 1
if rc is None: # not started (stop)
continue
@@ -916,6 +1043,13 @@ def compile_all(srcs, cmd_for, cache, jobs, env, progress=None, salt='', stop=No
cache.forget(dst)
else:
cache.record(dst, keys[dst], dep, src)
unsaved += 1
if unsaved >= CACHE_SAVE_EVERY and time.monotonic() - saved_at >= CACHE_SAVE_PAUSE:
try:
cache.save()
except OSError: # the save at the end still reports a disk that cannot be written
pass
unsaved, saved_at = 0, time.monotonic()
if progress:
progress(done, len(srcs))
cache.save()
@@ -997,6 +1131,107 @@ def preflight(msys, jobs, sources=None):
say(f'check: {len(sources)} runtime files and the libraries pass, {time.time() - t0:.0f} s')
def steps_to_run(state, identity, out, until=None):
"""The steps from exefs to shaders this build will run, as main's loop decides it: a step whose key is not the
recorded one, and every step after one that runs but those that read nothing of it (UNAFFECTED). until: the
step the build stops after (--until). For the checks made before the long steps (space_needs, probe_toolchain)."""
runs = []
for name in STEPS[:STEPS.index('shaders') + 1]:
if (any((earlier, name) not in UNAFFECTED for earlier in runs)
or state['steps'].get(name) != step_key(name, identity, out)):
runs.append(name)
if name == until:
break
return runs
GiB = 1 << 30
# Disk space the steps after the RomFS image take in the game folder, the least each is known to need: a build that
# has less is sure to stop, half an hour in when it is the shaders. Measured on a 0.7.1 game folder: the lifted C
# 1.16 GiB, its objects 0.34 GiB and the new exe beside the old one 0.3 GiB; with the lifted C already there only
# the changed files and the exe. A full shader preparation writes every translation as a loose file (a player's
# pack of two translators' translations held 8,290 MiB, 327,560 files) and the game's shader programs and pipeline
# cache, 0.84 GiB. Folding the translations into their pack takes as much again, but a pack that cannot be written
# does not stop the build (pack_shader_cache), so it is not asked for here.
SPACE_BUILD_CACHE, SPACE_REBUILD, SPACE_SHADERS = 2 * GiB, 1 * GiB, 4 * GiB
SPACE_IMAGE_MARGIN = 1 * GiB # beside the RomFS image, as step_romfs asks
def space_needs(runs, out, image_bytes=0, carried=False):
"""[(what, bytes)] this build will write into the game folder, for the steps it runs (steps_to_run).
image_bytes: the size of the RomFS image when that step runs; carried: the shaders step keeps the shaders
prepared before (step_shaders update) and writes none."""
needs = []
if 'romfs' in runs:
needs.append(('the RomFS image', image_bytes + SPACE_IMAGE_MARGIN))
if 'compile' in runs:
lifted = (out / 'build-cache' / 'lift' / 'layout.json').is_file()
needs.append(('the build cache (the translated code and its compiled files)',
SPACE_REBUILD if lifted else SPACE_BUILD_CACHE))
if 'shaders' in runs and not carried:
needs.append(('the prepared shaders', SPACE_SHADERS))
return needs
def check_space(out, needs):
"""Stops the build before its first long step when the game folder's disk cannot hold what the build will
write (space_needs), naming each part and how much is missing. Only the RomFS image's space was looked at,
and only when that step began: a disk that held the image but not the build cache or the shaders stopped the
build in the compile or, after half an hour, in the shader preparation. What an earlier run left of the image
counts as free: the RomFS step removes it (step_romfs)."""
need = sum(n for _, n in needs)
if not need:
return
free = shutil.disk_usage(out).free
try:
free += (out / 'romfs' / 'romfs.bin.part').stat().st_size
except OSError:
pass
if free >= need:
return
parts = ', '.join(f'{n / 1e9:.1f} GB for {what}' for what, n in needs)
fail(f'{out} has {free / 1e9:.1f} GB free and this build needs {need / 1e9:.1f} GB there: {parts}. Free '
f'{(need - free) / 1e9:.1f} GB more on that disk, or build into a folder on another one')
def probe_toolchain(msys, work):
"""Windows, before the long steps of a build whose compile step will run: an empty program is compiled and
linked with the installed toolchain, against the libraries the game links and with the linker the compile step
uses, in the folder work. Build tools that are incomplete, or that security software keeps from starting or
from writing an exe, stopped the build in its compile step, after the RomFS image and the lift, ten minutes and
more in; this takes a second. (Linux tries its compiler on the runtime's own sources: preflight.)"""
clang, clangxx, env, _ = toolchain(msys)
repair = (f'If the launcher installed these tools, delete the folder {msys.parent} and try again: it unpacks '
f'them anew. If you installed MSYS2 yourself, install the packages README-BuildKit.md lists')
shutil.rmtree(work, ignore_errors=True)
work.mkdir(parents=True)
try:
src, obj, exe = work / 'probe.c', work / 'probe.o', work / ('probe.exe' if WIN else 'probe')
src.write_text('int main(void) { return 0; }\n', encoding='ascii')
ld = ['-fuse-ld=lld'] if (msys / 'bin' / 'ld.lld.exe').is_file() else []
steps = (('compile', clang, [str(clang), '-c', '-std=gnu11', '-O1', '-o', str(obj), str(src)]),
('link', clangxx, [str(clangxx), *ld, '-o', str(exe), str(obj), '-L', str(msys / 'lib'), *LINK_LIBS]))
for what, tool, cmd in steps:
try:
p = subprocess.run(cmd, capture_output=True, text=True, env=env, errors='replace')
except OSError as e:
fail(f'{tool} does not start ({error_text(e)}): the build tools are damaged, or security software '
f'blocks them. {repair}')
if not p.returncode:
continue
said = (p.stderr.strip().splitlines() or [f'exit code {p.returncode}'])[-1][-300:]
if what == 'compile':
fail(f'{tool} cannot compile a test program ({said}): the build tools are incomplete, or security '
f'software blocks them. {repair}')
lacking = sorted(set(re.findall(r'unable to find library (-l\S+)', p.stderr)))
fail('the build tools cannot link the game: '
+ (f'{msys / "lib"} lacks {", ".join(lacking)}' if lacking else f'the linker says: {said}')
+ f'. {repair}')
finally:
shutil.rmtree(work, ignore_errors=True)
say('check: the build tools compile and link a test program')
def runtime_sources():
"""(source, object name) of every runtime file this OS's build compiles: the Win32 files on Windows, the SDL
window's (*_sdl.*) and platform/linux.c elsewhere (tools/build_game.py's foreign_os)."""
@@ -1502,19 +1737,24 @@ SHADER_STALL_S = 180 # no count moved for this long: the launcher says so (the
SHADER_BEAT_S = 10 # a progress event at least this often while the step runs
def shader_progress(phase, done, total, quiet_s, elapsed_s):
def shader_progress(phase, done, total, quiet_s, elapsed_s, build_only=False):
"""The shaders step's progress event fields: the translation fills the first half of the bar, the build into the
driver's cache the second (nx2navi.exe prints both counts, runtime/gfx_shader.c prepare_files and
runtime/vk_pipeline.c lb_warm_prepared); before either count, reading the game's shaders. quiet_s: seconds since a
count last moved, said once it passes SHADER_STALL_S, so a slow step never looks frozen without a word. The
detail ends without a full stop: the launcher adds one (launcher/ui/app.py)."""
detail ends without a full stop: the launcher adds one (launcher/ui/app.py).
build_only: a run that only builds (warm_driver_cache), whose count fills the whole bar: starting at one half,
the bar of a 20 minute build read 50% in its first second, and the launcher's time left, which follows the bar's
pace (launcher/jobs.py Job.eta), had half the bar to go by."""
clock = f'{int(elapsed_s) // 60}:{int(elapsed_s) % 60:02d}'
if phase is None:
fraction, detail = 0.0, f'Reading the game\'s shaders ({clock} so far)'
elif phase == 'translate':
fraction, detail = 0.5 * done / max(total, 1), f'Translated {done:,} of {total:,} shaders'
else:
fraction, detail = 0.5 + 0.5 * done / max(total, 1), f'Built {done:,} of {total:,} shaders for your graphics card'
part = done / max(total, 1)
fraction = part if build_only else 0.5 + 0.5 * part
detail = f'Built {done:,} of {total:,} shaders for your graphics card'
if quiet_s >= SHADER_STALL_S:
detail += (f'. No change for {int(quiet_s) // 60} min, but it is still running: one large shader can take '
'a while')
@@ -1574,10 +1814,10 @@ def shader_carry(out):
return (out / 'shader-prepared.bin').is_file() and bool(cache_identities(out))
def shader_run(cmd, cwd=None):
def shader_run(cmd, cwd=None, build_only=False):
"""Runs a shader preparation command and returns its exit code. Its output is read as it comes: the counts
nx2navi.exe prints become progress events, the other lines go on to the log, and while nothing moves an event
every few seconds keeps saying the step is running and for how long."""
every few seconds keeps saying the step is running and for how long. build_only: shader_progress."""
proc = subprocess.Popen(cmd, cwd=cwd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True,
encoding='utf-8', errors='replace', bufsize=1)
t0 = time.monotonic()
@@ -1588,7 +1828,7 @@ def shader_run(cmd, cwd=None):
def send():
now = time.monotonic()
emit('progress', **shader_progress(state['phase'], state['done'], state['total'], now - state['moved'],
now - t0))
now - t0, build_only))
def heartbeat():
while not stop.wait(SHADER_BEAT_S):
@@ -1651,7 +1891,7 @@ def warm_driver_cache(out):
say(" the prepared shaders are built into the graphics driver's cache again")
t0 = time.monotonic()
try:
rc = shader_run([str(exe), '--warm-shaders'], cwd=str(out))
rc = shader_run([str(exe), '--warm-shaders'], cwd=str(out), build_only=True)
except OSError as e:
rc = str(e)
if rc:
@@ -1678,6 +1918,8 @@ def step_shaders(out, update=False):
if update and shader_carry(out):
say(' the shaders prepared before this update stay in use; the game translates them again in the background')
if loose_shaders(out) >= PACK_LOOSE_MIN:
# minutes without a count (4 on a player's 328,000 files): the row says what runs meanwhile
emit('progress', step='shaders', fraction=0.0, detail='Folding the shader cache into one file')
pack_shader_cache(out)
warm_driver_cache(out)
return
@@ -1890,19 +2132,18 @@ def step_scan(out):
def step_key(step, identity, out=None):
"""What a step's output depends on. exefs and romfs: the game files only; lift: the unpacked ExeFS and the
lifter; compile: the lift, the runtime sources, this script and the processor (cpu_tag); prompts: the game files,
the kit's button art, the tools that draw it, the bind_ lines of nx2navi.ini, and whether every folder it writes
is still there."""
the kit's button art, the version of the tools that draw it (PROMPT_TOOLS), the bind_ lines of nx2navi.ini, and
whether every folder it writes is still there; scan: the game files, the check's version (SCAN_TOOL) and whether its report is still there."""
if step in ('exefs', 'romfs'):
return f'{step}-1|{identity}'
if step == 'scan':
written = 'written' if (out / 'romfs-check.json').is_file() else 'missing'
tool = KIT / 'tools' / 'romfs_check.py'
return f'scan-1|{identity}|{sha256_file(tool) if tool.is_file() else "no tool"}|{written}'
return f'scan-1|{identity}|{SCAN_TOOL if tool.is_file() else "no tool"}|{written}'
if step == 'prompts':
tools = tree_digest(KIT / 'tools', ('pad_prompts.py', 'prompt_art.py', 'button_icons.py'))
binds = sha256_bytes(json.dumps(bind_lines(out)).encode())
written = all((out / f / 'manifest.tsv').is_file() for f in prompt_art.OVERLAYS)
return (f'prompts-1|{identity}|{tree_digest(prompt_art_dir(), ("**/*.png",))}|{tools}|{binds}'
return (f'prompts-1|{identity}|{tree_digest(prompt_art_dir(), ("**/*.png",))}|{PROMPT_TOOLS}|{binds}'
f'|{"written" if written else "missing"}')
lift = f'lift|{exefs_digest(out)}|{lift_digest()}'
if step == 'lift':
@@ -1925,6 +2166,20 @@ def out_path(arg):
def main():
"""The build (build_main), with every error it does not report itself reported as fail reports one: the launcher
shows the message of the error event (launcher/build.py Run), and an exception that only printed its traceback
reached the player as "the build stopped (exit code 1)". The traceback still goes to the log."""
try:
build_main()
except nup.NupError as e:
fail(f'the dump is damaged: {e}. Copy the dump to this PC again, or dump the game again, and then try again')
except Exception as e: # noqa: BLE001 - whatever it is, the player gets its message
with OUT_LOCK:
traceback.print_exc()
fail(f'the build stopped: {error_text(e)}')
def build_main():
global EVENTS
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument('--nup', help='the game as one .nup file')
@@ -1939,13 +2194,17 @@ def main():
ap.add_argument('--check', action='store_true', help='only check the game\'s version; builds nothing')
ap.add_argument('--msys', default=r'C:\msys64\ucrt64',
help='MSYS2 UCRT64 folder (default C:\\msys64\\ucrt64; Windows only: Linux takes clang and lld from PATH)')
ap.add_argument('--jobs', type=int, default=os.cpu_count() or 4, help='parallel compiles')
ap.add_argument('--jobs', type=int,
help='parallel compiles (default: one per processor, fewer when the free memory is short)')
ap.add_argument('--events', action='store_true', help='machine-readable progress lines on stdout ("@@" + JSON)')
ap.add_argument('--make-runtime-lib', metavar='DIR', help='build the prebuilt runtime library of a release into DIR')
ap.add_argument('--obj', help='with --make-runtime-lib: the folder for its objects')
ap.add_argument('--until', choices=STEPS, help=argparse.SUPPRESS) # tests: stop after this step
a = ap.parse_args()
EVENTS = a.events
jobs_why = '--jobs'
if not a.jobs or a.jobs < 1:
a.jobs, jobs_why = default_jobs()
if a.make_runtime_lib:
make_runtime_lib(a.make_runtime_lib, Path(a.msys), a.jobs, a.obj)
return
@@ -1996,10 +2255,18 @@ def main():
# the compile step's key does not name the level: without this a finished compile reads as done before
state['steps'].pop('compile', None)
keep_awake()
# What can be known to fail later is looked at now: the disk space of every step that will run, and on Windows
# that the build tools compile and link (not for a build the tests stop before its compile step, --until)
ahead = steps_to_run(state, src.identity, out, a.until)
check_space(out, space_needs(ahead, out, sum(src.romfs_size(ref) for _, ref in src.romfs) if 'romfs' in ahead else 0,
a.rebuild and shader_carry(out)))
if WIN and 'compile' in ahead:
probe_toolchain(msys, cache / 'toolchain-check')
# Linux: the compiler, headers and libraries are the system's, so they are tried on the runtime before the long
# steps of a build that will reach the compile step (the first step whose key is not current makes the rest run)
if not WIN and any(state['steps'].get(n) != step_key(n, src.identity, out) for n in STEPS[:STEPS.index('compile') + 1]):
preflight(msys, a.jobs)
say(f'compiles at a time: {a.jobs} ({jobs_why})')
t0 = time.time()
romfs_job = None
runs = {'exefs': lambda: step_exefs(src, out), 'romfs': lambda: step_romfs(src, out, min(a.jobs, 8)),
@@ -2047,10 +2314,10 @@ def main():
emit('step', step=name, state='skip')
else:
# a step that runs makes every step after it run too: their keys hash its output, not its key (but
# for OWN_KEY_STEPS, whose keys name their inputs)
# for OWN_KEY_STEPS, whose keys name their inputs, and a step that reads nothing of it, UNAFFECTED)
with state_lock:
for later in STEPS[STEPS.index(name):]:
if later == name or not {name, later} & set(OWN_KEY_STEPS):
if later == name or not ({name, later} & set(OWN_KEY_STEPS) or (name, later) in UNAFFECTED):
state['steps'].pop(later, None)
say(f'{name}:')
emit('step', step=name, state='start')

24
tools/pad_prompts.py
View File

@@ -205,7 +205,7 @@ class Romfs:
def dictionary(self, dict_id):
if self.dicts is None:
self.dicts = {}
pack = zstandard.ZstdDecompressor().decompressobj().decompress(self.raw(DICTIONARIES))
pack = unpack(self.raw(DICTIONARIES))
for lo, hi in sarc_members(pack).values():
d = zstandard.ZstdCompressionDict(pack[lo:hi])
self.dicts[d.dict_id()] = d
@@ -218,7 +218,27 @@ class Romfs:
return raw, None
dict_id = zstandard.get_frame_parameters(raw).dict_id
d = self.dictionary(dict_id) if dict_id else None
return zstandard.ZstdDecompressor(dict_data=d).decompressobj().decompress(raw), d
return unpack(raw, d), d
# How much of a packed file unpack gives the decompressor at a time.
PIECE = 1 << 16
def unpack(raw, dictionary=None):
"""The contents of a zstd frame, with the dictionary it was packed with; bytes after the frame are left out.
The file goes to the decompressor in pieces of PIECE bytes and the pieces' outputs are joined once. Given the
whole file in one call, the library returns one bytes object that it grows by copying as the output comes: made-up
data of the size of the interface texture archive, 69 MB packed and 244 MB unpacked, took 21 s that way on a
fast PC against 0.3 s in pieces. Players' builds waited 44 to 52 s before this tool's first folder, most of
its minute."""
out, parts = zstandard.ZstdDecompressor(dict_data=dictionary).decompressobj(), []
for at in range(0, len(raw), PIECE):
parts.append(out.decompress(raw[at:at + PIECE]))
if out.eof:
break
return b''.join(parts)
def sarc_members(data):

9
tools/romfs_check.py
View File

@@ -35,7 +35,12 @@ import zstandard
STANDIN_MOD = 'Damaged file stand-ins'
DICTIONARIES = 'Pack/ZsDic.pack.zs'
ZSTD_MAGIC = b'\x28\xb5\x2f\xfd'
PIECE = 1 << 22
# How much of a file the decompressor is given at a time. Each call returns its output as one bytes object, which
# the library grows as the output comes: with 4 MiB of input a call's output is tens of MiB, grown by copying, and
# the copying holds the interpreter's lock, so the worker threads wait on each other. Measured on 32 MiB frames
# with 12 threads: about 120 MB/s of packed data with 4 MiB pieces, about 2,500 MB/s with 64 KiB pieces; files
# smaller than a piece are the same either way.
PIECE = 1 << 16
TEXTURE = re.compile(r'Tex/([^/]+)\.bntx\.zs')
@@ -110,6 +115,8 @@ def problem(image, entry):
size = 0
for at in range(0, len(data), PIECE):
size += len(out.decompress(data[at:at + PIECE]))
if out.eof: # the frame is whole: bytes after it are not the frame's, whichever piece they fall in
break
except zstandard.ZstdError as e:
return f'does not decompress ({e})'
if params.content_size and size != params.content_size: