Add the NX2navi launcher: a one-click setup that builds the game from the player's dump, starts it and keeps it up to date.

NX2navi-Setup.exe installs the launcher into %LOCALAPPDATA%\Programs\NX2navi without administrator rights, with
shortcuts and an Apps entry. The launcher finds the player's .nup, downloads the build tools itself (the MSYS2 UCRT64
packages toolchain.lock.json pins, checked by sha256 and unpacked without pacman, and zbic from PyPI) and runs
tools/make_game.py, whose steps it shows with their progress. Its window follows the game's options menu: colours
sampled from it, the menu's filigree, the white glowing cursor, the Triforce in the title band and as the programs'
icon, and all text in the game's Cetra Display, decoded from the player's own dump. Mouse, keyboard and an XInput
pad drive it.

Updates come from GitHub releases (NX2navi-update.json): the new version is downloaded, checked and staged beside
the running one, the stub restarts into it and falls back to the previous version when it fails to start, and the
game is rebuilt without the dump. make_game.py now keys its exefs and romfs steps on the game files alone, has
--rebuild and --events, compiles only the files whose source, headers, flags or compiler changed, and replaces
ocarina.exe only once the new one links. tools/make_launcher.py builds the release on Windows or Linux.

Tested: tests/launcher_test.py (19 tests), the real toolchain download and install, and under Wine 11 the setup,
the stub, the window on the bundled Windows Python, an update from a local feed by clicking Updates, the rollback
of a version that fails to start, and the downloaded clang compiling and linking a program against the game's
libraries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merged onto the current port: make_game.py keeps the launcher's step keys, cached link and in-place swap, and its
own runtime manifest (windres), AMD's FidelityFX library and the shader preparation, which becomes a step of its own
("shaders") that the launcher shows in its install, rebuild and update jobs. make_launcher.py ships the shader tools
and, with --ffx, AMD's library.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude
2026-11-07 19:14:39 +00:00
committed by chrg6
parent 514bb8eaa9
commit c403547c8b
35 changed files with 6422 additions and 83 deletions

View File

@@ -1,6 +1,6 @@
# NX2navi
Static recompilation of *Ocarina of Time* (Switch 2, ARM64, game version 1.1.1) into a native Windows x86-64 exe with a Vulkan renderer. Design: [docs/architecture.md](docs/architecture.md). Toolchain, build commands and tests: [docs/building.md](docs/building.md).
Static recompilation of *Ocarina of Time* (Switch 2, ARM64, game version 1.1.1) into a native Windows x86-64 exe with a Vulkan renderer. Design: [docs/architecture.md](docs/architecture.md). Toolchain, build commands and tests: [docs/building.md](docs/building.md). The players' launcher, setup and updates: [docs/launcher.md](docs/launcher.md).
## Game content boundary

View File

@@ -45,10 +45,12 @@ Report problems as issues, following [CONTRIBUTING.md](CONTRIBUTING.md).
## Installing
The build kit builds the game on your PC from your own dump (a `.nup` file, or ExeFS and RomFS folders) in about 10 to
15 minutes: [docs/README-BuildKit.md](docs/README-BuildKit.md). [docs/building.md](docs/building.md) describes building
from source. Settings are in the in-game menu and in `nx2navi.ini` ([nx2navi.ini.example](nx2navi.ini.example) lists every
key). Controls: [docs/controls.md](docs/controls.md).
Download `NX2navi-Setup-<version>.exe` from the releases and start it next to your `.nup` dump. The launcher it
installs finds the dump, downloads the build tools, builds the game on your PC (10 to 25 minutes, once) and starts
it; afterwards it keeps the port up to date and rebuilds only what changed ([docs/launcher.md](docs/launcher.md)).
[docs/README-BuildKit.md](docs/README-BuildKit.md) describes the same build by hand, [docs/building.md](docs/building.md)
building from source. Settings are in the in-game menu and in `nx2navi.ini` ([nx2navi.ini.example](nx2navi.ini.example)
lists every key). Controls: [docs/controls.md](docs/controls.md).
Requirements: Windows 10 or later, an x86-64 CPU with SSE4.2 (AVX2/FMA machines get a faster build), a GPU with Vulkan
1.3, `VK_EXT_external_memory_host` and `VK_EXT_mesh_shader`.

View File

@@ -107,3 +107,23 @@ folder's `licenses/` and in each component's own folder.
| MacNativeGames shader variant list | as in the installer release row above | MIT | `data/shader_list.tsv` |
| Weeping of the Empire menu cursor: eui's focus box read through ScreenMgr, BoxCursorMgr and the screens' node lists, AnimButton's hit test, the stale-hit-pane fallback to the parts panes, the three writes of moveBoxCursor, the structure offsets (1.0.0; the ScreenMgr instance placed in 1.4.3 by us) | https://github.com/daniel-madrid-07/weeping-of-the-empire @ `df0586cdc5702ca19bbebfa59697e0b6980f6237` (`engine/runtime/nx2navi_menu_cursor.cpp`), Copyright (c) 2026 daniel-madrid-07 | MIT | `runtime/menu_cursor.c` |
| AMD FidelityFX API headers (FSR 3.1 upscaling and frame generation) and its signed Vulkan library | https://github.com/GPUOpen-LibrariesAndSDKs/FidelityFX-SDK @ `v1.1.4` (`ffx-api/include/ffx_api/`, `PrebuiltSignedDLL/amd_fidelityfx_vk.dll`), Copyright (C) 2024 Advanced Micro Devices, Inc. | MIT | `runtime/third_party/ffx_api/` (unmodified, with `LICENSE.txt`), used by `runtime/vk_ffx.cpp`; the library ships in the build kit's `bin/` and beside the built game (never in this repository) |
## Distributed in the launcher release
`tools/make_launcher.py` assembles these into `NX2navi-Setup-<version>.exe` and the update archives of
`docs/launcher.md`. Versions and sha256 of the downloads are pinned in that script and in
`tools/requirements-launcher.txt`.
| Component | Source | License | Where in the release |
|---|---|---|---|
| CPython 3.14.9 embeddable package; includes `vcruntime140*.dll` (Microsoft, redistributable with Python), OpenSSL, libffi, SQLite, zlib, libmpdec | https://www.python.org/downloads/release/python-3149/ | PSF-2.0 | `python/` |
| `tkinter` and Tcl/Tk 9.0 (`_tkinter.pyd`, `tcl90.dll`, `tcl9tk90.dll`, `zlib1.dll`, `libtommath.dll`, `tcl/`), from python.org's 3.14.9 `amd64/tcltk.msi`, unpacked, never run | https://www.python.org/ftp/python/3.14.9/amd64/ | PSF-2.0; Tcl/Tk licence (BSD-style); libtommath public domain or BSD; zlib licence | `python/` |
| numpy 2.5.4 (with its bundled OpenBLAS), lz4 4.4.5, zstandard 0.25.0 | PyPI binary wheels, pins in `tools/requirements-launcher.txt` | BSD-3-Clause; lz4 BSD-3-Clause; zstandard BSD-3-Clause | `python/Lib/site-packages/` |
| zlib 1.4 (MSYS2 `mingw-w64-ucrt-x86_64-zlib`), static library | https://zlib.net, Copyright (C) Jean-loup Gailly and Mark Adler | zlib | linked into `NX2navi-Setup.exe` (its unzip) |
| mingw-w64 CRT startup code and winpthreads 14.0.0, GCC 16.2.0's `libgcc` (MSYS2 packages), static | https://www.mingw-w64.org, https://gcc.gnu.org | mingw-w64: ZPL-2.1 / MIT / public domain; libgcc: GPL-3.0 with the GCC Runtime Library Exception | linked into `NX2navi.exe` and `NX2navi-Setup.exe` |
Downloaded on the player's PC by the launcher, not distributed: the MSYS2 UCRT64 packages that
`toolchain.lock.json` pins (clang and LLVM, Apache-2.0 with LLVM exception; GCC and binutils, GPL-3.0; the
libraries of the "Compiler runtime in a source build" section and of `README-BuildKit.md`), from
https://repo.msys2.org/mingw/ucrt64/; and zbic 1.0.1.post1 (GPL-2.0), from PyPI. The game build links them as a
source build does.

View File

@@ -4,6 +4,9 @@ This kit builds *The Legend of Zelda: Ocarina of Time* (Nintendo Switch 2, game
program **on your PC, from your own copy of the game**. The kit contains no game code or data. Everything the build
writes (`ocarina.exe`, the RomFS image, the shader cache, your saves) comes from your game: keep it to yourself.
The NX2navi launcher (`NX2navi-Setup.exe`, [launcher.md](launcher.md)) does everything below for you, the
downloads included, and keeps the game up to date. This page is the same build by hand.
## What you need
- 64-bit Windows 10 or 11, 16 GB of RAM, a Vulkan 1.3 graphics card with mesh shaders and a current driver (tested on

View File

@@ -5,8 +5,8 @@ default build targets x86-64-v3 (AVX2/FMA); a CPU without them builds with `--cp
"Build"). Allow about 60 GB for the work folder. Extraction wants about 55 GB free while it runs and keeps about 35 GB
afterwards.
Players who only want to play can use the [installer](installer.md) instead: it builds the game on their machine
without this toolchain.
Players who only want to play use the [launcher](launcher.md) instead: it installs its own build tools, builds the
game on their machine and keeps it up to date.
## Toolchain

160
docs/launcher.md Normal file
View File

@@ -0,0 +1,160 @@
# Launcher
NX2navi's launcher installs everything the build needs, builds the game from the player's own dump, starts it, and
keeps the port up to date. Players download one file, `NX2navi-Setup-<version>.exe`; nothing of the game ships with
it. The code is `launcher/`, the release is made by `tools/make_launcher.py`, and the build itself is
`tools/make_game.py` (the build kit's script, [README-BuildKit.md](README-BuildKit.md)).
## What the player does
1. Starts `NX2navi-Setup.exe` (from the folder that holds their `.nup` dump, ideally). It installs the launcher into
`%LOCALAPPDATA%\Programs\NX2navi` without administrator rights, adds Start menu and Desktop shortcuts and an entry
in Settings > Apps, and opens the launcher. Windows SmartScreen warns once: the programs are not signed.
2. The launcher finds the dump by itself in the setup's folder, Downloads, Desktop, Documents or the top of each
drive, and proposes `<launcher>\game` as the install folder (or `X:\Games\NX2navi` on the drive with the most
free space when that drive has less than 40 GB). Both rows can be changed.
3. Selects **Begin Installation**. The launcher downloads the build tools (about 170 MB, once), checks the dump,
unpacks it, translates and compiles the game: 10 to 25 minutes. It keeps the PC awake meanwhile.
4. Selects **Play**. The launcher hides while the game runs and comes back when it exits.
From then on the launcher looks for a new version each time it opens (Settings can turn that off). When one is out,
the Updates row turns orange; selecting it downloads the new version, restarts the launcher in it and rebuilds the
game. Saves and settings are never touched.
## Look
The window follows the game's options menu, with colours sampled from it: a wooden title band with the Triforce and
the name, bronze label cells, dark value cells, orange On/Off choices, a white glowing cursor around the selected
row, the menu's filigree dividers and a description box for the selected row. All text is in the game's own Cetra
Display: the launcher decodes it from the player's dump or built game into `data/cache/fonts` (as
`runtime/menu_assets.c` does for the in-game menu) and uses the system's interface font until then. Every picture is
drawn in code (`launcher/ui/art.py`); no texture, font or sound of the game is in the repository or the release.
The Triforce is also the icon of `NX2navi.exe`, `NX2navi-Setup.exe`, the window and the taskbar
(`art.triforce_ico`, written into the release by `tools/make_launcher.py`).
Screens (`launcher/ui/app.py`): first run (dump, install folder, Desktop Shortcut, Check for Updates at Start,
Update Channel, Begin Installation); the steps of an install, update or rebuild with their progress, the build log's
last lines and Cancel, and after a failure Try Again and Copy Report; home (Play, Updates, Game Folder, Rebuild the
Game, Settings); settings (update check, Hide Launcher While Playing, Keep Build Cache, Update Channel, the folders,
Open Logs Folder, Uninstall, Back); uninstall (keep or remove the built game).
Controls: the mouse (hover moves the cursor, a click selects); the keyboard (arrows, Enter or Space, Esc, F5 checks
for updates); an XInput gamepad (d-pad or left stick, A, B, Y).
`python -m launcher --selftest --shots DIR` draws every screen with made-up states and saves a PNG of each.
## The launcher's folder
```
NX2navi.exe launcher/native/nx2navi.c: swaps in a staged Python, starts app\<current>\launcher\start.pyw
python/ python.org's embeddable Python 3.14.9 with tkinter, numpy, lz4 and zstandard
app/current.txt the version the stub starts; app/previous.txt the one before it
app/<version>/ launcher/, runtime/, recomp/, tools/, bin/nx2navi_xlate.exe, VERSION,
toolchain.lock.json (the pinned build tools), feed.json (where updates come from)
pydeps/ zbic, downloaded from PyPI on this PC
toolchain/msys64/ucrt64 MSYS2's clang and libraries, downloaded from the MSYS2 repository on this PC
toolchain/pkgs/ the downloaded package files
data/launcher.json settings; data/logs/launcher.log (and crash.log); data/cache/fonts
game/ the built game, unless another folder was chosen (its layout: README-BuildKit.md)
```
A path the Windows ANSI code page cannot hold breaks the MSYS2 tools, so the toolchain then goes to
`C:\NX2navi\toolchain` instead (`launcher/paths.py`).
## Installing
`jobs.InstallJob`, one row per step:
1. **Gather Build Tools** (`launcher/toolchain.py`): the packages `toolchain.lock.json` pins (MSYS2 UCRT64 clang,
gcc, SDL3, Opus, libvpx, dav1d, zstd, Vulkan headers and loader, VMA and their dependencies: 31 packages, 170 MB),
each checked against its sha256 and unpacked into `toolchain/msys64.new`, which replaces `toolchain/msys64` once
complete. No MSYS2 shell or pacman runs. Then zbic: the wheel for the bundled Python from PyPI, checked against
PyPI's sha256, into `pydeps/`.
2. **Inspect Your Game** to **Compile**: `tools/make_game.py --nup <dump> --out <game> --msys <toolchain> --events`,
whose `@@` progress lines drive the rows (check, exefs, romfs, lift, compile).
3. The game's Cetra Display is decoded for the window.
## Updating
A release carries `NX2navi-update.json` (version, notes, and the size and sha256 of each file). The feed
(`app/<version>/feed.json`) names a GitHub repository: the launcher reads its newest release (or pre-release, on the
Preview channel) that has the manifest. A private repository answers only with a token (`github_token` in
`data/launcher.json`); players need a public one. `feed_repo` in `data/launcher.json` points one launcher at another
repository for testing; `{"type": "url", "url": ".../NX2navi-update.json"}` in feed.json reads one manifest
directly.
`jobs.UpdateDownloadJob`, before the restart (`launcher/updater.py`):
1. **Download** `NX2navi-app-<version>.zip` (about 3 MB plus the shader translator) to `data/downloads`, resuming a cut
download, and check its size and sha256. When the manifest names a Python other than `python/runtime-id.txt`, also
`NX2navi-runtime-<id>.zip`, unpacked into `python.next`.
2. **Verify**: the archive unpacks into `app/<version>.part`, refusing names that leave the folder, holds the expected
`VERSION` and a launcher, and becomes `app/<version>`.
3. `app/previous.txt` gets the running version, `app/current.txt` the new one, and the launcher starts the stub with
`--continue-update <old>` and closes.
The stub swaps `python.next` in while no Python runs, then starts the new version, which finishes in
`jobs.ContinueUpdateJob`:
4. **Update Build Tools**: only when the new release pins other packages.
5. **Translate the Code** and **Compile Changed Files**: `make_game.py --rebuild`. The exefs and romfs steps depend
only on the game files, so they are kept and the dump is not needed again; the lift runs again and the compile
step compiles only the files whose source, included headers, flags or compiler changed (`build-cache/obj/
objects.json`, from clang's dependency files).
6. **Swap In the New Build**: the link writes `ocarina.exe.new`, which replaces `ocarina.exe` only once it is
complete. A failed rebuild leaves the old build playable; Rebuild the Game tries again.
7. Every version but the current and the previous one is deleted.
When the new version fails to start (exit code 86 from `start.pyw`, or any error within six seconds), the stub sets
`current.txt` back to `previous.txt`'s version and starts it with `--rolled-back`. The game is never rebuilt while it
runs. Keep Build Cache off deletes the lifted C and the objects (about 5 GB) after each build but keeps
`build-cache/state.json`, so updates still rebuild without the dump (compiling everything).
## Uninstalling
Settings > Uninstall NX2navi, or Settings > Apps > NX2navi (which runs `NX2navi.exe --uninstall`): the launcher
folder, its toolchain, the shortcuts and the Apps entry go; the built game and its saves stay unless "Remove the
Built Game" is On. The dump is never touched.
## Command line
`launcher/cli.py` runs the same jobs without the window (`python -m launcher.cli`, or in an installed launcher
`python\python.exe app\<version>\launcher\cli.py`): `check`, `update`, `tools`, `install --dump F --game D`,
`rebuild [--game D]`, `fonts`. `NX2navi.exe --desktop-shortcut on|off` and `--shortcuts on|off` are the stub's own.
`NX2navi-Setup.exe` takes `--dest DIR`, `--no-shortcuts`, `--no-launch` and `--quiet`.
## Releases
1. Bump `VERSION`; build the shader translator (`tools/build_xlate.py`).
2. With Python 3.14 and uv:
```powershell
.venv-py314\Scripts\python tools\make_launcher.py --xlate build\xlate --feed owner/name --notes notes.md
```
On Linux it cross-compiles the two programs with clang, lld and llvm-windres against the same MSYS2 packages and
unpacks Tcl/Tk with msiextract (msitools). It writes into `build/launcher-release`: `NX2navi-Setup-<version>.exe`
(the setup program with the launcher folder appended as a zip), `NX2navi-app-<version>.zip`,
`NX2navi-runtime-<id>.zip` and `NX2navi-update.json`. `check_content` runs over the folder first.
3. Upload those four files to GitHub release `v<version>` of the feed repository, by hand or with `--publish`
(`gh release create`; add `--draft` to look before it goes out). Launchers see it at their next start.
The release resolves the MSYS2 packages of the day into `toolchain.lock.json` (`--lock` reuses an older lock). The
repository keeps old package files, so a release keeps building with the compiler it was tested with.
## Tests
`python tests/launcher_test.py` (Python 3.14): versions, settings, the toolchain resolver and installer against a
local fake repository, resumed and checked downloads, the GitHub and URL feeds, staging, switching, rollback and
pruning, unsafe archives, the font decoding on a system font wrapped like the game's, `make_game.py` on a synthetic
`.nup` (events, skipped steps, `--rebuild` without the dump, step keys), the compile step's object cache with the
system's C compiler, the install job with a fake `make_game.py`, and the window's selftest when a display is there.
## Requirements and limits
- Windows 10 or 11, x86-64; the build's own requirements (README-BuildKit.md): 16 GB of RAM, about 30 GB free for
the game and 1.5 GB for the launcher and its tools, a Vulkan 1.3 GPU.
- The programs are not signed: SmartScreen warns on the first start of the setup.
- English only for now.
- The launcher builds the Windows game only; Linux builds use `docs/building.md`.

8
launcher/__init__.py Normal file
View File

@@ -0,0 +1,8 @@
"""The NX2navi launcher: installs the build tools, builds the game from the player's own dump, starts it and keeps
the port up to date (docs/launcher.md).
Standard library only, plus numpy for drawing the window (launcher/ui/art.py) and zstandard where Python lacks
compression.zstd. tkinter draws the window; everything that does work runs without it (launcher/cli.py).
"""
NAME = 'NX2navi'

6
launcher/__main__.py Normal file
View File

@@ -0,0 +1,6 @@
"""python -m launcher [--selftest [--shots DIR]] — the launcher window (launcher/ui/app.py)."""
import sys
from .ui.app import main
sys.exit(main())

135
launcher/build.py Normal file
View File

@@ -0,0 +1,135 @@
"""Running tools/make_game.py for the launcher: the command line, its "@@" progress lines (make_game.py --events)
and cancelling it with every compiler it started."""
import json
import os
import signal
import subprocess
import threading
# Runs make_game.py with pydeps/ (zbic) and the script's folder first on sys.path; the embeddable Python's ._pth
# file leaves both off.
RUN = ('import os, runpy, sys; p = sys.argv[1]; s = sys.argv[2]; sys.argv = sys.argv[2:]; '
'sys.path[:0] = [x for x in p.split(os.pathsep) if x] + [os.path.dirname(os.path.abspath(s))]; '
'runpy.run_path(s, run_name="__main__")')
CREATE_NO_WINDOW = 0x08000000
CREATE_NEW_PROCESS_GROUP = 0x00000200
def command(layout, args):
return [layout.python_for_builds(), '-I', '-c', RUN, str(layout.pydeps), str(layout.make_game), *map(str, args),
'--events']
def install_args(layout, dump, game_dir):
dump = str(dump)
if os.path.isdir(dump):
src = ['--exefs', os.path.join(dump, 'exefs'), '--romfs', os.path.join(dump, 'romfs')]
else:
src = ['--nup', dump]
return [*src, '--out', str(game_dir), '--msys', str(layout.msys)]
def rebuild_args(layout, game_dir):
return ['--rebuild', '--out', str(game_dir), '--msys', str(layout.msys)]
def check_args(dump):
dump = str(dump)
if os.path.isdir(dump):
return ['--exefs', os.path.join(dump, 'exefs'), '--romfs', os.path.join(dump, 'romfs'), '--check']
return ['--nup', dump, '--check']
class BuildError(Exception):
pass
class Run:
"""One make_game.py process. on_event(dict) for each "@@" line, on_line(str) for every other line; wait()
returns the exit code. cancel() ends the process and its children."""
def __init__(self, cmd, on_event, on_line, cwd=None):
self.error = None
kw = {}
if os.name == 'nt':
kw['creationflags'] = CREATE_NO_WINDOW | CREATE_NEW_PROCESS_GROUP
else:
kw['start_new_session'] = True
env = dict(os.environ, PYTHONUNBUFFERED='1', PYTHONIOENCODING='utf-8')
self.proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL,
cwd=cwd, env=env, text=True, encoding='utf-8', errors='replace', bufsize=1, **kw)
self.cancelled = False
self._reader = threading.Thread(target=self._read, args=(on_event, on_line), daemon=True)
self._reader.start()
def _read(self, on_event, on_line):
for line in self.proc.stdout:
line = line.rstrip('\r\n')
if line.startswith('@@'):
try:
ev = json.loads(line[2:])
except ValueError:
on_line(line)
continue
if ev.get('ev') == 'error':
self.error = ev.get('message')
on_event(ev)
elif line.startswith('error: ') and not self.error:
self.error = line[7:]
on_line(line)
else:
on_line(line)
def wait(self):
code = self.proc.wait()
self._reader.join(5)
return code
def cancel(self):
self.cancelled = True
if self.proc.poll() is not None:
return
if os.name == 'nt':
subprocess.run(['taskkill', '/T', '/F', '/PID', str(self.proc.pid)], capture_output=True,
creationflags=CREATE_NO_WINDOW)
else:
try:
os.killpg(self.proc.pid, signal.SIGTERM)
except (ProcessLookupError, PermissionError):
pass
def game_running(game_dir, exe='ocarina.exe'):
"""True when the built game runs from game_dir: Windows refuses to open a running exe for writing."""
path = os.path.join(str(game_dir), exe)
if os.name != 'nt' or not os.path.isfile(path):
return False
try:
with open(path, 'r+b'):
return False
except PermissionError:
return True
except OSError:
return False
def build_state(game_dir):
"""The game folder's build-cache/state.json (make_game.py), or {}."""
try:
with open(os.path.join(str(game_dir), 'build-cache', 'state.json'), encoding='utf-8') as f:
return json.load(f)
except (OSError, ValueError):
return {}
def built_version(game_dir):
"""The kit version that last built the game folder ('' when none finished)."""
st = build_state(game_dir)
if 'compile' in st.get('steps', {}) and os.path.isfile(os.path.join(str(game_dir), 'ocarina.exe')):
return st.get('kit', '')
return ''
def can_rebuild(game_dir):
st = build_state(game_dir)
return st.get('version') == 2 and all(s in st.get('steps', {}) for s in ('exefs', 'romfs'))

131
launcher/cli.py Normal file
View File

@@ -0,0 +1,131 @@
"""The launcher without its window, for scripts, tests and anyone who prefers a console:
python -m launcher.cli check is there a newer release?
python -m launcher.cli update download and stage it, then finish in the new version
python -m launcher.cli tools install the build tools and zbic
python -m launcher.cli install --dump F --game D build the game (the window's Begin Installation)
python -m launcher.cli rebuild [--game D] rebuild after an update (the window's Rebuild the Game)
python -m launcher.cli fonts [--dump F] decode the game's font for the window
python -m launcher.cli continue-update OLD what the new version runs after `update`
In an installed launcher: python\\python.exe app\\<version>\\launcher\\cli.py <command> (this file puts the release
folder on sys.path itself, as start.pyw does). Exit code 0 on success, 1 on failure, 3 when cancelled.
"""
import argparse
import os
import subprocess
import sys
import time
if __name__ == '__main__' and not __package__:
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
__package__ = 'launcher' # noqa: A001 — lets the relative imports below work when run as a script
from . import build, gamefiles, jobs, versions # noqa: E402
from .context import Context # noqa: E402
from .paths import Layout # noqa: E402
def follow(job):
"""Run a job here, printing each step's state as it changes."""
seen = {}
job.start()
while job.result is None:
for s in job.steps:
key = (s.state, s.value if s.state != jobs.RUNNING else int(s.fraction * 20))
if seen.get(s.id) != key:
seen[s.id] = key
print(f'{s.label}: {s.value}' + (f' ({s.detail})' if s.detail and s.state == jobs.RUNNING else ''),
flush=True)
time.sleep(0.2)
for s in job.steps:
if seen.get(s.id, (None,))[0] != s.state:
print(f'{s.label}: {s.value}', flush=True)
if job.result == 'failed':
print(f'error: {job.error}', file=sys.stderr, flush=True)
return {'ok': 0, 'failed': 1, 'cancelled': 3}[job.result]
def main(argv=None):
ap = argparse.ArgumentParser(prog='launcher.cli', description=__doc__.splitlines()[0])
ap.add_argument('--root', help='the launcher folder (default: this release\'s, or build/launcher-dev)')
sub = ap.add_subparsers(dest='cmd', required=True)
sub.add_parser('check')
sub.add_parser('update')
sub.add_parser('tools')
p = sub.add_parser('install')
p.add_argument('--dump', required=True)
p.add_argument('--game', required=True)
p = sub.add_parser('rebuild')
p.add_argument('--game')
p = sub.add_parser('fonts')
p.add_argument('--dump', default='')
p.add_argument('--game', default='')
p = sub.add_parser('continue-update')
p.add_argument('old')
a = ap.parse_args(argv)
ctx = Context(Layout(root=a.root) if a.root else None, echo=False)
s = ctx.settings
if a.cmd == 'check':
job = jobs.CheckJob(ctx).run_inline()
if job.result != 'ok':
print(f'error: {job.error}', file=sys.stderr)
return 1
r = job.release
if not r:
print('no release found')
elif jobs.update_available(ctx, r):
print(f'update available: {versions.display(r.version)} (you have {ctx.version})')
else:
print(f'up to date: {ctx.version} (newest release {versions.display(r.version)})')
return 0
if a.cmd == 'update':
check = jobs.CheckJob(ctx).run_inline()
if check.result != 'ok' or not jobs.update_available(ctx, check.release):
print(check.error or f'up to date: {ctx.version}')
return 0 if check.result == 'ok' else 1
job = jobs.UpdateDownloadJob(ctx, check.release)
code = follow(job)
if code or job.after != 'restart':
return code
new = ctx.layout.apps / versions.display(check.release.version)
cli = new / 'launcher' / 'cli.py'
print(f'finishing in {versions.display(check.release.version)}', flush=True)
return subprocess.call([ctx.layout.python_for_builds(), str(cli), '--root', str(ctx.layout.root),
'continue-update', ctx.version])
if a.cmd == 'continue-update':
gd = s.get('game_dir')
return follow(jobs.ContinueUpdateJob(ctx, gd if gd and build.built_version(gd) else '', a.old,
s.get('dump')))
if a.cmd == 'tools':
job = jobs.RebuildJob(ctx, '')
job.steps = job.steps[:1]
job.run = lambda: job.ensure_tools('tools_update')
return follow(job)
if a.cmd == 'install':
s['dump'], s['game_dir'] = os.path.abspath(a.dump), os.path.abspath(a.game)
s.save()
return follow(jobs.InstallJob(ctx, s['dump'], s['game_dir']))
if a.cmd == 'rebuild':
gd = os.path.abspath(a.game) if a.game else s.get('game_dir')
if not gd:
print('error: no game folder (give --game)', file=sys.stderr)
return 1
return follow(jobs.RebuildJob(ctx, gd, s.get('dump')))
if a.cmd == 'fonts':
got = gamefiles.extract_fonts(ctx.layout, a.game or s.get('game_dir'), a.dump or s.get('dump'))
for role, (path, fam) in got.items():
print(f'{role}: {fam} ({path})')
return 0 if got else 1
return 2
if __name__ == '__main__':
sys.exit(main())

48
launcher/context.py Normal file
View File

@@ -0,0 +1,48 @@
"""What every part of the launcher shares: the folder layout, the settings and the log (data/logs/launcher.log,
the run before it in launcher.1.log)."""
import os
import threading
import time
from .paths import Layout
from .settings import Settings
LOG_LIMIT = 4 << 20
class Context:
def __init__(self, layout=None, echo=False):
self.layout = (layout or Layout()).ensure()
self.settings = Settings(self.layout.settings)
self.version = self.layout.version
self.echo = echo
self._lock = threading.Lock()
path = self.layout.logs / 'launcher.log'
try:
if path.is_file() and path.stat().st_size > LOG_LIMIT:
os.replace(path, path.with_name('launcher.1.log'))
except OSError:
pass
try:
self._file = open(path, 'a', encoding='utf-8', errors='replace')
except OSError:
self._file = None
self.log(f'--- NX2navi launcher {self.version} ({"checkout" if self.layout.dev else "release"}) in '
f'{self.layout.root}')
def log(self, line):
stamp = time.strftime('%H:%M:%S')
with self._lock:
if self._file:
for part in str(line).splitlines() or ['']:
self._file.write(f'{stamp} {part}\n')
self._file.flush()
if self.echo:
print(line, flush=True)
def log_tail(self, n=80):
try:
with open(self.layout.logs / 'launcher.log', encoding='utf-8', errors='replace') as f:
return f.readlines()[-n:]
except OSError:
return []

144
launcher/feed.py Normal file
View File

@@ -0,0 +1,144 @@
"""Where updates come from.
Every release carries NX2navi-update.json (tools/make_launcher.py writes it):
{"format": 1, "version": "0.3.11", "build": "0.3.11+abc1234", "notes": "...",
"app": {"name": "NX2navi-app-0.3.11.zip", "size": n, "sha256": "..."},
"runtime": {"id": "py3.14.9-1", "name": "NX2navi-runtime-py3.14.9-1.zip", "size": n, "sha256": "..."},
"toolchain": "<lock digest>"}
The feed (app/<version>/feed.json, or the player's override in settings) is either
{"type": "github", "repo": "owner/name"} the newest GitHub release (or pre-release, on the preview channel)
that has the manifest; a token reads a private repository
{"type": "url", "url": "https://.../NX2navi-update.json"} one manifest; its files sit beside it
"""
import json
import urllib.parse
from . import net, versions
MANIFEST = 'NX2navi-update.json'
DEFAULT = {'type': 'github', 'repo': 'tangerineturtle1/NX2navi'}
GITHUB_API = 'https://api.github.com'
class FeedError(Exception):
pass
class Release:
def __init__(self, manifest, notes='', page='', assets=None, headers=None, prerelease=False):
self.manifest = manifest
self.version = manifest['version']
self.build = manifest.get('build', self.version)
self.notes = (manifest.get('notes') or notes or '').strip()
self.page = page
self.assets = assets or {} # name -> url
self.headers = headers or {} # extra request headers for the asset urls (a private repository's token)
self.prerelease = prerelease
def asset(self, key):
"""(url, headers, size, sha256) of the manifest's 'app' or 'runtime' file."""
entry = self.manifest.get(key)
if not entry:
raise FeedError(f'the {self.version} manifest has no {key} file')
url = self.assets.get(entry['name'])
if not url:
raise FeedError(f'the {self.version} release lacks {entry["name"]}')
return url, self.headers, entry.get('size'), entry.get('sha256')
def load(layout, settings):
feed = dict(DEFAULT)
try:
if layout.feed_file.is_file():
feed.update(json.loads(layout.feed_file.read_text(encoding='utf-8')))
except (OSError, ValueError):
pass
if settings.get('feed_repo'):
feed = {'type': 'github', 'repo': settings['feed_repo']}
return feed
def _check_manifest(m, where):
if not isinstance(m, dict) or m.get('format') != 1 or 'version' not in m or 'app' not in m:
raise FeedError(f'{where}: not an NX2navi update manifest')
versions.parse(m['version'])
return m
def _github(feed, channel, token, version):
api = feed.get('api', GITHUB_API).rstrip('/')
headers = {'Accept': 'application/vnd.github+json', 'X-GitHub-Api-Version': '2022-11-28'}
if token:
headers['Authorization'] = f'Bearer {token}'
try:
releases = net.get_json(f'{api}/repos/{feed["repo"]}/releases?per_page=20', headers, version=version)
except Exception as e:
code = getattr(e, 'code', None)
if code == 404:
raise FeedError(f'no releases found for {feed["repo"]} (a private repository needs a token)') from e
if code == 403:
raise FeedError('GitHub refused the request (rate limit); try again in an hour') from e
raise FeedError(f'GitHub could not be reached: {e}') from e
# The API lists the newest release first: the first one the channel takes that has a manifest is the answer.
for r in releases:
if r.get('draft') or (r.get('prerelease') and channel != 'preview'):
continue
assets = {a['name']: a for a in r.get('assets', [])}
if MANIFEST not in assets:
continue
if token: # a private repository's files come through the API with the token
urls = {n: a['url'] for n, a in assets.items()}
asset_headers = {'Accept': 'application/octet-stream', 'Authorization': f'Bearer {token}'}
else:
urls = {n: a['browser_download_url'] for n, a in assets.items()}
asset_headers = {}
try:
m = _check_manifest(json.loads(net.get_bytes(urls[MANIFEST], asset_headers, version=version)),
f'{r.get("tag_name")}/{MANIFEST}')
except (FeedError, ValueError, OSError) as e:
raise FeedError(f'{r.get("tag_name")}: {e}') from e
return Release(m, r.get('body') or '', r.get('html_url', ''), urls, asset_headers, bool(r.get('prerelease')))
return None
def _url(feed, version):
url = feed['url']
try:
m = _check_manifest(net.get_json(url, version=version), url)
except FeedError:
raise
except Exception as e:
raise FeedError(f'{url}: {e}') from e
base = url.rsplit('/', 1)[0] + '/'
assets = {e['name']: urllib.parse.urljoin(base, e['name']) for e in (m.get('app'), m.get('runtime')) if e}
return Release(m, assets=assets)
def latest(feed, channel='stable', token='', version='dev'):
"""The newest release the feed offers on the channel, or None when it has none."""
if feed.get('type') == 'url':
return _url(feed, version)
if feed.get('type') == 'github':
return _github(feed, channel, token, version)
raise FeedError(f'unknown feed type {feed.get("type")!r}')
def plain_notes(text, limit=12):
"""Release notes as short lines for the window: markdown bullets and headings without their marks."""
out = []
for line in (text or '').splitlines():
s = line.strip()
if not s or s.startswith(('<!--', '```', '---')):
continue
s = s.lstrip('#').strip()
for mark in ('- ', '* ', '+ '):
if s.startswith(mark):
s = s[2:]
s = s.replace('**', '').replace('`', '')
if s:
out.append(s)
if len(out) >= limit:
break
return out

322
launcher/gamefiles.py Normal file
View File

@@ -0,0 +1,322 @@
"""The player's game files, as far as the launcher reads them itself: finding a .nup dump, and the game's Cetra
Display font for the window (runtime/menu_assets.c reads the same font for the in-game menu's titles).
The font comes from the built game's romfs/ (romfs.bin and index.json, tools/make_game.py) or, before the first
build, straight from the .nup. They are decoded once into data/cache/fonts, which stays on the player's PC like the
rest of what is built from their game. Nothing of the game ships with the launcher.
- .zs files are zstd frames, most with one of the dictionaries of Pack/ZsDic.pack.zs (a SARC, compressed without a
dictionary; a frame names its dictionary by id).
- A .bfotf font is an OpenType file behind an 8-byte header, every big-endian 32-bit word XORed with one key: the
first word is 0x7f9a0218 under it, the second the payload size.
"""
import json
import os
import struct
import sys
from pathlib import Path
TITLE_ID = 0x0400486024DEC000
FONTS = { # role -> RomFS path; the window sets all its text in Cetra Display
'title': 'Font2/Scalable/CetraDisplayPro-Medium.Nin_Ounce_NVN2.bfotf.zs',
}
DICTS = 'Pack/ZsDic.pack.zs'
DICT_NAMES = ('zs.zsdic', 'pack.zsdic', 'bcett.byml.zsdic', 'bkprb.zsdic')
MAX_RAW = 64 << 20
def _nup_module(layout):
if str(layout.tools) not in sys.path:
sys.path.insert(0, str(layout.tools))
import nup
return nup
# ---------------------------------------------------------------- finding a dump
def candidate_dirs(extra=()):
home = Path.home()
dirs = [Path(d) for d in extra if d]
dirs += [home / 'Downloads', home / 'Desktop', home / 'Documents', home]
if os.name == 'nt':
import string
dirs += [Path(f'{c}:\\') for c in string.ascii_uppercase[2:] if os.path.isdir(f'{c}:\\')]
seen, out = set(), []
for d in dirs:
key = os.path.normcase(str(d))
if key not in seen and d.is_dir():
seen.add(key)
out.append(d)
return out
def is_game_nup(layout, path):
"""True when the file is a .nup holding Ocarina of Time (title 0400486024DEC000 or its update)."""
try:
n = _nup_module(layout).Nup(path)
except Exception:
return False
return any(p['title_id'] in (TITLE_ID, TITLE_ID | 0x800) for p in n.packages)
def find_dumps(layout, extra=(), depth=1):
""".nup files with the game in the usual places (the setup's folder first), newest first."""
found = []
for d in candidate_dirs(extra):
pattern_dirs = [d]
if depth > 0:
try:
pattern_dirs += [s for s in d.iterdir() if s.is_dir() and not s.name.startswith(('.', '$'))][:200]
except OSError:
pass
for pd in pattern_dirs:
try:
for f in pd.glob('*.nup'):
if f.is_file() and is_game_nup(layout, f):
found.append(f)
except OSError:
continue
uniq = {os.path.normcase(str(f)): f for f in found}
return sorted(uniq.values(), key=lambda f: f.stat().st_mtime, reverse=True)
def is_game_folder(path):
p = Path(path)
return (p / 'exefs' / 'main').is_file() and (p / 'romfs').is_dir()
# ---------------------------------------------------------------- reading RomFS files
class RomfsImage:
"""The built game's romfs/romfs.bin through its index.json."""
def __init__(self, folder):
folder = Path(folder)
index = json.loads((folder / 'index.json').read_text(encoding='utf-8'))
self.files = {e['path']: (e['offset'], e['size']) for e in index['files']}
self.bin = folder / 'romfs.bin'
def read(self, rel):
if rel not in self.files:
return None
off, size = self.files[rel]
with open(self.bin, 'rb') as f:
f.seek(off)
return f.read(size)
class NupRomfs:
"""The RomFS of the game's update (or base) package inside a .nup."""
def __init__(self, layout, path):
nup = _nup_module(layout)
self.n = nup.Nup(path)
pkgs = [p for p in self.n.packages if p['title_id'] in (TITLE_ID, TITLE_ID | 0x800)]
pkgs.sort(key=lambda p: (p['kind'] != 1, -p['version']))
self.trees = []
for p in pkgs:
if p['romfs'] < self.n.ndir:
self.trees.append(dict(self.n.tree(p['romfs'])))
def read(self, rel):
for t in self.trees:
if rel in t:
return self.n.read(t[rel])
return None
def open_romfs(layout, game_dir='', dump=''):
"""A RomFS reader for the built game, else the dump, else None."""
if game_dir and (Path(game_dir) / 'romfs' / 'index.json').is_file():
try:
return RomfsImage(Path(game_dir) / 'romfs')
except (OSError, ValueError, KeyError):
pass
if dump and Path(dump).is_file():
try:
return NupRomfs(layout, dump)
except Exception:
pass
if dump and is_game_folder(dump):
return FolderRomfs(Path(dump) / 'romfs')
return None
class FolderRomfs:
def __init__(self, folder):
self.folder = Path(folder)
def read(self, rel):
p = self.folder / rel
return p.read_bytes() if p.is_file() else None
# ---------------------------------------------------------------- zstd with the game's dictionaries
def _zstd():
try:
from compression import zstd # Python 3.14
return 'stdlib', zstd
except ImportError:
import zstandard
return 'zstandard', zstandard
def _frame_dict_id(data):
kind, z = _zstd()
if kind == 'stdlib':
return z.get_frame_info(data).dictionary_id
return z.get_frame_parameters(data).dict_id
def _decompress(data, dict_bytes=None):
kind, z = _zstd()
if kind == 'stdlib':
d = z.ZstdDict(dict_bytes) if dict_bytes else None
return z.decompress(data, zstd_dict=d)
dctx = z.ZstdDecompressor(dict_data=z.ZstdCompressionDict(dict_bytes) if dict_bytes else None)
return dctx.decompressobj().decompress(data)
def _dict_id(dict_bytes):
"""The id a zstd dictionary carries (magic 0xEC30A437, then the id)."""
if len(dict_bytes) >= 8 and struct.unpack_from('<I', dict_bytes, 0)[0] == 0xEC30A437:
return struct.unpack_from('<I', dict_bytes, 4)[0]
return 0
def sarc_members(data):
"""{name: bytes} of a SARC archive."""
if len(data) < 0x14 or data[:4] != b'SARC':
return {}
hl = struct.unpack_from('<H', data, 4)[0]
base = struct.unpack_from('<I', data, 0xC)[0]
if data[hl:hl + 4] != b'SFAT':
return {}
count = struct.unpack_from('<H', data, hl + 6)[0]
ents = hl + 0xC
names = ents + 16 * count + 8
if data[ents + 16 * count:ents + 16 * count + 4] != b'SFNT':
return {}
out = {}
for i in range(count):
_h, attr, start, end = struct.unpack_from('<IIII', data, ents + 16 * i)
no = names + (attr & 0xFFFF) * 4
name = data[no:data.index(b'\0', no)].decode('utf-8', 'replace')
if start <= end and base + end <= len(data):
out[name] = data[base + start:base + end]
return out
class ZsReader:
def __init__(self, romfs):
self.romfs = romfs
self._dicts = None
def dicts(self):
if self._dicts is None:
self._dicts = {}
raw = self.romfs.read(DICTS)
if raw:
members = sarc_members(_decompress(raw))
for name in DICT_NAMES:
if name in members:
self._dicts[_dict_id(members[name])] = members[name]
return self._dicts
def read(self, rel):
data = self.romfs.read(rel)
if data is None or not rel.endswith('.zs'):
return data
did = _frame_dict_id(data)
if did:
d = self.dicts().get(did)
if d is None:
return None
return _decompress(data, d)
return _decompress(data)
# ---------------------------------------------------------------- fonts
def bfotf_to_otf(b):
"""The OpenType (or TrueType) bytes inside a .bfotf/.bfttf, or None."""
if not b or len(b) < 12:
return None
key = struct.unpack_from('>I', b, 0)[0] ^ 0x7F9A0218
size = struct.unpack_from('>I', b, 4)[0] ^ key
if size < 12 or size > len(b) - 8 or size > MAX_RAW:
return None
body = b[8:8 + size] + b'\0' * (-size % 4)
words = struct.unpack(f'>{len(body) // 4}I', body)
out = struct.pack(f'>{len(words)}I', *(w ^ key for w in words))[:size]
if out[:4] != b'OTTO' and out[:4] != b'\x00\x01\x00\x00':
return None
return out
def font_family(otf):
"""The family name in an OpenType 'name' table (name id 16, else 1), or None."""
try:
num = struct.unpack_from('>H', otf, 4)[0]
for i in range(num):
tag, _cs, off, _len = struct.unpack_from('>4sIII', otf, 12 + 16 * i)
if tag != b'name':
continue
_fmt, count, strings = struct.unpack_from('>HHH', otf, off)
found = {}
for k in range(count):
pid, eid, lid, nid, ln, so = struct.unpack_from('>HHHHHH', otf, off + 6 + 12 * k)
if nid not in (1, 16):
continue
raw = otf[off + strings + so:off + strings + so + ln]
if pid == 3 or pid == 0:
name = raw.decode('utf-16-be', 'replace')
elif pid == 1:
name = raw.decode('mac_roman', 'replace')
else:
continue
rank = (nid == 16, pid == 3 and lid == 0x409)
if nid not in found or rank > found[nid][0]:
found[nid] = (rank, name)
for nid in (16, 1):
if nid in found:
return found[nid][1]
except (struct.error, IndexError):
return None
return None
def cached_fonts(layout):
"""{role: (path, family)} of the fonts decoded earlier."""
d = layout.cache / 'fonts'
out = {}
for role in FONTS:
p = d / f'{role}.otf'
meta = d / f'{role}.family'
if p.is_file() and meta.is_file():
out[role] = (p, meta.read_text(encoding='utf-8').strip())
return out
def extract_fonts(layout, game_dir='', dump=''):
"""Decode the game's Cetra Display into data/cache/fonts; {role: (path, family)} of the fonts found."""
romfs = open_romfs(layout, game_dir, dump)
if romfs is None:
return {}
zs = ZsReader(romfs)
d = layout.cache / 'fonts'
d.mkdir(parents=True, exist_ok=True)
out = {}
for role, rel in FONTS.items():
try:
otf = bfotf_to_otf(zs.read(rel))
except Exception:
otf = None
fam = font_family(otf) if otf else None
if not otf or not fam:
continue
(d / f'{role}.otf').write_bytes(otf)
(d / f'{role}.family').write_text(fam, encoding='utf-8')
out[role] = (d / f'{role}.otf', fam)
return out

368
launcher/jobs.py Normal file
View File

@@ -0,0 +1,368 @@
"""The launcher's long tasks, run on a worker thread: install, update, rebuild and the update check. Each job is a
list of steps the window shows as rows; the window reads the job's state on a timer (rev changes on every change)
and never blocks on it. The same jobs run headless from launcher/cli.py."""
import collections
import threading
import time
import traceback
from . import build, feed, gamefiles, net, toolchain, updater, versions
PENDING, RUNNING, DONE, SKIPPED, FAILED = 'pending', 'running', 'done', 'skipped', 'failed'
STEPS = {
'tools': ('Gather Build Tools', 'Downloading the compiler and libraries the build uses. This happens once and '
'takes about 170 MB.'),
'tools_update': ('Update Build Tools', 'Bringing the compiler and libraries to the versions this release was '
'tested with.'),
'check': ('Inspect Your Game', 'Making sure the dump is Ocarina of Time 1.1.1.'),
'exefs': ('Unpack the Program', "Unpacking the game's program files."),
'romfs': ('Pack the Game Data', 'Checking every game file against the dump and packing them into one image.'),
'lift': ('Translate the Code', "Each function of your game's program becomes C code for the compiler."),
'compile': ('Compile', 'Compiling the translated code and the port into a Windows program.'),
'compile_changed': ('Compile Changed Files', 'Compiling the files this version changed. Files that did not '
'change are kept from the last build.'),
'shaders': ('Prepare the Shaders', "Translating every shader of the game ahead of time, so play does not stop to "
'compile them.'),
'ready': ('Ready to Play', 'The game is built.'),
'download': ('Download {ver}', 'Downloading the new version of NX2navi.'),
'verify': ('Verify the Download', 'Checking the download against its checksum.'),
'swap': ('Swap In the New Build', 'Linking the new build. It replaces the old one only once it is complete.'),
}
WEIGHTS = {'tools': 12, 'tools_update': 4, 'check': 1, 'exefs': 2, 'romfs': 14, 'lift': 14, 'compile': 55,
'compile_changed': 55, 'shaders': 18, 'ready': 0, 'download': 8, 'verify': 1, 'swap': 6}
def mmss(seconds):
s = int(round(seconds))
return f'{s // 3600}:{s // 60 % 60:02d}:{s % 60:02d}' if s >= 3600 else f'{s // 60}:{s % 60:02d}'
class Step:
def __init__(self, sid, ver=''):
label, desc = STEPS[sid]
self.id, self.label, self.desc = sid, label.format(ver=ver), desc
self.state, self.value, self.fraction, self.detail = PENDING, 'Waiting', 0.0, ''
self.t0, self.seconds = None, 0.0
class Job:
kind = 'job'
title = ''
def __init__(self, ctx, step_ids, ver=''):
self.ctx = ctx
self.steps = [Step(s, ver) for s in step_ids]
self.rev = 0
self.cancel_flag = threading.Event()
self.proc = None
self.result = None # None while running, then 'ok', 'failed' or 'cancelled'
self.error = ''
self.lines = collections.deque(maxlen=500)
self.started = time.time()
self.finished = None
self.after = None # what the window does next ('restart', ...)
self._lock = threading.Lock()
# ---- state the window reads
def step(self, sid):
for s in self.steps:
if s.id == sid:
return s
return None
def current(self):
for s in self.steps:
if s.state in (RUNNING, FAILED):
return s
return next((s for s in self.steps if s.state == PENDING), self.steps[-1])
def overall(self):
total = sum(WEIGHTS[s.id] for s in self.steps) or 1
got = 0.0
for s in self.steps:
w = WEIGHTS[s.id]
got += w if s.state in (DONE, SKIPPED) else w * s.fraction if s.state == RUNNING else 0
return min(1.0, got / total)
def eta(self):
f = self.overall()
elapsed = time.time() - self.started
if f < 0.08 or elapsed < 20:
return None
return elapsed * (1 - f) / f
def elapsed(self):
return (self.finished or time.time()) - self.started
# ---- changes, from the worker
def _touch(self):
self.rev += 1
def log(self, line):
self.lines.append(line)
self.ctx.log(line)
self._touch()
def begin(self, sid, detail=''):
s = self.step(sid)
if s and s.state != RUNNING:
s.state, s.fraction, s.detail, s.t0, s.value = RUNNING, 0.0, detail, time.time(), '0%'
self._touch()
def progress(self, sid, fraction, detail=''):
s = self.step(sid)
if s:
if s.state == PENDING:
self.begin(sid)
s.fraction = max(0.0, min(1.0, fraction))
s.value = f'{int(s.fraction * 100)}%'
if detail:
s.detail = detail
self._touch()
def finish(self, sid, value=None):
s = self.step(sid)
if s:
s.seconds = time.time() - s.t0 if s.t0 else 0.0
s.state, s.fraction = DONE, 1.0
s.value = value or f'Done · {mmss(s.seconds)}'
self._touch()
def skip(self, sid, value='Unchanged'):
s = self.step(sid)
if s:
s.state, s.value = SKIPPED, value
self._touch()
def cancelled(self):
return self.cancel_flag.is_set()
def cancel(self):
self.cancel_flag.set()
if self.proc:
self.proc.cancel()
self._touch()
# ---- running
def start(self):
threading.Thread(target=self._main, name=f'job-{self.kind}', daemon=True).start()
return self
def run_inline(self):
self._main()
return self
def _main(self):
try:
self.run()
self.result = 'ok'
except net.Cancelled:
self.result = 'cancelled'
except Exception as e: # every failure ends the job with a message; the log has the traceback
if self.cancelled():
self.result = 'cancelled'
else:
self.result, self.error = 'failed', str(e) or type(e).__name__
self.ctx.log(traceback.format_exc())
finally:
for s in self.steps:
if s.state == RUNNING:
s.state = FAILED if self.result == 'failed' else PENDING
s.value = 'Stopped' if self.result == 'failed' else 'Cancelled'
self.finished = time.time()
self._touch()
def run(self):
raise NotImplementedError
# ---- shared pieces
def ensure_tools(self, sid):
layout = self.ctx.layout
self.begin(sid)
lock = toolchain.load_lock(layout, self.ctx.version)
if toolchain.is_current(layout, lock):
have_tools = True
else:
have_tools = False
def prog(phase, done, total):
if phase == 'download':
self.progress(sid, 0.9 * done / max(total, 1),
f'{done / 1e6:.0f} of {total / 1e6:.0f} MB downloaded')
else:
self.progress(sid, 0.9 + 0.1 * done / max(total, 1), f'Unpacking {done} of {total} packages')
toolchain.install(layout, lock, progress=prog, cancel=self.cancelled, log=self.log,
version=self.ctx.version)
toolchain.ensure_zbic(layout, cancel=self.cancelled, log=self.log, version=self.ctx.version)
if have_tools:
self.skip(sid, 'Unchanged' if sid == 'tools_update' else 'Ready')
else:
self.finish(sid)
def make_game(self, args, step_map, link_step=None):
"""Run make_game.py; its steps drive ours through step_map (make_game step -> our step id or None)."""
def on_event(ev):
kind = ev.get('ev')
sid = step_map.get(ev.get('step'))
if kind == 'step' and sid:
st = ev.get('state')
if st == 'start':
self.begin(sid)
elif st == 'done':
if sid == 'check':
self.finish(sid, 'Version 1.1.1')
elif link_step and sid in ('compile', 'compile_changed'):
self.finish(sid)
self.finish(link_step)
else:
self.finish(sid)
elif st == 'skip':
self.skip(sid, 'Kept' if sid in ('exefs', 'romfs') else 'Unchanged')
if link_step and sid in ('compile', 'compile_changed'):
self.skip(link_step, 'Unchanged')
elif kind == 'progress' and sid:
done, total, unit = ev.get('done', 0), max(ev.get('total', 1), 1), ev.get('unit', '')
if unit == 'bytes':
detail = f'{done / 1e9:.1f} of {total / 1e9:.1f} GB'
else:
detail = f'{done:,} of {total:,} {unit}'
self.progress(sid, done / total, detail)
elif kind == 'phase' and ev.get('phase') == 'link':
if link_step:
s = self.step(step_map.get('compile'))
if s and s.state == RUNNING:
self.finish(s.id)
self.begin(link_step, 'Linking')
else:
self.progress(sid or 'compile', 0.99, 'Linking')
elif kind == 'error':
self.error = ev.get('message', '')
self.proc = build.Run(build.command(self.ctx.layout, args), on_event, self.log)
code = self.proc.wait()
err = self.proc.error
self.proc = None
if self.cancelled():
raise net.Cancelled()
if code:
raise build.BuildError(err or f'the build stopped (exit code {code})')
class InstallJob(Job):
kind = 'install'
title = 'Installing'
def __init__(self, ctx, dump, game_dir):
super().__init__(ctx, ['tools', 'check', 'exefs', 'romfs', 'lift', 'compile', 'shaders', 'ready'])
self.dump, self.game_dir = str(dump), str(game_dir)
def run(self):
self.ensure_tools('tools')
self.make_game(build.install_args(self.ctx.layout, self.dump, self.game_dir),
{s: s for s in ('check', 'exefs', 'romfs', 'lift', 'compile', 'shaders')})
if self.step('shaders').state == PENDING:
self.skip('shaders')
self.finish('ready', 'Ready')
try:
gamefiles.extract_fonts(self.ctx.layout, self.game_dir)
except Exception as e:
self.log(f'fonts: {e}')
class RebuildJob(Job):
kind = 'rebuild'
title = 'Rebuilding'
def __init__(self, ctx, game_dir, dump=''):
super().__init__(ctx, ['tools_update', 'lift', 'compile_changed', 'swap', 'shaders'])
self.game_dir, self.dump = str(game_dir), dump
def run(self):
self.ensure_tools('tools_update')
layout = self.ctx.layout
if build.game_running(self.game_dir):
raise build.BuildError('the game is running: close it and try again')
if build.can_rebuild(self.game_dir):
args = build.rebuild_args(layout, self.game_dir)
elif self.dump:
args = build.install_args(layout, self.dump, self.game_dir)
else:
raise build.BuildError('the game folder has no earlier build: install the game again')
self.make_game(args, {'lift': 'lift', 'compile': 'compile_changed', 'shaders': 'shaders'}, link_step='swap')
for sid in ('lift', 'compile_changed', 'swap', 'shaders'):
if self.step(sid).state == PENDING:
self.skip(sid)
class UpdateDownloadJob(Job):
"""Before the restart: download, check and stage the release, then switch app/current.txt to it."""
kind = 'update'
def __init__(self, ctx, release):
ver = versions.display(release.version)
super().__init__(ctx, ['download', 'verify', 'tools_update', 'lift', 'compile_changed', 'swap', 'shaders'], ver)
self.release, self.ver = release, ver
self.title = f'Updating to {ver}'
def run(self):
layout = self.ctx.layout
self.begin('download')
prog = lambda d, t: self.progress('download', d / max(t, 1), f'{d / 1e6:.1f} of {t / 1e6:.1f} MB') # noqa: E731
folder = updater.stage_app(layout, self.release, progress=prog, cancel=self.cancelled,
version=self.ctx.version)
if updater.runtime_needed(layout, self.release):
updater.stage_runtime(layout, self.release, progress=prog, cancel=self.cancelled,
version=self.ctx.version)
self.finish('download')
self.begin('verify')
if not (folder / 'launcher' / 'start.pyw').is_file():
raise updater.UpdateError(f'{folder} is incomplete')
self.finish('verify', 'Checksum OK')
if layout.dev: # a checkout runs from the repository: nothing to switch to
self.after = 'staged'
return
updater.switch_to(layout, self.ver)
self.after = 'restart'
class ContinueUpdateJob(RebuildJob):
"""After the restart, in the new version: bring the tools to its lock and rebuild the game."""
kind = 'update'
def __init__(self, ctx, game_dir, old, dump=''):
Job.__init__(self, ctx, ['download', 'verify', 'tools_update', 'lift', 'compile_changed', 'swap', 'shaders'],
ctx.layout.version)
self.game_dir, self.dump, self.old = str(game_dir), dump, old
self.title = f'Updating to {versions.display(ctx.layout.version)}'
for sid, value in (('download', 'Done'), ('verify', 'Checksum OK')):
self.step(sid).state, self.step(sid).value, self.step(sid).fraction = DONE, value, 1.0
def run(self):
if self.game_dir and build.can_rebuild(self.game_dir):
RebuildJob.run(self)
else:
self.ensure_tools('tools_update')
for sid in ('lift', 'compile_changed', 'swap', 'shaders'):
self.skip(sid, 'No game yet')
updater.prune(self.ctx.layout)
class CheckJob(Job):
kind = 'check'
def __init__(self, ctx):
super().__init__(ctx, [])
self.release = None
def run(self):
s = self.ctx.settings
self.release = feed.latest(feed.load(self.ctx.layout, s), s.get('channel', 'stable'),
s.get('github_token', ''), self.ctx.version)
def overall(self):
return 0.0
def update_available(ctx, release):
return bool(release) and versions.newer(release.version, ctx.layout.version)

View File

@@ -0,0 +1,24 @@
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<!-- NX2navi.exe and NX2navi-Setup.exe: run as the user (no elevation prompt), draw at the display's real resolution,
use the current common controls and pass paths as UTF-8 to the C runtime. -->
<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
<trustInfo xmlns="urn:schemas-microsoft-com:asm.v3">
<security>
<requestedPrivileges>
<requestedExecutionLevel level="asInvoker" uiAccess="false"/>
</requestedPrivileges>
</security>
</trustInfo>
<application xmlns="urn:schemas-microsoft-com:asm.v3">
<windowsSettings>
<dpiAware xmlns="http://schemas.microsoft.com/SMI/2005/WindowsSettings">true/pm</dpiAware>
<dpiAwareness xmlns="http://schemas.microsoft.com/SMI/2016/WindowsSettings">PerMonitorV2</dpiAwareness>
<activeCodePage xmlns="http://schemas.microsoft.com/SMI/2019/WindowsSettings">UTF-8</activeCodePage>
</windowsSettings>
</application>
<compatibility xmlns="urn:schemas-microsoft-com:compatibility.v1">
<application>
<supportedOS Id="{8e0f7a12-bfb3-4fe8-b9a5-48fd50a15a9a}"/>
</application>
</compatibility>
</assembly>

166
launcher/native/common.h Normal file
View File

@@ -0,0 +1,166 @@
/* Shared by NX2navi.exe (nx2navi.c) and NX2navi-Setup.exe (setup.c): paths, small files, shortcuts and the
uninstall entry. Wide-character Win32 throughout, so folder names in any language work. */
#pragma once
#define COBJMACROS
#define WIN32_LEAN_AND_MEAN
#include <windows.h>
#include <objbase.h>
#include <shellapi.h>
#include <shlobj.h>
#include <shobjidl.h>
#include <stdio.h>
#include <string.h>
#include <wchar.h>
#define PATHN 1024
#define APP_NAME L"NX2navi"
#define UNINSTALL_KEY L"Software\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\NX2navi"
static inline void exe_dir(wchar_t *out, size_t n)
{
DWORD len = GetModuleFileNameW(NULL, out, (DWORD)n);
if (len == 0 || len >= n) { out[0] = 0; return; }
while (len > 0 && out[len - 1] != L'\\' && out[len - 1] != L'/') len--;
out[len > 0 ? len - 1 : 0] = 0;
}
static inline int path_exists(const wchar_t *p) { return GetFileAttributesW(p) != INVALID_FILE_ATTRIBUTES; }
static inline int is_dir(const wchar_t *p)
{
DWORD a = GetFileAttributesW(p);
return a != INVALID_FILE_ATTRIBUTES && (a & FILE_ATTRIBUTE_DIRECTORY);
}
/* The first line of a small UTF-8 text file, trimmed, as wide text; 0 when absent or empty. */
static inline int read_line(const wchar_t *path, wchar_t *out, size_t n)
{
char buf[256];
DWORD got = 0;
HANDLE f = CreateFileW(path, GENERIC_READ, FILE_SHARE_READ | FILE_SHARE_DELETE, NULL, OPEN_EXISTING, 0, NULL);
if (f == INVALID_HANDLE_VALUE) return 0;
if (!ReadFile(f, buf, sizeof buf - 1, &got, NULL)) got = 0;
CloseHandle(f);
buf[got] = 0;
char *s = buf;
if ((unsigned char)s[0] == 0xEF && (unsigned char)s[1] == 0xBB && (unsigned char)s[2] == 0xBF) s += 3;
while (*s == ' ' || *s == '\t') s++;
size_t len = strcspn(s, "\r\n");
while (len > 0 && (s[len - 1] == ' ' || s[len - 1] == '\t')) len--;
s[len] = 0;
if (!len) return 0;
return MultiByteToWideChar(CP_UTF8, 0, s, -1, out, (int)n) > 1;
}
static inline int write_line(const wchar_t *path, const wchar_t *text)
{
char buf[512];
int len = WideCharToMultiByte(CP_UTF8, 0, text, -1, buf, sizeof buf - 2, NULL, NULL);
if (len <= 0) return 0;
buf[len - 1] = '\n';
HANDLE f = CreateFileW(path, GENERIC_WRITE, 0, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
if (f == INVALID_HANDLE_VALUE) return 0;
DWORD put = 0;
BOOL ok = WriteFile(f, buf, (DWORD)len, &put, NULL);
CloseHandle(f);
return ok && put == (DWORD)len;
}
/* Compare dotted versions ("0.3.11" > "0.3.9"); build metadata after '+' is ignored. */
static inline int version_cmp(const wchar_t *a, const wchar_t *b)
{
for (;;) {
long x = 0, y = 0;
while (*a >= L'0' && *a <= L'9') x = x * 10 + (*a++ - L'0');
while (*b >= L'0' && *b <= L'9') y = y * 10 + (*b++ - L'0');
if (x != y) return x < y ? -1 : 1;
int ea = *a != L'.', eb = *b != L'.';
if (ea && eb) return 0;
if (!ea) a++;
if (!eb) b++;
if (ea && !eb) { while (*b == L'0' || *b == L'.') b++; return (*b >= L'1' && *b <= L'9') ? -1 : 0; }
if (eb && !ea) { while (*a == L'0' || *a == L'.') a++; return (*a >= L'1' && *a <= L'9') ? 1 : 0; }
}
}
static inline int known_folder(REFKNOWNFOLDERID id, wchar_t *out, size_t n)
{
PWSTR p = NULL;
if (FAILED(SHGetKnownFolderPath(id, 0, NULL, &p)) || !p) return 0;
int ok = wcslen(p) < n;
if (ok) wcscpy(out, p);
CoTaskMemFree(p);
return ok;
}
/* A .lnk at lnk that starts target (with its own icon). */
static inline int make_shortcut(const wchar_t *lnk, const wchar_t *target, const wchar_t *workdir, const wchar_t *desc)
{
IShellLinkW *sl = NULL;
IPersistFile *pf = NULL;
int ok = 0;
HRESULT init = CoInitializeEx(NULL, COINIT_APARTMENTTHREADED);
if (SUCCEEDED(CoCreateInstance(&CLSID_ShellLink, NULL, CLSCTX_INPROC_SERVER, &IID_IShellLinkW, (void **)&sl))) {
IShellLinkW_SetPath(sl, target);
IShellLinkW_SetWorkingDirectory(sl, workdir);
IShellLinkW_SetIconLocation(sl, target, 0);
IShellLinkW_SetDescription(sl, desc);
if (SUCCEEDED(IShellLinkW_QueryInterface(sl, &IID_IPersistFile, (void **)&pf))) {
ok = SUCCEEDED(IPersistFile_Save(pf, lnk, TRUE));
IPersistFile_Release(pf);
}
IShellLinkW_Release(sl);
}
if (SUCCEEDED(init)) CoUninitialize();
return ok;
}
/* The Desktop or Start menu shortcut's path. */
static inline int shortcut_path(int desktop, wchar_t *out, size_t n)
{
wchar_t dir[PATHN];
if (!known_folder(desktop ? &FOLDERID_Desktop : &FOLDERID_Programs, dir, PATHN)) return 0;
return swprintf(out, n, L"%ls\\%ls.lnk", dir, APP_NAME) > 0;
}
/* Create (on) or remove (off) the Desktop and/or Start menu shortcut to root\NX2navi.exe. */
static inline int set_shortcut(const wchar_t *root, int desktop, int on)
{
wchar_t lnk[PATHN], target[PATHN];
if (!shortcut_path(desktop, lnk, PATHN)) return 0;
if (!on) return DeleteFileW(lnk) || GetLastError() == ERROR_FILE_NOT_FOUND;
swprintf(target, PATHN, L"%ls\\NX2navi.exe", root);
return make_shortcut(lnk, target, root, L"Play and update NX2navi");
}
static inline void reg_str(HKEY k, const wchar_t *name, const wchar_t *value)
{
RegSetValueExW(k, name, 0, REG_SZ, (const BYTE *)value, (DWORD)((wcslen(value) + 1) * sizeof(wchar_t)));
}
static inline void reg_dword(HKEY k, const wchar_t *name, DWORD value)
{
RegSetValueExW(k, name, 0, REG_DWORD, (const BYTE *)&value, sizeof value);
}
/* The entry in Settings > Apps that runs "NX2navi.exe --uninstall". */
static inline int write_uninstall_entry(const wchar_t *root, const wchar_t *version, DWORD size_kb)
{
HKEY k;
wchar_t buf[PATHN + 64];
if (RegCreateKeyExW(HKEY_CURRENT_USER, UNINSTALL_KEY, 0, NULL, 0, KEY_SET_VALUE, NULL, &k, NULL) != ERROR_SUCCESS)
return 0;
reg_str(k, L"DisplayName", APP_NAME);
reg_str(k, L"DisplayVersion", version);
reg_str(k, L"Publisher", APP_NAME);
reg_str(k, L"InstallLocation", root);
swprintf(buf, PATHN + 64, L"\"%ls\\NX2navi.exe\",0", root);
reg_str(k, L"DisplayIcon", buf);
swprintf(buf, PATHN + 64, L"\"%ls\\NX2navi.exe\" --uninstall", root);
reg_str(k, L"UninstallString", buf);
reg_dword(k, L"NoModify", 1);
reg_dword(k, L"NoRepair", 1);
reg_dword(k, L"EstimatedSize", size_kb);
RegCloseKey(k);
return 1;
}

117
launcher/native/nx2navi.c Normal file
View File

@@ -0,0 +1,117 @@
/* NX2navi.exe: the launcher's own program, in the launcher folder (launcher/paths.py).
1. A Python staged by an update (python.next) replaces python\ while nothing runs from it.
2. The release named by app\current.txt starts: python\pythonw.exe app\<version>\launcher\start.pyw with this
program's arguments, the launcher folder as working folder and in NX2NAVI_LAUNCHER_ROOT.
3. When that release fails to start (exit code 86 from start.pyw, or any non-zero exit within six seconds) and
app\previous.txt names another complete release, current.txt goes back to it and it starts with
--rolled-back <version>.
Its own commands, for the launcher and the setup: --desktop-shortcut on|off, --shortcuts on|off (Desktop and Start
menu). No console window (-mwindows). */
#include "common.h"
#define STARTUP_FAILED 86
#define QUICK_MS 6000
static void message(const wchar_t *text)
{
MessageBoxW(NULL, text, APP_NAME, MB_OK | MB_ICONERROR);
}
static int release_ok(const wchar_t *root, const wchar_t *ver)
{
wchar_t p[PATHN];
swprintf(p, PATHN, L"%ls\\app\\%ls\\launcher\\start.pyw", root, ver);
return ver[0] && path_exists(p);
}
static void swap_python(const wchar_t *root)
{
wchar_t next[PATHN], cur[PATHN], old[PATHN];
swprintf(next, PATHN, L"%ls\\python.next", root);
if (!is_dir(next)) return;
swprintf(cur, PATHN, L"%ls\\python", root);
swprintf(old, PATHN, L"%ls\\python.old-%lu", root, (unsigned long)GetTickCount());
if (is_dir(cur) && !MoveFileExW(cur, old, 0)) return; /* in use: try again at the next start */
if (!MoveFileExW(next, cur, 0)) MoveFileExW(old, cur, 0);
}
/* Start the release; returns its exit code if it ended within QUICK_MS, else -1 (still running). */
static long start_release(const wchar_t *root, const wchar_t *ver, const wchar_t *args, const wchar_t *extra)
{
static wchar_t cmd[32768];
wchar_t python[PATHN];
swprintf(python, PATHN, L"%ls\\python\\pythonw.exe", root);
if (!path_exists(python)) {
message(L"The launcher's Python is missing. Run NX2navi-Setup.exe again to repair the installation.");
return 1;
}
swprintf(cmd, 32768, L"\"%ls\" \"%ls\\app\\%ls\\launcher\\start.pyw\" %ls %ls", python, root, ver, args, extra);
SetEnvironmentVariableW(L"NX2NAVI_LAUNCHER_ROOT", root);
STARTUPINFOW si = {.cb = sizeof si};
PROCESS_INFORMATION pi;
if (!CreateProcessW(python, cmd, NULL, NULL, FALSE, 0, NULL, root, &si, &pi)) {
message(L"The launcher could not be started. Run NX2navi-Setup.exe again to repair the installation.");
return 1;
}
CloseHandle(pi.hThread);
DWORD code = 0;
long result = -1;
if (WaitForSingleObject(pi.hProcess, QUICK_MS) == WAIT_OBJECT_0 && GetExitCodeProcess(pi.hProcess, &code))
result = (long)code;
CloseHandle(pi.hProcess);
return result;
}
static int shortcut_command(const wchar_t *root, int argc, wchar_t **argv)
{
for (int i = 1; i + 1 < argc; i++) {
int on = !wcscmp(argv[i + 1], L"on");
if (!wcscmp(argv[i], L"--desktop-shortcut")) return set_shortcut(root, 1, on) ? 0 : 1;
if (!wcscmp(argv[i], L"--shortcuts")) {
int a = set_shortcut(root, 1, on), b = set_shortcut(root, 0, on);
return a && b ? 0 : 1;
}
}
return -1;
}
int WINAPI wWinMain(HINSTANCE instance, HINSTANCE previous, PWSTR args, int show)
{
wchar_t root[PATHN], file[PATHN], ver[128], prev[128], extra[160];
(void)instance; (void)previous; (void)show;
exe_dir(root, PATHN);
if (!root[0]) return 1;
int argc = 0;
wchar_t **argv = CommandLineToArgvW(GetCommandLineW(), &argc);
if (argv) {
int r = shortcut_command(root, argc, argv);
LocalFree(argv);
if (r >= 0) return r;
}
swap_python(root);
swprintf(file, PATHN, L"%ls\\app\\current.txt", root);
if (!read_line(file, ver, 128) || !release_ok(root, ver)) {
swprintf(file, PATHN, L"%ls\\app\\previous.txt", root);
if (!read_line(file, ver, 128) || !release_ok(root, ver)) {
message(L"NX2navi's files are incomplete. Run NX2navi-Setup.exe again to repair the installation.");
return 1;
}
}
long code = start_release(root, ver, args ? args : L"", L"");
if (code == -1 || code == 0) return 0;
swprintf(file, PATHN, L"%ls\\app\\previous.txt", root);
if ((code == STARTUP_FAILED || code > 0) && read_line(file, prev, 128) && wcscmp(prev, ver) &&
release_ok(root, prev)) {
swprintf(file, PATHN, L"%ls\\app\\current.txt", root);
write_line(file, prev);
swprintf(file, PATHN, L"%ls\\app\\previous.txt", root);
DeleteFileW(file);
swprintf(extra, 160, L"--rolled-back %ls", ver);
start_release(root, prev, args ? args : L"", extra);
return 0;
}
if (code == STARTUP_FAILED) message(L"NX2navi could not start. data\\logs\\crash.log in its folder says why.");
return (int)code;
}

4
launcher/native/res.rc Normal file
View File

@@ -0,0 +1,4 @@
/* The icon (the Triforce, written by tools/make_launcher.py from launcher/ui/art.py triforce_ico into the build
folder, found through -I) and the manifest of NX2navi.exe and NX2navi-Setup.exe. */
1 ICON "nx2navi.ico"
1 24 "app.manifest"

382
launcher/native/setup.c Normal file
View File

@@ -0,0 +1,382 @@
/* NX2navi-Setup.exe: the one-click setup. This program with the launcher's folder appended as a zip.
It unpacks the zip into %LOCALAPPDATA%\Programs\NX2navi (no administrator rights), adds the Start menu and Desktop
shortcuts and the entry in Settings > Apps, and starts NX2navi.exe with --from <the setup's folder>, where the
launcher looks for the player's dump first. A small window in the launcher's colours shows while it unpacks.
When the folder already holds a newer version than the setup carries, the setup leaves the files alone (the
launcher updates itself) and only repairs the shortcuts.
The end of the file is a trailer: "NX2NSET1", the zip's offset and size (u64 little endian) and the version (24
bytes, NUL padded), written by tools/make_launcher.py. The zip is read here with zlib's inflate, so the setup
needs nothing from Windows beyond its own DLLs.
Options: --dest DIR, --no-shortcuts, --no-launch, --quiet (no window; for tests). */
#include "common.h"
#include <stdint.h>
#include <stdlib.h>
#include <zlib.h>
#define TRAILER 48
#define WM_DONE (WM_APP + 1)
typedef struct {
wchar_t exe[PATHN], dest[PATHN], src_dir[PATHN];
uint64_t off, size;
wchar_t version[32];
int shortcuts, launch, quiet;
int result; /* 0 ok, else an error message id */
wchar_t error[512];
uint64_t written;
} Setup;
static Setup S;
static HWND g_wnd;
static void fail_msg(const wchar_t *fmt, const wchar_t *arg)
{
swprintf(S.error, 512, fmt, arg ? arg : L"");
S.result = 1;
}
/* ---------------------------------------------------------------- reading our own file */
typedef struct { HANDLE f; uint64_t base, size; } Src;
static int src_read(Src *s, uint64_t at, void *buf, DWORD n)
{
LARGE_INTEGER li;
DWORD got = 0;
if (at + n > s->size) return 0;
li.QuadPart = (LONGLONG)(s->base + at);
if (!SetFilePointerEx(s->f, li, NULL, FILE_BEGIN)) return 0;
return ReadFile(s->f, buf, n, &got, NULL) && got == n;
}
static uint16_t le16(const unsigned char *p) { return (uint16_t)(p[0] | p[1] << 8); }
static uint32_t le32(const unsigned char *p) { return (uint32_t)p[0] | (uint32_t)p[1] << 8 | (uint32_t)p[2] << 16 | (uint32_t)p[3] << 24; }
static uint64_t le64(const unsigned char *p) { return (uint64_t)le32(p) | (uint64_t)le32(p + 4) << 32; }
static int read_trailer(void)
{
unsigned char t[TRAILER];
HANDLE f = CreateFileW(S.exe, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, 0, NULL);
if (f == INVALID_HANDLE_VALUE) return 0;
LARGE_INTEGER size, li;
DWORD got = 0;
int ok = GetFileSizeEx(f, &size) && size.QuadPart > TRAILER;
if (ok) {
li.QuadPart = size.QuadPart - TRAILER;
ok = SetFilePointerEx(f, li, NULL, FILE_BEGIN) && ReadFile(f, t, TRAILER, &got, NULL) && got == TRAILER &&
!memcmp(t, "NX2NSET1", 8);
}
CloseHandle(f);
if (!ok) return 0;
S.off = le64(t + 8);
S.size = le64(t + 16);
char v[25];
memcpy(v, t + 24, 24);
v[24] = 0;
MultiByteToWideChar(CP_UTF8, 0, v, -1, S.version, 32);
return S.off + S.size <= (uint64_t)size.QuadPart;
}
/* ---------------------------------------------------------------- unzip */
static int make_dirs(wchar_t *path) /* every folder above path (path is a file) */
{
for (wchar_t *p = path + 3; *p; p++) {
if (*p == L'\\') {
*p = 0;
if (!is_dir(path) && !CreateDirectoryW(path, NULL) && GetLastError() != ERROR_ALREADY_EXISTS) {
*p = L'\\';
return 0;
}
*p = L'\\';
}
}
return 1;
}
static int safe_name(const char *name)
{
if (!name[0] || name[0] == '/' || name[0] == '\\' || strchr(name, ':')) return 0;
for (const char *p = name; *p; p++)
if ((p == name || p[-1] == '/' || p[-1] == '\\') && p[0] == '.' && p[1] == '.' &&
(p[2] == '/' || p[2] == '\\' || !p[2]))
return 0;
return 1;
}
static int extract_entry(Src *z, uint64_t local, uint32_t csize, uint32_t usize, uint16_t method, uint32_t crc,
const wchar_t *target)
{
unsigned char lh[30];
if (!src_read(z, local, lh, 30) || le32(lh) != 0x04034b50) return 0;
uint64_t data = local + 30 + le16(lh + 26) + le16(lh + 28);
HANDLE out = CreateFileW(target, GENERIC_WRITE, 0, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
if (out == INVALID_HANDLE_VALUE) return 0;
static unsigned char in[1 << 16], buf[1 << 17];
uLong check = crc32(0L, Z_NULL, 0);
int ok = 1;
uint64_t left = csize, pos = data, total = 0;
z_stream st;
memset(&st, 0, sizeof st);
if (method == 8 && inflateInit2(&st, -MAX_WBITS) != Z_OK) ok = 0;
while (ok && (left || (method == 8 && st.avail_in))) {
DWORD n = (DWORD)(left < sizeof in ? left : sizeof in);
if (n && !src_read(z, pos, in, n)) { ok = 0; break; }
pos += n;
left -= n;
if (method == 0) {
DWORD put;
check = crc32(check, in, n);
ok = WriteFile(out, in, n, &put, NULL) && put == n;
total += n;
continue;
}
st.next_in = in;
st.avail_in = n;
do {
st.next_out = buf;
st.avail_out = sizeof buf;
int r = inflate(&st, Z_NO_FLUSH);
if (r != Z_OK && r != Z_STREAM_END && !(r == Z_BUF_ERROR && st.avail_in == 0)) { ok = 0; break; }
DWORD have = (DWORD)(sizeof buf - st.avail_out), put;
if (have) {
check = crc32(check, buf, have);
if (!WriteFile(out, buf, have, &put, NULL) || put != have) { ok = 0; break; }
total += have;
}
if (r == Z_STREAM_END) { left = 0; st.avail_in = 0; break; }
} while (st.avail_out == 0);
}
if (method == 8) inflateEnd(&st);
CloseHandle(out);
S.written += total;
return ok && total == usize && check == crc;
}
static int unzip(void)
{
Src z;
unsigned char eocd[22];
z.f = CreateFileW(S.exe, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, 0, NULL);
if (z.f == INVALID_HANDLE_VALUE) { fail_msg(L"The setup cannot read itself.", NULL); return 0; }
z.base = S.off;
z.size = S.size;
/* the end-of-central-directory record: the zip has no comment, so it is the last 22 bytes */
if (!src_read(&z, z.size - 22, eocd, 22) || le32(eocd) != 0x06054b50) {
CloseHandle(z.f);
fail_msg(L"The setup is damaged. Download it again.", NULL);
return 0;
}
uint32_t count = le16(eocd + 10), cd_size = le32(eocd + 12), cd_off = le32(eocd + 16);
unsigned char *cd = malloc(cd_size);
if (!cd || !src_read(&z, cd_off, cd, cd_size)) {
free(cd);
CloseHandle(z.f);
fail_msg(L"The setup is damaged. Download it again.", NULL);
return 0;
}
wchar_t target[PATHN], wname[PATHN];
char name[PATHN];
uint32_t at = 0;
int ok = 1;
for (uint32_t i = 0; ok && i < count; i++) {
if (at + 46 > cd_size || le32(cd + at) != 0x02014b50) { ok = 0; break; }
uint16_t method = le16(cd + at + 10), nlen = le16(cd + at + 28), xlen = le16(cd + at + 30),
clen = le16(cd + at + 32);
uint32_t crc = le32(cd + at + 16), csize = le32(cd + at + 20), usize = le32(cd + at + 24),
local = le32(cd + at + 42);
if (nlen >= PATHN || at + 46 + nlen > cd_size || (method != 0 && method != 8)) { ok = 0; break; }
memcpy(name, cd + at + 46, nlen);
name[nlen] = 0;
at += 46 + nlen + xlen + clen;
if (!safe_name(name)) { ok = 0; break; }
MultiByteToWideChar(CP_UTF8, 0, name, -1, wname, PATHN);
for (wchar_t *p = wname; *p; p++)
if (*p == L'/') *p = L'\\';
swprintf(target, PATHN, L"%ls\\%ls", S.dest, wname);
size_t tl = wcslen(target);
if (tl && target[tl - 1] == L'\\') {
target[tl - 1] = 0;
ok = make_dirs(wcscat(target, L"\\")) && (is_dir(target) || CreateDirectoryW(target, NULL));
continue;
}
if (!make_dirs(target)) { ok = 0; break; }
if (!extract_entry(&z, local, csize, usize, method, crc, target)) {
fail_msg(L"%ls could not be written. Close NX2navi if it is open, then run the setup again.", target);
free(cd);
CloseHandle(z.f);
return 0;
}
if (g_wnd && (i & 31) == 0) InvalidateRect(g_wnd, NULL, FALSE);
}
free(cd);
CloseHandle(z.f);
if (!ok) fail_msg(L"The setup is damaged. Download it again.", NULL);
return ok;
}
/* ---------------------------------------------------------------- the work */
static DWORD WINAPI work(void *arg)
{
wchar_t cur[PATHN], have[64];
(void)arg;
swprintf(cur, PATHN, L"%ls\\app\\current.txt", S.dest);
int newer_installed = read_line(cur, have, 64) && version_cmp(have, S.version) > 0;
wchar_t probe[PATHN];
swprintf(probe, PATHN, L"%ls\\x", S.dest);
if (!make_dirs(probe)) {
fail_msg(L"The folder %ls cannot be created.", S.dest);
} else if (!newer_installed) {
unzip();
}
if (!S.result) {
if (S.shortcuts) {
set_shortcut(S.dest, 0, 1);
set_shortcut(S.dest, 1, 1);
}
write_uninstall_entry(S.dest, newer_installed ? have : S.version, (DWORD)(S.written / 1024 + 1));
}
if (g_wnd) PostMessageW(g_wnd, WM_DONE, 0, 0);
return 0;
}
/* ---------------------------------------------------------------- the window */
static LRESULT CALLBACK wndproc(HWND w, UINT m, WPARAM wp, LPARAM lp)
{
switch (m) {
case WM_PAINT: {
PAINTSTRUCT ps;
HDC dc = BeginPaint(w, &ps);
RECT r;
GetClientRect(w, &r);
HBRUSH bg = CreateSolidBrush(RGB(0x0c, 0x2e, 0x1b)), edge = CreateSolidBrush(RGB(0xa9, 0x87, 0x4f));
FillRect(dc, &r, edge);
RECT in = {r.left + 2, r.top + 2, r.right - 2, r.bottom - 2};
FillRect(dc, &in, bg);
/* the Triforce: three gold triangles */
HBRUSH gold = CreateSolidBrush(RGB(0xe8, 0xaf, 0x3d));
HPEN none = CreatePen(PS_NULL, 0, 0);
HGDIOBJ ob = SelectObject(dc, gold), op = SelectObject(dc, none);
int cx = 52, top = 28, s = 44;
POINT t1[3] = {{cx, top}, {cx + s / 4, top + s * 433 / 1000}, {cx - s / 4, top + s * 433 / 1000}};
POINT t2[3] = {{cx - s / 4, top + s * 433 / 1000 + 1}, {cx, top + s * 866 / 1000}, {cx - s / 2, top + s * 866 / 1000}};
POINT t3[3] = {{cx + s / 4, top + s * 433 / 1000 + 1}, {cx + s / 2, top + s * 866 / 1000}, {cx, top + s * 866 / 1000}};
Polygon(dc, t1, 3);
Polygon(dc, t2, 3);
Polygon(dc, t3, 3);
SelectObject(dc, ob);
SelectObject(dc, op);
HFONT big = CreateFontW(-26, 0, 0, 0, FW_SEMIBOLD, 0, 0, 0, DEFAULT_CHARSET, 0, 0, CLEARTYPE_QUALITY, 0, L"Segoe UI");
HFONT small = CreateFontW(-16, 0, 0, 0, FW_NORMAL, 0, 0, 0, DEFAULT_CHARSET, 0, 0, CLEARTYPE_QUALITY, 0, L"Segoe UI");
SetBkMode(dc, TRANSPARENT);
SetTextColor(dc, RGB(0xf3, 0xec, 0xdc));
HGDIOBJ of = SelectObject(dc, big);
TextOutW(dc, 96, 30, APP_NAME, (int)wcslen(APP_NAME));
SelectObject(dc, small);
SetTextColor(dc, RGB(0xe9, 0xdc, 0xc0));
wchar_t line[96];
swprintf(line, 96, L"Installing version %ls\x2026 %llu MB", S.version, (unsigned long long)(S.written >> 20));
TextOutW(dc, 96, 66, line, (int)wcslen(line));
SelectObject(dc, of);
DeleteObject(big);
DeleteObject(small);
DeleteObject(gold);
DeleteObject(none);
DeleteObject(bg);
DeleteObject(edge);
EndPaint(w, &ps);
return 0;
}
case WM_TIMER:
InvalidateRect(w, NULL, FALSE);
return 0;
case WM_DONE:
DestroyWindow(w);
return 0;
case WM_DESTROY:
PostQuitMessage(0);
return 0;
}
return DefWindowProcW(w, m, wp, lp);
}
static void run_with_window(HINSTANCE inst)
{
WNDCLASSW wc = {0};
wc.lpfnWndProc = wndproc;
wc.hInstance = inst;
wc.hCursor = LoadCursor(NULL, IDC_WAIT);
wc.hIcon = LoadIconW(inst, MAKEINTRESOURCEW(1));
wc.lpszClassName = L"NX2naviSetup";
RegisterClassW(&wc);
int w = 440, h = 120;
int x = (GetSystemMetrics(SM_CXSCREEN) - w) / 2, y = (GetSystemMetrics(SM_CYSCREEN) - h) / 3;
g_wnd = CreateWindowExW(WS_EX_APPWINDOW, wc.lpszClassName, L"NX2navi Setup", WS_POPUP | WS_VISIBLE, x, y, w, h,
NULL, NULL, inst, NULL);
SetTimer(g_wnd, 1, 200, NULL);
HANDLE t = CreateThread(NULL, 0, work, NULL, 0, NULL);
MSG msg;
while (GetMessageW(&msg, NULL, 0, 0) > 0) {
TranslateMessage(&msg);
DispatchMessageW(&msg);
}
WaitForSingleObject(t, INFINITE);
CloseHandle(t);
}
int WINAPI wWinMain(HINSTANCE inst, HINSTANCE prev, PWSTR cmdline, int show)
{
(void)prev; (void)cmdline; (void)show;
GetModuleFileNameW(NULL, S.exe, PATHN);
exe_dir(S.src_dir, PATHN);
S.shortcuts = S.launch = 1;
S.dest[0] = 0;
int argc = 0;
wchar_t **argv = CommandLineToArgvW(GetCommandLineW(), &argc);
for (int i = 1; argv && i < argc; i++) {
if (!wcscmp(argv[i], L"--dest") && i + 1 < argc) {
GetFullPathNameW(argv[++i], PATHN, S.dest, NULL);
} else if (!wcscmp(argv[i], L"--no-shortcuts")) S.shortcuts = 0;
else if (!wcscmp(argv[i], L"--no-launch")) S.launch = 0;
else if (!wcscmp(argv[i], L"--quiet")) S.quiet = 1;
}
if (argv) LocalFree(argv);
if (!S.dest[0]) {
wchar_t base[PATHN];
if (!known_folder(&FOLDERID_UserProgramFiles, base, PATHN)) {
if (!GetEnvironmentVariableW(L"LOCALAPPDATA", base, PATHN)) return 1;
wcscat(base, L"\\Programs");
}
swprintf(S.dest, PATHN, L"%ls\\%ls", base, APP_NAME);
}
size_t dl = wcslen(S.dest);
while (dl > 3 && (S.dest[dl - 1] == L'\\' || S.dest[dl - 1] == L'/')) S.dest[--dl] = 0;
if (!read_trailer()) {
MessageBoxW(NULL, L"This setup is incomplete. Download NX2navi-Setup.exe again.", APP_NAME, MB_ICONERROR);
return 1;
}
if (S.quiet) work(NULL);
else run_with_window(inst);
if (S.result) {
if (!S.quiet) MessageBoxW(NULL, S.error, APP_NAME, MB_OK | MB_ICONERROR);
return 2;
}
if (S.launch) {
static wchar_t cmd[PATHN * 3];
wchar_t exe[PATHN];
swprintf(exe, PATHN, L"%ls\\NX2navi.exe", S.dest);
swprintf(cmd, PATHN * 3, L"\"%ls\" --from \"%ls\"", exe, S.src_dir);
STARTUPINFOW si = {.cb = sizeof si};
PROCESS_INFORMATION pi;
if (CreateProcessW(exe, cmd, NULL, NULL, FALSE, 0, NULL, S.dest, &si, &pi)) {
CloseHandle(pi.hThread);
CloseHandle(pi.hProcess);
}
}
return 0;
}

128
launcher/net.py Normal file
View File

@@ -0,0 +1,128 @@
"""HTTPS for the launcher: JSON requests and file downloads that resume, report progress, can be cancelled and are
checked against a sha256. Standard library only (urllib); Windows' own certificate store verifies the servers."""
import hashlib
import json
import os
import ssl
import time
import urllib.error
import urllib.request
from pathlib import Path
from . import NAME
CHUNK = 1 << 20
class Cancelled(Exception):
pass
class DownloadError(Exception):
pass
def user_agent(version='dev'):
return f'{NAME}-Launcher/{version}'
_CTX = None
def _context():
global _CTX
if _CTX is None:
_CTX = ssl.create_default_context()
return _CTX
def request(url, headers=None, timeout=30, version='dev'):
h = {'User-Agent': user_agent(version)}
h.update(headers or {})
req = urllib.request.Request(url, headers=h)
ctx = _context() if url.startswith('https:') else None
return urllib.request.urlopen(req, timeout=timeout, context=ctx)
def get_bytes(url, headers=None, timeout=30, version='dev', limit=64 << 20):
with request(url, headers, timeout, version) as r:
data = r.read(limit + 1)
if len(data) > limit:
raise DownloadError(f'{url}: more than {limit >> 20} MB')
return data
def get_json(url, headers=None, timeout=30, version='dev'):
return json.loads(get_bytes(url, headers, timeout, version).decode('utf-8'))
def sha256_file(path):
h = hashlib.sha256()
with open(path, 'rb') as f:
while b := f.read(CHUNK):
h.update(b)
return h.hexdigest()
def download(url, dest, sha256=None, size=None, progress=None, cancel=None, headers=None, retries=4, version='dev'):
"""Fetch url to dest through dest.part, resuming a part file a cut connection left. progress(done, total) after
every chunk; cancel() returning True raises Cancelled. A wrong size or sha256 deletes the file and raises
DownloadError. A dest already holding the expected sha256 is kept without a request."""
dest = Path(dest)
dest.parent.mkdir(parents=True, exist_ok=True)
if sha256 and dest.is_file() and sha256_file(dest) == sha256.lower():
if progress:
progress(dest.stat().st_size, dest.stat().st_size)
return dest
part = dest.with_name(dest.name + '.part')
last_error = None
for attempt in range(retries):
have = part.stat().st_size if part.is_file() else 0
h = dict(headers or {})
if have:
h['Range'] = f'bytes={have}-'
try:
with request(url, h, 60, version) as r:
status = getattr(r, 'status', 200)
if have and status != 206: # the server ignores Range: start over
have = 0
total = int(r.headers.get('Content-Length') or 0) + have or size or 0
with open(part, 'ab' if have else 'wb') as f:
done = have
while True:
if cancel and cancel():
raise Cancelled()
b = r.read(CHUNK)
if not b:
break
f.write(b)
done += len(b)
if progress:
progress(done, total or done)
break
except Cancelled:
raise
except urllib.error.HTTPError as e:
if e.code == 416 and have: # the part file is complete or longer than the file: start over
part.unlink(missing_ok=True)
last_error = e
continue
if e.code in (401, 403, 404, 410):
raise DownloadError(f'{url}: HTTP {e.code}') from e
last_error = e
except (urllib.error.URLError, OSError, TimeoutError) as e:
last_error = e
time.sleep(min(2 ** attempt, 16))
else:
raise DownloadError(f'{url}: {last_error}')
got = part.stat().st_size
if size and got != size:
part.unlink(missing_ok=True)
raise DownloadError(f'{url}: {got} bytes, expected {size}')
if sha256:
digest = sha256_file(part)
if digest != sha256.lower():
part.unlink(missing_ok=True)
raise DownloadError(f'{url}: sha256 {digest}, expected {sha256.lower()}')
os.replace(part, dest)
return dest

115
launcher/paths.py Normal file
View File

@@ -0,0 +1,115 @@
"""Where the launcher keeps everything.
An installed launcher (NX2navi-Setup.exe puts it in %LOCALAPPDATA%\\Programs\\NX2navi) is one folder:
NX2navi.exe launcher/native/nx2navi.c: starts python\\pythonw.exe on app\\<current>\\launcher\\start.pyw
python/ the embeddable Python with tkinter, numpy, lz4 and zstandard (tools/make_launcher.py)
app/current.txt the version the stub starts; app/previous.txt the one before it, kept for a rollback
app/<version>/ one release: launcher/, runtime/, recomp/, tools/, bin/, VERSION, toolchain.lock.json
pydeps/ zbic, downloaded from PyPI on this PC (launcher/toolchain.py ensure_zbic)
toolchain/msys64/ucrt64 MSYS2's clang and libraries, downloaded from the MSYS2 repository on this PC
toolchain/pkgs/ the downloaded package files, kept while the lock names them
data/launcher.json settings; data/logs/ logs; data/cache/ the fonts read from the player's game
game/ the built game, unless the player picks another folder
Run from a checkout of the repository (python -m launcher), the app folder is the repository and everything else goes
to build/launcher-dev/, which git ignores.
"""
import os
import sys
from pathlib import Path
APP = Path(__file__).resolve().parents[1]
def _is_release_app(app):
return app.parent.name == 'app' and (app.parent / 'current.txt').is_file()
def ansi_safe(path):
"""True when the path survives the Windows ANSI code page: the MSYS2 tools and the game's own build read their
arguments through it, so a folder name outside it breaks the build. Always True elsewhere."""
if os.name != 'nt':
return True
try:
str(path).encode('mbcs', errors='strict')
return True
except (UnicodeEncodeError, LookupError):
return False
class Layout:
def __init__(self, root=None, app=None):
self.app = Path(app or APP)
self.dev = not _is_release_app(self.app)
env = os.environ.get('NX2NAVI_LAUNCHER_ROOT')
if root:
self.root = Path(root)
elif env:
self.root = Path(env)
elif not self.dev:
self.root = self.app.parents[1]
else:
self.root = self.app / 'build' / 'launcher-dev'
self.apps = self.root / 'app'
self.python = self.root / 'python'
self.stub = self.root / 'NX2navi.exe'
self.pydeps = self.root / 'pydeps'
self.data = self.root / 'data'
self.logs = self.data / 'logs'
self.cache = self.data / 'cache'
self.downloads = self.data / 'downloads'
self.settings = self.data / 'launcher.json'
tool_root = self.root
if not ansi_safe(tool_root):
tool_root = Path(os.environ.get('SystemDrive', 'C:') + '\\') / 'NX2navi'
self.toolchain = tool_root / 'toolchain'
self.msys_root = self.toolchain / 'msys64'
self.msys = self.msys_root / 'ucrt64'
self.pkgs = self.toolchain / 'pkgs'
self.tools = self.app / 'tools'
self.make_game = self.tools / 'make_game.py'
self.lock_file = self.app / 'toolchain.lock.json'
self.feed_file = self.app / 'feed.json'
@property
def version(self):
v = self.app / 'VERSION'
return v.read_text(encoding='utf-8').strip() if v.is_file() else '0.0.0'
def default_game_dir(self):
"""<root>/game, or a Games folder on the fixed drive with the most free space when the root's drive has
less than 40 GB free or the root's path would break the build."""
here = self.root / 'game'
try:
import shutil
if shutil.disk_usage(self.root.anchor or self.root).free >= 40e9 and ansi_safe(here):
return here
except OSError:
return here
best = None
if os.name == 'nt':
import shutil
import string
for letter in string.ascii_uppercase[2:]:
drive = Path(f'{letter}:\\')
try:
free = shutil.disk_usage(drive).free
except OSError:
continue
if best is None or free > best[0]:
best = (free, drive / 'Games' / 'NX2navi')
return best[1] if best else here
def python_for_builds(self):
"""The console python.exe beside the launcher's pythonw.exe (no window: the build starts it with
CREATE_NO_WINDOW), or this interpreter in a checkout."""
exe = Path(sys.executable)
if exe.name.lower() == 'pythonw.exe' and (exe.parent / 'python.exe').is_file():
return str(exe.parent / 'python.exe')
return str(exe)
def ensure(self):
for d in (self.data, self.logs, self.cache, self.downloads):
d.mkdir(parents=True, exist_ok=True)
return self

43
launcher/settings.py Normal file
View File

@@ -0,0 +1,43 @@
"""data/launcher.json: the player's choices and what the launcher remembers between runs."""
import json
import os
from pathlib import Path
DEFAULTS = {
'game_dir': '', # the built game; empty until the first install picks one
'dump': '', # the .nup (or the folder with exefs/ and romfs/) the game was built from
'auto_check': True, # look for a new version when the launcher opens
'channel': 'stable', # 'stable' or 'preview' (GitHub pre-releases)
'hide_while_playing': True, # hide the window while the game runs, show it again when it exits
'keep_cache': True, # keep build-cache/ (the lift and objects) so an update recompiles only changes
'desktop_shortcut': True,
'feed_repo': '', # owner/name overriding feed.json, for testing a fork's releases
'github_token': '', # only for a private release repository; stored as plain text
'last_check': 0.0,
'last_seen': '', # the newest version the player was told about
}
class Settings(dict):
def __init__(self, path):
super().__init__(DEFAULTS)
self.path = Path(path)
try:
data = json.loads(self.path.read_text(encoding='utf-8'))
if isinstance(data, dict):
self.update({k: v for k, v in data.items() if k in DEFAULTS and type(v) is type(DEFAULTS[k])})
except (OSError, ValueError):
pass
def save(self):
self.path.parent.mkdir(parents=True, exist_ok=True)
tmp = self.path.with_name(self.path.name + '.tmp')
tmp.write_text(json.dumps(dict(self), indent=1), encoding='utf-8')
os.replace(tmp, self.path)
def reset(self):
"""Default Settings: the choices go back to their defaults; the game folder, dump and history stay."""
keep = {k: self[k] for k in ('game_dir', 'dump', 'last_check', 'last_seen', 'feed_repo', 'github_token')}
self.clear()
self.update(DEFAULTS)
self.update(keep)

45
launcher/start.pyw Normal file
View File

@@ -0,0 +1,45 @@
"""What NX2navi.exe runs: python\\pythonw.exe app\\<version>\\launcher\\start.pyw [arguments].
The embeddable Python puts only its ._pth entries on sys.path, so the release folder (the parent of launcher/) goes
first here. A failure before the window is up is written to data/logs/crash.log, shown in a message box and ends the
process with exit code 86, which tells the stub to go back to the previous version (launcher/native/nx2navi.c).
"""
import os
import sys
import traceback
APP = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, APP)
STARTUP_FAILED = 86
def crash(text):
root = os.environ.get('NX2NAVI_LAUNCHER_ROOT') or os.path.dirname(os.path.dirname(APP))
try:
logs = os.path.join(root, 'data', 'logs')
os.makedirs(logs, exist_ok=True)
with open(os.path.join(logs, 'crash.log'), 'a', encoding='utf-8') as f:
f.write(text + '\n')
except OSError:
pass
try:
import ctypes
ctypes.windll.user32.MessageBoxW(None, 'NX2navi could not start.\n\n' + text[-1500:], 'NX2navi', 0x10)
except Exception:
print(text, file=sys.stderr)
try:
from launcher.ui.app import main
except Exception:
crash(traceback.format_exc())
sys.exit(STARTUP_FAILED)
try:
code = main()
except SystemExit:
raise
except Exception:
crash(traceback.format_exc())
sys.exit(1)
sys.exit(code or 0)

288
launcher/toolchain.py Normal file
View File

@@ -0,0 +1,288 @@
"""The build tools, installed on the player's PC: MSYS2's UCRT64 clang and the libraries tools/make_game.py links
(the packages README-BuildKit.md lists), and the zbic Python package.
No MSYS2 shell and no pacman: the packages and everything they depend on are read from the repository's database,
downloaded, checked against the sha256 the database (or the release's lock) names, and unpacked into
toolchain/msys64/ucrt64 as pacman would lay them out. A release pins the exact package files in
toolchain.lock.json (make_lock, written by tools/make_launcher.py), so every player builds with the compiler the
release was tested with; the repository keeps old package files. A checkout without a lock resolves the newest
packages instead.
zbic (GPL-2.0) unpacks the game's compressed program files. The release does not carry it: the launcher downloads the
wheel for its own Python from PyPI and unpacks it into pydeps/, as `pip install zbic` would.
"""
import hashlib
import io
import json
import os
import re
import shutil
import sys
import tarfile
import time
import zipfile
from pathlib import Path
from . import net
REPO = 'https://repo.msys2.org/mingw/ucrt64/'
MIRRORS = (REPO, 'https://mirror.msys2.org/mingw/ucrt64/')
PREFIX = 'mingw-w64-ucrt-x86_64-'
ROOTS = ('clang', 'gcc', 'sdl3', 'opus', 'libvpx', 'dav1d', 'zstd', 'vulkan-headers', 'vulkan-loader',
'vulkan-memory-allocator')
REQUIRED = ('bin/clang.exe', 'bin/clang++.exe', 'bin/SDL3.dll', 'lib/libSDL3.dll.a', 'include/vma')
SKIP_SHARE = ('ucrt64/share/doc/', 'ucrt64/share/man/', 'ucrt64/share/info/', 'ucrt64/share/locale/',
'ucrt64/share/gtk-doc/', 'ucrt64/share/licenses/')
ZBIC = ('zbic', '1.0.1.post1')
LOCK_FORMAT = 1
class ToolchainError(Exception):
pass
# ---------------------------------------------------------------- the package database
def _zstd_decompress_stream(data):
try:
from compression import zstd # Python 3.14
return zstd.decompress(data)
except ImportError:
import zstandard
return zstandard.ZstdDecompressor().decompressobj().decompress(data)
def _open_tar(path_or_bytes):
"""A tarfile over a .tar.zst (or plain/gzip tar) file or bytes."""
data = path_or_bytes if isinstance(path_or_bytes, (bytes, bytearray)) else Path(path_or_bytes).read_bytes()
if data[:4] == b'\x28\xb5\x2f\xfd':
data = _zstd_decompress_stream(data)
return tarfile.open(fileobj=io.BytesIO(data), mode='r:*')
def parse_db(data):
"""{name: desc} for a pacman sync database (ucrt64.db): each desc is {FIELD: [lines]}."""
out = {}
with _open_tar(data) as t:
for m in t.getmembers():
if not m.name.endswith('/desc'):
continue
desc, field = {}, None
for line in t.extractfile(m).read().decode('utf-8').splitlines():
if line.startswith('%') and line.endswith('%') and len(line) > 2:
field = line[1:-1]
desc[field] = []
elif line and field:
desc[field].append(line)
if desc.get('NAME'):
out[desc['NAME'][0]] = desc
return out
def _dep_name(dep):
return re.split(r'[<>=:]', dep, maxsplit=1)[0].strip()
def resolve(db, roots=ROOTS):
"""Every package the roots need, roots first; names a package provides resolve to the package."""
provides = {}
for name, d in db.items():
for p in d.get('PROVIDES', []):
provides.setdefault(_dep_name(p), name)
order, seen = [], set()
todo = [r if r.startswith(PREFIX) else PREFIX + r for r in roots]
while todo:
n = _dep_name(todo.pop(0))
n = n if n in db else provides.get(n, n)
if n in seen:
continue
if n not in db:
raise ToolchainError(f'the package database has no {n}')
seen.add(n)
order.append(n)
todo += db[n].get('DEPENDS', [])
pkgs = []
for n in order:
d = db[n]
pkgs.append({'name': n, 'version': d['VERSION'][0], 'filename': d['FILENAME'][0],
'sha256': d['SHA256SUM'][0], 'csize': int(d['CSIZE'][0]), 'isize': int(d.get('ISIZE', ['0'])[0])})
return pkgs
def make_lock(repo=REPO, roots=ROOTS, version='dev'):
"""The lock a release ships: the resolved package files of the repository's current database."""
db = parse_db(net.get_bytes(repo + 'ucrt64.db', version=version))
return {'format': LOCK_FORMAT, 'repo': repo, 'roots': list(roots), 'packages': resolve(db, roots),
'zbic': {'name': ZBIC[0], 'version': ZBIC[1]}, 'made': time.strftime('%Y-%m-%d')}
def lock_digest(lock):
files = sorted(p['filename'] + ':' + p['sha256'] for p in lock['packages'])
return hashlib.sha256('\n'.join(files).encode()).hexdigest()
def load_lock(layout, version='dev'):
"""The release's toolchain.lock.json, or a lock resolved from the live database in a checkout."""
if layout.lock_file.is_file():
lock = json.loads(layout.lock_file.read_text(encoding='utf-8'))
if lock.get('format') != LOCK_FORMAT:
raise ToolchainError(f'{layout.lock_file}: unknown format {lock.get("format")}')
return lock
return make_lock(version=version)
# ---------------------------------------------------------------- install
def installed(layout):
p = layout.msys_root / 'installed.json'
try:
return json.loads(p.read_text(encoding='utf-8'))
except (OSError, ValueError):
return None
def missing_files(msys):
return [r for r in REQUIRED if not (Path(msys) / r).exists()]
def is_current(layout, lock):
inst = installed(layout)
return bool(inst) and inst.get('digest') == lock_digest(lock) and not missing_files(layout.msys)
def _extract_package(path, dest):
"""Unpack one .pkg.tar.zst into dest (the msys64 folder): only its ucrt64/ files, without the documentation.
The archive is read as a stream, member by member, so a 350 MB package never sits in memory whole."""
with open(path, 'rb') as raw:
zst = raw.read(4) == b'\x28\xb5\x2f\xfd'
raw.seek(0)
if zst:
try:
from compression import zstd # Python 3.14
fileobj = zstd.ZstdFile(raw)
except ImportError:
import zstandard
fileobj = zstandard.ZstdDecompressor().stream_reader(raw)
else:
fileobj = raw
with tarfile.open(fileobj=fileobj, mode='r|*') as t:
for m in t:
name = m.name.lstrip('./')
if name.startswith('ucrt64/') and not name.startswith(SKIP_SHARE):
t.extract(m, dest, filter='data')
def install(layout, lock, progress=None, cancel=None, log=print, version='dev'):
"""Download and unpack the lock's packages into toolchain/msys64 (through msys64.new, swapped in at the end).
progress(phase, done, total) with phase 'download' (bytes) or 'unpack' (packages)."""
pkgs = lock['packages']
layout.pkgs.mkdir(parents=True, exist_ok=True)
need = [p for p in pkgs if not ((layout.pkgs / p['filename']).is_file()
and (layout.pkgs / p['filename']).stat().st_size == p['csize'])]
total = sum(p['csize'] for p in need)
log(f'toolchain: {len(pkgs)} packages, {len(need)} to download ({total / 1e6:.0f} MB)')
done_before = 0
for p in need:
dest = layout.pkgs / p['filename']
last_error = None
for mirror in [lock.get('repo', REPO)] + [m for m in MIRRORS if m != lock.get('repo', REPO)]:
try:
net.download(mirror + p['filename'], dest, sha256=p['sha256'], size=p['csize'], cancel=cancel,
progress=(lambda d, t, base=done_before: progress and progress('download', base + d, total)),
version=version)
last_error = None
break
except net.DownloadError as e:
last_error = e
log(f'toolchain: {e}')
if last_error:
raise ToolchainError(f'{p["filename"]} could not be downloaded: {last_error}')
done_before += p['csize']
for p in pkgs: # files kept from an earlier download are checked too
if net.sha256_file(layout.pkgs / p['filename']) != p['sha256']:
(layout.pkgs / p['filename']).unlink()
raise ToolchainError(f'{p["filename"]} was damaged on disk; run the setup again to download it anew')
staging = layout.toolchain / 'msys64.new'
shutil.rmtree(staging, ignore_errors=True)
staging.mkdir(parents=True)
for i, p in enumerate(pkgs):
if cancel and cancel():
raise net.Cancelled()
if progress:
progress('unpack', i, len(pkgs))
_extract_package(layout.pkgs / p['filename'], staging)
if progress:
progress('unpack', len(pkgs), len(pkgs))
missing = missing_files(staging / 'ucrt64')
if missing:
raise ToolchainError(f'the packages lack {", ".join(missing)}')
(staging / 'installed.json').write_text(json.dumps({
'digest': lock_digest(lock), 'packages': [f'{p["name"]} {p["version"]}' for p in pkgs],
'installed': time.strftime('%Y-%m-%d %H:%M')}, indent=1), encoding='utf-8')
old = layout.toolchain / f'msys64.old-{int(time.time())}'
if layout.msys_root.exists():
os.replace(layout.msys_root, old)
os.replace(staging, layout.msys_root)
shutil.rmtree(old, ignore_errors=True)
keep = {p['filename'] for p in pkgs}
for f in layout.pkgs.iterdir():
if f.name not in keep:
f.unlink(missing_ok=True)
log(f'toolchain: installed in {layout.msys}')
# ---------------------------------------------------------------- zbic
def wheel_tag():
"""The wheel platform tags this interpreter loads, best first."""
v = f'cp{sys.version_info.major}{sys.version_info.minor}'
if os.name == 'nt':
return [f'{v}-{v}-win_amd64']
return [f'{v}-{v}-manylinux', f'{v}-{v}-musllinux']
def zbic_ok(layout):
"""zbic loads from pydeps/ in the interpreter the build runs with (a fresh process: this one may have cached
that pydeps/ did not exist)."""
import subprocess
if not layout.pydeps.is_dir():
return False
code = 'import sys; sys.path.insert(0, sys.argv[1]); import zbic; assert callable(zbic.decompress)'
flags = {'creationflags': 0x08000000} if os.name == 'nt' else {} # CREATE_NO_WINDOW
r = subprocess.run([layout.python_for_builds(), '-I', '-c', code, str(layout.pydeps)], capture_output=True,
**flags)
return r.returncode == 0
def ensure_zbic(layout, progress=None, cancel=None, log=print, version='dev'):
if zbic_ok(layout):
return
name, ver = ZBIC
info = net.get_json(f'https://pypi.org/pypi/{name}/{ver}/json', version=version)
files = [u for u in info.get('urls', []) if u['filename'].endswith('.whl')]
pick = None
for tag in wheel_tag():
for u in files:
if tag in u['filename'] and ('x86_64' in u['filename'] or 'amd64' in u['filename']):
pick = u
break
if pick:
break
if not pick:
raise ToolchainError(f'PyPI has no {name} {ver} wheel for Python {sys.version_info.major}.'
f'{sys.version_info.minor} on this system')
dest = layout.downloads / pick['filename']
net.download(pick['url'], dest, sha256=pick['digests']['sha256'], size=pick.get('size'), cancel=cancel,
progress=(lambda d, t: progress and progress('download', d, t)), version=version)
shutil.rmtree(layout.pydeps, ignore_errors=True)
layout.pydeps.mkdir(parents=True)
with zipfile.ZipFile(dest) as z:
for m in z.infolist():
target = (layout.pydeps / m.filename).resolve()
if not str(target).startswith(str(layout.pydeps.resolve())):
raise ToolchainError(f'{pick["filename"]}: unsafe path {m.filename}')
z.extractall(layout.pydeps)
dest.unlink(missing_ok=True)
if not zbic_ok(layout):
raise ToolchainError(f'{name} {ver} was unpacked into {layout.pydeps} but does not load')
log(f'zbic: {pick["filename"]}')

1
launcher/ui/__init__.py Normal file
View File

@@ -0,0 +1 @@
"""The launcher's window: art.py draws the sprites, app.py lays out the screens and handles input."""

1297
launcher/ui/app.py Normal file

File diff suppressed because it is too large Load Diff

538
launcher/ui/art.py Normal file
View File

@@ -0,0 +1,538 @@
"""The window's pictures, drawn with numpy into RGBA arrays and handed to Tk as PNG data.
Everything is drawn here from shapes and colours: the misty green backdrop, the wooden title band, the bronze label
cells with their mottled texture, the description box, the white glowing cursor, the filigree divider, the Triforce
and the small icons. Tk's canvas draws lines and polygons without antialiasing on Windows, so every curved or soft
shape is a picture; flat rectangles and text stay canvas items.
Colours follow the game's options menu (docs/launcher.md, "Look"). Sizes are in logical pixels times the display
scale s.
"""
import base64
import re
import struct
import zlib
import numpy as np
# ---------------------------------------------------------------- colours
PALETTE = {
'window': '#06180d',
'panel': '#3a2b22',
'label_top': '#7a5c32', 'label_mid': '#775930', 'label_bottom': '#72552d', 'label_border': '#6a4f29',
'ring': '#2a2019',
'value': '#2f2d22', 'value_border': '#45402f',
'orange': '#d69127', 'orange_light': '#e3a234', 'orange_dark': '#8f5f17',
'desc': '#403024', 'desc_border': '#4f3f30',
'text': '#e9dcc0', 'text_bright': '#f1e6cc', 'value_text': '#e3d5b6', 'desc_text': '#e9dfc8',
'muted': '#d4c6a6', 'dim': '#7d786a', 'off': '#5e5b51', 'on_text': '#fdf4de', 'salmon': '#e88f6c',
'ok': '#a7d68c', 'gold': '#e6b65a', 'band_text': '#f3ecdc', 'status': '#b9a988', 'log_text': '#cfc4ab',
'line': '#54442f', 'dot': '#5d4b36', 'filigree': '#7a6443', 'glow': '#e9ebee',
'tri_top': '#d49f36', 'tri_bottom': '#fcbf44',
}
def rgb(hexcolor):
h = hexcolor.lstrip('#')
return np.array([int(h[i:i + 2], 16) / 255.0 for i in (0, 2, 4)], dtype=np.float32)
# ---------------------------------------------------------------- PNG and Tk
def png_bytes(img):
"""PNG of an HxWx4 float array in 0..1 (straight alpha) or uint8 array."""
a = img if img.dtype == np.uint8 else (np.clip(img, 0, 1) * 255 + 0.5).astype(np.uint8)
h, w = a.shape[:2]
raw = b''.join(b'\0' + a[y].tobytes() for y in range(h))
def chunk(tag, data):
return struct.pack('>I', len(data)) + tag + data + struct.pack('>I', zlib.crc32(tag + data) & 0xFFFFFFFF)
return (b'\x89PNG\r\n\x1a\n' + chunk(b'IHDR', struct.pack('>IIBBBBB', w, h, 8, 6, 0, 0, 0))
+ chunk(b'IDAT', zlib.compress(raw, 3)) + chunk(b'IEND', b''))
def photo(master, img):
import tkinter as tk
return tk.PhotoImage(master=master, data=base64.b64encode(png_bytes(img)).decode('ascii'), format='png')
def ico_bytes(images):
"""A Windows .ico holding PNG-compressed images (one HxWx4 array per size, square)."""
pngs = [png_bytes(im) for im in images]
head = struct.pack('<HHH', 0, 1, len(pngs))
offset = 6 + 16 * len(pngs)
dirs, body = b'', b''
for im, data in zip(images, pngs):
n = im.shape[0]
dirs += struct.pack('<BBBBHHII', n % 256, n % 256, 0, 0, 1, 32, len(data), offset + len(body))
body += data
return head + dirs + body
# ---------------------------------------------------------------- drawing primitives
def new(h, w, color=None, alpha=0.0):
img = np.zeros((h, w, 4), dtype=np.float32)
if color is not None:
img[..., :3] = rgb(color)
img[..., 3] = alpha
return img
def over(dst, src, x=0, y=0):
"""Composite src over dst at (x, y), straight alpha, clipped to dst."""
h, w = src.shape[:2]
H, W = dst.shape[:2]
x0, y0, x1, y1 = max(x, 0), max(y, 0), min(x + w, W), min(y + h, H)
if x0 >= x1 or y0 >= y1:
return dst
s = src[y0 - y:y1 - y, x0 - x:x1 - x]
d = dst[y0:y1, x0:x1]
sa, da = s[..., 3:4], d[..., 3:4]
oa = sa + da * (1 - sa)
oc = np.where(oa > 0, (s[..., :3] * sa + d[..., :3] * da * (1 - sa)) / np.maximum(oa, 1e-6), 0)
d[..., :3], d[..., 3:4] = oc, oa
return dst
def paint(img, mask, color, alpha=1.0):
"""Paint a colour through a coverage mask (HxW, 0..1) onto img."""
layer = new(*mask.shape, color)
layer[..., 3] = np.clip(mask, 0, 1) * alpha
return over(img, layer)
def vgradient(h, w, stops):
"""HxWx3 colours down the height through (position 0..1, colour) stops."""
pos = np.array([p for p, _ in stops], dtype=np.float32)
cols = np.stack([rgb(c) for _, c in stops])
t = (np.arange(h, dtype=np.float32) + 0.5) / max(h, 1)
col = np.stack([np.interp(t, pos, cols[:, k]) for k in range(3)], axis=1)
return np.broadcast_to(col[:, None, :], (h, w, 3)).copy()
def hgradient_alpha(h, w, stops):
"""HxW alphas across the width through (position, alpha) stops."""
pos = np.array([p for p, _ in stops], dtype=np.float32)
al = np.array([a for _, a in stops], dtype=np.float32)
t = (np.arange(w, dtype=np.float32) + 0.5) / max(w, 1)
return np.broadcast_to(np.interp(t, pos, al)[None, :], (h, w)).copy()
def hgradient_rgba(h, w, stops):
"""An RGBA layer whose colour and alpha run across the width through (position, colour, alpha) stops."""
pos = np.array([p for p, _, _ in stops], dtype=np.float32)
cols = np.stack([rgb(c) for _, c, _ in stops])
al = np.array([a for _, _, a in stops], dtype=np.float32)
t = (np.arange(w, dtype=np.float32) + 0.5) / max(w, 1)
layer = new(h, w)
for k in range(3):
layer[..., k] = np.interp(t, pos, cols[:, k])[None, :]
layer[..., 3] = np.interp(t, pos, al)[None, :]
return layer
def value_noise(h, w, period_x, period_y=None, seed=0):
"""Smooth noise in 0..1 with features about period pixels apart."""
period_y = period_y or period_x
rng = np.random.default_rng(seed)
gh, gw = int(h / period_y) + 3, int(w / period_x) + 3
g = rng.random((gh, gw), dtype=np.float32)
ys = np.arange(h, dtype=np.float32) / period_y
xs = np.arange(w, dtype=np.float32) / period_x
y0, x0 = ys.astype(int), xs.astype(int)
fy, fx = ys - y0, xs - x0
fy, fx = fy * fy * (3 - 2 * fy), fx * fx * (3 - 2 * fx)
a, b = g[y0][:, x0], g[y0][:, x0 + 1]
c, d = g[y0 + 1][:, x0], g[y0 + 1][:, x0 + 1]
top = a + (b - a) * fx[None, :]
bot = c + (d - c) * fx[None, :]
return top + (bot - top) * fy[:, None]
def fractal(h, w, period_x, period_y=None, octaves=3, seed=0, contrast=0.4):
"""Fractal noise around mid grey, about contrast wide (the range of an SVG feTurbulence turned grey)."""
return 0.5 + (_fractal01(h, w, period_x, period_y, octaves, seed) - 0.5) * contrast
def _fractal01(h, w, period_x, period_y=None, octaves=3, seed=0):
period_y = period_y or period_x
out = np.zeros((h, w), dtype=np.float32)
amp, total = 1.0, 0.0
for o in range(octaves):
out += amp * value_noise(h, w, max(period_x / 2 ** o, 1.5), max(period_y / 2 ** o, 1.5), seed + o * 101)
total += amp
amp *= 0.5
out /= total
lo, hi = np.percentile(out, 1), np.percentile(out, 99)
return np.clip((out - lo) / max(hi - lo, 1e-6), 0, 1)
def overlay(rgb_arr, n, opacity):
"""CSS mix-blend-mode overlay of a grey noise n (HxW) over colours, at opacity."""
n = n[..., None]
blended = np.where(rgb_arr < 0.5, 2 * rgb_arr * n, 1 - 2 * (1 - rgb_arr) * (1 - n))
return rgb_arr + (blended - rgb_arr) * opacity
def box_blur(a, r):
"""Three box blurs of radius r over a 2-D array (close to a gaussian)."""
r = int(round(r))
if r < 1:
return a
out = a.astype(np.float32)
for _ in range(3):
for axis in (0, 1):
pad = [(0, 0), (0, 0)]
pad[axis] = (r + 1, r)
p = np.pad(out, pad, mode='edge')
c = np.cumsum(p, axis=axis, dtype=np.float64)
if axis == 0:
out = ((c[2 * r + 1:] - c[:-2 * r - 1]) / (2 * r + 1)).astype(np.float32)
else:
out = ((c[:, 2 * r + 1:] - c[:, :-2 * r - 1]) / (2 * r + 1)).astype(np.float32)
return out
def rect_mask(h, w, x0, y0, x1, y1):
"""Coverage of the rectangle [x0, x1) x [y0, y1) with fractional edges."""
xs = np.arange(w, dtype=np.float32)
ys = np.arange(h, dtype=np.float32)
cx = np.clip(np.minimum(xs + 1, x1) - np.maximum(xs, x0), 0, 1)
cy = np.clip(np.minimum(ys + 1, y1) - np.maximum(ys, y0), 0, 1)
return cy[:, None] * cx[None, :]
def ring_mask(h, w, x0, y0, x1, y1, width):
return np.clip(rect_mask(h, w, x0, y0, x1, y1) - rect_mask(h, w, x0 + width, y0 + width, x1 - width, y1 - width),
0, 1)
def polygon_mask(h, w, pts, ss=4):
"""Antialiased coverage of a convex polygon (pixel coordinates), supersampled ss x ss."""
ys = (np.arange(h * ss, dtype=np.float32) + 0.5) / ss
xs = (np.arange(w * ss, dtype=np.float32) + 0.5) / ss
X, Y = np.meshgrid(xs, ys)
inside = np.ones_like(X, dtype=bool)
n = len(pts)
area = sum(pts[i][0] * pts[(i + 1) % n][1] - pts[(i + 1) % n][0] * pts[i][1] for i in range(n))
sign = 1 if area > 0 else -1
for i in range(n):
(ax, ay), (bx, by) = pts[i], pts[(i + 1) % n]
inside &= sign * ((bx - ax) * (Y - ay) - (by - ay) * (X - ax)) >= 0
return inside.reshape(h, ss, w, ss).mean(axis=(1, 3)).astype(np.float32)
def stroke_mask(h, w, polylines, width):
"""Antialiased coverage of polylines (lists of (x, y)) stroked with round joins and caps."""
ys = np.arange(h, dtype=np.float32) + 0.5
xs = np.arange(w, dtype=np.float32) + 0.5
X, Y = np.meshgrid(xs, ys)
best = np.full((h, w), 1e9, dtype=np.float32)
for line in polylines:
P = np.asarray(line, dtype=np.float32)
if len(P) == 1:
P = np.vstack([P, P])
for (ax, ay), (bx, by) in zip(P[:-1], P[1:]):
dx, dy = bx - ax, by - ay
ll = dx * dx + dy * dy
if ll < 1e-9:
d = np.hypot(X - ax, Y - ay)
else:
t = np.clip(((X - ax) * dx + (Y - ay) * dy) / ll, 0, 1)
d = np.hypot(X - (ax + t * dx), Y - (ay + t * dy))
np.minimum(best, d, out=best)
return np.clip(width / 2 + 0.5 - best, 0, 1)
def shadow(h, w, mask, color, alpha, blur, dx=0, dy=0):
"""A soft shadow of a mask: blurred, offset, as an RGBA layer of the same size."""
m = box_blur(np.roll(np.roll(mask, int(dy), 0), int(dx), 1), blur / 2)
layer = new(h, w, color)
layer[..., 3] = np.clip(m, 0, 1) * alpha
return layer
# ---------------------------------------------------------------- SVG path data (the filigree)
def svg_polylines(d, scale=1.0, ox=0.0, oy=0.0, steps=10):
"""Points along an SVG path made of M, L, C and Z commands."""
toks = re.findall(r'[MLCZmlcz]|-?\d*\.?\d+', d)
lines, cur, pos, i, cmd = [], [], (0.0, 0.0), 0, None
start = (0.0, 0.0)
def num():
nonlocal i
v = float(toks[i])
i += 1
return v
while i < len(toks):
if toks[i] in 'MLCZmlcz':
cmd = toks[i]
i += 1
if cmd in 'Zz':
cur.append(start)
continue
if cmd in 'Mm':
if cur:
lines.append(cur)
pos = (num(), num())
start = pos
cur = [pos]
cmd = 'L'
elif cmd in 'Ll':
pos = (num(), num())
cur.append(pos)
elif cmd in 'Cc':
p0 = pos
c1, c2, p1 = (num(), num()), (num(), num()), (num(), num())
for k in range(1, steps + 1):
t = k / steps
u = 1 - t
cur.append((u ** 3 * p0[0] + 3 * u * u * t * c1[0] + 3 * u * t * t * c2[0] + t ** 3 * p1[0],
u ** 3 * p0[1] + 3 * u * u * t * c1[1] + 3 * u * t * t * c2[1] + t ** 3 * p1[1]))
pos = p1
if cur:
lines.append(cur)
return [[((x - ox) * scale, (y - oy) * scale) for x, y in line] for line in lines]
# The game's divider ornament, traced from the options menu: two pointed lens ends, S-waves into spiral hooks and a
# diamond knot under the middle, in a 120 x 38 box (centre x 60).
FILIGREE_PATHS = (
'M20 17.4 C20.4 17.3 21.6 16.8 22.2 16.9 C22.8 17 23.5 17.7 23.5 18.2 C23.5 18.7 23.1 19.4 22.4 19.8 C21.6 20.2 '
'20.3 20.4 19 20.4 C17.7 20.4 15.9 20.2 14.5 19.9 C13.1 19.6 10.6 19.4 10.5 18.6 C10.4 17.9 12.6 16.3 14 15.4 '
'C15.4 14.5 17.2 13.6 19 13.2 C20.8 12.8 23.2 12.8 25 13.1 C26.8 13.4 28.1 13.9 29.5 14.8 C30.9 15.7 32.2 17.4 '
'33.5 18.6 C34.8 19.8 36 21.2 37.5 22 C39 22.8 40.8 23.4 42.5 23.6 C44.2 23.8 46 23.5 47.5 23.1 C49 22.7 50.4 '
'21.9 51.2 21.2 C52 20.5 52.5 19.5 52.4 18.8 C52.3 18.1 51.5 17.3 50.6 16.9 C49.7 16.5 48.2 16.2 47 16.3 C45.8 '
'16.4 44.2 17.1 43.6 17.2',
'M100 17.4 C99.6 17.3 98.4 16.8 97.8 16.9 C97.2 17 96.5 17.7 96.5 18.2 C96.5 18.7 96.8 19.4 97.6 19.8 C98.3 20.2 '
'99.7 20.4 101 20.4 C102.3 20.4 104.1 20.2 105.5 19.9 C106.9 19.6 109.4 19.4 109.5 18.6 C109.6 17.9 107.4 16.3 '
'106 15.4 C104.6 14.5 102.8 13.6 101 13.2 C99.2 12.8 96.8 12.8 95 13.1 C93.2 13.4 91.9 13.9 90.5 14.8 C89.1 15.7 '
'87.8 17.4 86.5 18.6 C85.2 19.8 84 21.2 82.5 22 C81 22.8 79.2 23.4 77.5 23.6 C75.8 23.8 74 23.5 72.5 23.1 C71 '
'22.7 69.6 21.9 68.8 21.2 C68 20.5 67.5 19.5 67.6 18.8 C67.7 18.1 68.5 17.3 69.4 16.9 C70.3 16.5 71.8 16.2 73 '
'16.3 C74.2 16.4 75.8 17.1 76.4 17.2',
'M40.6 17.6 C40.7 17 40.7 14.9 41.2 14 C41.7 13.1 42.6 12.4 43.8 12 C45 11.6 46.7 11.4 48.2 11.5 C49.7 11.6 51.3 '
'11.8 52.6 12.3 C53.9 12.8 55 13.6 56 14.6 C57 15.6 58.2 17.6 58.6 18.2 L64.6 26 L60 31.6',
'M79.4 17.6 C79.3 17 79.3 14.9 78.8 14 C78.3 13.1 77.4 12.4 76.2 12 C75 11.6 73.3 11.4 71.8 11.5 C70.3 11.6 68.7 '
'11.8 67.4 12.3 C66.1 12.8 65 13.6 64 14.6 C63 15.6 61.8 17.6 61.4 18.2 L55.4 26 L60 31.6',
)
# ---------------------------------------------------------------- the window's pieces
def backdrop(W, H, s):
"""The whole window behind the panels: a dim misty field under a see-through dark green, darker at the sides."""
img = new(H, W, PALETTE['window'], 1.0)
m = int(30 * s)
scene = new(H, W - 2 * m)
scene[..., :3] = vgradient(H, W - 2 * m, [(0, '#394244'), (0.36, '#414b4c'), (0.60, '#474234'), (1, '#3e3523')])
scene[..., 3] = 1
over(img, scene, m, 0)
band = np.zeros((H, W), dtype=np.float32)
y0, y1 = int(250 * s), int(330 * s)
t = (np.arange(y1 - y0) + 0.5) / max(y1 - y0, 1)
band[y0:y1, m:W - m] = (np.interp(t, [0, 0.55, 1], [0, 0.85, 0]))[:, None]
paint(img, box_blur(band, 10 * s), '#262a22')
over(img, hgradient_rgba(H, W, [(0, '#05180d', 0.96), (0.14, '#092615', 0.82), (0.5, '#0c2e1b', 0.72),
(0.86, '#092615', 0.82), (1, '#05180d', 0.96)]))
inner = rect_mask(H, W, 0, 0, W, H)
vig = 1 - box_blur(np.pad(inner, int(45 * s), constant_values=0), 45 * s)[int(45 * s):-int(45 * s),
int(45 * s):-int(45 * s)]
paint(img, np.clip(vig * 1.6, 0, 1), '#020e07', 0.6)
return img
def band(w, h, s):
"""The wooden title band, its two-line bottom edge and the shadow under it; (image, extra height below)."""
pad = int(14 * s)
img = new(h + pad, w)
wood = vgradient(h, w, [(0, '#4c3b26'), (0.30, '#5a4529'), (0.65, '#6b5232'), (1, '#806239')])
wood = overlay(wood, fractal(h, w, 300 * s, 11 * s, 3, seed=9), 0.32)
wood = overlay(wood, fractal(h, w, 33 * s, None, 3, seed=4), 0.2)
shade = rect_mask(h + pad, w, 0, 0, w, h + 4 * s)
over(img, shadow(h + pad, w, shade, '#000000', 0.6, 16 * s, 0, 5 * s))
top = new(h, w)
top[..., :3], top[..., 3] = wood, 1
over(img, top)
paint(img, rect_mask(h + pad, w, 0, h, w, h + 2 * s), '#2c2217')
paint(img, rect_mask(h + pad, w, 0, h + 2 * s, w, h + 4 * s), '#a9874f')
return img
def panel(w, h, s):
"""The left list panel with the shadow on its right."""
sh = int(26 * s)
img = new(h, w + sh)
body = rect_mask(h, w + sh, 0, 0, w, h)
over(img, shadow(h, w + sh, body, '#000000', 0.35, 18 * s, 6 * s, 0))
col = np.broadcast_to(rgb(PALETTE['panel']), (h, w, 3)).copy()
col = overlay(col, fractal(h, w, 33 * s, None, 3, seed=4), 0.18)
layer = new(h, w)
layer[..., :3], layer[..., 3] = col, 1
over(img, layer)
return img
def label_cell(w, h, s, bright=False):
"""A bronze label cell: gradient, mottled texture, 1 px border and the 1 px dark ring around the row."""
r = max(1, int(round(s)))
img = new(h + 2 * r, w + 2 * r, PALETTE['ring'], 1.0)
col = vgradient(h, w, [(0, PALETTE['label_top']), (0.5, PALETTE['label_mid']), (1, PALETTE['label_bottom'])])
col = overlay(col, fractal(h, w, 33 * s, None, 3, seed=(w * 7 + h) % 997), 0.22)
if bright:
col = np.clip(col * 1.08, 0, 1)
cell = new(h, w)
cell[..., :3], cell[..., 3] = col, 1
over(img, cell, r, r)
paint(img, ring_mask(h + 2 * r, w + 2 * r, r, r, w + r, h + r, r), PALETTE['label_border'])
return img
def desc_box(w, h, s):
pad = int(20 * s)
img = new(h + 2 * pad, w + 2 * pad)
body = rect_mask(h + 2 * pad, w + 2 * pad, pad, pad, pad + w, pad + h)
over(img, shadow(h + 2 * pad, w + 2 * pad, body, '#000000', 0.35, 18 * s, 0, 6 * s))
col = np.broadcast_to(rgb(PALETTE['desc']), (h, w, 3)).copy()
col = overlay(col, fractal(h, w, 33 * s, None, 3, seed=11), 0.15)
layer = new(h, w)
layer[..., :3], layer[..., 3] = col, 1
over(img, layer, pad, pad)
paint(img, ring_mask(h + 2 * pad, w + 2 * pad, pad, pad, pad + w, pad + h, max(1, round(s))),
PALETTE['desc_border'])
return img, pad
def glow_frames(w, h, s, n=8):
"""The selection cursor around a w x h box: a 3 px white frame 6 px outside it with a soft blue glow inside and
out, n frames of one breath. Returns (frames, margin) where the box sits at (margin, margin)."""
gap, thick, spread = 6 * s, 3 * s, 16 * s
mg = int(gap + spread)
H, W = h + 2 * mg, w + 2 * mg
ring = ring_mask(H, W, mg - gap, mg - gap, mg + w + gap, mg + h + gap, thick)
soft_out = box_blur(ring, 6 * s)
soft_far = box_blur(ring, 12 * s)
frames = []
for k in range(n):
t = 0.5 - 0.5 * np.cos(2 * np.pi * k / n)
img = new(H, W)
paint(img, np.clip(soft_far * 2.2, 0, 1), '#96b4e6', 0.35 + 0.25 * t)
paint(img, np.clip(soft_out * 2.8, 0, 1), '#afc5e8', 0.7 + 0.3 * t)
paint(img, ring, PALETTE['glow'])
frames.append(img)
return frames, mg
def filigree(w, s):
"""The divider: a thin line with a dot at each end and the ornament in the middle; (image, height)."""
h = int(round(26 * s))
img = new(h, w)
mid = h / 2
orn_w = 78 * s
scale = orn_w / 108.0
ox = (w - orn_w) / 2
dot_r = 1.5 * s
gap = 5 * s
left_end, right_start = ox - gap, ox + orn_w + gap
xs0, xs1 = 4 * s + 2 * dot_r + gap, w - 4 * s - 2 * dot_r - gap
paint(img, rect_mask(h, w, xs0, mid - 0.5 * s, left_end, mid + 0.5 * s), PALETTE['line'])
paint(img, rect_mask(h, w, right_start, mid - 0.5 * s, xs1, mid + 0.5 * s), PALETTE['line'])
dots = stroke_mask(h, w, [[(4 * s + dot_r, mid)], [(w - 4 * s - dot_r, mid)]], 2 * dot_r)
paint(img, dots, PALETTE['dot'])
lines = []
for d in FILIGREE_PATHS:
for line in svg_polylines(d, scale, 6, 8):
lines.append([(x + ox, y + mid - 13 * scale) for x, y in line])
paint(img, stroke_mask(h, w, lines, 2.2 * scale), PALETTE['filigree'])
return img, h
def triforce(w):
"""The Triforce, w pixels wide: three triangles, each shading from top to bottom, on a transparent ground."""
h = int(round(w * 217 / 250))
k = w / 250.0
tris = [((125, 1), (186.2, 107), (63.8, 107)), ((62.2, 109), (123.4, 215), (1, 215)),
((187.8, 109), (249, 215), (126.6, 215))]
img = new(h, w)
top, bottom = rgb(PALETTE['tri_top']), rgb(PALETTE['tri_bottom'])
for tri in tris:
pts = [(x * k, y * k) for x, y in tri]
mask = polygon_mask(h, w, pts)
y0, y1 = min(p[1] for p in pts), max(p[1] for p in pts)
t = np.clip((np.arange(h, dtype=np.float32) + 0.5 - y0) / max(y1 - y0, 1e-6), 0, 1)[:, None, None]
layer = new(h, w)
layer[..., :3] = top + (bottom - top) * t
layer[..., 3] = mask
over(img, layer)
return img
def triforce_ico():
"""The launcher's icon at the sizes Windows asks for."""
imgs = []
for n in (16, 24, 32, 48, 64, 128, 256):
t = triforce(n)
sq = new(n, n)
over(sq, t, 0, (n - t.shape[0]) // 2)
imgs.append(sq)
return ico_bytes(imgs)
def check_mark(size, s, color=None):
img = new(size, size)
k = size / 24.0
paint(img, stroke_mask(size, size, [[(5 * k, 12.5 * k), (9.2 * k, 16.7 * k), (19 * k, 7 * k)]], 3 * k),
color or PALETTE['ok'])
return img
def chevron(w, h, color, left=False):
img = new(h, w)
pts = [(w - 0.5, 0), (0.5, h / 2), (w - 0.5, h)] if left else [(0.5, 0), (w - 0.5, h / 2), (0.5, h)]
paint(img, polygon_mask(h, w, pts), color)
return img
def framed_icon(size, s, kind):
"""The square-framed icon of a big row (play or refresh), like the icon frames of the game's menu rows."""
img = new(size, size)
t = 1.5 * s
paint(img, ring_mask(size, size, 0.5, 0.5, size - 0.5, size - 0.5, t), '#e9dcc0')
c = size / 2
if kind == 'play':
k = size / 32
paint(img, polygon_mask(size, size, [(c - 5 * k, c - 7 * k), (c + 7 * k, c), (c - 5 * k, c + 7 * k)]),
'#efe4ca')
elif kind == 'refresh':
import math
k = size / 32
r = 7.5 * k
arcs = []
for a0, a1 in ((200, 340), (20, 160)):
arcs.append([(c + r * math.cos(math.radians(a)), c + r * math.sin(math.radians(a)))
for a in range(a0, a1 + 1, 6)])
heads = [[(c + r * math.cos(math.radians(340)) - 3.5 * k, c + r * math.sin(math.radians(340)) - 1 * k),
(c + r * math.cos(math.radians(340)), c + r * math.sin(math.radians(340))),
(c + r * math.cos(math.radians(340)) + 0.5 * k, c + r * math.sin(math.radians(340)) - 4 * k)],
[(c + r * math.cos(math.radians(160)) + 3.5 * k, c + r * math.sin(math.radians(160)) + 1 * k),
(c + r * math.cos(math.radians(160)), c + r * math.sin(math.radians(160))),
(c + r * math.cos(math.radians(160)) - 0.5 * k, c + r * math.sin(math.radians(160)) + 4 * k)]]
paint(img, stroke_mask(size, size, arcs + heads, 2.2 * k), '#efe4ca')
return img
def badge(size, s):
"""The orange "!" disc beside Updates when a new version is ready (the "!" itself is text)."""
img = new(size + 4, size + 4)
ring = stroke_mask(size + 4, size + 4, [[((size + 4) / 2, (size + 4) / 2)]], size + 1.5 * s)
over(img, shadow(size + 4, size + 4, ring, '#fbbd45', 0.6, 6 * s))
paint(img, ring, '#3a2a14')
disc = stroke_mask(size + 4, size + 4, [[((size + 4) / 2, (size + 4) / 2)]], size)
layer = new(size + 4, size + 4)
layer[..., :3] = vgradient(size + 4, size + 4, [(0, '#fbbd45'), (1, '#e2942a')])
layer[..., 3] = disc
over(img, layer)
return img

173
launcher/ui/selftest.py Normal file
View File

@@ -0,0 +1,173 @@
"""--selftest [--shots DIR]: draw every screen with made-up states, optionally save a PNG of each, and close.
Nothing runs: the jobs are built and their steps set by hand."""
import os
import subprocess
import sys
import time
from pathlib import Path
from .. import feed, jobs
def grab_gdi(x, y, w, h, path):
"""Windows: copy the screen rectangle with GDI and write it as PNG (art.png_bytes); no extra packages."""
import ctypes
from ctypes import wintypes
import numpy as np
from . import art
user32, gdi32 = ctypes.windll.user32, ctypes.windll.gdi32
user32.GetDC.restype = wintypes.HDC
gdi32.CreateCompatibleDC.restype = wintypes.HDC
gdi32.CreateCompatibleBitmap.restype = wintypes.HBITMAP
gdi32.SelectObject.restype = wintypes.HGDIOBJ
screen = user32.GetDC(None)
mem = gdi32.CreateCompatibleDC(screen)
bmp = gdi32.CreateCompatibleBitmap(screen, w, h)
old = gdi32.SelectObject(mem, bmp)
gdi32.BitBlt(mem, 0, 0, w, h, screen, x, y, 0x00CC0020) # SRCCOPY
class BITMAPINFOHEADER(ctypes.Structure):
_fields_ = [('biSize', wintypes.DWORD), ('biWidth', wintypes.LONG), ('biHeight', wintypes.LONG),
('biPlanes', wintypes.WORD), ('biBitCount', wintypes.WORD), ('biCompression', wintypes.DWORD),
('biSizeImage', wintypes.DWORD), ('biXPelsPerMeter', wintypes.LONG),
('biYPelsPerMeter', wintypes.LONG), ('biClrUsed', wintypes.DWORD),
('biClrImportant', wintypes.DWORD)]
bi = BITMAPINFOHEADER(ctypes.sizeof(BITMAPINFOHEADER), w, -h, 1, 32, 0, 0, 0, 0, 0, 0)
buf = (ctypes.c_ubyte * (w * h * 4))()
gdi32.GetDIBits(mem, bmp, 0, h, buf, ctypes.byref(bi), 0)
gdi32.SelectObject(mem, old)
gdi32.DeleteObject(bmp)
gdi32.DeleteDC(mem)
user32.ReleaseDC(None, screen)
a = np.frombuffer(buf, dtype=np.uint8).reshape(h, w, 4)[..., [2, 1, 0, 3]].copy()
a[..., 3] = 255
Path(path).write_bytes(art.png_bytes(a))
return True
def grab(app, path):
"""A PNG of the window: GDI on Windows, else PIL's ImageGrab, else ImageMagick's import on X11."""
app.root.update()
x, y = app.root.winfo_rootx(), app.root.winfo_rooty()
w, h = app.root.winfo_width(), app.root.winfo_height()
if os.name == 'nt':
try:
return grab_gdi(x, y, w, h, path)
except Exception as e:
app.ctx.log(f'selftest: screen capture failed: {e!r}')
try:
from PIL import ImageGrab
ImageGrab.grab(bbox=(x, y, x + w, y + h), xdisplay=os.environ.get('DISPLAY') if os.name != 'nt' else None
).save(path)
return True
except Exception:
pass
try:
subprocess.run(['import', '-window', str(app.root.winfo_id()), str(path)], check=True, capture_output=True)
return True
except (OSError, subprocess.CalledProcessError):
return False
def fake_release(version='0.3.11'):
m = {'format': 1, 'version': version, 'app': {'name': f'NX2navi-app-{version}.zip', 'size': 1, 'sha256': '0'}}
notes = ('- Each program module comes from the .nup package that holds it whole.\n'
'- The PC stays awake while the game builds.\n- This launcher: one-click install, updates and play.')
return feed.Release(m, notes)
def states(app):
ctx = app.ctx
s = app.settings
s['dump'] = s.get('dump') or str(Path.home() / 'Downloads' / 'Ocarina of Time.nup')
s['game_dir'] = s.get('game_dir') or ('D:\\Games\\NX2navi' if os.name == 'nt' else '/games/NX2navi')
app.dumps = [Path(s['dump'])]
def first_run():
app.show('first_run', len(app.rows_first_run()) - 1)
def installing():
j = jobs.InstallJob(ctx, s['dump'], s['game_dir'])
for sid, val in (('tools', 'Done · 0:48'), ('check', 'Version 1.1.1'), ('exefs', 'Done · 0:04'),
('romfs', 'Done · 0:31')):
j.step(sid).state, j.step(sid).value, j.step(sid).fraction = jobs.DONE, val, 1.0
j.begin('lift')
j.progress('lift', 0.34, '412 of 1,204 chunks')
j.started = time.time() - 192
for ln in ('check: Ocarina of Time 1.1.1 (build e0f3ccfa)', 'romfs: 26.0 of 26.0 GB, 31 s', 'lift:',
' emitting with 16 workers', ' emitted 412/1204 chunks, 18s'):
j.lines.append(ln)
app.job, app.job_return, app.job_back = j, 'home', 'first_run'
app.show('job')
def home():
app.release, app.checked_at, app.check, app.check_error, app.message = None, time.time() - 5, None, '', ''
app.show('home', 0)
def update_ready():
app.release = fake_release()
app.show('home', 1)
def updating():
j = jobs.UpdateDownloadJob(ctx, fake_release())
for sid, val in (('download', 'Done · 0:06'), ('verify', 'Checksum OK')):
j.step(sid).state, j.step(sid).value, j.step(sid).fraction = jobs.DONE, val, 1.0
j.skip('tools_update', 'Unchanged')
j.step('lift').state, j.step('lift').value, j.step('lift').fraction = jobs.DONE, 'Done · 0:41', 1.0
j.begin('compile_changed')
j.progress('compile_changed', 0.61, '238 of 391 changed files')
j.started = time.time() - 104
app.job, app.job_return = j, 'home'
app.show('job')
def settings():
app.show('settings', 0)
def failed():
j = jobs.InstallJob(ctx, s['dump'], s['game_dir'])
for sid, val in (('tools', 'Done · 0:48'), ('check', 'Version 1.1.1'), ('exefs', 'Done · 0:04'),
('romfs', 'Done · 0:31'), ('lift', 'Done · 0:48')):
j.step(sid).state, j.step(sid).value, j.step(sid).fraction = jobs.DONE, val, 1.0
j.step('compile').state, j.step('compile').value = jobs.FAILED, 'Stopped'
j.result, j.error = 'failed', '3 files did not compile'
j.started, j.finished = time.time() - 362, time.time()
for ln in ('compile:', ' compiled 1200 of 1212, 236 s', 'error: 3 files did not compile'):
j.lines.append(ln)
app.job, app.job_return, app.job_back = j, 'home', 'first_run'
app.show('job')
def uninstall():
app.show('uninstall', 0)
return [('1-first-run', first_run), ('2-installing', installing), ('3-home', home),
('4-update-ready', update_ready), ('5-updating', updating), ('6-settings', settings),
('7-build-failed', failed), ('8-uninstall', uninstall)]
def run(app, shots):
out = Path(shots) if shots else None
if out:
out.mkdir(parents=True, exist_ok=True)
seq = states(app)
ok = True
def step(i=0):
nonlocal ok
if i >= len(seq):
app.ctx.log(f'selftest: {len(seq)} screens drawn')
print(f'selftest: {len(seq)} screens drawn' + (f', shots in {out}' if out else ''), flush=True)
app.root.after(100, app.root.destroy)
return
name, fn = seq[i]
try:
fn()
app.root.update()
except Exception as e:
ok = False
print(f'selftest: {name}: {e!r}', file=sys.stderr, flush=True)
raise
if out:
app.root.after(350, lambda: (grab(app, out / f'{name}.png'), step(i + 1)))
else:
app.root.after(50, lambda: step(i + 1))
step()

158
launcher/updater.py Normal file
View File

@@ -0,0 +1,158 @@
"""Updating the launcher's own release folder (launcher/paths.py, "app/").
An update is staged beside the running version and switched to with two small files, so a failed or interrupted
update never leaves a half-written release in use:
1. The app archive goes to data/downloads, is checked against the manifest's size and sha256 and unpacks into
app/<version>.part, which is renamed to app/<version> once it is complete and holds the expected VERSION.
2. A new Python (the manifest's runtime id differs from python/runtime-id.txt) unpacks into python.next; the stub
swaps it in at its next start, while no Python runs.
3. app/previous.txt gets the running version and app/current.txt the new one; the launcher starts the stub again,
which runs the new version. The stub goes back to previous.txt when the new version fails to start.
4. The new version rebuilds the game (launcher/jobs.py), then prune() removes every version but these two.
"""
import os
import shutil
import zipfile
from pathlib import Path
from . import net, versions
class UpdateError(Exception):
pass
def read_text(path, default=''):
try:
return Path(path).read_text(encoding='utf-8').strip()
except OSError:
return default
def write_text(path, text):
path = Path(path)
tmp = path.with_name(path.name + '.tmp')
tmp.write_text(text + '\n', encoding='utf-8')
os.replace(tmp, path)
def current(layout):
return read_text(layout.apps / 'current.txt') or layout.version
def previous(layout):
return read_text(layout.apps / 'previous.txt')
def runtime_id(layout):
return read_text(layout.python / 'runtime-id.txt')
def safe_extract(zip_path, dest):
"""Unpack a zip into dest, refusing absolute names and names that climb out of it."""
dest = Path(dest)
root = dest.resolve()
with zipfile.ZipFile(zip_path) as z:
for info in z.infolist():
name = info.filename.replace('\\', '/')
if name.startswith('/') or ':' in name.split('/')[0] or '..' in name.split('/'):
raise UpdateError(f'{Path(zip_path).name}: unsafe path {info.filename}')
target = (dest / name).resolve()
if target != root and root not in target.parents:
raise UpdateError(f'{Path(zip_path).name}: unsafe path {info.filename}')
z.extractall(dest)
def _fetch(release, key, layout, progress, cancel, version):
url, headers, size, sha = release.asset(key)
entry = release.manifest[key]
dest = layout.downloads / entry['name']
net.download(url, dest, sha256=sha, size=size, headers=headers, progress=progress, cancel=cancel,
version=version)
return dest
def stage_app(layout, release, progress=None, cancel=None, version='dev'):
"""Download, check and unpack the release's app archive into app/<version>; returns that folder."""
zip_path = _fetch(release, 'app', layout, progress, cancel, version)
ver = versions.display(release.version)
final = layout.apps / ver
part = layout.apps / f'{ver}.part'
shutil.rmtree(part, ignore_errors=True)
part.mkdir(parents=True)
try:
safe_extract(zip_path, part)
got = read_text(part / 'VERSION')
if versions.display(got) != ver or not (part / 'launcher' / 'start.pyw').is_file():
raise UpdateError(f'{zip_path.name} holds version {got or "?"} without a launcher, not {ver}')
if final.exists():
if final.resolve() == layout.app.resolve():
raise UpdateError(f'{ver} is the version that runs now')
shutil.rmtree(final)
os.replace(part, final)
except BaseException:
shutil.rmtree(part, ignore_errors=True)
raise
zip_path.unlink(missing_ok=True)
return final
def runtime_needed(layout, release):
rt = release.manifest.get('runtime')
return bool(rt) and not layout.dev and rt.get('id') != runtime_id(layout)
def stage_runtime(layout, release, progress=None, cancel=None, version='dev'):
"""Unpack a new Python into python.next for the stub to swap in; False when the running one is current."""
if not runtime_needed(layout, release):
return False
zip_path = _fetch(release, 'runtime', layout, progress, cancel, version)
nxt = layout.root / 'python.next'
shutil.rmtree(nxt, ignore_errors=True)
part = layout.root / 'python.next.part'
shutil.rmtree(part, ignore_errors=True)
part.mkdir()
safe_extract(zip_path, part)
if not (part / 'python.exe').is_file() or read_text(part / 'runtime-id.txt') != release.manifest['runtime']['id']:
shutil.rmtree(part, ignore_errors=True)
raise UpdateError(f'{zip_path.name} is not the runtime {release.manifest["runtime"]["id"]}')
os.replace(part, nxt)
zip_path.unlink(missing_ok=True)
return True
def switch_to(layout, ver):
"""Make ver the version the stub starts, keeping the running one as previous."""
if not (layout.apps / ver / 'launcher' / 'start.pyw').is_file():
raise UpdateError(f'app/{ver} is not a complete release')
now = current(layout)
if now != ver:
write_text(layout.apps / 'previous.txt', now)
write_text(layout.apps / 'current.txt', ver)
def rollback(layout):
"""Back to previous.txt's version; False when there is none to go back to."""
prev = previous(layout)
if not prev or not (layout.apps / prev / 'launcher' / 'start.pyw').is_file():
return False
write_text(layout.apps / 'current.txt', prev)
(layout.apps / 'previous.txt').unlink(missing_ok=True)
return True
def prune(layout):
"""Remove every app version but the current and previous one, an old Python the stub set aside, and stale
downloads."""
keep = {current(layout), previous(layout)}
if layout.apps.is_dir():
for d in layout.apps.iterdir():
if d.is_dir() and d.name not in keep:
shutil.rmtree(d, ignore_errors=True)
for d in layout.root.glob('python.old*'):
shutil.rmtree(d, ignore_errors=True)
if layout.downloads.is_dir():
for f in layout.downloads.iterdir():
if f.suffix in ('.part', '.zip'):
f.unlink(missing_ok=True)

25
launcher/versions.py Normal file
View File

@@ -0,0 +1,25 @@
"""Release versions: "0.3.11", optionally with build metadata ("0.3.11+abc1234") that never orders releases."""
import re
_NUM = re.compile(r'^\s*v?(\d+(?:\.\d+)*)(?:-([0-9A-Za-z.-]+))?(?:\+([0-9A-Za-z.-]+))?\s*$')
def parse(text):
"""(numbers, pre-release) for comparing; a pre-release sorts before its release. ValueError when malformed."""
m = _NUM.match(text or '')
if not m:
raise ValueError(f'not a version: {text!r}')
nums = tuple(int(x) for x in m.group(1).split('.'))
nums = nums + (0,) * (4 - len(nums)) if len(nums) < 4 else nums
pre = m.group(2)
return nums, (1,) if pre is None else (0, pre)
def newer(a, b):
"""True when version a is newer than version b."""
return parse(a) > parse(b)
def display(text):
"""The version without build metadata."""
return (text or '').split('+', 1)[0].strip()

192
launcher/winsys.py Normal file
View File

@@ -0,0 +1,192 @@
"""What the launcher needs from the operating system: crisp drawing on scaled displays, the game's fonts for this
process only, starting the game and the stub, shortcuts, the uninstall entry and a gamepad. Windows first; the
other branches exist so the window can be developed and tested on Linux."""
import ctypes
import os
import subprocess
import sys
from pathlib import Path
CREATE_NO_WINDOW = 0x08000000
DETACHED_PROCESS = 0x00000008
CREATE_NEW_PROCESS_GROUP = 0x00000200
UNINSTALL_KEY = r'Software\Microsoft\Windows\CurrentVersion\Uninstall\NX2navi'
def dpi_aware():
"""Draw at the display's real resolution instead of letting Windows stretch a 96-dpi window (blurry)."""
if os.name != 'nt':
return
try:
ctypes.windll.shcore.SetProcessDpiAwareness(1)
except (AttributeError, OSError):
try:
ctypes.windll.user32.SetProcessDPIAware()
except (AttributeError, OSError):
pass
def app_id():
"""Group the window under the launcher's own taskbar icon rather than python's."""
if os.name == 'nt':
try:
ctypes.windll.shell32.SetCurrentProcessExplicitAppUserModelID('NX2navi.Launcher')
except (AttributeError, OSError):
pass
def register_font(path):
"""Make a font file usable by this process only (Tk then finds it by family name). True on success."""
path = str(path)
if os.name == 'nt':
try:
return ctypes.windll.gdi32.AddFontResourceExW(ctypes.c_wchar_p(path), 0x10, None) > 0 # FR_PRIVATE
except (AttributeError, OSError):
return False
try:
fc = ctypes.CDLL('libfontconfig.so.1')
fc.FcConfigAppFontAddFile.argtypes = [ctypes.c_void_p, ctypes.c_char_p]
return bool(fc.FcConfigAppFontAddFile(None, path.encode()))
except OSError:
return False
def open_path(path):
path = str(path)
if os.name == 'nt':
os.startfile(path) # noqa: S606 — opens a folder in Explorer
else:
subprocess.Popen(['xdg-open', path], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
def start_game(exe):
"""Start the game from its own folder; returns the process to watch."""
exe = Path(exe)
flags = {'creationflags': CREATE_NEW_PROCESS_GROUP} if os.name == 'nt' else {'start_new_session': True}
return subprocess.Popen([str(exe)], cwd=str(exe.parent), stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL, close_fds=True, **flags)
def start_detached(cmd, cwd=None):
flags = {'creationflags': DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP} if os.name == 'nt' \
else {'start_new_session': True}
subprocess.Popen(cmd, cwd=cwd, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
close_fds=True, **flags)
def restart(layout, args=()):
"""Start the launcher again through its stub (which swaps in a staged Python and starts app/current.txt's
version). False in a checkout, which has no stub."""
if not layout.stub.is_file():
return False
start_detached([str(layout.stub), *args], cwd=str(layout.root))
return True
def stub_run(layout, args, timeout=30):
"""Run the stub for one of its own commands (shortcuts); True when it succeeded."""
if not layout.stub.is_file():
return False
flags = {'creationflags': CREATE_NO_WINDOW} if os.name == 'nt' else {}
try:
return subprocess.run([str(layout.stub), *args], timeout=timeout, capture_output=True, **flags).returncode == 0
except (OSError, subprocess.TimeoutExpired):
return False
def desktop_shortcut(layout, enable):
return stub_run(layout, ['--desktop-shortcut', 'on' if enable else 'off'])
def keep_awake(on):
if os.name != 'nt':
return
try:
ES_CONTINUOUS, ES_SYSTEM_REQUIRED = 0x80000000, 0x00000001
ctypes.windll.kernel32.SetThreadExecutionState(ES_CONTINUOUS | (ES_SYSTEM_REQUIRED if on else 0))
except (AttributeError, OSError):
pass
def uninstall(layout, remove_game='', log=print):
"""Remove the shortcuts, the uninstall entry and, once this process has exited, the launcher's folder (and the
game folder when remove_game names it). Saves inside a kept game folder stay."""
stub_run(layout, ['--shortcuts', 'off'])
if os.name == 'nt':
try:
import winreg
winreg.DeleteKey(winreg.HKEY_CURRENT_USER, UNINSTALL_KEY)
except OSError:
pass
targets = [str(layout.root)] + ([str(remove_game)] if remove_game else [])
if layout.toolchain.parent != layout.root:
targets.append(str(layout.toolchain.parent))
script = 'ping -n 3 127.0.0.1 >nul & ' + ' & '.join(f'rmdir /s /q "{t}"' for t in targets)
start_detached(['cmd.exe', '/c', script], cwd=os.environ.get('TEMP', 'C:\\'))
else:
log('uninstall: remove these folders yourself: ' + ', '.join([str(layout.root), str(remove_game)]))
# ---------------------------------------------------------------- gamepad (XInput)
class _XInputGamepad(ctypes.Structure):
_fields_ = [('wButtons', ctypes.c_ushort), ('bLeftTrigger', ctypes.c_ubyte), ('bRightTrigger', ctypes.c_ubyte),
('sThumbLX', ctypes.c_short), ('sThumbLY', ctypes.c_short), ('sThumbRX', ctypes.c_short),
('sThumbRY', ctypes.c_short)]
class _XInputState(ctypes.Structure):
_fields_ = [('dwPacketNumber', ctypes.c_uint), ('Gamepad', _XInputGamepad)]
BUTTONS = {0x0001: 'up', 0x0002: 'down', 0x0004: 'left', 0x0008: 'right', 0x1000: 'a', 0x2000: 'b', 0x8000: 'y',
0x4000: 'x', 0x0010: 'start'}
class Gamepad:
"""Polls the first XInput pad; poll() returns the buttons pressed since the last call (edges), the left stick
counted as the d-pad."""
def __init__(self):
self.dll = None
self.held = set()
if os.name == 'nt':
for name in ('xinput1_4', 'xinput1_3', 'xinput9_1_0'):
try:
self.dll = getattr(ctypes.windll, name)
break
except (AttributeError, OSError):
continue
def poll(self):
if not self.dll:
return set()
st = _XInputState()
down = set()
for pad in range(4):
try:
if self.dll.XInputGetState(pad, ctypes.byref(st)) == 0:
g = st.Gamepad
down = {name for bit, name in BUTTONS.items() if g.wButtons & bit}
if g.sThumbLY > 16000:
down.add('up')
elif g.sThumbLY < -16000:
down.add('down')
if g.sThumbLX > 16000:
down.add('right')
elif g.sThumbLX < -16000:
down.add('left')
break
except OSError:
return set()
pressed = down - self.held
self.held = down
return pressed
def is_windows():
return os.name == 'nt'
def frozen_python():
return Path(sys.executable).parent

678
tests/launcher_test.py Normal file
View File

@@ -0,0 +1,678 @@
"""launcher/ and tools/make_game.py's bookkeeping, without game files or Windows.
Run: python tests/launcher_test.py (Python 3.14 for the zstd tests; the window test needs tkinter, numpy and a display)
- versions, settings
- toolchain: the package database, dependency resolution and an install from a local fake repository
- net: downloads that resume, check their sha256 and stop on cancel
- feed: GitHub releases (stable, preview, private with a token) and a plain manifest URL, served locally
- updater: staging a release, switching, rollback, pruning, refusing unsafe archives
- gamefiles: the .bfotf font wrapping and zstd dictionaries, on a font of this system made into a fake RomFS
- make_game.py: a synthetic .nup through the check, exefs and romfs steps with --events, skipped steps on a second
run, --rebuild without the game files, and the object cache of the compile step with the system's C compiler
- jobs: an install driven by a fake make_game.py
- the window: --selftest draws every screen (skipped without a display)
"""
import hashlib
import http.server
import io
import json
import os
import shutil
import struct
import subprocess
import sys
import tarfile
import tempfile
import threading
import unittest
import zipfile
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(ROOT))
sys.path.insert(0, str(ROOT / 'tools'))
from launcher import build, feed, gamefiles, jobs, net, settings, toolchain, updater, versions # noqa: E402
from launcher.context import Context # noqa: E402
from launcher.paths import Layout # noqa: E402
try:
from compression import zstd as ZSTD
except ImportError:
ZSTD = None
# ---------------------------------------------------------------- helpers
class Server:
"""A local HTTP server for a folder, with extra routes (path -> (status, headers, body)) and Range support."""
def __init__(self, folder, routes=None):
folder, routes = Path(folder), routes if routes is not None else {}
self.hits = []
server = self
class H(http.server.BaseHTTPRequestHandler):
def log_message(self, *a):
pass
def do_GET(self):
server.hits.append((self.path, dict(self.headers)))
if self.path in routes:
status, headers, body = routes[self.path]
self.send_response(status)
for k, v in headers.items():
self.send_header(k, v)
self.send_header('Content-Length', str(len(body)))
self.end_headers()
self.wfile.write(body)
return
p = folder / self.path.lstrip('/').split('?')[0]
if not p.is_file():
self.send_error(404)
return
data = p.read_bytes()
rng = self.headers.get('Range')
if rng and rng.startswith('bytes='):
start = int(rng[6:].split('-')[0])
if start >= len(data):
self.send_error(416)
return
self.send_response(206)
self.send_header('Content-Range', f'bytes {start}-{len(data) - 1}/{len(data)}')
data = data[start:]
else:
self.send_response(200)
self.send_header('Content-Length', str(len(data)))
self.end_headers()
try:
self.wfile.write(data)
except ConnectionError: # the client stopped reading (the cancel test)
pass
self.httpd = http.server.ThreadingHTTPServer(('127.0.0.1', 0), H)
self.url = f'http://127.0.0.1:{self.httpd.server_address[1]}/'
threading.Thread(target=self.httpd.serve_forever, daemon=True).start()
def close(self):
self.httpd.shutdown()
def sha(data):
return hashlib.sha256(data).hexdigest()
def tar_bytes(files, compress=True):
"""A tar (zstd-compressed when the stdlib has zstd, else gzip) of {name: bytes}."""
buf = io.BytesIO()
mode = 'w:gz' if not (compress and ZSTD) else 'w'
with tarfile.open(fileobj=buf, mode=mode) as t:
for name, data in files.items():
info = tarfile.TarInfo(name)
info.size = len(data)
t.addfile(info, io.BytesIO(data))
data = buf.getvalue()
return ZSTD.compress(data) if compress and ZSTD else data
def desc(name, version, filename, data, depends=(), provides=()):
lines = ['%FILENAME%', filename, '', '%NAME%', name, '', '%VERSION%', version, '', '%CSIZE%', str(len(data)), '',
'%ISIZE%', str(len(data) * 3), '', '%SHA256SUM%', sha(data), '']
if depends:
lines += ['%DEPENDS%', *depends, '']
if provides:
lines += ['%PROVIDES%', *provides, '']
return '\n'.join(lines).encode()
def fake_repo(folder, damage=None):
"""A pacman repository with the ten root packages (each holding one of make_game.py's required files), one
dependency and one name provided by another package."""
P = toolchain.PREFIX
files = {'clang': {'ucrt64/bin/clang.exe': b'clang', 'ucrt64/bin/clang++.exe': b'clang++'},
'sdl3': {'ucrt64/bin/SDL3.dll': b'sdl', 'ucrt64/lib/libSDL3.dll.a': b'implib'},
'vulkan-memory-allocator': {'ucrt64/include/vma/vk_mem_alloc.h': b'// vma'},
'vulkan-loader': {'ucrt64/lib/libvulkan-1.dll.a': b'implib'},
'llvm-libs': {'ucrt64/bin/libLLVM.dll': b'llvm', 'ucrt64/share/doc/llvm/README': b'skipped'}}
deps = {'clang': [P + 'llvm-libs=23.1', P + 'gcc'], 'sdl3': [P + 'vulkan']}
entries = {}
for root in list(toolchain.ROOTS) + ['llvm-libs']:
name = P + root
fn = f'{name}-1.0-1-any.pkg.tar.zst'
data = tar_bytes({'.PKGINFO': b'pkg', **files.get(root, {f'ucrt64/share/{root}/x': b'x'})})
if damage == root:
(folder / fn).write_bytes(data[:-3] + b'bad')
else:
(folder / fn).write_bytes(data)
entries[f'{name}-1.0-1/desc'] = desc(name, '1.0-1', fn, data, deps.get(root, ()),
[P + 'vulkan'] if root == 'vulkan-loader' else ())
(folder / 'ucrt64.db').write_bytes(tar_bytes(entries))
class Tmp(unittest.TestCase):
def setUp(self):
self.tmp = Path(tempfile.mkdtemp(prefix='nx2navi-test-'))
self.addCleanup(shutil.rmtree, self.tmp, True)
self.layout = Layout(root=self.tmp / 'launcher', app=ROOT).ensure()
# ---------------------------------------------------------------- versions, settings
class VersionTest(unittest.TestCase):
def test_order(self):
self.assertTrue(versions.newer('0.3.11', '0.3.10'))
self.assertTrue(versions.newer('0.4', '0.3.99'))
self.assertFalse(versions.newer('0.3.10+abc', '0.3.10'))
self.assertTrue(versions.newer('1.0.0', '1.0.0-rc1'))
self.assertEqual(versions.display('0.3.11+abc1234'), '0.3.11')
with self.assertRaises(ValueError):
versions.parse('banana')
class SettingsTest(Tmp):
def test_roundtrip_and_reset(self):
s = settings.Settings(self.layout.settings)
self.assertTrue(s['auto_check'])
s['auto_check'], s['game_dir'], s['channel'] = False, 'D:/g', 'preview'
s.save()
t = settings.Settings(self.layout.settings)
self.assertEqual((t['auto_check'], t['game_dir'], t['channel']), (False, 'D:/g', 'preview'))
t.reset()
self.assertEqual((t['auto_check'], t['game_dir'], t['channel']), (True, 'D:/g', 'stable'))
def test_bad_types_ignored(self):
self.layout.settings.write_text(json.dumps({'auto_check': 'yes', 'channel': 'preview', 'junk': 1}))
s = settings.Settings(self.layout.settings)
self.assertEqual((s['auto_check'], s['channel']), (True, 'preview'))
self.assertNotIn('junk', s)
# ---------------------------------------------------------------- toolchain
class ToolchainTest(Tmp):
def test_resolve_and_install(self):
repo = self.tmp / 'repo'
repo.mkdir()
fake_repo(repo)
srv = Server(repo)
self.addCleanup(srv.close)
lock = toolchain.make_lock(srv.url)
names = [p['name'] for p in lock['packages']]
self.assertEqual(names[0], toolchain.PREFIX + 'clang')
self.assertIn(toolchain.PREFIX + 'llvm-libs', names) # a dependency with a version constraint
self.assertEqual(names.count(toolchain.PREFIX + 'vulkan-loader'), 1) # 'vulkan' resolves to its provider
self.assertFalse(toolchain.is_current(self.layout, lock))
phases = []
toolchain.install(self.layout, lock, progress=lambda ph, d, t: phases.append(ph), log=lambda m: None)
self.assertTrue(toolchain.is_current(self.layout, lock))
self.assertTrue((self.layout.msys / 'bin' / 'clang.exe').is_file())
self.assertFalse((self.layout.msys / 'share' / 'doc').exists()) # documentation left out
self.assertIn('download', phases)
self.assertIn('unpack', phases)
hits = len(srv.hits)
toolchain.install(self.layout, lock, log=lambda m: None) # cached package files: no download
self.assertEqual(len(srv.hits), hits)
self.assertEqual(toolchain.lock_digest(lock), toolchain.installed(self.layout)['digest'])
def test_damaged_package_refused(self):
repo = self.tmp / 'repo'
repo.mkdir()
fake_repo(repo, damage='opus')
srv = Server(repo)
self.addCleanup(srv.close)
lock = toolchain.make_lock(srv.url)
with self.assertRaises(toolchain.ToolchainError):
toolchain.install(self.layout, lock, log=lambda m: None)
self.assertFalse(toolchain.is_current(self.layout, lock))
def test_wheel_tag(self):
tag = toolchain.wheel_tag()[0]
self.assertTrue(tag.startswith(f'cp{sys.version_info.major}{sys.version_info.minor}-'))
# ---------------------------------------------------------------- net
class NetTest(Tmp):
def test_download_resume_and_checks(self):
data = os.urandom(3 * net.CHUNK + 123)
(self.tmp / 'f.bin').write_bytes(data)
srv = Server(self.tmp)
self.addCleanup(srv.close)
dest = self.tmp / 'out' / 'f.bin'
dest.parent.mkdir()
(dest.parent / 'f.bin.part').write_bytes(data[:1000]) # a part file from a cut connection
net.download(srv.url + 'f.bin', dest, sha256=sha(data), size=len(data))
self.assertEqual(dest.read_bytes(), data)
self.assertEqual(srv.hits[-1][1].get('Range'), 'bytes=1000-')
with self.assertRaises(net.DownloadError):
net.download(srv.url + 'f.bin', self.tmp / 'out' / 'g.bin', sha256='0' * 64)
self.assertFalse((self.tmp / 'out' / 'g.bin').exists())
with self.assertRaises(net.Cancelled):
net.download(srv.url + 'f.bin', self.tmp / 'out' / 'h.bin', cancel=lambda: True)
with self.assertRaises(net.DownloadError):
net.download(srv.url + 'missing.bin', self.tmp / 'out' / 'm.bin')
# ---------------------------------------------------------------- feed and updater
def make_app_zip(folder, ver, extra=None):
app = folder / f'app-{ver}'
(app / 'launcher').mkdir(parents=True)
(app / 'launcher' / 'start.pyw').write_text('# test\n')
(app / 'VERSION').write_text(ver + '\n')
z = folder / f'NX2navi-app-{ver}.zip'
with zipfile.ZipFile(z, 'w') as zf:
for p in app.rglob('*'):
if p.is_file():
zf.write(p, p.relative_to(app).as_posix())
for name, data in (extra or {}).items():
zf.writestr(name, data)
return z
def manifest_for(z, ver):
return {'format': 1, 'version': ver, 'notes': '- One\n- Two',
'app': {'name': z.name, 'size': z.stat().st_size, 'sha256': sha(z.read_bytes())}}
class FeedTest(Tmp):
def serve_github(self):
self.routes = {}
srv = Server(self.tmp, self.routes)
self.addCleanup(srv.close)
return srv
def publish(self, releases):
self.routes['/repos/o/r/releases?per_page=20'] = (200, {'Content-Type': 'application/json'},
json.dumps(releases).encode())
def release(self, srv, tag, ver, pre=False, manifest=True):
z = make_app_zip(self.tmp, ver)
m = self.tmp / f'm-{ver}.json'
m.write_text(json.dumps(manifest_for(z, ver)))
assets = [{'name': z.name, 'browser_download_url': srv.url + z.name, 'url': srv.url + z.name}]
if manifest:
assets.append({'name': feed.MANIFEST, 'browser_download_url': srv.url + m.name, 'url': srv.url + m.name})
return {'tag_name': tag, 'prerelease': pre, 'draft': False, 'body': 'notes', 'html_url': 'x', 'assets': assets}
def test_github_channels(self):
srv = self.serve_github()
self.publish([self.release(srv, 'v0.4.0-rc1', '0.4.0-rc1', pre=True),
self.release(srv, 'v0.3.12', '0.3.12', manifest=False), self.release(srv, 'v0.3.11', '0.3.11')])
f = {'type': 'github', 'repo': 'o/r', 'api': srv.url.rstrip('/')}
self.assertEqual(feed.latest(f, 'stable').version, '0.3.11') # 0.3.12 lacks a manifest
self.assertEqual(feed.latest(f, 'preview').version, '0.4.0-rc1')
r = feed.latest(f, 'stable', token='secret')
self.assertEqual(r.headers.get('Authorization'), 'Bearer secret')
self.assertTrue(any(h.get('Authorization') == 'Bearer secret' for _, h in srv.hits))
self.assertEqual(feed.plain_notes('## Head\n- **One**\n* `Two`\n<!-- x -->'), ['Head', 'One', 'Two'])
def test_url_feed_and_update(self):
z = make_app_zip(self.tmp, '0.9.0')
(self.tmp / feed.MANIFEST).write_text(json.dumps(manifest_for(z, '0.9.0')))
srv = Server(self.tmp)
self.addCleanup(srv.close)
rel = feed.latest({'type': 'url', 'url': srv.url + feed.MANIFEST})
self.assertEqual(rel.version, '0.9.0')
self.layout.apps.mkdir(parents=True)
(self.layout.apps / 'current.txt').write_text('0.8.0\n')
old = self.layout.apps / '0.8.0' / 'launcher'
old.mkdir(parents=True)
(old / 'start.pyw').write_text('')
for stale in ('0.7.0', '0.6.0'):
(self.layout.apps / stale).mkdir()
folder = updater.stage_app(self.layout, rel)
self.assertTrue((folder / 'launcher' / 'start.pyw').is_file())
updater.switch_to(self.layout, '0.9.0')
self.assertEqual((updater.current(self.layout), updater.previous(self.layout)), ('0.9.0', '0.8.0'))
updater.prune(self.layout)
self.assertEqual(sorted(p.name for p in self.layout.apps.iterdir() if p.is_dir()), ['0.8.0', '0.9.0'])
self.assertTrue(updater.rollback(self.layout))
self.assertEqual(updater.current(self.layout), '0.8.0')
self.assertFalse(updater.rollback(self.layout)) # nothing further back
def test_unsafe_or_wrong_archives(self):
z = make_app_zip(self.tmp, '1.0.0', extra={'../evil.txt': 'x'})
with self.assertRaises(updater.UpdateError):
updater.safe_extract(z, self.tmp / 'x')
self.assertFalse((self.tmp / 'evil.txt').exists())
z2 = make_app_zip(self.tmp, '1.0.1')
m = manifest_for(z2, '1.0.2') # the archive says 1.0.1
(self.tmp / feed.MANIFEST).write_text(json.dumps(m))
srv = Server(self.tmp)
self.addCleanup(srv.close)
rel = feed.Release(m, assets={z2.name: srv.url + z2.name})
with self.assertRaises(updater.UpdateError):
updater.stage_app(self.layout, rel)
self.assertFalse((self.layout.apps / '1.0.2').exists())
# ---------------------------------------------------------------- gamefiles
def sarc(files):
names = b''
offs = []
for n in files:
offs.append(len(names))
b = n.encode() + b'\0'
names += b + b'\0' * (-len(b) % 4)
data, ents = b'', b''
for (n, d), no in zip(files.items(), offs):
start = len(data)
data += d
ents += struct.pack('<IIII', 0, 0x01000000 | (no // 4), start, len(data))
data += b'\0' * (-len(data) % 8)
sfat = b'SFAT' + struct.pack('<HHI', 0xC, len(files), 0x65) + ents
sfnt = b'SFNT' + struct.pack('<HH', 8, 0) + names
base = 0x14 + len(sfat) + len(sfnt)
head = b'SARC' + struct.pack('<HHIIHH', 0x14, 0xFEFF, base + len(data), base, 0x100, 0)
return head + sfat + sfnt + data
def bfotf(otf, key=0x1234ABCD):
body = otf + b'\0' * (-len(otf) % 4)
words = struct.unpack(f'>{len(body) // 4}I', body)
return struct.pack('>II', 0x7F9A0218 ^ key, len(otf) ^ key) + struct.pack(f'>{len(words)}I',
*(w ^ key for w in words))
@unittest.skipUnless(ZSTD, 'needs compression.zstd (Python 3.14)')
class FontTest(Tmp):
FONT = Path('/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf')
@unittest.skipUnless(FONT.is_file(), 'needs DejaVu Sans')
def test_font_from_romfs_image(self):
otf = self.FONT.read_bytes()
self.assertEqual(gamefiles.bfotf_to_otf(bfotf(otf)), otf)
self.assertIsNone(gamefiles.bfotf_to_otf(b'\0' * 64))
samples = [otf[i:i + 512] for i in range(0, min(len(otf), 400000), 997)]
d = ZSTD.train_dict(samples, 16384)
packed = ZSTD.compress(bfotf(otf), zstd_dict=d)
self.assertNotEqual(ZSTD.get_frame_info(packed).dictionary_id, 0)
files = {'Pack/ZsDic.pack.zs': ZSTD.compress(sarc({'zs.zsdic': d.dict_content})),
gamefiles.FONTS['title']: packed}
romfs = self.tmp / 'game' / 'romfs'
romfs.mkdir(parents=True)
entries, blob = [], b''
for path, data in sorted(files.items()):
blob += b'\0' * (-len(blob) % 16)
entries.append({'path': path, 'offset': len(blob), 'size': len(data)})
blob += data
(romfs / 'romfs.bin').write_bytes(blob)
(romfs / 'index.json').write_text(json.dumps({'files': entries}))
got = gamefiles.extract_fonts(self.layout, self.tmp / 'game')
self.assertEqual(got['title'][1], 'DejaVu Sans')
self.assertEqual(Path(got['title'][0]).read_bytes(), otf)
self.assertEqual(gamefiles.cached_fonts(self.layout)['title'][1], 'DejaVu Sans')
# ---------------------------------------------------------------- make_game.py
def nso(build_id=b'\0' * 16, size=0x300):
head = bytearray(0x100)
head[0:4] = b'NSO0'
segs = [(0x100 + i * size, i * 0x10000, size) for i in range(3)]
for i, (fo, mo, sz) in enumerate(segs):
struct.pack_into('<III', head, 0x10 + i * 0x10, fo, mo, sz)
struct.pack_into('<I', head, 0x60 + i * 4, sz)
head[0x40:0x50] = build_id
return bytes(head) + b''.join(bytes([i + 1]) * size for i in range(3))
def write_nup(path, exefs, romfs, title_id=0x0400486024DEC000):
"""A .nup (tools/nup.py's format) with one base package: exefs {name: bytes}, romfs {path: bytes}."""
import nup as nupmod
pool = bytearray()
def name(s):
off = len(pool)
pool.extend(s.encode())
return off, len(s.encode())
dirs, files = [], []
def add_tree(tree):
"""tree: {name: bytes | dict}; returns the directory index."""
idx = len(dirs)
dirs.append(None)
kids = []
for n, v in sorted(tree.items()):
if isinstance(v, dict):
kids.append(('d', add_tree(v), n))
else:
files.append([n, v])
kids.append(('f', len(files) - 1, n))
dirs[idx] = kids
return idx
root = add_tree({'exefs': exefs, 'romfs': _nest(romfs)})
nd = len(dirs)
dir_rows, child_list = [], []
for kids in dirs:
first = len(child_list)
for kind, i, _n in kids:
child_list.append(i if kind == 'd' else nd + i)
dir_rows.append((first, len(kids)))
dir_names = {root: ''}
for kids in dirs:
for kind, i, n in kids:
if kind == 'd':
dir_names[i] = n
exefs_dir = next(i for k, i, n in dirs[root] if k == 'd' and n == 'exefs')
romfs_dir = next(i for k, i, n in dirs[root] if k == 'd' and n == 'romfs')
toc_head = 0x30 + nupmod.PACKAGE.size + nd * nupmod.DIRECTORY.size + len(files) * nupmod.FILE.size + \
4 * len(child_list)
blob_off = nupmod.SUPER.size
data = bytearray()
file_rows = []
for n, v in files:
no, nl = name(n)
file_rows.append((no, nl, len(v), blob_off + len(data), len(v), hashlib.sha256(v).digest(), 0))
data.extend(v)
dir_name_rows = [name(dir_names[i]) for i in range(nd)]
toc_off = blob_off + len(data)
toc = bytearray(struct.pack('<IIQQQQQ', 0x01434F54, 1, nd, len(files), toc_head, len(pool), 0))
toc += nupmod.PACKAGE.pack(title_id, root, exefs_dir, romfs_dir, 0, 0, 0, 0, 0, 0, 0, b'\0' * 20)
for i in range(nd):
toc += nupmod.DIRECTORY.pack(*dir_name_rows[i], *dir_rows[i], 0)
for no, nl, size, off, stored, digest, codec in file_rows:
toc += nupmod.FILE.pack(no, nl, size, off, stored, digest, codec, 0, 0, b'\0' * 12)
toc += struct.pack(f'<{len(child_list)}I', *child_list)
toc += pool
total = toc_off + len(toc)
sup = nupmod.SUPER.pack(b'NUP\x01', 0x80, 1, toc_off, len(toc), hashlib.sha256(bytes(toc)).digest(), blob_off,
total, 1, 0, b'\0' * 16, b'\0' * 32)
Path(path).write_bytes(sup + bytes(data) + bytes(toc))
def _nest(flat):
tree = {}
for p, v in flat.items():
parts = p.split('/')
node = tree
for part in parts[:-1]:
node = node.setdefault(part, {})
node[parts[-1]] = v
return tree
class MakeGameTest(Tmp):
def setUp(self):
super().setUp()
import make_game
self.mg = make_game
exefs = {'main': nso(bytes.fromhex(make_game.MAIN_BUILD_ID)), 'sdk': nso(), 'subsdk0': nso(), 'rtld': nso(),
'main.npdm': b'META' + b'\0' * 0x100}
self.romfs = {'a.txt': b'hello', 'dir/b.bin': os.urandom(5000), 'dir/sub/c': b'c' * 33}
self.nup = self.tmp / 'game.nup'
write_nup(self.nup, exefs, self.romfs)
self.msys = self.tmp / 'msys'
for need in ('bin/clang.exe', 'bin/clang++.exe', 'bin/SDL3.dll', 'lib/libSDL3.dll.a'):
(self.msys / need).parent.mkdir(parents=True, exist_ok=True)
(self.msys / need).write_bytes(b'')
(self.msys / 'include' / 'vma').mkdir(parents=True)
self.layout.pydeps.mkdir(parents=True, exist_ok=True)
(self.layout.pydeps / 'zbic.py').write_text('def decompress(b):\n return b\n')
self.out = self.tmp / 'game out'
def run_mg(self, *args):
events, lines = [], []
run = build.Run(build.command(self.layout, list(args)), events.append, lines.append)
code = run.wait()
return code, events, lines
def test_pipeline_rebuild_and_keys(self):
code, ev, lines = self.run_mg('--nup', self.nup, '--out', self.out, '--msys', self.msys, '--until', 'romfs')
self.assertEqual(code, 0, '\n'.join(lines))
states = [(e['step'], e['state']) for e in ev if e['ev'] == 'step']
self.assertEqual(states, [('check', 'start'), ('check', 'done'), ('exefs', 'start'), ('exefs', 'done'),
('romfs', 'start'), ('romfs', 'done')])
self.assertTrue(any(e['ev'] == 'progress' and e['step'] == 'romfs' and e['done'] == e['total'] for e in ev))
index = json.loads((self.out / 'romfs' / 'index.json').read_text())
blob = (self.out / 'romfs' / 'romfs.bin').read_bytes()
for e in index['files']:
self.assertEqual(blob[e['offset']:e['offset'] + e['size']], self.romfs[e['path']])
self.assertEqual(e['offset'] % 16, 0)
flat = (self.out / 'exefs' / 'main').read_bytes()
self.assertEqual(flat[0x40:0x50].hex(), self.mg.MAIN_BUILD_ID)
# a second run with the same input skips both steps
code, ev, _ = self.run_mg('--nup', self.nup, '--out', self.out, '--msys', self.msys, '--until', 'romfs')
self.assertEqual([e['state'] for e in ev if e['ev'] == 'step' and e['step'] in ('exefs', 'romfs')],
['skip', 'skip'])
# --rebuild needs no game files: it checks the unpacked ExeFS and keeps both steps
self.nup.rename(self.tmp / 'moved.nup')
code, ev, lines = self.run_mg('--rebuild', '--out', self.out, '--msys', self.msys, '--until', 'romfs')
self.assertEqual(code, 0, '\n'.join(lines))
self.assertEqual([e['state'] for e in ev if e['ev'] == 'step' and e['step'] in ('exefs', 'romfs')],
['skip', 'skip'])
# the exefs and romfs keys do not depend on the kit; the lift key does
st = json.loads((self.out / 'build-cache' / 'state.json').read_text())
self.assertEqual(st['steps']['exefs'], self.mg.step_key('exefs', st['input']))
self.assertNotIn(self.mg.kit_version(), st['steps']['romfs'])
def test_rebuild_without_build(self):
code, ev, lines = self.run_mg('--rebuild', '--out', self.tmp / 'empty', '--msys', self.msys)
self.assertEqual(code, 1)
self.assertTrue(any(e['ev'] == 'error' and 'rebuild' in e['message'] for e in ev))
def test_wrong_version_refused(self):
bad = self.tmp / 'bad.nup'
write_nup(bad, {'main': nso(b'\x11' * 16), 'sdk': nso(), 'subsdk0': nso(), 'rtld': nso(),
'main.npdm': b'META' + b'\0' * 0x100}, {'a': b'a'})
code, ev, _ = self.run_mg('--nup', bad, '--check')
self.assertEqual(code, 1)
self.assertTrue(any(e['ev'] == 'error' and '1.1.1' in e['message'] for e in ev))
def test_parse_deps(self):
text = 'C:/o/x.o: C:/src/x.c C:/inc/a\\ b.h \\\n C:/inc/c.h\n'
self.assertEqual(self.mg.parse_deps(text), ['C:/src/x.c', 'C:/inc/a b.h', 'C:/inc/c.h'])
@unittest.skipUnless(shutil.which('cc'), 'needs a C compiler')
def test_object_cache(self):
src = self.tmp / 'src'
src.mkdir()
(src / 'h.h').write_text('#define V 1\n')
(src / 'a.c').write_text('#include "h.h"\nint a(void) { return V; }\n')
(src / 'b.c').write_text('int b(void) { return 2; }\n')
obj = self.tmp / 'obj'
obj.mkdir()
srcs = [(src / 'a.c', obj / 'a.o'), (src / 'b.c', obj / 'b.o')]
def cmd_for(s, d):
return ['cc', '-c', '-I', str(src), '-o', str(d), str(s)]
def go():
cache = self.mg.ObjectCache(obj / 'objects.json')
return self.mg.compile_all(srcs, cmd_for, cache, 2, dict(os.environ), salt='cc')
self.assertEqual(go()[:2], (2, 0))
self.assertEqual(go()[:2], (0, 2))
(src / 'h.h').write_text('#define V 3\n') # a header: only a.c includes it
self.assertEqual(go()[:2], (1, 1))
(src / 'b.c').write_text('int b(void) { return syntax error }\n')
compiled, reused, bad = go()
self.assertEqual((compiled, len(bad)), (1, 1))
self.assertNotIn('b.o', json.loads((obj / 'objects.json').read_text()))
# ---------------------------------------------------------------- jobs
FAKE_MAKE_GAME = r'''
import json, sys, time
def ev(**k): print('@@' + json.dumps(k), flush=True)
if '--fail' in open(sys.argv[0] + '.mode').read():
ev(ev='step', step='check', state='start'); ev(ev='error', message='this is not game version 1.1.1'); sys.exit(1)
for s in ('check', 'exefs', 'romfs', 'lift', 'compile', 'shaders'):
ev(ev='step', step=s, state='start')
if s == 'compile':
ev(ev='progress', step=s, done=5, total=10, unit='files'); ev(ev='phase', step=s, phase='link')
print('working on', s, flush=True)
ev(ev='step', step=s, state='done', seconds=0.1)
ev(ev='end', exe='x')
'''
class JobTest(Tmp):
def make_ctx(self, mode):
app = self.tmp / 'app'
(app / 'tools').mkdir(parents=True)
(app / 'VERSION').write_text('0.3.10\n')
script = app / 'tools' / 'make_game.py'
script.write_text(FAKE_MAKE_GAME)
(app / 'tools' / 'make_game.py.mode').write_text(mode)
ctx = Context(Layout(root=self.tmp / 'root', app=app))
return ctx
def patch_tools(self):
saved = toolchain.load_lock, toolchain.is_current, toolchain.ensure_zbic
toolchain.load_lock = lambda layout, version='dev': {'packages': []}
toolchain.is_current = lambda layout, lock: True
toolchain.ensure_zbic = lambda *a, **k: None
self.addCleanup(lambda: setattr(toolchain, 'load_lock', saved[0]) or
setattr(toolchain, 'is_current', saved[1]) or setattr(toolchain, 'ensure_zbic', saved[2]))
def test_install_job(self):
self.patch_tools()
ctx = self.make_ctx('ok')
j = jobs.InstallJob(ctx, self.tmp / 'g.nup', self.tmp / 'game').run_inline()
self.assertEqual(j.result, 'ok', j.error)
self.assertEqual([s.state for s in j.steps], [jobs.SKIPPED] + [jobs.DONE] * 7) # tools ... shaders, ready
self.assertEqual(j.step('check').value, 'Version 1.1.1')
self.assertTrue(any('working on lift' in ln for ln in j.lines))
self.assertEqual(j.overall(), 1.0)
def test_failed_job(self):
self.patch_tools()
ctx = self.make_ctx('--fail')
j = jobs.InstallJob(ctx, self.tmp / 'g.nup', self.tmp / 'game').run_inline()
self.assertEqual(j.result, 'failed')
self.assertIn('1.1.1', j.error)
self.assertEqual(j.step('check').state, jobs.FAILED)
# ---------------------------------------------------------------- the window
def have_display():
if os.name == 'nt':
return True
if not os.environ.get('DISPLAY'):
return False
try:
import tkinter # noqa: F401
import numpy # noqa: F401
return True
except ImportError:
return False
@unittest.skipUnless(have_display(), 'needs tkinter, numpy and a display')
class WindowTest(Tmp):
def test_selftest(self):
r = subprocess.run([sys.executable, '-m', 'launcher', '--root', str(self.tmp / 'w'), '--selftest'],
cwd=ROOT, capture_output=True, text=True, timeout=180)
self.assertEqual(r.returncode, 0, r.stderr)
self.assertIn('8 screens drawn', r.stdout)
if __name__ == '__main__':
unittest.main(verbosity=2)

View File

@@ -3,8 +3,9 @@
python -I tools/make_game.py --nup "C:\\path\\game.nup" --out "C:\\Games\\Ocarina"
python -I tools/make_game.py --exefs "C:\\path\\exefs" --romfs "C:\\path\\romfs" --out "C:\\Games\\Ocarina"
python -I tools/make_game.py --nup "C:\\path\\game.nup" --check # only checks the game's version
python -I tools/make_game.py --rebuild --out "C:\\Games\\Ocarina" # a newer kit, the files of the last build
Steps, each skipped when a run before finished it for the same input (state in <out>/build-cache/state.json):
Steps, each skipped when a run before finished it for the same inputs (state in <out>/build-cache/state.json):
1. check the title (0400486024DEC000) and main's build id (game 1.1.1)
2. exefs the five ExeFS files, every module's segments unpacked (ZBIC through the player's own `zbic` package)
and stored plainly, as the runtime's loader reads them -> <out>/exefs
@@ -16,15 +17,28 @@ Steps, each skipped when a run before finished it for the same input (state in <
6. shaders every shader in the game files (tools/prepare_shaders.py, default bind state) translated, and each one's
GPU driver build done once, so play does not stop to compile them (`ocarina.exe --prepare-shaders`)
The exefs and romfs steps depend only on the game files; lift and compile depend on this kit's code too. A newer kit
therefore lifts and compiles again but keeps the unpacked game files, and --rebuild needs no game files at all. The
compile step keeps every object with the dependency list clang wrote for it (build-cache/obj/objects.json) and
compiles a file again only when it, a header it includes, the flags or the compiler changed. The new ocarina.exe
replaces the old one only once it links, so a failed build leaves the last good one in place.
Nothing is written outside --out. The game folder then holds only what this PC built from the player's files: never
share it. Needs Python 3.10+ with numpy, lz4, zstandard and zbic (pip), and MSYS2 UCRT64's clang and libraries
(README-BuildKit.md lists the commands). Delete <out>/build-cache afterwards to free about 5 GB.
(README-BuildKit.md lists the commands). Delete <out>/build-cache afterwards to free about 5 GB; the next build then
compiles everything again.
--events adds one machine-readable line per step and progress change to stdout, each starting with "@@" and followed
by a JSON object (launcher/build.py reads them): {"ev": "step", "step": ..., "state": "start" | "done" | "skip"},
{"ev": "progress", "step": ..., "done": n, "total": n, "unit": ...}, {"ev": "phase", ...}, {"ev": "error", "message":
...} and {"ev": "end", ...}.
"""
import argparse
import concurrent.futures
import hashlib
import json
import os
import re
import shutil
import struct
import subprocess
@@ -41,7 +55,8 @@ MAIN_BUILD_ID = 'e0f3ccfaf50065ae4ea01eb5aeb13b1a' # main of game 1.1.1
EXEFS_FILES = ('main', 'main.npdm', 'sdk', 'subsdk0', 'rtld')
MODULES = ('main', 'sdk', 'subsdk0', 'rtld')
ALIGN = 16
STATE_VERSION = 1
STATE_VERSION = 2
STEPS = ('exefs', 'romfs', 'lift', 'compile', 'shaders')
# Bring-up fixes in the lifted SDK, each anchored on exact generated text that must occur once.
LIFT_FIXES = [
@@ -61,17 +76,62 @@ LIFT_FIXES = [
LINK_LIBS = ['-lvulkan-1', '-lSDL3', '-lopus', '-lvpx', '-l:libdav1d.a', '-lzstd', '-l:libatomic.a', '-lbcrypt',
'-limm32', '-luuid', '-lole32', '-luser32', '-lshell32', '-lgdi32', '-ladvapi32', '-lwinmm']
SYSTEM_DLLS_SKIP = {'vulkan-1.dll'} # the graphics driver's loader, never a copy
EXE_NAME = 'ocarina.exe'
# Runs a script the way `python script.py` does, with the script's folder first on sys.path. The launcher's
# embeddable Python leaves the script folder off sys.path (its ._pth file), so every script this build starts goes
# through this.
RUN_SCRIPT = ('import os, runpy, sys; s = sys.argv[1]; sys.argv = sys.argv[1:]; '
'sys.path.insert(0, os.path.dirname(os.path.abspath(s))); runpy.run_path(s, run_name="__main__")')
EVENTS = False
def say(msg):
print(msg, flush=True)
def emit(ev, **fields):
"""One "@@{json}" line for the launcher (--events); nothing otherwise."""
if EVENTS:
print('@@' + json.dumps({'ev': ev, **fields}, ensure_ascii=False), flush=True)
def fail(msg):
emit('error', message=msg)
print(f'\nerror: {msg}', file=sys.stderr, flush=True)
sys.exit(1)
def py_cmd(script, *args):
"""The command line that runs a Python script of this kit with this interpreter (see RUN_SCRIPT)."""
flags = ['-I'] if sys.flags.isolated else []
return [sys.executable, *flags, '-c', RUN_SCRIPT, str(script), *map(str, args)]
def sha256_bytes(data):
return hashlib.sha256(data).hexdigest()
def sha256_file(path):
h = hashlib.sha256()
with open(path, 'rb') as f:
while b := f.read(8 << 20):
h.update(b)
return h.hexdigest()
def tree_digest(root, patterns):
"""sha256 over the relative names and contents of the files under root matching the glob patterns."""
h = hashlib.sha256()
files = sorted({p for pat in patterns for p in Path(root).glob(pat) if p.is_file() and '__pycache__' not in p.parts})
for p in files:
h.update(p.relative_to(root).as_posix().encode() + b'\0')
h.update(p.read_bytes())
h.update(b'\0')
return h.hexdigest()
# ---------------------------------------------------------------- input
class Source:
@@ -137,6 +197,27 @@ class Source:
yield b
class BuiltSource:
"""--rebuild: the ExeFS and RomFS an earlier run unpacked into the game folder, under that run's input identity.
The ExeFS modules are stored plainly there but keep their headers, so check() reads the same build id."""
def __init__(self, out, state):
self.out = out
if state.get('version') != STATE_VERSION or not state.get('input'):
fail(f'{out} holds no finished build of this kit to rebuild from: give --nup, or --exefs and --romfs')
for step in ('exefs', 'romfs'):
if step not in state.get('steps', {}):
fail(f'the last build in {out} did not finish its {step} step: give the game files again')
missing = [n for n in EXEFS_FILES if not (out / 'exefs' / n).is_file()]
missing += [f'romfs/{n}' for n in ('romfs.bin', 'index.json') if not (out / 'romfs' / n).is_file()]
if missing:
fail(f'{out} lacks {", ".join(missing)}: give the game files again')
self.identity = state['input']
def read_exefs(self, name):
return (self.out / 'exefs' / name).read_bytes()
def check(src):
main = src.read_exefs('main')
if main[:4] != b'NSO0':
@@ -182,11 +263,16 @@ def flat_nso(raw, name):
def step_exefs(src, out):
dst = out / 'exefs'
dst.mkdir(parents=True, exist_ok=True)
for name in EXEFS_FILES:
for i, name in enumerate(EXEFS_FILES):
raw = src.read_exefs(name)
data = flat_nso(raw, name) if name in MODULES else raw
(dst / name).write_bytes(data)
say(f' {name:9} {len(raw) / 1e6:6.1f} MB -> {len(data) / 1e6:6.1f} MB')
emit('progress', step='exefs', done=i + 1, total=len(EXEFS_FILES), unit='files')
def exefs_digest(out):
return tree_digest(out / 'exefs', EXEFS_FILES)
# ---------------------------------------------------------------- romfs
@@ -202,6 +288,7 @@ def step_romfs(src, out, jobs):
say(f' {len(files)} files, {total / 1e9:.1f} GB')
entries, h, pos, done, t0 = [], hashlib.sha256(), 0, 0, time.time()
part = dst / 'romfs.bin.part'
emit('progress', step='romfs', done=0, total=total, unit='bytes')
def load(e): # small files are read whole on a worker; big ones are streamed by the writer
rel, ref = e
@@ -224,6 +311,7 @@ def step_romfs(src, out, jobs):
h.update(chunk)
pos += size
done += size
emit('progress', step='romfs', done=done, total=total, unit='bytes')
if start // batch % 40 == 0:
say(f' {done / 1e9:5.1f} of {total / 1e9:.1f} GB, {time.time() - t0:.0f} s')
part.replace(dst / 'romfs.bin')
@@ -234,13 +322,31 @@ def step_romfs(src, out, jobs):
# ---------------------------------------------------------------- lift
EMITTED = re.compile(r'emitted (\d+)/(\d+) chunks')
def lift_digest():
"""The lifter's code, the patch table and the bring-up fixes: what the lifted C depends on besides the ExeFS."""
h = hashlib.sha256(tree_digest(KIT / 'recomp', ('*.py', 'patches-v111.txt')).encode())
h.update(repr(LIFT_FIXES).encode())
return h.hexdigest()
def step_lift(out, cache):
"""gen.py into build-cache/lift. The folder is kept: gen.py rewrites only the chunks whose text changed and
removes the ones it no longer writes."""
lift = cache / 'lift'
if lift.exists():
shutil.rmtree(lift)
cmd = [sys.executable, str(KIT / 'recomp' / 'gen.py'), '--exefs', str(out / 'exefs'), '--out', str(lift),
'--patches', str(KIT / 'recomp' / 'patches-v111.txt')]
if subprocess.run(cmd).returncode:
cmd = py_cmd(KIT / 'recomp' / 'gen.py', '--exefs', out / 'exefs', '--out', lift,
'--patches', KIT / 'recomp' / 'patches-v111.txt')
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, encoding='utf-8',
errors='replace', bufsize=1)
for line in p.stdout:
line = line.rstrip('\n')
say(f' {line}')
m = EMITTED.search(line)
if m:
emit('progress', step='lift', done=int(m.group(1)), total=int(m.group(2)), unit='chunks')
if p.wait():
fail('the code generator failed (see above)')
for mod, old, new in LIFT_FIXES:
hits = [p for p in sorted((lift / mod).glob('*.c')) if old in p.read_text(encoding='utf-8')]
@@ -252,14 +358,6 @@ def step_lift(out, cache):
# ---------------------------------------------------------------- compile
def sha256_file(path):
h = hashlib.sha256()
with open(path, 'rb') as f:
while b := f.read(8 << 20):
h.update(b)
return h.hexdigest()
def modules_c(lift, out, gen):
layout = json.loads((lift / 'layout.json').read_text(encoding='utf-8'))
L = ['#include "dispatch.h"', '#include "host.h"', '#include "layout.h"', '']
@@ -286,12 +384,17 @@ def modules_c(lift, out, gen):
'const int rt_nvn_api_version[2] = {231, 40}; /* NVN2: main asks for major 231, minor 40 or later */',
f'const char rt_port_version[] = {json.dumps(kit_version())};']
p = gen / 'modules.c'
p.write_text('\n'.join(L) + '\n', encoding='utf-8')
(gen / 'shim.c').write_text('/* Defined by the SDL window path only; this build uses the Win32 one. */\n'
'void menu_sdl_event(void *e) { (void)e; }\n', encoding='utf-8')
write_if_changed(p, '\n'.join(L) + '\n')
write_if_changed(gen / 'shim.c', '/* Defined by the SDL window path only; this build uses the Win32 one. */\n'
'void menu_sdl_event(void *e) { (void)e; }\n')
return p
def write_if_changed(path, text):
if not path.is_file() or path.read_text(encoding='utf-8') != text:
path.write_text(text, encoding='utf-8')
def kit_version():
v = KIT / 'VERSION'
return v.read_text(encoding='utf-8').strip() if v.is_file() else '0.0.1-ocarina'
@@ -323,6 +426,23 @@ def pe_imports(path):
return names
def copy_if_changed(src, dst):
"""Copy src over dst unless dst already has its size and modification time. A file the running game holds open
cannot be replaced on Windows: that stops the build with a message instead of a traceback."""
s = os.stat(src)
try:
d = os.stat(dst)
if d.st_size == s.st_size and int(d.st_mtime) == int(s.st_mtime):
return False
except FileNotFoundError:
pass
try:
shutil.copy2(src, dst)
except PermissionError:
fail(f'{dst} is in use: close the game and try again')
return True
def copy_dlls(exe, msys_bin, out):
seen, todo, copied = set(), [exe], []
while todo:
@@ -331,21 +451,136 @@ def copy_dlls(exe, msys_bin, out):
if name in seen or name in SYSTEM_DLLS_SKIP or not src.is_file():
continue
seen.add(name)
shutil.copy2(src, out / src.name)
copy_if_changed(src, out / src.name)
copied.append(src.name)
todo.append(src)
return copied
def step_compile(out, cache, msys, jobs):
lift, gen, obj = cache / 'lift', cache / 'gen', cache / 'obj'
def parse_deps(text):
"""The prerequisites of a make-style dependency file as clang -MMD writes it (the target dropped)."""
text = text.replace('\\\r\n', ' ').replace('\\\n', ' ')
tokens, cur, i = [], '', 0
while i < len(text):
c = text[i]
if c == '\\' and i + 1 < len(text) and text[i + 1] in ' #':
cur += text[i + 1]
i += 2
continue
if c == '$' and text[i + 1:i + 2] == '$':
cur += '$'
i += 2
continue
if c.isspace():
if cur:
tokens.append(cur)
cur = ''
else:
cur += c
i += 1
if cur:
tokens.append(cur)
for k, t in enumerate(tokens): # the target ends with ':' (a drive letter's colon is followed by a separator)
if t.endswith(':'):
return tokens[k + 1:]
return tokens
class ObjectCache:
"""Which objects are current: per object the key of its command and source, and the sha256 of every file its
dependency list names (build-cache/obj/objects.json)."""
def __init__(self, path):
self.path = path
try:
self.entries = json.loads(path.read_text(encoding='utf-8')) if path.is_file() else {}
except (OSError, ValueError):
self.entries = {}
self._sha = {}
def file_sha(self, p):
p = os.path.normcase(os.path.abspath(p))
if p not in self._sha:
try:
self._sha[p] = sha256_file(p)
except OSError:
self._sha[p] = None
return self._sha[p]
def current(self, obj, key):
e = self.entries.get(obj.name)
return (e is not None and e.get('key') == key and obj.is_file()
and all(self.file_sha(p) == h for p, h in e.get('deps', {}).items()))
def record(self, obj, key, dep_file, src):
try:
deps = parse_deps(Path(dep_file).read_text(encoding='utf-8', errors='replace'))
except OSError:
deps = []
deps = deps or [str(src)]
self.entries[obj.name] = {'key': key, 'deps': {p: self.file_sha(p) for p in deps}}
def forget(self, obj):
self.entries.pop(obj.name, None)
def save(self):
tmp = self.path.with_suffix('.tmp')
tmp.write_text(json.dumps(self.entries), encoding='utf-8')
tmp.replace(self.path)
def compiler_id(clang, env):
p = subprocess.run([str(clang), '--version'], capture_output=True, text=True, env=env)
st = os.stat(clang)
return sha256_bytes(f'{p.stdout}\n{st.st_size}\n{int(st.st_mtime)}'.encode())
def compile_all(srcs, cmd_for, cache, jobs, env, progress=None, salt=''):
"""Compile each (source, object) whose object is not current in cache. cmd_for(src, dst) -> argv without the
dependency-file options; salt identifies the compiler. Returns (compiled, reused, failures) with failures as
(src, stderr tail)."""
todo, keys = [], {}
for src, dst in srcs:
cmd = cmd_for(src, dst)
flags = [c for c in cmd if c not in (str(src), str(dst))]
keys[dst] = sha256_bytes(json.dumps([salt, flags, cache.file_sha(src)]).encode())
if not cache.current(dst, keys[dst]):
todo.append((src, dst))
reused = len(srcs) - len(todo)
def cc(job):
src, dst = job
dep = dst.with_suffix('.d')
cmd = cmd_for(src, dst)
cmd[1:1] = ['-MMD', '-MF', str(dep)]
p = subprocess.run(cmd, capture_output=True, text=True, env=env, errors='replace')
return src, dst, dep, p.returncode, p.stderr[-1500:]
done, bad = reused, []
if progress:
progress(done, len(srcs))
with concurrent.futures.ThreadPoolExecutor(max(1, jobs)) as ex:
for src, dst, dep, rc, err in ex.map(cc, todo):
done += 1
if rc:
bad.append((src, err))
cache.forget(dst)
else:
cache.record(dst, keys[dst], dep, src)
if progress:
progress(done, len(srcs))
cache.save()
return len(todo), reused, bad
def step_compile(out, cache_dir, msys, jobs):
lift, gen, obj = cache_dir / 'lift', cache_dir / 'gen', cache_dir / 'obj'
gen.mkdir(parents=True, exist_ok=True)
obj.mkdir(parents=True, exist_ok=True)
rt = KIT / 'runtime'
clang, clangxx = msys / 'bin' / 'clang.exe', msys / 'bin' / 'clang++.exe'
env = dict(os.environ, PATH=str(msys / 'bin') + os.pathsep + os.environ.get('PATH', ''))
if subprocess.run([sys.executable, str(KIT / 'tools' / 'romfs_index.py'), '--romfs', str(out / 'romfs'),
'--out', str(gen)]).returncode:
if subprocess.run(py_cmd(KIT / 'tools' / 'romfs_index.py', '--romfs', out / 'romfs', '--out', gen)).returncode:
fail('the RomFS index could not be written')
srcs = [(p, obj / f'{mod}_{p.stem}.o') for mod in ('main', 'sdk', 'subsdk0') for p in sorted((lift / mod).glob('*.c'))]
for pat in ('*.c', '*.cpp', 'platform/*.c', 'ui/*.cpp', 'third_party/imgui/*.cpp', 'third_party/imgui/backends/*.cpp'):
@@ -356,53 +591,74 @@ def step_compile(out, cache, msys, jobs):
srcs += [(modules_c(lift, out, gen), obj / 'modules.o'), (gen / 'shim.c', obj / 'shim.o')]
inc = ['-I', str(rt), '-I', str(lift), '-I', str(gen), '-I', str(rt / 'third_party' / 'imgui'),
'-I', str(msys / 'include' / 'vma')]
cid = compiler_id(clang, env)
def cc(job):
src, dst = job
def cmd_for(src, dst):
cpp = src.suffix in ('.cpp', '.cxx')
# the lifted code at -O1 (its size makes -O2 slow to build), the runtime at -O2; neither may assume strict aliasing:
# the lifted code reaches guest memory through casts of one address to several types (tools/build_game.py's flags)
guest = dst.name.startswith(('main_', 'sdk_', 'subsdk0_'))
cmd = [str(clangxx if cpp else clang), '-c', '-std=gnu++17' if cpp else '-std=gnu11', '-O1' if guest else '-O2',
'-fno-strict-aliasing', '-w', *inc, '-o', str(dst), str(src)]
p = subprocess.run(cmd, capture_output=True, text=True, env=env)
return src, p.returncode, p.stderr[-1500:]
return [str(clangxx if cpp else clang), '-c', '-std=gnu++17' if cpp else '-std=gnu11', '-O1' if guest else '-O2',
'-fno-strict-aliasing', '-w', *inc, '-o', str(dst), str(src)]
t0 = time.time()
last = [0.0]
def progress(done, total):
now = time.time()
if done == total or now - last[0] >= 0.25:
last[0] = now
emit('progress', step='compile', done=done, total=total, unit='files')
if done % 50 == 0 or done == total:
say(f' compiled {done} of {total}, {now - t0:.0f} s')
say(f' {len(srcs)} files on {jobs} threads')
t0, done, bad = time.time(), 0, []
with concurrent.futures.ThreadPoolExecutor(jobs) as ex:
for src, rc, err in ex.map(cc, srcs):
done += 1
if rc:
bad.append((src, err))
if done % 50 == 0 or done == len(srcs):
say(f' compiled {done} of {len(srcs)}, {time.time() - t0:.0f} s')
cache = ObjectCache(obj / 'objects.json')
compiled, reused, bad = compile_all(srcs, cmd_for, cache, jobs, env, progress, salt=cid)
say(f' {compiled} compiled, {reused} unchanged since the last build')
if bad:
for src, err in bad[:3]:
print(f'--- {src}\n{err}', file=sys.stderr)
fail(f'{len(bad)} files did not compile')
# the manifest (per-monitor DPI awareness: the picture at the monitor's real size, not one Windows stretches) and the
# icon, runtime/nx2navi.rc through MSYS2's windres
rco = obj / 'nx2navi_rc.o'
r = subprocess.run([str(msys / 'bin' / 'windres.exe'), '-I', str(rt), '-O', 'coff', '-i', str(rt / 'nx2navi.rc'),
'-o', str(rco)], capture_output=True, text=True, env=env)
'-o', str(rco)], capture_output=True, text=True, env=env, errors='replace')
if r.returncode:
fail(f'windres failed on runtime/nx2navi.rc: {r.stderr.strip()[:400]}')
rsp = cache / 'link.rsp'
rsp.write_text('\n'.join('"%s"' % o.as_posix() for o in [o for _, o in srcs] + [rco]), encoding='utf-8')
exe = out / 'ocarina.exe'
say(' linking')
p = subprocess.run([str(clangxx), '-o', str(exe), '@' + str(rsp), '-L', str(msys / 'lib'), *LINK_LIBS],
capture_output=True, text=True, env=env)
if p.returncode:
print(p.stderr[-4000:], file=sys.stderr)
fail('the link failed')
shutil.copy2(gen / 'romfs-index.bin', out / 'romfs-index.bin')
objs = [o for _, o in srcs] + [rco]
exe = out / EXE_NAME
rsp = cache_dir / 'link.rsp'
rsp.write_text('\n'.join('"%s"' % o.as_posix() for o in objs), encoding='utf-8')
link_key = sha256_bytes(json.dumps([LINK_LIBS, cid] + [[o.name, o.stat().st_size, o.stat().st_mtime_ns]
for o in objs]).encode())
state_link = cache_dir / 'link.key'
if exe.is_file() and state_link.is_file() and state_link.read_text(encoding='utf-8') == link_key:
say(' ocarina.exe is current, not linked again')
else:
emit('phase', step='compile', phase='link')
say(' linking')
tmp = out / (EXE_NAME + '.new')
p = subprocess.run([str(clangxx), '-o', str(tmp), '@' + str(rsp), '-L', str(msys / 'lib'), *LINK_LIBS],
capture_output=True, text=True, env=env, errors='replace')
if p.returncode:
print(p.stderr[-4000:], file=sys.stderr)
tmp.unlink(missing_ok=True)
fail('the link failed')
try:
os.replace(tmp, exe)
except PermissionError:
tmp.unlink(missing_ok=True)
fail(f'{exe} is in use: close the game and try again')
state_link.write_text(link_key, encoding='utf-8')
copy_if_changed(gen / 'romfs-index.bin', out / 'romfs-index.bin')
for name in ('nx2navi_xlate.exe', 'nx2navi_xlate.id'):
shutil.copy2(KIT / 'bin' / name, out / name)
copy_if_changed(KIT / 'bin' / name, out / name)
if (KIT / 'bin' / 'amd_fidelityfx_vk.dll').is_file(): # FSR 3.1 (runtime/vk_ffx.cpp)
shutil.copy2(KIT / 'bin' / 'amd_fidelityfx_vk.dll', out / 'amd_fidelityfx_vk.dll')
say(f' ocarina.exe {exe.stat().st_size / 1e6:.0f} MB; beside it: {", ".join(copy_dlls(exe, msys / "bin", out))}')
copy_if_changed(KIT / 'bin' / 'amd_fidelityfx_vk.dll', out / 'amd_fidelityfx_vk.dll')
say(f' {EXE_NAME} {exe.stat().st_size / 1e6:.0f} MB; beside it: {", ".join(copy_dlls(exe, msys / "bin", out))}')
(out / 'profile').mkdir(exist_ok=True)
@@ -432,56 +688,111 @@ def keep_awake():
pass
def load_state(path):
try:
state = json.loads(path.read_text(encoding='utf-8')) if path.is_file() else {}
except (OSError, ValueError):
state = {}
if state.get('version') == 1: # the first kit: a list of finished steps, all tied to one kit version
done = state.get('done', [])
state = {'version': STATE_VERSION, 'input': state.get('input'), 'steps': {}}
for step in ('exefs', 'romfs'):
if step in done:
state['steps'][step] = step_key(step, state['input'])
return state
def step_key(step, identity, out=None):
"""What a step's output depends on. exefs and romfs: the game files only; lift: the unpacked ExeFS and the
lifter; compile: the lift, the runtime sources and this script."""
if step in ('exefs', 'romfs'):
return f'{step}-1|{identity}'
lift = f'lift|{exefs_digest(out)}|{lift_digest()}'
if step == 'lift':
return lift
compile_key = f'compile|{lift}|{tree_digest(KIT / "runtime", ("**/*",))}|{sha256_file(Path(__file__))}|{kit_version()}'
if step == 'compile':
return compile_key
# shaders: the built game and the translator that prepared them
return f'shaders-1|{compile_key}|{sha256_file(KIT / "tools" / "prepare_shaders.py")}|{sha256_file(KIT / "bin" / "nx2navi_xlate.id")}'
def main():
global EVENTS
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument('--nup', help='the game as one .nup file')
ap.add_argument('--exefs', help='the game\'s exefs folder (with --romfs, instead of --nup)')
ap.add_argument('--romfs', help='the game\'s romfs folder')
ap.add_argument('--rebuild', action='store_true',
help='build again from the game files the last run unpacked into --out (after a kit update)')
ap.add_argument('--out', help='the folder for the finished game')
ap.add_argument('--check', action='store_true', help='only check the game\'s version; builds nothing')
ap.add_argument('--msys', default=r'C:\msys64\ucrt64', help='MSYS2 UCRT64 folder (default C:\\msys64\\ucrt64)')
ap.add_argument('--jobs', type=int, default=os.cpu_count() or 4, help='parallel compiles')
ap.add_argument('--events', action='store_true', help='machine-readable progress lines on stdout ("@@" + JSON)')
ap.add_argument('--until', choices=STEPS, help=argparse.SUPPRESS) # tests: stop after this step
a = ap.parse_args()
if bool(a.nup) == bool(a.exefs or a.romfs) or (not a.nup and not (a.exefs and a.romfs)):
ap.error('give --nup, or both --exefs and --romfs')
src = Source(a)
EVENTS = a.events
given = bool(a.nup) + bool(a.exefs or a.romfs) + bool(a.rebuild)
if given != 1 or (not a.nup and not a.rebuild and not (a.exefs and a.romfs)):
ap.error('give --nup, both --exefs and --romfs, or --rebuild')
if a.rebuild and not a.out:
ap.error('--rebuild needs --out')
out = Path(a.out).resolve() if a.out else None
cache = out / 'build-cache' if out else None
state_path = cache / 'state.json' if cache else None
state = load_state(state_path) if state_path else {}
emit('step', step='check', state='start')
src = BuiltSource(out, state) if a.rebuild else Source(a)
check(src)
say('check: Ocarina of Time 1.1.1 (build e0f3ccfa)')
emit('check', version='1.1.1', build=MAIN_BUILD_ID[:8])
emit('step', step='check', state='done')
if a.check:
return
if not a.out:
if not out:
ap.error('--out is needed to build')
msys = Path(a.msys)
for need in ('bin/clang.exe', 'bin/clang++.exe', 'bin/SDL3.dll', 'lib/libSDL3.dll.a', 'include/vma'):
if not (msys / need).exists():
fail(f'{msys / need} is missing: install MSYS2 and the packages README-BuildKit.md lists')
for mod in ('numpy', 'lz4', 'zbic'):
for mod in ('numpy', 'lz4') + (() if a.rebuild else ('zbic',)):
try:
__import__(mod)
except ImportError:
fail(f'the Python package {mod} is missing: pip install numpy lz4 zstandard zbic')
out = Path(a.out).resolve()
cache = out / 'build-cache'
cache.mkdir(parents=True, exist_ok=True)
state_path = cache / 'state.json'
state = json.loads(state_path.read_text(encoding='utf-8')) if state_path.is_file() else {}
if state.get('input') != src.identity or state.get('version') != STATE_VERSION or state.get('kit') != kit_version():
state = {'input': src.identity, 'version': STATE_VERSION, 'kit': kit_version(), 'done': []}
if state.get('version') != STATE_VERSION or state.get('input') != src.identity:
state = {'version': STATE_VERSION, 'input': src.identity, 'steps': {}}
state['kit'] = kit_version()
keep_awake()
t0 = time.time()
for name, run in (('exefs', lambda: step_exefs(src, out)), ('romfs', lambda: step_romfs(src, out, min(a.jobs, 8))),
('lift', lambda: step_lift(out, cache)), ('compile', lambda: step_compile(out, cache, msys, a.jobs)),
('shaders', lambda: step_shaders(out))):
if name in state['done']:
runs = {'exefs': lambda: step_exefs(src, out), 'romfs': lambda: step_romfs(src, out, min(a.jobs, 8)),
'lift': lambda: step_lift(out, cache), 'compile': lambda: step_compile(out, cache, msys, a.jobs),
'shaders': lambda: step_shaders(out)}
for name in STEPS:
key = step_key(name, src.identity, out)
if state['steps'].get(name) == key:
say(f'{name}: done before')
continue
say(f'{name}:')
t = time.time()
run()
state['done'].append(name)
state_path.write_text(json.dumps(state, indent=1), encoding='utf-8')
say(f'{name}: {time.time() - t:.0f} s')
say(f'\nfinished in {(time.time() - t0) / 60:.1f} min: {out / "ocarina.exe"}')
emit('step', step=name, state='skip')
else:
# a step that runs makes every step after it run too: their keys hash its output, not its key
for later in STEPS[STEPS.index(name):]:
state['steps'].pop(later, None)
say(f'{name}:')
emit('step', step=name, state='start')
t = time.time()
runs[name]()
state['steps'][name] = step_key(name, src.identity, out)
state_path.write_text(json.dumps(state, indent=1), encoding='utf-8')
say(f'{name}: {time.time() - t:.0f} s')
emit('step', step=name, state='done', seconds=round(time.time() - t, 1))
if name == a.until:
break
state_path.write_text(json.dumps(state, indent=1), encoding='utf-8')
say(f'\nfinished in {(time.time() - t0) / 60:.1f} min: {out / EXE_NAME}')
emit('end', exe=str(out / EXE_NAME), seconds=round(time.time() - t0, 1), kit=kit_version())
if __name__ == '__main__':

301
tools/make_launcher.py Normal file
View File

@@ -0,0 +1,301 @@
"""Build a release of the NX2navi launcher (docs/launcher.md, "Releases").
python tools/make_launcher.py --xlate build/xlate [--ffx amd_fidelityfx_vk.dll] [--out build/launcher-release]
[--feed owner/name]
[--notes notes.md] [--publish]
Needs Python 3.14 (tarfile and zstd for the MSYS2 packages), uv on PATH, network access on the first run, and
either Windows (msiexec, and the C compiler of MSYS2 it downloads itself) or Linux with clang, lld and llvm-windres
(it cross-compiles against the same MSYS2 packages) plus msitools. Downloads are cached in toolchain/launcher-dl/ and
checked against the sha256 in PINS.
The release folder (--out) holds:
NX2navi/ the launcher folder exactly as NX2navi-Setup.exe installs it:
NX2navi.exe launcher/native/nx2navi.c with the Triforce icon (launcher/ui/art.py)
python/ python.org's embeddable Python 3.14.9, tkinter and Tcl/Tk 9.0 from its tcltk.msi,
the packages of tools/requirements-launcher.txt, runtime-id.txt
app/current.txt, app/<version>/ launcher/ (without native/), runtime/, recomp/, the tools make_game.py uses,
bin/nx2navi_xlate.exe and .id (and amd_fidelityfx_vk.dll with --ffx), toolchain.lock.json (the MSYS2 packages of today,
pinned), feed.json, VERSION, the licence files
NX2navi-Setup-<version>.exe launcher/native/setup.c with the folder above appended as a zip: the one file
players download
NX2navi-app-<version>.zip app/<version> alone: what the launcher downloads to update itself
NX2navi-runtime-<id>.zip python/ alone: downloaded only when a release changes the Python
NX2navi-update.json the manifest the launcher reads (launcher/feed.py)
--publish uploads the four files as GitHub release v<version> of the --feed repository with the gh CLI (a draft
first is wiser: add --draft). check_content runs over everything before it is packed: the release holds no game data.
"""
import argparse
import hashlib
import json
import os
import shutil
import subprocess
import sys
import time
import zipfile
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT))
sys.path.insert(0, str(ROOT / 'tools'))
import check_content # noqa: E402
from launcher import toolchain # noqa: E402
from launcher.paths import Layout # noqa: E402
MAIN_ROOT = (ROOT / 'build').resolve().parent # build/ and toolchain/ are junctions in a worktree
DL = MAIN_ROOT / 'toolchain' / 'launcher-dl'
WORK = MAIN_ROOT / 'toolchain' / 'launcher-work'
PY_VERSION = '3.14.9'
PINS = {
'python-embed': (f'https://www.python.org/ftp/python/{PY_VERSION}/python-{PY_VERSION}-embed-amd64.zip',
'3c01d7cfc666f746a6f064f60fb2156ab0ec464d3d738ab796371a46a9bb24a2'),
# the installer's own Tcl/Tk and tkinter component, published beside it
'tcltk': (f'https://www.python.org/ftp/python/{PY_VERSION}/amd64/tcltk.msi',
'ac87609db3f715404dd41a989c2f1706ea1512f3d86217f938d11c63af8eb505'),
}
TOOLS = ('make_game.py', 'nup.py', 'romfs_index.py', 'check_content.py', 'hostmem.py', 'prepare_shaders.py',
'shader_census.py', 'shader_list.py', 'zstd_compat.py')
TOP = ('LICENSE', 'LICENSE-ADDITIONS', 'THIRD_PARTY_NOTICES.md', 'nx2navi.ini.example')
SKIP_DIRS = {'__pycache__', '.pytest_cache', 'native'}
SKIP_SUFFIXES = {'.o', '.obj', '.pyc', '.exe', '.dll', '.a', '.lib'}
TCLTK_FILES = ('_tkinter.pyd', 'tcl90.dll', 'tcl9tk90.dll', 'zlib1.dll', 'libtommath.dll')
DEFAULT_FEED = 'tangerineturtle1/NX2navi'
def say(msg):
print(msg, flush=True)
def sha256_file(path):
h = hashlib.sha256()
with open(path, 'rb') as f:
while b := f.read(1 << 20):
h.update(b)
return h.hexdigest()
def fetch(name):
url, want = PINS[name]
dest = DL / url.rsplit('/', 1)[1]
if dest.is_file() and sha256_file(dest) == want:
return dest
from launcher import net
say(f'download {url}')
net.download(url, dest, sha256=want)
return dest
def run(cmd, **kw):
say(' ' + ' '.join(str(c) for c in cmd))
subprocess.run([str(c) for c in cmd], check=True, **kw)
def copy_tree(src, dst, skip_dirs=SKIP_DIRS):
for p in sorted(src.rglob('*')):
rel = p.relative_to(src)
if any(part in skip_dirs for part in rel.parts) or p.is_dir() or p.suffix.lower() in SKIP_SUFFIXES:
continue
(dst / rel).parent.mkdir(parents=True, exist_ok=True)
shutil.copy2(p, dst / rel)
def version():
v = (ROOT / 'VERSION').read_text(encoding='utf-8').strip()
r = subprocess.run(['git', '-C', str(ROOT), 'rev-parse', '--short', 'HEAD'], capture_output=True, text=True)
return v, f'{v}+{r.stdout.strip()}' if r.returncode == 0 and r.stdout.strip() else v
# ---------------------------------------------------------------- python/
def tcltk_files():
dest = WORK / 'tcltk'
if (dest / 'DLLs' / '_tkinter.pyd').is_file():
return dest
msi = fetch('tcltk')
shutil.rmtree(dest, ignore_errors=True)
dest.mkdir(parents=True)
if os.name == 'nt': # an administrative image: copies the files, installs nothing
run(['msiexec', '/a', msi, '/qn', f'TARGETDIR={dest}'])
else:
run(['msiextract', '-C', dest, msi], stdout=subprocess.DEVNULL)
if not (dest / 'DLLs' / '_tkinter.pyd').is_file():
sys.exit(f'{dest}: no _tkinter.pyd in {msi.name}')
return dest
def runtime_id():
req = (ROOT / 'tools' / 'requirements-launcher.txt').read_bytes()
return f'py{PY_VERSION}-{hashlib.sha256(req + PINS["tcltk"][1].encode()).hexdigest()[:8]}'
def build_python(py):
with zipfile.ZipFile(fetch('python-embed')) as z:
z.extractall(py)
# The embeddable Python's sys.path is exactly this list; ../pydeps holds zbic (launcher/toolchain.py).
(py / 'python314._pth').write_text('python314.zip\n.\nLib/site-packages\n../pydeps\n', encoding='utf-8')
env = dict(os.environ, UV_CACHE_DIR=str(MAIN_ROOT / 'toolchain' / 'uv-cache'))
run(['uv', 'pip', 'install', '--target', py / 'Lib' / 'site-packages', '--python', sys.executable,
'--python-platform', 'windows', '--python-version', '3.14', '--only-binary', ':all:', '-q',
'-r', ROOT / 'tools' / 'requirements-launcher.txt'], env=env)
shutil.rmtree(py / 'Lib' / 'site-packages' / 'bin', ignore_errors=True)
tk = tcltk_files()
for dll in TCLTK_FILES:
if (tk / 'DLLs' / dll).is_file():
shutil.copy2(tk / 'DLLs' / dll, py / dll)
copy_tree(tk / 'Lib' / 'tkinter', py / 'tkinter', skip_dirs={'__pycache__', 'test'})
copy_tree(tk / 'tcl', py / 'tcl', skip_dirs={'__pycache__'})
(py / 'runtime-id.txt').write_text(runtime_id() + '\n', encoding='utf-8')
# ---------------------------------------------------------------- app/<version>/
def build_app(app, ver, xlate, feed_repo, lock, ffx=None):
copy_tree(ROOT / 'launcher', app / 'launcher')
for d in ('runtime', 'recomp'):
copy_tree(ROOT / d, app / d)
(app / 'tools').mkdir(parents=True)
for t in TOOLS:
shutil.copy2(ROOT / 'tools' / t, app / 'tools' / t)
for t in TOP:
if (ROOT / t).is_file():
shutil.copy2(ROOT / t, app / t)
(app / 'VERSION').write_text(ver + '\n', encoding='utf-8')
(app / 'bin').mkdir()
for n in ('nx2navi_xlate.exe', 'nx2navi_xlate.id'):
if not (xlate / n).is_file():
sys.exit(f'{xlate / n} is missing: build the shader translator first (tools/build_xlate.py)')
shutil.copy2(xlate / n, app / 'bin' / n)
if ffx: # FSR 3.1 loads it from beside the game (runtime/vk_ffx.cpp); make_game.py copies it there
shutil.copy2(ffx, app / 'bin' / 'amd_fidelityfx_vk.dll')
(app / 'toolchain.lock.json').write_text(json.dumps(lock, indent=1), encoding='utf-8')
(app / 'feed.json').write_text(json.dumps({'type': 'github', 'repo': feed_repo}, indent=1), encoding='utf-8')
# ---------------------------------------------------------------- the native programs
def native_compiler(lock):
"""(cc argv prefix, windres argv prefix) for x86_64 Windows: MSYS2's own clang on Windows, else the host's clang
against the same MSYS2 packages as a sysroot."""
lay = Layout(root=MAIN_ROOT / 'toolchain' / 'launcher-msys')
if not toolchain.is_current(lay, lock):
say('MSYS2 packages for the native programs')
toolchain.install(lay, lock, log=say)
sysroot = lay.msys
if os.name == 'nt':
return [str(sysroot / 'bin' / 'clang.exe')], [str(sysroot / 'bin' / 'windres.exe')]
for tool in ('clang', 'ld.lld'):
if not shutil.which(tool):
sys.exit(f'{tool} is needed to cross-compile the launcher programs')
windres = shutil.which('llvm-windres') or shutil.which('llvm-windres-18') or '/usr/lib/llvm-18/bin/llvm-windres'
if not Path(windres).exists():
sys.exit('llvm-windres is needed to compile the launcher programs\' resources')
return (['clang', '--target=x86_64-w64-mingw32', f'--sysroot={sysroot}', '-fuse-ld=lld'],
[windres, '--target=x86_64-w64-mingw32'])
def build_native(work, cc, windres):
from launcher.ui import art
work.mkdir(parents=True, exist_ok=True)
(work / 'nx2navi.ico').write_bytes(art.triforce_ico())
shutil.copy2(ROOT / 'launcher' / 'native' / 'app.manifest', work / 'app.manifest')
res = work / 'res.o'
run([*windres, '-I', work, ROOT / 'launcher' / 'native' / 'res.rc', '-O', 'coff', '-o', res])
flags = ['-municode', '-mwindows', '-O2', '-Wall', '-static', '-s']
stub = work / 'NX2navi.exe'
setup = work / 'setup-stub.exe'
run([*cc, *flags, ROOT / 'launcher' / 'native' / 'nx2navi.c', res, '-o', stub, '-lshell32', '-lole32', '-luuid'])
run([*cc, *flags, ROOT / 'launcher' / 'native' / 'setup.c', res, '-o', setup, '-l:libz.a', '-lshell32',
'-lole32', '-luuid', '-ladvapi32'])
return stub, setup
# ---------------------------------------------------------------- packing
def zip_dir(src, dest, prefix=''):
with zipfile.ZipFile(dest, 'w', zipfile.ZIP_DEFLATED, compresslevel=9) as z:
for p in sorted(src.rglob('*')):
if p.is_file():
z.write(p, (Path(prefix) / p.relative_to(src)).as_posix())
return dest
def make_setup(stub, payload, dest, ver):
"""setup-stub.exe + the payload zip + the 48-byte trailer launcher/native/setup.c reads."""
data = stub.read_bytes()
off = len(data)
z = payload.read_bytes()
trailer = b'NX2NSET1' + off.to_bytes(8, 'little') + len(z).to_bytes(8, 'little') + \
ver.encode('utf-8')[:24].ljust(24, b'\0')
dest.write_bytes(data + z + trailer)
return dest
def check_folder(folder):
bad = [f'{p.relative_to(folder)}: {m}' for p in sorted(folder.rglob('*')) if p.is_file()
for m in check_content.problems_for(p, allow_large=True)]
if bad:
sys.exit('check_content refuses these release files:\n ' + '\n '.join(bad[:20]))
def entry(path):
return {'name': path.name, 'size': path.stat().st_size, 'sha256': sha256_file(path)}
def main():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument('--xlate', type=Path, required=True, help='folder with nx2navi_xlate.exe and nx2navi_xlate.id')
ap.add_argument('--ffx', type=Path, help="AMD's signed amd_fidelityfx_vk.dll (FidelityFX SDK 1.1.4, MIT) for FSR 3.1")
ap.add_argument('--out', type=Path, default=MAIN_ROOT / 'build' / 'launcher-release')
ap.add_argument('--feed', default=DEFAULT_FEED, help='the GitHub repository the launcher looks for releases in')
ap.add_argument('--notes', type=Path, help='release notes (markdown) for the manifest and the GitHub release')
ap.add_argument('--lock', type=Path, help='a toolchain.lock.json to reuse instead of resolving today\'s packages')
ap.add_argument('--version', help='the release version instead of VERSION\'s (test releases)')
ap.add_argument('--publish', action='store_true', help='upload as GitHub release v<version> with gh')
ap.add_argument('--draft', action='store_true', help='with --publish: as a draft')
a = ap.parse_args()
if sys.version_info < (3, 14):
sys.exit('Python 3.14 or later is needed (compression.zstd for the MSYS2 packages)')
ver, build = version()
if a.version:
ver, build = a.version, a.version + build[len(ver):]
out = a.out
shutil.rmtree(out, ignore_errors=True)
folder = out / 'NX2navi'
t0 = time.time()
lock = json.loads(a.lock.read_text(encoding='utf-8')) if a.lock else toolchain.make_lock()
say(f'release {build}: toolchain lock with {len(lock["packages"])} packages')
build_app(folder / 'app' / ver, ver, a.xlate, a.feed, lock, a.ffx)
(folder / 'app' / 'current.txt').write_text(ver + '\n', encoding='utf-8')
build_python(folder / 'python')
cc, windres = native_compiler(lock)
stub, setup_stub = build_native(out / 'native', cc, windres)
shutil.copy2(stub, folder / 'NX2navi.exe')
check_folder(folder)
app_zip = zip_dir(folder / 'app' / ver, out / f'NX2navi-app-{ver}.zip')
rt_zip = zip_dir(folder / 'python', out / f'NX2navi-runtime-{runtime_id()}.zip')
payload = zip_dir(folder, out / 'native' / 'payload.zip')
setup = make_setup(setup_stub, payload, out / f'NX2navi-Setup-{ver}.exe', ver)
notes = a.notes.read_text(encoding='utf-8') if a.notes else ''
manifest = {'format': 1, 'version': ver, 'build': build, 'notes': notes,
'app': entry(app_zip), 'runtime': {'id': runtime_id(), **entry(rt_zip)},
'toolchain': toolchain.lock_digest(lock), 'setup': entry(setup),
'made': time.strftime('%Y-%m-%d %H:%M')}
man = out / 'NX2navi-update.json'
man.write_text(json.dumps(manifest, indent=1), encoding='utf-8')
for p in (setup, app_zip, rt_zip, man):
say(f'{p.name}: {p.stat().st_size / 1e6:.1f} MB')
say(f'release folder {out} in {time.time() - t0:.0f} s')
if a.publish:
cmd = ['gh', 'release', 'create', f'v{ver}', setup, app_zip, rt_zip, man, '--repo', a.feed,
'--title', f'NX2navi {ver}']
cmd += ['--notes-file', a.notes] if a.notes else ['--notes', f'NX2navi {ver}']
if a.draft:
cmd.append('--draft')
run(cmd)
if __name__ == '__main__':
main()

View File

@@ -0,0 +1,6 @@
# Packages of the launcher release's bundled Python 3.14.9 (tools/make_launcher.py): what the launcher and
# tools/make_game.py import. numpy: recomp/gen.py and the window's drawing (launcher/ui/art.py). lz4: NSO segments.
# zstandard: tools/nup.py. zbic is not bundled: the launcher downloads it from PyPI on the player's PC.
numpy==2.5.4
lz4==4.4.5
zstandard==0.25.0