mirror of
https://github.com/tangerineturtle1/NX2navi.git
synced 2026-11-15 17:09:57 -04:00
Add the NX2navi launcher: a one-click setup that builds the game from the player's dump, starts it and keeps it up to date.
NX2navi-Setup.exe installs the launcher into %LOCALAPPDATA%\Programs\NX2navi without administrator rights, with shortcuts and an Apps entry. The launcher finds the player's .nup, downloads the build tools itself (the MSYS2 UCRT64 packages toolchain.lock.json pins, checked by sha256 and unpacked without pacman, and zbic from PyPI) and runs tools/make_game.py, whose steps it shows with their progress. Its window follows the game's options menu: colours sampled from it, the menu's filigree, the white glowing cursor, the Triforce in the title band and as the programs' icon, and all text in the game's Cetra Display, decoded from the player's own dump. Mouse, keyboard and an XInput pad drive it. Updates come from GitHub releases (NX2navi-update.json): the new version is downloaded, checked and staged beside the running one, the stub restarts into it and falls back to the previous version when it fails to start, and the game is rebuilt without the dump. make_game.py now keys its exefs and romfs steps on the game files alone, has --rebuild and --events, compiles only the files whose source, headers, flags or compiler changed, and replaces ocarina.exe only once the new one links. tools/make_launcher.py builds the release on Windows or Linux. Tested: tests/launcher_test.py (19 tests), the real toolchain download and install, and under Wine 11 the setup, the stub, the window on the bundled Windows Python, an update from a local feed by clicking Updates, the rollback of a version that fails to start, and the downloaded clang compiling and linking a program against the game's libraries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Merged onto the current port: make_game.py keeps the launcher's step keys, cached link and in-place swap, and its own runtime manifest (windres), AMD's FidelityFX library and the shader preparation, which becomes a step of its own ("shaders") that the launcher shows in its install, rebuild and update jobs. make_launcher.py ships the shader tools and, with --ffx, AMD's library. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# NX2navi
|
||||
|
||||
Static recompilation of *Ocarina of Time* (Switch 2, ARM64, game version 1.1.1) into a native Windows x86-64 exe with a Vulkan renderer. Design: [docs/architecture.md](docs/architecture.md). Toolchain, build commands and tests: [docs/building.md](docs/building.md).
|
||||
Static recompilation of *Ocarina of Time* (Switch 2, ARM64, game version 1.1.1) into a native Windows x86-64 exe with a Vulkan renderer. Design: [docs/architecture.md](docs/architecture.md). Toolchain, build commands and tests: [docs/building.md](docs/building.md). The players' launcher, setup and updates: [docs/launcher.md](docs/launcher.md).
|
||||
|
||||
## Game content boundary
|
||||
|
||||
|
||||
10
README.md
10
README.md
@@ -45,10 +45,12 @@ Report problems as issues, following [CONTRIBUTING.md](CONTRIBUTING.md).
|
||||
|
||||
## Installing
|
||||
|
||||
The build kit builds the game on your PC from your own dump (a `.nup` file, or ExeFS and RomFS folders) in about 10 to
|
||||
15 minutes: [docs/README-BuildKit.md](docs/README-BuildKit.md). [docs/building.md](docs/building.md) describes building
|
||||
from source. Settings are in the in-game menu and in `nx2navi.ini` ([nx2navi.ini.example](nx2navi.ini.example) lists every
|
||||
key). Controls: [docs/controls.md](docs/controls.md).
|
||||
Download `NX2navi-Setup-<version>.exe` from the releases and start it next to your `.nup` dump. The launcher it
|
||||
installs finds the dump, downloads the build tools, builds the game on your PC (10 to 25 minutes, once) and starts
|
||||
it; afterwards it keeps the port up to date and rebuilds only what changed ([docs/launcher.md](docs/launcher.md)).
|
||||
[docs/README-BuildKit.md](docs/README-BuildKit.md) describes the same build by hand, [docs/building.md](docs/building.md)
|
||||
building from source. Settings are in the in-game menu and in `nx2navi.ini` ([nx2navi.ini.example](nx2navi.ini.example)
|
||||
lists every key). Controls: [docs/controls.md](docs/controls.md).
|
||||
|
||||
Requirements: Windows 10 or later, an x86-64 CPU with SSE4.2 (AVX2/FMA machines get a faster build), a GPU with Vulkan
|
||||
1.3, `VK_EXT_external_memory_host` and `VK_EXT_mesh_shader`.
|
||||
|
||||
@@ -107,3 +107,23 @@ folder's `licenses/` and in each component's own folder.
|
||||
| MacNativeGames shader variant list | as in the installer release row above | MIT | `data/shader_list.tsv` |
|
||||
| Weeping of the Empire menu cursor: eui's focus box read through ScreenMgr, BoxCursorMgr and the screens' node lists, AnimButton's hit test, the stale-hit-pane fallback to the parts panes, the three writes of moveBoxCursor, the structure offsets (1.0.0; the ScreenMgr instance placed in 1.4.3 by us) | https://github.com/daniel-madrid-07/weeping-of-the-empire @ `df0586cdc5702ca19bbebfa59697e0b6980f6237` (`engine/runtime/nx2navi_menu_cursor.cpp`), Copyright (c) 2026 daniel-madrid-07 | MIT | `runtime/menu_cursor.c` |
|
||||
| AMD FidelityFX API headers (FSR 3.1 upscaling and frame generation) and its signed Vulkan library | https://github.com/GPUOpen-LibrariesAndSDKs/FidelityFX-SDK @ `v1.1.4` (`ffx-api/include/ffx_api/`, `PrebuiltSignedDLL/amd_fidelityfx_vk.dll`), Copyright (C) 2024 Advanced Micro Devices, Inc. | MIT | `runtime/third_party/ffx_api/` (unmodified, with `LICENSE.txt`), used by `runtime/vk_ffx.cpp`; the library ships in the build kit's `bin/` and beside the built game (never in this repository) |
|
||||
|
||||
## Distributed in the launcher release
|
||||
|
||||
`tools/make_launcher.py` assembles these into `NX2navi-Setup-<version>.exe` and the update archives of
|
||||
`docs/launcher.md`. Versions and sha256 of the downloads are pinned in that script and in
|
||||
`tools/requirements-launcher.txt`.
|
||||
|
||||
| Component | Source | License | Where in the release |
|
||||
|---|---|---|---|
|
||||
| CPython 3.14.9 embeddable package; includes `vcruntime140*.dll` (Microsoft, redistributable with Python), OpenSSL, libffi, SQLite, zlib, libmpdec | https://www.python.org/downloads/release/python-3149/ | PSF-2.0 | `python/` |
|
||||
| `tkinter` and Tcl/Tk 9.0 (`_tkinter.pyd`, `tcl90.dll`, `tcl9tk90.dll`, `zlib1.dll`, `libtommath.dll`, `tcl/`), from python.org's 3.14.9 `amd64/tcltk.msi`, unpacked, never run | https://www.python.org/ftp/python/3.14.9/amd64/ | PSF-2.0; Tcl/Tk licence (BSD-style); libtommath public domain or BSD; zlib licence | `python/` |
|
||||
| numpy 2.5.4 (with its bundled OpenBLAS), lz4 4.4.5, zstandard 0.25.0 | PyPI binary wheels, pins in `tools/requirements-launcher.txt` | BSD-3-Clause; lz4 BSD-3-Clause; zstandard BSD-3-Clause | `python/Lib/site-packages/` |
|
||||
| zlib 1.4 (MSYS2 `mingw-w64-ucrt-x86_64-zlib`), static library | https://zlib.net, Copyright (C) Jean-loup Gailly and Mark Adler | zlib | linked into `NX2navi-Setup.exe` (its unzip) |
|
||||
| mingw-w64 CRT startup code and winpthreads 14.0.0, GCC 16.2.0's `libgcc` (MSYS2 packages), static | https://www.mingw-w64.org, https://gcc.gnu.org | mingw-w64: ZPL-2.1 / MIT / public domain; libgcc: GPL-3.0 with the GCC Runtime Library Exception | linked into `NX2navi.exe` and `NX2navi-Setup.exe` |
|
||||
|
||||
Downloaded on the player's PC by the launcher, not distributed: the MSYS2 UCRT64 packages that
|
||||
`toolchain.lock.json` pins (clang and LLVM, Apache-2.0 with LLVM exception; GCC and binutils, GPL-3.0; the
|
||||
libraries of the "Compiler runtime in a source build" section and of `README-BuildKit.md`), from
|
||||
https://repo.msys2.org/mingw/ucrt64/; and zbic 1.0.1.post1 (GPL-2.0), from PyPI. The game build links them as a
|
||||
source build does.
|
||||
|
||||
@@ -4,6 +4,9 @@ This kit builds *The Legend of Zelda: Ocarina of Time* (Nintendo Switch 2, game
|
||||
program **on your PC, from your own copy of the game**. The kit contains no game code or data. Everything the build
|
||||
writes (`ocarina.exe`, the RomFS image, the shader cache, your saves) comes from your game: keep it to yourself.
|
||||
|
||||
The NX2navi launcher (`NX2navi-Setup.exe`, [launcher.md](launcher.md)) does everything below for you, the
|
||||
downloads included, and keeps the game up to date. This page is the same build by hand.
|
||||
|
||||
## What you need
|
||||
|
||||
- 64-bit Windows 10 or 11, 16 GB of RAM, a Vulkan 1.3 graphics card with mesh shaders and a current driver (tested on
|
||||
|
||||
@@ -5,8 +5,8 @@ default build targets x86-64-v3 (AVX2/FMA); a CPU without them builds with `--cp
|
||||
"Build"). Allow about 60 GB for the work folder. Extraction wants about 55 GB free while it runs and keeps about 35 GB
|
||||
afterwards.
|
||||
|
||||
Players who only want to play can use the [installer](installer.md) instead: it builds the game on their machine
|
||||
without this toolchain.
|
||||
Players who only want to play use the [launcher](launcher.md) instead: it installs its own build tools, builds the
|
||||
game on their machine and keeps it up to date.
|
||||
|
||||
## Toolchain
|
||||
|
||||
|
||||
160
docs/launcher.md
Normal file
160
docs/launcher.md
Normal file
@@ -0,0 +1,160 @@
|
||||
# Launcher
|
||||
|
||||
NX2navi's launcher installs everything the build needs, builds the game from the player's own dump, starts it, and
|
||||
keeps the port up to date. Players download one file, `NX2navi-Setup-<version>.exe`; nothing of the game ships with
|
||||
it. The code is `launcher/`, the release is made by `tools/make_launcher.py`, and the build itself is
|
||||
`tools/make_game.py` (the build kit's script, [README-BuildKit.md](README-BuildKit.md)).
|
||||
|
||||
## What the player does
|
||||
|
||||
1. Starts `NX2navi-Setup.exe` (from the folder that holds their `.nup` dump, ideally). It installs the launcher into
|
||||
`%LOCALAPPDATA%\Programs\NX2navi` without administrator rights, adds Start menu and Desktop shortcuts and an entry
|
||||
in Settings > Apps, and opens the launcher. Windows SmartScreen warns once: the programs are not signed.
|
||||
2. The launcher finds the dump by itself in the setup's folder, Downloads, Desktop, Documents or the top of each
|
||||
drive, and proposes `<launcher>\game` as the install folder (or `X:\Games\NX2navi` on the drive with the most
|
||||
free space when that drive has less than 40 GB). Both rows can be changed.
|
||||
3. Selects **Begin Installation**. The launcher downloads the build tools (about 170 MB, once), checks the dump,
|
||||
unpacks it, translates and compiles the game: 10 to 25 minutes. It keeps the PC awake meanwhile.
|
||||
4. Selects **Play**. The launcher hides while the game runs and comes back when it exits.
|
||||
|
||||
From then on the launcher looks for a new version each time it opens (Settings can turn that off). When one is out,
|
||||
the Updates row turns orange; selecting it downloads the new version, restarts the launcher in it and rebuilds the
|
||||
game. Saves and settings are never touched.
|
||||
|
||||
## Look
|
||||
|
||||
The window follows the game's options menu, with colours sampled from it: a wooden title band with the Triforce and
|
||||
the name, bronze label cells, dark value cells, orange On/Off choices, a white glowing cursor around the selected
|
||||
row, the menu's filigree dividers and a description box for the selected row. All text is in the game's own Cetra
|
||||
Display: the launcher decodes it from the player's dump or built game into `data/cache/fonts` (as
|
||||
`runtime/menu_assets.c` does for the in-game menu) and uses the system's interface font until then. Every picture is
|
||||
drawn in code (`launcher/ui/art.py`); no texture, font or sound of the game is in the repository or the release.
|
||||
|
||||
The Triforce is also the icon of `NX2navi.exe`, `NX2navi-Setup.exe`, the window and the taskbar
|
||||
(`art.triforce_ico`, written into the release by `tools/make_launcher.py`).
|
||||
|
||||
Screens (`launcher/ui/app.py`): first run (dump, install folder, Desktop Shortcut, Check for Updates at Start,
|
||||
Update Channel, Begin Installation); the steps of an install, update or rebuild with their progress, the build log's
|
||||
last lines and Cancel, and after a failure Try Again and Copy Report; home (Play, Updates, Game Folder, Rebuild the
|
||||
Game, Settings); settings (update check, Hide Launcher While Playing, Keep Build Cache, Update Channel, the folders,
|
||||
Open Logs Folder, Uninstall, Back); uninstall (keep or remove the built game).
|
||||
|
||||
Controls: the mouse (hover moves the cursor, a click selects); the keyboard (arrows, Enter or Space, Esc, F5 checks
|
||||
for updates); an XInput gamepad (d-pad or left stick, A, B, Y).
|
||||
|
||||
`python -m launcher --selftest --shots DIR` draws every screen with made-up states and saves a PNG of each.
|
||||
|
||||
## The launcher's folder
|
||||
|
||||
```
|
||||
NX2navi.exe launcher/native/nx2navi.c: swaps in a staged Python, starts app\<current>\launcher\start.pyw
|
||||
python/ python.org's embeddable Python 3.14.9 with tkinter, numpy, lz4 and zstandard
|
||||
app/current.txt the version the stub starts; app/previous.txt the one before it
|
||||
app/<version>/ launcher/, runtime/, recomp/, tools/, bin/nx2navi_xlate.exe, VERSION,
|
||||
toolchain.lock.json (the pinned build tools), feed.json (where updates come from)
|
||||
pydeps/ zbic, downloaded from PyPI on this PC
|
||||
toolchain/msys64/ucrt64 MSYS2's clang and libraries, downloaded from the MSYS2 repository on this PC
|
||||
toolchain/pkgs/ the downloaded package files
|
||||
data/launcher.json settings; data/logs/launcher.log (and crash.log); data/cache/fonts
|
||||
game/ the built game, unless another folder was chosen (its layout: README-BuildKit.md)
|
||||
```
|
||||
|
||||
A path the Windows ANSI code page cannot hold breaks the MSYS2 tools, so the toolchain then goes to
|
||||
`C:\NX2navi\toolchain` instead (`launcher/paths.py`).
|
||||
|
||||
## Installing
|
||||
|
||||
`jobs.InstallJob`, one row per step:
|
||||
|
||||
1. **Gather Build Tools** (`launcher/toolchain.py`): the packages `toolchain.lock.json` pins (MSYS2 UCRT64 clang,
|
||||
gcc, SDL3, Opus, libvpx, dav1d, zstd, Vulkan headers and loader, VMA and their dependencies: 31 packages, 170 MB),
|
||||
each checked against its sha256 and unpacked into `toolchain/msys64.new`, which replaces `toolchain/msys64` once
|
||||
complete. No MSYS2 shell or pacman runs. Then zbic: the wheel for the bundled Python from PyPI, checked against
|
||||
PyPI's sha256, into `pydeps/`.
|
||||
2. **Inspect Your Game** to **Compile**: `tools/make_game.py --nup <dump> --out <game> --msys <toolchain> --events`,
|
||||
whose `@@` progress lines drive the rows (check, exefs, romfs, lift, compile).
|
||||
3. The game's Cetra Display is decoded for the window.
|
||||
|
||||
## Updating
|
||||
|
||||
A release carries `NX2navi-update.json` (version, notes, and the size and sha256 of each file). The feed
|
||||
(`app/<version>/feed.json`) names a GitHub repository: the launcher reads its newest release (or pre-release, on the
|
||||
Preview channel) that has the manifest. A private repository answers only with a token (`github_token` in
|
||||
`data/launcher.json`); players need a public one. `feed_repo` in `data/launcher.json` points one launcher at another
|
||||
repository for testing; `{"type": "url", "url": ".../NX2navi-update.json"}` in feed.json reads one manifest
|
||||
directly.
|
||||
|
||||
`jobs.UpdateDownloadJob`, before the restart (`launcher/updater.py`):
|
||||
|
||||
1. **Download** `NX2navi-app-<version>.zip` (about 3 MB plus the shader translator) to `data/downloads`, resuming a cut
|
||||
download, and check its size and sha256. When the manifest names a Python other than `python/runtime-id.txt`, also
|
||||
`NX2navi-runtime-<id>.zip`, unpacked into `python.next`.
|
||||
2. **Verify**: the archive unpacks into `app/<version>.part`, refusing names that leave the folder, holds the expected
|
||||
`VERSION` and a launcher, and becomes `app/<version>`.
|
||||
3. `app/previous.txt` gets the running version, `app/current.txt` the new one, and the launcher starts the stub with
|
||||
`--continue-update <old>` and closes.
|
||||
|
||||
The stub swaps `python.next` in while no Python runs, then starts the new version, which finishes in
|
||||
`jobs.ContinueUpdateJob`:
|
||||
|
||||
4. **Update Build Tools**: only when the new release pins other packages.
|
||||
5. **Translate the Code** and **Compile Changed Files**: `make_game.py --rebuild`. The exefs and romfs steps depend
|
||||
only on the game files, so they are kept and the dump is not needed again; the lift runs again and the compile
|
||||
step compiles only the files whose source, included headers, flags or compiler changed (`build-cache/obj/
|
||||
objects.json`, from clang's dependency files).
|
||||
6. **Swap In the New Build**: the link writes `ocarina.exe.new`, which replaces `ocarina.exe` only once it is
|
||||
complete. A failed rebuild leaves the old build playable; Rebuild the Game tries again.
|
||||
7. Every version but the current and the previous one is deleted.
|
||||
|
||||
When the new version fails to start (exit code 86 from `start.pyw`, or any error within six seconds), the stub sets
|
||||
`current.txt` back to `previous.txt`'s version and starts it with `--rolled-back`. The game is never rebuilt while it
|
||||
runs. Keep Build Cache off deletes the lifted C and the objects (about 5 GB) after each build but keeps
|
||||
`build-cache/state.json`, so updates still rebuild without the dump (compiling everything).
|
||||
|
||||
## Uninstalling
|
||||
|
||||
Settings > Uninstall NX2navi, or Settings > Apps > NX2navi (which runs `NX2navi.exe --uninstall`): the launcher
|
||||
folder, its toolchain, the shortcuts and the Apps entry go; the built game and its saves stay unless "Remove the
|
||||
Built Game" is On. The dump is never touched.
|
||||
|
||||
## Command line
|
||||
|
||||
`launcher/cli.py` runs the same jobs without the window (`python -m launcher.cli`, or in an installed launcher
|
||||
`python\python.exe app\<version>\launcher\cli.py`): `check`, `update`, `tools`, `install --dump F --game D`,
|
||||
`rebuild [--game D]`, `fonts`. `NX2navi.exe --desktop-shortcut on|off` and `--shortcuts on|off` are the stub's own.
|
||||
`NX2navi-Setup.exe` takes `--dest DIR`, `--no-shortcuts`, `--no-launch` and `--quiet`.
|
||||
|
||||
## Releases
|
||||
|
||||
1. Bump `VERSION`; build the shader translator (`tools/build_xlate.py`).
|
||||
2. With Python 3.14 and uv:
|
||||
|
||||
```powershell
|
||||
.venv-py314\Scripts\python tools\make_launcher.py --xlate build\xlate --feed owner/name --notes notes.md
|
||||
```
|
||||
|
||||
On Linux it cross-compiles the two programs with clang, lld and llvm-windres against the same MSYS2 packages and
|
||||
unpacks Tcl/Tk with msiextract (msitools). It writes into `build/launcher-release`: `NX2navi-Setup-<version>.exe`
|
||||
(the setup program with the launcher folder appended as a zip), `NX2navi-app-<version>.zip`,
|
||||
`NX2navi-runtime-<id>.zip` and `NX2navi-update.json`. `check_content` runs over the folder first.
|
||||
3. Upload those four files to GitHub release `v<version>` of the feed repository, by hand or with `--publish`
|
||||
(`gh release create`; add `--draft` to look before it goes out). Launchers see it at their next start.
|
||||
|
||||
The release resolves the MSYS2 packages of the day into `toolchain.lock.json` (`--lock` reuses an older lock). The
|
||||
repository keeps old package files, so a release keeps building with the compiler it was tested with.
|
||||
|
||||
## Tests
|
||||
|
||||
`python tests/launcher_test.py` (Python 3.14): versions, settings, the toolchain resolver and installer against a
|
||||
local fake repository, resumed and checked downloads, the GitHub and URL feeds, staging, switching, rollback and
|
||||
pruning, unsafe archives, the font decoding on a system font wrapped like the game's, `make_game.py` on a synthetic
|
||||
`.nup` (events, skipped steps, `--rebuild` without the dump, step keys), the compile step's object cache with the
|
||||
system's C compiler, the install job with a fake `make_game.py`, and the window's selftest when a display is there.
|
||||
|
||||
## Requirements and limits
|
||||
|
||||
- Windows 10 or 11, x86-64; the build's own requirements (README-BuildKit.md): 16 GB of RAM, about 30 GB free for
|
||||
the game and 1.5 GB for the launcher and its tools, a Vulkan 1.3 GPU.
|
||||
- The programs are not signed: SmartScreen warns on the first start of the setup.
|
||||
- English only for now.
|
||||
- The launcher builds the Windows game only; Linux builds use `docs/building.md`.
|
||||
8
launcher/__init__.py
Normal file
8
launcher/__init__.py
Normal file
@@ -0,0 +1,8 @@
|
||||
"""The NX2navi launcher: installs the build tools, builds the game from the player's own dump, starts it and keeps
|
||||
the port up to date (docs/launcher.md).
|
||||
|
||||
Standard library only, plus numpy for drawing the window (launcher/ui/art.py) and zstandard where Python lacks
|
||||
compression.zstd. tkinter draws the window; everything that does work runs without it (launcher/cli.py).
|
||||
"""
|
||||
|
||||
NAME = 'NX2navi'
|
||||
6
launcher/__main__.py
Normal file
6
launcher/__main__.py
Normal file
@@ -0,0 +1,6 @@
|
||||
"""python -m launcher [--selftest [--shots DIR]] — the launcher window (launcher/ui/app.py)."""
|
||||
import sys
|
||||
|
||||
from .ui.app import main
|
||||
|
||||
sys.exit(main())
|
||||
135
launcher/build.py
Normal file
135
launcher/build.py
Normal file
@@ -0,0 +1,135 @@
|
||||
"""Running tools/make_game.py for the launcher: the command line, its "@@" progress lines (make_game.py --events)
|
||||
and cancelling it with every compiler it started."""
|
||||
import json
|
||||
import os
|
||||
import signal
|
||||
import subprocess
|
||||
import threading
|
||||
|
||||
# Runs make_game.py with pydeps/ (zbic) and the script's folder first on sys.path; the embeddable Python's ._pth
|
||||
# file leaves both off.
|
||||
RUN = ('import os, runpy, sys; p = sys.argv[1]; s = sys.argv[2]; sys.argv = sys.argv[2:]; '
|
||||
'sys.path[:0] = [x for x in p.split(os.pathsep) if x] + [os.path.dirname(os.path.abspath(s))]; '
|
||||
'runpy.run_path(s, run_name="__main__")')
|
||||
CREATE_NO_WINDOW = 0x08000000
|
||||
CREATE_NEW_PROCESS_GROUP = 0x00000200
|
||||
|
||||
|
||||
def command(layout, args):
|
||||
return [layout.python_for_builds(), '-I', '-c', RUN, str(layout.pydeps), str(layout.make_game), *map(str, args),
|
||||
'--events']
|
||||
|
||||
|
||||
def install_args(layout, dump, game_dir):
|
||||
dump = str(dump)
|
||||
if os.path.isdir(dump):
|
||||
src = ['--exefs', os.path.join(dump, 'exefs'), '--romfs', os.path.join(dump, 'romfs')]
|
||||
else:
|
||||
src = ['--nup', dump]
|
||||
return [*src, '--out', str(game_dir), '--msys', str(layout.msys)]
|
||||
|
||||
|
||||
def rebuild_args(layout, game_dir):
|
||||
return ['--rebuild', '--out', str(game_dir), '--msys', str(layout.msys)]
|
||||
|
||||
|
||||
def check_args(dump):
|
||||
dump = str(dump)
|
||||
if os.path.isdir(dump):
|
||||
return ['--exefs', os.path.join(dump, 'exefs'), '--romfs', os.path.join(dump, 'romfs'), '--check']
|
||||
return ['--nup', dump, '--check']
|
||||
|
||||
|
||||
class BuildError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class Run:
|
||||
"""One make_game.py process. on_event(dict) for each "@@" line, on_line(str) for every other line; wait()
|
||||
returns the exit code. cancel() ends the process and its children."""
|
||||
|
||||
def __init__(self, cmd, on_event, on_line, cwd=None):
|
||||
self.error = None
|
||||
kw = {}
|
||||
if os.name == 'nt':
|
||||
kw['creationflags'] = CREATE_NO_WINDOW | CREATE_NEW_PROCESS_GROUP
|
||||
else:
|
||||
kw['start_new_session'] = True
|
||||
env = dict(os.environ, PYTHONUNBUFFERED='1', PYTHONIOENCODING='utf-8')
|
||||
self.proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL,
|
||||
cwd=cwd, env=env, text=True, encoding='utf-8', errors='replace', bufsize=1, **kw)
|
||||
self.cancelled = False
|
||||
self._reader = threading.Thread(target=self._read, args=(on_event, on_line), daemon=True)
|
||||
self._reader.start()
|
||||
|
||||
def _read(self, on_event, on_line):
|
||||
for line in self.proc.stdout:
|
||||
line = line.rstrip('\r\n')
|
||||
if line.startswith('@@'):
|
||||
try:
|
||||
ev = json.loads(line[2:])
|
||||
except ValueError:
|
||||
on_line(line)
|
||||
continue
|
||||
if ev.get('ev') == 'error':
|
||||
self.error = ev.get('message')
|
||||
on_event(ev)
|
||||
elif line.startswith('error: ') and not self.error:
|
||||
self.error = line[7:]
|
||||
on_line(line)
|
||||
else:
|
||||
on_line(line)
|
||||
|
||||
def wait(self):
|
||||
code = self.proc.wait()
|
||||
self._reader.join(5)
|
||||
return code
|
||||
|
||||
def cancel(self):
|
||||
self.cancelled = True
|
||||
if self.proc.poll() is not None:
|
||||
return
|
||||
if os.name == 'nt':
|
||||
subprocess.run(['taskkill', '/T', '/F', '/PID', str(self.proc.pid)], capture_output=True,
|
||||
creationflags=CREATE_NO_WINDOW)
|
||||
else:
|
||||
try:
|
||||
os.killpg(self.proc.pid, signal.SIGTERM)
|
||||
except (ProcessLookupError, PermissionError):
|
||||
pass
|
||||
|
||||
|
||||
def game_running(game_dir, exe='ocarina.exe'):
|
||||
"""True when the built game runs from game_dir: Windows refuses to open a running exe for writing."""
|
||||
path = os.path.join(str(game_dir), exe)
|
||||
if os.name != 'nt' or not os.path.isfile(path):
|
||||
return False
|
||||
try:
|
||||
with open(path, 'r+b'):
|
||||
return False
|
||||
except PermissionError:
|
||||
return True
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
|
||||
def build_state(game_dir):
|
||||
"""The game folder's build-cache/state.json (make_game.py), or {}."""
|
||||
try:
|
||||
with open(os.path.join(str(game_dir), 'build-cache', 'state.json'), encoding='utf-8') as f:
|
||||
return json.load(f)
|
||||
except (OSError, ValueError):
|
||||
return {}
|
||||
|
||||
|
||||
def built_version(game_dir):
|
||||
"""The kit version that last built the game folder ('' when none finished)."""
|
||||
st = build_state(game_dir)
|
||||
if 'compile' in st.get('steps', {}) and os.path.isfile(os.path.join(str(game_dir), 'ocarina.exe')):
|
||||
return st.get('kit', '')
|
||||
return ''
|
||||
|
||||
|
||||
def can_rebuild(game_dir):
|
||||
st = build_state(game_dir)
|
||||
return st.get('version') == 2 and all(s in st.get('steps', {}) for s in ('exefs', 'romfs'))
|
||||
131
launcher/cli.py
Normal file
131
launcher/cli.py
Normal file
@@ -0,0 +1,131 @@
|
||||
"""The launcher without its window, for scripts, tests and anyone who prefers a console:
|
||||
|
||||
python -m launcher.cli check is there a newer release?
|
||||
python -m launcher.cli update download and stage it, then finish in the new version
|
||||
python -m launcher.cli tools install the build tools and zbic
|
||||
python -m launcher.cli install --dump F --game D build the game (the window's Begin Installation)
|
||||
python -m launcher.cli rebuild [--game D] rebuild after an update (the window's Rebuild the Game)
|
||||
python -m launcher.cli fonts [--dump F] decode the game's font for the window
|
||||
python -m launcher.cli continue-update OLD what the new version runs after `update`
|
||||
|
||||
In an installed launcher: python\\python.exe app\\<version>\\launcher\\cli.py <command> (this file puts the release
|
||||
folder on sys.path itself, as start.pyw does). Exit code 0 on success, 1 on failure, 3 when cancelled.
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
if __name__ == '__main__' and not __package__:
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
__package__ = 'launcher' # noqa: A001 — lets the relative imports below work when run as a script
|
||||
|
||||
from . import build, gamefiles, jobs, versions # noqa: E402
|
||||
from .context import Context # noqa: E402
|
||||
from .paths import Layout # noqa: E402
|
||||
|
||||
|
||||
def follow(job):
|
||||
"""Run a job here, printing each step's state as it changes."""
|
||||
seen = {}
|
||||
job.start()
|
||||
while job.result is None:
|
||||
for s in job.steps:
|
||||
key = (s.state, s.value if s.state != jobs.RUNNING else int(s.fraction * 20))
|
||||
if seen.get(s.id) != key:
|
||||
seen[s.id] = key
|
||||
print(f'{s.label}: {s.value}' + (f' ({s.detail})' if s.detail and s.state == jobs.RUNNING else ''),
|
||||
flush=True)
|
||||
time.sleep(0.2)
|
||||
for s in job.steps:
|
||||
if seen.get(s.id, (None,))[0] != s.state:
|
||||
print(f'{s.label}: {s.value}', flush=True)
|
||||
if job.result == 'failed':
|
||||
print(f'error: {job.error}', file=sys.stderr, flush=True)
|
||||
return {'ok': 0, 'failed': 1, 'cancelled': 3}[job.result]
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser(prog='launcher.cli', description=__doc__.splitlines()[0])
|
||||
ap.add_argument('--root', help='the launcher folder (default: this release\'s, or build/launcher-dev)')
|
||||
sub = ap.add_subparsers(dest='cmd', required=True)
|
||||
sub.add_parser('check')
|
||||
sub.add_parser('update')
|
||||
sub.add_parser('tools')
|
||||
p = sub.add_parser('install')
|
||||
p.add_argument('--dump', required=True)
|
||||
p.add_argument('--game', required=True)
|
||||
p = sub.add_parser('rebuild')
|
||||
p.add_argument('--game')
|
||||
p = sub.add_parser('fonts')
|
||||
p.add_argument('--dump', default='')
|
||||
p.add_argument('--game', default='')
|
||||
p = sub.add_parser('continue-update')
|
||||
p.add_argument('old')
|
||||
a = ap.parse_args(argv)
|
||||
ctx = Context(Layout(root=a.root) if a.root else None, echo=False)
|
||||
s = ctx.settings
|
||||
|
||||
if a.cmd == 'check':
|
||||
job = jobs.CheckJob(ctx).run_inline()
|
||||
if job.result != 'ok':
|
||||
print(f'error: {job.error}', file=sys.stderr)
|
||||
return 1
|
||||
r = job.release
|
||||
if not r:
|
||||
print('no release found')
|
||||
elif jobs.update_available(ctx, r):
|
||||
print(f'update available: {versions.display(r.version)} (you have {ctx.version})')
|
||||
else:
|
||||
print(f'up to date: {ctx.version} (newest release {versions.display(r.version)})')
|
||||
return 0
|
||||
|
||||
if a.cmd == 'update':
|
||||
check = jobs.CheckJob(ctx).run_inline()
|
||||
if check.result != 'ok' or not jobs.update_available(ctx, check.release):
|
||||
print(check.error or f'up to date: {ctx.version}')
|
||||
return 0 if check.result == 'ok' else 1
|
||||
job = jobs.UpdateDownloadJob(ctx, check.release)
|
||||
code = follow(job)
|
||||
if code or job.after != 'restart':
|
||||
return code
|
||||
new = ctx.layout.apps / versions.display(check.release.version)
|
||||
cli = new / 'launcher' / 'cli.py'
|
||||
print(f'finishing in {versions.display(check.release.version)}', flush=True)
|
||||
return subprocess.call([ctx.layout.python_for_builds(), str(cli), '--root', str(ctx.layout.root),
|
||||
'continue-update', ctx.version])
|
||||
|
||||
if a.cmd == 'continue-update':
|
||||
gd = s.get('game_dir')
|
||||
return follow(jobs.ContinueUpdateJob(ctx, gd if gd and build.built_version(gd) else '', a.old,
|
||||
s.get('dump')))
|
||||
|
||||
if a.cmd == 'tools':
|
||||
job = jobs.RebuildJob(ctx, '')
|
||||
job.steps = job.steps[:1]
|
||||
job.run = lambda: job.ensure_tools('tools_update')
|
||||
return follow(job)
|
||||
|
||||
if a.cmd == 'install':
|
||||
s['dump'], s['game_dir'] = os.path.abspath(a.dump), os.path.abspath(a.game)
|
||||
s.save()
|
||||
return follow(jobs.InstallJob(ctx, s['dump'], s['game_dir']))
|
||||
|
||||
if a.cmd == 'rebuild':
|
||||
gd = os.path.abspath(a.game) if a.game else s.get('game_dir')
|
||||
if not gd:
|
||||
print('error: no game folder (give --game)', file=sys.stderr)
|
||||
return 1
|
||||
return follow(jobs.RebuildJob(ctx, gd, s.get('dump')))
|
||||
|
||||
if a.cmd == 'fonts':
|
||||
got = gamefiles.extract_fonts(ctx.layout, a.game or s.get('game_dir'), a.dump or s.get('dump'))
|
||||
for role, (path, fam) in got.items():
|
||||
print(f'{role}: {fam} ({path})')
|
||||
return 0 if got else 1
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
48
launcher/context.py
Normal file
48
launcher/context.py
Normal file
@@ -0,0 +1,48 @@
|
||||
"""What every part of the launcher shares: the folder layout, the settings and the log (data/logs/launcher.log,
|
||||
the run before it in launcher.1.log)."""
|
||||
import os
|
||||
import threading
|
||||
import time
|
||||
|
||||
from .paths import Layout
|
||||
from .settings import Settings
|
||||
|
||||
LOG_LIMIT = 4 << 20
|
||||
|
||||
|
||||
class Context:
|
||||
def __init__(self, layout=None, echo=False):
|
||||
self.layout = (layout or Layout()).ensure()
|
||||
self.settings = Settings(self.layout.settings)
|
||||
self.version = self.layout.version
|
||||
self.echo = echo
|
||||
self._lock = threading.Lock()
|
||||
path = self.layout.logs / 'launcher.log'
|
||||
try:
|
||||
if path.is_file() and path.stat().st_size > LOG_LIMIT:
|
||||
os.replace(path, path.with_name('launcher.1.log'))
|
||||
except OSError:
|
||||
pass
|
||||
try:
|
||||
self._file = open(path, 'a', encoding='utf-8', errors='replace')
|
||||
except OSError:
|
||||
self._file = None
|
||||
self.log(f'--- NX2navi launcher {self.version} ({"checkout" if self.layout.dev else "release"}) in '
|
||||
f'{self.layout.root}')
|
||||
|
||||
def log(self, line):
|
||||
stamp = time.strftime('%H:%M:%S')
|
||||
with self._lock:
|
||||
if self._file:
|
||||
for part in str(line).splitlines() or ['']:
|
||||
self._file.write(f'{stamp} {part}\n')
|
||||
self._file.flush()
|
||||
if self.echo:
|
||||
print(line, flush=True)
|
||||
|
||||
def log_tail(self, n=80):
|
||||
try:
|
||||
with open(self.layout.logs / 'launcher.log', encoding='utf-8', errors='replace') as f:
|
||||
return f.readlines()[-n:]
|
||||
except OSError:
|
||||
return []
|
||||
144
launcher/feed.py
Normal file
144
launcher/feed.py
Normal file
@@ -0,0 +1,144 @@
|
||||
"""Where updates come from.
|
||||
|
||||
Every release carries NX2navi-update.json (tools/make_launcher.py writes it):
|
||||
|
||||
{"format": 1, "version": "0.3.11", "build": "0.3.11+abc1234", "notes": "...",
|
||||
"app": {"name": "NX2navi-app-0.3.11.zip", "size": n, "sha256": "..."},
|
||||
"runtime": {"id": "py3.14.9-1", "name": "NX2navi-runtime-py3.14.9-1.zip", "size": n, "sha256": "..."},
|
||||
"toolchain": "<lock digest>"}
|
||||
|
||||
The feed (app/<version>/feed.json, or the player's override in settings) is either
|
||||
{"type": "github", "repo": "owner/name"} the newest GitHub release (or pre-release, on the preview channel)
|
||||
that has the manifest; a token reads a private repository
|
||||
{"type": "url", "url": "https://.../NX2navi-update.json"} one manifest; its files sit beside it
|
||||
"""
|
||||
import json
|
||||
import urllib.parse
|
||||
|
||||
from . import net, versions
|
||||
|
||||
MANIFEST = 'NX2navi-update.json'
|
||||
DEFAULT = {'type': 'github', 'repo': 'tangerineturtle1/NX2navi'}
|
||||
GITHUB_API = 'https://api.github.com'
|
||||
|
||||
|
||||
class FeedError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class Release:
|
||||
def __init__(self, manifest, notes='', page='', assets=None, headers=None, prerelease=False):
|
||||
self.manifest = manifest
|
||||
self.version = manifest['version']
|
||||
self.build = manifest.get('build', self.version)
|
||||
self.notes = (manifest.get('notes') or notes or '').strip()
|
||||
self.page = page
|
||||
self.assets = assets or {} # name -> url
|
||||
self.headers = headers or {} # extra request headers for the asset urls (a private repository's token)
|
||||
self.prerelease = prerelease
|
||||
|
||||
def asset(self, key):
|
||||
"""(url, headers, size, sha256) of the manifest's 'app' or 'runtime' file."""
|
||||
entry = self.manifest.get(key)
|
||||
if not entry:
|
||||
raise FeedError(f'the {self.version} manifest has no {key} file')
|
||||
url = self.assets.get(entry['name'])
|
||||
if not url:
|
||||
raise FeedError(f'the {self.version} release lacks {entry["name"]}')
|
||||
return url, self.headers, entry.get('size'), entry.get('sha256')
|
||||
|
||||
|
||||
def load(layout, settings):
|
||||
feed = dict(DEFAULT)
|
||||
try:
|
||||
if layout.feed_file.is_file():
|
||||
feed.update(json.loads(layout.feed_file.read_text(encoding='utf-8')))
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
if settings.get('feed_repo'):
|
||||
feed = {'type': 'github', 'repo': settings['feed_repo']}
|
||||
return feed
|
||||
|
||||
|
||||
def _check_manifest(m, where):
|
||||
if not isinstance(m, dict) or m.get('format') != 1 or 'version' not in m or 'app' not in m:
|
||||
raise FeedError(f'{where}: not an NX2navi update manifest')
|
||||
versions.parse(m['version'])
|
||||
return m
|
||||
|
||||
|
||||
def _github(feed, channel, token, version):
|
||||
api = feed.get('api', GITHUB_API).rstrip('/')
|
||||
headers = {'Accept': 'application/vnd.github+json', 'X-GitHub-Api-Version': '2022-11-28'}
|
||||
if token:
|
||||
headers['Authorization'] = f'Bearer {token}'
|
||||
try:
|
||||
releases = net.get_json(f'{api}/repos/{feed["repo"]}/releases?per_page=20', headers, version=version)
|
||||
except Exception as e:
|
||||
code = getattr(e, 'code', None)
|
||||
if code == 404:
|
||||
raise FeedError(f'no releases found for {feed["repo"]} (a private repository needs a token)') from e
|
||||
if code == 403:
|
||||
raise FeedError('GitHub refused the request (rate limit); try again in an hour') from e
|
||||
raise FeedError(f'GitHub could not be reached: {e}') from e
|
||||
# The API lists the newest release first: the first one the channel takes that has a manifest is the answer.
|
||||
for r in releases:
|
||||
if r.get('draft') or (r.get('prerelease') and channel != 'preview'):
|
||||
continue
|
||||
assets = {a['name']: a for a in r.get('assets', [])}
|
||||
if MANIFEST not in assets:
|
||||
continue
|
||||
if token: # a private repository's files come through the API with the token
|
||||
urls = {n: a['url'] for n, a in assets.items()}
|
||||
asset_headers = {'Accept': 'application/octet-stream', 'Authorization': f'Bearer {token}'}
|
||||
else:
|
||||
urls = {n: a['browser_download_url'] for n, a in assets.items()}
|
||||
asset_headers = {}
|
||||
try:
|
||||
m = _check_manifest(json.loads(net.get_bytes(urls[MANIFEST], asset_headers, version=version)),
|
||||
f'{r.get("tag_name")}/{MANIFEST}')
|
||||
except (FeedError, ValueError, OSError) as e:
|
||||
raise FeedError(f'{r.get("tag_name")}: {e}') from e
|
||||
return Release(m, r.get('body') or '', r.get('html_url', ''), urls, asset_headers, bool(r.get('prerelease')))
|
||||
return None
|
||||
|
||||
|
||||
def _url(feed, version):
|
||||
url = feed['url']
|
||||
try:
|
||||
m = _check_manifest(net.get_json(url, version=version), url)
|
||||
except FeedError:
|
||||
raise
|
||||
except Exception as e:
|
||||
raise FeedError(f'{url}: {e}') from e
|
||||
base = url.rsplit('/', 1)[0] + '/'
|
||||
assets = {e['name']: urllib.parse.urljoin(base, e['name']) for e in (m.get('app'), m.get('runtime')) if e}
|
||||
return Release(m, assets=assets)
|
||||
|
||||
|
||||
def latest(feed, channel='stable', token='', version='dev'):
|
||||
"""The newest release the feed offers on the channel, or None when it has none."""
|
||||
if feed.get('type') == 'url':
|
||||
return _url(feed, version)
|
||||
if feed.get('type') == 'github':
|
||||
return _github(feed, channel, token, version)
|
||||
raise FeedError(f'unknown feed type {feed.get("type")!r}')
|
||||
|
||||
|
||||
def plain_notes(text, limit=12):
|
||||
"""Release notes as short lines for the window: markdown bullets and headings without their marks."""
|
||||
out = []
|
||||
for line in (text or '').splitlines():
|
||||
s = line.strip()
|
||||
if not s or s.startswith(('<!--', '```', '---')):
|
||||
continue
|
||||
s = s.lstrip('#').strip()
|
||||
for mark in ('- ', '* ', '+ '):
|
||||
if s.startswith(mark):
|
||||
s = s[2:]
|
||||
s = s.replace('**', '').replace('`', '')
|
||||
if s:
|
||||
out.append(s)
|
||||
if len(out) >= limit:
|
||||
break
|
||||
return out
|
||||
322
launcher/gamefiles.py
Normal file
322
launcher/gamefiles.py
Normal file
@@ -0,0 +1,322 @@
|
||||
"""The player's game files, as far as the launcher reads them itself: finding a .nup dump, and the game's Cetra
|
||||
Display font for the window (runtime/menu_assets.c reads the same font for the in-game menu's titles).
|
||||
|
||||
The font comes from the built game's romfs/ (romfs.bin and index.json, tools/make_game.py) or, before the first
|
||||
build, straight from the .nup. They are decoded once into data/cache/fonts, which stays on the player's PC like the
|
||||
rest of what is built from their game. Nothing of the game ships with the launcher.
|
||||
|
||||
- .zs files are zstd frames, most with one of the dictionaries of Pack/ZsDic.pack.zs (a SARC, compressed without a
|
||||
dictionary; a frame names its dictionary by id).
|
||||
- A .bfotf font is an OpenType file behind an 8-byte header, every big-endian 32-bit word XORed with one key: the
|
||||
first word is 0x7f9a0218 under it, the second the payload size.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import struct
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
TITLE_ID = 0x0400486024DEC000
|
||||
FONTS = { # role -> RomFS path; the window sets all its text in Cetra Display
|
||||
'title': 'Font2/Scalable/CetraDisplayPro-Medium.Nin_Ounce_NVN2.bfotf.zs',
|
||||
}
|
||||
DICTS = 'Pack/ZsDic.pack.zs'
|
||||
DICT_NAMES = ('zs.zsdic', 'pack.zsdic', 'bcett.byml.zsdic', 'bkprb.zsdic')
|
||||
MAX_RAW = 64 << 20
|
||||
|
||||
|
||||
def _nup_module(layout):
|
||||
if str(layout.tools) not in sys.path:
|
||||
sys.path.insert(0, str(layout.tools))
|
||||
import nup
|
||||
return nup
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- finding a dump
|
||||
|
||||
def candidate_dirs(extra=()):
|
||||
home = Path.home()
|
||||
dirs = [Path(d) for d in extra if d]
|
||||
dirs += [home / 'Downloads', home / 'Desktop', home / 'Documents', home]
|
||||
if os.name == 'nt':
|
||||
import string
|
||||
dirs += [Path(f'{c}:\\') for c in string.ascii_uppercase[2:] if os.path.isdir(f'{c}:\\')]
|
||||
seen, out = set(), []
|
||||
for d in dirs:
|
||||
key = os.path.normcase(str(d))
|
||||
if key not in seen and d.is_dir():
|
||||
seen.add(key)
|
||||
out.append(d)
|
||||
return out
|
||||
|
||||
|
||||
def is_game_nup(layout, path):
|
||||
"""True when the file is a .nup holding Ocarina of Time (title 0400486024DEC000 or its update)."""
|
||||
try:
|
||||
n = _nup_module(layout).Nup(path)
|
||||
except Exception:
|
||||
return False
|
||||
return any(p['title_id'] in (TITLE_ID, TITLE_ID | 0x800) for p in n.packages)
|
||||
|
||||
|
||||
def find_dumps(layout, extra=(), depth=1):
|
||||
""".nup files with the game in the usual places (the setup's folder first), newest first."""
|
||||
found = []
|
||||
for d in candidate_dirs(extra):
|
||||
pattern_dirs = [d]
|
||||
if depth > 0:
|
||||
try:
|
||||
pattern_dirs += [s for s in d.iterdir() if s.is_dir() and not s.name.startswith(('.', '$'))][:200]
|
||||
except OSError:
|
||||
pass
|
||||
for pd in pattern_dirs:
|
||||
try:
|
||||
for f in pd.glob('*.nup'):
|
||||
if f.is_file() and is_game_nup(layout, f):
|
||||
found.append(f)
|
||||
except OSError:
|
||||
continue
|
||||
uniq = {os.path.normcase(str(f)): f for f in found}
|
||||
return sorted(uniq.values(), key=lambda f: f.stat().st_mtime, reverse=True)
|
||||
|
||||
|
||||
def is_game_folder(path):
|
||||
p = Path(path)
|
||||
return (p / 'exefs' / 'main').is_file() and (p / 'romfs').is_dir()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- reading RomFS files
|
||||
|
||||
class RomfsImage:
|
||||
"""The built game's romfs/romfs.bin through its index.json."""
|
||||
|
||||
def __init__(self, folder):
|
||||
folder = Path(folder)
|
||||
index = json.loads((folder / 'index.json').read_text(encoding='utf-8'))
|
||||
self.files = {e['path']: (e['offset'], e['size']) for e in index['files']}
|
||||
self.bin = folder / 'romfs.bin'
|
||||
|
||||
def read(self, rel):
|
||||
if rel not in self.files:
|
||||
return None
|
||||
off, size = self.files[rel]
|
||||
with open(self.bin, 'rb') as f:
|
||||
f.seek(off)
|
||||
return f.read(size)
|
||||
|
||||
|
||||
class NupRomfs:
|
||||
"""The RomFS of the game's update (or base) package inside a .nup."""
|
||||
|
||||
def __init__(self, layout, path):
|
||||
nup = _nup_module(layout)
|
||||
self.n = nup.Nup(path)
|
||||
pkgs = [p for p in self.n.packages if p['title_id'] in (TITLE_ID, TITLE_ID | 0x800)]
|
||||
pkgs.sort(key=lambda p: (p['kind'] != 1, -p['version']))
|
||||
self.trees = []
|
||||
for p in pkgs:
|
||||
if p['romfs'] < self.n.ndir:
|
||||
self.trees.append(dict(self.n.tree(p['romfs'])))
|
||||
|
||||
def read(self, rel):
|
||||
for t in self.trees:
|
||||
if rel in t:
|
||||
return self.n.read(t[rel])
|
||||
return None
|
||||
|
||||
|
||||
def open_romfs(layout, game_dir='', dump=''):
|
||||
"""A RomFS reader for the built game, else the dump, else None."""
|
||||
if game_dir and (Path(game_dir) / 'romfs' / 'index.json').is_file():
|
||||
try:
|
||||
return RomfsImage(Path(game_dir) / 'romfs')
|
||||
except (OSError, ValueError, KeyError):
|
||||
pass
|
||||
if dump and Path(dump).is_file():
|
||||
try:
|
||||
return NupRomfs(layout, dump)
|
||||
except Exception:
|
||||
pass
|
||||
if dump and is_game_folder(dump):
|
||||
return FolderRomfs(Path(dump) / 'romfs')
|
||||
return None
|
||||
|
||||
|
||||
class FolderRomfs:
|
||||
def __init__(self, folder):
|
||||
self.folder = Path(folder)
|
||||
|
||||
def read(self, rel):
|
||||
p = self.folder / rel
|
||||
return p.read_bytes() if p.is_file() else None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- zstd with the game's dictionaries
|
||||
|
||||
def _zstd():
|
||||
try:
|
||||
from compression import zstd # Python 3.14
|
||||
return 'stdlib', zstd
|
||||
except ImportError:
|
||||
import zstandard
|
||||
return 'zstandard', zstandard
|
||||
|
||||
|
||||
def _frame_dict_id(data):
|
||||
kind, z = _zstd()
|
||||
if kind == 'stdlib':
|
||||
return z.get_frame_info(data).dictionary_id
|
||||
return z.get_frame_parameters(data).dict_id
|
||||
|
||||
|
||||
def _decompress(data, dict_bytes=None):
|
||||
kind, z = _zstd()
|
||||
if kind == 'stdlib':
|
||||
d = z.ZstdDict(dict_bytes) if dict_bytes else None
|
||||
return z.decompress(data, zstd_dict=d)
|
||||
dctx = z.ZstdDecompressor(dict_data=z.ZstdCompressionDict(dict_bytes) if dict_bytes else None)
|
||||
return dctx.decompressobj().decompress(data)
|
||||
|
||||
|
||||
def _dict_id(dict_bytes):
|
||||
"""The id a zstd dictionary carries (magic 0xEC30A437, then the id)."""
|
||||
if len(dict_bytes) >= 8 and struct.unpack_from('<I', dict_bytes, 0)[0] == 0xEC30A437:
|
||||
return struct.unpack_from('<I', dict_bytes, 4)[0]
|
||||
return 0
|
||||
|
||||
|
||||
def sarc_members(data):
|
||||
"""{name: bytes} of a SARC archive."""
|
||||
if len(data) < 0x14 or data[:4] != b'SARC':
|
||||
return {}
|
||||
hl = struct.unpack_from('<H', data, 4)[0]
|
||||
base = struct.unpack_from('<I', data, 0xC)[0]
|
||||
if data[hl:hl + 4] != b'SFAT':
|
||||
return {}
|
||||
count = struct.unpack_from('<H', data, hl + 6)[0]
|
||||
ents = hl + 0xC
|
||||
names = ents + 16 * count + 8
|
||||
if data[ents + 16 * count:ents + 16 * count + 4] != b'SFNT':
|
||||
return {}
|
||||
out = {}
|
||||
for i in range(count):
|
||||
_h, attr, start, end = struct.unpack_from('<IIII', data, ents + 16 * i)
|
||||
no = names + (attr & 0xFFFF) * 4
|
||||
name = data[no:data.index(b'\0', no)].decode('utf-8', 'replace')
|
||||
if start <= end and base + end <= len(data):
|
||||
out[name] = data[base + start:base + end]
|
||||
return out
|
||||
|
||||
|
||||
class ZsReader:
|
||||
def __init__(self, romfs):
|
||||
self.romfs = romfs
|
||||
self._dicts = None
|
||||
|
||||
def dicts(self):
|
||||
if self._dicts is None:
|
||||
self._dicts = {}
|
||||
raw = self.romfs.read(DICTS)
|
||||
if raw:
|
||||
members = sarc_members(_decompress(raw))
|
||||
for name in DICT_NAMES:
|
||||
if name in members:
|
||||
self._dicts[_dict_id(members[name])] = members[name]
|
||||
return self._dicts
|
||||
|
||||
def read(self, rel):
|
||||
data = self.romfs.read(rel)
|
||||
if data is None or not rel.endswith('.zs'):
|
||||
return data
|
||||
did = _frame_dict_id(data)
|
||||
if did:
|
||||
d = self.dicts().get(did)
|
||||
if d is None:
|
||||
return None
|
||||
return _decompress(data, d)
|
||||
return _decompress(data)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- fonts
|
||||
|
||||
def bfotf_to_otf(b):
|
||||
"""The OpenType (or TrueType) bytes inside a .bfotf/.bfttf, or None."""
|
||||
if not b or len(b) < 12:
|
||||
return None
|
||||
key = struct.unpack_from('>I', b, 0)[0] ^ 0x7F9A0218
|
||||
size = struct.unpack_from('>I', b, 4)[0] ^ key
|
||||
if size < 12 or size > len(b) - 8 or size > MAX_RAW:
|
||||
return None
|
||||
body = b[8:8 + size] + b'\0' * (-size % 4)
|
||||
words = struct.unpack(f'>{len(body) // 4}I', body)
|
||||
out = struct.pack(f'>{len(words)}I', *(w ^ key for w in words))[:size]
|
||||
if out[:4] != b'OTTO' and out[:4] != b'\x00\x01\x00\x00':
|
||||
return None
|
||||
return out
|
||||
|
||||
|
||||
def font_family(otf):
|
||||
"""The family name in an OpenType 'name' table (name id 16, else 1), or None."""
|
||||
try:
|
||||
num = struct.unpack_from('>H', otf, 4)[0]
|
||||
for i in range(num):
|
||||
tag, _cs, off, _len = struct.unpack_from('>4sIII', otf, 12 + 16 * i)
|
||||
if tag != b'name':
|
||||
continue
|
||||
_fmt, count, strings = struct.unpack_from('>HHH', otf, off)
|
||||
found = {}
|
||||
for k in range(count):
|
||||
pid, eid, lid, nid, ln, so = struct.unpack_from('>HHHHHH', otf, off + 6 + 12 * k)
|
||||
if nid not in (1, 16):
|
||||
continue
|
||||
raw = otf[off + strings + so:off + strings + so + ln]
|
||||
if pid == 3 or pid == 0:
|
||||
name = raw.decode('utf-16-be', 'replace')
|
||||
elif pid == 1:
|
||||
name = raw.decode('mac_roman', 'replace')
|
||||
else:
|
||||
continue
|
||||
rank = (nid == 16, pid == 3 and lid == 0x409)
|
||||
if nid not in found or rank > found[nid][0]:
|
||||
found[nid] = (rank, name)
|
||||
for nid in (16, 1):
|
||||
if nid in found:
|
||||
return found[nid][1]
|
||||
except (struct.error, IndexError):
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def cached_fonts(layout):
|
||||
"""{role: (path, family)} of the fonts decoded earlier."""
|
||||
d = layout.cache / 'fonts'
|
||||
out = {}
|
||||
for role in FONTS:
|
||||
p = d / f'{role}.otf'
|
||||
meta = d / f'{role}.family'
|
||||
if p.is_file() and meta.is_file():
|
||||
out[role] = (p, meta.read_text(encoding='utf-8').strip())
|
||||
return out
|
||||
|
||||
|
||||
def extract_fonts(layout, game_dir='', dump=''):
|
||||
"""Decode the game's Cetra Display into data/cache/fonts; {role: (path, family)} of the fonts found."""
|
||||
romfs = open_romfs(layout, game_dir, dump)
|
||||
if romfs is None:
|
||||
return {}
|
||||
zs = ZsReader(romfs)
|
||||
d = layout.cache / 'fonts'
|
||||
d.mkdir(parents=True, exist_ok=True)
|
||||
out = {}
|
||||
for role, rel in FONTS.items():
|
||||
try:
|
||||
otf = bfotf_to_otf(zs.read(rel))
|
||||
except Exception:
|
||||
otf = None
|
||||
fam = font_family(otf) if otf else None
|
||||
if not otf or not fam:
|
||||
continue
|
||||
(d / f'{role}.otf').write_bytes(otf)
|
||||
(d / f'{role}.family').write_text(fam, encoding='utf-8')
|
||||
out[role] = (d / f'{role}.otf', fam)
|
||||
return out
|
||||
368
launcher/jobs.py
Normal file
368
launcher/jobs.py
Normal file
@@ -0,0 +1,368 @@
|
||||
"""The launcher's long tasks, run on a worker thread: install, update, rebuild and the update check. Each job is a
|
||||
list of steps the window shows as rows; the window reads the job's state on a timer (rev changes on every change)
|
||||
and never blocks on it. The same jobs run headless from launcher/cli.py."""
|
||||
import collections
|
||||
import threading
|
||||
import time
|
||||
import traceback
|
||||
|
||||
from . import build, feed, gamefiles, net, toolchain, updater, versions
|
||||
|
||||
PENDING, RUNNING, DONE, SKIPPED, FAILED = 'pending', 'running', 'done', 'skipped', 'failed'
|
||||
|
||||
STEPS = {
|
||||
'tools': ('Gather Build Tools', 'Downloading the compiler and libraries the build uses. This happens once and '
|
||||
'takes about 170 MB.'),
|
||||
'tools_update': ('Update Build Tools', 'Bringing the compiler and libraries to the versions this release was '
|
||||
'tested with.'),
|
||||
'check': ('Inspect Your Game', 'Making sure the dump is Ocarina of Time 1.1.1.'),
|
||||
'exefs': ('Unpack the Program', "Unpacking the game's program files."),
|
||||
'romfs': ('Pack the Game Data', 'Checking every game file against the dump and packing them into one image.'),
|
||||
'lift': ('Translate the Code', "Each function of your game's program becomes C code for the compiler."),
|
||||
'compile': ('Compile', 'Compiling the translated code and the port into a Windows program.'),
|
||||
'compile_changed': ('Compile Changed Files', 'Compiling the files this version changed. Files that did not '
|
||||
'change are kept from the last build.'),
|
||||
'shaders': ('Prepare the Shaders', "Translating every shader of the game ahead of time, so play does not stop to "
|
||||
'compile them.'),
|
||||
'ready': ('Ready to Play', 'The game is built.'),
|
||||
'download': ('Download {ver}', 'Downloading the new version of NX2navi.'),
|
||||
'verify': ('Verify the Download', 'Checking the download against its checksum.'),
|
||||
'swap': ('Swap In the New Build', 'Linking the new build. It replaces the old one only once it is complete.'),
|
||||
}
|
||||
WEIGHTS = {'tools': 12, 'tools_update': 4, 'check': 1, 'exefs': 2, 'romfs': 14, 'lift': 14, 'compile': 55,
|
||||
'compile_changed': 55, 'shaders': 18, 'ready': 0, 'download': 8, 'verify': 1, 'swap': 6}
|
||||
|
||||
|
||||
def mmss(seconds):
|
||||
s = int(round(seconds))
|
||||
return f'{s // 3600}:{s // 60 % 60:02d}:{s % 60:02d}' if s >= 3600 else f'{s // 60}:{s % 60:02d}'
|
||||
|
||||
|
||||
class Step:
|
||||
def __init__(self, sid, ver=''):
|
||||
label, desc = STEPS[sid]
|
||||
self.id, self.label, self.desc = sid, label.format(ver=ver), desc
|
||||
self.state, self.value, self.fraction, self.detail = PENDING, 'Waiting', 0.0, ''
|
||||
self.t0, self.seconds = None, 0.0
|
||||
|
||||
|
||||
class Job:
|
||||
kind = 'job'
|
||||
title = ''
|
||||
|
||||
def __init__(self, ctx, step_ids, ver=''):
|
||||
self.ctx = ctx
|
||||
self.steps = [Step(s, ver) for s in step_ids]
|
||||
self.rev = 0
|
||||
self.cancel_flag = threading.Event()
|
||||
self.proc = None
|
||||
self.result = None # None while running, then 'ok', 'failed' or 'cancelled'
|
||||
self.error = ''
|
||||
self.lines = collections.deque(maxlen=500)
|
||||
self.started = time.time()
|
||||
self.finished = None
|
||||
self.after = None # what the window does next ('restart', ...)
|
||||
self._lock = threading.Lock()
|
||||
|
||||
# ---- state the window reads
|
||||
def step(self, sid):
|
||||
for s in self.steps:
|
||||
if s.id == sid:
|
||||
return s
|
||||
return None
|
||||
|
||||
def current(self):
|
||||
for s in self.steps:
|
||||
if s.state in (RUNNING, FAILED):
|
||||
return s
|
||||
return next((s for s in self.steps if s.state == PENDING), self.steps[-1])
|
||||
|
||||
def overall(self):
|
||||
total = sum(WEIGHTS[s.id] for s in self.steps) or 1
|
||||
got = 0.0
|
||||
for s in self.steps:
|
||||
w = WEIGHTS[s.id]
|
||||
got += w if s.state in (DONE, SKIPPED) else w * s.fraction if s.state == RUNNING else 0
|
||||
return min(1.0, got / total)
|
||||
|
||||
def eta(self):
|
||||
f = self.overall()
|
||||
elapsed = time.time() - self.started
|
||||
if f < 0.08 or elapsed < 20:
|
||||
return None
|
||||
return elapsed * (1 - f) / f
|
||||
|
||||
def elapsed(self):
|
||||
return (self.finished or time.time()) - self.started
|
||||
|
||||
# ---- changes, from the worker
|
||||
def _touch(self):
|
||||
self.rev += 1
|
||||
|
||||
def log(self, line):
|
||||
self.lines.append(line)
|
||||
self.ctx.log(line)
|
||||
self._touch()
|
||||
|
||||
def begin(self, sid, detail=''):
|
||||
s = self.step(sid)
|
||||
if s and s.state != RUNNING:
|
||||
s.state, s.fraction, s.detail, s.t0, s.value = RUNNING, 0.0, detail, time.time(), '0%'
|
||||
self._touch()
|
||||
|
||||
def progress(self, sid, fraction, detail=''):
|
||||
s = self.step(sid)
|
||||
if s:
|
||||
if s.state == PENDING:
|
||||
self.begin(sid)
|
||||
s.fraction = max(0.0, min(1.0, fraction))
|
||||
s.value = f'{int(s.fraction * 100)}%'
|
||||
if detail:
|
||||
s.detail = detail
|
||||
self._touch()
|
||||
|
||||
def finish(self, sid, value=None):
|
||||
s = self.step(sid)
|
||||
if s:
|
||||
s.seconds = time.time() - s.t0 if s.t0 else 0.0
|
||||
s.state, s.fraction = DONE, 1.0
|
||||
s.value = value or f'Done · {mmss(s.seconds)}'
|
||||
self._touch()
|
||||
|
||||
def skip(self, sid, value='Unchanged'):
|
||||
s = self.step(sid)
|
||||
if s:
|
||||
s.state, s.value = SKIPPED, value
|
||||
self._touch()
|
||||
|
||||
def cancelled(self):
|
||||
return self.cancel_flag.is_set()
|
||||
|
||||
def cancel(self):
|
||||
self.cancel_flag.set()
|
||||
if self.proc:
|
||||
self.proc.cancel()
|
||||
self._touch()
|
||||
|
||||
# ---- running
|
||||
def start(self):
|
||||
threading.Thread(target=self._main, name=f'job-{self.kind}', daemon=True).start()
|
||||
return self
|
||||
|
||||
def run_inline(self):
|
||||
self._main()
|
||||
return self
|
||||
|
||||
def _main(self):
|
||||
try:
|
||||
self.run()
|
||||
self.result = 'ok'
|
||||
except net.Cancelled:
|
||||
self.result = 'cancelled'
|
||||
except Exception as e: # every failure ends the job with a message; the log has the traceback
|
||||
if self.cancelled():
|
||||
self.result = 'cancelled'
|
||||
else:
|
||||
self.result, self.error = 'failed', str(e) or type(e).__name__
|
||||
self.ctx.log(traceback.format_exc())
|
||||
finally:
|
||||
for s in self.steps:
|
||||
if s.state == RUNNING:
|
||||
s.state = FAILED if self.result == 'failed' else PENDING
|
||||
s.value = 'Stopped' if self.result == 'failed' else 'Cancelled'
|
||||
self.finished = time.time()
|
||||
self._touch()
|
||||
|
||||
def run(self):
|
||||
raise NotImplementedError
|
||||
|
||||
# ---- shared pieces
|
||||
def ensure_tools(self, sid):
|
||||
layout = self.ctx.layout
|
||||
self.begin(sid)
|
||||
lock = toolchain.load_lock(layout, self.ctx.version)
|
||||
if toolchain.is_current(layout, lock):
|
||||
have_tools = True
|
||||
else:
|
||||
have_tools = False
|
||||
|
||||
def prog(phase, done, total):
|
||||
if phase == 'download':
|
||||
self.progress(sid, 0.9 * done / max(total, 1),
|
||||
f'{done / 1e6:.0f} of {total / 1e6:.0f} MB downloaded')
|
||||
else:
|
||||
self.progress(sid, 0.9 + 0.1 * done / max(total, 1), f'Unpacking {done} of {total} packages')
|
||||
toolchain.install(layout, lock, progress=prog, cancel=self.cancelled, log=self.log,
|
||||
version=self.ctx.version)
|
||||
toolchain.ensure_zbic(layout, cancel=self.cancelled, log=self.log, version=self.ctx.version)
|
||||
if have_tools:
|
||||
self.skip(sid, 'Unchanged' if sid == 'tools_update' else 'Ready')
|
||||
else:
|
||||
self.finish(sid)
|
||||
|
||||
def make_game(self, args, step_map, link_step=None):
|
||||
"""Run make_game.py; its steps drive ours through step_map (make_game step -> our step id or None)."""
|
||||
def on_event(ev):
|
||||
kind = ev.get('ev')
|
||||
sid = step_map.get(ev.get('step'))
|
||||
if kind == 'step' and sid:
|
||||
st = ev.get('state')
|
||||
if st == 'start':
|
||||
self.begin(sid)
|
||||
elif st == 'done':
|
||||
if sid == 'check':
|
||||
self.finish(sid, 'Version 1.1.1')
|
||||
elif link_step and sid in ('compile', 'compile_changed'):
|
||||
self.finish(sid)
|
||||
self.finish(link_step)
|
||||
else:
|
||||
self.finish(sid)
|
||||
elif st == 'skip':
|
||||
self.skip(sid, 'Kept' if sid in ('exefs', 'romfs') else 'Unchanged')
|
||||
if link_step and sid in ('compile', 'compile_changed'):
|
||||
self.skip(link_step, 'Unchanged')
|
||||
elif kind == 'progress' and sid:
|
||||
done, total, unit = ev.get('done', 0), max(ev.get('total', 1), 1), ev.get('unit', '')
|
||||
if unit == 'bytes':
|
||||
detail = f'{done / 1e9:.1f} of {total / 1e9:.1f} GB'
|
||||
else:
|
||||
detail = f'{done:,} of {total:,} {unit}'
|
||||
self.progress(sid, done / total, detail)
|
||||
elif kind == 'phase' and ev.get('phase') == 'link':
|
||||
if link_step:
|
||||
s = self.step(step_map.get('compile'))
|
||||
if s and s.state == RUNNING:
|
||||
self.finish(s.id)
|
||||
self.begin(link_step, 'Linking')
|
||||
else:
|
||||
self.progress(sid or 'compile', 0.99, 'Linking')
|
||||
elif kind == 'error':
|
||||
self.error = ev.get('message', '')
|
||||
|
||||
self.proc = build.Run(build.command(self.ctx.layout, args), on_event, self.log)
|
||||
code = self.proc.wait()
|
||||
err = self.proc.error
|
||||
self.proc = None
|
||||
if self.cancelled():
|
||||
raise net.Cancelled()
|
||||
if code:
|
||||
raise build.BuildError(err or f'the build stopped (exit code {code})')
|
||||
|
||||
|
||||
class InstallJob(Job):
|
||||
kind = 'install'
|
||||
title = 'Installing'
|
||||
|
||||
def __init__(self, ctx, dump, game_dir):
|
||||
super().__init__(ctx, ['tools', 'check', 'exefs', 'romfs', 'lift', 'compile', 'shaders', 'ready'])
|
||||
self.dump, self.game_dir = str(dump), str(game_dir)
|
||||
|
||||
def run(self):
|
||||
self.ensure_tools('tools')
|
||||
self.make_game(build.install_args(self.ctx.layout, self.dump, self.game_dir),
|
||||
{s: s for s in ('check', 'exefs', 'romfs', 'lift', 'compile', 'shaders')})
|
||||
if self.step('shaders').state == PENDING:
|
||||
self.skip('shaders')
|
||||
self.finish('ready', 'Ready')
|
||||
try:
|
||||
gamefiles.extract_fonts(self.ctx.layout, self.game_dir)
|
||||
except Exception as e:
|
||||
self.log(f'fonts: {e}')
|
||||
|
||||
|
||||
class RebuildJob(Job):
|
||||
kind = 'rebuild'
|
||||
title = 'Rebuilding'
|
||||
|
||||
def __init__(self, ctx, game_dir, dump=''):
|
||||
super().__init__(ctx, ['tools_update', 'lift', 'compile_changed', 'swap', 'shaders'])
|
||||
self.game_dir, self.dump = str(game_dir), dump
|
||||
|
||||
def run(self):
|
||||
self.ensure_tools('tools_update')
|
||||
layout = self.ctx.layout
|
||||
if build.game_running(self.game_dir):
|
||||
raise build.BuildError('the game is running: close it and try again')
|
||||
if build.can_rebuild(self.game_dir):
|
||||
args = build.rebuild_args(layout, self.game_dir)
|
||||
elif self.dump:
|
||||
args = build.install_args(layout, self.dump, self.game_dir)
|
||||
else:
|
||||
raise build.BuildError('the game folder has no earlier build: install the game again')
|
||||
self.make_game(args, {'lift': 'lift', 'compile': 'compile_changed', 'shaders': 'shaders'}, link_step='swap')
|
||||
for sid in ('lift', 'compile_changed', 'swap', 'shaders'):
|
||||
if self.step(sid).state == PENDING:
|
||||
self.skip(sid)
|
||||
|
||||
|
||||
class UpdateDownloadJob(Job):
|
||||
"""Before the restart: download, check and stage the release, then switch app/current.txt to it."""
|
||||
kind = 'update'
|
||||
|
||||
def __init__(self, ctx, release):
|
||||
ver = versions.display(release.version)
|
||||
super().__init__(ctx, ['download', 'verify', 'tools_update', 'lift', 'compile_changed', 'swap', 'shaders'], ver)
|
||||
self.release, self.ver = release, ver
|
||||
self.title = f'Updating to {ver}'
|
||||
|
||||
def run(self):
|
||||
layout = self.ctx.layout
|
||||
self.begin('download')
|
||||
prog = lambda d, t: self.progress('download', d / max(t, 1), f'{d / 1e6:.1f} of {t / 1e6:.1f} MB') # noqa: E731
|
||||
folder = updater.stage_app(layout, self.release, progress=prog, cancel=self.cancelled,
|
||||
version=self.ctx.version)
|
||||
if updater.runtime_needed(layout, self.release):
|
||||
updater.stage_runtime(layout, self.release, progress=prog, cancel=self.cancelled,
|
||||
version=self.ctx.version)
|
||||
self.finish('download')
|
||||
self.begin('verify')
|
||||
if not (folder / 'launcher' / 'start.pyw').is_file():
|
||||
raise updater.UpdateError(f'{folder} is incomplete')
|
||||
self.finish('verify', 'Checksum OK')
|
||||
if layout.dev: # a checkout runs from the repository: nothing to switch to
|
||||
self.after = 'staged'
|
||||
return
|
||||
updater.switch_to(layout, self.ver)
|
||||
self.after = 'restart'
|
||||
|
||||
|
||||
class ContinueUpdateJob(RebuildJob):
|
||||
"""After the restart, in the new version: bring the tools to its lock and rebuild the game."""
|
||||
kind = 'update'
|
||||
|
||||
def __init__(self, ctx, game_dir, old, dump=''):
|
||||
Job.__init__(self, ctx, ['download', 'verify', 'tools_update', 'lift', 'compile_changed', 'swap', 'shaders'],
|
||||
ctx.layout.version)
|
||||
self.game_dir, self.dump, self.old = str(game_dir), dump, old
|
||||
self.title = f'Updating to {versions.display(ctx.layout.version)}'
|
||||
for sid, value in (('download', 'Done'), ('verify', 'Checksum OK')):
|
||||
self.step(sid).state, self.step(sid).value, self.step(sid).fraction = DONE, value, 1.0
|
||||
|
||||
def run(self):
|
||||
if self.game_dir and build.can_rebuild(self.game_dir):
|
||||
RebuildJob.run(self)
|
||||
else:
|
||||
self.ensure_tools('tools_update')
|
||||
for sid in ('lift', 'compile_changed', 'swap', 'shaders'):
|
||||
self.skip(sid, 'No game yet')
|
||||
updater.prune(self.ctx.layout)
|
||||
|
||||
|
||||
class CheckJob(Job):
|
||||
kind = 'check'
|
||||
|
||||
def __init__(self, ctx):
|
||||
super().__init__(ctx, [])
|
||||
self.release = None
|
||||
|
||||
def run(self):
|
||||
s = self.ctx.settings
|
||||
self.release = feed.latest(feed.load(self.ctx.layout, s), s.get('channel', 'stable'),
|
||||
s.get('github_token', ''), self.ctx.version)
|
||||
|
||||
def overall(self):
|
||||
return 0.0
|
||||
|
||||
|
||||
def update_available(ctx, release):
|
||||
return bool(release) and versions.newer(release.version, ctx.layout.version)
|
||||
24
launcher/native/app.manifest
Normal file
24
launcher/native/app.manifest
Normal file
@@ -0,0 +1,24 @@
|
||||
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
|
||||
<!-- NX2navi.exe and NX2navi-Setup.exe: run as the user (no elevation prompt), draw at the display's real resolution,
|
||||
use the current common controls and pass paths as UTF-8 to the C runtime. -->
|
||||
<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
|
||||
<trustInfo xmlns="urn:schemas-microsoft-com:asm.v3">
|
||||
<security>
|
||||
<requestedPrivileges>
|
||||
<requestedExecutionLevel level="asInvoker" uiAccess="false"/>
|
||||
</requestedPrivileges>
|
||||
</security>
|
||||
</trustInfo>
|
||||
<application xmlns="urn:schemas-microsoft-com:asm.v3">
|
||||
<windowsSettings>
|
||||
<dpiAware xmlns="http://schemas.microsoft.com/SMI/2005/WindowsSettings">true/pm</dpiAware>
|
||||
<dpiAwareness xmlns="http://schemas.microsoft.com/SMI/2016/WindowsSettings">PerMonitorV2</dpiAwareness>
|
||||
<activeCodePage xmlns="http://schemas.microsoft.com/SMI/2019/WindowsSettings">UTF-8</activeCodePage>
|
||||
</windowsSettings>
|
||||
</application>
|
||||
<compatibility xmlns="urn:schemas-microsoft-com:compatibility.v1">
|
||||
<application>
|
||||
<supportedOS Id="{8e0f7a12-bfb3-4fe8-b9a5-48fd50a15a9a}"/>
|
||||
</application>
|
||||
</compatibility>
|
||||
</assembly>
|
||||
166
launcher/native/common.h
Normal file
166
launcher/native/common.h
Normal file
@@ -0,0 +1,166 @@
|
||||
/* Shared by NX2navi.exe (nx2navi.c) and NX2navi-Setup.exe (setup.c): paths, small files, shortcuts and the
|
||||
uninstall entry. Wide-character Win32 throughout, so folder names in any language work. */
|
||||
#pragma once
|
||||
#define COBJMACROS
|
||||
#define WIN32_LEAN_AND_MEAN
|
||||
#include <windows.h>
|
||||
#include <objbase.h>
|
||||
#include <shellapi.h>
|
||||
#include <shlobj.h>
|
||||
#include <shobjidl.h>
|
||||
#include <stdio.h>
|
||||
#include <string.h>
|
||||
#include <wchar.h>
|
||||
|
||||
#define PATHN 1024
|
||||
#define APP_NAME L"NX2navi"
|
||||
#define UNINSTALL_KEY L"Software\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\NX2navi"
|
||||
|
||||
static inline void exe_dir(wchar_t *out, size_t n)
|
||||
{
|
||||
DWORD len = GetModuleFileNameW(NULL, out, (DWORD)n);
|
||||
if (len == 0 || len >= n) { out[0] = 0; return; }
|
||||
while (len > 0 && out[len - 1] != L'\\' && out[len - 1] != L'/') len--;
|
||||
out[len > 0 ? len - 1 : 0] = 0;
|
||||
}
|
||||
|
||||
static inline int path_exists(const wchar_t *p) { return GetFileAttributesW(p) != INVALID_FILE_ATTRIBUTES; }
|
||||
|
||||
static inline int is_dir(const wchar_t *p)
|
||||
{
|
||||
DWORD a = GetFileAttributesW(p);
|
||||
return a != INVALID_FILE_ATTRIBUTES && (a & FILE_ATTRIBUTE_DIRECTORY);
|
||||
}
|
||||
|
||||
/* The first line of a small UTF-8 text file, trimmed, as wide text; 0 when absent or empty. */
|
||||
static inline int read_line(const wchar_t *path, wchar_t *out, size_t n)
|
||||
{
|
||||
char buf[256];
|
||||
DWORD got = 0;
|
||||
HANDLE f = CreateFileW(path, GENERIC_READ, FILE_SHARE_READ | FILE_SHARE_DELETE, NULL, OPEN_EXISTING, 0, NULL);
|
||||
if (f == INVALID_HANDLE_VALUE) return 0;
|
||||
if (!ReadFile(f, buf, sizeof buf - 1, &got, NULL)) got = 0;
|
||||
CloseHandle(f);
|
||||
buf[got] = 0;
|
||||
char *s = buf;
|
||||
if ((unsigned char)s[0] == 0xEF && (unsigned char)s[1] == 0xBB && (unsigned char)s[2] == 0xBF) s += 3;
|
||||
while (*s == ' ' || *s == '\t') s++;
|
||||
size_t len = strcspn(s, "\r\n");
|
||||
while (len > 0 && (s[len - 1] == ' ' || s[len - 1] == '\t')) len--;
|
||||
s[len] = 0;
|
||||
if (!len) return 0;
|
||||
return MultiByteToWideChar(CP_UTF8, 0, s, -1, out, (int)n) > 1;
|
||||
}
|
||||
|
||||
static inline int write_line(const wchar_t *path, const wchar_t *text)
|
||||
{
|
||||
char buf[512];
|
||||
int len = WideCharToMultiByte(CP_UTF8, 0, text, -1, buf, sizeof buf - 2, NULL, NULL);
|
||||
if (len <= 0) return 0;
|
||||
buf[len - 1] = '\n';
|
||||
HANDLE f = CreateFileW(path, GENERIC_WRITE, 0, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
|
||||
if (f == INVALID_HANDLE_VALUE) return 0;
|
||||
DWORD put = 0;
|
||||
BOOL ok = WriteFile(f, buf, (DWORD)len, &put, NULL);
|
||||
CloseHandle(f);
|
||||
return ok && put == (DWORD)len;
|
||||
}
|
||||
|
||||
/* Compare dotted versions ("0.3.11" > "0.3.9"); build metadata after '+' is ignored. */
|
||||
static inline int version_cmp(const wchar_t *a, const wchar_t *b)
|
||||
{
|
||||
for (;;) {
|
||||
long x = 0, y = 0;
|
||||
while (*a >= L'0' && *a <= L'9') x = x * 10 + (*a++ - L'0');
|
||||
while (*b >= L'0' && *b <= L'9') y = y * 10 + (*b++ - L'0');
|
||||
if (x != y) return x < y ? -1 : 1;
|
||||
int ea = *a != L'.', eb = *b != L'.';
|
||||
if (ea && eb) return 0;
|
||||
if (!ea) a++;
|
||||
if (!eb) b++;
|
||||
if (ea && !eb) { while (*b == L'0' || *b == L'.') b++; return (*b >= L'1' && *b <= L'9') ? -1 : 0; }
|
||||
if (eb && !ea) { while (*a == L'0' || *a == L'.') a++; return (*a >= L'1' && *a <= L'9') ? 1 : 0; }
|
||||
}
|
||||
}
|
||||
|
||||
static inline int known_folder(REFKNOWNFOLDERID id, wchar_t *out, size_t n)
|
||||
{
|
||||
PWSTR p = NULL;
|
||||
if (FAILED(SHGetKnownFolderPath(id, 0, NULL, &p)) || !p) return 0;
|
||||
int ok = wcslen(p) < n;
|
||||
if (ok) wcscpy(out, p);
|
||||
CoTaskMemFree(p);
|
||||
return ok;
|
||||
}
|
||||
|
||||
/* A .lnk at lnk that starts target (with its own icon). */
|
||||
static inline int make_shortcut(const wchar_t *lnk, const wchar_t *target, const wchar_t *workdir, const wchar_t *desc)
|
||||
{
|
||||
IShellLinkW *sl = NULL;
|
||||
IPersistFile *pf = NULL;
|
||||
int ok = 0;
|
||||
HRESULT init = CoInitializeEx(NULL, COINIT_APARTMENTTHREADED);
|
||||
if (SUCCEEDED(CoCreateInstance(&CLSID_ShellLink, NULL, CLSCTX_INPROC_SERVER, &IID_IShellLinkW, (void **)&sl))) {
|
||||
IShellLinkW_SetPath(sl, target);
|
||||
IShellLinkW_SetWorkingDirectory(sl, workdir);
|
||||
IShellLinkW_SetIconLocation(sl, target, 0);
|
||||
IShellLinkW_SetDescription(sl, desc);
|
||||
if (SUCCEEDED(IShellLinkW_QueryInterface(sl, &IID_IPersistFile, (void **)&pf))) {
|
||||
ok = SUCCEEDED(IPersistFile_Save(pf, lnk, TRUE));
|
||||
IPersistFile_Release(pf);
|
||||
}
|
||||
IShellLinkW_Release(sl);
|
||||
}
|
||||
if (SUCCEEDED(init)) CoUninitialize();
|
||||
return ok;
|
||||
}
|
||||
|
||||
/* The Desktop or Start menu shortcut's path. */
|
||||
static inline int shortcut_path(int desktop, wchar_t *out, size_t n)
|
||||
{
|
||||
wchar_t dir[PATHN];
|
||||
if (!known_folder(desktop ? &FOLDERID_Desktop : &FOLDERID_Programs, dir, PATHN)) return 0;
|
||||
return swprintf(out, n, L"%ls\\%ls.lnk", dir, APP_NAME) > 0;
|
||||
}
|
||||
|
||||
/* Create (on) or remove (off) the Desktop and/or Start menu shortcut to root\NX2navi.exe. */
|
||||
static inline int set_shortcut(const wchar_t *root, int desktop, int on)
|
||||
{
|
||||
wchar_t lnk[PATHN], target[PATHN];
|
||||
if (!shortcut_path(desktop, lnk, PATHN)) return 0;
|
||||
if (!on) return DeleteFileW(lnk) || GetLastError() == ERROR_FILE_NOT_FOUND;
|
||||
swprintf(target, PATHN, L"%ls\\NX2navi.exe", root);
|
||||
return make_shortcut(lnk, target, root, L"Play and update NX2navi");
|
||||
}
|
||||
|
||||
static inline void reg_str(HKEY k, const wchar_t *name, const wchar_t *value)
|
||||
{
|
||||
RegSetValueExW(k, name, 0, REG_SZ, (const BYTE *)value, (DWORD)((wcslen(value) + 1) * sizeof(wchar_t)));
|
||||
}
|
||||
|
||||
static inline void reg_dword(HKEY k, const wchar_t *name, DWORD value)
|
||||
{
|
||||
RegSetValueExW(k, name, 0, REG_DWORD, (const BYTE *)&value, sizeof value);
|
||||
}
|
||||
|
||||
/* The entry in Settings > Apps that runs "NX2navi.exe --uninstall". */
|
||||
static inline int write_uninstall_entry(const wchar_t *root, const wchar_t *version, DWORD size_kb)
|
||||
{
|
||||
HKEY k;
|
||||
wchar_t buf[PATHN + 64];
|
||||
if (RegCreateKeyExW(HKEY_CURRENT_USER, UNINSTALL_KEY, 0, NULL, 0, KEY_SET_VALUE, NULL, &k, NULL) != ERROR_SUCCESS)
|
||||
return 0;
|
||||
reg_str(k, L"DisplayName", APP_NAME);
|
||||
reg_str(k, L"DisplayVersion", version);
|
||||
reg_str(k, L"Publisher", APP_NAME);
|
||||
reg_str(k, L"InstallLocation", root);
|
||||
swprintf(buf, PATHN + 64, L"\"%ls\\NX2navi.exe\",0", root);
|
||||
reg_str(k, L"DisplayIcon", buf);
|
||||
swprintf(buf, PATHN + 64, L"\"%ls\\NX2navi.exe\" --uninstall", root);
|
||||
reg_str(k, L"UninstallString", buf);
|
||||
reg_dword(k, L"NoModify", 1);
|
||||
reg_dword(k, L"NoRepair", 1);
|
||||
reg_dword(k, L"EstimatedSize", size_kb);
|
||||
RegCloseKey(k);
|
||||
return 1;
|
||||
}
|
||||
117
launcher/native/nx2navi.c
Normal file
117
launcher/native/nx2navi.c
Normal file
@@ -0,0 +1,117 @@
|
||||
/* NX2navi.exe: the launcher's own program, in the launcher folder (launcher/paths.py).
|
||||
|
||||
1. A Python staged by an update (python.next) replaces python\ while nothing runs from it.
|
||||
2. The release named by app\current.txt starts: python\pythonw.exe app\<version>\launcher\start.pyw with this
|
||||
program's arguments, the launcher folder as working folder and in NX2NAVI_LAUNCHER_ROOT.
|
||||
3. When that release fails to start (exit code 86 from start.pyw, or any non-zero exit within six seconds) and
|
||||
app\previous.txt names another complete release, current.txt goes back to it and it starts with
|
||||
--rolled-back <version>.
|
||||
|
||||
Its own commands, for the launcher and the setup: --desktop-shortcut on|off, --shortcuts on|off (Desktop and Start
|
||||
menu). No console window (-mwindows). */
|
||||
#include "common.h"
|
||||
|
||||
#define STARTUP_FAILED 86
|
||||
#define QUICK_MS 6000
|
||||
|
||||
static void message(const wchar_t *text)
|
||||
{
|
||||
MessageBoxW(NULL, text, APP_NAME, MB_OK | MB_ICONERROR);
|
||||
}
|
||||
|
||||
static int release_ok(const wchar_t *root, const wchar_t *ver)
|
||||
{
|
||||
wchar_t p[PATHN];
|
||||
swprintf(p, PATHN, L"%ls\\app\\%ls\\launcher\\start.pyw", root, ver);
|
||||
return ver[0] && path_exists(p);
|
||||
}
|
||||
|
||||
static void swap_python(const wchar_t *root)
|
||||
{
|
||||
wchar_t next[PATHN], cur[PATHN], old[PATHN];
|
||||
swprintf(next, PATHN, L"%ls\\python.next", root);
|
||||
if (!is_dir(next)) return;
|
||||
swprintf(cur, PATHN, L"%ls\\python", root);
|
||||
swprintf(old, PATHN, L"%ls\\python.old-%lu", root, (unsigned long)GetTickCount());
|
||||
if (is_dir(cur) && !MoveFileExW(cur, old, 0)) return; /* in use: try again at the next start */
|
||||
if (!MoveFileExW(next, cur, 0)) MoveFileExW(old, cur, 0);
|
||||
}
|
||||
|
||||
/* Start the release; returns its exit code if it ended within QUICK_MS, else -1 (still running). */
|
||||
static long start_release(const wchar_t *root, const wchar_t *ver, const wchar_t *args, const wchar_t *extra)
|
||||
{
|
||||
static wchar_t cmd[32768];
|
||||
wchar_t python[PATHN];
|
||||
swprintf(python, PATHN, L"%ls\\python\\pythonw.exe", root);
|
||||
if (!path_exists(python)) {
|
||||
message(L"The launcher's Python is missing. Run NX2navi-Setup.exe again to repair the installation.");
|
||||
return 1;
|
||||
}
|
||||
swprintf(cmd, 32768, L"\"%ls\" \"%ls\\app\\%ls\\launcher\\start.pyw\" %ls %ls", python, root, ver, args, extra);
|
||||
SetEnvironmentVariableW(L"NX2NAVI_LAUNCHER_ROOT", root);
|
||||
STARTUPINFOW si = {.cb = sizeof si};
|
||||
PROCESS_INFORMATION pi;
|
||||
if (!CreateProcessW(python, cmd, NULL, NULL, FALSE, 0, NULL, root, &si, &pi)) {
|
||||
message(L"The launcher could not be started. Run NX2navi-Setup.exe again to repair the installation.");
|
||||
return 1;
|
||||
}
|
||||
CloseHandle(pi.hThread);
|
||||
DWORD code = 0;
|
||||
long result = -1;
|
||||
if (WaitForSingleObject(pi.hProcess, QUICK_MS) == WAIT_OBJECT_0 && GetExitCodeProcess(pi.hProcess, &code))
|
||||
result = (long)code;
|
||||
CloseHandle(pi.hProcess);
|
||||
return result;
|
||||
}
|
||||
|
||||
static int shortcut_command(const wchar_t *root, int argc, wchar_t **argv)
|
||||
{
|
||||
for (int i = 1; i + 1 < argc; i++) {
|
||||
int on = !wcscmp(argv[i + 1], L"on");
|
||||
if (!wcscmp(argv[i], L"--desktop-shortcut")) return set_shortcut(root, 1, on) ? 0 : 1;
|
||||
if (!wcscmp(argv[i], L"--shortcuts")) {
|
||||
int a = set_shortcut(root, 1, on), b = set_shortcut(root, 0, on);
|
||||
return a && b ? 0 : 1;
|
||||
}
|
||||
}
|
||||
return -1;
|
||||
}
|
||||
|
||||
int WINAPI wWinMain(HINSTANCE instance, HINSTANCE previous, PWSTR args, int show)
|
||||
{
|
||||
wchar_t root[PATHN], file[PATHN], ver[128], prev[128], extra[160];
|
||||
(void)instance; (void)previous; (void)show;
|
||||
exe_dir(root, PATHN);
|
||||
if (!root[0]) return 1;
|
||||
int argc = 0;
|
||||
wchar_t **argv = CommandLineToArgvW(GetCommandLineW(), &argc);
|
||||
if (argv) {
|
||||
int r = shortcut_command(root, argc, argv);
|
||||
LocalFree(argv);
|
||||
if (r >= 0) return r;
|
||||
}
|
||||
swap_python(root);
|
||||
swprintf(file, PATHN, L"%ls\\app\\current.txt", root);
|
||||
if (!read_line(file, ver, 128) || !release_ok(root, ver)) {
|
||||
swprintf(file, PATHN, L"%ls\\app\\previous.txt", root);
|
||||
if (!read_line(file, ver, 128) || !release_ok(root, ver)) {
|
||||
message(L"NX2navi's files are incomplete. Run NX2navi-Setup.exe again to repair the installation.");
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
long code = start_release(root, ver, args ? args : L"", L"");
|
||||
if (code == -1 || code == 0) return 0;
|
||||
swprintf(file, PATHN, L"%ls\\app\\previous.txt", root);
|
||||
if ((code == STARTUP_FAILED || code > 0) && read_line(file, prev, 128) && wcscmp(prev, ver) &&
|
||||
release_ok(root, prev)) {
|
||||
swprintf(file, PATHN, L"%ls\\app\\current.txt", root);
|
||||
write_line(file, prev);
|
||||
swprintf(file, PATHN, L"%ls\\app\\previous.txt", root);
|
||||
DeleteFileW(file);
|
||||
swprintf(extra, 160, L"--rolled-back %ls", ver);
|
||||
start_release(root, prev, args ? args : L"", extra);
|
||||
return 0;
|
||||
}
|
||||
if (code == STARTUP_FAILED) message(L"NX2navi could not start. data\\logs\\crash.log in its folder says why.");
|
||||
return (int)code;
|
||||
}
|
||||
4
launcher/native/res.rc
Normal file
4
launcher/native/res.rc
Normal file
@@ -0,0 +1,4 @@
|
||||
/* The icon (the Triforce, written by tools/make_launcher.py from launcher/ui/art.py triforce_ico into the build
|
||||
folder, found through -I) and the manifest of NX2navi.exe and NX2navi-Setup.exe. */
|
||||
1 ICON "nx2navi.ico"
|
||||
1 24 "app.manifest"
|
||||
382
launcher/native/setup.c
Normal file
382
launcher/native/setup.c
Normal file
@@ -0,0 +1,382 @@
|
||||
/* NX2navi-Setup.exe: the one-click setup. This program with the launcher's folder appended as a zip.
|
||||
|
||||
It unpacks the zip into %LOCALAPPDATA%\Programs\NX2navi (no administrator rights), adds the Start menu and Desktop
|
||||
shortcuts and the entry in Settings > Apps, and starts NX2navi.exe with --from <the setup's folder>, where the
|
||||
launcher looks for the player's dump first. A small window in the launcher's colours shows while it unpacks.
|
||||
When the folder already holds a newer version than the setup carries, the setup leaves the files alone (the
|
||||
launcher updates itself) and only repairs the shortcuts.
|
||||
|
||||
The end of the file is a trailer: "NX2NSET1", the zip's offset and size (u64 little endian) and the version (24
|
||||
bytes, NUL padded), written by tools/make_launcher.py. The zip is read here with zlib's inflate, so the setup
|
||||
needs nothing from Windows beyond its own DLLs.
|
||||
|
||||
Options: --dest DIR, --no-shortcuts, --no-launch, --quiet (no window; for tests). */
|
||||
#include "common.h"
|
||||
#include <stdint.h>
|
||||
#include <stdlib.h>
|
||||
#include <zlib.h>
|
||||
|
||||
#define TRAILER 48
|
||||
#define WM_DONE (WM_APP + 1)
|
||||
|
||||
typedef struct {
|
||||
wchar_t exe[PATHN], dest[PATHN], src_dir[PATHN];
|
||||
uint64_t off, size;
|
||||
wchar_t version[32];
|
||||
int shortcuts, launch, quiet;
|
||||
int result; /* 0 ok, else an error message id */
|
||||
wchar_t error[512];
|
||||
uint64_t written;
|
||||
} Setup;
|
||||
|
||||
static Setup S;
|
||||
static HWND g_wnd;
|
||||
|
||||
static void fail_msg(const wchar_t *fmt, const wchar_t *arg)
|
||||
{
|
||||
swprintf(S.error, 512, fmt, arg ? arg : L"");
|
||||
S.result = 1;
|
||||
}
|
||||
|
||||
/* ---------------------------------------------------------------- reading our own file */
|
||||
|
||||
typedef struct { HANDLE f; uint64_t base, size; } Src;
|
||||
|
||||
static int src_read(Src *s, uint64_t at, void *buf, DWORD n)
|
||||
{
|
||||
LARGE_INTEGER li;
|
||||
DWORD got = 0;
|
||||
if (at + n > s->size) return 0;
|
||||
li.QuadPart = (LONGLONG)(s->base + at);
|
||||
if (!SetFilePointerEx(s->f, li, NULL, FILE_BEGIN)) return 0;
|
||||
return ReadFile(s->f, buf, n, &got, NULL) && got == n;
|
||||
}
|
||||
|
||||
static uint16_t le16(const unsigned char *p) { return (uint16_t)(p[0] | p[1] << 8); }
|
||||
static uint32_t le32(const unsigned char *p) { return (uint32_t)p[0] | (uint32_t)p[1] << 8 | (uint32_t)p[2] << 16 | (uint32_t)p[3] << 24; }
|
||||
static uint64_t le64(const unsigned char *p) { return (uint64_t)le32(p) | (uint64_t)le32(p + 4) << 32; }
|
||||
|
||||
static int read_trailer(void)
|
||||
{
|
||||
unsigned char t[TRAILER];
|
||||
HANDLE f = CreateFileW(S.exe, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, 0, NULL);
|
||||
if (f == INVALID_HANDLE_VALUE) return 0;
|
||||
LARGE_INTEGER size, li;
|
||||
DWORD got = 0;
|
||||
int ok = GetFileSizeEx(f, &size) && size.QuadPart > TRAILER;
|
||||
if (ok) {
|
||||
li.QuadPart = size.QuadPart - TRAILER;
|
||||
ok = SetFilePointerEx(f, li, NULL, FILE_BEGIN) && ReadFile(f, t, TRAILER, &got, NULL) && got == TRAILER &&
|
||||
!memcmp(t, "NX2NSET1", 8);
|
||||
}
|
||||
CloseHandle(f);
|
||||
if (!ok) return 0;
|
||||
S.off = le64(t + 8);
|
||||
S.size = le64(t + 16);
|
||||
char v[25];
|
||||
memcpy(v, t + 24, 24);
|
||||
v[24] = 0;
|
||||
MultiByteToWideChar(CP_UTF8, 0, v, -1, S.version, 32);
|
||||
return S.off + S.size <= (uint64_t)size.QuadPart;
|
||||
}
|
||||
|
||||
/* ---------------------------------------------------------------- unzip */
|
||||
|
||||
static int make_dirs(wchar_t *path) /* every folder above path (path is a file) */
|
||||
{
|
||||
for (wchar_t *p = path + 3; *p; p++) {
|
||||
if (*p == L'\\') {
|
||||
*p = 0;
|
||||
if (!is_dir(path) && !CreateDirectoryW(path, NULL) && GetLastError() != ERROR_ALREADY_EXISTS) {
|
||||
*p = L'\\';
|
||||
return 0;
|
||||
}
|
||||
*p = L'\\';
|
||||
}
|
||||
}
|
||||
return 1;
|
||||
}
|
||||
|
||||
static int safe_name(const char *name)
|
||||
{
|
||||
if (!name[0] || name[0] == '/' || name[0] == '\\' || strchr(name, ':')) return 0;
|
||||
for (const char *p = name; *p; p++)
|
||||
if ((p == name || p[-1] == '/' || p[-1] == '\\') && p[0] == '.' && p[1] == '.' &&
|
||||
(p[2] == '/' || p[2] == '\\' || !p[2]))
|
||||
return 0;
|
||||
return 1;
|
||||
}
|
||||
|
||||
static int extract_entry(Src *z, uint64_t local, uint32_t csize, uint32_t usize, uint16_t method, uint32_t crc,
|
||||
const wchar_t *target)
|
||||
{
|
||||
unsigned char lh[30];
|
||||
if (!src_read(z, local, lh, 30) || le32(lh) != 0x04034b50) return 0;
|
||||
uint64_t data = local + 30 + le16(lh + 26) + le16(lh + 28);
|
||||
HANDLE out = CreateFileW(target, GENERIC_WRITE, 0, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
|
||||
if (out == INVALID_HANDLE_VALUE) return 0;
|
||||
static unsigned char in[1 << 16], buf[1 << 17];
|
||||
uLong check = crc32(0L, Z_NULL, 0);
|
||||
int ok = 1;
|
||||
uint64_t left = csize, pos = data, total = 0;
|
||||
z_stream st;
|
||||
memset(&st, 0, sizeof st);
|
||||
if (method == 8 && inflateInit2(&st, -MAX_WBITS) != Z_OK) ok = 0;
|
||||
while (ok && (left || (method == 8 && st.avail_in))) {
|
||||
DWORD n = (DWORD)(left < sizeof in ? left : sizeof in);
|
||||
if (n && !src_read(z, pos, in, n)) { ok = 0; break; }
|
||||
pos += n;
|
||||
left -= n;
|
||||
if (method == 0) {
|
||||
DWORD put;
|
||||
check = crc32(check, in, n);
|
||||
ok = WriteFile(out, in, n, &put, NULL) && put == n;
|
||||
total += n;
|
||||
continue;
|
||||
}
|
||||
st.next_in = in;
|
||||
st.avail_in = n;
|
||||
do {
|
||||
st.next_out = buf;
|
||||
st.avail_out = sizeof buf;
|
||||
int r = inflate(&st, Z_NO_FLUSH);
|
||||
if (r != Z_OK && r != Z_STREAM_END && !(r == Z_BUF_ERROR && st.avail_in == 0)) { ok = 0; break; }
|
||||
DWORD have = (DWORD)(sizeof buf - st.avail_out), put;
|
||||
if (have) {
|
||||
check = crc32(check, buf, have);
|
||||
if (!WriteFile(out, buf, have, &put, NULL) || put != have) { ok = 0; break; }
|
||||
total += have;
|
||||
}
|
||||
if (r == Z_STREAM_END) { left = 0; st.avail_in = 0; break; }
|
||||
} while (st.avail_out == 0);
|
||||
}
|
||||
if (method == 8) inflateEnd(&st);
|
||||
CloseHandle(out);
|
||||
S.written += total;
|
||||
return ok && total == usize && check == crc;
|
||||
}
|
||||
|
||||
static int unzip(void)
|
||||
{
|
||||
Src z;
|
||||
unsigned char eocd[22];
|
||||
z.f = CreateFileW(S.exe, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, 0, NULL);
|
||||
if (z.f == INVALID_HANDLE_VALUE) { fail_msg(L"The setup cannot read itself.", NULL); return 0; }
|
||||
z.base = S.off;
|
||||
z.size = S.size;
|
||||
/* the end-of-central-directory record: the zip has no comment, so it is the last 22 bytes */
|
||||
if (!src_read(&z, z.size - 22, eocd, 22) || le32(eocd) != 0x06054b50) {
|
||||
CloseHandle(z.f);
|
||||
fail_msg(L"The setup is damaged. Download it again.", NULL);
|
||||
return 0;
|
||||
}
|
||||
uint32_t count = le16(eocd + 10), cd_size = le32(eocd + 12), cd_off = le32(eocd + 16);
|
||||
unsigned char *cd = malloc(cd_size);
|
||||
if (!cd || !src_read(&z, cd_off, cd, cd_size)) {
|
||||
free(cd);
|
||||
CloseHandle(z.f);
|
||||
fail_msg(L"The setup is damaged. Download it again.", NULL);
|
||||
return 0;
|
||||
}
|
||||
wchar_t target[PATHN], wname[PATHN];
|
||||
char name[PATHN];
|
||||
uint32_t at = 0;
|
||||
int ok = 1;
|
||||
for (uint32_t i = 0; ok && i < count; i++) {
|
||||
if (at + 46 > cd_size || le32(cd + at) != 0x02014b50) { ok = 0; break; }
|
||||
uint16_t method = le16(cd + at + 10), nlen = le16(cd + at + 28), xlen = le16(cd + at + 30),
|
||||
clen = le16(cd + at + 32);
|
||||
uint32_t crc = le32(cd + at + 16), csize = le32(cd + at + 20), usize = le32(cd + at + 24),
|
||||
local = le32(cd + at + 42);
|
||||
if (nlen >= PATHN || at + 46 + nlen > cd_size || (method != 0 && method != 8)) { ok = 0; break; }
|
||||
memcpy(name, cd + at + 46, nlen);
|
||||
name[nlen] = 0;
|
||||
at += 46 + nlen + xlen + clen;
|
||||
if (!safe_name(name)) { ok = 0; break; }
|
||||
MultiByteToWideChar(CP_UTF8, 0, name, -1, wname, PATHN);
|
||||
for (wchar_t *p = wname; *p; p++)
|
||||
if (*p == L'/') *p = L'\\';
|
||||
swprintf(target, PATHN, L"%ls\\%ls", S.dest, wname);
|
||||
size_t tl = wcslen(target);
|
||||
if (tl && target[tl - 1] == L'\\') {
|
||||
target[tl - 1] = 0;
|
||||
ok = make_dirs(wcscat(target, L"\\")) && (is_dir(target) || CreateDirectoryW(target, NULL));
|
||||
continue;
|
||||
}
|
||||
if (!make_dirs(target)) { ok = 0; break; }
|
||||
if (!extract_entry(&z, local, csize, usize, method, crc, target)) {
|
||||
fail_msg(L"%ls could not be written. Close NX2navi if it is open, then run the setup again.", target);
|
||||
free(cd);
|
||||
CloseHandle(z.f);
|
||||
return 0;
|
||||
}
|
||||
if (g_wnd && (i & 31) == 0) InvalidateRect(g_wnd, NULL, FALSE);
|
||||
}
|
||||
free(cd);
|
||||
CloseHandle(z.f);
|
||||
if (!ok) fail_msg(L"The setup is damaged. Download it again.", NULL);
|
||||
return ok;
|
||||
}
|
||||
|
||||
/* ---------------------------------------------------------------- the work */
|
||||
|
||||
static DWORD WINAPI work(void *arg)
|
||||
{
|
||||
wchar_t cur[PATHN], have[64];
|
||||
(void)arg;
|
||||
swprintf(cur, PATHN, L"%ls\\app\\current.txt", S.dest);
|
||||
int newer_installed = read_line(cur, have, 64) && version_cmp(have, S.version) > 0;
|
||||
wchar_t probe[PATHN];
|
||||
swprintf(probe, PATHN, L"%ls\\x", S.dest);
|
||||
if (!make_dirs(probe)) {
|
||||
fail_msg(L"The folder %ls cannot be created.", S.dest);
|
||||
} else if (!newer_installed) {
|
||||
unzip();
|
||||
}
|
||||
if (!S.result) {
|
||||
if (S.shortcuts) {
|
||||
set_shortcut(S.dest, 0, 1);
|
||||
set_shortcut(S.dest, 1, 1);
|
||||
}
|
||||
write_uninstall_entry(S.dest, newer_installed ? have : S.version, (DWORD)(S.written / 1024 + 1));
|
||||
}
|
||||
if (g_wnd) PostMessageW(g_wnd, WM_DONE, 0, 0);
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* ---------------------------------------------------------------- the window */
|
||||
|
||||
static LRESULT CALLBACK wndproc(HWND w, UINT m, WPARAM wp, LPARAM lp)
|
||||
{
|
||||
switch (m) {
|
||||
case WM_PAINT: {
|
||||
PAINTSTRUCT ps;
|
||||
HDC dc = BeginPaint(w, &ps);
|
||||
RECT r;
|
||||
GetClientRect(w, &r);
|
||||
HBRUSH bg = CreateSolidBrush(RGB(0x0c, 0x2e, 0x1b)), edge = CreateSolidBrush(RGB(0xa9, 0x87, 0x4f));
|
||||
FillRect(dc, &r, edge);
|
||||
RECT in = {r.left + 2, r.top + 2, r.right - 2, r.bottom - 2};
|
||||
FillRect(dc, &in, bg);
|
||||
/* the Triforce: three gold triangles */
|
||||
HBRUSH gold = CreateSolidBrush(RGB(0xe8, 0xaf, 0x3d));
|
||||
HPEN none = CreatePen(PS_NULL, 0, 0);
|
||||
HGDIOBJ ob = SelectObject(dc, gold), op = SelectObject(dc, none);
|
||||
int cx = 52, top = 28, s = 44;
|
||||
POINT t1[3] = {{cx, top}, {cx + s / 4, top + s * 433 / 1000}, {cx - s / 4, top + s * 433 / 1000}};
|
||||
POINT t2[3] = {{cx - s / 4, top + s * 433 / 1000 + 1}, {cx, top + s * 866 / 1000}, {cx - s / 2, top + s * 866 / 1000}};
|
||||
POINT t3[3] = {{cx + s / 4, top + s * 433 / 1000 + 1}, {cx + s / 2, top + s * 866 / 1000}, {cx, top + s * 866 / 1000}};
|
||||
Polygon(dc, t1, 3);
|
||||
Polygon(dc, t2, 3);
|
||||
Polygon(dc, t3, 3);
|
||||
SelectObject(dc, ob);
|
||||
SelectObject(dc, op);
|
||||
HFONT big = CreateFontW(-26, 0, 0, 0, FW_SEMIBOLD, 0, 0, 0, DEFAULT_CHARSET, 0, 0, CLEARTYPE_QUALITY, 0, L"Segoe UI");
|
||||
HFONT small = CreateFontW(-16, 0, 0, 0, FW_NORMAL, 0, 0, 0, DEFAULT_CHARSET, 0, 0, CLEARTYPE_QUALITY, 0, L"Segoe UI");
|
||||
SetBkMode(dc, TRANSPARENT);
|
||||
SetTextColor(dc, RGB(0xf3, 0xec, 0xdc));
|
||||
HGDIOBJ of = SelectObject(dc, big);
|
||||
TextOutW(dc, 96, 30, APP_NAME, (int)wcslen(APP_NAME));
|
||||
SelectObject(dc, small);
|
||||
SetTextColor(dc, RGB(0xe9, 0xdc, 0xc0));
|
||||
wchar_t line[96];
|
||||
swprintf(line, 96, L"Installing version %ls\x2026 %llu MB", S.version, (unsigned long long)(S.written >> 20));
|
||||
TextOutW(dc, 96, 66, line, (int)wcslen(line));
|
||||
SelectObject(dc, of);
|
||||
DeleteObject(big);
|
||||
DeleteObject(small);
|
||||
DeleteObject(gold);
|
||||
DeleteObject(none);
|
||||
DeleteObject(bg);
|
||||
DeleteObject(edge);
|
||||
EndPaint(w, &ps);
|
||||
return 0;
|
||||
}
|
||||
case WM_TIMER:
|
||||
InvalidateRect(w, NULL, FALSE);
|
||||
return 0;
|
||||
case WM_DONE:
|
||||
DestroyWindow(w);
|
||||
return 0;
|
||||
case WM_DESTROY:
|
||||
PostQuitMessage(0);
|
||||
return 0;
|
||||
}
|
||||
return DefWindowProcW(w, m, wp, lp);
|
||||
}
|
||||
|
||||
static void run_with_window(HINSTANCE inst)
|
||||
{
|
||||
WNDCLASSW wc = {0};
|
||||
wc.lpfnWndProc = wndproc;
|
||||
wc.hInstance = inst;
|
||||
wc.hCursor = LoadCursor(NULL, IDC_WAIT);
|
||||
wc.hIcon = LoadIconW(inst, MAKEINTRESOURCEW(1));
|
||||
wc.lpszClassName = L"NX2naviSetup";
|
||||
RegisterClassW(&wc);
|
||||
int w = 440, h = 120;
|
||||
int x = (GetSystemMetrics(SM_CXSCREEN) - w) / 2, y = (GetSystemMetrics(SM_CYSCREEN) - h) / 3;
|
||||
g_wnd = CreateWindowExW(WS_EX_APPWINDOW, wc.lpszClassName, L"NX2navi Setup", WS_POPUP | WS_VISIBLE, x, y, w, h,
|
||||
NULL, NULL, inst, NULL);
|
||||
SetTimer(g_wnd, 1, 200, NULL);
|
||||
HANDLE t = CreateThread(NULL, 0, work, NULL, 0, NULL);
|
||||
MSG msg;
|
||||
while (GetMessageW(&msg, NULL, 0, 0) > 0) {
|
||||
TranslateMessage(&msg);
|
||||
DispatchMessageW(&msg);
|
||||
}
|
||||
WaitForSingleObject(t, INFINITE);
|
||||
CloseHandle(t);
|
||||
}
|
||||
|
||||
int WINAPI wWinMain(HINSTANCE inst, HINSTANCE prev, PWSTR cmdline, int show)
|
||||
{
|
||||
(void)prev; (void)cmdline; (void)show;
|
||||
GetModuleFileNameW(NULL, S.exe, PATHN);
|
||||
exe_dir(S.src_dir, PATHN);
|
||||
S.shortcuts = S.launch = 1;
|
||||
S.dest[0] = 0;
|
||||
int argc = 0;
|
||||
wchar_t **argv = CommandLineToArgvW(GetCommandLineW(), &argc);
|
||||
for (int i = 1; argv && i < argc; i++) {
|
||||
if (!wcscmp(argv[i], L"--dest") && i + 1 < argc) {
|
||||
GetFullPathNameW(argv[++i], PATHN, S.dest, NULL);
|
||||
} else if (!wcscmp(argv[i], L"--no-shortcuts")) S.shortcuts = 0;
|
||||
else if (!wcscmp(argv[i], L"--no-launch")) S.launch = 0;
|
||||
else if (!wcscmp(argv[i], L"--quiet")) S.quiet = 1;
|
||||
}
|
||||
if (argv) LocalFree(argv);
|
||||
if (!S.dest[0]) {
|
||||
wchar_t base[PATHN];
|
||||
if (!known_folder(&FOLDERID_UserProgramFiles, base, PATHN)) {
|
||||
if (!GetEnvironmentVariableW(L"LOCALAPPDATA", base, PATHN)) return 1;
|
||||
wcscat(base, L"\\Programs");
|
||||
}
|
||||
swprintf(S.dest, PATHN, L"%ls\\%ls", base, APP_NAME);
|
||||
}
|
||||
size_t dl = wcslen(S.dest);
|
||||
while (dl > 3 && (S.dest[dl - 1] == L'\\' || S.dest[dl - 1] == L'/')) S.dest[--dl] = 0;
|
||||
if (!read_trailer()) {
|
||||
MessageBoxW(NULL, L"This setup is incomplete. Download NX2navi-Setup.exe again.", APP_NAME, MB_ICONERROR);
|
||||
return 1;
|
||||
}
|
||||
if (S.quiet) work(NULL);
|
||||
else run_with_window(inst);
|
||||
if (S.result) {
|
||||
if (!S.quiet) MessageBoxW(NULL, S.error, APP_NAME, MB_OK | MB_ICONERROR);
|
||||
return 2;
|
||||
}
|
||||
if (S.launch) {
|
||||
static wchar_t cmd[PATHN * 3];
|
||||
wchar_t exe[PATHN];
|
||||
swprintf(exe, PATHN, L"%ls\\NX2navi.exe", S.dest);
|
||||
swprintf(cmd, PATHN * 3, L"\"%ls\" --from \"%ls\"", exe, S.src_dir);
|
||||
STARTUPINFOW si = {.cb = sizeof si};
|
||||
PROCESS_INFORMATION pi;
|
||||
if (CreateProcessW(exe, cmd, NULL, NULL, FALSE, 0, NULL, S.dest, &si, &pi)) {
|
||||
CloseHandle(pi.hThread);
|
||||
CloseHandle(pi.hProcess);
|
||||
}
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
128
launcher/net.py
Normal file
128
launcher/net.py
Normal file
@@ -0,0 +1,128 @@
|
||||
"""HTTPS for the launcher: JSON requests and file downloads that resume, report progress, can be cancelled and are
|
||||
checked against a sha256. Standard library only (urllib); Windows' own certificate store verifies the servers."""
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import ssl
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
from . import NAME
|
||||
|
||||
CHUNK = 1 << 20
|
||||
|
||||
|
||||
class Cancelled(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class DownloadError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def user_agent(version='dev'):
|
||||
return f'{NAME}-Launcher/{version}'
|
||||
|
||||
|
||||
_CTX = None
|
||||
|
||||
|
||||
def _context():
|
||||
global _CTX
|
||||
if _CTX is None:
|
||||
_CTX = ssl.create_default_context()
|
||||
return _CTX
|
||||
|
||||
|
||||
def request(url, headers=None, timeout=30, version='dev'):
|
||||
h = {'User-Agent': user_agent(version)}
|
||||
h.update(headers or {})
|
||||
req = urllib.request.Request(url, headers=h)
|
||||
ctx = _context() if url.startswith('https:') else None
|
||||
return urllib.request.urlopen(req, timeout=timeout, context=ctx)
|
||||
|
||||
|
||||
def get_bytes(url, headers=None, timeout=30, version='dev', limit=64 << 20):
|
||||
with request(url, headers, timeout, version) as r:
|
||||
data = r.read(limit + 1)
|
||||
if len(data) > limit:
|
||||
raise DownloadError(f'{url}: more than {limit >> 20} MB')
|
||||
return data
|
||||
|
||||
|
||||
def get_json(url, headers=None, timeout=30, version='dev'):
|
||||
return json.loads(get_bytes(url, headers, timeout, version).decode('utf-8'))
|
||||
|
||||
|
||||
def sha256_file(path):
|
||||
h = hashlib.sha256()
|
||||
with open(path, 'rb') as f:
|
||||
while b := f.read(CHUNK):
|
||||
h.update(b)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def download(url, dest, sha256=None, size=None, progress=None, cancel=None, headers=None, retries=4, version='dev'):
|
||||
"""Fetch url to dest through dest.part, resuming a part file a cut connection left. progress(done, total) after
|
||||
every chunk; cancel() returning True raises Cancelled. A wrong size or sha256 deletes the file and raises
|
||||
DownloadError. A dest already holding the expected sha256 is kept without a request."""
|
||||
dest = Path(dest)
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
if sha256 and dest.is_file() and sha256_file(dest) == sha256.lower():
|
||||
if progress:
|
||||
progress(dest.stat().st_size, dest.stat().st_size)
|
||||
return dest
|
||||
part = dest.with_name(dest.name + '.part')
|
||||
last_error = None
|
||||
for attempt in range(retries):
|
||||
have = part.stat().st_size if part.is_file() else 0
|
||||
h = dict(headers or {})
|
||||
if have:
|
||||
h['Range'] = f'bytes={have}-'
|
||||
try:
|
||||
with request(url, h, 60, version) as r:
|
||||
status = getattr(r, 'status', 200)
|
||||
if have and status != 206: # the server ignores Range: start over
|
||||
have = 0
|
||||
total = int(r.headers.get('Content-Length') or 0) + have or size or 0
|
||||
with open(part, 'ab' if have else 'wb') as f:
|
||||
done = have
|
||||
while True:
|
||||
if cancel and cancel():
|
||||
raise Cancelled()
|
||||
b = r.read(CHUNK)
|
||||
if not b:
|
||||
break
|
||||
f.write(b)
|
||||
done += len(b)
|
||||
if progress:
|
||||
progress(done, total or done)
|
||||
break
|
||||
except Cancelled:
|
||||
raise
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code == 416 and have: # the part file is complete or longer than the file: start over
|
||||
part.unlink(missing_ok=True)
|
||||
last_error = e
|
||||
continue
|
||||
if e.code in (401, 403, 404, 410):
|
||||
raise DownloadError(f'{url}: HTTP {e.code}') from e
|
||||
last_error = e
|
||||
except (urllib.error.URLError, OSError, TimeoutError) as e:
|
||||
last_error = e
|
||||
time.sleep(min(2 ** attempt, 16))
|
||||
else:
|
||||
raise DownloadError(f'{url}: {last_error}')
|
||||
got = part.stat().st_size
|
||||
if size and got != size:
|
||||
part.unlink(missing_ok=True)
|
||||
raise DownloadError(f'{url}: {got} bytes, expected {size}')
|
||||
if sha256:
|
||||
digest = sha256_file(part)
|
||||
if digest != sha256.lower():
|
||||
part.unlink(missing_ok=True)
|
||||
raise DownloadError(f'{url}: sha256 {digest}, expected {sha256.lower()}')
|
||||
os.replace(part, dest)
|
||||
return dest
|
||||
115
launcher/paths.py
Normal file
115
launcher/paths.py
Normal file
@@ -0,0 +1,115 @@
|
||||
"""Where the launcher keeps everything.
|
||||
|
||||
An installed launcher (NX2navi-Setup.exe puts it in %LOCALAPPDATA%\\Programs\\NX2navi) is one folder:
|
||||
|
||||
NX2navi.exe launcher/native/nx2navi.c: starts python\\pythonw.exe on app\\<current>\\launcher\\start.pyw
|
||||
python/ the embeddable Python with tkinter, numpy, lz4 and zstandard (tools/make_launcher.py)
|
||||
app/current.txt the version the stub starts; app/previous.txt the one before it, kept for a rollback
|
||||
app/<version>/ one release: launcher/, runtime/, recomp/, tools/, bin/, VERSION, toolchain.lock.json
|
||||
pydeps/ zbic, downloaded from PyPI on this PC (launcher/toolchain.py ensure_zbic)
|
||||
toolchain/msys64/ucrt64 MSYS2's clang and libraries, downloaded from the MSYS2 repository on this PC
|
||||
toolchain/pkgs/ the downloaded package files, kept while the lock names them
|
||||
data/launcher.json settings; data/logs/ logs; data/cache/ the fonts read from the player's game
|
||||
game/ the built game, unless the player picks another folder
|
||||
|
||||
Run from a checkout of the repository (python -m launcher), the app folder is the repository and everything else goes
|
||||
to build/launcher-dev/, which git ignores.
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
APP = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def _is_release_app(app):
|
||||
return app.parent.name == 'app' and (app.parent / 'current.txt').is_file()
|
||||
|
||||
|
||||
def ansi_safe(path):
|
||||
"""True when the path survives the Windows ANSI code page: the MSYS2 tools and the game's own build read their
|
||||
arguments through it, so a folder name outside it breaks the build. Always True elsewhere."""
|
||||
if os.name != 'nt':
|
||||
return True
|
||||
try:
|
||||
str(path).encode('mbcs', errors='strict')
|
||||
return True
|
||||
except (UnicodeEncodeError, LookupError):
|
||||
return False
|
||||
|
||||
|
||||
class Layout:
|
||||
def __init__(self, root=None, app=None):
|
||||
self.app = Path(app or APP)
|
||||
self.dev = not _is_release_app(self.app)
|
||||
env = os.environ.get('NX2NAVI_LAUNCHER_ROOT')
|
||||
if root:
|
||||
self.root = Path(root)
|
||||
elif env:
|
||||
self.root = Path(env)
|
||||
elif not self.dev:
|
||||
self.root = self.app.parents[1]
|
||||
else:
|
||||
self.root = self.app / 'build' / 'launcher-dev'
|
||||
self.apps = self.root / 'app'
|
||||
self.python = self.root / 'python'
|
||||
self.stub = self.root / 'NX2navi.exe'
|
||||
self.pydeps = self.root / 'pydeps'
|
||||
self.data = self.root / 'data'
|
||||
self.logs = self.data / 'logs'
|
||||
self.cache = self.data / 'cache'
|
||||
self.downloads = self.data / 'downloads'
|
||||
self.settings = self.data / 'launcher.json'
|
||||
tool_root = self.root
|
||||
if not ansi_safe(tool_root):
|
||||
tool_root = Path(os.environ.get('SystemDrive', 'C:') + '\\') / 'NX2navi'
|
||||
self.toolchain = tool_root / 'toolchain'
|
||||
self.msys_root = self.toolchain / 'msys64'
|
||||
self.msys = self.msys_root / 'ucrt64'
|
||||
self.pkgs = self.toolchain / 'pkgs'
|
||||
self.tools = self.app / 'tools'
|
||||
self.make_game = self.tools / 'make_game.py'
|
||||
self.lock_file = self.app / 'toolchain.lock.json'
|
||||
self.feed_file = self.app / 'feed.json'
|
||||
|
||||
@property
|
||||
def version(self):
|
||||
v = self.app / 'VERSION'
|
||||
return v.read_text(encoding='utf-8').strip() if v.is_file() else '0.0.0'
|
||||
|
||||
def default_game_dir(self):
|
||||
"""<root>/game, or a Games folder on the fixed drive with the most free space when the root's drive has
|
||||
less than 40 GB free or the root's path would break the build."""
|
||||
here = self.root / 'game'
|
||||
try:
|
||||
import shutil
|
||||
if shutil.disk_usage(self.root.anchor or self.root).free >= 40e9 and ansi_safe(here):
|
||||
return here
|
||||
except OSError:
|
||||
return here
|
||||
best = None
|
||||
if os.name == 'nt':
|
||||
import shutil
|
||||
import string
|
||||
for letter in string.ascii_uppercase[2:]:
|
||||
drive = Path(f'{letter}:\\')
|
||||
try:
|
||||
free = shutil.disk_usage(drive).free
|
||||
except OSError:
|
||||
continue
|
||||
if best is None or free > best[0]:
|
||||
best = (free, drive / 'Games' / 'NX2navi')
|
||||
return best[1] if best else here
|
||||
|
||||
def python_for_builds(self):
|
||||
"""The console python.exe beside the launcher's pythonw.exe (no window: the build starts it with
|
||||
CREATE_NO_WINDOW), or this interpreter in a checkout."""
|
||||
exe = Path(sys.executable)
|
||||
if exe.name.lower() == 'pythonw.exe' and (exe.parent / 'python.exe').is_file():
|
||||
return str(exe.parent / 'python.exe')
|
||||
return str(exe)
|
||||
|
||||
def ensure(self):
|
||||
for d in (self.data, self.logs, self.cache, self.downloads):
|
||||
d.mkdir(parents=True, exist_ok=True)
|
||||
return self
|
||||
43
launcher/settings.py
Normal file
43
launcher/settings.py
Normal file
@@ -0,0 +1,43 @@
|
||||
"""data/launcher.json: the player's choices and what the launcher remembers between runs."""
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
DEFAULTS = {
|
||||
'game_dir': '', # the built game; empty until the first install picks one
|
||||
'dump': '', # the .nup (or the folder with exefs/ and romfs/) the game was built from
|
||||
'auto_check': True, # look for a new version when the launcher opens
|
||||
'channel': 'stable', # 'stable' or 'preview' (GitHub pre-releases)
|
||||
'hide_while_playing': True, # hide the window while the game runs, show it again when it exits
|
||||
'keep_cache': True, # keep build-cache/ (the lift and objects) so an update recompiles only changes
|
||||
'desktop_shortcut': True,
|
||||
'feed_repo': '', # owner/name overriding feed.json, for testing a fork's releases
|
||||
'github_token': '', # only for a private release repository; stored as plain text
|
||||
'last_check': 0.0,
|
||||
'last_seen': '', # the newest version the player was told about
|
||||
}
|
||||
|
||||
|
||||
class Settings(dict):
|
||||
def __init__(self, path):
|
||||
super().__init__(DEFAULTS)
|
||||
self.path = Path(path)
|
||||
try:
|
||||
data = json.loads(self.path.read_text(encoding='utf-8'))
|
||||
if isinstance(data, dict):
|
||||
self.update({k: v for k, v in data.items() if k in DEFAULTS and type(v) is type(DEFAULTS[k])})
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
|
||||
def save(self):
|
||||
self.path.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = self.path.with_name(self.path.name + '.tmp')
|
||||
tmp.write_text(json.dumps(dict(self), indent=1), encoding='utf-8')
|
||||
os.replace(tmp, self.path)
|
||||
|
||||
def reset(self):
|
||||
"""Default Settings: the choices go back to their defaults; the game folder, dump and history stay."""
|
||||
keep = {k: self[k] for k in ('game_dir', 'dump', 'last_check', 'last_seen', 'feed_repo', 'github_token')}
|
||||
self.clear()
|
||||
self.update(DEFAULTS)
|
||||
self.update(keep)
|
||||
45
launcher/start.pyw
Normal file
45
launcher/start.pyw
Normal file
@@ -0,0 +1,45 @@
|
||||
"""What NX2navi.exe runs: python\\pythonw.exe app\\<version>\\launcher\\start.pyw [arguments].
|
||||
|
||||
The embeddable Python puts only its ._pth entries on sys.path, so the release folder (the parent of launcher/) goes
|
||||
first here. A failure before the window is up is written to data/logs/crash.log, shown in a message box and ends the
|
||||
process with exit code 86, which tells the stub to go back to the previous version (launcher/native/nx2navi.c).
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
import traceback
|
||||
|
||||
APP = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
sys.path.insert(0, APP)
|
||||
STARTUP_FAILED = 86
|
||||
|
||||
|
||||
def crash(text):
|
||||
root = os.environ.get('NX2NAVI_LAUNCHER_ROOT') or os.path.dirname(os.path.dirname(APP))
|
||||
try:
|
||||
logs = os.path.join(root, 'data', 'logs')
|
||||
os.makedirs(logs, exist_ok=True)
|
||||
with open(os.path.join(logs, 'crash.log'), 'a', encoding='utf-8') as f:
|
||||
f.write(text + '\n')
|
||||
except OSError:
|
||||
pass
|
||||
try:
|
||||
import ctypes
|
||||
ctypes.windll.user32.MessageBoxW(None, 'NX2navi could not start.\n\n' + text[-1500:], 'NX2navi', 0x10)
|
||||
except Exception:
|
||||
print(text, file=sys.stderr)
|
||||
|
||||
|
||||
try:
|
||||
from launcher.ui.app import main
|
||||
except Exception:
|
||||
crash(traceback.format_exc())
|
||||
sys.exit(STARTUP_FAILED)
|
||||
|
||||
try:
|
||||
code = main()
|
||||
except SystemExit:
|
||||
raise
|
||||
except Exception:
|
||||
crash(traceback.format_exc())
|
||||
sys.exit(1)
|
||||
sys.exit(code or 0)
|
||||
288
launcher/toolchain.py
Normal file
288
launcher/toolchain.py
Normal file
@@ -0,0 +1,288 @@
|
||||
"""The build tools, installed on the player's PC: MSYS2's UCRT64 clang and the libraries tools/make_game.py links
|
||||
(the packages README-BuildKit.md lists), and the zbic Python package.
|
||||
|
||||
No MSYS2 shell and no pacman: the packages and everything they depend on are read from the repository's database,
|
||||
downloaded, checked against the sha256 the database (or the release's lock) names, and unpacked into
|
||||
toolchain/msys64/ucrt64 as pacman would lay them out. A release pins the exact package files in
|
||||
toolchain.lock.json (make_lock, written by tools/make_launcher.py), so every player builds with the compiler the
|
||||
release was tested with; the repository keeps old package files. A checkout without a lock resolves the newest
|
||||
packages instead.
|
||||
|
||||
zbic (GPL-2.0) unpacks the game's compressed program files. The release does not carry it: the launcher downloads the
|
||||
wheel for its own Python from PyPI and unpacks it into pydeps/, as `pip install zbic` would.
|
||||
"""
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
import tarfile
|
||||
import time
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
from . import net
|
||||
|
||||
REPO = 'https://repo.msys2.org/mingw/ucrt64/'
|
||||
MIRRORS = (REPO, 'https://mirror.msys2.org/mingw/ucrt64/')
|
||||
PREFIX = 'mingw-w64-ucrt-x86_64-'
|
||||
ROOTS = ('clang', 'gcc', 'sdl3', 'opus', 'libvpx', 'dav1d', 'zstd', 'vulkan-headers', 'vulkan-loader',
|
||||
'vulkan-memory-allocator')
|
||||
REQUIRED = ('bin/clang.exe', 'bin/clang++.exe', 'bin/SDL3.dll', 'lib/libSDL3.dll.a', 'include/vma')
|
||||
SKIP_SHARE = ('ucrt64/share/doc/', 'ucrt64/share/man/', 'ucrt64/share/info/', 'ucrt64/share/locale/',
|
||||
'ucrt64/share/gtk-doc/', 'ucrt64/share/licenses/')
|
||||
ZBIC = ('zbic', '1.0.1.post1')
|
||||
LOCK_FORMAT = 1
|
||||
|
||||
|
||||
class ToolchainError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- the package database
|
||||
|
||||
def _zstd_decompress_stream(data):
|
||||
try:
|
||||
from compression import zstd # Python 3.14
|
||||
return zstd.decompress(data)
|
||||
except ImportError:
|
||||
import zstandard
|
||||
return zstandard.ZstdDecompressor().decompressobj().decompress(data)
|
||||
|
||||
|
||||
def _open_tar(path_or_bytes):
|
||||
"""A tarfile over a .tar.zst (or plain/gzip tar) file or bytes."""
|
||||
data = path_or_bytes if isinstance(path_or_bytes, (bytes, bytearray)) else Path(path_or_bytes).read_bytes()
|
||||
if data[:4] == b'\x28\xb5\x2f\xfd':
|
||||
data = _zstd_decompress_stream(data)
|
||||
return tarfile.open(fileobj=io.BytesIO(data), mode='r:*')
|
||||
|
||||
|
||||
def parse_db(data):
|
||||
"""{name: desc} for a pacman sync database (ucrt64.db): each desc is {FIELD: [lines]}."""
|
||||
out = {}
|
||||
with _open_tar(data) as t:
|
||||
for m in t.getmembers():
|
||||
if not m.name.endswith('/desc'):
|
||||
continue
|
||||
desc, field = {}, None
|
||||
for line in t.extractfile(m).read().decode('utf-8').splitlines():
|
||||
if line.startswith('%') and line.endswith('%') and len(line) > 2:
|
||||
field = line[1:-1]
|
||||
desc[field] = []
|
||||
elif line and field:
|
||||
desc[field].append(line)
|
||||
if desc.get('NAME'):
|
||||
out[desc['NAME'][0]] = desc
|
||||
return out
|
||||
|
||||
|
||||
def _dep_name(dep):
|
||||
return re.split(r'[<>=:]', dep, maxsplit=1)[0].strip()
|
||||
|
||||
|
||||
def resolve(db, roots=ROOTS):
|
||||
"""Every package the roots need, roots first; names a package provides resolve to the package."""
|
||||
provides = {}
|
||||
for name, d in db.items():
|
||||
for p in d.get('PROVIDES', []):
|
||||
provides.setdefault(_dep_name(p), name)
|
||||
order, seen = [], set()
|
||||
todo = [r if r.startswith(PREFIX) else PREFIX + r for r in roots]
|
||||
while todo:
|
||||
n = _dep_name(todo.pop(0))
|
||||
n = n if n in db else provides.get(n, n)
|
||||
if n in seen:
|
||||
continue
|
||||
if n not in db:
|
||||
raise ToolchainError(f'the package database has no {n}')
|
||||
seen.add(n)
|
||||
order.append(n)
|
||||
todo += db[n].get('DEPENDS', [])
|
||||
pkgs = []
|
||||
for n in order:
|
||||
d = db[n]
|
||||
pkgs.append({'name': n, 'version': d['VERSION'][0], 'filename': d['FILENAME'][0],
|
||||
'sha256': d['SHA256SUM'][0], 'csize': int(d['CSIZE'][0]), 'isize': int(d.get('ISIZE', ['0'])[0])})
|
||||
return pkgs
|
||||
|
||||
|
||||
def make_lock(repo=REPO, roots=ROOTS, version='dev'):
|
||||
"""The lock a release ships: the resolved package files of the repository's current database."""
|
||||
db = parse_db(net.get_bytes(repo + 'ucrt64.db', version=version))
|
||||
return {'format': LOCK_FORMAT, 'repo': repo, 'roots': list(roots), 'packages': resolve(db, roots),
|
||||
'zbic': {'name': ZBIC[0], 'version': ZBIC[1]}, 'made': time.strftime('%Y-%m-%d')}
|
||||
|
||||
|
||||
def lock_digest(lock):
|
||||
files = sorted(p['filename'] + ':' + p['sha256'] for p in lock['packages'])
|
||||
return hashlib.sha256('\n'.join(files).encode()).hexdigest()
|
||||
|
||||
|
||||
def load_lock(layout, version='dev'):
|
||||
"""The release's toolchain.lock.json, or a lock resolved from the live database in a checkout."""
|
||||
if layout.lock_file.is_file():
|
||||
lock = json.loads(layout.lock_file.read_text(encoding='utf-8'))
|
||||
if lock.get('format') != LOCK_FORMAT:
|
||||
raise ToolchainError(f'{layout.lock_file}: unknown format {lock.get("format")}')
|
||||
return lock
|
||||
return make_lock(version=version)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- install
|
||||
|
||||
def installed(layout):
|
||||
p = layout.msys_root / 'installed.json'
|
||||
try:
|
||||
return json.loads(p.read_text(encoding='utf-8'))
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def missing_files(msys):
|
||||
return [r for r in REQUIRED if not (Path(msys) / r).exists()]
|
||||
|
||||
|
||||
def is_current(layout, lock):
|
||||
inst = installed(layout)
|
||||
return bool(inst) and inst.get('digest') == lock_digest(lock) and not missing_files(layout.msys)
|
||||
|
||||
|
||||
def _extract_package(path, dest):
|
||||
"""Unpack one .pkg.tar.zst into dest (the msys64 folder): only its ucrt64/ files, without the documentation.
|
||||
The archive is read as a stream, member by member, so a 350 MB package never sits in memory whole."""
|
||||
with open(path, 'rb') as raw:
|
||||
zst = raw.read(4) == b'\x28\xb5\x2f\xfd'
|
||||
raw.seek(0)
|
||||
if zst:
|
||||
try:
|
||||
from compression import zstd # Python 3.14
|
||||
fileobj = zstd.ZstdFile(raw)
|
||||
except ImportError:
|
||||
import zstandard
|
||||
fileobj = zstandard.ZstdDecompressor().stream_reader(raw)
|
||||
else:
|
||||
fileobj = raw
|
||||
with tarfile.open(fileobj=fileobj, mode='r|*') as t:
|
||||
for m in t:
|
||||
name = m.name.lstrip('./')
|
||||
if name.startswith('ucrt64/') and not name.startswith(SKIP_SHARE):
|
||||
t.extract(m, dest, filter='data')
|
||||
|
||||
|
||||
def install(layout, lock, progress=None, cancel=None, log=print, version='dev'):
|
||||
"""Download and unpack the lock's packages into toolchain/msys64 (through msys64.new, swapped in at the end).
|
||||
progress(phase, done, total) with phase 'download' (bytes) or 'unpack' (packages)."""
|
||||
pkgs = lock['packages']
|
||||
layout.pkgs.mkdir(parents=True, exist_ok=True)
|
||||
need = [p for p in pkgs if not ((layout.pkgs / p['filename']).is_file()
|
||||
and (layout.pkgs / p['filename']).stat().st_size == p['csize'])]
|
||||
total = sum(p['csize'] for p in need)
|
||||
log(f'toolchain: {len(pkgs)} packages, {len(need)} to download ({total / 1e6:.0f} MB)')
|
||||
done_before = 0
|
||||
for p in need:
|
||||
dest = layout.pkgs / p['filename']
|
||||
last_error = None
|
||||
for mirror in [lock.get('repo', REPO)] + [m for m in MIRRORS if m != lock.get('repo', REPO)]:
|
||||
try:
|
||||
net.download(mirror + p['filename'], dest, sha256=p['sha256'], size=p['csize'], cancel=cancel,
|
||||
progress=(lambda d, t, base=done_before: progress and progress('download', base + d, total)),
|
||||
version=version)
|
||||
last_error = None
|
||||
break
|
||||
except net.DownloadError as e:
|
||||
last_error = e
|
||||
log(f'toolchain: {e}')
|
||||
if last_error:
|
||||
raise ToolchainError(f'{p["filename"]} could not be downloaded: {last_error}')
|
||||
done_before += p['csize']
|
||||
for p in pkgs: # files kept from an earlier download are checked too
|
||||
if net.sha256_file(layout.pkgs / p['filename']) != p['sha256']:
|
||||
(layout.pkgs / p['filename']).unlink()
|
||||
raise ToolchainError(f'{p["filename"]} was damaged on disk; run the setup again to download it anew')
|
||||
staging = layout.toolchain / 'msys64.new'
|
||||
shutil.rmtree(staging, ignore_errors=True)
|
||||
staging.mkdir(parents=True)
|
||||
for i, p in enumerate(pkgs):
|
||||
if cancel and cancel():
|
||||
raise net.Cancelled()
|
||||
if progress:
|
||||
progress('unpack', i, len(pkgs))
|
||||
_extract_package(layout.pkgs / p['filename'], staging)
|
||||
if progress:
|
||||
progress('unpack', len(pkgs), len(pkgs))
|
||||
missing = missing_files(staging / 'ucrt64')
|
||||
if missing:
|
||||
raise ToolchainError(f'the packages lack {", ".join(missing)}')
|
||||
(staging / 'installed.json').write_text(json.dumps({
|
||||
'digest': lock_digest(lock), 'packages': [f'{p["name"]} {p["version"]}' for p in pkgs],
|
||||
'installed': time.strftime('%Y-%m-%d %H:%M')}, indent=1), encoding='utf-8')
|
||||
old = layout.toolchain / f'msys64.old-{int(time.time())}'
|
||||
if layout.msys_root.exists():
|
||||
os.replace(layout.msys_root, old)
|
||||
os.replace(staging, layout.msys_root)
|
||||
shutil.rmtree(old, ignore_errors=True)
|
||||
keep = {p['filename'] for p in pkgs}
|
||||
for f in layout.pkgs.iterdir():
|
||||
if f.name not in keep:
|
||||
f.unlink(missing_ok=True)
|
||||
log(f'toolchain: installed in {layout.msys}')
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- zbic
|
||||
|
||||
def wheel_tag():
|
||||
"""The wheel platform tags this interpreter loads, best first."""
|
||||
v = f'cp{sys.version_info.major}{sys.version_info.minor}'
|
||||
if os.name == 'nt':
|
||||
return [f'{v}-{v}-win_amd64']
|
||||
return [f'{v}-{v}-manylinux', f'{v}-{v}-musllinux']
|
||||
|
||||
|
||||
def zbic_ok(layout):
|
||||
"""zbic loads from pydeps/ in the interpreter the build runs with (a fresh process: this one may have cached
|
||||
that pydeps/ did not exist)."""
|
||||
import subprocess
|
||||
if not layout.pydeps.is_dir():
|
||||
return False
|
||||
code = 'import sys; sys.path.insert(0, sys.argv[1]); import zbic; assert callable(zbic.decompress)'
|
||||
flags = {'creationflags': 0x08000000} if os.name == 'nt' else {} # CREATE_NO_WINDOW
|
||||
r = subprocess.run([layout.python_for_builds(), '-I', '-c', code, str(layout.pydeps)], capture_output=True,
|
||||
**flags)
|
||||
return r.returncode == 0
|
||||
|
||||
|
||||
def ensure_zbic(layout, progress=None, cancel=None, log=print, version='dev'):
|
||||
if zbic_ok(layout):
|
||||
return
|
||||
name, ver = ZBIC
|
||||
info = net.get_json(f'https://pypi.org/pypi/{name}/{ver}/json', version=version)
|
||||
files = [u for u in info.get('urls', []) if u['filename'].endswith('.whl')]
|
||||
pick = None
|
||||
for tag in wheel_tag():
|
||||
for u in files:
|
||||
if tag in u['filename'] and ('x86_64' in u['filename'] or 'amd64' in u['filename']):
|
||||
pick = u
|
||||
break
|
||||
if pick:
|
||||
break
|
||||
if not pick:
|
||||
raise ToolchainError(f'PyPI has no {name} {ver} wheel for Python {sys.version_info.major}.'
|
||||
f'{sys.version_info.minor} on this system')
|
||||
dest = layout.downloads / pick['filename']
|
||||
net.download(pick['url'], dest, sha256=pick['digests']['sha256'], size=pick.get('size'), cancel=cancel,
|
||||
progress=(lambda d, t: progress and progress('download', d, t)), version=version)
|
||||
shutil.rmtree(layout.pydeps, ignore_errors=True)
|
||||
layout.pydeps.mkdir(parents=True)
|
||||
with zipfile.ZipFile(dest) as z:
|
||||
for m in z.infolist():
|
||||
target = (layout.pydeps / m.filename).resolve()
|
||||
if not str(target).startswith(str(layout.pydeps.resolve())):
|
||||
raise ToolchainError(f'{pick["filename"]}: unsafe path {m.filename}')
|
||||
z.extractall(layout.pydeps)
|
||||
dest.unlink(missing_ok=True)
|
||||
if not zbic_ok(layout):
|
||||
raise ToolchainError(f'{name} {ver} was unpacked into {layout.pydeps} but does not load')
|
||||
log(f'zbic: {pick["filename"]}')
|
||||
1
launcher/ui/__init__.py
Normal file
1
launcher/ui/__init__.py
Normal file
@@ -0,0 +1 @@
|
||||
"""The launcher's window: art.py draws the sprites, app.py lays out the screens and handles input."""
|
||||
1297
launcher/ui/app.py
Normal file
1297
launcher/ui/app.py
Normal file
File diff suppressed because it is too large
Load Diff
538
launcher/ui/art.py
Normal file
538
launcher/ui/art.py
Normal file
@@ -0,0 +1,538 @@
|
||||
"""The window's pictures, drawn with numpy into RGBA arrays and handed to Tk as PNG data.
|
||||
|
||||
Everything is drawn here from shapes and colours: the misty green backdrop, the wooden title band, the bronze label
|
||||
cells with their mottled texture, the description box, the white glowing cursor, the filigree divider, the Triforce
|
||||
and the small icons. Tk's canvas draws lines and polygons without antialiasing on Windows, so every curved or soft
|
||||
shape is a picture; flat rectangles and text stay canvas items.
|
||||
|
||||
Colours follow the game's options menu (docs/launcher.md, "Look"). Sizes are in logical pixels times the display
|
||||
scale s.
|
||||
"""
|
||||
import base64
|
||||
import re
|
||||
import struct
|
||||
import zlib
|
||||
|
||||
import numpy as np
|
||||
|
||||
# ---------------------------------------------------------------- colours
|
||||
|
||||
PALETTE = {
|
||||
'window': '#06180d',
|
||||
'panel': '#3a2b22',
|
||||
'label_top': '#7a5c32', 'label_mid': '#775930', 'label_bottom': '#72552d', 'label_border': '#6a4f29',
|
||||
'ring': '#2a2019',
|
||||
'value': '#2f2d22', 'value_border': '#45402f',
|
||||
'orange': '#d69127', 'orange_light': '#e3a234', 'orange_dark': '#8f5f17',
|
||||
'desc': '#403024', 'desc_border': '#4f3f30',
|
||||
'text': '#e9dcc0', 'text_bright': '#f1e6cc', 'value_text': '#e3d5b6', 'desc_text': '#e9dfc8',
|
||||
'muted': '#d4c6a6', 'dim': '#7d786a', 'off': '#5e5b51', 'on_text': '#fdf4de', 'salmon': '#e88f6c',
|
||||
'ok': '#a7d68c', 'gold': '#e6b65a', 'band_text': '#f3ecdc', 'status': '#b9a988', 'log_text': '#cfc4ab',
|
||||
'line': '#54442f', 'dot': '#5d4b36', 'filigree': '#7a6443', 'glow': '#e9ebee',
|
||||
'tri_top': '#d49f36', 'tri_bottom': '#fcbf44',
|
||||
}
|
||||
|
||||
|
||||
def rgb(hexcolor):
|
||||
h = hexcolor.lstrip('#')
|
||||
return np.array([int(h[i:i + 2], 16) / 255.0 for i in (0, 2, 4)], dtype=np.float32)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- PNG and Tk
|
||||
|
||||
def png_bytes(img):
|
||||
"""PNG of an HxWx4 float array in 0..1 (straight alpha) or uint8 array."""
|
||||
a = img if img.dtype == np.uint8 else (np.clip(img, 0, 1) * 255 + 0.5).astype(np.uint8)
|
||||
h, w = a.shape[:2]
|
||||
raw = b''.join(b'\0' + a[y].tobytes() for y in range(h))
|
||||
|
||||
def chunk(tag, data):
|
||||
return struct.pack('>I', len(data)) + tag + data + struct.pack('>I', zlib.crc32(tag + data) & 0xFFFFFFFF)
|
||||
return (b'\x89PNG\r\n\x1a\n' + chunk(b'IHDR', struct.pack('>IIBBBBB', w, h, 8, 6, 0, 0, 0))
|
||||
+ chunk(b'IDAT', zlib.compress(raw, 3)) + chunk(b'IEND', b''))
|
||||
|
||||
|
||||
def photo(master, img):
|
||||
import tkinter as tk
|
||||
return tk.PhotoImage(master=master, data=base64.b64encode(png_bytes(img)).decode('ascii'), format='png')
|
||||
|
||||
|
||||
def ico_bytes(images):
|
||||
"""A Windows .ico holding PNG-compressed images (one HxWx4 array per size, square)."""
|
||||
pngs = [png_bytes(im) for im in images]
|
||||
head = struct.pack('<HHH', 0, 1, len(pngs))
|
||||
offset = 6 + 16 * len(pngs)
|
||||
dirs, body = b'', b''
|
||||
for im, data in zip(images, pngs):
|
||||
n = im.shape[0]
|
||||
dirs += struct.pack('<BBBBHHII', n % 256, n % 256, 0, 0, 1, 32, len(data), offset + len(body))
|
||||
body += data
|
||||
return head + dirs + body
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- drawing primitives
|
||||
|
||||
def new(h, w, color=None, alpha=0.0):
|
||||
img = np.zeros((h, w, 4), dtype=np.float32)
|
||||
if color is not None:
|
||||
img[..., :3] = rgb(color)
|
||||
img[..., 3] = alpha
|
||||
return img
|
||||
|
||||
|
||||
def over(dst, src, x=0, y=0):
|
||||
"""Composite src over dst at (x, y), straight alpha, clipped to dst."""
|
||||
h, w = src.shape[:2]
|
||||
H, W = dst.shape[:2]
|
||||
x0, y0, x1, y1 = max(x, 0), max(y, 0), min(x + w, W), min(y + h, H)
|
||||
if x0 >= x1 or y0 >= y1:
|
||||
return dst
|
||||
s = src[y0 - y:y1 - y, x0 - x:x1 - x]
|
||||
d = dst[y0:y1, x0:x1]
|
||||
sa, da = s[..., 3:4], d[..., 3:4]
|
||||
oa = sa + da * (1 - sa)
|
||||
oc = np.where(oa > 0, (s[..., :3] * sa + d[..., :3] * da * (1 - sa)) / np.maximum(oa, 1e-6), 0)
|
||||
d[..., :3], d[..., 3:4] = oc, oa
|
||||
return dst
|
||||
|
||||
|
||||
def paint(img, mask, color, alpha=1.0):
|
||||
"""Paint a colour through a coverage mask (HxW, 0..1) onto img."""
|
||||
layer = new(*mask.shape, color)
|
||||
layer[..., 3] = np.clip(mask, 0, 1) * alpha
|
||||
return over(img, layer)
|
||||
|
||||
|
||||
def vgradient(h, w, stops):
|
||||
"""HxWx3 colours down the height through (position 0..1, colour) stops."""
|
||||
pos = np.array([p for p, _ in stops], dtype=np.float32)
|
||||
cols = np.stack([rgb(c) for _, c in stops])
|
||||
t = (np.arange(h, dtype=np.float32) + 0.5) / max(h, 1)
|
||||
col = np.stack([np.interp(t, pos, cols[:, k]) for k in range(3)], axis=1)
|
||||
return np.broadcast_to(col[:, None, :], (h, w, 3)).copy()
|
||||
|
||||
|
||||
def hgradient_alpha(h, w, stops):
|
||||
"""HxW alphas across the width through (position, alpha) stops."""
|
||||
pos = np.array([p for p, _ in stops], dtype=np.float32)
|
||||
al = np.array([a for _, a in stops], dtype=np.float32)
|
||||
t = (np.arange(w, dtype=np.float32) + 0.5) / max(w, 1)
|
||||
return np.broadcast_to(np.interp(t, pos, al)[None, :], (h, w)).copy()
|
||||
|
||||
|
||||
def hgradient_rgba(h, w, stops):
|
||||
"""An RGBA layer whose colour and alpha run across the width through (position, colour, alpha) stops."""
|
||||
pos = np.array([p for p, _, _ in stops], dtype=np.float32)
|
||||
cols = np.stack([rgb(c) for _, c, _ in stops])
|
||||
al = np.array([a for _, _, a in stops], dtype=np.float32)
|
||||
t = (np.arange(w, dtype=np.float32) + 0.5) / max(w, 1)
|
||||
layer = new(h, w)
|
||||
for k in range(3):
|
||||
layer[..., k] = np.interp(t, pos, cols[:, k])[None, :]
|
||||
layer[..., 3] = np.interp(t, pos, al)[None, :]
|
||||
return layer
|
||||
|
||||
|
||||
def value_noise(h, w, period_x, period_y=None, seed=0):
|
||||
"""Smooth noise in 0..1 with features about period pixels apart."""
|
||||
period_y = period_y or period_x
|
||||
rng = np.random.default_rng(seed)
|
||||
gh, gw = int(h / period_y) + 3, int(w / period_x) + 3
|
||||
g = rng.random((gh, gw), dtype=np.float32)
|
||||
ys = np.arange(h, dtype=np.float32) / period_y
|
||||
xs = np.arange(w, dtype=np.float32) / period_x
|
||||
y0, x0 = ys.astype(int), xs.astype(int)
|
||||
fy, fx = ys - y0, xs - x0
|
||||
fy, fx = fy * fy * (3 - 2 * fy), fx * fx * (3 - 2 * fx)
|
||||
a, b = g[y0][:, x0], g[y0][:, x0 + 1]
|
||||
c, d = g[y0 + 1][:, x0], g[y0 + 1][:, x0 + 1]
|
||||
top = a + (b - a) * fx[None, :]
|
||||
bot = c + (d - c) * fx[None, :]
|
||||
return top + (bot - top) * fy[:, None]
|
||||
|
||||
|
||||
def fractal(h, w, period_x, period_y=None, octaves=3, seed=0, contrast=0.4):
|
||||
"""Fractal noise around mid grey, about contrast wide (the range of an SVG feTurbulence turned grey)."""
|
||||
return 0.5 + (_fractal01(h, w, period_x, period_y, octaves, seed) - 0.5) * contrast
|
||||
|
||||
|
||||
def _fractal01(h, w, period_x, period_y=None, octaves=3, seed=0):
|
||||
period_y = period_y or period_x
|
||||
out = np.zeros((h, w), dtype=np.float32)
|
||||
amp, total = 1.0, 0.0
|
||||
for o in range(octaves):
|
||||
out += amp * value_noise(h, w, max(period_x / 2 ** o, 1.5), max(period_y / 2 ** o, 1.5), seed + o * 101)
|
||||
total += amp
|
||||
amp *= 0.5
|
||||
out /= total
|
||||
lo, hi = np.percentile(out, 1), np.percentile(out, 99)
|
||||
return np.clip((out - lo) / max(hi - lo, 1e-6), 0, 1)
|
||||
|
||||
|
||||
def overlay(rgb_arr, n, opacity):
|
||||
"""CSS mix-blend-mode overlay of a grey noise n (HxW) over colours, at opacity."""
|
||||
n = n[..., None]
|
||||
blended = np.where(rgb_arr < 0.5, 2 * rgb_arr * n, 1 - 2 * (1 - rgb_arr) * (1 - n))
|
||||
return rgb_arr + (blended - rgb_arr) * opacity
|
||||
|
||||
|
||||
def box_blur(a, r):
|
||||
"""Three box blurs of radius r over a 2-D array (close to a gaussian)."""
|
||||
r = int(round(r))
|
||||
if r < 1:
|
||||
return a
|
||||
out = a.astype(np.float32)
|
||||
for _ in range(3):
|
||||
for axis in (0, 1):
|
||||
pad = [(0, 0), (0, 0)]
|
||||
pad[axis] = (r + 1, r)
|
||||
p = np.pad(out, pad, mode='edge')
|
||||
c = np.cumsum(p, axis=axis, dtype=np.float64)
|
||||
if axis == 0:
|
||||
out = ((c[2 * r + 1:] - c[:-2 * r - 1]) / (2 * r + 1)).astype(np.float32)
|
||||
else:
|
||||
out = ((c[:, 2 * r + 1:] - c[:, :-2 * r - 1]) / (2 * r + 1)).astype(np.float32)
|
||||
return out
|
||||
|
||||
|
||||
def rect_mask(h, w, x0, y0, x1, y1):
|
||||
"""Coverage of the rectangle [x0, x1) x [y0, y1) with fractional edges."""
|
||||
xs = np.arange(w, dtype=np.float32)
|
||||
ys = np.arange(h, dtype=np.float32)
|
||||
cx = np.clip(np.minimum(xs + 1, x1) - np.maximum(xs, x0), 0, 1)
|
||||
cy = np.clip(np.minimum(ys + 1, y1) - np.maximum(ys, y0), 0, 1)
|
||||
return cy[:, None] * cx[None, :]
|
||||
|
||||
|
||||
def ring_mask(h, w, x0, y0, x1, y1, width):
|
||||
return np.clip(rect_mask(h, w, x0, y0, x1, y1) - rect_mask(h, w, x0 + width, y0 + width, x1 - width, y1 - width),
|
||||
0, 1)
|
||||
|
||||
|
||||
def polygon_mask(h, w, pts, ss=4):
|
||||
"""Antialiased coverage of a convex polygon (pixel coordinates), supersampled ss x ss."""
|
||||
ys = (np.arange(h * ss, dtype=np.float32) + 0.5) / ss
|
||||
xs = (np.arange(w * ss, dtype=np.float32) + 0.5) / ss
|
||||
X, Y = np.meshgrid(xs, ys)
|
||||
inside = np.ones_like(X, dtype=bool)
|
||||
n = len(pts)
|
||||
area = sum(pts[i][0] * pts[(i + 1) % n][1] - pts[(i + 1) % n][0] * pts[i][1] for i in range(n))
|
||||
sign = 1 if area > 0 else -1
|
||||
for i in range(n):
|
||||
(ax, ay), (bx, by) = pts[i], pts[(i + 1) % n]
|
||||
inside &= sign * ((bx - ax) * (Y - ay) - (by - ay) * (X - ax)) >= 0
|
||||
return inside.reshape(h, ss, w, ss).mean(axis=(1, 3)).astype(np.float32)
|
||||
|
||||
|
||||
def stroke_mask(h, w, polylines, width):
|
||||
"""Antialiased coverage of polylines (lists of (x, y)) stroked with round joins and caps."""
|
||||
ys = np.arange(h, dtype=np.float32) + 0.5
|
||||
xs = np.arange(w, dtype=np.float32) + 0.5
|
||||
X, Y = np.meshgrid(xs, ys)
|
||||
best = np.full((h, w), 1e9, dtype=np.float32)
|
||||
for line in polylines:
|
||||
P = np.asarray(line, dtype=np.float32)
|
||||
if len(P) == 1:
|
||||
P = np.vstack([P, P])
|
||||
for (ax, ay), (bx, by) in zip(P[:-1], P[1:]):
|
||||
dx, dy = bx - ax, by - ay
|
||||
ll = dx * dx + dy * dy
|
||||
if ll < 1e-9:
|
||||
d = np.hypot(X - ax, Y - ay)
|
||||
else:
|
||||
t = np.clip(((X - ax) * dx + (Y - ay) * dy) / ll, 0, 1)
|
||||
d = np.hypot(X - (ax + t * dx), Y - (ay + t * dy))
|
||||
np.minimum(best, d, out=best)
|
||||
return np.clip(width / 2 + 0.5 - best, 0, 1)
|
||||
|
||||
|
||||
def shadow(h, w, mask, color, alpha, blur, dx=0, dy=0):
|
||||
"""A soft shadow of a mask: blurred, offset, as an RGBA layer of the same size."""
|
||||
m = box_blur(np.roll(np.roll(mask, int(dy), 0), int(dx), 1), blur / 2)
|
||||
layer = new(h, w, color)
|
||||
layer[..., 3] = np.clip(m, 0, 1) * alpha
|
||||
return layer
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- SVG path data (the filigree)
|
||||
|
||||
def svg_polylines(d, scale=1.0, ox=0.0, oy=0.0, steps=10):
|
||||
"""Points along an SVG path made of M, L, C and Z commands."""
|
||||
toks = re.findall(r'[MLCZmlcz]|-?\d*\.?\d+', d)
|
||||
lines, cur, pos, i, cmd = [], [], (0.0, 0.0), 0, None
|
||||
start = (0.0, 0.0)
|
||||
|
||||
def num():
|
||||
nonlocal i
|
||||
v = float(toks[i])
|
||||
i += 1
|
||||
return v
|
||||
|
||||
while i < len(toks):
|
||||
if toks[i] in 'MLCZmlcz':
|
||||
cmd = toks[i]
|
||||
i += 1
|
||||
if cmd in 'Zz':
|
||||
cur.append(start)
|
||||
continue
|
||||
if cmd in 'Mm':
|
||||
if cur:
|
||||
lines.append(cur)
|
||||
pos = (num(), num())
|
||||
start = pos
|
||||
cur = [pos]
|
||||
cmd = 'L'
|
||||
elif cmd in 'Ll':
|
||||
pos = (num(), num())
|
||||
cur.append(pos)
|
||||
elif cmd in 'Cc':
|
||||
p0 = pos
|
||||
c1, c2, p1 = (num(), num()), (num(), num()), (num(), num())
|
||||
for k in range(1, steps + 1):
|
||||
t = k / steps
|
||||
u = 1 - t
|
||||
cur.append((u ** 3 * p0[0] + 3 * u * u * t * c1[0] + 3 * u * t * t * c2[0] + t ** 3 * p1[0],
|
||||
u ** 3 * p0[1] + 3 * u * u * t * c1[1] + 3 * u * t * t * c2[1] + t ** 3 * p1[1]))
|
||||
pos = p1
|
||||
if cur:
|
||||
lines.append(cur)
|
||||
return [[((x - ox) * scale, (y - oy) * scale) for x, y in line] for line in lines]
|
||||
|
||||
|
||||
# The game's divider ornament, traced from the options menu: two pointed lens ends, S-waves into spiral hooks and a
|
||||
# diamond knot under the middle, in a 120 x 38 box (centre x 60).
|
||||
FILIGREE_PATHS = (
|
||||
'M20 17.4 C20.4 17.3 21.6 16.8 22.2 16.9 C22.8 17 23.5 17.7 23.5 18.2 C23.5 18.7 23.1 19.4 22.4 19.8 C21.6 20.2 '
|
||||
'20.3 20.4 19 20.4 C17.7 20.4 15.9 20.2 14.5 19.9 C13.1 19.6 10.6 19.4 10.5 18.6 C10.4 17.9 12.6 16.3 14 15.4 '
|
||||
'C15.4 14.5 17.2 13.6 19 13.2 C20.8 12.8 23.2 12.8 25 13.1 C26.8 13.4 28.1 13.9 29.5 14.8 C30.9 15.7 32.2 17.4 '
|
||||
'33.5 18.6 C34.8 19.8 36 21.2 37.5 22 C39 22.8 40.8 23.4 42.5 23.6 C44.2 23.8 46 23.5 47.5 23.1 C49 22.7 50.4 '
|
||||
'21.9 51.2 21.2 C52 20.5 52.5 19.5 52.4 18.8 C52.3 18.1 51.5 17.3 50.6 16.9 C49.7 16.5 48.2 16.2 47 16.3 C45.8 '
|
||||
'16.4 44.2 17.1 43.6 17.2',
|
||||
'M100 17.4 C99.6 17.3 98.4 16.8 97.8 16.9 C97.2 17 96.5 17.7 96.5 18.2 C96.5 18.7 96.8 19.4 97.6 19.8 C98.3 20.2 '
|
||||
'99.7 20.4 101 20.4 C102.3 20.4 104.1 20.2 105.5 19.9 C106.9 19.6 109.4 19.4 109.5 18.6 C109.6 17.9 107.4 16.3 '
|
||||
'106 15.4 C104.6 14.5 102.8 13.6 101 13.2 C99.2 12.8 96.8 12.8 95 13.1 C93.2 13.4 91.9 13.9 90.5 14.8 C89.1 15.7 '
|
||||
'87.8 17.4 86.5 18.6 C85.2 19.8 84 21.2 82.5 22 C81 22.8 79.2 23.4 77.5 23.6 C75.8 23.8 74 23.5 72.5 23.1 C71 '
|
||||
'22.7 69.6 21.9 68.8 21.2 C68 20.5 67.5 19.5 67.6 18.8 C67.7 18.1 68.5 17.3 69.4 16.9 C70.3 16.5 71.8 16.2 73 '
|
||||
'16.3 C74.2 16.4 75.8 17.1 76.4 17.2',
|
||||
'M40.6 17.6 C40.7 17 40.7 14.9 41.2 14 C41.7 13.1 42.6 12.4 43.8 12 C45 11.6 46.7 11.4 48.2 11.5 C49.7 11.6 51.3 '
|
||||
'11.8 52.6 12.3 C53.9 12.8 55 13.6 56 14.6 C57 15.6 58.2 17.6 58.6 18.2 L64.6 26 L60 31.6',
|
||||
'M79.4 17.6 C79.3 17 79.3 14.9 78.8 14 C78.3 13.1 77.4 12.4 76.2 12 C75 11.6 73.3 11.4 71.8 11.5 C70.3 11.6 68.7 '
|
||||
'11.8 67.4 12.3 C66.1 12.8 65 13.6 64 14.6 C63 15.6 61.8 17.6 61.4 18.2 L55.4 26 L60 31.6',
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- the window's pieces
|
||||
|
||||
def backdrop(W, H, s):
|
||||
"""The whole window behind the panels: a dim misty field under a see-through dark green, darker at the sides."""
|
||||
img = new(H, W, PALETTE['window'], 1.0)
|
||||
m = int(30 * s)
|
||||
scene = new(H, W - 2 * m)
|
||||
scene[..., :3] = vgradient(H, W - 2 * m, [(0, '#394244'), (0.36, '#414b4c'), (0.60, '#474234'), (1, '#3e3523')])
|
||||
scene[..., 3] = 1
|
||||
over(img, scene, m, 0)
|
||||
band = np.zeros((H, W), dtype=np.float32)
|
||||
y0, y1 = int(250 * s), int(330 * s)
|
||||
t = (np.arange(y1 - y0) + 0.5) / max(y1 - y0, 1)
|
||||
band[y0:y1, m:W - m] = (np.interp(t, [0, 0.55, 1], [0, 0.85, 0]))[:, None]
|
||||
paint(img, box_blur(band, 10 * s), '#262a22')
|
||||
over(img, hgradient_rgba(H, W, [(0, '#05180d', 0.96), (0.14, '#092615', 0.82), (0.5, '#0c2e1b', 0.72),
|
||||
(0.86, '#092615', 0.82), (1, '#05180d', 0.96)]))
|
||||
inner = rect_mask(H, W, 0, 0, W, H)
|
||||
vig = 1 - box_blur(np.pad(inner, int(45 * s), constant_values=0), 45 * s)[int(45 * s):-int(45 * s),
|
||||
int(45 * s):-int(45 * s)]
|
||||
paint(img, np.clip(vig * 1.6, 0, 1), '#020e07', 0.6)
|
||||
return img
|
||||
|
||||
|
||||
def band(w, h, s):
|
||||
"""The wooden title band, its two-line bottom edge and the shadow under it; (image, extra height below)."""
|
||||
pad = int(14 * s)
|
||||
img = new(h + pad, w)
|
||||
wood = vgradient(h, w, [(0, '#4c3b26'), (0.30, '#5a4529'), (0.65, '#6b5232'), (1, '#806239')])
|
||||
wood = overlay(wood, fractal(h, w, 300 * s, 11 * s, 3, seed=9), 0.32)
|
||||
wood = overlay(wood, fractal(h, w, 33 * s, None, 3, seed=4), 0.2)
|
||||
shade = rect_mask(h + pad, w, 0, 0, w, h + 4 * s)
|
||||
over(img, shadow(h + pad, w, shade, '#000000', 0.6, 16 * s, 0, 5 * s))
|
||||
top = new(h, w)
|
||||
top[..., :3], top[..., 3] = wood, 1
|
||||
over(img, top)
|
||||
paint(img, rect_mask(h + pad, w, 0, h, w, h + 2 * s), '#2c2217')
|
||||
paint(img, rect_mask(h + pad, w, 0, h + 2 * s, w, h + 4 * s), '#a9874f')
|
||||
return img
|
||||
|
||||
|
||||
def panel(w, h, s):
|
||||
"""The left list panel with the shadow on its right."""
|
||||
sh = int(26 * s)
|
||||
img = new(h, w + sh)
|
||||
body = rect_mask(h, w + sh, 0, 0, w, h)
|
||||
over(img, shadow(h, w + sh, body, '#000000', 0.35, 18 * s, 6 * s, 0))
|
||||
col = np.broadcast_to(rgb(PALETTE['panel']), (h, w, 3)).copy()
|
||||
col = overlay(col, fractal(h, w, 33 * s, None, 3, seed=4), 0.18)
|
||||
layer = new(h, w)
|
||||
layer[..., :3], layer[..., 3] = col, 1
|
||||
over(img, layer)
|
||||
return img
|
||||
|
||||
|
||||
def label_cell(w, h, s, bright=False):
|
||||
"""A bronze label cell: gradient, mottled texture, 1 px border and the 1 px dark ring around the row."""
|
||||
r = max(1, int(round(s)))
|
||||
img = new(h + 2 * r, w + 2 * r, PALETTE['ring'], 1.0)
|
||||
col = vgradient(h, w, [(0, PALETTE['label_top']), (0.5, PALETTE['label_mid']), (1, PALETTE['label_bottom'])])
|
||||
col = overlay(col, fractal(h, w, 33 * s, None, 3, seed=(w * 7 + h) % 997), 0.22)
|
||||
if bright:
|
||||
col = np.clip(col * 1.08, 0, 1)
|
||||
cell = new(h, w)
|
||||
cell[..., :3], cell[..., 3] = col, 1
|
||||
over(img, cell, r, r)
|
||||
paint(img, ring_mask(h + 2 * r, w + 2 * r, r, r, w + r, h + r, r), PALETTE['label_border'])
|
||||
return img
|
||||
|
||||
|
||||
def desc_box(w, h, s):
|
||||
pad = int(20 * s)
|
||||
img = new(h + 2 * pad, w + 2 * pad)
|
||||
body = rect_mask(h + 2 * pad, w + 2 * pad, pad, pad, pad + w, pad + h)
|
||||
over(img, shadow(h + 2 * pad, w + 2 * pad, body, '#000000', 0.35, 18 * s, 0, 6 * s))
|
||||
col = np.broadcast_to(rgb(PALETTE['desc']), (h, w, 3)).copy()
|
||||
col = overlay(col, fractal(h, w, 33 * s, None, 3, seed=11), 0.15)
|
||||
layer = new(h, w)
|
||||
layer[..., :3], layer[..., 3] = col, 1
|
||||
over(img, layer, pad, pad)
|
||||
paint(img, ring_mask(h + 2 * pad, w + 2 * pad, pad, pad, pad + w, pad + h, max(1, round(s))),
|
||||
PALETTE['desc_border'])
|
||||
return img, pad
|
||||
|
||||
|
||||
def glow_frames(w, h, s, n=8):
|
||||
"""The selection cursor around a w x h box: a 3 px white frame 6 px outside it with a soft blue glow inside and
|
||||
out, n frames of one breath. Returns (frames, margin) where the box sits at (margin, margin)."""
|
||||
gap, thick, spread = 6 * s, 3 * s, 16 * s
|
||||
mg = int(gap + spread)
|
||||
H, W = h + 2 * mg, w + 2 * mg
|
||||
ring = ring_mask(H, W, mg - gap, mg - gap, mg + w + gap, mg + h + gap, thick)
|
||||
soft_out = box_blur(ring, 6 * s)
|
||||
soft_far = box_blur(ring, 12 * s)
|
||||
frames = []
|
||||
for k in range(n):
|
||||
t = 0.5 - 0.5 * np.cos(2 * np.pi * k / n)
|
||||
img = new(H, W)
|
||||
paint(img, np.clip(soft_far * 2.2, 0, 1), '#96b4e6', 0.35 + 0.25 * t)
|
||||
paint(img, np.clip(soft_out * 2.8, 0, 1), '#afc5e8', 0.7 + 0.3 * t)
|
||||
paint(img, ring, PALETTE['glow'])
|
||||
frames.append(img)
|
||||
return frames, mg
|
||||
|
||||
|
||||
def filigree(w, s):
|
||||
"""The divider: a thin line with a dot at each end and the ornament in the middle; (image, height)."""
|
||||
h = int(round(26 * s))
|
||||
img = new(h, w)
|
||||
mid = h / 2
|
||||
orn_w = 78 * s
|
||||
scale = orn_w / 108.0
|
||||
ox = (w - orn_w) / 2
|
||||
dot_r = 1.5 * s
|
||||
gap = 5 * s
|
||||
left_end, right_start = ox - gap, ox + orn_w + gap
|
||||
xs0, xs1 = 4 * s + 2 * dot_r + gap, w - 4 * s - 2 * dot_r - gap
|
||||
paint(img, rect_mask(h, w, xs0, mid - 0.5 * s, left_end, mid + 0.5 * s), PALETTE['line'])
|
||||
paint(img, rect_mask(h, w, right_start, mid - 0.5 * s, xs1, mid + 0.5 * s), PALETTE['line'])
|
||||
dots = stroke_mask(h, w, [[(4 * s + dot_r, mid)], [(w - 4 * s - dot_r, mid)]], 2 * dot_r)
|
||||
paint(img, dots, PALETTE['dot'])
|
||||
lines = []
|
||||
for d in FILIGREE_PATHS:
|
||||
for line in svg_polylines(d, scale, 6, 8):
|
||||
lines.append([(x + ox, y + mid - 13 * scale) for x, y in line])
|
||||
paint(img, stroke_mask(h, w, lines, 2.2 * scale), PALETTE['filigree'])
|
||||
return img, h
|
||||
|
||||
|
||||
def triforce(w):
|
||||
"""The Triforce, w pixels wide: three triangles, each shading from top to bottom, on a transparent ground."""
|
||||
h = int(round(w * 217 / 250))
|
||||
k = w / 250.0
|
||||
tris = [((125, 1), (186.2, 107), (63.8, 107)), ((62.2, 109), (123.4, 215), (1, 215)),
|
||||
((187.8, 109), (249, 215), (126.6, 215))]
|
||||
img = new(h, w)
|
||||
top, bottom = rgb(PALETTE['tri_top']), rgb(PALETTE['tri_bottom'])
|
||||
for tri in tris:
|
||||
pts = [(x * k, y * k) for x, y in tri]
|
||||
mask = polygon_mask(h, w, pts)
|
||||
y0, y1 = min(p[1] for p in pts), max(p[1] for p in pts)
|
||||
t = np.clip((np.arange(h, dtype=np.float32) + 0.5 - y0) / max(y1 - y0, 1e-6), 0, 1)[:, None, None]
|
||||
layer = new(h, w)
|
||||
layer[..., :3] = top + (bottom - top) * t
|
||||
layer[..., 3] = mask
|
||||
over(img, layer)
|
||||
return img
|
||||
|
||||
|
||||
def triforce_ico():
|
||||
"""The launcher's icon at the sizes Windows asks for."""
|
||||
imgs = []
|
||||
for n in (16, 24, 32, 48, 64, 128, 256):
|
||||
t = triforce(n)
|
||||
sq = new(n, n)
|
||||
over(sq, t, 0, (n - t.shape[0]) // 2)
|
||||
imgs.append(sq)
|
||||
return ico_bytes(imgs)
|
||||
|
||||
|
||||
def check_mark(size, s, color=None):
|
||||
img = new(size, size)
|
||||
k = size / 24.0
|
||||
paint(img, stroke_mask(size, size, [[(5 * k, 12.5 * k), (9.2 * k, 16.7 * k), (19 * k, 7 * k)]], 3 * k),
|
||||
color or PALETTE['ok'])
|
||||
return img
|
||||
|
||||
|
||||
def chevron(w, h, color, left=False):
|
||||
img = new(h, w)
|
||||
pts = [(w - 0.5, 0), (0.5, h / 2), (w - 0.5, h)] if left else [(0.5, 0), (w - 0.5, h / 2), (0.5, h)]
|
||||
paint(img, polygon_mask(h, w, pts), color)
|
||||
return img
|
||||
|
||||
|
||||
def framed_icon(size, s, kind):
|
||||
"""The square-framed icon of a big row (play or refresh), like the icon frames of the game's menu rows."""
|
||||
img = new(size, size)
|
||||
t = 1.5 * s
|
||||
paint(img, ring_mask(size, size, 0.5, 0.5, size - 0.5, size - 0.5, t), '#e9dcc0')
|
||||
c = size / 2
|
||||
if kind == 'play':
|
||||
k = size / 32
|
||||
paint(img, polygon_mask(size, size, [(c - 5 * k, c - 7 * k), (c + 7 * k, c), (c - 5 * k, c + 7 * k)]),
|
||||
'#efe4ca')
|
||||
elif kind == 'refresh':
|
||||
import math
|
||||
k = size / 32
|
||||
r = 7.5 * k
|
||||
arcs = []
|
||||
for a0, a1 in ((200, 340), (20, 160)):
|
||||
arcs.append([(c + r * math.cos(math.radians(a)), c + r * math.sin(math.radians(a)))
|
||||
for a in range(a0, a1 + 1, 6)])
|
||||
heads = [[(c + r * math.cos(math.radians(340)) - 3.5 * k, c + r * math.sin(math.radians(340)) - 1 * k),
|
||||
(c + r * math.cos(math.radians(340)), c + r * math.sin(math.radians(340))),
|
||||
(c + r * math.cos(math.radians(340)) + 0.5 * k, c + r * math.sin(math.radians(340)) - 4 * k)],
|
||||
[(c + r * math.cos(math.radians(160)) + 3.5 * k, c + r * math.sin(math.radians(160)) + 1 * k),
|
||||
(c + r * math.cos(math.radians(160)), c + r * math.sin(math.radians(160))),
|
||||
(c + r * math.cos(math.radians(160)) - 0.5 * k, c + r * math.sin(math.radians(160)) + 4 * k)]]
|
||||
paint(img, stroke_mask(size, size, arcs + heads, 2.2 * k), '#efe4ca')
|
||||
return img
|
||||
|
||||
|
||||
def badge(size, s):
|
||||
"""The orange "!" disc beside Updates when a new version is ready (the "!" itself is text)."""
|
||||
img = new(size + 4, size + 4)
|
||||
ring = stroke_mask(size + 4, size + 4, [[((size + 4) / 2, (size + 4) / 2)]], size + 1.5 * s)
|
||||
over(img, shadow(size + 4, size + 4, ring, '#fbbd45', 0.6, 6 * s))
|
||||
paint(img, ring, '#3a2a14')
|
||||
disc = stroke_mask(size + 4, size + 4, [[((size + 4) / 2, (size + 4) / 2)]], size)
|
||||
layer = new(size + 4, size + 4)
|
||||
layer[..., :3] = vgradient(size + 4, size + 4, [(0, '#fbbd45'), (1, '#e2942a')])
|
||||
layer[..., 3] = disc
|
||||
over(img, layer)
|
||||
return img
|
||||
173
launcher/ui/selftest.py
Normal file
173
launcher/ui/selftest.py
Normal file
@@ -0,0 +1,173 @@
|
||||
"""--selftest [--shots DIR]: draw every screen with made-up states, optionally save a PNG of each, and close.
|
||||
Nothing runs: the jobs are built and their steps set by hand."""
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
from .. import feed, jobs
|
||||
|
||||
|
||||
def grab_gdi(x, y, w, h, path):
|
||||
"""Windows: copy the screen rectangle with GDI and write it as PNG (art.png_bytes); no extra packages."""
|
||||
import ctypes
|
||||
from ctypes import wintypes
|
||||
import numpy as np
|
||||
from . import art
|
||||
user32, gdi32 = ctypes.windll.user32, ctypes.windll.gdi32
|
||||
user32.GetDC.restype = wintypes.HDC
|
||||
gdi32.CreateCompatibleDC.restype = wintypes.HDC
|
||||
gdi32.CreateCompatibleBitmap.restype = wintypes.HBITMAP
|
||||
gdi32.SelectObject.restype = wintypes.HGDIOBJ
|
||||
screen = user32.GetDC(None)
|
||||
mem = gdi32.CreateCompatibleDC(screen)
|
||||
bmp = gdi32.CreateCompatibleBitmap(screen, w, h)
|
||||
old = gdi32.SelectObject(mem, bmp)
|
||||
gdi32.BitBlt(mem, 0, 0, w, h, screen, x, y, 0x00CC0020) # SRCCOPY
|
||||
|
||||
class BITMAPINFOHEADER(ctypes.Structure):
|
||||
_fields_ = [('biSize', wintypes.DWORD), ('biWidth', wintypes.LONG), ('biHeight', wintypes.LONG),
|
||||
('biPlanes', wintypes.WORD), ('biBitCount', wintypes.WORD), ('biCompression', wintypes.DWORD),
|
||||
('biSizeImage', wintypes.DWORD), ('biXPelsPerMeter', wintypes.LONG),
|
||||
('biYPelsPerMeter', wintypes.LONG), ('biClrUsed', wintypes.DWORD),
|
||||
('biClrImportant', wintypes.DWORD)]
|
||||
bi = BITMAPINFOHEADER(ctypes.sizeof(BITMAPINFOHEADER), w, -h, 1, 32, 0, 0, 0, 0, 0, 0)
|
||||
buf = (ctypes.c_ubyte * (w * h * 4))()
|
||||
gdi32.GetDIBits(mem, bmp, 0, h, buf, ctypes.byref(bi), 0)
|
||||
gdi32.SelectObject(mem, old)
|
||||
gdi32.DeleteObject(bmp)
|
||||
gdi32.DeleteDC(mem)
|
||||
user32.ReleaseDC(None, screen)
|
||||
a = np.frombuffer(buf, dtype=np.uint8).reshape(h, w, 4)[..., [2, 1, 0, 3]].copy()
|
||||
a[..., 3] = 255
|
||||
Path(path).write_bytes(art.png_bytes(a))
|
||||
return True
|
||||
|
||||
|
||||
def grab(app, path):
|
||||
"""A PNG of the window: GDI on Windows, else PIL's ImageGrab, else ImageMagick's import on X11."""
|
||||
app.root.update()
|
||||
x, y = app.root.winfo_rootx(), app.root.winfo_rooty()
|
||||
w, h = app.root.winfo_width(), app.root.winfo_height()
|
||||
if os.name == 'nt':
|
||||
try:
|
||||
return grab_gdi(x, y, w, h, path)
|
||||
except Exception as e:
|
||||
app.ctx.log(f'selftest: screen capture failed: {e!r}')
|
||||
try:
|
||||
from PIL import ImageGrab
|
||||
ImageGrab.grab(bbox=(x, y, x + w, y + h), xdisplay=os.environ.get('DISPLAY') if os.name != 'nt' else None
|
||||
).save(path)
|
||||
return True
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
subprocess.run(['import', '-window', str(app.root.winfo_id()), str(path)], check=True, capture_output=True)
|
||||
return True
|
||||
except (OSError, subprocess.CalledProcessError):
|
||||
return False
|
||||
|
||||
|
||||
def fake_release(version='0.3.11'):
|
||||
m = {'format': 1, 'version': version, 'app': {'name': f'NX2navi-app-{version}.zip', 'size': 1, 'sha256': '0'}}
|
||||
notes = ('- Each program module comes from the .nup package that holds it whole.\n'
|
||||
'- The PC stays awake while the game builds.\n- This launcher: one-click install, updates and play.')
|
||||
return feed.Release(m, notes)
|
||||
|
||||
|
||||
def states(app):
|
||||
ctx = app.ctx
|
||||
s = app.settings
|
||||
s['dump'] = s.get('dump') or str(Path.home() / 'Downloads' / 'Ocarina of Time.nup')
|
||||
s['game_dir'] = s.get('game_dir') or ('D:\\Games\\NX2navi' if os.name == 'nt' else '/games/NX2navi')
|
||||
app.dumps = [Path(s['dump'])]
|
||||
|
||||
def first_run():
|
||||
app.show('first_run', len(app.rows_first_run()) - 1)
|
||||
|
||||
def installing():
|
||||
j = jobs.InstallJob(ctx, s['dump'], s['game_dir'])
|
||||
for sid, val in (('tools', 'Done · 0:48'), ('check', 'Version 1.1.1'), ('exefs', 'Done · 0:04'),
|
||||
('romfs', 'Done · 0:31')):
|
||||
j.step(sid).state, j.step(sid).value, j.step(sid).fraction = jobs.DONE, val, 1.0
|
||||
j.begin('lift')
|
||||
j.progress('lift', 0.34, '412 of 1,204 chunks')
|
||||
j.started = time.time() - 192
|
||||
for ln in ('check: Ocarina of Time 1.1.1 (build e0f3ccfa)', 'romfs: 26.0 of 26.0 GB, 31 s', 'lift:',
|
||||
' emitting with 16 workers', ' emitted 412/1204 chunks, 18s'):
|
||||
j.lines.append(ln)
|
||||
app.job, app.job_return, app.job_back = j, 'home', 'first_run'
|
||||
app.show('job')
|
||||
|
||||
def home():
|
||||
app.release, app.checked_at, app.check, app.check_error, app.message = None, time.time() - 5, None, '', ''
|
||||
app.show('home', 0)
|
||||
|
||||
def update_ready():
|
||||
app.release = fake_release()
|
||||
app.show('home', 1)
|
||||
|
||||
def updating():
|
||||
j = jobs.UpdateDownloadJob(ctx, fake_release())
|
||||
for sid, val in (('download', 'Done · 0:06'), ('verify', 'Checksum OK')):
|
||||
j.step(sid).state, j.step(sid).value, j.step(sid).fraction = jobs.DONE, val, 1.0
|
||||
j.skip('tools_update', 'Unchanged')
|
||||
j.step('lift').state, j.step('lift').value, j.step('lift').fraction = jobs.DONE, 'Done · 0:41', 1.0
|
||||
j.begin('compile_changed')
|
||||
j.progress('compile_changed', 0.61, '238 of 391 changed files')
|
||||
j.started = time.time() - 104
|
||||
app.job, app.job_return = j, 'home'
|
||||
app.show('job')
|
||||
|
||||
def settings():
|
||||
app.show('settings', 0)
|
||||
|
||||
def failed():
|
||||
j = jobs.InstallJob(ctx, s['dump'], s['game_dir'])
|
||||
for sid, val in (('tools', 'Done · 0:48'), ('check', 'Version 1.1.1'), ('exefs', 'Done · 0:04'),
|
||||
('romfs', 'Done · 0:31'), ('lift', 'Done · 0:48')):
|
||||
j.step(sid).state, j.step(sid).value, j.step(sid).fraction = jobs.DONE, val, 1.0
|
||||
j.step('compile').state, j.step('compile').value = jobs.FAILED, 'Stopped'
|
||||
j.result, j.error = 'failed', '3 files did not compile'
|
||||
j.started, j.finished = time.time() - 362, time.time()
|
||||
for ln in ('compile:', ' compiled 1200 of 1212, 236 s', 'error: 3 files did not compile'):
|
||||
j.lines.append(ln)
|
||||
app.job, app.job_return, app.job_back = j, 'home', 'first_run'
|
||||
app.show('job')
|
||||
|
||||
def uninstall():
|
||||
app.show('uninstall', 0)
|
||||
|
||||
return [('1-first-run', first_run), ('2-installing', installing), ('3-home', home),
|
||||
('4-update-ready', update_ready), ('5-updating', updating), ('6-settings', settings),
|
||||
('7-build-failed', failed), ('8-uninstall', uninstall)]
|
||||
|
||||
|
||||
def run(app, shots):
|
||||
out = Path(shots) if shots else None
|
||||
if out:
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
seq = states(app)
|
||||
ok = True
|
||||
|
||||
def step(i=0):
|
||||
nonlocal ok
|
||||
if i >= len(seq):
|
||||
app.ctx.log(f'selftest: {len(seq)} screens drawn')
|
||||
print(f'selftest: {len(seq)} screens drawn' + (f', shots in {out}' if out else ''), flush=True)
|
||||
app.root.after(100, app.root.destroy)
|
||||
return
|
||||
name, fn = seq[i]
|
||||
try:
|
||||
fn()
|
||||
app.root.update()
|
||||
except Exception as e:
|
||||
ok = False
|
||||
print(f'selftest: {name}: {e!r}', file=sys.stderr, flush=True)
|
||||
raise
|
||||
if out:
|
||||
app.root.after(350, lambda: (grab(app, out / f'{name}.png'), step(i + 1)))
|
||||
else:
|
||||
app.root.after(50, lambda: step(i + 1))
|
||||
step()
|
||||
158
launcher/updater.py
Normal file
158
launcher/updater.py
Normal file
@@ -0,0 +1,158 @@
|
||||
"""Updating the launcher's own release folder (launcher/paths.py, "app/").
|
||||
|
||||
An update is staged beside the running version and switched to with two small files, so a failed or interrupted
|
||||
update never leaves a half-written release in use:
|
||||
|
||||
1. The app archive goes to data/downloads, is checked against the manifest's size and sha256 and unpacks into
|
||||
app/<version>.part, which is renamed to app/<version> once it is complete and holds the expected VERSION.
|
||||
2. A new Python (the manifest's runtime id differs from python/runtime-id.txt) unpacks into python.next; the stub
|
||||
swaps it in at its next start, while no Python runs.
|
||||
3. app/previous.txt gets the running version and app/current.txt the new one; the launcher starts the stub again,
|
||||
which runs the new version. The stub goes back to previous.txt when the new version fails to start.
|
||||
4. The new version rebuilds the game (launcher/jobs.py), then prune() removes every version but these two.
|
||||
"""
|
||||
import os
|
||||
import shutil
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
from . import net, versions
|
||||
|
||||
|
||||
class UpdateError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def read_text(path, default=''):
|
||||
try:
|
||||
return Path(path).read_text(encoding='utf-8').strip()
|
||||
except OSError:
|
||||
return default
|
||||
|
||||
|
||||
def write_text(path, text):
|
||||
path = Path(path)
|
||||
tmp = path.with_name(path.name + '.tmp')
|
||||
tmp.write_text(text + '\n', encoding='utf-8')
|
||||
os.replace(tmp, path)
|
||||
|
||||
|
||||
def current(layout):
|
||||
return read_text(layout.apps / 'current.txt') or layout.version
|
||||
|
||||
|
||||
def previous(layout):
|
||||
return read_text(layout.apps / 'previous.txt')
|
||||
|
||||
|
||||
def runtime_id(layout):
|
||||
return read_text(layout.python / 'runtime-id.txt')
|
||||
|
||||
|
||||
def safe_extract(zip_path, dest):
|
||||
"""Unpack a zip into dest, refusing absolute names and names that climb out of it."""
|
||||
dest = Path(dest)
|
||||
root = dest.resolve()
|
||||
with zipfile.ZipFile(zip_path) as z:
|
||||
for info in z.infolist():
|
||||
name = info.filename.replace('\\', '/')
|
||||
if name.startswith('/') or ':' in name.split('/')[0] or '..' in name.split('/'):
|
||||
raise UpdateError(f'{Path(zip_path).name}: unsafe path {info.filename}')
|
||||
target = (dest / name).resolve()
|
||||
if target != root and root not in target.parents:
|
||||
raise UpdateError(f'{Path(zip_path).name}: unsafe path {info.filename}')
|
||||
z.extractall(dest)
|
||||
|
||||
|
||||
def _fetch(release, key, layout, progress, cancel, version):
|
||||
url, headers, size, sha = release.asset(key)
|
||||
entry = release.manifest[key]
|
||||
dest = layout.downloads / entry['name']
|
||||
net.download(url, dest, sha256=sha, size=size, headers=headers, progress=progress, cancel=cancel,
|
||||
version=version)
|
||||
return dest
|
||||
|
||||
|
||||
def stage_app(layout, release, progress=None, cancel=None, version='dev'):
|
||||
"""Download, check and unpack the release's app archive into app/<version>; returns that folder."""
|
||||
zip_path = _fetch(release, 'app', layout, progress, cancel, version)
|
||||
ver = versions.display(release.version)
|
||||
final = layout.apps / ver
|
||||
part = layout.apps / f'{ver}.part'
|
||||
shutil.rmtree(part, ignore_errors=True)
|
||||
part.mkdir(parents=True)
|
||||
try:
|
||||
safe_extract(zip_path, part)
|
||||
got = read_text(part / 'VERSION')
|
||||
if versions.display(got) != ver or not (part / 'launcher' / 'start.pyw').is_file():
|
||||
raise UpdateError(f'{zip_path.name} holds version {got or "?"} without a launcher, not {ver}')
|
||||
if final.exists():
|
||||
if final.resolve() == layout.app.resolve():
|
||||
raise UpdateError(f'{ver} is the version that runs now')
|
||||
shutil.rmtree(final)
|
||||
os.replace(part, final)
|
||||
except BaseException:
|
||||
shutil.rmtree(part, ignore_errors=True)
|
||||
raise
|
||||
zip_path.unlink(missing_ok=True)
|
||||
return final
|
||||
|
||||
|
||||
def runtime_needed(layout, release):
|
||||
rt = release.manifest.get('runtime')
|
||||
return bool(rt) and not layout.dev and rt.get('id') != runtime_id(layout)
|
||||
|
||||
|
||||
def stage_runtime(layout, release, progress=None, cancel=None, version='dev'):
|
||||
"""Unpack a new Python into python.next for the stub to swap in; False when the running one is current."""
|
||||
if not runtime_needed(layout, release):
|
||||
return False
|
||||
zip_path = _fetch(release, 'runtime', layout, progress, cancel, version)
|
||||
nxt = layout.root / 'python.next'
|
||||
shutil.rmtree(nxt, ignore_errors=True)
|
||||
part = layout.root / 'python.next.part'
|
||||
shutil.rmtree(part, ignore_errors=True)
|
||||
part.mkdir()
|
||||
safe_extract(zip_path, part)
|
||||
if not (part / 'python.exe').is_file() or read_text(part / 'runtime-id.txt') != release.manifest['runtime']['id']:
|
||||
shutil.rmtree(part, ignore_errors=True)
|
||||
raise UpdateError(f'{zip_path.name} is not the runtime {release.manifest["runtime"]["id"]}')
|
||||
os.replace(part, nxt)
|
||||
zip_path.unlink(missing_ok=True)
|
||||
return True
|
||||
|
||||
|
||||
def switch_to(layout, ver):
|
||||
"""Make ver the version the stub starts, keeping the running one as previous."""
|
||||
if not (layout.apps / ver / 'launcher' / 'start.pyw').is_file():
|
||||
raise UpdateError(f'app/{ver} is not a complete release')
|
||||
now = current(layout)
|
||||
if now != ver:
|
||||
write_text(layout.apps / 'previous.txt', now)
|
||||
write_text(layout.apps / 'current.txt', ver)
|
||||
|
||||
|
||||
def rollback(layout):
|
||||
"""Back to previous.txt's version; False when there is none to go back to."""
|
||||
prev = previous(layout)
|
||||
if not prev or not (layout.apps / prev / 'launcher' / 'start.pyw').is_file():
|
||||
return False
|
||||
write_text(layout.apps / 'current.txt', prev)
|
||||
(layout.apps / 'previous.txt').unlink(missing_ok=True)
|
||||
return True
|
||||
|
||||
|
||||
def prune(layout):
|
||||
"""Remove every app version but the current and previous one, an old Python the stub set aside, and stale
|
||||
downloads."""
|
||||
keep = {current(layout), previous(layout)}
|
||||
if layout.apps.is_dir():
|
||||
for d in layout.apps.iterdir():
|
||||
if d.is_dir() and d.name not in keep:
|
||||
shutil.rmtree(d, ignore_errors=True)
|
||||
for d in layout.root.glob('python.old*'):
|
||||
shutil.rmtree(d, ignore_errors=True)
|
||||
if layout.downloads.is_dir():
|
||||
for f in layout.downloads.iterdir():
|
||||
if f.suffix in ('.part', '.zip'):
|
||||
f.unlink(missing_ok=True)
|
||||
25
launcher/versions.py
Normal file
25
launcher/versions.py
Normal file
@@ -0,0 +1,25 @@
|
||||
"""Release versions: "0.3.11", optionally with build metadata ("0.3.11+abc1234") that never orders releases."""
|
||||
import re
|
||||
|
||||
_NUM = re.compile(r'^\s*v?(\d+(?:\.\d+)*)(?:-([0-9A-Za-z.-]+))?(?:\+([0-9A-Za-z.-]+))?\s*$')
|
||||
|
||||
|
||||
def parse(text):
|
||||
"""(numbers, pre-release) for comparing; a pre-release sorts before its release. ValueError when malformed."""
|
||||
m = _NUM.match(text or '')
|
||||
if not m:
|
||||
raise ValueError(f'not a version: {text!r}')
|
||||
nums = tuple(int(x) for x in m.group(1).split('.'))
|
||||
nums = nums + (0,) * (4 - len(nums)) if len(nums) < 4 else nums
|
||||
pre = m.group(2)
|
||||
return nums, (1,) if pre is None else (0, pre)
|
||||
|
||||
|
||||
def newer(a, b):
|
||||
"""True when version a is newer than version b."""
|
||||
return parse(a) > parse(b)
|
||||
|
||||
|
||||
def display(text):
|
||||
"""The version without build metadata."""
|
||||
return (text or '').split('+', 1)[0].strip()
|
||||
192
launcher/winsys.py
Normal file
192
launcher/winsys.py
Normal file
@@ -0,0 +1,192 @@
|
||||
"""What the launcher needs from the operating system: crisp drawing on scaled displays, the game's fonts for this
|
||||
process only, starting the game and the stub, shortcuts, the uninstall entry and a gamepad. Windows first; the
|
||||
other branches exist so the window can be developed and tested on Linux."""
|
||||
import ctypes
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
CREATE_NO_WINDOW = 0x08000000
|
||||
DETACHED_PROCESS = 0x00000008
|
||||
CREATE_NEW_PROCESS_GROUP = 0x00000200
|
||||
UNINSTALL_KEY = r'Software\Microsoft\Windows\CurrentVersion\Uninstall\NX2navi'
|
||||
|
||||
|
||||
def dpi_aware():
|
||||
"""Draw at the display's real resolution instead of letting Windows stretch a 96-dpi window (blurry)."""
|
||||
if os.name != 'nt':
|
||||
return
|
||||
try:
|
||||
ctypes.windll.shcore.SetProcessDpiAwareness(1)
|
||||
except (AttributeError, OSError):
|
||||
try:
|
||||
ctypes.windll.user32.SetProcessDPIAware()
|
||||
except (AttributeError, OSError):
|
||||
pass
|
||||
|
||||
|
||||
def app_id():
|
||||
"""Group the window under the launcher's own taskbar icon rather than python's."""
|
||||
if os.name == 'nt':
|
||||
try:
|
||||
ctypes.windll.shell32.SetCurrentProcessExplicitAppUserModelID('NX2navi.Launcher')
|
||||
except (AttributeError, OSError):
|
||||
pass
|
||||
|
||||
|
||||
def register_font(path):
|
||||
"""Make a font file usable by this process only (Tk then finds it by family name). True on success."""
|
||||
path = str(path)
|
||||
if os.name == 'nt':
|
||||
try:
|
||||
return ctypes.windll.gdi32.AddFontResourceExW(ctypes.c_wchar_p(path), 0x10, None) > 0 # FR_PRIVATE
|
||||
except (AttributeError, OSError):
|
||||
return False
|
||||
try:
|
||||
fc = ctypes.CDLL('libfontconfig.so.1')
|
||||
fc.FcConfigAppFontAddFile.argtypes = [ctypes.c_void_p, ctypes.c_char_p]
|
||||
return bool(fc.FcConfigAppFontAddFile(None, path.encode()))
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
|
||||
def open_path(path):
|
||||
path = str(path)
|
||||
if os.name == 'nt':
|
||||
os.startfile(path) # noqa: S606 — opens a folder in Explorer
|
||||
else:
|
||||
subprocess.Popen(['xdg-open', path], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
|
||||
|
||||
def start_game(exe):
|
||||
"""Start the game from its own folder; returns the process to watch."""
|
||||
exe = Path(exe)
|
||||
flags = {'creationflags': CREATE_NEW_PROCESS_GROUP} if os.name == 'nt' else {'start_new_session': True}
|
||||
return subprocess.Popen([str(exe)], cwd=str(exe.parent), stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL, close_fds=True, **flags)
|
||||
|
||||
|
||||
def start_detached(cmd, cwd=None):
|
||||
flags = {'creationflags': DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP} if os.name == 'nt' \
|
||||
else {'start_new_session': True}
|
||||
subprocess.Popen(cmd, cwd=cwd, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
|
||||
close_fds=True, **flags)
|
||||
|
||||
|
||||
def restart(layout, args=()):
|
||||
"""Start the launcher again through its stub (which swaps in a staged Python and starts app/current.txt's
|
||||
version). False in a checkout, which has no stub."""
|
||||
if not layout.stub.is_file():
|
||||
return False
|
||||
start_detached([str(layout.stub), *args], cwd=str(layout.root))
|
||||
return True
|
||||
|
||||
|
||||
def stub_run(layout, args, timeout=30):
|
||||
"""Run the stub for one of its own commands (shortcuts); True when it succeeded."""
|
||||
if not layout.stub.is_file():
|
||||
return False
|
||||
flags = {'creationflags': CREATE_NO_WINDOW} if os.name == 'nt' else {}
|
||||
try:
|
||||
return subprocess.run([str(layout.stub), *args], timeout=timeout, capture_output=True, **flags).returncode == 0
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
return False
|
||||
|
||||
|
||||
def desktop_shortcut(layout, enable):
|
||||
return stub_run(layout, ['--desktop-shortcut', 'on' if enable else 'off'])
|
||||
|
||||
|
||||
def keep_awake(on):
|
||||
if os.name != 'nt':
|
||||
return
|
||||
try:
|
||||
ES_CONTINUOUS, ES_SYSTEM_REQUIRED = 0x80000000, 0x00000001
|
||||
ctypes.windll.kernel32.SetThreadExecutionState(ES_CONTINUOUS | (ES_SYSTEM_REQUIRED if on else 0))
|
||||
except (AttributeError, OSError):
|
||||
pass
|
||||
|
||||
|
||||
def uninstall(layout, remove_game='', log=print):
|
||||
"""Remove the shortcuts, the uninstall entry and, once this process has exited, the launcher's folder (and the
|
||||
game folder when remove_game names it). Saves inside a kept game folder stay."""
|
||||
stub_run(layout, ['--shortcuts', 'off'])
|
||||
if os.name == 'nt':
|
||||
try:
|
||||
import winreg
|
||||
winreg.DeleteKey(winreg.HKEY_CURRENT_USER, UNINSTALL_KEY)
|
||||
except OSError:
|
||||
pass
|
||||
targets = [str(layout.root)] + ([str(remove_game)] if remove_game else [])
|
||||
if layout.toolchain.parent != layout.root:
|
||||
targets.append(str(layout.toolchain.parent))
|
||||
script = 'ping -n 3 127.0.0.1 >nul & ' + ' & '.join(f'rmdir /s /q "{t}"' for t in targets)
|
||||
start_detached(['cmd.exe', '/c', script], cwd=os.environ.get('TEMP', 'C:\\'))
|
||||
else:
|
||||
log('uninstall: remove these folders yourself: ' + ', '.join([str(layout.root), str(remove_game)]))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- gamepad (XInput)
|
||||
|
||||
class _XInputGamepad(ctypes.Structure):
|
||||
_fields_ = [('wButtons', ctypes.c_ushort), ('bLeftTrigger', ctypes.c_ubyte), ('bRightTrigger', ctypes.c_ubyte),
|
||||
('sThumbLX', ctypes.c_short), ('sThumbLY', ctypes.c_short), ('sThumbRX', ctypes.c_short),
|
||||
('sThumbRY', ctypes.c_short)]
|
||||
|
||||
|
||||
class _XInputState(ctypes.Structure):
|
||||
_fields_ = [('dwPacketNumber', ctypes.c_uint), ('Gamepad', _XInputGamepad)]
|
||||
|
||||
|
||||
BUTTONS = {0x0001: 'up', 0x0002: 'down', 0x0004: 'left', 0x0008: 'right', 0x1000: 'a', 0x2000: 'b', 0x8000: 'y',
|
||||
0x4000: 'x', 0x0010: 'start'}
|
||||
|
||||
|
||||
class Gamepad:
|
||||
"""Polls the first XInput pad; poll() returns the buttons pressed since the last call (edges), the left stick
|
||||
counted as the d-pad."""
|
||||
|
||||
def __init__(self):
|
||||
self.dll = None
|
||||
self.held = set()
|
||||
if os.name == 'nt':
|
||||
for name in ('xinput1_4', 'xinput1_3', 'xinput9_1_0'):
|
||||
try:
|
||||
self.dll = getattr(ctypes.windll, name)
|
||||
break
|
||||
except (AttributeError, OSError):
|
||||
continue
|
||||
|
||||
def poll(self):
|
||||
if not self.dll:
|
||||
return set()
|
||||
st = _XInputState()
|
||||
down = set()
|
||||
for pad in range(4):
|
||||
try:
|
||||
if self.dll.XInputGetState(pad, ctypes.byref(st)) == 0:
|
||||
g = st.Gamepad
|
||||
down = {name for bit, name in BUTTONS.items() if g.wButtons & bit}
|
||||
if g.sThumbLY > 16000:
|
||||
down.add('up')
|
||||
elif g.sThumbLY < -16000:
|
||||
down.add('down')
|
||||
if g.sThumbLX > 16000:
|
||||
down.add('right')
|
||||
elif g.sThumbLX < -16000:
|
||||
down.add('left')
|
||||
break
|
||||
except OSError:
|
||||
return set()
|
||||
pressed = down - self.held
|
||||
self.held = down
|
||||
return pressed
|
||||
|
||||
|
||||
def is_windows():
|
||||
return os.name == 'nt'
|
||||
|
||||
|
||||
def frozen_python():
|
||||
return Path(sys.executable).parent
|
||||
678
tests/launcher_test.py
Normal file
678
tests/launcher_test.py
Normal file
@@ -0,0 +1,678 @@
|
||||
"""launcher/ and tools/make_game.py's bookkeeping, without game files or Windows.
|
||||
|
||||
Run: python tests/launcher_test.py (Python 3.14 for the zstd tests; the window test needs tkinter, numpy and a display)
|
||||
|
||||
- versions, settings
|
||||
- toolchain: the package database, dependency resolution and an install from a local fake repository
|
||||
- net: downloads that resume, check their sha256 and stop on cancel
|
||||
- feed: GitHub releases (stable, preview, private with a token) and a plain manifest URL, served locally
|
||||
- updater: staging a release, switching, rollback, pruning, refusing unsafe archives
|
||||
- gamefiles: the .bfotf font wrapping and zstd dictionaries, on a font of this system made into a fake RomFS
|
||||
- make_game.py: a synthetic .nup through the check, exefs and romfs steps with --events, skipped steps on a second
|
||||
run, --rebuild without the game files, and the object cache of the compile step with the system's C compiler
|
||||
- jobs: an install driven by a fake make_game.py
|
||||
- the window: --selftest draws every screen (skipped without a display)
|
||||
"""
|
||||
import hashlib
|
||||
import http.server
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import struct
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
import tempfile
|
||||
import threading
|
||||
import unittest
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
sys.path.insert(0, str(ROOT))
|
||||
sys.path.insert(0, str(ROOT / 'tools'))
|
||||
|
||||
from launcher import build, feed, gamefiles, jobs, net, settings, toolchain, updater, versions # noqa: E402
|
||||
from launcher.context import Context # noqa: E402
|
||||
from launcher.paths import Layout # noqa: E402
|
||||
|
||||
try:
|
||||
from compression import zstd as ZSTD
|
||||
except ImportError:
|
||||
ZSTD = None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- helpers
|
||||
|
||||
class Server:
|
||||
"""A local HTTP server for a folder, with extra routes (path -> (status, headers, body)) and Range support."""
|
||||
|
||||
def __init__(self, folder, routes=None):
|
||||
folder, routes = Path(folder), routes if routes is not None else {}
|
||||
self.hits = []
|
||||
server = self
|
||||
|
||||
class H(http.server.BaseHTTPRequestHandler):
|
||||
def log_message(self, *a):
|
||||
pass
|
||||
|
||||
def do_GET(self):
|
||||
server.hits.append((self.path, dict(self.headers)))
|
||||
if self.path in routes:
|
||||
status, headers, body = routes[self.path]
|
||||
self.send_response(status)
|
||||
for k, v in headers.items():
|
||||
self.send_header(k, v)
|
||||
self.send_header('Content-Length', str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
return
|
||||
p = folder / self.path.lstrip('/').split('?')[0]
|
||||
if not p.is_file():
|
||||
self.send_error(404)
|
||||
return
|
||||
data = p.read_bytes()
|
||||
rng = self.headers.get('Range')
|
||||
if rng and rng.startswith('bytes='):
|
||||
start = int(rng[6:].split('-')[0])
|
||||
if start >= len(data):
|
||||
self.send_error(416)
|
||||
return
|
||||
self.send_response(206)
|
||||
self.send_header('Content-Range', f'bytes {start}-{len(data) - 1}/{len(data)}')
|
||||
data = data[start:]
|
||||
else:
|
||||
self.send_response(200)
|
||||
self.send_header('Content-Length', str(len(data)))
|
||||
self.end_headers()
|
||||
try:
|
||||
self.wfile.write(data)
|
||||
except ConnectionError: # the client stopped reading (the cancel test)
|
||||
pass
|
||||
|
||||
self.httpd = http.server.ThreadingHTTPServer(('127.0.0.1', 0), H)
|
||||
self.url = f'http://127.0.0.1:{self.httpd.server_address[1]}/'
|
||||
threading.Thread(target=self.httpd.serve_forever, daemon=True).start()
|
||||
|
||||
def close(self):
|
||||
self.httpd.shutdown()
|
||||
|
||||
|
||||
def sha(data):
|
||||
return hashlib.sha256(data).hexdigest()
|
||||
|
||||
|
||||
def tar_bytes(files, compress=True):
|
||||
"""A tar (zstd-compressed when the stdlib has zstd, else gzip) of {name: bytes}."""
|
||||
buf = io.BytesIO()
|
||||
mode = 'w:gz' if not (compress and ZSTD) else 'w'
|
||||
with tarfile.open(fileobj=buf, mode=mode) as t:
|
||||
for name, data in files.items():
|
||||
info = tarfile.TarInfo(name)
|
||||
info.size = len(data)
|
||||
t.addfile(info, io.BytesIO(data))
|
||||
data = buf.getvalue()
|
||||
return ZSTD.compress(data) if compress and ZSTD else data
|
||||
|
||||
|
||||
def desc(name, version, filename, data, depends=(), provides=()):
|
||||
lines = ['%FILENAME%', filename, '', '%NAME%', name, '', '%VERSION%', version, '', '%CSIZE%', str(len(data)), '',
|
||||
'%ISIZE%', str(len(data) * 3), '', '%SHA256SUM%', sha(data), '']
|
||||
if depends:
|
||||
lines += ['%DEPENDS%', *depends, '']
|
||||
if provides:
|
||||
lines += ['%PROVIDES%', *provides, '']
|
||||
return '\n'.join(lines).encode()
|
||||
|
||||
|
||||
def fake_repo(folder, damage=None):
|
||||
"""A pacman repository with the ten root packages (each holding one of make_game.py's required files), one
|
||||
dependency and one name provided by another package."""
|
||||
P = toolchain.PREFIX
|
||||
files = {'clang': {'ucrt64/bin/clang.exe': b'clang', 'ucrt64/bin/clang++.exe': b'clang++'},
|
||||
'sdl3': {'ucrt64/bin/SDL3.dll': b'sdl', 'ucrt64/lib/libSDL3.dll.a': b'implib'},
|
||||
'vulkan-memory-allocator': {'ucrt64/include/vma/vk_mem_alloc.h': b'// vma'},
|
||||
'vulkan-loader': {'ucrt64/lib/libvulkan-1.dll.a': b'implib'},
|
||||
'llvm-libs': {'ucrt64/bin/libLLVM.dll': b'llvm', 'ucrt64/share/doc/llvm/README': b'skipped'}}
|
||||
deps = {'clang': [P + 'llvm-libs=23.1', P + 'gcc'], 'sdl3': [P + 'vulkan']}
|
||||
entries = {}
|
||||
for root in list(toolchain.ROOTS) + ['llvm-libs']:
|
||||
name = P + root
|
||||
fn = f'{name}-1.0-1-any.pkg.tar.zst'
|
||||
data = tar_bytes({'.PKGINFO': b'pkg', **files.get(root, {f'ucrt64/share/{root}/x': b'x'})})
|
||||
if damage == root:
|
||||
(folder / fn).write_bytes(data[:-3] + b'bad')
|
||||
else:
|
||||
(folder / fn).write_bytes(data)
|
||||
entries[f'{name}-1.0-1/desc'] = desc(name, '1.0-1', fn, data, deps.get(root, ()),
|
||||
[P + 'vulkan'] if root == 'vulkan-loader' else ())
|
||||
(folder / 'ucrt64.db').write_bytes(tar_bytes(entries))
|
||||
|
||||
|
||||
class Tmp(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = Path(tempfile.mkdtemp(prefix='nx2navi-test-'))
|
||||
self.addCleanup(shutil.rmtree, self.tmp, True)
|
||||
self.layout = Layout(root=self.tmp / 'launcher', app=ROOT).ensure()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- versions, settings
|
||||
|
||||
class VersionTest(unittest.TestCase):
|
||||
def test_order(self):
|
||||
self.assertTrue(versions.newer('0.3.11', '0.3.10'))
|
||||
self.assertTrue(versions.newer('0.4', '0.3.99'))
|
||||
self.assertFalse(versions.newer('0.3.10+abc', '0.3.10'))
|
||||
self.assertTrue(versions.newer('1.0.0', '1.0.0-rc1'))
|
||||
self.assertEqual(versions.display('0.3.11+abc1234'), '0.3.11')
|
||||
with self.assertRaises(ValueError):
|
||||
versions.parse('banana')
|
||||
|
||||
|
||||
class SettingsTest(Tmp):
|
||||
def test_roundtrip_and_reset(self):
|
||||
s = settings.Settings(self.layout.settings)
|
||||
self.assertTrue(s['auto_check'])
|
||||
s['auto_check'], s['game_dir'], s['channel'] = False, 'D:/g', 'preview'
|
||||
s.save()
|
||||
t = settings.Settings(self.layout.settings)
|
||||
self.assertEqual((t['auto_check'], t['game_dir'], t['channel']), (False, 'D:/g', 'preview'))
|
||||
t.reset()
|
||||
self.assertEqual((t['auto_check'], t['game_dir'], t['channel']), (True, 'D:/g', 'stable'))
|
||||
|
||||
def test_bad_types_ignored(self):
|
||||
self.layout.settings.write_text(json.dumps({'auto_check': 'yes', 'channel': 'preview', 'junk': 1}))
|
||||
s = settings.Settings(self.layout.settings)
|
||||
self.assertEqual((s['auto_check'], s['channel']), (True, 'preview'))
|
||||
self.assertNotIn('junk', s)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- toolchain
|
||||
|
||||
class ToolchainTest(Tmp):
|
||||
def test_resolve_and_install(self):
|
||||
repo = self.tmp / 'repo'
|
||||
repo.mkdir()
|
||||
fake_repo(repo)
|
||||
srv = Server(repo)
|
||||
self.addCleanup(srv.close)
|
||||
lock = toolchain.make_lock(srv.url)
|
||||
names = [p['name'] for p in lock['packages']]
|
||||
self.assertEqual(names[0], toolchain.PREFIX + 'clang')
|
||||
self.assertIn(toolchain.PREFIX + 'llvm-libs', names) # a dependency with a version constraint
|
||||
self.assertEqual(names.count(toolchain.PREFIX + 'vulkan-loader'), 1) # 'vulkan' resolves to its provider
|
||||
self.assertFalse(toolchain.is_current(self.layout, lock))
|
||||
phases = []
|
||||
toolchain.install(self.layout, lock, progress=lambda ph, d, t: phases.append(ph), log=lambda m: None)
|
||||
self.assertTrue(toolchain.is_current(self.layout, lock))
|
||||
self.assertTrue((self.layout.msys / 'bin' / 'clang.exe').is_file())
|
||||
self.assertFalse((self.layout.msys / 'share' / 'doc').exists()) # documentation left out
|
||||
self.assertIn('download', phases)
|
||||
self.assertIn('unpack', phases)
|
||||
hits = len(srv.hits)
|
||||
toolchain.install(self.layout, lock, log=lambda m: None) # cached package files: no download
|
||||
self.assertEqual(len(srv.hits), hits)
|
||||
self.assertEqual(toolchain.lock_digest(lock), toolchain.installed(self.layout)['digest'])
|
||||
|
||||
def test_damaged_package_refused(self):
|
||||
repo = self.tmp / 'repo'
|
||||
repo.mkdir()
|
||||
fake_repo(repo, damage='opus')
|
||||
srv = Server(repo)
|
||||
self.addCleanup(srv.close)
|
||||
lock = toolchain.make_lock(srv.url)
|
||||
with self.assertRaises(toolchain.ToolchainError):
|
||||
toolchain.install(self.layout, lock, log=lambda m: None)
|
||||
self.assertFalse(toolchain.is_current(self.layout, lock))
|
||||
|
||||
def test_wheel_tag(self):
|
||||
tag = toolchain.wheel_tag()[0]
|
||||
self.assertTrue(tag.startswith(f'cp{sys.version_info.major}{sys.version_info.minor}-'))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- net
|
||||
|
||||
class NetTest(Tmp):
|
||||
def test_download_resume_and_checks(self):
|
||||
data = os.urandom(3 * net.CHUNK + 123)
|
||||
(self.tmp / 'f.bin').write_bytes(data)
|
||||
srv = Server(self.tmp)
|
||||
self.addCleanup(srv.close)
|
||||
dest = self.tmp / 'out' / 'f.bin'
|
||||
dest.parent.mkdir()
|
||||
(dest.parent / 'f.bin.part').write_bytes(data[:1000]) # a part file from a cut connection
|
||||
net.download(srv.url + 'f.bin', dest, sha256=sha(data), size=len(data))
|
||||
self.assertEqual(dest.read_bytes(), data)
|
||||
self.assertEqual(srv.hits[-1][1].get('Range'), 'bytes=1000-')
|
||||
with self.assertRaises(net.DownloadError):
|
||||
net.download(srv.url + 'f.bin', self.tmp / 'out' / 'g.bin', sha256='0' * 64)
|
||||
self.assertFalse((self.tmp / 'out' / 'g.bin').exists())
|
||||
with self.assertRaises(net.Cancelled):
|
||||
net.download(srv.url + 'f.bin', self.tmp / 'out' / 'h.bin', cancel=lambda: True)
|
||||
with self.assertRaises(net.DownloadError):
|
||||
net.download(srv.url + 'missing.bin', self.tmp / 'out' / 'm.bin')
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- feed and updater
|
||||
|
||||
def make_app_zip(folder, ver, extra=None):
|
||||
app = folder / f'app-{ver}'
|
||||
(app / 'launcher').mkdir(parents=True)
|
||||
(app / 'launcher' / 'start.pyw').write_text('# test\n')
|
||||
(app / 'VERSION').write_text(ver + '\n')
|
||||
z = folder / f'NX2navi-app-{ver}.zip'
|
||||
with zipfile.ZipFile(z, 'w') as zf:
|
||||
for p in app.rglob('*'):
|
||||
if p.is_file():
|
||||
zf.write(p, p.relative_to(app).as_posix())
|
||||
for name, data in (extra or {}).items():
|
||||
zf.writestr(name, data)
|
||||
return z
|
||||
|
||||
|
||||
def manifest_for(z, ver):
|
||||
return {'format': 1, 'version': ver, 'notes': '- One\n- Two',
|
||||
'app': {'name': z.name, 'size': z.stat().st_size, 'sha256': sha(z.read_bytes())}}
|
||||
|
||||
|
||||
class FeedTest(Tmp):
|
||||
def serve_github(self):
|
||||
self.routes = {}
|
||||
srv = Server(self.tmp, self.routes)
|
||||
self.addCleanup(srv.close)
|
||||
return srv
|
||||
|
||||
def publish(self, releases):
|
||||
self.routes['/repos/o/r/releases?per_page=20'] = (200, {'Content-Type': 'application/json'},
|
||||
json.dumps(releases).encode())
|
||||
|
||||
def release(self, srv, tag, ver, pre=False, manifest=True):
|
||||
z = make_app_zip(self.tmp, ver)
|
||||
m = self.tmp / f'm-{ver}.json'
|
||||
m.write_text(json.dumps(manifest_for(z, ver)))
|
||||
assets = [{'name': z.name, 'browser_download_url': srv.url + z.name, 'url': srv.url + z.name}]
|
||||
if manifest:
|
||||
assets.append({'name': feed.MANIFEST, 'browser_download_url': srv.url + m.name, 'url': srv.url + m.name})
|
||||
return {'tag_name': tag, 'prerelease': pre, 'draft': False, 'body': 'notes', 'html_url': 'x', 'assets': assets}
|
||||
|
||||
def test_github_channels(self):
|
||||
srv = self.serve_github()
|
||||
self.publish([self.release(srv, 'v0.4.0-rc1', '0.4.0-rc1', pre=True),
|
||||
self.release(srv, 'v0.3.12', '0.3.12', manifest=False), self.release(srv, 'v0.3.11', '0.3.11')])
|
||||
f = {'type': 'github', 'repo': 'o/r', 'api': srv.url.rstrip('/')}
|
||||
self.assertEqual(feed.latest(f, 'stable').version, '0.3.11') # 0.3.12 lacks a manifest
|
||||
self.assertEqual(feed.latest(f, 'preview').version, '0.4.0-rc1')
|
||||
r = feed.latest(f, 'stable', token='secret')
|
||||
self.assertEqual(r.headers.get('Authorization'), 'Bearer secret')
|
||||
self.assertTrue(any(h.get('Authorization') == 'Bearer secret' for _, h in srv.hits))
|
||||
self.assertEqual(feed.plain_notes('## Head\n- **One**\n* `Two`\n<!-- x -->'), ['Head', 'One', 'Two'])
|
||||
|
||||
def test_url_feed_and_update(self):
|
||||
z = make_app_zip(self.tmp, '0.9.0')
|
||||
(self.tmp / feed.MANIFEST).write_text(json.dumps(manifest_for(z, '0.9.0')))
|
||||
srv = Server(self.tmp)
|
||||
self.addCleanup(srv.close)
|
||||
rel = feed.latest({'type': 'url', 'url': srv.url + feed.MANIFEST})
|
||||
self.assertEqual(rel.version, '0.9.0')
|
||||
self.layout.apps.mkdir(parents=True)
|
||||
(self.layout.apps / 'current.txt').write_text('0.8.0\n')
|
||||
old = self.layout.apps / '0.8.0' / 'launcher'
|
||||
old.mkdir(parents=True)
|
||||
(old / 'start.pyw').write_text('')
|
||||
for stale in ('0.7.0', '0.6.0'):
|
||||
(self.layout.apps / stale).mkdir()
|
||||
folder = updater.stage_app(self.layout, rel)
|
||||
self.assertTrue((folder / 'launcher' / 'start.pyw').is_file())
|
||||
updater.switch_to(self.layout, '0.9.0')
|
||||
self.assertEqual((updater.current(self.layout), updater.previous(self.layout)), ('0.9.0', '0.8.0'))
|
||||
updater.prune(self.layout)
|
||||
self.assertEqual(sorted(p.name for p in self.layout.apps.iterdir() if p.is_dir()), ['0.8.0', '0.9.0'])
|
||||
self.assertTrue(updater.rollback(self.layout))
|
||||
self.assertEqual(updater.current(self.layout), '0.8.0')
|
||||
self.assertFalse(updater.rollback(self.layout)) # nothing further back
|
||||
|
||||
def test_unsafe_or_wrong_archives(self):
|
||||
z = make_app_zip(self.tmp, '1.0.0', extra={'../evil.txt': 'x'})
|
||||
with self.assertRaises(updater.UpdateError):
|
||||
updater.safe_extract(z, self.tmp / 'x')
|
||||
self.assertFalse((self.tmp / 'evil.txt').exists())
|
||||
z2 = make_app_zip(self.tmp, '1.0.1')
|
||||
m = manifest_for(z2, '1.0.2') # the archive says 1.0.1
|
||||
(self.tmp / feed.MANIFEST).write_text(json.dumps(m))
|
||||
srv = Server(self.tmp)
|
||||
self.addCleanup(srv.close)
|
||||
rel = feed.Release(m, assets={z2.name: srv.url + z2.name})
|
||||
with self.assertRaises(updater.UpdateError):
|
||||
updater.stage_app(self.layout, rel)
|
||||
self.assertFalse((self.layout.apps / '1.0.2').exists())
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- gamefiles
|
||||
|
||||
def sarc(files):
|
||||
names = b''
|
||||
offs = []
|
||||
for n in files:
|
||||
offs.append(len(names))
|
||||
b = n.encode() + b'\0'
|
||||
names += b + b'\0' * (-len(b) % 4)
|
||||
data, ents = b'', b''
|
||||
for (n, d), no in zip(files.items(), offs):
|
||||
start = len(data)
|
||||
data += d
|
||||
ents += struct.pack('<IIII', 0, 0x01000000 | (no // 4), start, len(data))
|
||||
data += b'\0' * (-len(data) % 8)
|
||||
sfat = b'SFAT' + struct.pack('<HHI', 0xC, len(files), 0x65) + ents
|
||||
sfnt = b'SFNT' + struct.pack('<HH', 8, 0) + names
|
||||
base = 0x14 + len(sfat) + len(sfnt)
|
||||
head = b'SARC' + struct.pack('<HHIIHH', 0x14, 0xFEFF, base + len(data), base, 0x100, 0)
|
||||
return head + sfat + sfnt + data
|
||||
|
||||
|
||||
def bfotf(otf, key=0x1234ABCD):
|
||||
body = otf + b'\0' * (-len(otf) % 4)
|
||||
words = struct.unpack(f'>{len(body) // 4}I', body)
|
||||
return struct.pack('>II', 0x7F9A0218 ^ key, len(otf) ^ key) + struct.pack(f'>{len(words)}I',
|
||||
*(w ^ key for w in words))
|
||||
|
||||
|
||||
@unittest.skipUnless(ZSTD, 'needs compression.zstd (Python 3.14)')
|
||||
class FontTest(Tmp):
|
||||
FONT = Path('/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf')
|
||||
|
||||
@unittest.skipUnless(FONT.is_file(), 'needs DejaVu Sans')
|
||||
def test_font_from_romfs_image(self):
|
||||
otf = self.FONT.read_bytes()
|
||||
self.assertEqual(gamefiles.bfotf_to_otf(bfotf(otf)), otf)
|
||||
self.assertIsNone(gamefiles.bfotf_to_otf(b'\0' * 64))
|
||||
samples = [otf[i:i + 512] for i in range(0, min(len(otf), 400000), 997)]
|
||||
d = ZSTD.train_dict(samples, 16384)
|
||||
packed = ZSTD.compress(bfotf(otf), zstd_dict=d)
|
||||
self.assertNotEqual(ZSTD.get_frame_info(packed).dictionary_id, 0)
|
||||
files = {'Pack/ZsDic.pack.zs': ZSTD.compress(sarc({'zs.zsdic': d.dict_content})),
|
||||
gamefiles.FONTS['title']: packed}
|
||||
romfs = self.tmp / 'game' / 'romfs'
|
||||
romfs.mkdir(parents=True)
|
||||
entries, blob = [], b''
|
||||
for path, data in sorted(files.items()):
|
||||
blob += b'\0' * (-len(blob) % 16)
|
||||
entries.append({'path': path, 'offset': len(blob), 'size': len(data)})
|
||||
blob += data
|
||||
(romfs / 'romfs.bin').write_bytes(blob)
|
||||
(romfs / 'index.json').write_text(json.dumps({'files': entries}))
|
||||
got = gamefiles.extract_fonts(self.layout, self.tmp / 'game')
|
||||
self.assertEqual(got['title'][1], 'DejaVu Sans')
|
||||
self.assertEqual(Path(got['title'][0]).read_bytes(), otf)
|
||||
self.assertEqual(gamefiles.cached_fonts(self.layout)['title'][1], 'DejaVu Sans')
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- make_game.py
|
||||
|
||||
def nso(build_id=b'\0' * 16, size=0x300):
|
||||
head = bytearray(0x100)
|
||||
head[0:4] = b'NSO0'
|
||||
segs = [(0x100 + i * size, i * 0x10000, size) for i in range(3)]
|
||||
for i, (fo, mo, sz) in enumerate(segs):
|
||||
struct.pack_into('<III', head, 0x10 + i * 0x10, fo, mo, sz)
|
||||
struct.pack_into('<I', head, 0x60 + i * 4, sz)
|
||||
head[0x40:0x50] = build_id
|
||||
return bytes(head) + b''.join(bytes([i + 1]) * size for i in range(3))
|
||||
|
||||
|
||||
def write_nup(path, exefs, romfs, title_id=0x0400486024DEC000):
|
||||
"""A .nup (tools/nup.py's format) with one base package: exefs {name: bytes}, romfs {path: bytes}."""
|
||||
import nup as nupmod
|
||||
pool = bytearray()
|
||||
|
||||
def name(s):
|
||||
off = len(pool)
|
||||
pool.extend(s.encode())
|
||||
return off, len(s.encode())
|
||||
|
||||
dirs, files = [], []
|
||||
|
||||
def add_tree(tree):
|
||||
"""tree: {name: bytes | dict}; returns the directory index."""
|
||||
idx = len(dirs)
|
||||
dirs.append(None)
|
||||
kids = []
|
||||
for n, v in sorted(tree.items()):
|
||||
if isinstance(v, dict):
|
||||
kids.append(('d', add_tree(v), n))
|
||||
else:
|
||||
files.append([n, v])
|
||||
kids.append(('f', len(files) - 1, n))
|
||||
dirs[idx] = kids
|
||||
return idx
|
||||
|
||||
root = add_tree({'exefs': exefs, 'romfs': _nest(romfs)})
|
||||
nd = len(dirs)
|
||||
dir_rows, child_list = [], []
|
||||
for kids in dirs:
|
||||
first = len(child_list)
|
||||
for kind, i, _n in kids:
|
||||
child_list.append(i if kind == 'd' else nd + i)
|
||||
dir_rows.append((first, len(kids)))
|
||||
dir_names = {root: ''}
|
||||
for kids in dirs:
|
||||
for kind, i, n in kids:
|
||||
if kind == 'd':
|
||||
dir_names[i] = n
|
||||
exefs_dir = next(i for k, i, n in dirs[root] if k == 'd' and n == 'exefs')
|
||||
romfs_dir = next(i for k, i, n in dirs[root] if k == 'd' and n == 'romfs')
|
||||
toc_head = 0x30 + nupmod.PACKAGE.size + nd * nupmod.DIRECTORY.size + len(files) * nupmod.FILE.size + \
|
||||
4 * len(child_list)
|
||||
blob_off = nupmod.SUPER.size
|
||||
data = bytearray()
|
||||
file_rows = []
|
||||
for n, v in files:
|
||||
no, nl = name(n)
|
||||
file_rows.append((no, nl, len(v), blob_off + len(data), len(v), hashlib.sha256(v).digest(), 0))
|
||||
data.extend(v)
|
||||
dir_name_rows = [name(dir_names[i]) for i in range(nd)]
|
||||
toc_off = blob_off + len(data)
|
||||
toc = bytearray(struct.pack('<IIQQQQQ', 0x01434F54, 1, nd, len(files), toc_head, len(pool), 0))
|
||||
toc += nupmod.PACKAGE.pack(title_id, root, exefs_dir, romfs_dir, 0, 0, 0, 0, 0, 0, 0, b'\0' * 20)
|
||||
for i in range(nd):
|
||||
toc += nupmod.DIRECTORY.pack(*dir_name_rows[i], *dir_rows[i], 0)
|
||||
for no, nl, size, off, stored, digest, codec in file_rows:
|
||||
toc += nupmod.FILE.pack(no, nl, size, off, stored, digest, codec, 0, 0, b'\0' * 12)
|
||||
toc += struct.pack(f'<{len(child_list)}I', *child_list)
|
||||
toc += pool
|
||||
total = toc_off + len(toc)
|
||||
sup = nupmod.SUPER.pack(b'NUP\x01', 0x80, 1, toc_off, len(toc), hashlib.sha256(bytes(toc)).digest(), blob_off,
|
||||
total, 1, 0, b'\0' * 16, b'\0' * 32)
|
||||
Path(path).write_bytes(sup + bytes(data) + bytes(toc))
|
||||
|
||||
|
||||
def _nest(flat):
|
||||
tree = {}
|
||||
for p, v in flat.items():
|
||||
parts = p.split('/')
|
||||
node = tree
|
||||
for part in parts[:-1]:
|
||||
node = node.setdefault(part, {})
|
||||
node[parts[-1]] = v
|
||||
return tree
|
||||
|
||||
|
||||
class MakeGameTest(Tmp):
|
||||
def setUp(self):
|
||||
super().setUp()
|
||||
import make_game
|
||||
self.mg = make_game
|
||||
exefs = {'main': nso(bytes.fromhex(make_game.MAIN_BUILD_ID)), 'sdk': nso(), 'subsdk0': nso(), 'rtld': nso(),
|
||||
'main.npdm': b'META' + b'\0' * 0x100}
|
||||
self.romfs = {'a.txt': b'hello', 'dir/b.bin': os.urandom(5000), 'dir/sub/c': b'c' * 33}
|
||||
self.nup = self.tmp / 'game.nup'
|
||||
write_nup(self.nup, exefs, self.romfs)
|
||||
self.msys = self.tmp / 'msys'
|
||||
for need in ('bin/clang.exe', 'bin/clang++.exe', 'bin/SDL3.dll', 'lib/libSDL3.dll.a'):
|
||||
(self.msys / need).parent.mkdir(parents=True, exist_ok=True)
|
||||
(self.msys / need).write_bytes(b'')
|
||||
(self.msys / 'include' / 'vma').mkdir(parents=True)
|
||||
self.layout.pydeps.mkdir(parents=True, exist_ok=True)
|
||||
(self.layout.pydeps / 'zbic.py').write_text('def decompress(b):\n return b\n')
|
||||
self.out = self.tmp / 'game out'
|
||||
|
||||
def run_mg(self, *args):
|
||||
events, lines = [], []
|
||||
run = build.Run(build.command(self.layout, list(args)), events.append, lines.append)
|
||||
code = run.wait()
|
||||
return code, events, lines
|
||||
|
||||
def test_pipeline_rebuild_and_keys(self):
|
||||
code, ev, lines = self.run_mg('--nup', self.nup, '--out', self.out, '--msys', self.msys, '--until', 'romfs')
|
||||
self.assertEqual(code, 0, '\n'.join(lines))
|
||||
states = [(e['step'], e['state']) for e in ev if e['ev'] == 'step']
|
||||
self.assertEqual(states, [('check', 'start'), ('check', 'done'), ('exefs', 'start'), ('exefs', 'done'),
|
||||
('romfs', 'start'), ('romfs', 'done')])
|
||||
self.assertTrue(any(e['ev'] == 'progress' and e['step'] == 'romfs' and e['done'] == e['total'] for e in ev))
|
||||
index = json.loads((self.out / 'romfs' / 'index.json').read_text())
|
||||
blob = (self.out / 'romfs' / 'romfs.bin').read_bytes()
|
||||
for e in index['files']:
|
||||
self.assertEqual(blob[e['offset']:e['offset'] + e['size']], self.romfs[e['path']])
|
||||
self.assertEqual(e['offset'] % 16, 0)
|
||||
flat = (self.out / 'exefs' / 'main').read_bytes()
|
||||
self.assertEqual(flat[0x40:0x50].hex(), self.mg.MAIN_BUILD_ID)
|
||||
# a second run with the same input skips both steps
|
||||
code, ev, _ = self.run_mg('--nup', self.nup, '--out', self.out, '--msys', self.msys, '--until', 'romfs')
|
||||
self.assertEqual([e['state'] for e in ev if e['ev'] == 'step' and e['step'] in ('exefs', 'romfs')],
|
||||
['skip', 'skip'])
|
||||
# --rebuild needs no game files: it checks the unpacked ExeFS and keeps both steps
|
||||
self.nup.rename(self.tmp / 'moved.nup')
|
||||
code, ev, lines = self.run_mg('--rebuild', '--out', self.out, '--msys', self.msys, '--until', 'romfs')
|
||||
self.assertEqual(code, 0, '\n'.join(lines))
|
||||
self.assertEqual([e['state'] for e in ev if e['ev'] == 'step' and e['step'] in ('exefs', 'romfs')],
|
||||
['skip', 'skip'])
|
||||
# the exefs and romfs keys do not depend on the kit; the lift key does
|
||||
st = json.loads((self.out / 'build-cache' / 'state.json').read_text())
|
||||
self.assertEqual(st['steps']['exefs'], self.mg.step_key('exefs', st['input']))
|
||||
self.assertNotIn(self.mg.kit_version(), st['steps']['romfs'])
|
||||
|
||||
def test_rebuild_without_build(self):
|
||||
code, ev, lines = self.run_mg('--rebuild', '--out', self.tmp / 'empty', '--msys', self.msys)
|
||||
self.assertEqual(code, 1)
|
||||
self.assertTrue(any(e['ev'] == 'error' and 'rebuild' in e['message'] for e in ev))
|
||||
|
||||
def test_wrong_version_refused(self):
|
||||
bad = self.tmp / 'bad.nup'
|
||||
write_nup(bad, {'main': nso(b'\x11' * 16), 'sdk': nso(), 'subsdk0': nso(), 'rtld': nso(),
|
||||
'main.npdm': b'META' + b'\0' * 0x100}, {'a': b'a'})
|
||||
code, ev, _ = self.run_mg('--nup', bad, '--check')
|
||||
self.assertEqual(code, 1)
|
||||
self.assertTrue(any(e['ev'] == 'error' and '1.1.1' in e['message'] for e in ev))
|
||||
|
||||
def test_parse_deps(self):
|
||||
text = 'C:/o/x.o: C:/src/x.c C:/inc/a\\ b.h \\\n C:/inc/c.h\n'
|
||||
self.assertEqual(self.mg.parse_deps(text), ['C:/src/x.c', 'C:/inc/a b.h', 'C:/inc/c.h'])
|
||||
|
||||
@unittest.skipUnless(shutil.which('cc'), 'needs a C compiler')
|
||||
def test_object_cache(self):
|
||||
src = self.tmp / 'src'
|
||||
src.mkdir()
|
||||
(src / 'h.h').write_text('#define V 1\n')
|
||||
(src / 'a.c').write_text('#include "h.h"\nint a(void) { return V; }\n')
|
||||
(src / 'b.c').write_text('int b(void) { return 2; }\n')
|
||||
obj = self.tmp / 'obj'
|
||||
obj.mkdir()
|
||||
srcs = [(src / 'a.c', obj / 'a.o'), (src / 'b.c', obj / 'b.o')]
|
||||
|
||||
def cmd_for(s, d):
|
||||
return ['cc', '-c', '-I', str(src), '-o', str(d), str(s)]
|
||||
|
||||
def go():
|
||||
cache = self.mg.ObjectCache(obj / 'objects.json')
|
||||
return self.mg.compile_all(srcs, cmd_for, cache, 2, dict(os.environ), salt='cc')
|
||||
self.assertEqual(go()[:2], (2, 0))
|
||||
self.assertEqual(go()[:2], (0, 2))
|
||||
(src / 'h.h').write_text('#define V 3\n') # a header: only a.c includes it
|
||||
self.assertEqual(go()[:2], (1, 1))
|
||||
(src / 'b.c').write_text('int b(void) { return syntax error }\n')
|
||||
compiled, reused, bad = go()
|
||||
self.assertEqual((compiled, len(bad)), (1, 1))
|
||||
self.assertNotIn('b.o', json.loads((obj / 'objects.json').read_text()))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- jobs
|
||||
|
||||
FAKE_MAKE_GAME = r'''
|
||||
import json, sys, time
|
||||
def ev(**k): print('@@' + json.dumps(k), flush=True)
|
||||
if '--fail' in open(sys.argv[0] + '.mode').read():
|
||||
ev(ev='step', step='check', state='start'); ev(ev='error', message='this is not game version 1.1.1'); sys.exit(1)
|
||||
for s in ('check', 'exefs', 'romfs', 'lift', 'compile', 'shaders'):
|
||||
ev(ev='step', step=s, state='start')
|
||||
if s == 'compile':
|
||||
ev(ev='progress', step=s, done=5, total=10, unit='files'); ev(ev='phase', step=s, phase='link')
|
||||
print('working on', s, flush=True)
|
||||
ev(ev='step', step=s, state='done', seconds=0.1)
|
||||
ev(ev='end', exe='x')
|
||||
'''
|
||||
|
||||
|
||||
class JobTest(Tmp):
|
||||
def make_ctx(self, mode):
|
||||
app = self.tmp / 'app'
|
||||
(app / 'tools').mkdir(parents=True)
|
||||
(app / 'VERSION').write_text('0.3.10\n')
|
||||
script = app / 'tools' / 'make_game.py'
|
||||
script.write_text(FAKE_MAKE_GAME)
|
||||
(app / 'tools' / 'make_game.py.mode').write_text(mode)
|
||||
ctx = Context(Layout(root=self.tmp / 'root', app=app))
|
||||
return ctx
|
||||
|
||||
def patch_tools(self):
|
||||
saved = toolchain.load_lock, toolchain.is_current, toolchain.ensure_zbic
|
||||
toolchain.load_lock = lambda layout, version='dev': {'packages': []}
|
||||
toolchain.is_current = lambda layout, lock: True
|
||||
toolchain.ensure_zbic = lambda *a, **k: None
|
||||
self.addCleanup(lambda: setattr(toolchain, 'load_lock', saved[0]) or
|
||||
setattr(toolchain, 'is_current', saved[1]) or setattr(toolchain, 'ensure_zbic', saved[2]))
|
||||
|
||||
def test_install_job(self):
|
||||
self.patch_tools()
|
||||
ctx = self.make_ctx('ok')
|
||||
j = jobs.InstallJob(ctx, self.tmp / 'g.nup', self.tmp / 'game').run_inline()
|
||||
self.assertEqual(j.result, 'ok', j.error)
|
||||
self.assertEqual([s.state for s in j.steps], [jobs.SKIPPED] + [jobs.DONE] * 7) # tools ... shaders, ready
|
||||
self.assertEqual(j.step('check').value, 'Version 1.1.1')
|
||||
self.assertTrue(any('working on lift' in ln for ln in j.lines))
|
||||
self.assertEqual(j.overall(), 1.0)
|
||||
|
||||
def test_failed_job(self):
|
||||
self.patch_tools()
|
||||
ctx = self.make_ctx('--fail')
|
||||
j = jobs.InstallJob(ctx, self.tmp / 'g.nup', self.tmp / 'game').run_inline()
|
||||
self.assertEqual(j.result, 'failed')
|
||||
self.assertIn('1.1.1', j.error)
|
||||
self.assertEqual(j.step('check').state, jobs.FAILED)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- the window
|
||||
|
||||
def have_display():
|
||||
if os.name == 'nt':
|
||||
return True
|
||||
if not os.environ.get('DISPLAY'):
|
||||
return False
|
||||
try:
|
||||
import tkinter # noqa: F401
|
||||
import numpy # noqa: F401
|
||||
return True
|
||||
except ImportError:
|
||||
return False
|
||||
|
||||
|
||||
@unittest.skipUnless(have_display(), 'needs tkinter, numpy and a display')
|
||||
class WindowTest(Tmp):
|
||||
def test_selftest(self):
|
||||
r = subprocess.run([sys.executable, '-m', 'launcher', '--root', str(self.tmp / 'w'), '--selftest'],
|
||||
cwd=ROOT, capture_output=True, text=True, timeout=180)
|
||||
self.assertEqual(r.returncode, 0, r.stderr)
|
||||
self.assertIn('8 screens drawn', r.stdout)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main(verbosity=2)
|
||||
@@ -3,8 +3,9 @@
|
||||
python -I tools/make_game.py --nup "C:\\path\\game.nup" --out "C:\\Games\\Ocarina"
|
||||
python -I tools/make_game.py --exefs "C:\\path\\exefs" --romfs "C:\\path\\romfs" --out "C:\\Games\\Ocarina"
|
||||
python -I tools/make_game.py --nup "C:\\path\\game.nup" --check # only checks the game's version
|
||||
python -I tools/make_game.py --rebuild --out "C:\\Games\\Ocarina" # a newer kit, the files of the last build
|
||||
|
||||
Steps, each skipped when a run before finished it for the same input (state in <out>/build-cache/state.json):
|
||||
Steps, each skipped when a run before finished it for the same inputs (state in <out>/build-cache/state.json):
|
||||
1. check the title (0400486024DEC000) and main's build id (game 1.1.1)
|
||||
2. exefs the five ExeFS files, every module's segments unpacked (ZBIC through the player's own `zbic` package)
|
||||
and stored plainly, as the runtime's loader reads them -> <out>/exefs
|
||||
@@ -16,15 +17,28 @@ Steps, each skipped when a run before finished it for the same input (state in <
|
||||
6. shaders every shader in the game files (tools/prepare_shaders.py, default bind state) translated, and each one's
|
||||
GPU driver build done once, so play does not stop to compile them (`ocarina.exe --prepare-shaders`)
|
||||
|
||||
The exefs and romfs steps depend only on the game files; lift and compile depend on this kit's code too. A newer kit
|
||||
therefore lifts and compiles again but keeps the unpacked game files, and --rebuild needs no game files at all. The
|
||||
compile step keeps every object with the dependency list clang wrote for it (build-cache/obj/objects.json) and
|
||||
compiles a file again only when it, a header it includes, the flags or the compiler changed. The new ocarina.exe
|
||||
replaces the old one only once it links, so a failed build leaves the last good one in place.
|
||||
|
||||
Nothing is written outside --out. The game folder then holds only what this PC built from the player's files: never
|
||||
share it. Needs Python 3.10+ with numpy, lz4, zstandard and zbic (pip), and MSYS2 UCRT64's clang and libraries
|
||||
(README-BuildKit.md lists the commands). Delete <out>/build-cache afterwards to free about 5 GB.
|
||||
(README-BuildKit.md lists the commands). Delete <out>/build-cache afterwards to free about 5 GB; the next build then
|
||||
compiles everything again.
|
||||
|
||||
--events adds one machine-readable line per step and progress change to stdout, each starting with "@@" and followed
|
||||
by a JSON object (launcher/build.py reads them): {"ev": "step", "step": ..., "state": "start" | "done" | "skip"},
|
||||
{"ev": "progress", "step": ..., "done": n, "total": n, "unit": ...}, {"ev": "phase", ...}, {"ev": "error", "message":
|
||||
...} and {"ev": "end", ...}.
|
||||
"""
|
||||
import argparse
|
||||
import concurrent.futures
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import struct
|
||||
import subprocess
|
||||
@@ -41,7 +55,8 @@ MAIN_BUILD_ID = 'e0f3ccfaf50065ae4ea01eb5aeb13b1a' # main of game 1.1.1
|
||||
EXEFS_FILES = ('main', 'main.npdm', 'sdk', 'subsdk0', 'rtld')
|
||||
MODULES = ('main', 'sdk', 'subsdk0', 'rtld')
|
||||
ALIGN = 16
|
||||
STATE_VERSION = 1
|
||||
STATE_VERSION = 2
|
||||
STEPS = ('exefs', 'romfs', 'lift', 'compile', 'shaders')
|
||||
|
||||
# Bring-up fixes in the lifted SDK, each anchored on exact generated text that must occur once.
|
||||
LIFT_FIXES = [
|
||||
@@ -61,17 +76,62 @@ LIFT_FIXES = [
|
||||
LINK_LIBS = ['-lvulkan-1', '-lSDL3', '-lopus', '-lvpx', '-l:libdav1d.a', '-lzstd', '-l:libatomic.a', '-lbcrypt',
|
||||
'-limm32', '-luuid', '-lole32', '-luser32', '-lshell32', '-lgdi32', '-ladvapi32', '-lwinmm']
|
||||
SYSTEM_DLLS_SKIP = {'vulkan-1.dll'} # the graphics driver's loader, never a copy
|
||||
EXE_NAME = 'ocarina.exe'
|
||||
|
||||
# Runs a script the way `python script.py` does, with the script's folder first on sys.path. The launcher's
|
||||
# embeddable Python leaves the script folder off sys.path (its ._pth file), so every script this build starts goes
|
||||
# through this.
|
||||
RUN_SCRIPT = ('import os, runpy, sys; s = sys.argv[1]; sys.argv = sys.argv[1:]; '
|
||||
'sys.path.insert(0, os.path.dirname(os.path.abspath(s))); runpy.run_path(s, run_name="__main__")')
|
||||
|
||||
EVENTS = False
|
||||
|
||||
|
||||
def say(msg):
|
||||
print(msg, flush=True)
|
||||
|
||||
|
||||
def emit(ev, **fields):
|
||||
"""One "@@{json}" line for the launcher (--events); nothing otherwise."""
|
||||
if EVENTS:
|
||||
print('@@' + json.dumps({'ev': ev, **fields}, ensure_ascii=False), flush=True)
|
||||
|
||||
|
||||
def fail(msg):
|
||||
emit('error', message=msg)
|
||||
print(f'\nerror: {msg}', file=sys.stderr, flush=True)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def py_cmd(script, *args):
|
||||
"""The command line that runs a Python script of this kit with this interpreter (see RUN_SCRIPT)."""
|
||||
flags = ['-I'] if sys.flags.isolated else []
|
||||
return [sys.executable, *flags, '-c', RUN_SCRIPT, str(script), *map(str, args)]
|
||||
|
||||
|
||||
def sha256_bytes(data):
|
||||
return hashlib.sha256(data).hexdigest()
|
||||
|
||||
|
||||
def sha256_file(path):
|
||||
h = hashlib.sha256()
|
||||
with open(path, 'rb') as f:
|
||||
while b := f.read(8 << 20):
|
||||
h.update(b)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def tree_digest(root, patterns):
|
||||
"""sha256 over the relative names and contents of the files under root matching the glob patterns."""
|
||||
h = hashlib.sha256()
|
||||
files = sorted({p for pat in patterns for p in Path(root).glob(pat) if p.is_file() and '__pycache__' not in p.parts})
|
||||
for p in files:
|
||||
h.update(p.relative_to(root).as_posix().encode() + b'\0')
|
||||
h.update(p.read_bytes())
|
||||
h.update(b'\0')
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- input
|
||||
|
||||
class Source:
|
||||
@@ -137,6 +197,27 @@ class Source:
|
||||
yield b
|
||||
|
||||
|
||||
class BuiltSource:
|
||||
"""--rebuild: the ExeFS and RomFS an earlier run unpacked into the game folder, under that run's input identity.
|
||||
The ExeFS modules are stored plainly there but keep their headers, so check() reads the same build id."""
|
||||
|
||||
def __init__(self, out, state):
|
||||
self.out = out
|
||||
if state.get('version') != STATE_VERSION or not state.get('input'):
|
||||
fail(f'{out} holds no finished build of this kit to rebuild from: give --nup, or --exefs and --romfs')
|
||||
for step in ('exefs', 'romfs'):
|
||||
if step not in state.get('steps', {}):
|
||||
fail(f'the last build in {out} did not finish its {step} step: give the game files again')
|
||||
missing = [n for n in EXEFS_FILES if not (out / 'exefs' / n).is_file()]
|
||||
missing += [f'romfs/{n}' for n in ('romfs.bin', 'index.json') if not (out / 'romfs' / n).is_file()]
|
||||
if missing:
|
||||
fail(f'{out} lacks {", ".join(missing)}: give the game files again')
|
||||
self.identity = state['input']
|
||||
|
||||
def read_exefs(self, name):
|
||||
return (self.out / 'exefs' / name).read_bytes()
|
||||
|
||||
|
||||
def check(src):
|
||||
main = src.read_exefs('main')
|
||||
if main[:4] != b'NSO0':
|
||||
@@ -182,11 +263,16 @@ def flat_nso(raw, name):
|
||||
def step_exefs(src, out):
|
||||
dst = out / 'exefs'
|
||||
dst.mkdir(parents=True, exist_ok=True)
|
||||
for name in EXEFS_FILES:
|
||||
for i, name in enumerate(EXEFS_FILES):
|
||||
raw = src.read_exefs(name)
|
||||
data = flat_nso(raw, name) if name in MODULES else raw
|
||||
(dst / name).write_bytes(data)
|
||||
say(f' {name:9} {len(raw) / 1e6:6.1f} MB -> {len(data) / 1e6:6.1f} MB')
|
||||
emit('progress', step='exefs', done=i + 1, total=len(EXEFS_FILES), unit='files')
|
||||
|
||||
|
||||
def exefs_digest(out):
|
||||
return tree_digest(out / 'exefs', EXEFS_FILES)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- romfs
|
||||
@@ -202,6 +288,7 @@ def step_romfs(src, out, jobs):
|
||||
say(f' {len(files)} files, {total / 1e9:.1f} GB')
|
||||
entries, h, pos, done, t0 = [], hashlib.sha256(), 0, 0, time.time()
|
||||
part = dst / 'romfs.bin.part'
|
||||
emit('progress', step='romfs', done=0, total=total, unit='bytes')
|
||||
|
||||
def load(e): # small files are read whole on a worker; big ones are streamed by the writer
|
||||
rel, ref = e
|
||||
@@ -224,6 +311,7 @@ def step_romfs(src, out, jobs):
|
||||
h.update(chunk)
|
||||
pos += size
|
||||
done += size
|
||||
emit('progress', step='romfs', done=done, total=total, unit='bytes')
|
||||
if start // batch % 40 == 0:
|
||||
say(f' {done / 1e9:5.1f} of {total / 1e9:.1f} GB, {time.time() - t0:.0f} s')
|
||||
part.replace(dst / 'romfs.bin')
|
||||
@@ -234,13 +322,31 @@ def step_romfs(src, out, jobs):
|
||||
|
||||
# ---------------------------------------------------------------- lift
|
||||
|
||||
EMITTED = re.compile(r'emitted (\d+)/(\d+) chunks')
|
||||
|
||||
|
||||
def lift_digest():
|
||||
"""The lifter's code, the patch table and the bring-up fixes: what the lifted C depends on besides the ExeFS."""
|
||||
h = hashlib.sha256(tree_digest(KIT / 'recomp', ('*.py', 'patches-v111.txt')).encode())
|
||||
h.update(repr(LIFT_FIXES).encode())
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def step_lift(out, cache):
|
||||
"""gen.py into build-cache/lift. The folder is kept: gen.py rewrites only the chunks whose text changed and
|
||||
removes the ones it no longer writes."""
|
||||
lift = cache / 'lift'
|
||||
if lift.exists():
|
||||
shutil.rmtree(lift)
|
||||
cmd = [sys.executable, str(KIT / 'recomp' / 'gen.py'), '--exefs', str(out / 'exefs'), '--out', str(lift),
|
||||
'--patches', str(KIT / 'recomp' / 'patches-v111.txt')]
|
||||
if subprocess.run(cmd).returncode:
|
||||
cmd = py_cmd(KIT / 'recomp' / 'gen.py', '--exefs', out / 'exefs', '--out', lift,
|
||||
'--patches', KIT / 'recomp' / 'patches-v111.txt')
|
||||
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, encoding='utf-8',
|
||||
errors='replace', bufsize=1)
|
||||
for line in p.stdout:
|
||||
line = line.rstrip('\n')
|
||||
say(f' {line}')
|
||||
m = EMITTED.search(line)
|
||||
if m:
|
||||
emit('progress', step='lift', done=int(m.group(1)), total=int(m.group(2)), unit='chunks')
|
||||
if p.wait():
|
||||
fail('the code generator failed (see above)')
|
||||
for mod, old, new in LIFT_FIXES:
|
||||
hits = [p for p in sorted((lift / mod).glob('*.c')) if old in p.read_text(encoding='utf-8')]
|
||||
@@ -252,14 +358,6 @@ def step_lift(out, cache):
|
||||
|
||||
# ---------------------------------------------------------------- compile
|
||||
|
||||
def sha256_file(path):
|
||||
h = hashlib.sha256()
|
||||
with open(path, 'rb') as f:
|
||||
while b := f.read(8 << 20):
|
||||
h.update(b)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def modules_c(lift, out, gen):
|
||||
layout = json.loads((lift / 'layout.json').read_text(encoding='utf-8'))
|
||||
L = ['#include "dispatch.h"', '#include "host.h"', '#include "layout.h"', '']
|
||||
@@ -286,12 +384,17 @@ def modules_c(lift, out, gen):
|
||||
'const int rt_nvn_api_version[2] = {231, 40}; /* NVN2: main asks for major 231, minor 40 or later */',
|
||||
f'const char rt_port_version[] = {json.dumps(kit_version())};']
|
||||
p = gen / 'modules.c'
|
||||
p.write_text('\n'.join(L) + '\n', encoding='utf-8')
|
||||
(gen / 'shim.c').write_text('/* Defined by the SDL window path only; this build uses the Win32 one. */\n'
|
||||
'void menu_sdl_event(void *e) { (void)e; }\n', encoding='utf-8')
|
||||
write_if_changed(p, '\n'.join(L) + '\n')
|
||||
write_if_changed(gen / 'shim.c', '/* Defined by the SDL window path only; this build uses the Win32 one. */\n'
|
||||
'void menu_sdl_event(void *e) { (void)e; }\n')
|
||||
return p
|
||||
|
||||
|
||||
def write_if_changed(path, text):
|
||||
if not path.is_file() or path.read_text(encoding='utf-8') != text:
|
||||
path.write_text(text, encoding='utf-8')
|
||||
|
||||
|
||||
def kit_version():
|
||||
v = KIT / 'VERSION'
|
||||
return v.read_text(encoding='utf-8').strip() if v.is_file() else '0.0.1-ocarina'
|
||||
@@ -323,6 +426,23 @@ def pe_imports(path):
|
||||
return names
|
||||
|
||||
|
||||
def copy_if_changed(src, dst):
|
||||
"""Copy src over dst unless dst already has its size and modification time. A file the running game holds open
|
||||
cannot be replaced on Windows: that stops the build with a message instead of a traceback."""
|
||||
s = os.stat(src)
|
||||
try:
|
||||
d = os.stat(dst)
|
||||
if d.st_size == s.st_size and int(d.st_mtime) == int(s.st_mtime):
|
||||
return False
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
try:
|
||||
shutil.copy2(src, dst)
|
||||
except PermissionError:
|
||||
fail(f'{dst} is in use: close the game and try again')
|
||||
return True
|
||||
|
||||
|
||||
def copy_dlls(exe, msys_bin, out):
|
||||
seen, todo, copied = set(), [exe], []
|
||||
while todo:
|
||||
@@ -331,21 +451,136 @@ def copy_dlls(exe, msys_bin, out):
|
||||
if name in seen or name in SYSTEM_DLLS_SKIP or not src.is_file():
|
||||
continue
|
||||
seen.add(name)
|
||||
shutil.copy2(src, out / src.name)
|
||||
copy_if_changed(src, out / src.name)
|
||||
copied.append(src.name)
|
||||
todo.append(src)
|
||||
return copied
|
||||
|
||||
|
||||
def step_compile(out, cache, msys, jobs):
|
||||
lift, gen, obj = cache / 'lift', cache / 'gen', cache / 'obj'
|
||||
def parse_deps(text):
|
||||
"""The prerequisites of a make-style dependency file as clang -MMD writes it (the target dropped)."""
|
||||
text = text.replace('\\\r\n', ' ').replace('\\\n', ' ')
|
||||
tokens, cur, i = [], '', 0
|
||||
while i < len(text):
|
||||
c = text[i]
|
||||
if c == '\\' and i + 1 < len(text) and text[i + 1] in ' #':
|
||||
cur += text[i + 1]
|
||||
i += 2
|
||||
continue
|
||||
if c == '$' and text[i + 1:i + 2] == '$':
|
||||
cur += '$'
|
||||
i += 2
|
||||
continue
|
||||
if c.isspace():
|
||||
if cur:
|
||||
tokens.append(cur)
|
||||
cur = ''
|
||||
else:
|
||||
cur += c
|
||||
i += 1
|
||||
if cur:
|
||||
tokens.append(cur)
|
||||
for k, t in enumerate(tokens): # the target ends with ':' (a drive letter's colon is followed by a separator)
|
||||
if t.endswith(':'):
|
||||
return tokens[k + 1:]
|
||||
return tokens
|
||||
|
||||
|
||||
class ObjectCache:
|
||||
"""Which objects are current: per object the key of its command and source, and the sha256 of every file its
|
||||
dependency list names (build-cache/obj/objects.json)."""
|
||||
|
||||
def __init__(self, path):
|
||||
self.path = path
|
||||
try:
|
||||
self.entries = json.loads(path.read_text(encoding='utf-8')) if path.is_file() else {}
|
||||
except (OSError, ValueError):
|
||||
self.entries = {}
|
||||
self._sha = {}
|
||||
|
||||
def file_sha(self, p):
|
||||
p = os.path.normcase(os.path.abspath(p))
|
||||
if p not in self._sha:
|
||||
try:
|
||||
self._sha[p] = sha256_file(p)
|
||||
except OSError:
|
||||
self._sha[p] = None
|
||||
return self._sha[p]
|
||||
|
||||
def current(self, obj, key):
|
||||
e = self.entries.get(obj.name)
|
||||
return (e is not None and e.get('key') == key and obj.is_file()
|
||||
and all(self.file_sha(p) == h for p, h in e.get('deps', {}).items()))
|
||||
|
||||
def record(self, obj, key, dep_file, src):
|
||||
try:
|
||||
deps = parse_deps(Path(dep_file).read_text(encoding='utf-8', errors='replace'))
|
||||
except OSError:
|
||||
deps = []
|
||||
deps = deps or [str(src)]
|
||||
self.entries[obj.name] = {'key': key, 'deps': {p: self.file_sha(p) for p in deps}}
|
||||
|
||||
def forget(self, obj):
|
||||
self.entries.pop(obj.name, None)
|
||||
|
||||
def save(self):
|
||||
tmp = self.path.with_suffix('.tmp')
|
||||
tmp.write_text(json.dumps(self.entries), encoding='utf-8')
|
||||
tmp.replace(self.path)
|
||||
|
||||
|
||||
def compiler_id(clang, env):
|
||||
p = subprocess.run([str(clang), '--version'], capture_output=True, text=True, env=env)
|
||||
st = os.stat(clang)
|
||||
return sha256_bytes(f'{p.stdout}\n{st.st_size}\n{int(st.st_mtime)}'.encode())
|
||||
|
||||
|
||||
def compile_all(srcs, cmd_for, cache, jobs, env, progress=None, salt=''):
|
||||
"""Compile each (source, object) whose object is not current in cache. cmd_for(src, dst) -> argv without the
|
||||
dependency-file options; salt identifies the compiler. Returns (compiled, reused, failures) with failures as
|
||||
(src, stderr tail)."""
|
||||
todo, keys = [], {}
|
||||
for src, dst in srcs:
|
||||
cmd = cmd_for(src, dst)
|
||||
flags = [c for c in cmd if c not in (str(src), str(dst))]
|
||||
keys[dst] = sha256_bytes(json.dumps([salt, flags, cache.file_sha(src)]).encode())
|
||||
if not cache.current(dst, keys[dst]):
|
||||
todo.append((src, dst))
|
||||
reused = len(srcs) - len(todo)
|
||||
|
||||
def cc(job):
|
||||
src, dst = job
|
||||
dep = dst.with_suffix('.d')
|
||||
cmd = cmd_for(src, dst)
|
||||
cmd[1:1] = ['-MMD', '-MF', str(dep)]
|
||||
p = subprocess.run(cmd, capture_output=True, text=True, env=env, errors='replace')
|
||||
return src, dst, dep, p.returncode, p.stderr[-1500:]
|
||||
|
||||
done, bad = reused, []
|
||||
if progress:
|
||||
progress(done, len(srcs))
|
||||
with concurrent.futures.ThreadPoolExecutor(max(1, jobs)) as ex:
|
||||
for src, dst, dep, rc, err in ex.map(cc, todo):
|
||||
done += 1
|
||||
if rc:
|
||||
bad.append((src, err))
|
||||
cache.forget(dst)
|
||||
else:
|
||||
cache.record(dst, keys[dst], dep, src)
|
||||
if progress:
|
||||
progress(done, len(srcs))
|
||||
cache.save()
|
||||
return len(todo), reused, bad
|
||||
|
||||
|
||||
def step_compile(out, cache_dir, msys, jobs):
|
||||
lift, gen, obj = cache_dir / 'lift', cache_dir / 'gen', cache_dir / 'obj'
|
||||
gen.mkdir(parents=True, exist_ok=True)
|
||||
obj.mkdir(parents=True, exist_ok=True)
|
||||
rt = KIT / 'runtime'
|
||||
clang, clangxx = msys / 'bin' / 'clang.exe', msys / 'bin' / 'clang++.exe'
|
||||
env = dict(os.environ, PATH=str(msys / 'bin') + os.pathsep + os.environ.get('PATH', ''))
|
||||
if subprocess.run([sys.executable, str(KIT / 'tools' / 'romfs_index.py'), '--romfs', str(out / 'romfs'),
|
||||
'--out', str(gen)]).returncode:
|
||||
if subprocess.run(py_cmd(KIT / 'tools' / 'romfs_index.py', '--romfs', out / 'romfs', '--out', gen)).returncode:
|
||||
fail('the RomFS index could not be written')
|
||||
srcs = [(p, obj / f'{mod}_{p.stem}.o') for mod in ('main', 'sdk', 'subsdk0') for p in sorted((lift / mod).glob('*.c'))]
|
||||
for pat in ('*.c', '*.cpp', 'platform/*.c', 'ui/*.cpp', 'third_party/imgui/*.cpp', 'third_party/imgui/backends/*.cpp'):
|
||||
@@ -356,53 +591,74 @@ def step_compile(out, cache, msys, jobs):
|
||||
srcs += [(modules_c(lift, out, gen), obj / 'modules.o'), (gen / 'shim.c', obj / 'shim.o')]
|
||||
inc = ['-I', str(rt), '-I', str(lift), '-I', str(gen), '-I', str(rt / 'third_party' / 'imgui'),
|
||||
'-I', str(msys / 'include' / 'vma')]
|
||||
cid = compiler_id(clang, env)
|
||||
|
||||
def cc(job):
|
||||
src, dst = job
|
||||
def cmd_for(src, dst):
|
||||
cpp = src.suffix in ('.cpp', '.cxx')
|
||||
# the lifted code at -O1 (its size makes -O2 slow to build), the runtime at -O2; neither may assume strict aliasing:
|
||||
# the lifted code reaches guest memory through casts of one address to several types (tools/build_game.py's flags)
|
||||
guest = dst.name.startswith(('main_', 'sdk_', 'subsdk0_'))
|
||||
cmd = [str(clangxx if cpp else clang), '-c', '-std=gnu++17' if cpp else '-std=gnu11', '-O1' if guest else '-O2',
|
||||
'-fno-strict-aliasing', '-w', *inc, '-o', str(dst), str(src)]
|
||||
p = subprocess.run(cmd, capture_output=True, text=True, env=env)
|
||||
return src, p.returncode, p.stderr[-1500:]
|
||||
return [str(clangxx if cpp else clang), '-c', '-std=gnu++17' if cpp else '-std=gnu11', '-O1' if guest else '-O2',
|
||||
'-fno-strict-aliasing', '-w', *inc, '-o', str(dst), str(src)]
|
||||
|
||||
t0 = time.time()
|
||||
last = [0.0]
|
||||
|
||||
def progress(done, total):
|
||||
now = time.time()
|
||||
if done == total or now - last[0] >= 0.25:
|
||||
last[0] = now
|
||||
emit('progress', step='compile', done=done, total=total, unit='files')
|
||||
if done % 50 == 0 or done == total:
|
||||
say(f' compiled {done} of {total}, {now - t0:.0f} s')
|
||||
|
||||
say(f' {len(srcs)} files on {jobs} threads')
|
||||
t0, done, bad = time.time(), 0, []
|
||||
with concurrent.futures.ThreadPoolExecutor(jobs) as ex:
|
||||
for src, rc, err in ex.map(cc, srcs):
|
||||
done += 1
|
||||
if rc:
|
||||
bad.append((src, err))
|
||||
if done % 50 == 0 or done == len(srcs):
|
||||
say(f' compiled {done} of {len(srcs)}, {time.time() - t0:.0f} s')
|
||||
cache = ObjectCache(obj / 'objects.json')
|
||||
compiled, reused, bad = compile_all(srcs, cmd_for, cache, jobs, env, progress, salt=cid)
|
||||
say(f' {compiled} compiled, {reused} unchanged since the last build')
|
||||
if bad:
|
||||
for src, err in bad[:3]:
|
||||
print(f'--- {src}\n{err}', file=sys.stderr)
|
||||
fail(f'{len(bad)} files did not compile')
|
||||
|
||||
# the manifest (per-monitor DPI awareness: the picture at the monitor's real size, not one Windows stretches) and the
|
||||
# icon, runtime/nx2navi.rc through MSYS2's windres
|
||||
rco = obj / 'nx2navi_rc.o'
|
||||
r = subprocess.run([str(msys / 'bin' / 'windres.exe'), '-I', str(rt), '-O', 'coff', '-i', str(rt / 'nx2navi.rc'),
|
||||
'-o', str(rco)], capture_output=True, text=True, env=env)
|
||||
'-o', str(rco)], capture_output=True, text=True, env=env, errors='replace')
|
||||
if r.returncode:
|
||||
fail(f'windres failed on runtime/nx2navi.rc: {r.stderr.strip()[:400]}')
|
||||
rsp = cache / 'link.rsp'
|
||||
rsp.write_text('\n'.join('"%s"' % o.as_posix() for o in [o for _, o in srcs] + [rco]), encoding='utf-8')
|
||||
exe = out / 'ocarina.exe'
|
||||
say(' linking')
|
||||
p = subprocess.run([str(clangxx), '-o', str(exe), '@' + str(rsp), '-L', str(msys / 'lib'), *LINK_LIBS],
|
||||
capture_output=True, text=True, env=env)
|
||||
if p.returncode:
|
||||
print(p.stderr[-4000:], file=sys.stderr)
|
||||
fail('the link failed')
|
||||
shutil.copy2(gen / 'romfs-index.bin', out / 'romfs-index.bin')
|
||||
objs = [o for _, o in srcs] + [rco]
|
||||
exe = out / EXE_NAME
|
||||
rsp = cache_dir / 'link.rsp'
|
||||
rsp.write_text('\n'.join('"%s"' % o.as_posix() for o in objs), encoding='utf-8')
|
||||
link_key = sha256_bytes(json.dumps([LINK_LIBS, cid] + [[o.name, o.stat().st_size, o.stat().st_mtime_ns]
|
||||
for o in objs]).encode())
|
||||
state_link = cache_dir / 'link.key'
|
||||
if exe.is_file() and state_link.is_file() and state_link.read_text(encoding='utf-8') == link_key:
|
||||
say(' ocarina.exe is current, not linked again')
|
||||
else:
|
||||
emit('phase', step='compile', phase='link')
|
||||
say(' linking')
|
||||
tmp = out / (EXE_NAME + '.new')
|
||||
p = subprocess.run([str(clangxx), '-o', str(tmp), '@' + str(rsp), '-L', str(msys / 'lib'), *LINK_LIBS],
|
||||
capture_output=True, text=True, env=env, errors='replace')
|
||||
if p.returncode:
|
||||
print(p.stderr[-4000:], file=sys.stderr)
|
||||
tmp.unlink(missing_ok=True)
|
||||
fail('the link failed')
|
||||
try:
|
||||
os.replace(tmp, exe)
|
||||
except PermissionError:
|
||||
tmp.unlink(missing_ok=True)
|
||||
fail(f'{exe} is in use: close the game and try again')
|
||||
state_link.write_text(link_key, encoding='utf-8')
|
||||
copy_if_changed(gen / 'romfs-index.bin', out / 'romfs-index.bin')
|
||||
for name in ('nx2navi_xlate.exe', 'nx2navi_xlate.id'):
|
||||
shutil.copy2(KIT / 'bin' / name, out / name)
|
||||
copy_if_changed(KIT / 'bin' / name, out / name)
|
||||
if (KIT / 'bin' / 'amd_fidelityfx_vk.dll').is_file(): # FSR 3.1 (runtime/vk_ffx.cpp)
|
||||
shutil.copy2(KIT / 'bin' / 'amd_fidelityfx_vk.dll', out / 'amd_fidelityfx_vk.dll')
|
||||
say(f' ocarina.exe {exe.stat().st_size / 1e6:.0f} MB; beside it: {", ".join(copy_dlls(exe, msys / "bin", out))}')
|
||||
copy_if_changed(KIT / 'bin' / 'amd_fidelityfx_vk.dll', out / 'amd_fidelityfx_vk.dll')
|
||||
say(f' {EXE_NAME} {exe.stat().st_size / 1e6:.0f} MB; beside it: {", ".join(copy_dlls(exe, msys / "bin", out))}')
|
||||
(out / 'profile').mkdir(exist_ok=True)
|
||||
|
||||
|
||||
@@ -432,56 +688,111 @@ def keep_awake():
|
||||
pass
|
||||
|
||||
|
||||
def load_state(path):
|
||||
try:
|
||||
state = json.loads(path.read_text(encoding='utf-8')) if path.is_file() else {}
|
||||
except (OSError, ValueError):
|
||||
state = {}
|
||||
if state.get('version') == 1: # the first kit: a list of finished steps, all tied to one kit version
|
||||
done = state.get('done', [])
|
||||
state = {'version': STATE_VERSION, 'input': state.get('input'), 'steps': {}}
|
||||
for step in ('exefs', 'romfs'):
|
||||
if step in done:
|
||||
state['steps'][step] = step_key(step, state['input'])
|
||||
return state
|
||||
|
||||
|
||||
def step_key(step, identity, out=None):
|
||||
"""What a step's output depends on. exefs and romfs: the game files only; lift: the unpacked ExeFS and the
|
||||
lifter; compile: the lift, the runtime sources and this script."""
|
||||
if step in ('exefs', 'romfs'):
|
||||
return f'{step}-1|{identity}'
|
||||
lift = f'lift|{exefs_digest(out)}|{lift_digest()}'
|
||||
if step == 'lift':
|
||||
return lift
|
||||
compile_key = f'compile|{lift}|{tree_digest(KIT / "runtime", ("**/*",))}|{sha256_file(Path(__file__))}|{kit_version()}'
|
||||
if step == 'compile':
|
||||
return compile_key
|
||||
# shaders: the built game and the translator that prepared them
|
||||
return f'shaders-1|{compile_key}|{sha256_file(KIT / "tools" / "prepare_shaders.py")}|{sha256_file(KIT / "bin" / "nx2navi_xlate.id")}'
|
||||
|
||||
|
||||
def main():
|
||||
global EVENTS
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument('--nup', help='the game as one .nup file')
|
||||
ap.add_argument('--exefs', help='the game\'s exefs folder (with --romfs, instead of --nup)')
|
||||
ap.add_argument('--romfs', help='the game\'s romfs folder')
|
||||
ap.add_argument('--rebuild', action='store_true',
|
||||
help='build again from the game files the last run unpacked into --out (after a kit update)')
|
||||
ap.add_argument('--out', help='the folder for the finished game')
|
||||
ap.add_argument('--check', action='store_true', help='only check the game\'s version; builds nothing')
|
||||
ap.add_argument('--msys', default=r'C:\msys64\ucrt64', help='MSYS2 UCRT64 folder (default C:\\msys64\\ucrt64)')
|
||||
ap.add_argument('--jobs', type=int, default=os.cpu_count() or 4, help='parallel compiles')
|
||||
ap.add_argument('--events', action='store_true', help='machine-readable progress lines on stdout ("@@" + JSON)')
|
||||
ap.add_argument('--until', choices=STEPS, help=argparse.SUPPRESS) # tests: stop after this step
|
||||
a = ap.parse_args()
|
||||
if bool(a.nup) == bool(a.exefs or a.romfs) or (not a.nup and not (a.exefs and a.romfs)):
|
||||
ap.error('give --nup, or both --exefs and --romfs')
|
||||
src = Source(a)
|
||||
EVENTS = a.events
|
||||
given = bool(a.nup) + bool(a.exefs or a.romfs) + bool(a.rebuild)
|
||||
if given != 1 or (not a.nup and not a.rebuild and not (a.exefs and a.romfs)):
|
||||
ap.error('give --nup, both --exefs and --romfs, or --rebuild')
|
||||
if a.rebuild and not a.out:
|
||||
ap.error('--rebuild needs --out')
|
||||
out = Path(a.out).resolve() if a.out else None
|
||||
cache = out / 'build-cache' if out else None
|
||||
state_path = cache / 'state.json' if cache else None
|
||||
state = load_state(state_path) if state_path else {}
|
||||
|
||||
emit('step', step='check', state='start')
|
||||
src = BuiltSource(out, state) if a.rebuild else Source(a)
|
||||
check(src)
|
||||
say('check: Ocarina of Time 1.1.1 (build e0f3ccfa)')
|
||||
emit('check', version='1.1.1', build=MAIN_BUILD_ID[:8])
|
||||
emit('step', step='check', state='done')
|
||||
if a.check:
|
||||
return
|
||||
if not a.out:
|
||||
if not out:
|
||||
ap.error('--out is needed to build')
|
||||
msys = Path(a.msys)
|
||||
for need in ('bin/clang.exe', 'bin/clang++.exe', 'bin/SDL3.dll', 'lib/libSDL3.dll.a', 'include/vma'):
|
||||
if not (msys / need).exists():
|
||||
fail(f'{msys / need} is missing: install MSYS2 and the packages README-BuildKit.md lists')
|
||||
for mod in ('numpy', 'lz4', 'zbic'):
|
||||
for mod in ('numpy', 'lz4') + (() if a.rebuild else ('zbic',)):
|
||||
try:
|
||||
__import__(mod)
|
||||
except ImportError:
|
||||
fail(f'the Python package {mod} is missing: pip install numpy lz4 zstandard zbic')
|
||||
out = Path(a.out).resolve()
|
||||
cache = out / 'build-cache'
|
||||
cache.mkdir(parents=True, exist_ok=True)
|
||||
state_path = cache / 'state.json'
|
||||
state = json.loads(state_path.read_text(encoding='utf-8')) if state_path.is_file() else {}
|
||||
if state.get('input') != src.identity or state.get('version') != STATE_VERSION or state.get('kit') != kit_version():
|
||||
state = {'input': src.identity, 'version': STATE_VERSION, 'kit': kit_version(), 'done': []}
|
||||
if state.get('version') != STATE_VERSION or state.get('input') != src.identity:
|
||||
state = {'version': STATE_VERSION, 'input': src.identity, 'steps': {}}
|
||||
state['kit'] = kit_version()
|
||||
keep_awake()
|
||||
t0 = time.time()
|
||||
for name, run in (('exefs', lambda: step_exefs(src, out)), ('romfs', lambda: step_romfs(src, out, min(a.jobs, 8))),
|
||||
('lift', lambda: step_lift(out, cache)), ('compile', lambda: step_compile(out, cache, msys, a.jobs)),
|
||||
('shaders', lambda: step_shaders(out))):
|
||||
if name in state['done']:
|
||||
runs = {'exefs': lambda: step_exefs(src, out), 'romfs': lambda: step_romfs(src, out, min(a.jobs, 8)),
|
||||
'lift': lambda: step_lift(out, cache), 'compile': lambda: step_compile(out, cache, msys, a.jobs),
|
||||
'shaders': lambda: step_shaders(out)}
|
||||
for name in STEPS:
|
||||
key = step_key(name, src.identity, out)
|
||||
if state['steps'].get(name) == key:
|
||||
say(f'{name}: done before')
|
||||
continue
|
||||
say(f'{name}:')
|
||||
t = time.time()
|
||||
run()
|
||||
state['done'].append(name)
|
||||
state_path.write_text(json.dumps(state, indent=1), encoding='utf-8')
|
||||
say(f'{name}: {time.time() - t:.0f} s')
|
||||
say(f'\nfinished in {(time.time() - t0) / 60:.1f} min: {out / "ocarina.exe"}')
|
||||
emit('step', step=name, state='skip')
|
||||
else:
|
||||
# a step that runs makes every step after it run too: their keys hash its output, not its key
|
||||
for later in STEPS[STEPS.index(name):]:
|
||||
state['steps'].pop(later, None)
|
||||
say(f'{name}:')
|
||||
emit('step', step=name, state='start')
|
||||
t = time.time()
|
||||
runs[name]()
|
||||
state['steps'][name] = step_key(name, src.identity, out)
|
||||
state_path.write_text(json.dumps(state, indent=1), encoding='utf-8')
|
||||
say(f'{name}: {time.time() - t:.0f} s')
|
||||
emit('step', step=name, state='done', seconds=round(time.time() - t, 1))
|
||||
if name == a.until:
|
||||
break
|
||||
state_path.write_text(json.dumps(state, indent=1), encoding='utf-8')
|
||||
say(f'\nfinished in {(time.time() - t0) / 60:.1f} min: {out / EXE_NAME}')
|
||||
emit('end', exe=str(out / EXE_NAME), seconds=round(time.time() - t0, 1), kit=kit_version())
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
||||
301
tools/make_launcher.py
Normal file
301
tools/make_launcher.py
Normal file
@@ -0,0 +1,301 @@
|
||||
"""Build a release of the NX2navi launcher (docs/launcher.md, "Releases").
|
||||
|
||||
python tools/make_launcher.py --xlate build/xlate [--ffx amd_fidelityfx_vk.dll] [--out build/launcher-release]
|
||||
[--feed owner/name]
|
||||
[--notes notes.md] [--publish]
|
||||
|
||||
Needs Python 3.14 (tarfile and zstd for the MSYS2 packages), uv on PATH, network access on the first run, and
|
||||
either Windows (msiexec, and the C compiler of MSYS2 it downloads itself) or Linux with clang, lld and llvm-windres
|
||||
(it cross-compiles against the same MSYS2 packages) plus msitools. Downloads are cached in toolchain/launcher-dl/ and
|
||||
checked against the sha256 in PINS.
|
||||
|
||||
The release folder (--out) holds:
|
||||
|
||||
NX2navi/ the launcher folder exactly as NX2navi-Setup.exe installs it:
|
||||
NX2navi.exe launcher/native/nx2navi.c with the Triforce icon (launcher/ui/art.py)
|
||||
python/ python.org's embeddable Python 3.14.9, tkinter and Tcl/Tk 9.0 from its tcltk.msi,
|
||||
the packages of tools/requirements-launcher.txt, runtime-id.txt
|
||||
app/current.txt, app/<version>/ launcher/ (without native/), runtime/, recomp/, the tools make_game.py uses,
|
||||
bin/nx2navi_xlate.exe and .id (and amd_fidelityfx_vk.dll with --ffx), toolchain.lock.json (the MSYS2 packages of today,
|
||||
pinned), feed.json, VERSION, the licence files
|
||||
NX2navi-Setup-<version>.exe launcher/native/setup.c with the folder above appended as a zip: the one file
|
||||
players download
|
||||
NX2navi-app-<version>.zip app/<version> alone: what the launcher downloads to update itself
|
||||
NX2navi-runtime-<id>.zip python/ alone: downloaded only when a release changes the Python
|
||||
NX2navi-update.json the manifest the launcher reads (launcher/feed.py)
|
||||
|
||||
--publish uploads the four files as GitHub release v<version> of the --feed repository with the gh CLI (a draft
|
||||
first is wiser: add --draft). check_content runs over everything before it is packed: the release holds no game data.
|
||||
"""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT))
|
||||
sys.path.insert(0, str(ROOT / 'tools'))
|
||||
import check_content # noqa: E402
|
||||
from launcher import toolchain # noqa: E402
|
||||
from launcher.paths import Layout # noqa: E402
|
||||
|
||||
MAIN_ROOT = (ROOT / 'build').resolve().parent # build/ and toolchain/ are junctions in a worktree
|
||||
DL = MAIN_ROOT / 'toolchain' / 'launcher-dl'
|
||||
WORK = MAIN_ROOT / 'toolchain' / 'launcher-work'
|
||||
PY_VERSION = '3.14.9'
|
||||
PINS = {
|
||||
'python-embed': (f'https://www.python.org/ftp/python/{PY_VERSION}/python-{PY_VERSION}-embed-amd64.zip',
|
||||
'3c01d7cfc666f746a6f064f60fb2156ab0ec464d3d738ab796371a46a9bb24a2'),
|
||||
# the installer's own Tcl/Tk and tkinter component, published beside it
|
||||
'tcltk': (f'https://www.python.org/ftp/python/{PY_VERSION}/amd64/tcltk.msi',
|
||||
'ac87609db3f715404dd41a989c2f1706ea1512f3d86217f938d11c63af8eb505'),
|
||||
}
|
||||
TOOLS = ('make_game.py', 'nup.py', 'romfs_index.py', 'check_content.py', 'hostmem.py', 'prepare_shaders.py',
|
||||
'shader_census.py', 'shader_list.py', 'zstd_compat.py')
|
||||
TOP = ('LICENSE', 'LICENSE-ADDITIONS', 'THIRD_PARTY_NOTICES.md', 'nx2navi.ini.example')
|
||||
SKIP_DIRS = {'__pycache__', '.pytest_cache', 'native'}
|
||||
SKIP_SUFFIXES = {'.o', '.obj', '.pyc', '.exe', '.dll', '.a', '.lib'}
|
||||
TCLTK_FILES = ('_tkinter.pyd', 'tcl90.dll', 'tcl9tk90.dll', 'zlib1.dll', 'libtommath.dll')
|
||||
DEFAULT_FEED = 'tangerineturtle1/NX2navi'
|
||||
|
||||
|
||||
def say(msg):
|
||||
print(msg, flush=True)
|
||||
|
||||
|
||||
def sha256_file(path):
|
||||
h = hashlib.sha256()
|
||||
with open(path, 'rb') as f:
|
||||
while b := f.read(1 << 20):
|
||||
h.update(b)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def fetch(name):
|
||||
url, want = PINS[name]
|
||||
dest = DL / url.rsplit('/', 1)[1]
|
||||
if dest.is_file() and sha256_file(dest) == want:
|
||||
return dest
|
||||
from launcher import net
|
||||
say(f'download {url}')
|
||||
net.download(url, dest, sha256=want)
|
||||
return dest
|
||||
|
||||
|
||||
def run(cmd, **kw):
|
||||
say(' ' + ' '.join(str(c) for c in cmd))
|
||||
subprocess.run([str(c) for c in cmd], check=True, **kw)
|
||||
|
||||
|
||||
def copy_tree(src, dst, skip_dirs=SKIP_DIRS):
|
||||
for p in sorted(src.rglob('*')):
|
||||
rel = p.relative_to(src)
|
||||
if any(part in skip_dirs for part in rel.parts) or p.is_dir() or p.suffix.lower() in SKIP_SUFFIXES:
|
||||
continue
|
||||
(dst / rel).parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy2(p, dst / rel)
|
||||
|
||||
|
||||
def version():
|
||||
v = (ROOT / 'VERSION').read_text(encoding='utf-8').strip()
|
||||
r = subprocess.run(['git', '-C', str(ROOT), 'rev-parse', '--short', 'HEAD'], capture_output=True, text=True)
|
||||
return v, f'{v}+{r.stdout.strip()}' if r.returncode == 0 and r.stdout.strip() else v
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- python/
|
||||
|
||||
def tcltk_files():
|
||||
dest = WORK / 'tcltk'
|
||||
if (dest / 'DLLs' / '_tkinter.pyd').is_file():
|
||||
return dest
|
||||
msi = fetch('tcltk')
|
||||
shutil.rmtree(dest, ignore_errors=True)
|
||||
dest.mkdir(parents=True)
|
||||
if os.name == 'nt': # an administrative image: copies the files, installs nothing
|
||||
run(['msiexec', '/a', msi, '/qn', f'TARGETDIR={dest}'])
|
||||
else:
|
||||
run(['msiextract', '-C', dest, msi], stdout=subprocess.DEVNULL)
|
||||
if not (dest / 'DLLs' / '_tkinter.pyd').is_file():
|
||||
sys.exit(f'{dest}: no _tkinter.pyd in {msi.name}')
|
||||
return dest
|
||||
|
||||
|
||||
def runtime_id():
|
||||
req = (ROOT / 'tools' / 'requirements-launcher.txt').read_bytes()
|
||||
return f'py{PY_VERSION}-{hashlib.sha256(req + PINS["tcltk"][1].encode()).hexdigest()[:8]}'
|
||||
|
||||
|
||||
def build_python(py):
|
||||
with zipfile.ZipFile(fetch('python-embed')) as z:
|
||||
z.extractall(py)
|
||||
# The embeddable Python's sys.path is exactly this list; ../pydeps holds zbic (launcher/toolchain.py).
|
||||
(py / 'python314._pth').write_text('python314.zip\n.\nLib/site-packages\n../pydeps\n', encoding='utf-8')
|
||||
env = dict(os.environ, UV_CACHE_DIR=str(MAIN_ROOT / 'toolchain' / 'uv-cache'))
|
||||
run(['uv', 'pip', 'install', '--target', py / 'Lib' / 'site-packages', '--python', sys.executable,
|
||||
'--python-platform', 'windows', '--python-version', '3.14', '--only-binary', ':all:', '-q',
|
||||
'-r', ROOT / 'tools' / 'requirements-launcher.txt'], env=env)
|
||||
shutil.rmtree(py / 'Lib' / 'site-packages' / 'bin', ignore_errors=True)
|
||||
tk = tcltk_files()
|
||||
for dll in TCLTK_FILES:
|
||||
if (tk / 'DLLs' / dll).is_file():
|
||||
shutil.copy2(tk / 'DLLs' / dll, py / dll)
|
||||
copy_tree(tk / 'Lib' / 'tkinter', py / 'tkinter', skip_dirs={'__pycache__', 'test'})
|
||||
copy_tree(tk / 'tcl', py / 'tcl', skip_dirs={'__pycache__'})
|
||||
(py / 'runtime-id.txt').write_text(runtime_id() + '\n', encoding='utf-8')
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- app/<version>/
|
||||
|
||||
def build_app(app, ver, xlate, feed_repo, lock, ffx=None):
|
||||
copy_tree(ROOT / 'launcher', app / 'launcher')
|
||||
for d in ('runtime', 'recomp'):
|
||||
copy_tree(ROOT / d, app / d)
|
||||
(app / 'tools').mkdir(parents=True)
|
||||
for t in TOOLS:
|
||||
shutil.copy2(ROOT / 'tools' / t, app / 'tools' / t)
|
||||
for t in TOP:
|
||||
if (ROOT / t).is_file():
|
||||
shutil.copy2(ROOT / t, app / t)
|
||||
(app / 'VERSION').write_text(ver + '\n', encoding='utf-8')
|
||||
(app / 'bin').mkdir()
|
||||
for n in ('nx2navi_xlate.exe', 'nx2navi_xlate.id'):
|
||||
if not (xlate / n).is_file():
|
||||
sys.exit(f'{xlate / n} is missing: build the shader translator first (tools/build_xlate.py)')
|
||||
shutil.copy2(xlate / n, app / 'bin' / n)
|
||||
if ffx: # FSR 3.1 loads it from beside the game (runtime/vk_ffx.cpp); make_game.py copies it there
|
||||
shutil.copy2(ffx, app / 'bin' / 'amd_fidelityfx_vk.dll')
|
||||
(app / 'toolchain.lock.json').write_text(json.dumps(lock, indent=1), encoding='utf-8')
|
||||
(app / 'feed.json').write_text(json.dumps({'type': 'github', 'repo': feed_repo}, indent=1), encoding='utf-8')
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- the native programs
|
||||
|
||||
def native_compiler(lock):
|
||||
"""(cc argv prefix, windres argv prefix) for x86_64 Windows: MSYS2's own clang on Windows, else the host's clang
|
||||
against the same MSYS2 packages as a sysroot."""
|
||||
lay = Layout(root=MAIN_ROOT / 'toolchain' / 'launcher-msys')
|
||||
if not toolchain.is_current(lay, lock):
|
||||
say('MSYS2 packages for the native programs')
|
||||
toolchain.install(lay, lock, log=say)
|
||||
sysroot = lay.msys
|
||||
if os.name == 'nt':
|
||||
return [str(sysroot / 'bin' / 'clang.exe')], [str(sysroot / 'bin' / 'windres.exe')]
|
||||
for tool in ('clang', 'ld.lld'):
|
||||
if not shutil.which(tool):
|
||||
sys.exit(f'{tool} is needed to cross-compile the launcher programs')
|
||||
windres = shutil.which('llvm-windres') or shutil.which('llvm-windres-18') or '/usr/lib/llvm-18/bin/llvm-windres'
|
||||
if not Path(windres).exists():
|
||||
sys.exit('llvm-windres is needed to compile the launcher programs\' resources')
|
||||
return (['clang', '--target=x86_64-w64-mingw32', f'--sysroot={sysroot}', '-fuse-ld=lld'],
|
||||
[windres, '--target=x86_64-w64-mingw32'])
|
||||
|
||||
|
||||
def build_native(work, cc, windres):
|
||||
from launcher.ui import art
|
||||
work.mkdir(parents=True, exist_ok=True)
|
||||
(work / 'nx2navi.ico').write_bytes(art.triforce_ico())
|
||||
shutil.copy2(ROOT / 'launcher' / 'native' / 'app.manifest', work / 'app.manifest')
|
||||
res = work / 'res.o'
|
||||
run([*windres, '-I', work, ROOT / 'launcher' / 'native' / 'res.rc', '-O', 'coff', '-o', res])
|
||||
flags = ['-municode', '-mwindows', '-O2', '-Wall', '-static', '-s']
|
||||
stub = work / 'NX2navi.exe'
|
||||
setup = work / 'setup-stub.exe'
|
||||
run([*cc, *flags, ROOT / 'launcher' / 'native' / 'nx2navi.c', res, '-o', stub, '-lshell32', '-lole32', '-luuid'])
|
||||
run([*cc, *flags, ROOT / 'launcher' / 'native' / 'setup.c', res, '-o', setup, '-l:libz.a', '-lshell32',
|
||||
'-lole32', '-luuid', '-ladvapi32'])
|
||||
return stub, setup
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- packing
|
||||
|
||||
def zip_dir(src, dest, prefix=''):
|
||||
with zipfile.ZipFile(dest, 'w', zipfile.ZIP_DEFLATED, compresslevel=9) as z:
|
||||
for p in sorted(src.rglob('*')):
|
||||
if p.is_file():
|
||||
z.write(p, (Path(prefix) / p.relative_to(src)).as_posix())
|
||||
return dest
|
||||
|
||||
|
||||
def make_setup(stub, payload, dest, ver):
|
||||
"""setup-stub.exe + the payload zip + the 48-byte trailer launcher/native/setup.c reads."""
|
||||
data = stub.read_bytes()
|
||||
off = len(data)
|
||||
z = payload.read_bytes()
|
||||
trailer = b'NX2NSET1' + off.to_bytes(8, 'little') + len(z).to_bytes(8, 'little') + \
|
||||
ver.encode('utf-8')[:24].ljust(24, b'\0')
|
||||
dest.write_bytes(data + z + trailer)
|
||||
return dest
|
||||
|
||||
|
||||
def check_folder(folder):
|
||||
bad = [f'{p.relative_to(folder)}: {m}' for p in sorted(folder.rglob('*')) if p.is_file()
|
||||
for m in check_content.problems_for(p, allow_large=True)]
|
||||
if bad:
|
||||
sys.exit('check_content refuses these release files:\n ' + '\n '.join(bad[:20]))
|
||||
|
||||
|
||||
def entry(path):
|
||||
return {'name': path.name, 'size': path.stat().st_size, 'sha256': sha256_file(path)}
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument('--xlate', type=Path, required=True, help='folder with nx2navi_xlate.exe and nx2navi_xlate.id')
|
||||
ap.add_argument('--ffx', type=Path, help="AMD's signed amd_fidelityfx_vk.dll (FidelityFX SDK 1.1.4, MIT) for FSR 3.1")
|
||||
ap.add_argument('--out', type=Path, default=MAIN_ROOT / 'build' / 'launcher-release')
|
||||
ap.add_argument('--feed', default=DEFAULT_FEED, help='the GitHub repository the launcher looks for releases in')
|
||||
ap.add_argument('--notes', type=Path, help='release notes (markdown) for the manifest and the GitHub release')
|
||||
ap.add_argument('--lock', type=Path, help='a toolchain.lock.json to reuse instead of resolving today\'s packages')
|
||||
ap.add_argument('--version', help='the release version instead of VERSION\'s (test releases)')
|
||||
ap.add_argument('--publish', action='store_true', help='upload as GitHub release v<version> with gh')
|
||||
ap.add_argument('--draft', action='store_true', help='with --publish: as a draft')
|
||||
a = ap.parse_args()
|
||||
if sys.version_info < (3, 14):
|
||||
sys.exit('Python 3.14 or later is needed (compression.zstd for the MSYS2 packages)')
|
||||
ver, build = version()
|
||||
if a.version:
|
||||
ver, build = a.version, a.version + build[len(ver):]
|
||||
out = a.out
|
||||
shutil.rmtree(out, ignore_errors=True)
|
||||
folder = out / 'NX2navi'
|
||||
t0 = time.time()
|
||||
lock = json.loads(a.lock.read_text(encoding='utf-8')) if a.lock else toolchain.make_lock()
|
||||
say(f'release {build}: toolchain lock with {len(lock["packages"])} packages')
|
||||
build_app(folder / 'app' / ver, ver, a.xlate, a.feed, lock, a.ffx)
|
||||
(folder / 'app' / 'current.txt').write_text(ver + '\n', encoding='utf-8')
|
||||
build_python(folder / 'python')
|
||||
cc, windres = native_compiler(lock)
|
||||
stub, setup_stub = build_native(out / 'native', cc, windres)
|
||||
shutil.copy2(stub, folder / 'NX2navi.exe')
|
||||
check_folder(folder)
|
||||
app_zip = zip_dir(folder / 'app' / ver, out / f'NX2navi-app-{ver}.zip')
|
||||
rt_zip = zip_dir(folder / 'python', out / f'NX2navi-runtime-{runtime_id()}.zip')
|
||||
payload = zip_dir(folder, out / 'native' / 'payload.zip')
|
||||
setup = make_setup(setup_stub, payload, out / f'NX2navi-Setup-{ver}.exe', ver)
|
||||
notes = a.notes.read_text(encoding='utf-8') if a.notes else ''
|
||||
manifest = {'format': 1, 'version': ver, 'build': build, 'notes': notes,
|
||||
'app': entry(app_zip), 'runtime': {'id': runtime_id(), **entry(rt_zip)},
|
||||
'toolchain': toolchain.lock_digest(lock), 'setup': entry(setup),
|
||||
'made': time.strftime('%Y-%m-%d %H:%M')}
|
||||
man = out / 'NX2navi-update.json'
|
||||
man.write_text(json.dumps(manifest, indent=1), encoding='utf-8')
|
||||
for p in (setup, app_zip, rt_zip, man):
|
||||
say(f'{p.name}: {p.stat().st_size / 1e6:.1f} MB')
|
||||
say(f'release folder {out} in {time.time() - t0:.0f} s')
|
||||
if a.publish:
|
||||
cmd = ['gh', 'release', 'create', f'v{ver}', setup, app_zip, rt_zip, man, '--repo', a.feed,
|
||||
'--title', f'NX2navi {ver}']
|
||||
cmd += ['--notes-file', a.notes] if a.notes else ['--notes', f'NX2navi {ver}']
|
||||
if a.draft:
|
||||
cmd.append('--draft')
|
||||
run(cmd)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
6
tools/requirements-launcher.txt
Normal file
6
tools/requirements-launcher.txt
Normal file
@@ -0,0 +1,6 @@
|
||||
# Packages of the launcher release's bundled Python 3.14.9 (tools/make_launcher.py): what the launcher and
|
||||
# tools/make_game.py import. numpy: recomp/gen.py and the window's drawing (launcher/ui/art.py). lz4: NSO segments.
|
||||
# zstandard: tools/nup.py. zbic is not bundled: the launcher downloads it from PyPI on the player's PC.
|
||||
numpy==2.5.4
|
||||
lz4==4.4.5
|
||||
zstandard==0.25.0
|
||||
Reference in New Issue
Block a user