mirror of
https://github.com/tangerineturtle1/NX2navi.git
synced 2026-11-16 15:59:59 -04:00
The review found that dynres.c dynres_frame compared a changed target with a static that started at 0 and outlived a restart: with fps_limit = display the first refresh-rate wander under 1 % after the start was logged, against the comment's rule, and after dyn_res off and on a move back to the earlier run's last logged target said nothing, so the start line and the live lines disagreed. A test of the menu's path found a third case: a limit or dyn_res_target changed in the menu bumps cfg_version, dynres_read takes the new target first, and dynres_frame, which logged only when its own reading differed from D.target_ns, never named it. The last logged target now lives in D (logged_ns), the start line sets it, and dynres_frame logs any move of more than 1 % from it, naming the set target as "as set". No lock and no wait; the target and the steps are what they were. The review's stale "(dyn_res_target 16.7 in 1..100)" in cfg.c misfit now reads 0..100, as the row's set has been since the commit that changed the default target. tests/dynres_check.py gains these cases (a menu limit, a menu target, a wander after the start, the earlier run's target after off and on); they fail on the old dynres.c and pass. The -Wall compile of dynres.c and cfg.c shows no warning, and cfg_rows_check reports 0 problems. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
990 lines
82 KiB
Markdown
990 lines
82 KiB
Markdown
# Building from source
|
|
|
|
Windows 10 or later, an x86-64 CPU with SSE4.2, and a GPU with Vulkan 1.3 and `VK_EXT_external_memory_host`. The
|
|
default build targets x86-64-v3 (AVX2/FMA); a CPU without them builds with `--cpu v2` instead and runs slower (see
|
|
"Build"). Allow about 60 GB for the work folder. Extraction wants about 55 GB free while it runs and keeps about 35 GB
|
|
afterwards.
|
|
|
|
Players who only want to play use the [launcher](launcher.md) instead: it installs its own build tools, builds the
|
|
game on their machine and keeps it up to date.
|
|
|
|
## Toolchain
|
|
|
|
Versions are the ones the build was verified with; newer patch releases are fine.
|
|
|
|
| Tool | Version | Install | Used for |
|
|
|---|---|---|---|
|
|
| MSYS2 UCRT64 | clang/lld 23.1.3 | msys2.org installer to `C:\msys64`, packages below | compiling the generated C and the runtime |
|
|
| Visual Studio Build Tools | 2026, C++ workload, Windows SDK 10.0.26100 | winget `Microsoft.VisualStudio.BuildTools` with `--add Microsoft.VisualStudio.Workload.VCTools --includeRecommended` | the linker .NET NativeAOT calls |
|
|
| .NET SDK | 10.0 | winget `Microsoft.DotNet.SDK.10` | the shader translator (`nx2navi_xlate.exe`) |
|
|
| Vulkan SDK | 1.4.363 | winget `KhronosGroup.VulkanSDK` | validation layers, `vulkaninfo` |
|
|
| uv | 0.13 | standalone installer from astral.sh | the two Python environments |
|
|
| hactool | 1.4.0 @ `1d64a83` | `.venv\Scripts\python tools\build_hactool.py`, see below | NCA and update extraction |
|
|
|
|
MSYS2 packages, after `pacman -Syuu` (run it twice; the first pass updates the core and exits):
|
|
|
|
```sh
|
|
pacman -S --needed --noconfirm mingw-w64-ucrt-x86_64-{clang,lld,ninja,gcc,make,pkgconf,sdl3,opus,libvpx,dav1d,zstd,vulkan-devel,\
|
|
vulkan-memory-allocator,vulkan-validation-layers,shaderc,spirv-tools} base-devel git unzip
|
|
```
|
|
|
|
`ffmpeg` from MSYS2 is a GPLv3 build and is never linked; the movies go through libvpx.
|
|
|
|
PATH:
|
|
|
|
- Add `C:\Program Files (x86)\Microsoft Visual Studio\Installer` to the user PATH: NativeAOT needs `vswhere.exe`, or
|
|
the publish fails at the link step (MSB3073).
|
|
- Another `cmake` or `ninja` earlier on PATH (a 32-bit winlibs MinGW, for example) is picked up by accident. The build
|
|
scripts call Ninja from `C:\msys64\ucrt64\bin` themselves; from a shell, put `C:\msys64\ucrt64\bin` first.
|
|
|
|
### hactool
|
|
|
|
`tools/build_hactool.py` fetches hactool at `1d64a83` into `toolchain\src\hactool`, applies the three patches in
|
|
`tools/hactool/` and builds a static `toolchain\hactool\hactool.exe` with MSYS2 UCRT64 gcc (a few seconds):
|
|
|
|
- `portable.patch`: the changes a portable Windows build needs, the `MNG_TITLE_KEYS` variable and
|
|
the two BKTR fixes in `nca.c` for the subsection table padding and the subsection layout of newer cumulative updates;
|
|
- `unicode.patch`: a manifest with `<activeCodePage>UTF-8</activeCodePage>`, so paths under a non-ASCII user profile
|
|
reach it intact;
|
|
- `mbedtls-asm-clobbers.patch`: the `memory` and `cc` clobbers the bundled mbedtls 2.6.1's x86-64 multiply asm lacks.
|
|
Without them a clang build (the Linux release) fails every RSA signature check and the update merge stops.
|
|
|
|
hactool reads `-k <prod.keys>`; MSYS2 shells set `HOME`, which hides its default key location, so pass `-k` there.
|
|
|
|
## Build
|
|
|
|
From the repository root, in PowerShell. Every step that depends on the game version takes it as `<ver>`: `v143` for
|
|
1.4.3, `v121` for 1.2.1 or `v110` for 1.1.0. The tools default to `v121` when `--version` is left out. Saves load on any
|
|
of them.
|
|
|
|
1. Python environments (once). `.venv` runs the lifter and the build; `.venv-py314` runs the extraction scripts, which
|
|
need the stdlib `compression.zstd` of Python 3.14:
|
|
|
|
```powershell
|
|
uv venv --python 3.12 .venv
|
|
uv pip install --python .venv/Scripts/python.exe -r tools/requirements.txt
|
|
uv venv --python 3.14 .venv-py314
|
|
uv pip install --python .venv-py314/Scripts/python.exe -r tools/requirements-py314.txt
|
|
```
|
|
|
|
2. Copy `config.example.toml` to `config.local.toml` and set the paths to your base game, the update and your console's
|
|
keys file. `config.local.toml` is ignored by git. The tools read the keys file themselves.
|
|
|
|
3. Extract and verify. This writes `build\extracted\v0` (1.0.0) and `build\extracted\<ver>` for each update in
|
|
`updates`, identified by its hash; a hash mismatch stops the run. With another `[work] dir` in `config.local.toml`,
|
|
`extracted` goes there instead, and `build_game.py` reads the ExeFS and RomFS from there:
|
|
|
|
```powershell
|
|
.venv-py314\Scripts\python tools\extract.py
|
|
```
|
|
|
|
4. Shader translator (no game files; again after a change to its sources, and `tools\make_release.py` stops on an older
|
|
one). This publishes `build\xlate\nx2navi_xlate.exe`:
|
|
|
|
```powershell
|
|
.venv\Scripts\python tools\build_xlate.py
|
|
```
|
|
|
|
5. Lift the modules to C, write the RomFS overlay (the frame-rate fix), then compile and link
|
|
`build\<ver>\nx2navi.exe`:
|
|
|
|
```powershell
|
|
.venv\Scripts\python recomp\gen.py --exefs build\extracted\<ver>\exefs --out generated\<ver> --patches recomp\patches-<ver>.txt
|
|
.venv-py314\Scripts\python tools\make_romfs_patches.py --version <ver>
|
|
.venv\Scripts\python tools\build_game.py --version <ver>
|
|
```
|
|
|
|
The menu's font comes into `runtime/ui/menu.cpp` with `#embed`; for a compiler without it (before clang 19, which
|
|
a Linux build with the distribution's compiler can meet) the same step writes the font's bytes as
|
|
`runtime/ui/roboto_ttf.h` (git ignores it), which `menu.cpp` then includes.
|
|
The runtime's own shaders (upscaling, HDR, texture decode and the like) are in the repository as source only:
|
|
GLSL and SPIR-V assembly in `runtime\shaders\`, and two small ones assembled by `tools\make_dlaa_mask.py` and
|
|
`tools\make_mvobj_fs.py`. `build_game.py` compiles them into `runtime\<name>_spv.h` (git ignores these) before it
|
|
compiles the runtime, with `glslc`, `spirv-as` and `spirv-val` from the `shaderc` and `spirv-tools` packages above,
|
|
and again whenever a source changed. `python tools\build_shaders.py` does that step alone; `--check` only reports
|
|
the headers that are missing or stale.
|
|
|
|
`--cpu v2` targets x86-64-v2 (SSE4.2, no AVX2/FMA) for older CPUs; the default `v3` needs AVX2. The v2 build links
|
|
`runtime/fma.c`, a correctly rounded software fma the FP lowering needs, and runs slower. Inline assembly,
|
|
target attributes and intrinsics are not bound by `-march`, so a v2 build runs `tools/check_isa.py` over the runtime
|
|
objects before it links: any AVX, BMI or other instruction above x86-64-v2 outside a function listed there as
|
|
dispatched by a runtime CPU check stops the build. `python tests\check_isa_test.py` builds the files with such
|
|
functions both ways and checks that they pass.
|
|
|
|
`gen.py` sizes its emit pool from the CPU count and the free memory; `--jobs N` sets it, and so does the environment variable `NX2NAVI_LIFT_JOBS=N`, which the installer's lift step honours too (it has no `--jobs`). When a worker runs out of memory, the pool restarts with half the workers (`emit: out of memory with N workers, retrying with M`), down to one; only an out-of-memory error with one worker fails the step.
|
|
|
|
Lifting takes about 15 s, a clean compile about 200 s on 24 threads; later builds recompile only what changed.
|
|
A compiler cache in front of clang shortens it further: with `SCCACHE` set to an sccache executable, `build_game.py`
|
|
writes it into the ninja file ahead of `cc`, and every lift rewrites the generated chunks with the same contents, so
|
|
a second build of the same version hits all of their objects:
|
|
|
|
```powershell
|
|
$env:SCCACHE = "path\to\sccache.exe"
|
|
.venv\Scripts\python tools\build_game.py --version <ver>
|
|
```
|
|
`--patches` builds in the frame-rate, resolution and other patch sets, switchable at run time. The link copies
|
|
`SDL3.dll`, `libiconv-2.dll` and `libopus-0.dll` next to the exe.
|
|
For Xbox button prompts, run `make_romfs_patches.py` twice more, with `--prompts xbox` and `--prompts xbox-label`:
|
|
they write `build\<ver>\romfs-patches-xbox` and `romfs-patches-xbox-label` ([controls.md](controls.md), "Xbox
|
|
button prompts"), which `build_game.py` copies next to `nx2navi.exe` like `romfs-patches`. `--prompts pc` writes
|
|
`romfs-patches-pc`, the keyboard and mouse prompts ([controls.md](controls.md), "PC button prompts").
|
|
|
|
6. Prepare the shaders (once per build folder and translator). It needs `refs\MacNativeGames` (`tools\fetch_refs.sh`)
|
|
for the shared shader list:
|
|
|
|
```powershell
|
|
.venv-py314\Scripts\python tools\prepare_shaders.py build\<ver> --version <ver>
|
|
```
|
|
|
|
Without this step the first launch compiles shaders and pipelines during play and stutters until the caches fill.
|
|
The AMD driver keeps its pipeline cache per exe path and holds a limited amount: preparing another build folder can
|
|
evict this one's. `nx2navi.exe --warm-shaders` rebuilds it from the shader cache. A driver update empties the
|
|
driver's cache too, and a change to what a stage library is built with gives every library another entry in it.
|
|
So an update that carries the prepared shaders over runs `--warm-shaders` (`tools/make_game.py`
|
|
`warm_driver_cache`: seconds while the libraries are there, minutes when they are not), and the game itself checks
|
|
at every start: it builds a sample of the prepared fragment libraries and, when their median time says the cache
|
|
is cold, builds the whole prepared list into the driver's cache again on idle-priority threads while it runs
|
|
(`runtime/vk_pipeline.c` `lw_start`, `NX2NAVI_LIB_WARM`; `shader-warm.txt` next to the caches says where that
|
|
stands). With the cache cold every pipeline a draw first needs compiles in play: on an RX 7900 XT a library took
|
|
93 ms (vertex) and 242 ms (fragment) on average against 0.3 and 0.6 ms from the cache, and a first walk through a
|
|
village skipped 11,180 draws and made 94 draws wait 5.5 s in all, against 1,605 skipped (1,473 of them in the
|
|
scene's first frames) and 60 ms of waits with the cache warm.
|
|
A frame that still lacks draws when it is presented is not shown: the previous picture stays on screen
|
|
(`runtime/vk_backend.c` `frame_hold`, `NX2NAVI_FRAME_HOLD_MS`, an allowance of 120 ms that fills again at an eighth
|
|
of the clock's rate). On a first walk through a village with nothing recorded, 21 of the 23 such frames were held
|
|
back; in a player's session a hit by a boulder had shown one black frame, its lighting pass skipped after 50 ms.
|
|
When the driver's shader compiler crashes on a variant (NVIDIA 615.78.08 on an RTX 5080 does, issue #30),
|
|
`prepare_shaders.py` runs the driver step again with `--warm-shaders`, up to three times, without the variants the
|
|
driver was building at the crash. `shader-prepare-skip.txt` next to the exe lists them with the GPU and driver
|
|
version; they build at their first draw instead, another driver version builds them again, and so does deleting the
|
|
file.
|
|
|
|
What this step leaves for play, and the shared pipeline list. The step translates every shader's default variant
|
|
and the shader list's variants and builds each one's stage library or compute pipeline. A whole graphics pipeline
|
|
is built ahead at the start of a game folder that has drawn it before (`pipeline-keys.bin`) or whose release
|
|
listed it (below); a pipeline met for the first time is linked from its stage libraries in 2 to 8 ms, of which its
|
|
first draw waits 3 (`pipe_link_draw_ms`, from the pipeline's first miss; `runtime/vk_exec_draw.c` `link_budget`),
|
|
except a mesh
|
|
pipeline, which has no libraries and compiles whole (0.2 to 1.1 s, its draws left out of the picture meanwhile),
|
|
and many of them on a first visit add up to hitches. `tools/pipeline_list.py merge` folds the `pipeline-keys.bin`
|
|
and `shader-answers.bin` of game folders that were played, and lists made before, into a pipeline list, with the
|
|
compute variants those folders dispatched (no pipeline record names a compute shader, and a variant the shader list
|
|
lacks is translated on the frame that first dispatches it); `prepare_shaders.py` reads it from
|
|
`tools\data\pipeline_list.bin` (`data\pipeline_list.bin` in a release folder, or `--pipeline-list`), adds the
|
|
shader variants it names to the seed and writes its records next to the game as `seed-pipeline-keys.bin`, and
|
|
`--prepare-shaders` builds every one of them into the pipeline cache and the driver's (`runtime/vk_pipeline.c`
|
|
`seed_pipelines`; `NX2NAVI_PREPARE_PIPELINES=mesh` builds only the mesh pipelines). Every start then counts the
|
|
list as known pipelines beside the folder's own and builds them before play, from those caches (`keys_open_seed`;
|
|
`pipe_seed = 0` in `nx2navi.ini` leaves the list out). `--warm-shaders` builds them again, and
|
|
`nx2navi.exe --prepare-pipelines` does that step alone. The list is
|
|
derived from game data: it is not in the repository (`tools/check_content.py` refuses it), and a kit, a launcher
|
|
release or an installer release takes one with `--pipeline-list` (`tools/make_kit.py`, `make_launcher.py`,
|
|
`make_release.py`). A list belongs to the record layout of the runtime that wrote it; another runtime says so and
|
|
builds nothing from it. `python tests\pipeline_list_test.py` checks the tool on made-up folders.
|
|
|
|
What a game folder's shader cache holds can be checked without the game:
|
|
`python tools\validate_shader_cache.py <game folder>\shader-cache` reads every translated shader from the pack and
|
|
the loose files, applies the checks the game makes when it loads one (those that fail are translated again at
|
|
their next use) and runs `spirv-val` on the rest; `--every N` samples, `--out FILE` lists the failed ones. It writes
|
|
nothing to the cache. `python tests\validate_shader_cache_test.py` checks the tool on a made-up cache.
|
|
|
|
7. Run. The game keeps running when its window loses the focus (`assume_focus = 1` in `nx2navi.ini`, the default; the
|
|
Game tab switches it). With 0 it pauses, as it does on the Switch when Home is pressed.
|
|
|
|
```powershell
|
|
build\<ver>\nx2navi.exe
|
|
```
|
|
|
|
Settings live in `nx2navi.ini` next to `nx2navi.exe` ([nx2navi.ini.example](../nx2navi.ini.example) lists them). Saves go to
|
|
`build\profile` for `v121` and `build\profile-<ver>` for other versions (`build_game.py --profile` changes it).
|
|
|
|
### Portable folder
|
|
|
|
`.venv\Scripts\python tools\package_build.py --version <ver>` writes `build\<ver>-portable`, which runs on another of
|
|
your machines without MSYS2, Python or .NET. `romfs.bin` stays out unless you pass `--with-romfs`; otherwise put it in
|
|
`romfs\` next to the exe or point `NX2NAVI_ROMFS` at its folder. Run `nx2navi.exe --prepare-shaders` once in the copied
|
|
folder. The folder holds code and data derived from your game: for your own machines only.
|
|
|
|
### Profile-guided build
|
|
|
|
A build with an execution profile compiles the lifted code at -O2 with the counts of a play session, and the runtime
|
|
with them too: hot paths laid out straight, hot calls inlined, functions that never ran compiled for size. On 1.4.3
|
|
the fight bench ran 12-15% faster per frame and a fight outside the training about 6%. Game 1.1.1 has no profile yet
|
|
and no measurement: "Training a profile for 1.1.1" below is the session that makes both.
|
|
|
|
Game 1.1.1 is built by `tools/make_game.py`, on the player's PC through the launcher. It compiles with
|
|
`data\pgo-v111.profdata` of its kit (`app\<version>\data\` of a launcher release, which `make_launcher.py --pgo
|
|
<file>` puts there and builds the prebuilt runtime with; `build\pgo\` in a source checkout) and says so in the
|
|
compile step's log. The profile is left out, and the build is the one a kit without a profile makes (the lifted code
|
|
at -O1), when:
|
|
|
|
- the kit has no profile, or `NX2NAVI_PGO=off` is in the environment (`make_game.py --pgo off`);
|
|
- its record (`pgo-v111.json`) is missing or names another `recomp/` than the kit's (`make_game.py` `pgo_record`).
|
|
Functions are matched by name and control-flow hash, so a function whose C changed loses its counts without a
|
|
warning; the record holds `recomp/` by content (the lifter's `.py` files and `patches-v111.txt`), and any change
|
|
there means a new training;
|
|
- the compiler cannot read the file: one trial compile at the start of the compile step, by each compiler that will
|
|
read it (`pgo_trial`). The log then names what the compiler said.
|
|
|
|
A profile is written by the `llvm-profdata` of one LLVM release and read by the clang of the build tools, which a
|
|
release pins in `toolchain.lock.json` (clang 23.1.5 in 0.9.2). The same release or a newer clang reads it; an older
|
|
one than the file's format refuses it with an error in every compile, which is what the trial compile is for. Train
|
|
with the compiler the release pins: then the instrumented game, the merge and the players' build are one LLVM release.
|
|
|
|
An instrumented build links the compiler's profile runtime, which MSYS2 packs apart from clang and neither
|
|
`C:\msys64` (after the packages above) nor the launcher's build tools hold: the package
|
|
`mingw-w64-ucrt-x86_64-compiler-rt` in the version of the clang it goes with, unpacked into the `msys64` folder of
|
|
those build tools (`pacman -S` for `C:\msys64`; for the launcher's `toolchain\msys64`, the package file from the
|
|
MSYS2 repository unpacked there). `make_game.py` links a test program first and names the package when it is not
|
|
there (`pgo_gen_check`). The merge needs `llvm-profdata` of the same LLVM release: MSYS2's
|
|
`mingw-w64-ucrt-x86_64-llvm-tools`, or the `bin\` of an LLVM installation of that release.
|
|
|
|
#### Training a profile for 1.1.1
|
|
|
|
`<game>` is a game folder `make_game.py` built (the launcher's `game\`, or one built from a checkout), `<ucrt64>` the
|
|
UCRT64 folder of the build tools with the profile runtime, `python` a Python with numpy and lz4 (`.venv`). A
|
|
checkout also needs the shader translator in `build\xlate` (step 4 of "Build"). Run every step from one commit with
|
|
a clean `recomp/`.
|
|
|
|
1. Measure the build without a profile (step 6) while the game folder still holds it.
|
|
2. Build the instrumented game into the same folder. In a folder this checkout built only the compile step runs: the
|
|
lifted code at -O2, every runtime file (the prebuilt runtime counts nothing), and the link.
|
|
|
|
```powershell
|
|
python tools\make_game.py --rebuild --out <game> --msys <ucrt64> --pgo gen
|
|
```
|
|
|
|
From the launcher instead: start `NX2navi.exe` with `NX2NAVI_PGO=gen` in its environment and choose Rebuild the
|
|
Game; the launcher hands the variable on to its build.
|
|
3. Delete `<game>\pgo\*.profraw` left from an earlier instrumented build, then play: start `<game>\ocarina.exe` (or
|
|
Play in the launcher) and play for 20 to 30 minutes with the settings players have by default. Cover the title
|
|
screen and the file select, Kokiri Forest, Hyrule Field on foot and on horseback, a town, a dungeon with fights and
|
|
a boss, the menus and the map, a cutscene, a save and several area changes. What is not played counts as never
|
|
run and is compiled for size. The game is slower than a normal build. Several sessions add up.
|
|
4. The game adds its counts to `<game>\pgo\default_<id>.profraw` every minute and when it ends
|
|
(`runtime/platform/win32.c` `pgo_flush_thread`, `plat_exit`), so a crash loses at most a minute; look for the
|
|
file after the first minute of play. `<game>\pgo\training.json` names the `recomp/` and the compiler the game was
|
|
built from. The game's own runs inside a build (the shader preparation) count into
|
|
`<game>\build-cache\pgo-build\` and are not merged.
|
|
5. Merge, with the `llvm-profdata` of the LLVM release that built the game:
|
|
|
|
```powershell
|
|
python tools\pgo.py merge <game> --version v111 --toolchain <folder with bin\llvm-profdata.exe>
|
|
```
|
|
|
|
This writes `build\pgo\pgo-v111.profdata` and its record `pgo-v111.json`, and stops when the checkout's
|
|
`recomp/` is not the one the game was built from. Then build with it; the log must say `built with the execution
|
|
profile pgo-v111.profdata`, or it says why not:
|
|
|
|
```powershell
|
|
python tools\make_game.py --rebuild --out <game> --msys <ucrt64>
|
|
```
|
|
|
|
6. Compare the two builds with the runtime's own profiler (`runtime/prof.h`, `runtime/prof_win.c`), in the same
|
|
place from the same save, the camera left alone, with the same settings. In `<game>\nx2navi.ini` (or as
|
|
`NX2NAVI_TRACE`, `NX2NAVI_PROFILE` and `NX2NAVI_PROFILE_FRAMES` in the environment):
|
|
|
|
```ini
|
|
trace = perf
|
|
profile = <game>\logs\before.prof
|
|
profile_frames = 7200-10800
|
|
```
|
|
|
|
`profile` (`NX2NAVI_PROFILE`) samples every thread each millisecond; `profile_frames` limits that to the presented
|
|
frames 7200 to 10800, the third minute at 60 frames a second, and the profile is written when the window ends, so
|
|
be in place before it starts. Use `after.prof` for the second build. Then, for each, before the next build
|
|
replaces `ocarina.map` (the report names the functions from the map of the build that was sampled):
|
|
|
|
```powershell
|
|
python tools\prof_report.py <game>\logs\before.prof <game>\build-cache\ocarina.map
|
|
python tools\perf_summary.py <game>\logs\nx2navi.log 7200 10800
|
|
```
|
|
|
|
At the default 60 frames a second both builds draw the same frames, so the sample counts of `guest (f_*)` (the
|
|
lifted code) and `runtime host (exe)` in the report are processor time for the same work: the profile's gain is
|
|
how far they fall. For the frame time itself set `fps_limit = unlimited` and compare the `interval avg` of the
|
|
`perf:` lines in a place the processor bounds, such as a town at a low `render_scale`. Measure a place that was
|
|
in the training and one that was not.
|
|
|
|
The profile and the raw counts are derived from the game's code: they stay under `build\` and in the game folder,
|
|
`tools/check_content.py` refuses them in a commit, and a release carries the profile only as `make_launcher.py --pgo`
|
|
puts it there. `python tests\pgo_build_test.py` checks which flags a build takes for a kit with and without a
|
|
profile, for a record of another tree and for a profile the compiler refuses, with a stand-in for the compiler.
|
|
|
|
The other game versions build with `tools/build_game.py`: `--pgo gen` with an llvm-mingw toolchain makes the
|
|
instrumented `nx2navi.exe`, which writes `<out>\pgo\*.profraw`; `tools\pgo.py merge <out> --version <ver>` merges
|
|
them, and `--pgo build\pgo\pgo-<ver>.profdata` compiles with the result. `make_release.py` ships every profile of
|
|
`build\pgo` in `data\`, where the installer's build takes the one of its update, and warns when a record names
|
|
another `recomp/` tree than the one released.
|
|
|
|
## Linux (in progress)
|
|
|
|
The runtime builds and runs on x86-64 Linux with a window, input and menu on SDL3 (`*_sdl.c`) in place of the Win32
|
|
files (`*_win.c`); `runtime/platform/linux.c` implements `platform.h`. Status: the selftest and the probes pass; 1.2.1
|
|
plays at 60 fps on RADV (RX 7800 XT) and NVIDIA (RTX 3060); 1.4.3 builds, probes and plays on RADV (RX 6600, and the
|
|
Vega 7 of a Ryzen 5 4600G at about 34 fps). Long play sessions and the Steam Deck itself are not verified yet.
|
|
|
|
Requirements:
|
|
|
|
- Linux 6.7 or later: the write watch uses userfaultfd write protection with `PAGEMAP_SCAN`. SteamOS 3.8 qualifies.
|
|
- A Vulkan 1.3 driver with `VK_EXT_external_memory_host` (RADV, NVIDIA).
|
|
- Packages on Arch:
|
|
|
|
```sh
|
|
pacman -S --needed clang lld ninja make sdl3 opus libvpx dav1d zstd vulkan-headers vulkan-icd-loader \
|
|
vulkan-validation-layers shaderc spirv-tools dotnet-sdk uv git
|
|
```
|
|
|
|
Vulkan Memory Allocator is a single header (AUR `vulkan-memory-allocator`). Without it, `build_game.py` fetches the
|
|
version the Windows release pins (3.4.0) into `toolchain/vma/` on first use and checks its hash.
|
|
The Vulkan headers must be release 363 (1.4.363, the one MSYS2 supplies on Windows) or newer: the runtime names
|
|
extensions older headers lack. Where the system's are older, `build_game.py` and `make_game.py` fetch that release
|
|
into `toolchain/vulkan-headers/` on first use, check its hash and put it ahead of the system's (`tools/vulkan_pin.py`,
|
|
`build_game.py` `vulkan_include`).
|
|
|
|
The steps are the Windows ones with POSIX paths: `.venv/bin/python` in place of `.venv\Scripts\python`, and `nx2navi` in
|
|
place of `nx2navi.exe`. `build_game.py` takes clang, lld and Ninja from PATH (or the ones in `<dir>/bin` with
|
|
`--toolchain <dir>`) and links against the system libraries, with `libstdc++` static and an `$ORIGIN` rpath;
|
|
`build_hactool.py` builds `hactool` with make. `tools/prepare_shaders.py` reads the shader list from
|
|
`refs/MacNativeGames` at `origin/HEAD`, which must be the pinned commit with its history (the list format changed later).
|
|
|
|
The probes open a window, so from an SSH session they need the desktop session's display, or they stop at `gpu 0
|
|
skipped: no graphics+compute+present queue family`: on Wayland `XDG_RUNTIME_DIR=/run/user/$(id -u)
|
|
WAYLAND_DISPLAY=wayland-1 SDL_VIDEODRIVER=wayland` (the socket name is in `$XDG_RUNTIME_DIR`), on X11 `DISPLAY=:0`.
|
|
|
|
NVIDIA's Linux driver compiles into a disk cache of its own. A process adds at most `__GL_SHADER_DISK_CACHE_SIZE` to it
|
|
(1 GiB by default; at a start the files after the first are deleted while the folder is over that size), and the default
|
|
folder, `~/.cache/nvidia`, is shared with every program. `--prepare-shaders` of game 1.1.1 builds 42,142 variants, about
|
|
1.5 GB: on the default limit about 30,700 of them were stored and the rest compiled again at their first draw in play.
|
|
`runtime/vk_device.c` therefore points `__GL_SHADER_DISK_CACHE_PATH` at `nvidia-cache` beside `ocarina` and the size at
|
|
8 GiB, each unless the environment sets it, as it does `MESA_SHADER_CACHE_DIR` for RADV, and only while NVIDIA's kernel
|
|
driver is loaded (`/proc/driver/nvidia/version`). The driver does not make that folder and caches nothing in a folder it
|
|
cannot write, so the game makes it, and when it cannot make or write it sets neither variable. A game run sets them in
|
|
`main` before any of its threads starts, since a thread can load the Vulkan driver before the instance is made. The
|
|
folder can be deleted; `ocarina --warm-shaders` fills it again.
|
|
|
|
Memory: on Windows the GPU imports guest memory and `GetWriteWatch` tracks the CPU's writes to it. On Linux,
|
|
re-protecting imported memory invalidates the driver's userptr buffer on every submit (about 1.3 ms per 16 MiB on
|
|
amdgpu), and importing pins the pages, which drops their write protection. Where guest memory cannot be imported as
|
|
dma-bufs (below), each CPU-visible pool therefore gets a
|
|
host-visible copy (shadow storage, `runtime/vk_memory.c`): the pages the CPU wrote go in before each submit, and the GPU's
|
|
writes come back at its fence. Pages both sides write keep a base copy, so each direction moves only its own bytes;
|
|
a VRAM mirror's write-back merge marks the pages it changed, and only those are compared at the fence; a lookup that
|
|
reads guest memory itself (the uniform snapshot, `vk_mem_lookup`) does not queue its pool for the copy-in. In Kokiri
|
|
Forest (1.1.1, RTX 4090) that costs about 1 ms a frame (copy-in 0.5 ms, write-back 0.6 ms), and the copies hold about
|
|
5 GB. The GPU's bytes reach guest memory in the fence's completion callback, later than an import's, so the wait
|
|
after a frame's first submission covers the write-backs there (`runtime/vk_sync.c`, "the executed timeline"): the game
|
|
rewrites what those passes wrote without a fence, and a write-back that landed after the rewrite left the GPU adding
|
|
light onto its own last output, the whole scene brighter for a frame. `NX2NAVI_SHADOW=1` turns it on under Windows too. The Steam Deck's RADV has `VK_EXT_external_memory_host` as
|
|
well: the shadow copies are chosen for the measured cost above, not for a missing extension. NVIDIA's own driver on Linux
|
|
imports instead (`NX2NAVI_SHADOW=1` brings the copies back): it imports guest memory without that cost, so there the
|
|
copies are only the slower path. In Kokiri Forest on 0.7.1 (RTX 4090, driver 615.78), imports with VRAM mirrors ran at
|
|
51.9 frames a second and the copies at 27.4; on that version the copies also flickered between two looks almost every
|
|
frame, which the wait for the write-backs under shadow storage (`vk_sync_exec_wait`) has since addressed.
|
|
|
|
On Linux the pools are imports after all where the system allows dma-bufs of guest memory (`runtime/platform/linux.c`,
|
|
"Guest memory as a memfd's"; `runtime/vk_memory.c`, "dma-buf imports"). The watched reservations are then shared
|
|
mappings of one memfd, `/dev/udmabuf` makes dma-bufs of page ranges of it, and the device imports those
|
|
(`VK_KHR_external_memory_fd`, `VK_EXT_external_memory_dma_buf`). A dma-buf pins its pages once and is not tied to the
|
|
process's page tables, so the write watch and the freezes leave the GPU's mapping alone, and nothing is copied. RADV
|
|
puts every imported memory object into every submission (about 0.6 us each), and the game keeps over 4,000 pools, so
|
|
pools share their imports: guest addresses are cut into spans of 64 MiB (`NX2NAVI_DMABUF_MB`), and a pool is a buffer
|
|
bound into its span's memory at its offset. A new span is cut back at the spans already there, so spans overlap only
|
|
where a pool itself lies across an older span's edge: amdgpu counts every span's pages in its GTT use, an overlap twice.
|
|
A span pins all of its pages, those the game never touched included, until some frames after its last pool is gone. It
|
|
takes `/dev/udmabuf` readable and writable by the player (recent systemd gives it to the logged-in user with its
|
|
`uaccess` tag; elsewhere it is often `root` and group `kvm` only), udmabuf's `size_limit_mb`
|
|
(`/sys/module/udmabuf/parameters/size_limit_mb`) at 1024 or more, since the largest pools need about 300 MiB in one
|
|
dma-buf (where it is lower, `udmabuf.size_limit_mb=1024` on the kernel command line raises it), a device with both
|
|
extensions, and NVIDIA's kernel module not loaded: NVIDIA's own driver imports host pointers without amdgpu's cost, as
|
|
above. A start-up probe imports one page of guest memory as the pools will be; where anything is missing or refused, the
|
|
pools keep their shadow copies and the log says why (`mem: dma-buf imports off`, `mem: guest memory is not a memfd's`).
|
|
`NX2NAVI_DMABUF=0` keeps the copies, and `NX2NAVI_DMABUF=1` makes guest memory a memfd's even where NVIDIA's module is
|
|
loaded; both only from the environment, since the reservations are made before `nx2navi.ini` is read. `NX2NAVI_SHADOW=1`
|
|
still forces the copies. In Kokiri Forest (1.1.1, RX 6800 XT, Mesa 26.2) the imports ran at 60.7 frames a second and the
|
|
copies at 40.3, in Hyrule Field at 55.7 and 42.4; the spans then held 8.0 to 9.6 GiB, and amdgpu counted 8.6 to 10.1 GiB
|
|
of its 15.8 GiB host heap in use, against 5.2 to 5.6 GiB with the copies, which are memory of their own beside guest
|
|
memory.
|
|
|
|
The device address table (`runtime/vk_gpa.c`) names a pool's shadow copy as it names an import, so shaders follow guest
|
|
pointers into either. A shader that reads through the table resolves nothing, so the pool it reaches is not queued for
|
|
the copy-in: a guest submission whose record binds the table (`vk_exec_draw.c` `ssbo_info`) brings the shadow of
|
|
every pool in the table up to date with the pages the CPU wrote, once, before the first part recorded after the
|
|
binding is submitted, and queues the same pages for those pools' VRAM mirrors (`vk_memory.c` `vk_mem_shadow_sync`).
|
|
That copies the written pages of every CPU-visible pool once in each such guest submission, which the cost above does
|
|
not include. What a shader stores through the table lands in the shadow copy
|
|
and in the pool's VRAM mirror, and no binding names it, so nothing notes it as a GPU write: it does not come back to
|
|
guest memory, and the next whole copy-in of its page (the CPU rewrites other words of the same records) puts guest
|
|
memory's stale bytes back over it in both copies. In the runs measured so far that copy-in always came before the
|
|
frame's stores, which does no harm. `NX2NAVI_SHADOW_TABLE_STORES=1` keeps the stores instead: the copy-ins then keep a
|
|
hash of each page they write, a page whose shadow no longer matches its hash was stored into, keeps a base copy from
|
|
then on like a page with a noted GPU write, and has its stores written back to guest memory at the fence of every
|
|
submission that uses the table. It is off by default (`runtime/vk_memory.c`, "Pages a shader stores into", says why).
|
|
`NX2NAVI_SHADOW_CHECK=1` logs every 60 frames how often a copy-in overwrote such a store and for which pools, how many
|
|
write-backs landed on pages the CPU had written since their copy-in, and what the copy-ins cost and where they went.
|
|
`--probe-vkmem` checks the sync: the shadow of a pool that is never bound must hold `0x13579BDF`, keep it through a
|
|
plain CPU write and a guest submission that does not bind the table, and hold the written `0x2468ACE0` after one that
|
|
does; and with `NX2NAVI_SHADOW_TABLE_STORES=1` the store: a GPU
|
|
write into a shadow page that nothing notes must survive the CPU's writes of that page and reach guest memory. On
|
|
Linux, run it for each backing (with `NX2NAVI_DMABUF=1` and `NX2NAVI_SHADOW=0` the pools are dma-buf imports, with
|
|
`NX2NAVI_DMABUF=0` and `NX2NAVI_SHADOW=0` host-pointer imports):
|
|
|
|
```sh
|
|
for dmabuf in 0 1; do for shadow in 0 1; do for memory in import mirror; do
|
|
NX2NAVI_DMABUF=$dmabuf NX2NAVI_GPA=1 NX2NAVI_SHADOW=$shadow NX2NAVI_MEMORY=$memory ./ocarina --probe-vkmem
|
|
done; done; done
|
|
```
|
|
|
|
The Deck's GPU (Van Gogh) has no native ASTC, so its textures go through the compute decoder (`runtime/vk_astc.c`,
|
|
`NX2NAVI_ASTC`) as on desktop AMD GPUs without it; frame times measured there include it.
|
|
|
|
## Checks
|
|
|
|
Before a commit:
|
|
|
|
- `python tools/check_content.py --staged`: nothing game-derived and no compiled shader is staged. CI runs the same check;
|
|
`git config core.hooksPath .githooks` runs it before each local commit.
|
|
- `build\<ver>\nx2navi.exe --selftest`: layout, dispatch table and host-function registry, without game files.
|
|
- `python tools\check_platform.py` after a runtime change outside `runtime/platform/`: no shared file uses Win32 outside
|
|
a branch that only a Windows build compiles (`#ifdef _WIN32`, the `#else` of `#ifndef _WIN32`).
|
|
`python tests\check_platform_test.py` checks the scanner itself on made-up files.
|
|
|
|
A change to the in-game menu's look (`runtime/ui/menu_skin.cpp`, `option_rows.cpp`, `perf_panel.cpp`) can be looked at
|
|
without game files: `python tools/menu_preview.py` builds the menu's own drawing code with the host's clang++ and
|
|
writes the Screen tab, the full performance overlay and the boot screen to `build/menu-preview/` (`menu.png`,
|
|
`overlay.png`, `boot.png`; a stand-in font, and the window's plain border in place of the game's ornamented frame).
|
|
With `--romfs <extracted RomFS folder>` it draws the game's interface textures as the game does, and
|
|
`tools/ui_textures.py <folder>` lists the textures there and dumps those matching `--match` to `build/ui-textures/`;
|
|
both decode the game's files, so they run with `.venv-py314\Scripts\python` (`compression.zstd` and numpy). What they
|
|
write is decoded game data: keep it under `build/`.
|
|
|
|
A change to the shared save (`runtime/save_zip.c`, `runtime/save_share.c`, the save import in `runtime/fs_host.c`)
|
|
passes `python tests\save_zip_check.py` and `python tests\save_share_check.py`. The first builds `save_zip.c` alone
|
|
and checks its deflate, inflate, zip writer and reader against Python's zlib and zipfile both ways, and that the
|
|
launcher imports a zip it wrote. The second builds `fs_host.c` and `save_share.c` with the rest of the runtime stubbed
|
|
and runs each step as a start of the game on a profile it makes: the export, an import from a zip the launcher
|
|
exported, its backup and swap at the next start, the zips it refuses and Cancel. No game files.
|
|
|
|
A change to the step Ocarina is given each frame (`runtime/frame_rate.c` `ocarina_step`), or to the whole numbers the
|
|
game's counters get out of it (`whole_take`, `frame_rate_day_host`, `frame_rate_water_host`, `frame_rate_sim_host`), passes
|
|
`python tests\frame_step_check.py`, which builds `frame_rate.c` with the rest of the runtime stubbed and feeds it
|
|
display times: a held 60 fps limit gives exactly one frame every frame, a 240 fps limit never less than a quarter
|
|
frame, uneven frame times one steady step, and game time follows the clock. While RivaTuner Statistics Server holds
|
|
the limit the pacer keeps the 240 ceiling, the limit's own period once no host present went out for 200 ms, and a
|
|
movie's rate. It calls the two hooks as their patch
|
|
sites do: at 144 fps each of 64 water objects gets its ripple steps and the day its seconds, at 120, 144 and 240 fps
|
|
the water managers and floors of the set `owater` get 60 fixed steps a second with only s0 changed, and a step of one
|
|
frame gives what the game computes. It also checks that each of those sites in `recomp/patches-v111.txt` ends in one
|
|
`bl` to its page slot, and runs the physics solver step count's words of the original and the sets `osolver` and
|
|
`osolverfull` (`physics_steps` 0, 1 and 2), lifted as the generator lifts them: the game's 4 and 8 at 60 and 30 fps
|
|
for all three, rounded up by `osolver` and at least 4 every update by `osolverfull` above 60 fps. For the set `ofollow`
|
|
(`floor_carry`) it checks the two words of each of the three floor-carry gates and runs the gate's compare with the
|
|
game's words and the set's: the page's cut is 100 at 60 and 30 fps, and from 90 to 240 fps the set cuts a floor's
|
|
velocity at the time scales where the game cuts it at 60 fps and nowhere else. No game files.
|
|
|
|
A change to how the frame limit goes to RivaTuner Statistics Server (`runtime/rtss_limit.c`, `runtime/rtss_decide.h`)
|
|
passes `python tests\rtss_limit_check.py`. It builds `rtss_decide.h` alone and checks its reader of RTSS's shared
|
|
memory against a fake `RTSSSharedMemoryV2` (a bad header, short entries, an entry past the memory or past the 4096th
|
|
find nothing; the flags and the period where RTSS keeps them) and the worker's decisions poll by poll against a fake
|
|
profile and a fake RTSS clock: RTSS's injection delay through listed with no API, the profile not read yet and no
|
|
period measured, RTSS quitting and coming back, a read-only profile, a refused limit, the frame count rule (1.05 times
|
|
the limit over three new periods, a frozen period never counted), an RTSS that lists the game and never shows its
|
|
hold (given up after 30 s, a minimised window not counted), no picture going out, kept-out wants and their
|
|
reasons, a want that must stay two polls and low latency switching at every poll, a movie at 144, 60 and 40, a closed
|
|
run, a hooks library that does not load, and two limiters. It also checks the pacer's period beside RTSS
|
|
(`rtss_pace_period`), the limit's rounding without a denominator, and the state file `rtss-profile.txt`: its lines,
|
|
its rewrite and what a start does with a line a run that did not close left, the value written before the newest one
|
|
included. Statically, a crash and a fatal error
|
|
(`main.c` `crash_handler`, `dispatch.c` `fatal_end`) go through `rtss_limit_crash`. On Windows it also builds the real
|
|
`rtss_limit.c` against a fake RTSS (its registry key, hooks library, profile files and shared memory) and runs each
|
|
scenario in a process of its own, among them a want after the exit's restore and one the worker read before it, a
|
|
worker polling without pause while the want switches, the registry and file calls a present must not make, and a run
|
|
killed while RTSS held its limit followed by the next start, each in a process of its own (a next start with a limit,
|
|
with frame generation on and with `NX2NAVI_RTSS=0`, and a run killed inside the save of its second limit), put backs
|
|
that do not land, a state file that does not read, and a crash on RTSS's own thread with no line in the file. No GPU,
|
|
no RTSS, no game files.
|
|
|
|
A change to how the mixer applies volumes (`runtime/audio_mixer.c` `accumulate_ramp`, the voice and splitter paths
|
|
of `audio_mixer_render`) or to what it owes the device after a stall (`ring_pull`, `audio_mixer_owed`) passes
|
|
`python tests\audio_mixer_check.py`, which builds `audio_mixer.c` with the rest of the runtime stubbed. It mixes
|
|
voices of one constant sample, so a mix buffer shows the gain sample by sample: a
|
|
volume stepped from 0 to 1 rises in equal steps across the next 5 ms frame and holds after it, for a voice into a mix,
|
|
a voice into a splitter destination and a submix into one; a voice that starts, was silent in the frame before or was
|
|
moved to another mix, a destination that is new or was moved, and the first frame after the renderer starts take the
|
|
new volume at once; `NX2NAVI_AUDIO_RAMP=0` applies every volume at once. It also makes the device's requests by
|
|
hand: the silence of an underrun, up to the prefill's return, is owed in whole render frames, which are rendered and
|
|
not queued; the silence before the first prefill and across a stop is not owed, and more than 3 seconds of it is given
|
|
up. The render thread's own timing (`runtime/audren.c` `render_thread`) is not built by it. No game files and no audio
|
|
device.
|
|
|
|
A change to what the audio renderer does with values the game sends (`runtime/audren.c` `update_check`, `request_update`
|
|
and `audio_bad`; `runtime/audio_mixer.c` `splitters_walk`, `audio_mixer_routing_check`, `audio_mixer_mix_layout_check`,
|
|
`audio_mixer_update_effects` and the render-pass effects; `runtime/audio_effects.c` `audio_effect_invalid`;
|
|
`runtime/audio_reverb.c` `audio_reverb_invalid`) passes `python tests\audio_params_check.py`. It builds `audren.c` and
|
|
`audio_mixer.c` with the rest of the runtime stubbed and the real effect DSPs linked, and sends hand-built
|
|
RequestUpdates through `renderer_handler`. An update whose sections do not match the renderer (sizes, magic words,
|
|
overruns, the output buffer, a mix layout with a negative count or more buffers than the renderer has, a splitter id run
|
|
whose length wraps in 32 bits) answers 0x5299, a result the SDK hands back to the game (it aborts on any result but
|
|
0x5299, 0x5499 and 0x5699), and writes no output, changes no renderer state and places no C buffers. A channel, voice or
|
|
splitter record with an id the renderer does not have, a wave buffer index of 4, a filter that is not a finite number
|
|
and a changed mix with a foreign id are left out of an update that is otherwise applied; a splitter destination id the
|
|
renderer does not have is never read (a guard entry before the destination table would make it heard); a splitter
|
|
section whose framing breaks between the check and the apply keeps what came before the fault and is logged as the rest
|
|
of the section left out, not as a refused update. Each effect and sink whose values the renderer cannot use is passed
|
|
through or skipped in every render pass, with nothing written to the game's rings or sink buffer, and runs again once
|
|
its values are usable. Each kind is logged once and counted every time; valid effects of every type over 20 passes leave
|
|
nothing out. With `NX2NAVI_AUDIO_STRICT=1` such a value ends the process, in two runs of their own. No game files and no
|
|
audio device.
|
|
|
|
A change to the sprint button (`runtime/sprint.c`, the set `sprint` of `recomp/patches-v111.txt`, its call in
|
|
`runtime/ov_input.c` `npad_states`) passes `python tests\sprint_check.py`, which builds `sprint.c` with the rest of the
|
|
runtime stubbed and plays frames against a stand-in Link: toggle and hold on each kind of button, the states that keep
|
|
and end a sprint, ZL, the button kept from the game only during play (the bottom button only while Link moves) and a
|
|
Link the player does not control, and the set switched off with the button off but never over a start that kept it
|
|
off. No game files.
|
|
|
|
A change to the text speed (`runtime/text_speed.c`, the set `text_speed` of `recomp/patches-v111.txt`, its call in
|
|
`runtime/ov_input.c` `npad_states`) passes `python tests\text_speed_check.py`, which lifts the set's replacement words
|
|
with `recomp/lifter.py` and runs them with `text_speed.c` against a stand-in message window and typewriter: the page
|
|
words each setting writes and the set it switches, the skip site's decide bit and typewriter word for a message the
|
|
player may skip and one the game keeps from skipping, the speed site's factor for that typewriter and no other, the
|
|
registers and flags both sites leave alone, and a start that kept the set off. No game files.
|
|
|
|
A change to the ocarina recorder's clock (the set `orecord` of `recomp/patches-v111.txt`, `ocarina_timing`) passes
|
|
`python tests\ocarina_timing_check.py --cc <clang>` (`.venv`, Unicorn): it checks the set's four sites, their original
|
|
word and nine replacement words, the page word resolved to the patch page at each site and no other set's word on the
|
|
page's +264; it lifts the words and runs them against the game's word and a float32 model: steps of one and two frames
|
|
give the game's count with the fraction left at 0, the steps of the 40 to 240 fps limits count 3600 frames a minute
|
|
(0 and 1 in turn at 120, 5 in every 12 updates at 144, 1 in every 4 at 240), NaN, negative, infinite and huge steps
|
|
give the game's count, and only x8, v16-v18 and FPSR change; and it runs the nine words in Unicorn at each site against
|
|
the lifted C over 10,000 random steps and fractions, comparing registers and the page bit for bit. It does not cover
|
|
the sites inside their functions: a regenerate of the game code does, and its check of the original words. No game
|
|
files.
|
|
|
|
A change to the mouse camera (`runtime/mouse_direct.c`, the mouse path of `runtime/input.c`, the set `mouse_direct` of
|
|
`recomp/patches-v111.txt` or `patches-v143.txt`) passes `python tests\mouse_direct_check.py --cc <clang> --deps
|
|
<ucrt64>\include`, which builds `mouse_direct.c`, and `input.c` with `--deps`, with the rest of the runtime stubbed and
|
|
calls the camera hooks as their patch sites do, for each game version: the registers a site hands over and gets back,
|
|
the same turn for the same mouse distance at every frame rate and stick camera speed, a single count, the idle bit, the
|
|
pause after a camera reset, the capture and the mode, and scripted mouse counts. No game files.
|
|
|
|
A change to which waits a signalled event or an ended thread wakes (`runtime/kernel.c` `wake_sync_waiters`, the
|
|
`WaitSynchronization` service) passes `python tests\sync_wake_check.py`, which builds `kernel.c` with the platform
|
|
layer and the rest of the runtime stubbed and runs guest threads whose entries are host functions: a signal wakes the
|
|
waits on its event and leaves the others asleep, a wait on several handles reports the right one, a thread's end ends
|
|
the wait on it, timed waits keep their time, a cancel ends any wait, and 4,000 signals to eight looping waiters are
|
|
each seen. It runs a second time with `NX2NAVI_SYNC_WAKE_ALL=1`. No game files.
|
|
|
|
A change to what a host thread gives back when it ends (`runtime/platform/win32.c` `thread_main`,
|
|
`plat_thread_end_hook`) passes `python tests\thread_end_check.py`, which builds the platform layer alone: 416 threads
|
|
sleep, read a file and end, half joined and half detached; the end hook runs once on each, and on Windows the process
|
|
holds as many handles afterwards as before (each thread's sleep timer and read event are closed). No game files.
|
|
|
|
A change to the census of the process's handles by kind (`runtime/platform/win32.c` `plat_handle_census`, which
|
|
`NX2NAVI_HANDLE_CENSUS` logs) passes `python tests\handle_census_check.py`, which builds the platform layer alone: 20
|
|
events and 5 sections made show as Event +20 and Section +5, the total stays within 2 of the process's handle count,
|
|
the figures go back once they are closed, and the kinds past the room given are summed in the last entry. Windows
|
|
only. No game files.
|
|
|
|
A file a detached thread writes piece by piece (the flight recorder's dump, `runtime/flight.c` `dump_main`) goes
|
|
through `runtime/part_file.h`: it is written as `<name>.part` and renamed when complete, because the window's close
|
|
ends the process under such a thread and a file cut short there would read as a whole one. A file made in memory
|
|
first uses `plat_file_write_cached` or `plat_file_write_atomic`, which do the same in one call. A change to the
|
|
helper passes `python tests\part_file_check.py`, which builds it with the platform layer: nothing stands under the
|
|
name while a file is written, an older file stays until the new one is complete and after a write that was given up,
|
|
and a writer killed in the middle leaves the older file whole and the half as `.part`. No game files.
|
|
|
|
A change to the log queue (`runtime/logq.c`) passes `python tests\logq_check.py`, which builds it with the platform
|
|
layer and a destination of its own: 160,000 lines from eight threads arrive whole and in each thread's order, a line
|
|
logged while the queue's lock is taken waits and does not pass the lines queued before it, a destination that takes
|
|
30 ms a write does not hold the threads that log, the line about slow writes comes once in ten seconds, a destination
|
|
that stops while four times the queue's size is logged loses nothing, lines logged together are written in one write
|
|
a quarter of a second after the first and not before, a queue that is half full is written at once, and `logq_sync`
|
|
returns with everything written, the lines the log thread was still gathering included. The program then runs twice
|
|
more as a process that logs 200 lines to a file and ends at once, by an exit and as a fatal error does: each file
|
|
holds every line, and the fatal line last.
|
|
The test passes on a busy machine too: it waits for the queue to empty instead of for a time, counts slow lines
|
|
instead of timing one, leaves out a check its own thread was too late for, and tries the half-full write three times.
|
|
No game files.
|
|
|
|
A change to the button prompts (`tools/make_romfs_patches.py --prompts`, `tools/button_icons.py`) passes
|
|
`.venv-py314\Scripts\python tests\romfs_prompts_test.py`, which runs the tool on a RomFS it makes itself (no game
|
|
files); a change to the settings passes `nx2navi.exe --probe-cfg`.
|
|
|
|
A change to the action prompt's diamond of face buttons (`runtime/action_prompt.c`) or to the SARC and zstd dictionary
|
|
helpers (`runtime/sarc.c`) passes `python tests\action_prompt_check.py`. It builds them alone, with the
|
|
undefined-behaviour sanitizer, on a layout file, archives and dictionaries it makes from the format `action_prompt.c`'s
|
|
header comment describes, and checks that:
|
|
|
|
- by label every place, cross, flip and bubble moves and no other byte, in every copy of the layout in the archive;
|
|
- a second run changes nothing, and by label then by position gives the game's file back;
|
|
- a file that is not the expected one is refused and left as it was: one value off, half laid out, one copy of four,
|
|
damaged offsets, a pane record laid over the records that lead to another field, more than 16 copies, copies that
|
|
share some of their bytes;
|
|
- the packed file names the dictionary the original named, found by its id in the pack of dictionaries.
|
|
|
|
No game files. `--romfs <folder with romfs.bin and index.json>` also runs it on the game's own archive: four copies laid
|
|
out, 208 fields, the same dictionary and unpacked size, and no byte changed outside the layout's copies.
|
|
|
|
A change to what the library warm logs when the driver's cache of stage libraries is cold (`runtime/vk_warm_hint.h`,
|
|
`runtime/vk_pipeline.c` `lw_main`) passes `python tests\warm_hint_check.py`, which builds the header alone and checks
|
|
the added text by GPU vendor and by why the cache was cold. No game files.
|
|
|
|
A change to how a run ends when the GPU device is lost (`runtime/vk_device.c` `vk_device_lost_check`, the checks of
|
|
`vk_lost` in the replay, upload and present paths, `runtime/vk_sync.c` `completion_main`) passes
|
|
`python tests\device_lost_check.py <game exe>`, which runs the executable's `--probe-devlost`: with the device marked
|
|
lost no completion callback runs and the submit, upload and wait paths return at once, and four threads that report a
|
|
loss at the same moment end the process with exit code 72, one device report and one FATAL line. It needs a Vulkan
|
|
device and opens no window; the device is not really lost, so what a driver answers after a loss is not covered.
|
|
`device_lost_at = <n>` in `nx2navi.ini` ends a real run the same way at its n-th hand-over to the GPU queue.
|
|
|
|
A change to how a fault ends the process (`runtime/main.c` `crash_handler`, `mode_done`) passes
|
|
`python tests\exit_fault_check.py <game exe>`, which runs two crash probes: a fault inside a library while a mode runs
|
|
is a crash (one FATAL line, exit code 71), and the same fault after the mode has returned, while the process is ending,
|
|
leaves the mode's own exit code and one `exit:` line. The second is what a library another program put into the
|
|
process does when it faults in its detach: an injector's Vulkan layer did so at the end of every shader preparation on
|
|
a player's PC, and the launcher took each finished preparation for a failed one. No game files, no GPU.
|
|
|
|
A change to the movie timestamp sites of `recomp/patches-v111.txt` (set `omovie`) passes
|
|
`python tests\movie_timeline_check.py --cc <clang>` (`.venv`, Unicorn): it lifts the twelve replacement words on
|
|
their own and compares them with Unicorn over repeated, skipped and rounded timestamps, registers, flags and memory.
|
|
It does not cover the sites as `gen.py` emits them inside their two functions, nor the add and store after them:
|
|
a regenerate of the game code and a compile of the chunk that holds main+0x3b5c7c4 and main+0x3b5c908 does.
|
|
|
|
A change to the memory gate of the background pipeline builds (`runtime/vk_build_gate.h`, `runtime/vk_pipeline.c`
|
|
`gate_enter`) passes `python tests\build_gate_check.py [<game exe>]`, which builds the header alone and checks what it
|
|
admits and refuses and that builds under way count through their reservations. With the executable it also runs
|
|
`--probe-buildgate` (no device, no window, no build) with the settings in effect, with `build_free_mb = 0` and with a
|
|
value no machine has free. What the builders do when they are refused (`warm_builder`, `pipe_worker`, `lb_worker`,
|
|
`lw_worker`) shows in a run's log: start the game with `build_free_mb` set above the machine's free commit and read the
|
|
`vk_pipeline:` lines that name the commit headroom.
|
|
|
|
A change to when the pipeline caches are saved in play (`runtime/vk_save_retry.h`, `runtime/vk_pipeline.c`
|
|
`autosave_once`) passes `python tests\save_retry_check.py`, which builds the header alone: a save for new content, none
|
|
without, and after a save that did not happen a bounded row of tries at the save interval.
|
|
|
|
A change to how FSR 3.1's pipelines reach the pipeline caches (`runtime/vk_ffx.cpp` `vk_ffx_proc_addr`,
|
|
`runtime/vk_pipeline.c` `vk_pipeline_create_foreign`) passes `python tests\ffx_cache_check.py <game exe>`. It runs
|
|
`--probe-ffx-cache` (one upscaling context on the device; no window, no game) twice on pipeline cache files of its own
|
|
and once with `ffx_pipe_cache = 0`: the library's pipelines must come through the runtime's creates, the second run
|
|
must compile none of them, and with the setting off none may come through. The executable needs
|
|
`amd_fidelityfx_vk.dll` beside it; without it the probe and the test skip. The graphics driver's own cache can serve
|
|
the first run too, so to see the files alone do it, run the probe from a copy of the executable under a name the
|
|
driver has not seen: its first run compiles, its second does not.
|
|
|
|
A change to the start's memory warning (`runtime/mem_need.h`, `runtime/memory.c` `mem_start_note`) passes
|
|
`python tests\mem_need_check.py [<game exe>]`, which builds the header alone and checks when the game's need counts
|
|
as short. With the executable it also runs `--probe-memneed` (it prints the machine's commit and the answer and
|
|
starts nothing) with `mem_need_mb` at 0, at 1 and at a value no machine has, and reads the warning line of the last.
|
|
A change to when a start does the whole warm on the boot screen (`runtime/boot_warm.h`, `runtime/vk_pipeline.c`
|
|
`bf_decide`) passes `python tests\boot_warm_check.py`, which builds the header alone: the warm after a driver change,
|
|
its continuation by a later start, the end of the tries, no second warm once one is done, and what a marker left by
|
|
a start that ended in the boot's builds means with and without the crash handler's note. The warm itself shows
|
|
in `--probe-prewarm` on a copy of an install's caches whose `pipeline-cache.bin` has another driver's header (bytes
|
|
8 to 15 of the file are the vendor and device ids, the 16 after them the driver's cache id): the log names the whole
|
|
warm, the compute pipelines on every core and the library warm's answer, and `boot-warm-owed.txt` ends as "done".
|
|
Run under an exe name the driver has not seen, the probe compiles everything, as after a real driver update. What
|
|
the boot screen shows, and the builders' step down when the wait is cut short, need a start of the game.
|
|
A change to how `pipeline-cache.bin` is read (`runtime/vk_cache_file.h`, `runtime/vk_pipeline.c` `cache_load`) passes
|
|
`python tests\pipeline_cache_file_check.py`, which builds the header alone and reads a sparse file just over 2 GB
|
|
through it: measured and read whole without a size limit, refused by its real size under one. No game files, no GPU;
|
|
about 2 GB of memory for a few seconds.
|
|
|
|
A change to how `pipeline-binaries.bin` is indexed or cut back (`runtime/vk_pipeline.c` `pb_index`, `pb_keep`,
|
|
`pb_init`) passes `python tests\pb_torn_check.py`, which cuts the record layout, the index walk and `pb_keep` out of
|
|
`vk_pipeline.c` and walks files of whole records, with a last record cut short at many lengths and with a malformed
|
|
record: the whole records before it are indexed and the walk ends where they end, `pb_keep` cuts the file back to
|
|
there, a second walk reaches the end of what is left, and a record appended after the cut is read back. No game files,
|
|
no GPU.
|
|
|
|
A change to the layout of `shader-prepared.bin` (`runtime/prp_file.h`, `runtime/gfx_shader.c`
|
|
`gfx_shader_prepare_seed`, `tools/make_game.py` `prepared_from`) passes `python tests\prp_file_check.py`, which builds
|
|
the header alone, checks the parse of both versions and has `make_game.py` read the list the C side wrote. No game
|
|
files, no GPU.
|
|
|
|
A change to which translations of a previous translator are served (`runtime/gfx_shader.c` `carry_usable`,
|
|
`carry_load`, `carry_find_iid`, `carry_last`) passes `python tests\carry_pool_check.py`, which cuts the carry lookups
|
|
out of `gfx_shader.c` and runs them against stand-in caches: a translation whose texture pool array has the earlier
|
|
length (32768) is freed and the lookup goes on, one with the current length or no pool is served, and once the current
|
|
translator has failed the shader the refused translation serves from the memory cache. No game files, no GPU.
|
|
|
|
A change to how the translator children are read and restarted (`runtime/platform/win32.c` `plat_child_read`,
|
|
`runtime/gfx_shader.c` `child_exchange_raw`, `svc_request`) passes `python tests\child_timeout_check.py`, which builds
|
|
those functions with a 300 ms read timeout and runs `cmd.exe` children through them: one that never answers costs one
|
|
timeout and is not asked again, one that exits is restarted once, one that answers gives its line. Windows only; no
|
|
game files, no GPU.
|
|
|
|
A change to where the lazy write-back is on by default (`runtime/wb_lazy_choice.h`, `runtime/vk_memory.c`
|
|
`vk_mem_lazy_decide`) passes `python tests\wb_lazy_check.py`, which builds the header alone and checks the choice and
|
|
its reason by platform, by GPU driver and by what the player set. No game files, no GPU.
|
|
|
|
A change to the lazy write-back's arms, faults, releases or fetches (`runtime/memory.c` "Lazy write-back" and "Ranges
|
|
in flight", `runtime/vk_memory.c` `lazy_fetch`, `mirror_evict`, `hot_check`) passes `python tests\lazy_fetch_check.py`,
|
|
which builds memory.c with the Windows fault handler and a fetch that only waits, and runs each scenario as its own
|
|
process, so an access violation fails it. A thread that reads a range over and over while it is armed and given back
|
|
again never reaches the crash handler, also when the arm's clean test takes 100 ms, and a read of a range being armed
|
|
does not either when a range in flight in its region ends during that test. A fetch holds up nothing but the touches
|
|
of its range: freezes, thaws, arms, releases and copies of other ranges return at once meanwhile, also while a thaw
|
|
fetches the armed ranges of its own regions, two ranges fetch at once, and more touching threads than the flight table
|
|
holds all end. The touches of a range in flight wait for it and run again, a write among them without releasing the
|
|
region's other ranges, also when the flight ended while the write waited for the freeze lock; thaws, mapping changes,
|
|
releases of its buffer, arms and freezes see it. The same with
|
|
`wb_lazy_chunk = 1`, and with `wb_lazy_unlocked = 0` the fetch holds the freeze lock again, a thaw's too, and a
|
|
release fetches one range at a time (the table of ranges in flight stays: a touch of one still waits for it alone).
|
|
The "mem: lazy write-back" lines give the longest fetch, the longest wait of a fault for the freeze lock and the
|
|
touches that waited for a range in flight. `vk_memory.c`'s fetch slots, and its evictions (`mirror_evict`,
|
|
`hot_check`: their fetch before `pool_lock` is taken with `wb_lazy_unlocked = 1`, under it with 0, and their order
|
|
against a pool's finalize), have no check without a device. Windows only; no game files, no GPU.
|
|
|
|
A change to when the freezing of idle guest regions waits (`runtime/freeze_hold.h`, `runtime/vk_memory.c`
|
|
`freeze_pass`) passes `python tests\freeze_hold_check.py`, which builds the header alone: the backoff after a slow
|
|
freeze doubles while slow ones come within the quiet time, fast freezes between them included, stops at its cap and
|
|
starts over after the quiet time (with a quiet time of 0, at the first fast freeze); and an area load holds the passes
|
|
while the texture pool loads and for the hold after it or after the area's load. No GPU, no game files.
|
|
|
|
A change to what writes `nx2navi.ini` outside the game (`launcher/gameini.py`, `installer/ini.py`, `tools/make_game.py`,
|
|
`launcher/updater.py`) passes the settings-file tests of `tests\launcher_test.py` (`PlayerIniTest`, and the tests named
|
|
`..._settings_file` of `BackgroundUpdateTest` and `MakeGameTest`): a player's file with lines that are no settings of
|
|
the game comes through an update and a rebuild byte for byte. The game's own writer, the settings menu, is checked by
|
|
`--probe-cfg` with the same lines.
|
|
|
|
A change to the mouse camera's stick deflection (`runtime/input_map.c` `input_mouse_stick`) passes
|
|
`python tests\mouse_stick_check.py`, which builds that file with the rest of the runtime stubbed and checks the
|
|
deflection at each camera speed: a slow movement is divided by the factor, a fast flick reaches full tilt. No game
|
|
files.
|
|
|
|
A change to how a gamepad's axes are read (`runtime/input.c` `pad_axis`: an axis counts once it has been seen at rest)
|
|
passes `python tests\pad_rest_check.py`, which builds that function with stand-ins for SDL. No game files.
|
|
|
|
A change to the shake key (`runtime/input.c` `shake_stick`) passes `python tests\shake_key_check.py`, which builds that
|
|
function with stand-ins for the clock and the game's present count: each side lasts two of the game's updates from 20
|
|
to 240 fps, also with two reads an update, a press swings for 800 ms, a hold keeps swinging until its release, and a
|
|
swing without presents keeps its side. No game files.
|
|
|
|
A change to HD rumble's encoding or report schedule (`runtime/rumble_hd.c`: a vibration value as the bytes of a Switch
|
|
controller's rumble report, and which reports are written when) passes `python tests\rumble_hd_check.py`, which builds
|
|
that file alone: known values and the neutral one, the frequency and amplitude clamps (no amplitude step above the
|
|
actuators' safe maximum), a report's data for two actuators and for one, and a batch of values reduced to the reports
|
|
written (the newest value at once, a held one again every 30 ms, the neutral value three times once the game stops).
|
|
The script then decodes a sweep of encoded values with formulae of its own. No game files, no SDL, no controller: what
|
|
a controller does with the reports is not covered.
|
|
|
|
A change to how a mesh pipeline's fragment module gets the inputs no mesh output feeds (`runtime/vk_pipeline.c`
|
|
`spirv_mesh_outputs`, `spirv_fs_fill_inputs`) passes `python tests\fs_fill_inputs_check.py`, which builds both
|
|
functions and runs them on mesh and fragment shaders it compiles with glslang, then validates the result. No game
|
|
files.
|
|
|
|
A change to which render passes count as drawn frame after frame (`runtime/vk_target_frames.h`: a draw into one is
|
|
skipped while its pipeline builds, a draw into any other waits) passes `python tests\target_frames_check.py`, which
|
|
builds the header alone and replays an area change after a loading screen from a player's log: the scene, probe and
|
|
half-resolution passes are per-frame again at once, a lookup table, a volume's slices and an icon capture still wait,
|
|
and an image taken over by another texture starts without the history of the one before. No GPU, no game files.
|
|
|
|
A change to how the holes in the allocator's blocks are counted or to which blocks a texture compaction empties
|
|
(`runtime/vk_tex_compact.h`: `tcp_tally`, `tcp_stats`, `tcp_plan`) passes `python tests\tex_compact_check.py`, which
|
|
builds the header alone: a block with one allocation that stays is never chosen, the block that takes the fewest moved
|
|
bytes goes first, what moves fits the free bytes outside the chosen blocks, a plan under its smallest gain is no plan,
|
|
and 400 random tables keep those rules. The part of `runtime/vk_texture.c` that builds the table and moves the images
|
|
(`NX2NAVI_TEX_COMPACT`) is not built by it. No GPU, no game files.
|
|
|
|
A change to which entry of the upscaler stand-in's scratch images serves a new size (`runtime/vk_scratch_lru.h`,
|
|
`runtime/vk_exec_draw.c` `fallback_scratch`) passes `python tests\scratch_lru_check.py`, which builds the header alone
|
|
and checks that a match is a hit, a ninth size takes a free entry or the one used longest ago instead of being
|
|
refused, and two images of one frame never evict each other. No GPU, no game files.
|
|
|
|
A change to dynamic resolution's target or step hold (`runtime/dynres.c`) passes `python tests\dynres_check.py`, which
|
|
builds that file with stand-ins for the settings, the GPU's busy time, the clock and the game's rate under the limit:
|
|
the default target is the frame time of 60, 144 and 240 fps and 16.7 ms for an unknown or unlimited rate, a target
|
|
that is set wins, a limit or target changed while it runs (in the menu too) moves the target with one log line, a
|
|
refresh rate wandering under 1 % after the start line logs none, and each step is held for `dyn_res_step` 60ths of a
|
|
second at 60, 144 and 240 fps, down while the GPU is over the target and back up once it is under three quarters of
|
|
it. No GPU, no game files.
|
|
|
|
A change to the ASTC decoder's scratch trim (`runtime/vk_astc.c` `vk_astc_scratch_trim`, with the upload lock it only
|
|
tries, `runtime/vk_memory.c` `vk_defer_buffer_try`) passes `python tests\astc_trim_check.py`, which builds `vk_astc.c`
|
|
with the platform layer's recursive lock and the rest of the runtime stubbed: the trim gives way while another thread
|
|
holds the upload lock, goes through once it is free, and never deadlocks with an upload that asks for the scratch
|
|
meanwhile. No GPU, no game files.
|
|
|
|
A change to the copy of each event-ordered queue's first held wait (`runtime/vk_evo_peek.h`, `runtime/vk_exec.c`
|
|
`vk_exec_held_ready`) passes `python tests\evo_peek_check.py`, which builds the header alone: one writer republishes
|
|
while three readers copy, no copy mixes two publishes, and no reader waits for the writer. No GPU, no game files.
|
|
|
|
A change to the present that waits for held work (`runtime/vk_present_held.h`, `runtime/vk_backend.c` `submit_sync`,
|
|
`vk_be_present`, `ph_out`) passes `python tests\present_held_check.py`, which builds the header with the platform layer:
|
|
one presenting and three submitting threads over 200,000 frames, every frame out once, in order and never two at once,
|
|
each reason a held present goes out seen; step by step what keeps and what puts out a held present; and with the lock
|
|
held elsewhere a submitting thread gives up at once while the presenting thread waits and counts its wait. No GPU, no
|
|
game files.
|
|
|
|
A change to where a frame's bookkeeping at its present runs (`runtime/vk_frame_owed.h`, `runtime/vk_backend.c`
|
|
`present_image`, `present_frame_closed` and `frame_end`, `runtime/vk_sync.c` `submit`, `vk_sync_try_locked` and
|
|
`vk_sync_locked`) passes `python tests\frame_owed_check.py`, which builds the header with the platform layer: one
|
|
presenting thread, three replaying threads, some replays holding the lock 40 ms, and a present thread that blits each
|
|
frame handed to it, over 20,000 frames; every frame's bookkeeping runs once, in order, never during a replay, before
|
|
any replay that took the lock after its present and before the frame's blit, and no present waits for a replay. No
|
|
GPU, no game files.
|
|
|
|
A change to a synchronous submission's deferred binds (`runtime/nvn_submit_guard.h`, `runtime/nvn_api_cmd.c`
|
|
`nvn_submit_prepare`, `nvn_submit_done`, `resolve_deferred`) passes `python tests\submit_clone_check.py`, which builds the
|
|
header with the platform layer: four threads submit two recording copies, a quarter of the calls giving theirs twice,
|
|
while another rewrites the arrays their deferred binds read, and no replay sees its handles change while the in-place,
|
|
the given-again and the clone path run; then step by step a copy given twice in one call (in place both times, cloned
|
|
both times when another submission has it), and the bind resolution (changed and unchanged arrays, binds it leaves
|
|
alone, calls 16 deep and not 17, a corrupt record). No GPU, no game files.
|
|
|
|
A change to how `runtime/memory.c` `mem_copy_readable` reads the lazy heap (`NX2NAVI_HEAP_LAZY=1`) passes
|
|
`python tests\heap_lazy_copy_check.py`, which builds `memory.c` with the Windows platform layer and its fault handler:
|
|
every copy returns, a heap page nothing touched yet reads zero and stays reserved, and pages past the heap's end are
|
|
left out. Windows only, no game files.
|
|
|
|
A change to the page write generations (`runtime/vk_memory.c`, from its "page write generations" banner to its "dirty
|
|
census" banner, and the readers of its tables `cpu_wrote_since`, `shadow_pool_clean`, `lazy_clean`, `page_dirty`)
|
|
passes `python tests\gen_wrap_check.py`, which builds those parts of the file alone and runs the generations across
|
|
2^32 (where 32-bit generations wrapped), 2^40 and 2^63: texture checks, the snapshot path and the region check, the
|
|
CPU-side tables, and mirror and shadow cleanliness answer as below the boundary, and one line is logged when the count
|
|
reaches 2^32. It also takes the start from `NX2NAVI_GEN_SEED` (`gen_seed`): read as 64 bits, decimal or `0x`, capped
|
|
at 2^62, a value that is no number ignored, each with its log line. No GPU, no game files.
|
|
|
|
A change to `runtime/vk_memory.c`, `runtime/vk_texture.c` or `runtime/tex_probe.c` passes
|
|
`python tests\gen_width_check.py`, which compiles the three files with the build's flags and `-Wshorten-64-to-32`,
|
|
`-Wformat` and `-Wincompatible-pointer-types` as errors: the page write generations are 64 bits, and a local, field or
|
|
format left at 32 bits would bring their wrap back without a word. Any narrowing in these files takes an explicit cast.
|
|
It needs the shader headers (`tools/build_shaders.py`); no GPU, no game files.
|
|
|
|
A change to which frames FSR 3.1 frame generation is told are camera cuts (`runtime/vk_fg_reset.h`, `runtime/vk_fg.cpp`
|
|
`generate` and `vk_fg_frame`) passes `python tests\fg_reset_check.py`, which builds the header alone and checks that each
|
|
frame id gets its own frame's cut, that an id never noted or no longer held counts as a cut, and that a slot rewritten
|
|
on another thread never answers an old id with a newer frame's value. No GPU, no game files.
|
|
|
|
A change to who holds the frame limit with DLSS Frame Generation (`runtime/dlssg_pace.h`, `runtime/vk_dlssg.c`
|
|
`vk_dlssg_pace`) passes `python tests\dlssg_pace_check.py`, which builds the header alone and checks that Reflex holds
|
|
the limit only once Streamline took exactly that limit, that the game's own pacer keeps it during a movie, before the
|
|
first host present, after 200 ms without one and with the setting off, that frames coming too soon over a window of 60
|
|
give it to the pacer for the rest of the run, and that frames the multiplier periods apart switch Reflex to the frames
|
|
on screen once and to the pacer the second time. No GPU, no game files.
|
|
|
|
A change to `runtime/dlssg/nx2_sl.h` or to the export check in `tools/build_dlssg.py` (`declared_exports`,
|
|
`pe_exports`) passes `python tests\dlssg_exports_check.py`, which builds a small DLL with MSYS2's clang and checks that
|
|
the export it lacks is the one named; that `build_dlssg.check` stops on that DLL, a missing file and a file that is no
|
|
DLL, and passes a DLL with every export `nx2_sl.h` declares; that every function `nx2_sl.h` declares is read; and that
|
|
`tools/make_launcher.py` and `tools/make_kit.py` make that check of their `--dlssg` DLL before they download anything.
|
|
The DLL itself is checked when `tools/build_dlssg.py` builds it, or with `--check <dll>`. No MSVC, no Streamline SDK,
|
|
no game files.
|
|
A change to the list of what the upscalers let go of while the GPU may still use it (`runtime/vk_up_defer.h`) passes
|
|
`python tests\up_defer_check.py`, which builds the header alone, as C and as C++, and checks that an entry goes only
|
|
once the submission that last named it is done and its present gate is passed, that the ready ones go in the order
|
|
they were put and the rest keep theirs, that a full list refuses without overwriting, and that an entry a release puts
|
|
waits for the next take. No GPU, no game files.
|
|
|
|
A change to where the upscalers free what they let go of (`runtime/vk_dlaa.c` "what the upscalers let go of while the
|
|
GPU may still use it", `runtime/vk_ffx.cpp` and `runtime/vk_ffx4.cpp` `context_for`) passes
|
|
`python tests\upscaler_idle_check.py`, which reads the sources: `vkDeviceWaitIdle` only in `vk_upscaler_device_idle`
|
|
and the `--probe-ffx-cache` path, that function only behind the rollback switch, a full list and a refused creation,
|
|
every place that lets something go calling `vk_upscaler_let_go`, every creation of a DLSS feature or an FSR context
|
|
(`vk_dlaa.c` `evaluate`, `vk_ffx.cpp` and `vk_ffx4.cpp` `context_for`) trying a refusal once more after a wait for the
|
|
GPU to pass what was let go of (`vk_upscaler_gpu_wait`), no place that lets something go (those, the `retire_*`
|
|
helpers, the FSR trims and `release_idle` calls) calling a destroy that frees at once (`destroy_size`, `destroy_snap`,
|
|
`hl_destroy`, `pb_destroy`, `feat_release`, `N.release`, `vkDestroyImageView`, `vmaDestroyImage`, the FSR files'
|
|
`A.destroy`), except after the first `image2d` of `hudless_from`, `pb_images` and `snap_for` for their half-made
|
|
images, a whole list taken (`upd_take_all`) only at teardown or after `vk_upscaler_settle` for a full list,
|
|
`upscaler_idle_wait`'s `sections[]` entry and `nx2navi.ini.example` line, a name in `perf.c` for each perf mark kind,
|
|
and `pb_images` noting the images' last use after it makes them. Any Python, no game files.
|
|
|
|
A change to which frames the flight recorder takes for a one-frame flash (`runtime/flight_flash.h`, `runtime/flight.c`
|
|
`flight_cells`) passes `python tests\flight_flash_check.py`, which builds the header alone, draws pictures of the
|
|
thumbnails' size and checks that a patch or a whole frame shown once is a flash, that a camera cut, a fade, a pan and
|
|
shapes crossing the picture at any speed are none, where each of the test's rules decides, and that a frame beside one
|
|
without a thumbnail is not judged. With `time` it prints what the cell means of one thumbnail cost. No GPU, no game
|
|
files.
|
|
|
|
A change to the present side's log lines (`runtime/vk_backend.c` `present_main` and `present_window`, the window line
|
|
of `runtime/perf.c` `perf_present`) or to how `tools/perf_summary.py` reads them passes
|
|
`python tests\perf_summary_check.py`. It checks that the format strings it copies are still the runtime's, makes lines
|
|
with them and checks the present percentiles, the present thread's averages and the hand-off share the tool prints,
|
|
and that the lines of older logs, without those fields, still parse. Any Python, no game files.
|
|
|
|
A change to what the frame-time summary charges to an address-space walk of `NX2NAVI_MEM_CENSUS`
|
|
(`runtime/perf_census.h` `census_charge`, `runtime/perf.c` `ft_frame`) passes `python tests\census_charge_check.py`,
|
|
which builds the header alone: a walk before, after or around a frame, one still running, a torn read of the last walk's
|
|
start (taken as no walk), the charge taken from the replay part first, capped by the overlap and the parts, and 100,000
|
|
random frames in which the census part never passes the frame and the parts keep their sum. No GPU, no game files.
|
|
|
|
A change to how the on/off switches read their value (`runtime/perf.c` `rt_knob`) passes `python tests\rt_knob_check.py`,
|
|
which cuts the function out of the file with its warning table and builds it with stand-ins for the settings and the
|
|
log: 0, 1 and ab read as before without a line, false, off and no in any case read as 0 and true, on and yes as 1
|
|
without a line, and any other value reads as 1 and is logged once per switch. No game files.
|
|
|
|
A change to how nx2navi.ini and the environment are read (`runtime/cfg.c` `cfg_load`, `cut_comment`, `switch_word`,
|
|
`cfg_bool`, `check_value`) passes `python tests\cfg_parse_check.py`, which builds `cfg.c` with stand-ins for what it
|
|
links with and loads scratch files: a comment after a value of a setting with a set of values is cut and logged once
|
|
with its line, free text, a value in double quotes and the environment keep theirs; switches written as words read as
|
|
0 or 1 through every reader, the low-end mode's switch (`lowend_of`, read apart from `cfg_get`) included; values
|
|
outside their sets are logged with their line, at most 16 lines and a count; a full settings table logs one line at
|
|
load and one for the menu. The launcher's reader (`launcher/gameini.py`) has the same comment lines in
|
|
`tests\launcher_test.py` `PlayerIniTest`, and `tools/make_game.py`'s own reads of `shader_carry` and `low_end_gpu`
|
|
(`shader_carry`, `guest_opt_ini`) theirs in `MakeGameTest`. No game files.
|
|
|
|
The lifter (`recomp/`) and the CPU headers (`runtime/cpu.h`, `runtime/simd.h`) are checked against Unicorn. Every lifter
|
|
change passes these, all run with `.venv`:
|
|
|
|
| Test | Checks | Game files |
|
|
|---|---|---|
|
|
| `tests\diff\lift.py` | instructions sampled from the extracted modules; `--coverage` lists every word the lifter rejects or traps | yes |
|
|
| `tests\diff\simd.py` | the whole FP/SIMD encoding space | no |
|
|
| `tests\diff\atomic.py` | exclusive load/store sequences and ARMv8.1 atomics | no |
|
|
| `tests\diff\func.py` | whole functions as the generator emits them, loops, jump tables and calls included (calls go to a stub that checks the registers at the call) | yes |
|
|
| `tests\flags_check.py` | the function-local flag helpers of `runtime\cpu.h` against its packed ones | no |
|
|
| `tests\ds3_check.py` | the DualShock 3 report decoding and output report of `runtime\input_ds3.c` | no |
|
|
| `tests\gpa_check.py` | the device address table of `runtime\vk_gpa.c` as a shader reads it: mapping, mirrors and twins, overlapping imports, and blocks given back and taken again while pool addresses climb (no GPU; any Python) | no |
|
|
| `tests\excl_mt.py` | the exclusive monitor helpers under contention, a changed value, and the change-and-restore case the compare-and-swap monitor does not see, on several threads | no |
|
|
| `tests\jumptables_check.py` | `recomp/jumptables.py` against a Unicorn oracle; must report 0 disagreements, on 1.4.3 too | yes |
|
|
|
|
A `--cpu v2` build passes the same differential tests with `--cflags=-march=x86-64-v2`; they compile `runtime/fma.c`
|
|
too, so they check the software fma the v2 build links rather than the C runtime's.
|
|
|
|
On NVIDIA Linux, `NX2NAVI_DLAA=1 ./ocarina --probe-vkdevice` logs NGX initialization and shutdown when available.
|
|
|
|
Runtime probes, each a mode of `nx2navi.exe` (the list is at the top of `runtime/main.c`): `--probe-kernel`,
|
|
`--probe-vkexec`, `--probe-nvn-tex` and `--probe-shader` cover most runtime changes. `--probe-shader` and the draw
|
|
part of `--probe-vkexec` translate shaders of the census below: they look for `build/<ver>/shaders/jobs.json` of the
|
|
exe's own version first, then of the other one.
|
|
`--probe-vkmem` also checks CPU-backed texture uploads and rewrites, GPU-written texture aliases, and memory write watches.
|
|
`--probe-meshfb` (no game files) draws mesh programs of the runtime's own, with and without a task stage and with
|
|
direct and GPU-written workgroup counts, through `VK_EXT_mesh_shader` and through the mesh fallback, and compares the
|
|
pictures; on a device without mesh shaders only the fallback draws. A change to `runtime/mesh_fb.c`,
|
|
`runtime/vk_mesh_fb.c` or the fallback's part of `runtime/vk_exec_draw.c` passes it. With
|
|
`NX2NAVI_PROBE_MESHFB_DIR=<folder>` it converts the game's own task and mesh translations instead, for `spirv-val`
|
|
(the top of `runtime/mesh_fb_probe.c` says what the folder holds).
|
|
A change to the texture detail (`runtime/vk_texture.c` "texture detail": `detail_skip`, `upload`, `astc_bl_regions`)
|
|
passes it three times, with `NX2NAVI_TEXTURE_DETAIL` set to `full`, `reduced` and `low`: each time ASTC textures are
|
|
uploaded and rewritten, and every level and layer of their images must hold the game's level it stands for.
|
|
`--probe-prewarm` runs the boot's pipeline stages without the game, from the caches next to the exe (it skips when
|
|
there are none): the pipeline cache load, the shader prewarm, the recorded pipelines and the compute pipeline prebuild,
|
|
with a `mem: boot stage` log line after each that gives the device memory and the process commit so far. It then asks
|
|
for compute pipelines the boot did not keep, as their first dispatch would, and prints what one costs. The device
|
|
memory after the compute stage should be what it was before the stage, apart from the kept pipelines. A change to what
|
|
the boot builds or keeps (`runtime/vk_pipeline.c`: `prewarm_run`, `lb_compute`, the `compute-used.bin` record) passes it.
|
|
|
|
Shaders: `.venv-py314\Scripts\python tools\shader_census.py --version <ver> --translate` extracts every Maxwell shader
|
|
to `build\<ver>\shaders\` and runs it through the translator and `spirv-val`; failure groups go to `failures.txt`.
|
|
`tools\xlate_smoke.py` checks the translator alone, without game files.
|
|
|
|
A change to the SM86 tensor lowering (`tools/xlate/AmpereLowering.cs` Hmma, Ldsm, Movm, Half) passes
|
|
`python tests\tensor_lowering_check.py`: it lowers GPU-captured vectors with `nx2navi_xlate --lower` and runs them on
|
|
`tests\maxwell_warp_emu.py`, a model of what the shader translator makes of Maxwell code (the docstring names the
|
|
vector file and what is checked). `tests\maxwell_warp_emu_selftest.py` checks the emulator itself. No game files.
|
|
|
|
A change to how the packaging scripts check the translator they ship (`tools/build_xlate.py` `source_identity`,
|
|
`check_built`; `make_launcher.py`, `make_release.py`, `make_appimage.py`, `make_kit.py`; `tools/xlate_smoke.py`
|
|
`pool_check`) passes `python tests\xlate_package_check.py`: a folder whose `nx2navi_xlate.id` names this checkout's
|
|
sources passes, a missing file, another identity and a folder built before a patch changed are refused,
|
|
`build_xlate.py` writes no `.id` without a build, each script stops before its slow steps, and on Windows
|
|
`make_launcher.py` also refuses an exe that does not run beside a matching `.id`. It needs git on PATH; no game files.
|
|
`tools\xlate_smoke.py --exe <nx2navi_xlate.exe>` checks a translator's texture pool length against
|
|
`tools/xlate/Program.cs` `TexturePoolLength`: one older than its sources fails it.
|
|
|
|
## Reverse engineering (optional)
|
|
|
|
- Ghidra 12.1.2 with the Ghidra-Switch-Loader extension in `toolchain\ghidra\` (JDK 21) opens the game modules (NSO).
|