From 32901bb337b183d2a40c3ddc4b2e3b8568e98615 Mon Sep 17 00:00:00 2001 From: chrg6 Date: Tue, 10 Nov 2026 06:16:58 -0500 Subject: [PATCH 01/18] Build the runtime's shader headers before make_game.py compiles, so a source checkout builds again. tools/make_game.py has its own compile step and did not build runtime/*_spv.h, which the repository no longer holds: from a source checkout (the Linux path of docs/README-BuildKit.md) every runtime file that includes one failed to compile. A kit carries the headers built, so nothing is stale there and no shader compiler is looked for. Pull request #405 by mooki000, taken as it is. Not run on Linux here. Co-Authored-By: Claude Fable 5.2 --- docs/README-BuildKit.md | 4 +++- tools/make_game.py | 8 ++++++++ 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/README-BuildKit.md b/docs/README-BuildKit.md index cb48cdb9..bf37ea75 100644 --- a/docs/README-BuildKit.md +++ b/docs/README-BuildKit.md @@ -80,7 +80,9 @@ RADV). It takes clang, clang++ and lld from `PATH` and links against the system' pinned 3.4.0 into `toolchain/vma/` and checks its hash. 2. The shader translator: a kit with `bin/nx2navi_xlate` has it; from a source checkout, build it once with - `python tools/build_xlate.py` (needs the .NET 10 SDK). The build copies it beside the game. + `python tools/build_xlate.py` (needs the .NET 10 SDK). The build copies it beside the game. A kit also carries the + runtime's own shaders compiled; from a source checkout the build compiles them first, with `glslc`, `spirv-as` and + `spirv-val` (on Arch the packages `shaderc` and `spirv-tools`). 3. Build, then play: diff --git a/tools/make_game.py b/tools/make_game.py index 65ed86be..deaa1c69 100644 --- a/tools/make_game.py +++ b/tools/make_game.py @@ -1199,6 +1199,14 @@ def step_compile(out, cache_dir, msys, jobs, romfs_job=None, update=False): gen.mkdir(parents=True, exist_ok=True) obj.mkdir(parents=True, exist_ok=True) rt = KIT / 'runtime' + # The runtime's own shaders are in the repository as source only: from a source checkout their headers are not + # there until they are built, and every runtime file that includes one would fail to compile. A kit carries them + # built, so nothing is stale there and no shader compiler is looked for. + import build_shaders + stale = build_shaders.stale(rt) + if stale: + say(f' compiling {len(stale)} of the runtime\'s own shaders (runtime/*_spv.h)') + build_shaders.build(rt, stale) clang, clangxx, env, vma = toolchain(msys) if not romfs_job: romfs_index(out, gen) From c117cd8999a0ec65ec605f5ee3892085d8e45939 Mon Sep 17 00:00:00 2001 From: chrg6 Date: Tue, 10 Nov 2026 06:17:13 -0500 Subject: [PATCH 02/18] Set NVIDIA's shader disk cache on Linux only with NVIDIA's driver, in a folder that could be made and written, before any thread of the game starts. The first part of this pull request (the cache beside the exe with an 8 GiB limit) was already in. This is the rest: nothing is set without NVIDIA's kernel driver, neither variable is set when the folder cannot be made or written (the driver caches nothing in such a folder), the log says which folder and limit hold, and a game run sets the drivers' cache variables in main before any of its threads can load a Vulkan driver (vk_device.c vk_driver_cache_env); the other modes still set them in create_instance. On Windows that only moves the MESA_SHADER_CACHE_DIR setenv earlier. Pull request #369 by mooki000, taken as it is. The Linux part is not compiled or run here. Co-Authored-By: Claude Fable 5.2 --- docs/README-BuildKit.md | 8 ++++--- docs/building.md | 5 ++++- docs/installer.md | 7 +++--- runtime/host.h | 3 +++ runtime/main.c | 1 + runtime/vk_device.c | 50 ++++++++++++++++++++++++++++++++--------- 6 files changed, 57 insertions(+), 17 deletions(-) diff --git a/docs/README-BuildKit.md b/docs/README-BuildKit.md index bf37ea75..c5330081 100644 --- a/docs/README-BuildKit.md +++ b/docs/README-BuildKit.md @@ -99,9 +99,11 @@ On NVIDIA, start the game with `NX2NAVI_SHADOW=0` if the 3D world stays black wh profile, logs and settings are in the `--out` folder as on Windows. On NVIDIA the driver keeps the shaders it compiled in `nvidia-cache` beside `ocarina`, not in `~/.cache/nvidia`, where -one run adds at most 1 GiB while the preparation at the end of the build makes about 1.5 GB. The game sets the folder -and a limit of 8 GiB itself (`__GL_SHADER_DISK_CACHE_PATH` and `__GL_SHADER_DISK_CACHE_SIZE` in the environment take -precedence). Deleting the folder costs one slow start; `./ocarina --warm-shaders` fills it again. +one run adds at most 1 GiB while the preparation at the end of the build makes about 1.5 GB. The game makes and sets +the folder unless the environment sets `__GL_SHADER_DISK_CACHE_PATH`, and a limit of 8 GiB unless it sets +`__GL_SHADER_DISK_CACHE_SIZE`. When it cannot make or write the folder, it sets neither and the driver keeps its own +folder and limit; the log says which (`vk: NVIDIA's shader disk cache`). Deleting the folder costs one slow start; +`./ocarina --warm-shaders` fills it again. `tools/make_appimage.py` packs this kit into one `NX2navi-Linux--x86_64.AppImage` with its own Python, for players without a source checkout: it checks for the packages above, installs zbic with pip on its first run, picks diff --git a/docs/building.md b/docs/building.md index 3b561114..47243cf7 100644 --- a/docs/building.md +++ b/docs/building.md @@ -235,7 +235,10 @@ NVIDIA's Linux driver compiles into a disk cache of its own. A process adds at m folder, `~/.cache/nvidia`, is shared with every program. `--prepare-shaders` of game 1.1.1 builds 42,142 variants, about 1.5 GB: on the default limit about 30,700 of them were stored and the rest compiled again at their first draw in play. `runtime/vk_device.c` therefore points `__GL_SHADER_DISK_CACHE_PATH` at `nvidia-cache` beside `ocarina` and the size at -8 GiB before the first Vulkan call, each unless the environment sets it, as it does `MESA_SHADER_CACHE_DIR` for RADV. The +8 GiB, each unless the environment sets it, as it does `MESA_SHADER_CACHE_DIR` for RADV, and only while NVIDIA's kernel +driver is loaded (`/proc/driver/nvidia/version`). The driver does not make that folder and caches nothing in a folder it +cannot write, so the game makes it, and when it cannot make or write it sets neither variable. A game run sets them in +`main` before any of its threads starts, since a thread can load the Vulkan driver before the instance is made. The folder can be deleted; `ocarina --warm-shaders` fills it again. Memory: on Windows the GPU imports guest memory and `GetWriteWatch` tracks the CPU's writes to it. On Linux, diff --git a/docs/installer.md b/docs/installer.md index ae1d5cc3..7587c906 100644 --- a/docs/installer.md +++ b/docs/installer.md @@ -202,9 +202,10 @@ bin/ nx2navi_xlate, hactool, ninja, libSDL3.so.0 - The setup's temp folder is `.setup-temp` inside the install folder (`--temp` changes it): the AppImage is mounted read-only, and a Steam Deck's `/home` is often too full for the generated C while the game goes to the SD card. - The game keeps Mesa's shader cache in `mesa-cache` beside `nx2navi` unless `MESA_SHADER_CACHE_DIR` is set, for the - same reason: RADV fills its default folder in `~/.cache` up to 1 GB. NVIDIA's driver cache goes to `nvidia-cache` - beside it with a limit of 8 GiB unless `__GL_SHADER_DISK_CACHE_PATH` or `__GL_SHADER_DISK_CACHE_SIZE` is set: the - default limit of 1 GiB is smaller than what the shader preparation builds. + same reason: RADV fills its default folder in `~/.cache` up to 1 GB. With NVIDIA's driver, its cache goes to + `nvidia-cache` beside it unless `__GL_SHADER_DISK_CACHE_PATH` is set, with a limit of 8 GiB unless + `__GL_SHADER_DISK_CACHE_SIZE` is set: the default limit of 1 GiB is smaller than what the shader preparation builds. + When that folder cannot be made or written, the driver keeps its own folder and limit. - The window follows `LANG` for English or Russian, cancels through SIGTERM to the CLI's process group (the CLI handles it as Ctrl+Break on Windows) and opens the finished folder with `xdg-open`. - Requirements beyond Windows': Linux 6.7 or later (the write watch, `docs/building.md`) and glibc 2.34. diff --git a/runtime/host.h b/runtime/host.h index 81effc41..85155251 100644 --- a/runtime/host.h +++ b/runtime/host.h @@ -389,4 +389,7 @@ int probe_movie(void); // movie_probe.c: hos int vk_pipeline_prepare_shaders(void); // vk_pipeline.c: --prepare-shaders (tools/prepare_shaders.py) int vk_pipeline_warm_shaders(void); // vk_pipeline.c: --warm-shaders (tools/build_game.py) int vk_pipeline_probe_prewarm(void); // vk_pipeline.c: --probe-prewarm +// vk_device.c: the drivers' own shader caches beside the exe (MESA_SHADER_CACHE_DIR, on NVIDIA's Linux driver +// __GL_SHADER_DISK_CACHE_PATH and _SIZE) in the environment, once; before any thread can load a Vulkan driver +void vk_driver_cache_env(void); int vk_pipeline_spirv_selftest(void); // vk_pipeline.c: --selftest, the SPIR-V rewrites without a device diff --git a/runtime/main.c b/runtime/main.c index 9f1e6d1d..db0a5e98 100644 --- a/runtime/main.c +++ b/runtime/main.c @@ -611,6 +611,7 @@ int main(int argc, char **argv) { log_open(!mode); if (!mode) { cfg.boot = cfg.seconds > 0; // --seconds starts the game like --boot, with its checkpoints + vk_driver_cache_env(); // before the threads below and rt_boot's, any of which may load the Vulkan driver discord_init(); // the Rich Presence thread (discord_rpc.c); it reads nx2navi.ini itself and idles while off rt_boot(); return 0; diff --git a/runtime/vk_device.c b/runtime/vk_device.c index 55a2cff9..9167b87f 100644 --- a/runtime/vk_device.c +++ b/runtime/vk_device.c @@ -4,6 +4,10 @@ #include #include #include +#ifdef __linux__ +#include +#include +#endif #include "cfg.h" #include "host.h" @@ -79,30 +83,56 @@ static void mesa_cache_beside_exe(void) { // variants of game 1.1.1 into it in one process: with the default limit it stored about 30,700 of them and the rest // were compiled again by the first start after it. So unless the environment sets them, the cache goes to nvidia-cache // beside the exe, which nothing else writes, with a limit of NV_CACHE_BYTES. The driver makes GLCache inside that -// folder but not the folder, so it is made here. +// folder but not the folder, so it is made here; pointed at a folder that is missing or not writable it caches nothing +// at all, so when that folder cannot be made or written neither variable is set and the driver keeps its own folder +// and limit. Without NVIDIA's kernel driver (no /proc/driver/nvidia/version, the test play.sh makes) the variables do +// nothing and the folder would stay empty, so nothing is done. #define NV_CACHE_BYTES "8589934592" // 8 GiB static void nvidia_cache_beside_exe(void) { + if (plat_fs_kind("/proc/driver/nvidia/version") != PLAT_FS_FILE) return; char dir[PLAT_PATH_MAX * 2], size[32]; - if (!plat_getenv("__GL_SHADER_DISK_CACHE_PATH", dir, sizeof dir)) { + int own_dir = !plat_getenv("__GL_SHADER_DISK_CACHE_PATH", dir, sizeof dir); + if (own_dir) { int len = plat_exe_path(dir, sizeof dir - 16); char *cut = len ? plat_last_sep(dir) : NULL; - if (cut) { - strcpy(cut + 1, "nvidia-cache"); - plat_fs_mkdir(dir); // already there is fine - plat_setenv("__GL_SHADER_DISK_CACHE_PATH", dir); + if (!cut) return; + strcpy(cut + 1, "nvidia-cache"); + int made = plat_fs_mkdir(dir), there = made == PLAT_FS_OK || made == PLAT_FS_EXISTS; + int err = !there ? plat_fs_os_error() : access(dir, W_OK | X_OK) ? errno : 0; + if (err) { + rt_log("vk: NVIDIA's shader disk cache stays in the driver's own folder with its own limit: %s %s (%s)", dir, + there ? "is not writable" : "cannot be made", strerror(err)); + return; } + plat_setenv("__GL_SHADER_DISK_CACHE_PATH", dir); } - if (!plat_getenv("__GL_SHADER_DISK_CACHE_SIZE", size, sizeof size)) - plat_setenv("__GL_SHADER_DISK_CACHE_SIZE", NV_CACHE_BYTES); + int own_size = !plat_getenv("__GL_SHADER_DISK_CACHE_SIZE", size, sizeof size); + if (own_size) plat_setenv("__GL_SHADER_DISK_CACHE_SIZE", NV_CACHE_BYTES); + rt_log("vk: NVIDIA's shader disk cache in %s (%s), up to %s bytes (%s)", dir, + own_dir ? "beside the exe" : "__GL_SHADER_DISK_CACHE_PATH", own_size ? NV_CACHE_BYTES : size, + own_size ? "the game's limit" : "__GL_SHADER_DISK_CACHE_SIZE"); } #endif -static int create_instance(void) { - vk_dlssg_early(); // DLSS Frame Generation: Streamline before any Vulkan call of the process (its slInit asks so) +// A driver may read these variables when it loads, and a thread can load the Vulkan driver before create_instance: +// widescreen.c's wide_init asks for the monitor's size (fullscreen with a monitor chosen and the picture at the +// window's size), and SDL's load of the Vulkan library there lists the instance extensions, which loads the drivers, +// while the prewarm thread may be in create_instance; setenv beside other threads' getenv is not safe either. main +// therefore calls this for a game run before any thread of the game starts; create_instance calls it for the other +// modes. Both calls happen before other threads exist or under vk_init's lock, so the flag needs no atomics. +void vk_driver_cache_env(void) { + static int done; + if (done) return; + done = 1; mesa_cache_beside_exe(); #ifdef __linux__ nvidia_cache_beside_exe(); #endif +} + +static int create_instance(void) { + vk_driver_cache_env(); + vk_dlssg_early(); // DLSS Frame Generation: Streamline before any Vulkan call of the process (its slInit asks so) const char *ext[12]; uint32_t n_ext = (uint32_t)gfx_window_vk_extensions(ext, 7); const char *layers[] = {"VK_LAYER_KHRONOS_validation"}; From 19eeb29cdab7294d00289b743dd30a0d1d4bb583 Mon Sep 17 00:00:00 2001 From: chrg6 Date: Tue, 10 Nov 2026 06:17:42 -0500 Subject: [PATCH 03/18] Let tools/run_game.py run a Linux game folder and a save from a crash report. The scripted-run tool finds the game under each name the build tools give it, counts other copies through /proc where there is no tasklist, uses the game folder's own profile when it has one, takes a save folder or zip from elsewhere through the launcher's save search (a single slot replaces only that slot), and --wall ends a run that never reaches the world clock. Pull request #310 by mooki000, taken as it is except for its note that game time is not counted on 1.1.1, which no longer holds. --help runs on Windows; nothing was run on Linux here. Co-Authored-By: Claude Fable 5.2 --- docs/controls.md | 2 + tools/run_game.py | 127 +++++++++++++++++++++++++++++++++++----------- 2 files changed, 99 insertions(+), 30 deletions(-) diff --git a/docs/controls.md b/docs/controls.md index add8c150..8fd82b9a 100644 --- a/docs/controls.md +++ b/docs/controls.md @@ -148,6 +148,8 @@ The PS button is not used. The Controls tab binds and remaps its buttons like an `NX2NAVI_INPUT_RECORD=` (`input_record` in `nx2navi.ini`) writes the merged input in the same format, one line per button press and per stick segment. The file replays as a script, which makes it the repro to attach to a bug report. +`tools/run_game.py --save --seconds --wall [--script ] [--shot ]` runs the game on a copy of its profile with a save swapped in, under the script (default `tools/input/load_save.txt`: title screen to the first save slot). `` names one of the game's save backups (`profile/save-backup/