The prepared list named its shader variants and nothing about where they came from, so an update that keeps the prepared shaders (shader_carry) kept the old list whatever seed the new kit writes: variants only the new seed names, from a newer pipeline list or a change to which variants are prepared, were translated when play first drew them. The existing identities do not cover this: nx2navi_xlate.id and identities.txt name translators, which decide cache keys and what is carried over, not which variants are listed.
The list is now version 2 (runtime/prp_file.h): after the head come the sha256 of seed-programs.bin, the sha256 of seed-answers.bin and the translator identity, written by gfx_shader_prepare_seed; the three readers go through gfx_shader_prepared_read and still read a version 1 list. At an update step_shaders writes the seed first (prepare_shaders.py --no-run, about 4 s) and keeps the list only when its digests are that seed's (make_game.py seed_changed); another seed, or a list that does not say, is prepared again. Another translator alone still carries over: input ids do not depend on the translator, and preparing everything again for it is what carrying was made to avoid.
Checked with tests/prp_file_check.py (the C layout against make_game's reader), three launcher tests, and the exe's --prepare-shaders on a seed without records in a folder of its own: the digests it wrote are those of the seed files, one byte more in the seed reads as changed, and after preparing again it does not. A version 1 list of 42,142 variants and the same list as version 2 are both read whole by --probe-prewarm.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Each player keeps their own game and save. One player makes a room, the others join it with its secret, and story flags and, by the room's rules, items, songs, quest items, small keys (a shared pool), rupees and ammunition reach every save. The layer is a host thread with one UDP socket to a relay the players run themselves (tools/coop_relay.py, Python's standard library only); the messages between the players are sealed with a key from the room's secret (XChaCha20-Poly1305, Monocypher 4.0.2 vendored with its licence and a row in THIRD_PARTY_NOTICES.md), every change is signed by the player who made it, a player accepts a room by its members' keys and the key of its rules, the save is copied to coop-backup before anything is applied, and values are set through the game's own routines between frames. The Co-op tab makes, joins, accepts, removes and leaves, and the menu takes typed text for the name, the relay and the secret. This is the contributor's pull request #518 on this line.
Nothing reaches the network while co-op is off: the socket is opened, and the relay's name looked up, only after co-op is on, a relay is set and a room is made or joined. There is no default relay.
Left out of the pull request: the other players drawn in the world (the "puppets") with the copied engine they need. They were unfinished, added two patch sites to the game's per-frame update that could not be put through the differential tests here, and their file helpers clashed with the ones this line already has. So no position is sent: the POS message's codec stays, nothing sends it.
Changed against the pull request:
- The menu's test hook drives the Co-op tab only when it comes from the environment. From nx2navi.ini it could turn co-op on, join a room and accept its offer.
- The log's label for a room is hashed from the secret instead of being its first three characters.
- A game that has not saved yet has no save to copy, and takes nothing from the room until it has saved once; that used to count as a copy made.
- A PING's token no longer carries the PC's uptime.
- On Windows the maker's key file and its folder get an access list with the player's account alone.
- The tests that link the platform layer get the socket library, and the save test a stand-in for the title screen's hook.
Checked on Windows, where the pull request had never been compiled: the runtime builds and links; --probe-coop passes (sockets on 127.0.0.1, the codec against its byte vectors, a fuzz of every parser, two sessions through a hostile mock relay, a relay restart, the sync engine); tests/coop_relay_test.py passes its 61 tests, with the probe through the real relay on 127.0.0.1 at 0 and 30 % loss; tests/coop_page_check.py passes and its deliberate fault fails 48 checks. Not checked: two games in one room (one game runs at a time here), any real network, the tab in the game.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
make_game.py passed the map as -Wl,--Map=<game folder>/build-cache/ocarina.map, and -Wl, cuts its argument at every comma: with a comma in the folder's path the part after it reached the linker as an input file and an update ended with "the link failed" (issue #470). -Xlinker hands the path over as one argument. build_game.py already did so on Windows and now does on Linux too.
With MSYS2's clang and ld here, a link into a folder named "a, b" fails with the -Wl, form ("cannot find b/t.map") and writes the map with -Xlinker.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The shader step's progress event rounded its share to four decimals, and the launcher shows the whole percent below it: 42,140 of 42,142 shaders built read as 100% while a long driver compile was still running. The event now carries the share as computed, so the bar stays at 99% until the count is complete. The test feeds the last counts of a 100,000-shader build and fails with the rounding.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The launcher asked for 31 GB where the build's own check now asks for about 34, so a disk between the two passed the first check and was refused seconds later. Both say 34 GB now. The build no longer asks for the whole shader preparation's space when the game folder already holds translations. The texts said 10 to 25 minutes for an install that took players 27 to 41, and 5 GB for a build cache that measures 1.6. A build error's message goes into the launcher's log as one line.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
pad_prompts.py gave each packed game file to the decompressor in one
call, which returns one bytes object grown by copying. On made-up data
of the size of the interface texture archive, 69 MB packed and 244 MB
unpacked, that takes 21 s on a fast PC; players' builds waited 44 to 52 s
before the step's first folder, most of its minute. In 64 KiB pieces
joined once the same bytes take 0.3 s.
The prompts step's key held the digest of the three scripts' bytes, so
this edit would have drawn every install's prompts again, as two of the
last four releases did. It now holds a constant changed by hand, whose
value is the scripts' digest in 0.7.0 and 0.7.1, and a test fails when
the scripts' logic changes, to ask whether the constant must change.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Only the RomFS image's space was looked at, and only when that step
began: a disk that held the image but not the build cache or the shaders
stopped the build in the compile or, half an hour in, in the shader
preparation. The build now adds up what the steps it will run write into
the game folder (the image and 1 GiB, 2 GiB for the build cache, 4 GiB
for a full shader preparation, the least each is known to need) and
stops at once when the disk has less, naming each part and how much is
missing.
On Windows a build whose compile step will run first compiles and links
an empty program with the installed tools, against the libraries the
game links. Tools that are incomplete or blocked stopped the build only
in its compile step, after the image and the lift; the message says what
failed and how to get the tools back.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
romfs_check.py gave the decompressor 4 MiB of a file at a time. Each call
returns its output as one bytes object grown by copying, under the
interpreter's lock: on 32 MiB frames that ran at about 120 MB/s of packed
data on twelve threads, and at about 2,500 MB/s with 64 KiB pieces; on a
1 MiB file at 490 against 3,000. Files smaller than a piece are as fast
as before. The check now stops at the frame's end, so what it says of a
file does not depend on the piece size; a test compares both sizes and
the check as it was on damaged, cut and whole frames.
The scan step's key held the sha256 of the script's bytes, so this edit
would have made every install check its 26 GB again. The key now holds a
constant changed by hand, whose value is the sha256 the script had in
0.7.0 and 0.7.1: the key of every install that passed stays what it is.
A test fails when the script's logic changes, to ask whether the constant
must change.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
modules_c wrote the paths of the game folder as C strings with JSON's
default escapes, which turn a character outside the Basic Multilingual
Plane into two surrogate escapes. C refuses those ("invalid universal
character"), so a game folder with an emoji or a rare Han character in
its path failed at the end of the compile. The characters now stand as
themselves in the UTF-8 source, and a test compiles the generated file.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
make_game.py ran one compile per logical processor whatever the memory.
Without --jobs it now runs as many as 80 % of the free memory holds at
768 MiB a compile, the peak build_game.py records for one, never more
than the processors and never fewer than one, on Windows and Linux alike.
The build's first lines say the number and what decided it.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
An exception make_game.py did not report itself only printed a traceback,
and the launcher showed "the build stopped (exit code 1)". main now sends
every such error through fail with the exception's message (the system's
text and the file for a file error, the full disk named as such), and the
traceback still goes to the log.
A dump whose data does not match its own checksums ended the RomFS step
that way. The step now names the damaged file and says to copy or dump
the game again, and it removes the partial image of a failed run.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The Updates row promised that an update rebuilds the game "in a few
minutes". The code cannot keep that: a version that changes the lifter
compiles the game's code again, and one that changes what the shaders are
built with builds all of them into the driver's cache again. Players'
updates took 24, 34 and 45 minutes. The row now says it takes a few
minutes when little changed and half an hour or more otherwise.
The warm-up of the driver's cache after an update only builds, but its
count was drawn in the second half of the bar as in a full preparation,
so a 20 minute run read 50 % in its first second. Its count now fills the
whole bar, and the row says so while the shader cache is packed before it.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A step that runs drops the record of every step after it, and the RomFS
step comes before the lift although the lift reads only the unpacked
ExeFS. An install that was cut short while the image was still being
packed, the lift beside it already done, therefore lifted the whole game
again at the next try. The RomFS step now leaves the lift's record; the
compile and the shaders, which read the image, still follow it.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
compile_all wrote the object cache only when every file was done, so
Cancel or a crash in the middle of the compile lost the record of every
object and the next run compiled the whole game again. The cache now goes
to disk every 25 objects, at most every two seconds, moved into place
whole. Results are taken as the compiles end instead of in the order they
started: the largest file starts first, and in that order nothing was
recorded until it had finished.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A RomFS step that was cut short (Cancel, a full disk, a power cut) leaves
romfs.bin.part, up to 26 GB. The next run writes the image anew, but
make_game.py's step_romfs and the launcher's Begin Installation measured
the free space first, with that file counted as used: on a nearly full
disk the retry was refused although it needs no more room than the first
try. Both remove the file before they measure, and the first-run screen
counts its size as free.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The carry path of an update does not run the shader preparation: it runs --warm-shaders, which builds the listed pipelines from the seed-pipeline-keys.bin the game folder already has. make_game.py now writes that file from the kit's list first, so a newer list reaches an updated install. Its shader variants are not translated on this path; a record whose variant the folder has neither prepared nor met is left out and counted.
On a GPU that has no mesh shaders every mesh record failed with a log line each. They are counted in the step's line instead.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The preparation translates every shader and builds each one's stage library, so a pipeline a draw meets for the first time is linked in 2 to 6 ms. A mesh pipeline has no stage libraries: on an install that never drew it, it compiles whole in play, 0.2 to 1.1 s with its draws left out of the picture (18 of them on a first visit of Kokiri Forest). Whole pipelines were built ahead only from the folder's own pipeline-keys.bin.
tools/pipeline_list.py merge folds the pipeline-keys.bin and shader-answers.bin of game folders that were played into one list file: every key record once, with the answers record of each shader it names. A record names its shaders by input id, which depends on the game's code and the answers alone, and holds the game's pipeline state; what depends on the GPU is derived when the pipeline is built, so a record made on one machine is good on another with the same game version and runtime. Four folders on this PC give 5,432 records, 317 of them mesh pipelines (147 KB of keys).
tools/prepare_shaders.py reads the list (tools/data/pipeline_list.bin, data/pipeline_list.bin in a release folder, or --pipeline-list), adds its shader variants to the seed and writes its records as seed-pipeline-keys.bin. nx2navi.exe --prepare-shaders builds the mesh pipelines among them after the seed is translated and saves the pipeline cache before the stage libraries are built (NX2NAVI_PREPARE_PIPELINES=all builds the vertex pipelines too); --warm-shaders does it again after an update or a new driver, and --prepare-pipelines is that step alone. The pipelines go into the caches and are not kept or added to the folder's own key file, so they cost no start-up time or video memory until a draw meets them. A marker file keeps a driver that crashes on one of them from failing every preparation.
The list is derived from game data and stays out of the repository: check_content.py refuses files recorded from a game folder by their first bytes, and make_kit.py, make_launcher.py and make_release.py carry a list given with --pipeline-list. All three now also carry pipeline_list.py, which prepare_shaders.py imports.
Checked without the game: tests/pipeline_list_test.py, and prepare_shaders.py --no-run on the game files with a list of 317 mesh pipelines (all 266 shader variants found in this game version, 177 new to the seed). The build step in nx2navi.exe compiles and links and has not been run.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A player who keeps the game on a mapped network drive could not finish a build. Three things change in tools/make_game.py:
- The output folder keeps its drive letter on Windows. It was resolved to the share's UNC name, which is a second connection to the same server, and some servers drop that one under the compilers' load while the mapped drive stays up.
- When a source's size cannot be read because its folder stopped answering, the build says so and asks again for up to a minute before it stops with a message that names the drive. It used to end in a traceback. A file that is gone from a folder that answers still ends the build at once.
- A finished file is put in place through a temporary file beside it instead of a file in a temporary folder. Such a folder is made for its owner alone, and some shares refuse that while they take ordinary files; the build then reported a permissions fault in the output folder after everything had compiled.
From pull request #419: the first and third change are the contributor's, the wait is written again so that a file that is really gone does not wait a minute, and the launcher tests cover all three.
Co-Authored-By: Matthias <dev@sleepyyui.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A player's AppImage build of 0.7.1 stopped in its compile step after eight minutes of RomFS work: four
files did not compile, because the kit builds with the system's compiler and headers and theirs were
older than the runtime needs. vk_pipeline.c and vk_device.c name VK_KHR_pipeline_binary,
VK_KHR_dynamic_rendering_local_read and VK_KHR_shader_maximal_reconvergence, which their Vulkan headers
did not declare, and ui/menu.cpp takes its font in with #embed, which clang has from version 19.
- Vulkan headers: tools/vulkan_pin.py holds the release the other builds use (1.4.363, the archive and
hash tools/make_release_linux.py already pinned, which now reads them from there). The AppImage carries
its C headers in toolchain/vulkan-headers and builds against them whatever the system has; a kit or a
source checkout uses the system's when they are at least that release and fetches the pinned ones
otherwise (build_game.py vulkan_include). Their folder goes first on the include path of the runtime's
sources only, so the lifted code's objects keep their keys, and a runtime object compiled before
against the system's headers is taken over instead of compiled again. AppRun no longer asks for the
system's Vulkan headers.
- #embed: tools/build_shaders.py also writes the font's bytes as runtime/ui/roboto_ttf.h (ignored by git,
carried by a kit), and menu.cpp includes that where the compiler has no __has_embed. With #embed
nothing changes.
- Before its long steps a Linux build that will compile now checks every runtime source that needs no
header of the install with the compile step's own command (clang -fsyntax-only) and links an empty
program against the game's libraries (make_game.py preflight): an old compiler, an old header or a
missing development package stops it in the first seconds with the compiler's own errors. A failed
compile names every file that failed, not only the three whose errors it prints.
Checked on Windows: tests/launcher_test.py passes with four new tests (117); the header of bytes and
#embed give the same 162,588 bytes, and both branches of menu.cpp compile; the early check passes on the
Windows file set with clang 23 and stops on files made to fail; the 34 local headers of release 363 are
the upstream tag's files by git blob id. Not run: anything on Linux, the pinned archive's download, a
compiler older than 23.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The launcher release carries the runtime's shader headers built and none of the tools that build them, and
make_game.py's compile step imported build_shaders unconditionally: every update to the 0.7.0 build stopped at
"Compile Changed Files" with ModuleNotFoundError. The import is now tried, and without it the step that rebuilds stale
headers is left out, as its comment intended. A launcher test checks that every tool make_game.py imports is in the
launcher release unless the import is guarded or only runs on Linux.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Controls tab's Button prompts row and nx2navi.ini take steamdeck: the Deck's own labels in the game's
text, action prompt and ocarina notes, which are A B X Y where an Xbox pad has them, L1 R1 L2 R2 and L3 R3,
Menu and View. The style is drawn by tools/pad_prompts.py like the others, into romfs-patches-steamdeck
and romfs-patches-steamdeck-label, and every build with button art writes them.
Pull request #107 by loteran asked for PlayStation and Steam Deck prompts; it was closed, and the tree
took the PlayStation half in another form, drawn from button art instead of redrawn font glyphs. This is
the Steam Deck half in that form, written here: the art has no Deck style, so its entry in
tools/prompt_art.py names the Xbox pictures for the face buttons, Menu, View, the d-pad and the sticks
and the PlayStation pictures for the shoulders and stick clicks. It needs four pictures of the Xbox set
that no other style used (the white letters on a light button, for the ocarina's notes); a release made
with --prompt-art copies them with the rest.
The Windows files compile and tests/launcher_test.py passes, with the prompts step's test following the
number of folders (seven now); every picture the style names is in the art used for releases. The style
was not drawn here (that needs the game's font), and --probe-cfg, which has a new check, was not run.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Where a sandbox or a compiler wrapper hides the processor from clang -march=native, tools/make_game.py found
no x86-64 level and built the lifted code as generic x86-64. It now falls back to the features every
logical processor lists in /proc/cpuinfo and takes a level only when they prove it. cpu_tag reads the same
list through the new cpu_flags.
From pull request #295 by repparw, whose three build fixes came with its FSR 3.1 work. The other two were
in already (the processor in the compile key, the shaders step keyed by the translator in use), and the
wrapper case that prompted this one is also handled by the probe's own environment; this is the fallback
for what that does not reach. The pull request's main part, FSR 3.1 upscaling on Linux from vendored
sources, is not taken: 366 of its 494 vendored files are compiled SPIR-V shaders as C headers, which this
repository does not hold.
Checked on Windows with made-up cpuinfo files (v3 on every core, one core at v2, v2, no flags line);
tests/launcher_test.py passes.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
On Linux the game looks for libnvidia-ngx-dlss.so.<version> beside the executable, and the driver brings
only NGX's core, so the DLSS modes did nothing in every Linux build. tools/dlss_pin.py pins the Linux
library of the DLSS version it pins for Windows, with its own size and SHA-256; tools/make_appimage.py
puts it in the kit's bin/ with its notice (downloaded from NVIDIA's DLSS SDK repository and checked, or
--dlss; --no-dlss leaves it out), and tools/make_game.py copies it beside the game on Linux as it copies
the DLL on Windows, keeping a library the player put there.
Pull request #297 by mooki000, taken as it is; the builder's description is merged with the button art
option that came with the release workflow. On Windows place_dlss does what it did. No NVIDIA file is in
the repository.
tests/launcher_test.py passes with its two new tests (112 tests, Python 3.12 on Windows); the AppImage
was not built here.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
tools/make_game.py has its own compile step and did not build runtime/*_spv.h, which the repository
no longer holds: from a source checkout (the Linux path of docs/README-BuildKit.md) every runtime file
that includes one failed to compile. A kit carries the headers built, so nothing is stale there and no
shader compiler is looked for.
Pull request #405 by mooki000, taken as it is. Not run on Linux here.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The preparation builds every stage library once so the driver's disk cache holds it and a library a draw needs later takes under a millisecond. Turning dynamic vertex input off for AMD's RDNA 3 and newer driver changed the dynamic states every library lists, which gave each one another entry in the driver's cache, and nothing built them again: an update carried the prepared shaders over without the driver step. On an RX 7900 XT (26.10.1) all 42,142 prepared libraries were cold, 93 ms a vertex and 242 ms a fragment library on average. With no pipeline recorded, a walk through a village skipped 11,180 draws while their pipelines built and made 94 draws wait 5.5 s in all, four of them 1.0 to 2.0 s; a player's session log shows up to 114,834 draws skipped in 600 frames and 181 waits of 251 ms on average. Skipped ground and walls appear and disappear, and a skipped lighting pass is a black frame with the interface intact.
- The game checks the driver's cache at every start (vk_pipeline.c lw_start): a sample of the prepared fragment libraries is built, and when their median time says the cache is cold the whole prepared list is built into it again on idle-priority threads while the game runs. shader-warm.txt records a warm in progress, so the next start goes on with one a run ended, and stops the tries on a driver that does not keep the libraries. This also covers a driver update and a cache the driver emptied.
- An update that carries the prepared shaders over runs the game's --warm-shaders (tools/make_game.py warm_driver_cache): 11 s with the libraries there, about 8 minutes with all of them cold on that card.
- A draw whose pipeline is missing waits for its fast link from a budget of 6 ms a presented frame, 3 ms at most each (NX2NAVI_PIPE_LINK_MS), before it is skipped: with warm libraries the link is done within that.
- Full-screen draws wait from their 50 ms budget again instead of being skipped (NX2NAVI_PIPE_SKIP = any is the default; fullscreen stays selectable). With warm libraries the wait is a millisecond.
The same walk afterwards: libraries 0.3 and 0.7 ms on average, 1,605 draws skipped, 1,473 of them in the scene's first 600 frames and 8 to 80 in each 600 after, and 19 waits of 60 ms in all. First visits to four other saves (a mountain pass, the Lost Woods, Kokiri Forest, a river) waited 12 to 84 ms in all except the Lost Woods, where mesh shader pipelines, which have no stage libraries, and vertex libraries that need added outputs still compiled once: 23 waits, 2.1 s in all, the longest 776 ms.
The texture cache counted its waits in presents chosen at 60 a second: at 165 presents a second the 1800 that stood for thirty seconds were 11 s, so a released render target went sooner than meant and a texture out of view for 11 s was retired and showed its stand-in for some frames when it came back. The counts are now taken times the present rate over 60 (presents_for), and a texture is retired after two minutes without a sample instead of thirty seconds.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
An update decided the lifted code was 0.5.0's generic build from any old-format entry in objects.json, and 0.5.0's
runtime objects stay there for good, so 0.5.0 to 0.6.0 to 0.6.1 recompiled the whole game for generic x86-64. Only
the lifted objects' entries decide now, and entries of objects a build no longer makes are dropped. Rebuild the Game
could not undo it: the compile step's key names no level, so the step read as done before; it now runs when the
recorded level is not this processor's. With Keep Build Cache off the lift is deleted but its key still matched, and
the next compile found nothing to compile: the lift runs again when the compile step will. The two reads of the
compiler's version no longer fail on bytes outside the system's code page.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
On Linux the three host probes of the build did nothing. available_ram returned None, so the lift started one worker per CPU whatever the memory and could run the machine out of it; keep_awake did nothing, so the machine could sleep in the middle of a build of 10 to 25 minutes; cpu_tag returned 'any', so the compile step's key ignored the processor and a game folder moved to a CPU without AVX2 kept its x86-64-v3 objects.
available_ram now reads MemAvailable from /proc/meminfo and, when the process runs under a cgroup v2 memory limit (its own or a parent's), caps it by memory.max less memory.current. keep_awake starts systemd-inhibit with a lock on sleep (not idle: the screen may still blank and lock, as with ES_SYSTEM_REQUIRED on Windows) around a cat that reads a pipe this process holds open, so the lock ends with the process even when it is killed; without systemd-inhibit it does nothing. cpu_tag reads the flags of /proc/cpuinfo and returns the digits the Windows branch returns for SSE4.2, AVX2 and AVX-512F, counting a feature only when every logical processor has it.
The compile key now holds the real tag instead of 'any', so an existing Linux build compiles once more. The compile --jobs defaults are unchanged. tests/host_probes_test.py covers the three probes over fake /proc and cgroup files and a fake systemd-inhibit.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Use the native toolchain and archiver for runtime libraries, accept VMA_INCLUDE, and report missing Linux tools before compiling. Read the host CPU through compiler wrappers without changing the environment used by real compiles.
Compare copied tools by content rather than normalized timestamps, then replace them atomically so read-only POSIX installs can update without changing executable modes. Cover equal-metadata copies and translator refresh during shader preparation, and correct the patch-application comment.
This brings in the test branch up to the shader reads, not the texture eviction by disuse still in progress there:
- black frames: a released or taken-back texture keeps its content, and a new texture over a live one of another size gets the region both cover;
- the menu font atlas uploaded through the runtime's checked upload batch;
- the game held to 240 frames a second, which stops the quick menu's crash;
- the FSR 3.1 contexts evicted instead of turning FSR off after a few render scales;
- AMD frame generation kept off while NVIDIA's holds the device;
- less device memory held that only costs the system's commit;
- shaders read from the disk cache off the render thread past a small budget;
- the exit code 70 text;
- pull requests #260, #261, #270 and #271.
The tool lists in make_kit.py and make_launcher.py and the setting names in cfg.c take both sides: romfs_check.py and pack_shader_cache.py, FOG_CULL, SCENE_SHARE and GPA_TWIN_MAX_KB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A profiled traversal session had 9 of its 16 worst hitches (100 to 500 ms) inside the render thread's shader lookup,
reading <key>.res.json and <key>.spv of shaders met for the first time: opening a loose file costs about 3 ms the first
time on Windows. get_async_one now reads in place while those reads stay within 1/8 of the wall time (2 ms at once);
past that three loader threads read them and the draws wait as for a translation. The perf line counts both.
The launcher's updates carry the shaders over and never packed the cache, which held 44,000 loose translations in the
test install. An update now packs it once it holds 1,000 loose ones, and the pack tool reads the loose files on 16
threads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A dump can hold files whose first pages are zeros, with a container checksum that passes because the damage was packed along with them. The game then opens such a file over and over and gives up on whatever needs it: one damaged normal map left the Great Fairy Fountain without walls.
The game build's new last step, tools/romfs_check.py, decompresses every file of the game data once, in worker threads and never while the game runs, and writes romfs-check.json beside the game. A damaged texture gets a stand-in, an undamaged texture of the same kind renamed to it, in an ordinary mod folder the Mods menu can turn off, so the game can load what needs it. Other damaged files are reported, since only a clean dump restores them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The shaders step key hashed only the kit's bin/nx2navi_xlate.id, but the build takes the translator from the kit's bin/ or, in a source checkout, from build/xlate/ (xlate_source). From a source checkout, main() worked out the shaders key right after the compile step and stopped with a FileNotFoundError on bin/nx2navi_xlate.id before any shader was prepared.
The key now hashes the identity file beside the translator xlate_source finds, so a source checkout is keyed by build/xlate/nx2navi_xlate.id and a kit's key is unchanged. A build without a translator stops with the existing message that names tools/build_xlate.py. The prompts test kit gets a translator stand-in beside its identity file, and a new test covers a kit that has the translator only under build/xlate/.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The launcher builds through tools/make_game.py, whose shaders step never ran tools/pack_shader_cache.py, so a
launcher install keeps about 175,000 loose files (4.9 GB) in shader-cache/. The game looks them up one by one at
every start and reads the use masks of 42,000 of them for 20 s after it. make_game.py now packs the cache after the
preparation; a failure is reported and the build goes on, since the game still reads loose files. The kit and the
launcher ship the tool.
windres (binutils) opens its files and starts its preprocessor through the system's ANSI code page. A build or user
folder whose name holds a character outside that code page (Victor with an accent on a Cyrillic Windows, a Japanese
user name, a folder named with a full-width colon) failed the Compile step with "Invalid argument" or "Cannot create
temporary file". make_game.py now writes runtime/nx2navi.rc's two resources as a .res file itself (resource_file) and
hands it to lld, which builds the resource section from it and opens paths as Unicode. The resources come out as
windres made them: the manifest as 24/1, the icon's images as RT_ICON 1..7 and their group as RT_GROUP_ICON 1, US
English. Without lld the windres path stays.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A shader cache made before the game carried translations over has no identities.txt, so after the update that brings
the carrying the game could carry nothing, yet the update skipped the shader step whenever shaders had been prepared.
Before a new translator goes beside the game (the compile step, and the shader step a translator-only kit runs alone),
note_cache_translator writes identities.txt naming the translator still there, in the form runtime/gfx_shader.c
reads; a file already there is left alone. The shader step is skipped only when the cache names a translator.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An update recompiled every file of the game's code whenever anything in its keys changed, though the compile was the
same. The compiler's id held the file time of clang.exe, so a tool update that put the same compiler back made every
object stale; it is now the version text and the size. The fast CLANG64 compiler, whose objects are UCRT64's byte for
byte, was part of the key through its id and --sysroot; it no longer is.
So that this change itself costs nothing, an object whose entry has the key an earlier version gave the same compile
is taken over (ObjectCache.adopt) when its sources and headers read as they did: 0.6.0's form, and the form of 0.5.0
and earlier, which held the kit's own folder (app/<version>) in its paths; a header in that folder is read in this kit.
Checked read-only against two real game folders: all 551 lifted objects of a 0.6.0 install and of a 0.3.10 install
match their form and would be taken over.
The objects of 0.5.0 and earlier were compiled for generic x86-64, so an update keeps compiling the lifted code for the
level the objects were compiled for (build-cache/obj/lifted-build.json, lifted_level); --retune, for Rebuild the
Game, compiles it for this processor's level. An update also leaves the shaders prepared before in use and lets the
game translate them again in the background (shader_carry, NX2NAVI_SHADER_CARRY), instead of a full preparation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the local fixes package (fix 11), measured in Kokiri Forest at 4K on an RTX 4090 over PCIe 3.0: 40.5 to 62.6 fps
native and 45.6 to 74.9 fps with DLSS Quality, the mirror copies from 6.5 to 0.3 ms of GPU time a frame.
- Pages a VRAM mirror wrote back into guest memory are not copied up into it again: the mirror sync reads page stamps
of their own that a mirror's write-back leaves alone (vk_gpu_written_back, decided at record time by
vk_mem_mirrored). This replaces the lookup table NX2NAVI_WB_NO_REUP had, keeps its name, and is now on by default.
- The game's flushes of mirrored memory are not copied where the write watch covers the region, as every page the game
wrote there is stamped already (47-57 MiB a frame); NX2NAVI_FLUSH_UPLOAD=1 copies them as before.
- Written storage ranges up to NX2NAVI_WB_MERGE_MAX (now 32 MiB) are merged in 4 MiB pieces, so a range is no longer
bound to what one dispatch covers; the snapshot pool keeps its budget-scaled size with 128 slots.
- A submission's first part goes to the GPU at the first pass boundary while the GPU is idle (split_idle), long passes
are split too (split_in_pass), and the part's time limit reads the clock less often.
- A render target that needs an image of its own takes the image of a released one with the same parameters, released
two or more frames before (tex_reuse); memory requirements are kept per parameter set; a released entry another
descriptor's entry replaced at its address is kept and taken back.
- The memory check reads a page's write-watch bit and stamp under the lock that keeps them together.
- The launcher reads the gamepad only while its own window is in front, and does not start a second copy of the game.
- make_game.py says why a Python package did not load, not only that it is missing.
The fix's per-pass GPU timing is left out: NX2NAVI_GPU_PASSES does the same in main.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The compile step copies the kit's translator next to the game, but a kit that changes only the translator runs the
shader step alone (its step key holds the translator's identity), which then prepared with the old copy. From the
local fixes package (fix 9); its translator build guard is in main already.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The commit census of the perf trace walked the address space on the presenting thread outside mem_census, which
brought back the hitch mem_census was added to remove; it now runs only with mem_census = 1. DLSS's output pass at
the mode's share fell back to no pass at all when the scene was not at that share (the first presents, a dynamic
resolution step): after such a frame the first pass wider than the scene is taken again for 120 frames. The launcher's
nx2navi.ini editor no longer rewrites a file it cannot read, keeps the byte order mark and bytes that are not UTF-8,
reads an empty line as the game does, and leaves no temporary file behind. The build kit carries the two modules
build_game.py imports on Linux, the Linux steps list the right packages and Python, and stale comments and notes are
corrected (Discord on by default, two prebuild threads, Enhanced HDR gone).
DLSS Frame Generation needs nx2navi_dlssg.dll, NVIDIA Streamline's runtime and nvngx_dlssg.dll beside the game, which
no release carried, so the menu offered it and the game turned it off. tools/make_launcher.py --dlssg and --streamline
put them with their licences and a notice in the release's bin/dlssg/, and tools/make_game.py place_dlssg copies the
whole set beside the game. THIRD_PARTY_NOTICES.md has the row; the notes and README say RTX 40 and 50.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Linux build linked libunwind for a frame walk the runtime no longer uses: platform/linux.c walks the stack with
libgcc's unwinder. make_game.py's Linux toolchain imports build_game.vma_include, so the kit carries that file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
make_game.py only knew MSYS2. On Linux it now does what tools/build_game.py's Linux path does:
- clang, clang++ and lld from PATH (checked at start), the system's headers and libraries, vk_mem_alloc.h from the
system or the pinned copy build_game.py fetches into toolchain/vma;
- the runtime's *_sdl.* files and platform/linux.c in place of the Win32 ones; no shim.c (ui/menu_sdl.cpp defines
menu_sdl_event), no windres, no DLLs to copy;
- -fdeclspec for C++, as third_party/ffx_api/ffx_api.h declares __declspec(dllexport) unconditionally;
- build_game.py's Linux link flags (-no-pie, noexecstack, rpath $ORIGIN, libstdc++ static) and libraries, plus
dav1d (the AV1 movies), libatomic and libunwind (the previous commit's plat_frame_capture);
- ocarina and nx2navi_xlate without .exe; the translator from the kit's bin/ or a checkout's build/xlate.
Windows is unchanged: every difference is under os.name.
tools/make_appimage.py packs the kit (tools, recomp, runtime, the built translator, VMA) with a relocatable CPython 3.14
and numpy, lz4, zstandard and fonttools into NX2navi-Linux-<version>-x86_64.AppImage. Its AppRun checks for the
compiler and headers and names the packages for Arch, Fedora and Debian/Ubuntu, installs zbic (GPL-2.0, not bundled)
with pip into ~/.local/share/nx2navi/pydeps on the first run, picks --jobs from the memory, builds and writes play.sh
beside the game. README-BuildKit.md has a Linux section.
Tested on CachyOS (kernel 7.3, clang 23, RTX 3080, NVIDIA 620.70.06, 16 GB): make_game.py --nup ... --jobs 4 builds
ocarina in 9.9 minutes from a 1.1.1 dump, which passes --probe-cfg; a build with the same sources and flags
(compiled by hand before this script could) starts, loads a save and plays. With one compile per thread on 16 GB
the build filled memory and swap and the kernel stopped desktop processes, hence the --jobs advice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
make_launcher.py and make_kit.py take --prompt-art (folders holding XGamepad, P5Gamepad and Keyboard_Mouse, or those
folders) and copy only the pictures pad_prompts.py reads into the release's prompt-art folder; tools/prompt_art.py
holds the art tables and that list, with no imaging packages. make_game.py's new last step, prompts, runs
pad_prompts.py on that art into the game folder's romfs-patches-* folders. Its key names everything it reads (the
RomFS, the art, the tools, the bind_ lines of nx2navi.ini, the folders being there), so it runs again only when one of
those changed, and it makes no other step run. A kit without art skips it. The launcher's Python gets fonttools and
pillow, and its jobs show a Draw the Button Prompts row when the version has art.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The preparation translates the game's shaders in their default bind state and builds their stage libraries and
compute pipelines into the driver's own cache. Variants met in play with another bind state are still translated then,
pipelines are still linked in play, and the driver cache is kept per executable and is lost when the driver clears
it. The launcher's step text, the README, make_game.py and docs/launcher.md now say it makes such stops far rarer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The release carried AMD's FSR 3.1 library but not nvngx_dlss.dll, so DLSS ran only where the NVIDIA driver supplied the model itself; elsewhere players had to fetch the DLL by hand. tools/make_launcher.py and tools/make_kit.py now download NVIDIA's own build from the DLSS SDK repository at a pinned tag (tools/dlss_pin.py: 310.9.1, checked against its size and SHA-256, cached), and put it in the release's bin/ with a notice naming the NVIDIA RTX SDKs License that covers it. --dlss FILE takes a copy on disk instead, --no-dlss leaves it out.
tools/make_game.py copies the DLL and its notice beside the game. A different nvngx_dlss.dll the player put there is kept; the one an earlier build placed (named by the notice's sha256 line) is replaced, so a newer release can bring a newer DLSS. The compile step's key covers make_game.py and the kit version, so existing installs get the DLL with their next update.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>