Commit Graph

94 Commits

Author SHA1 Message Date
Tobim6
0b4fdac38d Defaults from the RX 580 runs: asynchronous replay, mesh emulation, mirror_hot 25, freeze_idle 1200, native zstd, game code at -O2; freeze and zstd and replay rows in the Lama patches tab.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:22:57 +01:00
Tobim6
93e68d57f1 Pool marks: a fragment shader stores its marks where it ends, not between a sample's coordinates and the sample.
On AMD's driver for Polaris (RX 580, 23.10) a store between a pool texture's access chain and its sample left the
quad's helper lanes out of the code in between: derivatives at a triangle's edge were garbage and edge pixels came out
dark, the dotted lines along the paths' triangles since 0.7.0 (frame dump f1714: the colour pass covers every pixel,
the depth test is not what drops them). Each access now notes its slot in a Private variable; before each return of
the entry function and before a kill, terminate or demote every noted slot is marked, an unset one going to a spare
slot past the pool. Accesses in loops or outside the entry function note theirs in a ring of 8, marked in a loop at
the way out. Checked with spirv-val on the 45 marked shaders of the dump.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:22:34 +01:00
Tobim6
bb654470b2 Crash report: the log names what the crashing thread was building, and a library link names its case.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:22:24 +01:00
Tobim6
1768d5f607 Mesh emulation: the vertex shader declares the fragment shader's inputs one for one.
The generated vertex shader wrote one vector per location. A fragment shader that packs two inputs into one location
(Component decorations: a vec2 at component 0 and another at component 2) then had nothing matching its second input,
whose value is undefined: wrong colours on the emulated foliage. The vertex shader now declares an output for every
input variable of the fragment shader, with its Location, Component, type and Flat, filled from those components of
the mesh stage's vector at that location (zero where the mesh stage writes none).

(cherry picked from commit c38eb31889314ebfb164ff9dffad5796eba46ed4)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:22:08 +01:00
Tobim6
8b78e64539 Mesh emulation: point meshes too, and a warning for draws from a nonzero workgroup.
The 8 point-list mesh programs of Ocarina 1.1.1 (32 points, position and point size) were left unemulated. Their
indices are single uints and the point size goes to an extra vertex slot, which the generated vertex shader writes to
PointSize; the draw asks for a point list (the pipeline key's topology class follows the draw's primitive for mesh
draws), chosen from the mesh stage's OutputPoints. A mesh draw from a nonzero first workgroup is drawn from 0 as
before, now with a warning in the emulated path too. All 1016 programs convert and pass spirv-val.

(cherry picked from commit 8900aeaffa3264e4cc655ffd487cb2d2f3e3dd17)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:22:07 +01:00
Tobim6
aec5b5a840 Emulate mesh shaders with compute on GPUs without VK_EXT_mesh_shader (NX2NAVI_MESH_EMULATION, off by default).
Without the extension every mesh draw was skipped ("mesh draws need VK_EXT_mesh_shader: skipped"): on an RX 580 the
game's grass and foliage were missing. The translator already builds each task and mesh program as compute code that
MeshSpirv.cs wraps into TaskEXT / MeshEXT modules; runtime/mesh_emu_spirv.c turns those modules into GLCompute ones:
outputs and payload move to workgroup memory, SetMeshOutputsEXT's counts are kept, and the entry function's end
reserves vertex slots and indices in an arena with atomics and copies the vertices and triangles out (task: the payload
and the mesh group count go to a task record, and the mesh dispatch to the draw record). A generated vertex shader
reads the vertices back for the game's fragment shader. Checked with spirv-val (Vulkan 1.3) on the 1016 task and mesh
programs of Ocarina of Time 1.1.1: 1008 convert and validate; the 8 point-list meshes stay unemulated.

vk_pipeline.c gives an emulated mesh pipeline a layout whose task and mesh bindings are visible to compute, with push
constants and a compute-side descriptor template, builds the generated vertex shader's graphics pipeline and the two
compute pipelines; vk_exec_draw.c prepares each mesh draw as before, then outside the pass zeroes the arena counters
(first emulated draw of a submission), fills a draw record, pushes the same descriptors to the compute bind point,
dispatches the task and mesh stages (direct, or from the indirect ring entry) and draws DrawIndexedIndirect from the
record. mesh_emulation = 1 turns it on; mesh_emu_mb sizes the arena (160 MiB).

(cherry picked from commit efc43b177694d8f8ce5700fcf28d80e889a45589)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:22:03 +01:00
chrg6
d31c3a1036 Make the water corrections in the runtime, on by default, instead of loading replacement modules from a folder.
The coordinate pair and the absorption guard needed modules made on each PC by tools/projective_recipe.py and a folder named in NX2NAVI_SPV_OVERRIDE, so no player had them. runtime/spirv_water.c makes the recipe's edits on the translation in memory, in the recipe's order: on the NVIDIA profile its three modules are byte for byte the recipe's (RTX 5090, game 1.1.1). vk_pipeline.c keeps the edited modules in a small table, made at first use, and builds a pair from them only when both of its stages were made; a module of another shape is built as it is, with a log line. NX2NAVI_WATER_PROJECTIVE=0 turns the corrections off. tests/water_projective_check.py compares the runtime's edits with the recipe's on made-up modules.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 07:55:57 -05:00
chrg6
a13af7a8c2 Replace a water pair's vertex module before the vertex output union's additions, keep pairs with dual-source blending original, and log each replacement.
The paired modules were looked up by the hash of the module as built, so a vertex module that had outputs added for its union (which differs from one install to the next) or a fragment module changed for dual-source blending ended the run. The vertex replacement now stands for the translation itself and takes the added outputs as any module does; a pair with dual-source factors is built from its original modules in both stages.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 06:37:04 -05:00
ChurchOfIlias
37e8e6dbda Guard water absorption distance alongside paired coordinates
Add the exact-material nonnegative distance guard and the recorded NVIDIA translation profiles. Keep generated shaders and the private native patcher out of the repository. Document reported cross-vendor testing and the unresolved origin of invalid absorption distance.
2026-11-12 06:34:27 -05:00
ChurchOfIlias
d0d0112c3a Add an opt-in paired projective coordinate correction for water
Carry clip coordinates through perspective interpolation for the identified water shader pair. Keep generated modules private and document the observed results and remaining portability limits.
2026-11-12 06:34:27 -05:00
chrg6
6ccd1e224d Build the shared pipeline list before play on every start, and build every listed pipeline at install.
A new install met every pipeline in play: in Hyrule Field one 600-frame stretch of a first visit had 1,769 draws wait 461 ms in all for pipelines, the longest frame 88 ms. The boot now reads seed-pipeline-keys.bin, the release's list, as recorded pipelines after the install's own (keys_open_seed, NX2NAVI_PIPE_SEED, default 1): their pairs go into the vertex output unions and the prewarm builds them as it builds the install's. The preparation builds every listed pipeline (NX2NAVI_PREPARE_PIPELINES now defaults to all), with the unions of the same list, so the first boot of a new install finds them in the caches. The list's mesh pipelines are left out at the boot on a GPU without mesh shaders, as the preparation leaves them out. Nothing of the list goes into pipeline-keys.bin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 06:19:39 -05:00
chrg6
677a69378f Fill the fragment inputs a mesh program does not write, with the point sprite's coordinate for point meshes.
A mesh pipeline whose fragment stage reads a location the mesh program never writes now gets a fragment module where
that input is a private variable set to (0, 0, 0, 1) at entry, or to (s, t, 0, 1) of gl_PointCoord when the mesh
program draws points. The fragment shaders recorded with two of the game's point-drawing mesh programs read location
5, which those programs never write; Vulkan leaves such an input undefined.

Checks: tests/fs_fill_inputs_check.py passes; 237 recorded fragment modules rewritten offline pass spirv-val; a
60 s Kakariko run with the change shows the area as before with no Vulkan errors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 19:38:18 -05:00
chrg6
ef64f341a0 Retry a graphics pipeline the driver refuses with VK_ERROR_UNKNOWN without its optimiser, and add VK_NOOPT=mesh.
AMD's Windows driver answers VK_ERROR_UNKNOWN for some pipelines its compiler gives up on, and the draws that need
such a pipeline were left out for good. create_graphics now tries once more with VK_PIPELINE_CREATE_DISABLE_OPTIMIZATION_BIT
(slower shaders for that pipeline instead of a missing draw) and logs the first such retry. NX2NAVI_VK_NOOPT gains a
value, mesh: only pipelines with task and mesh stages, the foliage's, are built without the optimiser, whose
optimisation is a suspect in device losses some minutes into a forest on AMD GPUs; with it a report can be narrowed
without slowing every pipeline.

Checked: the runtime compiles. A 60 s run with VK_NOOPT=mesh on an RTX GPU logs the mesh-only line, has no error and
the same GPU time as the default (4.09 ms a frame against 4.09), 134.5 fps. The retry cannot be triggered on this
GPU.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 19:38:18 -05:00
chrg6
8c353cdf21 Build the boot's first recorded pipeline alone before its builder threads start.
The boot's prewarm and the shared list's preparation started a builder on every core at once, so the driver began
that many graphics pipelines at the same moment, its first ones of the run among them. NVIDIA's 621.14 was reported to
fault in a sizable share of starts that way; a driver's first graphics pipeline sets up its compiler. warm_builder
now takes a one-job mode, and both callers build one queued pipeline on the starting thread before they start the
others.

Checked on an RTX GPU with --probe-prewarm in a copy of a dev folder (its prewarm skip marker removed), twice each with
the baseline and this build, alternating: no fault in four runs. With the 20 s budget the first pass built 2,203 of
3,226 recorded pipelines before and 2,135 after; the second pass built all 3,226 in both. A fault rate of a few
percent cannot be shown in four runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 19:38:18 -05:00
chrg6
276c886143 Prepare every pipeline of the shared list where the driver has no graphics pipeline library.
NX2NAVI_PREPARE_PIPELINES=mesh, the default, builds only the list's mesh pipelines, because a vertex pipeline is
fast-linked from its stage libraries in a few milliseconds and its vertex module depends on the output union this
install has recorded. Neither holds where the library is off (vk_caps_gpl: AMD's Windows driver for RDNA 2 and older,
or gpl = 0): a vertex pipeline a draw misses is compiled whole while that draw is left out, and its vertex outputs come
from its own fragment shader, since independent layouts need the library, so the listed pipeline is the one the draw
asks for. There mesh now builds every record, and the summary line says so.

Checked: all runtime files compile. Not run: no install on this PC holds the shared list, and the case is a GPU without
the library (NX2NAVI_GPL=0 would show it on any GPU).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 19:38:18 -05:00
chrg6
15bd64f4b8 Hold new pipeline builds back during the save at exit on every driver, instead of copying the cache while NVIDIA's builds run.
The save at exit now copies the cache only once the builds already in the driver have finished, with new ones held back meanwhile, as it already did on other drivers. Intel's driver crashed copying a cache while builds ran into it, and when the window closes the game's save is written after the pipeline cache, so a crash in that copy would lose it. The case the save at exit missed, a window closed while the boot's prewarm workers built pipelines back to back, is covered by holding new builds back: a build from the file takes about half a millisecond. The prebuild's comment keeps why its default is two threads, with the measurements of one thread for the rest of the list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 18:12:47 -05:00
automata.
27f93a314a Save completed pipeline cache data during NVIDIA background builds 2026-11-11 18:10:38 -05:00
automata.
60ac5f5982 Prioritize recent compute pipelines before reducing background concurrency 2026-11-11 18:10:37 -05:00
chrg6
295922c224 Sync the device address table's shadows once per guest submission that binds the table, and take out the port of pull request #353.
The author of pull request #353 closed it on 2026-11-11 with the words that they give no permission to use its code. Its port (3134f524) goes: the mark of a pipeline that declares the table (VkPipeInfo.gpa, vk_pipeline.c pipe_add) and of the part that draws with it (vk_mem_gpa_use in vk_exec_draw.c bind_resources and mo_emit), the sync of the table's pools in every such part's prologue (vk_mem_gpa_sync), the submission with a callback in the replay's place as it stood (vk_sync_probe_submit), and --probe-vkmem's read through the table with its shader (test_gpa_read, runtime/shaders/gpa_read.spvasm).

In its place, after pull request #551, which was written without that code: a shadow pool goes into the table and onto a list of the table's shadows (vk_memory.c shadow_table_add); the record of a binding of the table notes the guest submission (vk_exec_draw.c ssbo_info, vk_mem_shadow_table_read); and vk_mem_shadow_sync brings the CPU's writes into every shadow on the list once for that guest submission, before the part is submitted. Once is enough, because the copy-ins read the guest submission's write-watch snapshots, which a second sync would find unchanged. Measured by #551's author on Linux (RTX 4090, Kokiri Forest): 113 and 76 frames over 25 ms in 75 s against 217 and 429 with the port, 43 against 41 fps, the picture complete in both.

What this line had built on the port stays, on the new sync: the test that passes over a pool with nothing written (shadow_pool_clean), the upload of the same pages into a table pool's VRAM mirror, the write-back of stores through the table (NX2NAVI_SHADOW_TABLE_STORES, table_wb_cb, now asked once per guest submission that binds the table) and the counts of NX2NAVI_SHADOW_CHECK. The probes that record their own commands (the lost-device probe, --probe-vkmem's table parts) call vk_sync_submit_own, written anew. vk_mem_shadow_storage stays: one line that returns the mode. Finalized pools in the table were already kept for vk_mem_shutdown here.

--probe-vkmem has a check of its own for the sync (test_table_sync): the shadow of a pool that is never bound keeps its old word through a CPU write and a guest submission that does not bind the table, and holds the new one after a guest submission that does. It passes with NX2NAVI_SHADOW=1 on Windows (RTX 5090), as do the store check with NX2NAVI_SHADOW_TABLE_STORES=1 and the probe without shadow storage; --probe-devlost ends as before. Not run: the game, with or without shadow storage, and anything on Linux.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 06:10:09 -05:00
chrg6
31e0a02901 Merge the mesh shader fallback: task and mesh programs drawn through compute passes and an ordinary draw where the card has no mesh shaders.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 06:01:39 -05:00
chrg6
33d605fb13 Merge the texture origin diagnostic and the fill of a second descriptor's image from its storage, and let the play file replace a pipeline cache that is over the limit.
A pipeline cache over the size limit was dropped at every start and, since the file is no longer written in play, never replaced: every start began empty. The play file is now folded in at the next start whatever its size when the file beside it is over the limit.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 05:58:56 -05:00
chrg6
a8c6e992ee Draw task and mesh programs through compute passes when the mesh fallback is on.
A mesh draw becomes its record (the workgroup count, copied on the GPU for an indirect draw), the task dispatch, a
scan that lists the mesh workgroups the tasks started, the mesh dispatch, and an instanced indirect draw with a
generated vertex stage and the program's fragment stage. The compute modules bind the resources of the task and mesh
stages, and their pipelines are made on a thread of their own while the program's draws are skipped.

The indirect mesh ring and the texture pool marks are released at a probe's shutdown.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 05:58:42 -05:00
chrg6
f9219180b9 Try a pipeline cache save again when it did not happen, up to six times at the save interval.
A save that found no memory for its copy, could not write its file or met a build that had just begun was not tried again until the next new pipeline was built, so a run that built none and then ended without its orderly exit lost what it had compiled since the last file. The autosave thread now repeats the try, and the same bound holds for the save owed after the play file was folded in, which before was tried at every interval for as long as it failed. tests/save_retry_check.py checks the rule.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 05:25:38 -05:00
chrg6
d073d80e70 Create FSR 3.1's pipelines through the pipeline caches kept across runs, so a context is not compiled again at every start the driver's own cache does not serve.
The library creates its pipelines itself and names no pipeline cache. It is now handed pipeline creates of the runtime's in the place of the device's two, and a create without a cache is asked of the pipeline cache first and compiled into the play cache otherwise. ffx_pipe_cache = 0 turns it off. --probe-ffx-cache makes one upscaling context without a window or the game and says where its pipelines came from, and tests/ffx_cache_check.py runs it.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 05:22:56 -05:00
chrg6
2e28ba12df Merge the stutter fixes: a pipeline cache past 2 GB is read, the prepared shader list knows its seed, the pipeline cache is written only when it changed with a small file for what play adds, and a queued build a draw waits for goes first.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 04:59:05 -05:00
chrg6
70afa1584b Start a whole pipeline build a draw waits for ahead of the other queued builds.
A draw that waits for a pipeline with no fast link (a mesh pipeline, one without stage libraries, or one whose link failed) needs the whole build, and its job stood where it was queued: the draw queue is first in, first out, so on a burst of new pipelines the build a draw stood waiting for started after the fast links and builds of the draws before it, and the draw was skipped at the end of its wait all the same. graphics now hands the job to job_wanted: while no worker has taken it, it goes to the head of the draw queue and a worker is woken. A wait longer than 500 ms, which is no frame budget (a one-shot draw with oneshot_budget = 0), builds the job on the waiting thread at once when no worker is idle, as a stage library no worker took already is. A wait inside a frame's budget is never turned into a build on the replay thread: the budget is there so that thread is not held for a compile. The worker's build and bookkeeping moved into job_build for both to use; the log's statistics count both cases.

The other half of this item was there: the boot's prebuild asks the caches for a large compute shader's pipeline (compute_create, cached only) before it queues the background build (lb_compute), and with the play cache that question now reaches both caches.

Checked: it builds, --probe-cfg is ok, and --probe-prewarm builds the 3,212 recorded pipelines through the moved code as before. The queue jump and the build on the waiting thread need a draw that waits, which no probe makes: not run.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 04:57:21 -05:00
chrg6
719ee94e67 Write a pipeline cache only when its data grew, and give what a run compiles a cache file of its own.
pipeline-cache.bin was copied out of the driver and written whole at every save after any pipeline was created, also one the cache had held: tonight's logs show three saves of 358.1 MB in a run that added 0.3 MB, and 302 to 306 MB written in each of seven sessions that added about 0.5 MB each. The library warm in play built its 37,572 stage libraries into the same cache, which grew it by 1.6 GB: a save of 2026.7 MB in play, a file over pipeline_cache_max_mb, and the next start dropped it whole and compiled the pipelines it had held.

cache_save now asks a cache for the size of its data first and neither copies nor writes one whose size is what its file holds. What a run compiles goes into a second cache with its own file, pipeline-cache-play.bin (pipeline_cache_play, default 1; needs pipelineCreationCacheControl): pipe_create_in asks the first cache with VK_PIPELINE_CREATE_FAIL_ON_PIPELINE_COMPILE_REQUIRED_BIT and builds into the play cache only when that says a compile is needed, and in play only the play file is saved. At a start the play file is read back and merged into the first cache in memory; once it reached pipeline_cache_play_mb (64) it is folded in for good: pipeline-cache.bin is written once, before play, and the play file removed. The preparation modes fold it at any size and build into the one file as before. The library warm's threads create with no cache of ours, so their libraries reach the driver's own cache alone.

Checked with --probe-prewarm on a copy of a dev folder's caches (3,212 recorded pipelines, 333 compute shaders), no game run: with a filled pipeline-cache.bin every pipeline came from it and neither file was written; with none and a cold driver cache the compiles went into pipeline-cache-play.bin (75.9 MB written, pipeline-cache.bin not created); the next start merged it and wrote pipeline-cache.bin once (191.1 MB) and removed it; kept below the fold size, a play file of 68.8 MB gave 1,445 pipelines at 0.3 ms from the first cache where 470 hit without the merge, and was not written again. Not checked in play: the saves during a session, the library warm without a cache, and drivers other than NVIDIA's.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 04:56:26 -05:00
chrg6
39fa0be78c Make every pipeline built ahead of its first use pass a memory gate that counts the builds already running.
Only the large compute shaders' background builder looked at the system's commit before it started a build, and it
saw nothing of what the builds beside it were about to take. The boot's recorded pipelines (up to 64 builders), the
pool's background builds during play, the boot's compute pipelines and stage libraries and the library warm asked
nobody.

Each of them now asks the gate before a build. A build that starts reserves an estimate of the driver compiler's
working memory until it ends (build_reserve_mb, 64 MB; 4096 MB for a large compute shader), and the gate admits a
build only while the commit headroom holds the part kept free (build_free_mb, 4096 MB), the reservations and its own.
A refused boot builder ends and leaves its pipelines to the pool; the pool leaves a background build queued and asks
again every two seconds; the stage library prebuild and the library warm wait up to 30 seconds and then end, and the
warm goes on at the next start. A pipeline a draw or a dispatch waits for is never refused. The preparation of the
pipeline list runs without the gate: no game runs beside it.

tests/build_gate_check.py checks the gate's arithmetic and, through --probe-buildgate, its answers with the machine's
own commit. What the builders do when refused is not exercised by a test.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 04:42:07 -05:00
chrg6
cba95f8d0a Record in shader-prepared.bin which seed it was prepared from, and prepare again at an update when the seed changed.
The prepared list named its shader variants and nothing about where they came from, so an update that keeps the prepared shaders (shader_carry) kept the old list whatever seed the new kit writes: variants only the new seed names, from a newer pipeline list or a change to which variants are prepared, were translated when play first drew them. The existing identities do not cover this: nx2navi_xlate.id and identities.txt name translators, which decide cache keys and what is carried over, not which variants are listed.

The list is now version 2 (runtime/prp_file.h): after the head come the sha256 of seed-programs.bin, the sha256 of seed-answers.bin and the translator identity, written by gfx_shader_prepare_seed; the three readers go through gfx_shader_prepared_read and still read a version 1 list. At an update step_shaders writes the seed first (prepare_shaders.py --no-run, about 4 s) and keeps the list only when its digests are that seed's (make_game.py seed_changed); another seed, or a list that does not say, is prepared again. Another translator alone still carries over: input ids do not depend on the translator, and preparing everything again for it is what carrying was made to avoid.

Checked with tests/prp_file_check.py (the C layout against make_game's reader), three launcher tests, and the exe's --prepare-shaders on a seed without records in a folder of its own: the digests it wrote are those of the seed files, one byte more in the seed reads as changed, and after preparing again it does not. A version 1 list of 42,142 variants and the same list as version 2 are both read whole by --probe-prewarm.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 04:36:33 -05:00
chrg6
871b06e2e7 Read a pipeline cache larger than 2 GB on Windows, and log every refused file with its size.
cache_load measured pipeline-cache.bin with ftell into a long, which is 32 bits on Windows: a file past 2 GB measured as -1, so the run started with an empty cache and the log said nothing, not even that the file was over pipeline_cache_max_mb. The read now lives in runtime/vk_cache_file.h: the length comes from the 64-bit tell, the read runs in 64 MB pieces, and a refusal returns its reason and the file's size, which cache_load logs. The Windows atomic write no longer cuts a block of 4 GB or more to a 32-bit count.

tests/pipeline_cache_file_check.py reads a sparse file just over 2 GB through the header. With --probe-prewarm and such a file carrying this GPU's header, the log reads "holds 2048 MB, over NX2NAVI_PIPELINE_CACHE_MAX_MB 512" under the default limit and "2048.0 MB read" with the limit off; the exe before this change logged nothing for the same file.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 04:29:31 -05:00
chrg6
fc1a50f0db Merge the audit: its map and ledger, the perf-trace markers and size lines, and the fixes of its Phase 4.
Verified by runs in the audit: the Market's periodic stall (a pool slot was listed once per refresh pass instead of once; the list fell from 160,385 entries to at most 3,192 and no frame over 20 ms remained while standing), orderly exits delete the pipeline prebuild's running marker (left behind, the next start wrote a skip file and every later start skipped the prebuild), the log and crash report name an area a save asks for that the game does not have, and the menu atlas hash no longer overflows a signed multiply. The other fixes compile and pass the regression route but their faults were never reproduced: see AUDIT.md section 13 for each.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 00:39:08 -05:00
chrg6
e887a1d18b Delete the boot prebuild's running marker at every orderly exit, so closing the game during boot is not taken for a driver crash.
The marker was deleted only when the boot's builds ended. A window closed before that left it, the next start read it as a start that died in the driver's compiler, and every later start on that GPU and driver skipped the prebuild and built each pipeline at its first draw.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-10 23:22:27 -05:00
chrg6
9dc5fa8b36 Check the size of pipeline-cache.bin before reading it.
A file over pipeline_cache_max_mb starts the run empty, but it was read into memory whole before its size was compared, which is the memory the limit is there to spare on a 16 GB machine. The size is now compared first and a file that is dropped is never read. The limit itself stays at 512 MB: sizing it from the machine's memory, as the contributor proposed, was not run by them or here.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-10 23:14:00 -05:00
chrg6
ded13958de Log the sizes of the caches, pools and tables under the perf trace, every 600 presents and with each memory snapshot.
Four "mem: sizes" lines give the entries and bytes of what no other line counted: the guest heap, physical maps, kernel region, lazily written ranges and the fixed kernel, setjmp, save-file, RomFS and zstd tables; pool records, mirrors, deferred destructions, upload and staging buffers, the device address table and the texture, sampler and slot caches; image views, descriptor spares, the completion queue, uniform snapshot arenas, recording copies and pins; and pipelines, layouts, stage libraries, job queues, the shader memory cache and the translator processes. Each subsystem reads its own counters under its lock; settings now count the replaced values they keep. Segment and scene lines say "-" for the world clock before it starts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 19:51:36 -05:00
chrg6
66aedffcdd Name the work behind each frame on any thread under the perf trace: RomFS reads, zstd decodes, pipeline creates, guest memory maps, pool initialize and finalize, and device memory allocations.
A slow frame's line and the 60-frame window's line now say how long each kind took on any thread, how often, how many bytes and the longest single one, so a hitch outside the render thread's own waits has a named cause. While NX2NAVI_TRACE=perf is off each site pays one branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 19:27:58 -05:00
chrg6
87551c7618 An update that carries the shaders over builds the kit's current pipeline list, and a GPU without mesh shaders skips the list's mesh pipelines.
The carry path of an update does not run the shader preparation: it runs --warm-shaders, which builds the listed pipelines from the seed-pipeline-keys.bin the game folder already has. make_game.py now writes that file from the kit's list first, so a newer list reaches an updated install. Its shader variants are not translated on this path; a record whose variant the folder has neither prepared nor met is left out and counted.

On a GPU that has no mesh shaders every mesh record failed with a log line each. They are counted in the step's line instead.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-10 14:56:56 -05:00
chrg6
918d399d55 Add the shared pipeline list: mesh pipelines players met are built at the shader preparation, before the first start.
The preparation translates every shader and builds each one's stage library, so a pipeline a draw meets for the first time is linked in 2 to 6 ms. A mesh pipeline has no stage libraries: on an install that never drew it, it compiles whole in play, 0.2 to 1.1 s with its draws left out of the picture (18 of them on a first visit of Kokiri Forest). Whole pipelines were built ahead only from the folder's own pipeline-keys.bin.

tools/pipeline_list.py merge folds the pipeline-keys.bin and shader-answers.bin of game folders that were played into one list file: every key record once, with the answers record of each shader it names. A record names its shaders by input id, which depends on the game's code and the answers alone, and holds the game's pipeline state; what depends on the GPU is derived when the pipeline is built, so a record made on one machine is good on another with the same game version and runtime. Four folders on this PC give 5,432 records, 317 of them mesh pipelines (147 KB of keys).

tools/prepare_shaders.py reads the list (tools/data/pipeline_list.bin, data/pipeline_list.bin in a release folder, or --pipeline-list), adds its shader variants to the seed and writes its records as seed-pipeline-keys.bin. nx2navi.exe --prepare-shaders builds the mesh pipelines among them after the seed is translated and saves the pipeline cache before the stage libraries are built (NX2NAVI_PREPARE_PIPELINES=all builds the vertex pipelines too); --warm-shaders does it again after an update or a new driver, and --prepare-pipelines is that step alone. The pipelines go into the caches and are not kept or added to the folder's own key file, so they cost no start-up time or video memory until a draw meets them. A marker file keeps a driver that crashes on one of them from failing every preparation.

The list is derived from game data and stays out of the repository: check_content.py refuses files recorded from a game folder by their first bytes, and make_kit.py, make_launcher.py and make_release.py carry a list given with --pipeline-list. All three now also carry pipeline_list.py, which prepare_shaders.py imports.

Checked without the game: tests/pipeline_list_test.py, and prepare_shaders.py --no-run on the game files with a list of 317 mesh pipelines (all 266 shader variants found in this game version, 177 new to the seed). The build step in nx2navi.exe compiles and links and has not been run.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-10 14:52:46 -05:00
chrg6
e8f5f8e01e Merge branch 'main' of https://github.com/tangerineturtle1/NX2navi into test-all-fixes
# Conflicts:
#	runtime/cfg.c
2026-11-10 14:27:47 -05:00
epicmeme546
e38f42a959 Test for NaN on the bits of the number on AMD's driver for RX 5000 and 6000 cards, so that no shader compiler can leave a test out.
The reports of black and white blocks on Navi, on Link next to her and on bright surfaces seen from inside a house come from RX 5000 and 6000 cards, whose Windows driver compiles with "AMD proprietary shader compiler" (vk_caps.amd_sc); an RX 7900 XT (LLPC) draws the same scenes clean. The game's shaders test for NaN with OpIsNan in 48,717 of an install's 54,959 translated modules, 26 tests each on average, and the translator's guard that keeps NaN out of every float colour output, so that one bad texel does not spread through blur and bloom into blocks, is such a test too. SignedZeroInfNanPreserve asks the driver to keep those tests; whether that compiler does is not known.

With that driver, vk_pipeline.c spirv_nan_bits now turns every 32-bit isnan, scalar or vector, into (bits & 0x7fffffff) > 0x7f800000 on the number's bits, and a float select with a constant whose bits are a NaN into a select of the bits (issue #156), when the module is created. NX2NAVI_VK_NAN_BITS=1 does it on every driver, 0 on none. Run over every module of an install's shader cache, the rewrite changed 1,281,675 tests and 57 selects with every output well formed (each id defined once, below the bound, the uint types declared once); --selftest checks it on a small module, and Link's house with it forced on (RX 7900 XT) draws as without it. Not yet tried on an RX 5000 or 6000 card.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 13:08:03 -05:00
chrg6
e9ec6de6bc A render target's frame count follows its storage, not the guest's texture object, and a mesh draw no longer waits for a pipeline that is known to be compiling.
Found in the first run of the previous two commits (a first visit of Kokiri Forest from a state that had only seen Hyrule Field):

- The count of frames a target was rendered in started again whenever the guest's texture object changed, and the game initializes its render target objects anew every frame at the same storage: the log showed `seen 2129/2128 in 1 frames`, so no target ever became steady and the rule did nothing. The owner of an image is now the storage address the texture cache keys it by. An image handed on to a texture elsewhere still starts at none.
- A one-shot mesh draw waited its 8 ms in every frame that met the same pipeline while it compiled for 150 to 1100 ms: 17 waits, 170 ms, every draw skipped at the end, up to four in one frame. A mesh pipeline that has been building for longer than that wait is a compile, and the draws after the first do not wait for it (vk_pipeline_building_ms).
- The last wait of a full-screen draw with no picture to hold was 40 ms a draw, and two such draws made 86 ms in one frame. It is 40 ms a frame now.
- A one-shot vertex draw skipped at the end of its wait gets its log line however short the wait was (seven were skipped with no line, so their targets could not be named).

That run against the same cell of the earlier comparison (same exe path, cache state, route and settings; one run each): the load's worst wait of the draws in one frame 99 ms against 735 ms, no mesh wait over 13 ms against 524 and 206 ms. Two long frames remain in both runs (628 and 862 ms here, 660 and 854 ms there): two compute pipelines built at their first dispatch, about 395 and 397 ms.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-10 12:14:29 -05:00
chrg6
9d257edf5a The log says which missed pipelines were slow and why, and a slow frame that waited for a pipeline is charged to the pipeline.
A session's log gave the counts of slow pipelines and slow frames but not what they were. Now:

- Each pipeline a draw missed that took 50 ms or longer to become usable gets a line (the first 48 of a run): mesh or vertex, fast-linked or built whole, whether pipeline-keys.bin named it, its shaders' input ids and where their translations came from (the shader cache, a previous translator's, or translated in this session). The summary line counts the same: how many were mesh pipelines, how many the key file named and the prebuild had not reached, how many had a shader translated in this session.
- Each shader library that took 25 ms or longer gets a line (the first 32): its shader, the translation's origin, and whether outputs were added for its fragment shader, which is a module the preparation never built.
- The count of draw lookups that "found shader libraries missing" says what it counts: a library not made in this process yet, not one the driver's cache lacks.
- After the boot prewarm one line says how many of the recorded pipelines' shaders are a previous translator's translations, beside the pipeline cache hits that show whether any pipeline was compiled again.
- The frame-time summary's "over 25 ms by cause" took the replay's whole time as one part, so every frame that waited for a pipeline, a translation or a texture upload inside the replay counted as "replay". The replay's part is now its time without those waits.

gfx_shader_origin gives a translation's origin.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-10 12:02:24 -05:00
chrg6
0e16e58cf3 Merge the shader stutter fixes into the combined test line.
Eight commits: a draw that misses its pipeline has the rest of its submission read ahead, the library workers link
the pipelines of skipped draws, a pipeline whose fragment shader reads a target of its own pass is fast-linked like
any other, and diagnostics. The merge was clean. Their before and after matrix had not finished when this was merged.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-10 11:05:17 -05:00
chrg6
fdf2fb89d3 A draw that misses its pipeline has the rest of its submission read ahead, and the stage libraries of the pipelines the later draws need are queued at once.
The next draw with a new pipeline was only found when the replay reached it, so the draws of an area's first frame,
or of a turn that brings several new materials into view, paid for their pipelines one after another while the
workers that could have built them together sat idle. The submission is recorded before it is replayed: the first
draw of a submission that finds a shader or pipeline missing now has the ops after it read (vk_exec.c vk_exec_ahead)
with the binds that decide a draw's shaders and shader libraries applied to a copy of the replay state. Each draw
there looks its shaders up as the real draw will, which also queues a shader that is not in memory, and names its
state to vk_pipeline_hint, which queues the pre-rasterization and fragment shader libraries that are missing. No
pipeline is entered and no key recorded, so a wrong guess costs two libraries at most. The draw's own lookup then
links on its thread when both libraries are there.

On a first visit to Kokiri Forest a scan read 6,900 ops in 0.78 ms on average (the longest 5.5 ms; a scan ends
after 3 ms), and the scans of the load and the walk named 377 states without a pipeline and queued 104 libraries
for them.

pipe_ahead = 0 (NX2NAVI_PIPE_AHEAD) turns it off.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-10 10:00:20 -05:00
chrg6
b121f90e41 The library workers link the pipelines of skipped draws, so a burst of new materials is linked on several threads instead of by the one pool worker that is not inside an optimised build.
A draw that is skipped while its pipeline builds leaves the pipeline to the pool: a job is fast-linked first and goes
back in the queue for its optimised build. The pool keeps one worker out of optimised builds; with the other three
inside one (100 to 350 ms each) that worker linked every new pipeline of a burst one after another. The library
workers, which so far served only draws that wait, now take the fast link of a draw-queue job too (vk_pipeline.c
fast_take, lib_worker), and there are as many of them as pool workers (four on sixteen logical processors, two
before).

Measured on a first visit to Kokiri Forest (RTX 5090, libraries in the driver's cache): of the 444 pipelines draws
missed, 180 took over 50 ms to become usable, the longest 1.3 s, and their draws were missing from every frame until
then. With this change and the two beside it (feedback loops, the look-ahead) 23 did, 19 of them mesh pipelines,
which have no fast link.

pipe_fast_workers = 0 (NX2NAVI_PIPE_FAST_WORKERS) gives the old behaviour.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-10 10:00:19 -05:00
chrg6
1d9b0dc962 A pipeline whose fragment shader reads a target of its own pass is fast-linked like any other: the feedback loop flags are part of the fragment output library.
Such a pipeline (soft particles and water reading the depth buffer, full-screen passes that read the target they
write) had no fast link: gpl_link refused it, so its draws were skipped during, or waited for, the whole optimised
build. The flags belong to the fragment output state alone, so the fragment output library carries them and its key
names them (FoKey.fbl); the shader libraries are the ones the same shaders use without a loop.

Measured on a first visit to Kokiri Forest (RTX 5090, stage libraries in the driver's cache, DLSS Frame Generation
on, where no frame is held back and a full-screen draw waits on): full-screen draws of such pipelines waited 267, 42
and 61 ms in the area's first frame and 410 ms in the middle of the walk; fast-linked they wait 2 to 3 ms.

gpl_feedback = 0 in nx2navi.ini (NX2NAVI_GPL_FEEDBACK) gives the old behaviour.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-10 10:00:19 -05:00
chrg6
eeeeec683b The pipeline statistics say how long a missed pipeline took to become usable and which stage libraries were slow, and a draw counts as skipped for its pipeline only when it was skipped.
- Every pipeline a draw missed is timed from the miss to its first usable pipeline, the fast link or, without one,
  the build itself (vk_pipeline.c ready_note): the gpl lines of a save give the counts under 4, 8, 17, 50 and 150 ms
  and over, the average, the longest, and how many had no fast link. That time is how long the pipeline's draws were
  missing from the picture or waited.
- The stage library lines add the libraries of 25 ms or more and their sum, and the pre-rasterization libraries that
  had outputs added for their fragment shaders: those are another module than the one the preparation built, so the
  driver's cache has none of them.
- "a draw waited" lines name a pipeline with a feedback loop.
- The "draws skipped for a pipeline" counter (overlay, perf lines, flight records) was raised by every lookup that
  found the pipeline building, also when the draw then waited for its link and was drawn. It is raised where the draw
  is skipped now (vk_exec_draw.c draw_skipped).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-10 10:00:18 -05:00
chrg6
a1fcbb0b07 Name NVIDIA's Shader Cache Size limit when the library warm finds the driver's cache cold.
The driver keeps the prepared shaders' libraries in its shader cache, and under Windows that cache has a size limit
set in NVIDIA's settings. A limit smaller than the set drops the oldest libraries as new ones come in (a 2048 MB cache
could not hold them on the reporter's RTX 5070 Ti), so every start builds them again in the background and play
stutters until that is done. The cold lines of the library warm now say so on NVIDIA under Windows, most plainly when
the cache is cold right after a warm that finished on the same driver; the build kit's notes and the example file say
it too. tests/warm_hint_check.py checks which hint each case gets.

From pull request #403 by tarballz; two conflicts resolved (an include and the build kit's notes).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-10 09:13:35 -05:00
chrg6
3134f52475 Let shaders follow guest pointers into shadow copies, and keep those copies current for the device address table's readers.
With shadow storage (Linux, and Windows builds under Wine) a CPU-visible pool is a host-visible copy with
no memory object of its own, and only pools that owned one went into the device address table: pointers
the game's shaders read from memory reached nothing. Shadow copies are now in the table. A shader that
reads through the table resolves no pool, so nothing queued the pool it reaches for the copy-in: the part
is marked where its pipeline is drawn or dispatched (VkPipeInfo.gpa, vk_mem_gpa_use), and its prologue
brings the shadow of every pool in the table up to date with the CPU's writes and queues the same pages
for those pools' VRAM mirrors (vk_memory.c vk_mem_gpa_sync). --probe-vkmem reads a word through the table
from a pool that is never bound, before and after a plain CPU write.

Pull request #353 by TophC7, ported onto the base copies of the black-world fix:
- taken as it is: the table registration, the mark in vk_exec_draw.c and vk_pipeline.c, the prologue
  call and vk_sync_probe_submit in vk_sync.c (two hunks merged by hand), the probe's test;
- written again: vk_mem_gpa_sync, on top of the snapshot-based shadow_sync_pool; a lookup no longer
  queues a pool for being in the table, which every shadow pool now is;
- narrowed: the sync runs only with shadow storage. Without it (Windows) a mirrored pool stays unsynced
  for the table's readers as before, so the frame there is unchanged, and the probe makes the source's
  mirror only with shadow storage and skips when the device has no address table;
- the probe's shader was an array of SPIR-V words in tex_probe.c; it is now the source
  runtime/shaders/gpa_read.spvasm, built like the other runtime shaders;
- the exact re-upload count in the texture probe stays checked on Windows.
Files other sessions are changing: runtime/vk_memory.c (vk_pool_init's registration, mem_resolve's touch,
the new section before vk_mem_use).

The Windows files compile and the shader assembles and validates; no probe was run, and nothing was run on
Linux here.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-10 06:30:06 -05:00
chrg6
314aea08b3 Let the background builder of large compute shaders always keep its pipeline again.
Only a shader a run dispatched reaches that builder now, so its pipeline stays, entered before its cache is merged, as before the usage record existed. The branch that destroyed an unkept pipeline before the merge could no longer be reached; had it been, a dispatch in that window would have found the pipeline in no table and in no cache and compiled it again.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-10 04:58:09 -05:00
chrg6
0f334310ec Write the first-dispatch statistics after the pipeline lock is released, and make the record of dispatched compute shaders safe to write from two threads.
Found by a review of the compute prebuild change:
- cf_log formatted and wrote its log line while holding the pipeline lock shared, on the autosave thread, which runs below normal priority; every dispatch needs that lock. It now copies the numbers and writes after the unlock.
- The record could be written by the autosave thread and by a save at exit at the same time, so the older copy could land on the newer file; and a failed write cleared the changed mark, so nothing tried again. One writer at a time now, the copy is allocated outside the lock a dispatch takes, and a failed write leaves the record marked.
- compute_keep read anything that was neither "all" nor a number as 0, which keeps nothing; it now reads as the default.
- The statistics called a pipeline "read from the pipeline cache file" when the driver said the pipeline cache held it, which is also true of what the same run built into the cache earlier. The lines now say the pipeline cache held it. The probe's skip line names the settings that also lead to it.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-10 04:56:26 -05:00