Players' games crashed with three workers on by default in 0.9.1 and 0.9.2: an RX 6700 XT lost its device after
about 18,000 frames, its last mark a compute dispatch of the replay, with an occlusion query slot found unavailable
after its fence just before. The cause is not known yet, so replay_workers is 0 again unless a player sets it. The new
Replay workers row (0 to 8, experimental) writes replay_workers and waits for a restart, as the other restart-only rows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts: the settings key lists take both sides; frame generation keeps the camera-cut handling and logs its counters every 600 frames with the cuts among them; the pass-time mark moves into draw_viewport; mo_desc keeps the pose depth and notes the reader.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
With the HUD hidden the screen showed the scene at its render size instead of the upscaler's picture. The upscaler ran at the pass after the game's own upscaler, taken for the pass the HUD is drawn into. Without a HUD the game draws no such pass: the next one is the frame's last, which encodes the stand-in's scaled scene into the 8-bit output. The upscaler's picture went into that target and the last draw drew over it. A player's save with the HUD off showed blocks of render pixels, a picture jumping by a render pixel between frames, and the undrawn edge row in white, at every FSR mode and at Native AA; every test save until now had the HUD on. vk_dlaa_upscaler now runs the upscaler right after the stand-in of the game's upscaler wrote its output image, and puts the picture there (NX2NAVI_DLAA_AT_UPSCALER=0: as before).
The edge fill moves to the end of each pass that took the jitter (vk_dlaa_pass_end), for the pass's colour targets and depth, in one copy of eight regions per image. Filled only before the upscaler, the undrawn row had already been read by the water, the bloom and the anti-aliasing pass.
On a copy of the player's save (HUD off), FSR 4 Ultra Performance, world slowed to 1/20, present screenshots 0.25 s apart: the picture moves 0.00 to 0.02 px between shots (old placement: a jump of one render pixel in 5 of 20), the change on edges is 1.9 against 16.9, and no shot has the white row (half had it, with HDR on). Native AA, Performance with HDR, and FSR 3.1 are steady and clean as well; the save with the HUD on is unchanged. A run through a stream: 0 of 106 shots with a bright bottom edge (20 of 102). Frame rate 58.3 fps against 58.6.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
vk_exec_pass_end took tfr_lock and then, in vk_texture_gpu_written, the texture lock. A chunk recorded by another
thread can hold the texture lock in the pool sync and end its pass there (a lookup's alias copy through
vk_exec_inline_begin), and then wait for tfr_lock. The two chunks waited for each other and the other chunks spun in
their turns, so loading after Continue never finished with replay_workers = 3. The targets are now marked written
before tfr_lock is taken, which guards only the target history.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A split hands the part to the queue and seals its texture pool set; the next part takes another set of the ring and
refreshes its entries first. The splits for the mesh stages made 5 parts a submission and 6 set switches a frame at
2.3 ms of replay each (Kokiri Forest on 0.7.1, an RX 580: replay 32-38 ms a frame, the replay thread blocked on fences
26-35% of the time, 79 waits for a set the GPU still used). A cut (vk_sync_cut) ends the command buffer without
submitting it: the part and its pre-pass command buffer wait in a chain and go to the queue, in order, with the next
part that does, so the set is the same one throughout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each emulated mesh draw ended the pass, ran its stages between four barriers and began the pass again: a pass of 20
emulated draws became 20 passes, and on an RX 580 6 to 7 ms of a frame's 30 ms of GPU time went to the gaps between
passes (gpu_passes in Kokiri Forest: 8.7-10 ms per frame outside the game's passes and dispatches, 6.3-7.7 ms of it in
273-306 gaps with barriers only; 71 emulated draws a frame, 36 of them indirect). Now the first emulated draw of a pass
ends the part there (vk_exec_split_now), so the pass begins a new part, and the stages of every emulated draw of that
pass, the indirect ones' ring entries included, go into the part's pre-pass command buffer (vk_sync_pre), which the
queue runs after the mirror prologue and before the part; the pass is not split again. NX2NAVI_MESH_EMU_BATCH=0 (Lama
patches: Mesh emulation before the pass) brings the split back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A dynamic resolution that keeps its targets at the largest size and draws the scene into part of them shows up as a
viewport smaller than the pass; one that resizes its targets shows up in the pass size. NX2NAVI_SCENE_SCALE=33 left
the handheld scene pass at 1440x810, so this tells which of the two the game does there.
(cherry picked from commit af7abe36b0b15c5a63b354d2870543987d1f4b1a)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On an i9-14900KF (8 performance and 16 efficiency cores, 32 logical processors), with the replay on its own thread,
the core NX2NAVI_RECORDER_CORE=auto reserved (logical processors 14 and 15) left the woken replay thread unscheduled
for about 200 ms roughly once a second while the game stood in Kakariko. During each stall its queue held 40 to 50
items, its lock was free, the wakes had been sent and it used no CPU time, while a game thread waited on a fence of the
work in that queue: 27 frames a minute had a 150 ms or longer frame. With NX2NAVI_RECORDER_CORE=0 the same minute had
none. The default is now 0; auto and 1 still reserve the core, and the log says when none is reserved.
On an RX 7900 XT with a Ryzen 7 5800X3D, Hyrule Field standing went from 41.2 to 50.4 fps in one batch, and the
pictures matched those with the workers off there and in Kokiri Forest, Kakariko, Goron City, a graveyard house and
Gerudo Valley. replay_workers = 0 in nx2navi.ini turns them off.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The workers have no chunk to record while the snapshot copies the uniform ranges a submission binds (about 700
regions, 0.7 MiB in Hyrule Field), so the replay thread now hands them slices of it through vk_exec_par, a loop the
workers take between groups. Without workers the copy runs on the replay thread as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hyrule Field, standing, 3 workers: with 1, 2 or 4 groups out the frame rate stayed within one frame per second of
the 50 the mode without pipelining makes, over three batches, though the replay is shorter. A frame there is the
replay and then about 6.6 ms of GPU work after it, which the next frame waits for; the design comment in vk_exec.c
now says so. Sending a part's finished command buffers while later groups record was tried and gained nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With NX2NAVI_REPLAY_PIPE=1 (off by default) the submitting thread hands a group's chunks to the workers, takes chunks
of it itself until none is left to take, and goes on with the ops after the group in a new command buffer after the
chunks' places in the part (vk_sync_append_later), also into the next group's front pass.
Up to four groups are handed out and not in their part yet; their chunks wait in one queue in the order of the stream.
The submitting thread waits for them only where the order needs it: for room, before a part goes to the GPU, before
copies and texture or buffer clears, where DLAA's run or the contact shadows' hook acts, and where a lookup would make
a texture entry or take back a retired one (the op goes again afterwards, as a chunk's does in turn).
What the chunks read stays as the stream had it while the submitting thread runs ahead: uniform snapshot drops count
from their place in the stream, a chunk reads only the inline uniform updates before its group, the HDR present's
source goes by stream position, the texture pool set syncs under the chunks' lock, each group keeps its own render
conditions, occlusion segments and closed query slots, and the queries after a group's last report go into the next
segment made after it. A group's first chunk runs its pass hooks after the group before began its pass; turns wait
for every earlier chunk, of earlier groups too. Locks the chunks take, the submitting thread takes while it is behind.
Draws keep the sampled-texture descriptors they resolved per draw context and reuse them without the texture lock
while their slot is unchanged; the workers waited for that lock at nearly every draw. Texture lookups that change
nothing no longer wait for their turn. The perf line says where the submitting thread waited, what the chunks took,
and which texture lookups waited; NX2NAVI_GPU_TIMELINE also shows the chunks and the waits.
Off by default because it does not pay yet: in Hyrule Field with 3 workers it shortens the replay (11.5 against 12.1
ms a frame) but not the frame (49.2 against 49.6 fps in the same batch). The GPU gets a group's commands only once its
last chunk is recorded, so the main thread's wait for the GPU after the frame's first submission grew by 0.8 ms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Mesh draws ended a group and stayed on the submitting thread: 119 a frame in Hyrule Field at about 13 us each, and the
groups around them were cut short. They are recorded in the chunks now: the entries filled ahead for the indirect ones
are the draw context's, the ring index is taken atomically and the ring is made before the chunks record, each chunk
counts the ops that break a run of them on its own, and the lazy lookups of the mesh and shading-rate entry points store
their pointer before the flag that says it is there. With the mesh fallback, and while NX2NAVI_GPU_PASSES copies the
workgroup counts, they stay on the submitting thread.
A chunk's closed occlusion query slots go into a list of its own and into their segments once the group is done, in
chunk order, instead of under a lock per query. The per-object motion check decides alone per draw context (its
variant memo is per context) and takes its lock only to list a draw. The occlusion query ring has 16384 slots: the
chunks' blocks are sized to their draws, and with 4096 the ring came round to slots not reset yet.
The perf line splits the submitting thread's time with workers on: ops outside groups by kind, the front pass, its own
chunks and the wait for the others, and which ops ended the groups.
Hyrule Field, RX 7900 XT, 3440x1440, FSR 4 Native AA, same batch: standing 51.7 fps with 3 workers against 43.6 with
the workers off (replay 11.2 against 15.1 ms); along the field route 48.9 against 44.7. Screenshots standing and along
the route match the replay without workers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The chunks of a parallel group took query slots from the ring as they recorded, so the slots of the group's chunks
interleaved. The completion thread reads and resets slots in runs that follow the ring, and with interleaved slots it
made four times the result calls and a reset per slot: the wait after a frame's first submission ended 3.5 ms after
the GPU had run it instead of 0.6 ms, and the replay time the workers saved went into that wait (44 fps with 3 workers
as without). The front pass now gives each chunk a block of slots, as many as its draws that count samples and render,
in the order of the stream; a segment's slots are taken in ring order. Standing in Hyrule Field, 3 workers: 49.8 fps
against 43.7 with the workers off.
NX2NAVI_GPU_TIMELINE=<frame> writes four frames from that present on to the log: each replay, each timed part's submit
and GPU run, the waits for the GPU and the workers' groups, so a run shows whether the GPU waits for the replay, a wait,
or the CPU behind one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Samples-passed resets and reports ended most groups in Hyrule Field (the game reports twice without a reset there, so
counting stays on): the front pass now replays them in the order of the stream, makes each occlusion segment and
queues its completion callback in that order, and each worker follows the counting from its chunk's first op and adds
the queries it closes to the segment the stream gives them. Per-object motion takes a lock instead of waiting for the
earlier chunks: its list's order does not change the picture. The locks the chunks share are taken only inside a
chunk, as outside groups the submitting thread alone replays; taken always, they cost the replay about 1 ms a frame.
The perf line counts the draws by what ended their group.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stage 2 of NX2NAVI_REPLAY_WORKERS (default 0, the replay as before). A run of draws is cut into groups: a group ends
where the render targets change after its first draw or clear, and before an inline uniform update, a mesh draw and a
samples-passed counter op, which the submitting thread replays itself in the order of the stream. A group of 64 draws
or more is first walked by a front pass on the submitting thread: it applies the state and binds, settles the CPU side
of the timestamp reports and decides the render conditions in the order of the stream, and copies the replay state at
the first op of each chunk. The workers and the submitting thread then take the chunks in order and record each into a
command buffer of its own, which vk_sync_append puts into the part in the order of the stream.
The frame's pass hooks (contact shadows, DLSS) run in a group's first chunk, which the others start after. What still
depends on the order between chunks waits until every earlier chunk of the group is done: a texture lookup that would
make or take back an entry returns nothing and the chunk replays the op again in turn, and so do the contact shadows'
pass, per-object motion and the HDR pass's source. The tables the chunks share take locks (the shader table, the
descriptor lists, mirror use and GPU write notes, uniform snapshot drops, target frames, the DLAA camera votes and the
previous-frame motion notes); each chunk keeps its own occlusion query open, its own feedback memo and no pass marks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reading the thread-local pointer in the functions that run per binding and per descriptor cost about 0.3 ms of
replay a frame in Hyrule Field (45.0 against 45.8 fps). VkExec.dc now names the context; the draw and look-ahead
entry points fill it from the calling thread's on first use, the functions that hold the state read it from there and
the few helpers without it take it as a parameter. A replay worker's copy of the state carries the worker's context.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stage 1 of recording a submission on several threads. A submission's draws come in runs between ops that have to
stay in the replay's order on its own thread (dispatches, copies, texture and buffer clears, events, the game's
barriers, calls). With NX2NAVI_REPLAY_WORKERS=1 a run of 16 draws or more is recorded by a worker thread: it takes a
copy of the replay state, records the run into a command buffer of its own pool with a draw context of its own, ends
it behind a full barrier and hands the state back. vk_sync_append puts that command buffer into the part between the
replay's own pieces before and after it, so the GPU runs everything in the op stream's order. The submitting thread
waits for each run, so every side effect keeps the replay's order; this stage checks that a run recorded apart from
a copy of the state gives the same pictures. The default, 0, replays as before.
walk's handling of the ops that do not steer it moves into op_one, which the worker uses too. NX2NAVI_REPLAY_RUNS=1
logs every 600 frames how the draws fall into such runs and which ops break them. The draw replay's hot functions load
their recording context once instead of at every use.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vk_exec_draw.c kept what a draw, bind or dispatch carries to the next one in file statics: the state the command
buffer holds, the descriptor signatures and skips, the per-binding reuse, the shader and pipeline memos, the shader
lookup's caches and the submission's transient resources. They now live in one DrawCtx, and every thread starts with
the replay's own, so the replay behaves as before. A later stage can give each recording thread a context of its own.
vk_exec.c's inline commands (vk_exec_inline_begin) find the replay state the calling thread records with through a
thread-local pointer too, instead of the file's single state and a flag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cache_die moved the process to the logical processors of the largest L3 cache whenever the L3 sizes differed. That is
right for a two-die X3D desktop part with 8 cores a die, but it also held the game to the 4 large cores of a mobile
part whose 8 small cores share a smaller L3, and to the 6 cores of a 12-core X3D part, where the recording thread's
reserved core then left the game 5. The die is now used only when it has 8 or more physical cores (perf.c
cpu_placement logs when one is passed over). And while the process runs on that die alone, RECORDER_CORE=auto
reserves no core for the recording thread, since the game's threads already have the die to themselves; =1 still
reserves one.
Checked: all runtime files compile. This PC has a single-die processor (one L3), where neither rule applies: a run
still reserves the recording core (logical processors 14 and 15 of 16) and logs no cache die.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two descriptors can name one CPU-written storage: one whose content the cache takes from the CPU, and a second with a texture flag for which it does not. The second has no upload of its own and its slot shows the first one's image while that entry is live. Once the first is released, a lookup of the second's slot makes it an entry of its own, start_at destroys the released first one, and the new image starts cleared and is never uploaded; the slot shows it from then on, also after the first descriptor is registered again. The same happens when both are released and the texture pool refresh looks the second's slot up before the next texture arrives in that memory.
With tex_split_fill = 1 (the default) start_at gives such an entry the storage's bytes: by a copy from the released CPU-written entry it is about to destroy when that image is current and alias_compatible allows, else by an upload from the storage. The image then follows the storage as a CPU-written texture's does until the GPU or WriteTexels writes it; 2 fills it once, 0 is the behaviour before. Depth, stencil and multisample images, pools without CPU access and storage over 32 MiB are left out, and an entry that is to be filled gets a new image instead of a released entry's, since a reused image goes through its first transition after the upload batch. The perf trace counts the copies, the fills with their bytes and time, the fills made again after a write and the entries left empty.
--probe-vkmem makes both sequences with the three values and reads the second descriptor's image back: zeros with 0, the storage's bytes with 1, and with 2 the bytes of the texture that was in the memory before when both descriptors were let go of. One case makes the entry inside a recorded submission, through a hook that lets the probe stand in for a replay, and there the content comes by the copy.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
For a day that wait ended on a second timeline, when the GPU had run the submission's own commands (wb_split = 1,
then the default), on the thought that it only left out write-backs nothing needed yet. It left out the completion
thread's side effects for the submissions before it as well, and the game reads some of those next: the occlusion
counts, counter reports and event values, and the stamps the texture checks read. A frame's first submission is small
and runs right behind the last frame's large one, so the game went on while that one's callbacks were still running.
Measured in Hyrule Field with the camera turning (RTX 5090, lazy write-back on): 2,267 of 2,320 such waits returned
before those side effects had run; they were through 0.45 to 0.8 ms later on average, 12.6 ms at most, and in that
time 770 to 1,015 occlusion counts, 21 to 28 counter reports and 17 event values a frame reached guest memory. The
submission the game waits for wrote nothing back in any of 14 runs, so its own write-backs were never the matter.
A player saw a patch of grass near the camera change from frame to frame there and less of it with the fence's wait.
No scripted run showed such a frame with either wait, so the link to what the player saw is not shown.
The fence's wait takes 0.65 to 0.7 ms more of a frame there, about 6 % at 85 frames a second (single runs beside
other work). wb_split = 1 still gives the other wait, and 2 forces it with shadow storage.
NX2NAVI_TRACE=perf now counts what a wait on that timeline leaves behind and the completion thread's time by callback.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
(cherry picked from commit 4d108b706e72c001ddbe20608f06469885df584b)
A lost device used to end the process from whichever submit or wait noticed it first, while the present only logged it
and went on and the other threads kept calling the driver. The completion thread could take the submissions' timeline
as reached and run the callbacks of submissions that never finished.
vk_device_lost_check now takes the result of every submit, present, acquire and wait. The first thread to see the loss
sets a flag, writes the report and ends the process as a fatal error does (the log written, a save commit under way
finished), with an exit code of its own; later threads stop in that call, and a watchdog ends the process after ten
seconds should the first thread hang. The replay, its submit, the uploads, the present and the lazy fetch read the flag
and return without touching the device. The completion thread asks the device with a call of its own before a new
value counts and runs no callback once the flag is set.
The executable's --probe-devlost and tests/device_lost_check.py check this without a lost device, and
device_lost_at in nx2navi.ini ends a real run the same way.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
An indirect mesh draw reads its workgroup count through a ring entry that a buffer copy fills, and a copy cannot be
recorded inside a pass: every such draw ended its pass, 45 times a frame in a village. The first draw of a run now
fills the entries of the mesh draws that follow it with only state, draws, counter reports and render conditions
between them, and those draws stay in the pass (mesh_ahead, default 1; 0 is the old way, ab switches in A/B blocks).
An entry is taken only by the draw it was filled for, at the same record address, while no other op was replayed
since. Counter reports count as staying in the pass because the replay closes their query inside it and writes the
value from the completion thread; the game reports one after most of these draws, and without that only 38% of the
draws were served.
Measured in the village: 42 of the 45 draws a frame are served by an entry filled ahead; gaps with these copies
41 to 3 a frame (0.22 to 0.04 ms of GPU time); passes and dispatches 248 to 208 a frame (302 to 254 in Hyrule
Field). With the piece release of the commit before, one run each: village 142.6 fps and GPU busy 3.87 ms (138.0
and 135.4 fps, 4.15 ms without this change), Hyrule Field 136.8 fps and 4.34 ms (131.5 and 131.3, 4.70 ms). In A/B
blocks of one run with the narrower rule (counter reports still ending a run) the GPU's busy time went from 4.81 to
4.64 ms.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Every pass begin and every compute dispatch recorded a full barrier, which makes the GPU finish everything before each of them. With barrier_game = 1 (NX2NAVI_BARRIER_GAME) a pass after an earlier pass gets a barrier for its attachments only and a dispatch after other dispatches gets none; a change between drawing and compute keeps the full barrier, and so do the game's own barriers and the replay's copies, clears, fills and uniform updates. On an RTX 5090 at 2560x1440 the GPU's time for a frame fell from 5.4 to 4.9 ms standing in Kokiri Forest (two in-run A/B runs, the control's blocks differ by 0.7 %) and from 5.1 to 4.4 ms on a walk through it. The frame rate did not change there, because the replay thread is the limit on that machine; it should rise where the GPU is.
The default is 1 on NVIDIA graphics cards only, and 0 on every other vendor's: the game placed its barriers for an NVIDIA GPU, Vulkan asks for more of them than the game recorded, and no AMD or Intel driver has run this. 34 screenshots across the A and B blocks of one run show the same picture on NVIDIA. NX2NAVI_TRACE=perf logs the barriers a frame by kind.
From pull request #389, which had it on for every vendor; the default by vendor and the settings text are added here. The validation layer's synchronization check could not be run on this machine.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Both are about the time between two frames. Ocarina waits for the previous frame's fence before it lets the next frame's GPU work start: its main thread signals an event from the CPU, and the main queue's commands begin with a wait for it.
The completion thread ran a submission's callbacks before it let the submission's waiters go, and the occlusion counts were among them: one vkGetQueryPoolResults and one vkResetQueryPool per reported query, about a hundred queries a frame in Hyrule Field. On an RX 7900 XT (26.10.1) each call goes into the kernel, 3.9 ms of every frame, so the fence was reached 4 to 6 ms after the GPU had finished with every game thread waiting. Slots come from the ring in order, so one call now reads up to 256 slots from the one asked for, the available results are kept for the segments that follow, and the slots taken are reset in runs (vk_exec_xfer.c oq_take, NX2NAVI_OQ_BATCH). Standing in Hyrule Field at 3440x1440 with FSR: 52.7 fps against 44.6, the longest frame of the measured minute 46 ms against 55; the completion thread spends 0.3 ms a frame in the callback.
When the CPU's signal came a millisecond or two after the main queue's submit call, the submission took its copies, stopped at the wait and was held; the render thread polled the event, presented, and the present replayed the frame under a value the game's fence does not stand for. The next frame then skipped its wait for the GPU and was replayed inside its submit call, and frames of 23 and 15 ms took turns. A submission that begins with a wait for an event the CPU has signalled before now waits for the signal in the submit call, before its copies are taken, 10 ms at most (vk_exec.c vk_exec_gate_wait, NX2NAVI_GATE_WAIT_MS): 54.5 fps and a longest frame of 33 ms in the same place, 58.6 fps with the recorder thread on. With a camera sweep the pictures went out 21.0 ms apart on average with a 99th percentile of 55 ms, against 25.4 and 64. A village went from 48.6 to 53.7 fps and a graveyard from 56.5 to 64.2. 900 consecutive frames of a still scene show no dark or changed frame, with the recorder thread and without.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Found in the first run of the previous two commits (a first visit of Kokiri Forest from a state that had only seen Hyrule Field):
- The count of frames a target was rendered in started again whenever the guest's texture object changed, and the game initializes its render target objects anew every frame at the same storage: the log showed `seen 2129/2128 in 1 frames`, so no target ever became steady and the rule did nothing. The owner of an image is now the storage address the texture cache keys it by. An image handed on to a texture elsewhere still starts at none.
- A one-shot mesh draw waited its 8 ms in every frame that met the same pipeline while it compiled for 150 to 1100 ms: 17 waits, 170 ms, every draw skipped at the end, up to four in one frame. A mesh pipeline that has been building for longer than that wait is a compile, and the draws after the first do not wait for it (vk_pipeline_building_ms).
- The last wait of a full-screen draw with no picture to hold was 40 ms a draw, and two such draws made 86 ms in one frame. It is 40 ms a frame now.
- A one-shot vertex draw skipped at the end of its wait gets its log line however short the wait was (seven were skipped with no line, so their targets could not be named).
That run against the same cell of the earlier comparison (same exe path, cache state, route and settings; one run each): the load's worst wait of the draws in one frame 99 ms against 735 ms, no mesh wait over 13 ms against 524 and 206 ms. Two long frames remain in both runs (628 and 862 ms here, 660 and 854 ms there): two compute pipelines built at their first dispatch, about 395 and 397 ms.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A draw whose pipeline is still building is skipped when its pass is rendered every frame and waits otherwise, so that a lookup table or a volume generated once is not left empty. "Every frame" meant "in the last four frames", so after a loading screen or a full-screen menu every pass of the first frame read as new: each draw with a new pipeline waited on the replay thread, up to 30 s each, and a mesh draw, whose pipeline has no fast link, for its whole compile. A player's 0.7.1 log shows it at three area changes: frames of 954, 1015 and 684 ms with 853, 960 and 663 ms of pipeline wait, six mesh draws waiting 52 to 660 ms in passes whose targets had been rendered in thousands of frames.
The target history (now runtime/vk_target_frames.h) also counts the frames each target was rendered in, for as long as it is the same guest texture. From eight frames on its pass is per-frame however long ago the last one was, and its draws are skipped and built by the workers like any other (pipe_skip_steady, on by default). A target rendered for the first time still waits, now from one pool of 50 ms a presented frame that the full-screen draws share (pipe_frame_wait_ms; oneshot_budget = 0 gives the old wait back), a mesh draw 8 ms at most, and the wait for a one-shot draw's shader comes from the same pool (gfx_shader_await takes a limit). A full-screen draw with no picture left to hold waits 40 ms more instead of 500. One frame so waits under 100 ms for pipelines in all.
Every 600 frames with a miss the log says how many draws found a shader or pipeline missing, how long they held the replay thread in all and in the worst frame, by the kind of wait, and how many were skipped at its end.
tests/target_frames_check.py replays the area change of that log against the header. Not yet run in the game.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Eight commits: a draw that misses its pipeline has the rest of its submission read ahead, the library workers link
the pipelines of skipped draws, a pipeline whose fragment shader reads a target of its own pass is fast-linked like
any other, and diagnostics. The merge was clean. Their before and after matrix had not finished when this was merged.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Each frame the game sets one of two events to 0 from the CPU, submits its worker queues and the main queue, whose first command waits for that event to be 1, and signals 1 with nvnEventSignal once it has written what the frame's commands read. The recording of a submission takes the VRAM mirror sync and the uniform snapshot at its start and tested the wait a moment later, so a signal that arrived in between passed the wait and the frame was drawn from copies taken before the signal: single frames with the whole scene's shadowing missing (6 to 7 of 255 brighter, the sky and the interface unchanged) or a black scene, more of them the busier the CPU.
Every nvnEventSignal is now numbered and keeps the value it replaced, the recording notes the number before it takes its copies, and a signal with a later number does not count for that submission. The submission stops at the wait like any wait reached before its signal and goes on at once in a submission of its own, whose copies are taken after the signal. NX2NAVI_EVENT_AT_SYNC=0 gives the earlier test. NX2NAVI_EVO_TRACE=1 logs the order in which every frame's submissions, waits and signals were replayed, which is how the frames were found.
Measured with the flight recorder's thumbnails, 1,800 presented frames a run, standing in Hyrule Field by day with the lazy write-back on, DLSS Performance, frame generation 4x and HDR: the build before this change showed 34 single-frame spikes with eight busy threads beside the game, 20 with four and 8 with six, most of them in frames whose signal came during the replay; with this change two runs with eight busy threads showed 0 and 0, with 667 and 724 such frames. A session of the user's with the lazy write-back off ran 25,200 frames without a spike in its three automatic dumps while the change stopped 4,744 wait tests.
Not measured: runs of this same build with the switch off (the two tried failed for other reasons, a device loss during the boot beside another game and a load with cold caches), any run with the lazy write-back off under load, the frame rate and GPU time against the switch off, and the differential tests, which this change does not touch. The flicker with the lazy write-back off that was reported has not been reproduced in a capture; its mechanism is the same signal and occurs there too.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The next draw with a new pipeline was only found when the replay reached it, so the draws of an area's first frame,
or of a turn that brings several new materials into view, paid for their pipelines one after another while the
workers that could have built them together sat idle. The submission is recorded before it is replayed: the first
draw of a submission that finds a shader or pipeline missing now has the ops after it read (vk_exec.c vk_exec_ahead)
with the binds that decide a draw's shaders and shader libraries applied to a copy of the replay state. Each draw
there looks its shaders up as the real draw will, which also queues a shader that is not in memory, and names its
state to vk_pipeline_hint, which queues the pre-rasterization and fragment shader libraries that are missing. No
pipeline is entered and no key recorded, so a wrong guess costs two libraries at most. The draw's own lookup then
links on its thread when both libraries are there.
On a first visit to Kokiri Forest a scan read 6,900 ops in 0.78 ms on average (the longest 5.5 ms; a scan ends
after 3 ms), and the scans of the load and the walk named 377 states without a pipeline and queued 104 libraries
for them.
pipe_ahead = 0 (NX2NAVI_PIPE_AHEAD) turns it off.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
On Linux recorder_core reserved the last logical processor of the mask (an efficiency core on a hybrid
CPU) without its sibling, for the first thread to call vk_exec_record, and the first process-wide affinity
call took it back. plat_cpu_reserve there now takes the fast core that clocks highest with both of its
logical processors (none when fewer than four cores are fast), threads the holder starts get the other
processors, plat_process_affinity leaves the core with its holder, and a sweep every 300 presents moves
threads that inherited the core and gives it back once the holder has ended.
Shared code, so Windows changes too: the priority and the core go to the first thread that has spent half
a second in vk_exec_record (vk_exec.c rec_claim) instead of the first to call it, and the log names the
logical processors.
Pull request #358 by mooki000, taken as it is. The Windows files compile; nothing was run, and the Linux
code is not compiled here.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Every frame the render thread waited about 5 ms for the GPU after the frame's first submission, with the game
CPU-bound in every area. The wait is there so that the GPU has run that submission's passes before the game
rewrites what they read. But the submission also ends with the write-backs of what its shaders wrote, 60 MiB merged
into guest memory in 3 to 4 ms of GPU time, and the wait covered those too.
The write-backs now go into a part of their own. The part before them signals a second timeline, the executed
timeline, and the first-submission wait waits on that; the write-back part signals the timeline as before, so every
fence the game asks for still covers them, in the same queue order. wb_split = 0 keeps one part and the old wait.
Not measured in the game yet.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Main brought the fixes of eleven pull requests (the texture pool set sealed when an early replay part is submitted, every location reflected in interface arrays, RGB9E5 texels recognised for integer storage views, the upscaler a frame started with kept, GPU pass timestamps without vendor breadcrumbs, the Linux build, window and NGX fixes), render targets released when drawn over at another address, the vertex layout kept in the pipeline on newer AMD drivers, and the upscaler choice read once per evaluation.
Four hunks conflicted and take both sides: pool_sync returns for a sealed set before the per-frame creation budget's clock (vk_exec_draw.c); make_module keeps the test line's early_ok argument and main's module hash (vk_pipeline.c); a texture taken back counts as sampled again and records how long it was away, and the texture line logs the disuse counters and the overwritten render targets (vk_texture.c).
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A replay part handed to the queue early kept reading the bindless pool set while later parts of the same submission wrote new entries into it. That updates descriptors a pending command buffer reads, and on RADV a texture whose memory was mapped after the part was submitted is absent from its page tables. This caused a gfxhub read fault in the 128x128 pass of the first world frame in Kokiri Forest.
Seal the set when a part goes, take another ring set at the next sync that has entries to write, and keep recording the same part while every other set belongs to the submission. Document the descriptor lifetime contract in maxwell.md 4.9.
Fixes#257.
A session with per-pass GPU timestamps showed the game's own render passes and compute dispatches at 1.6 to 2.3 ms of GPU time a frame while the perf line's GPU time was 7.7 to 13.5 ms: most of the frame's GPU time is in commands the replay records between the marks, and nothing said which.
The timed frames now also get a timestamp at the start and the end of each part of a submission, and every stretch between two marks (or a mark and the part's edge) is a gap. The replay names the kind of commands it records in a gap: buffer and texture copies, fills and clears, the temporal upscaler, write-back merges and copies, depth twin copies, the texture pool marks, the mesh draw counts and block-compressed target copies, each with a size where it has one. The report adds the gaps' time per frame summed by the kinds noted in them (a gap nobody noted holds barriers and state only), the parts' own time for comparison with the perf line, and each kind's count and size per frame. Nothing runs outside the timed frames.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The texture pool refresh looks up every registered slot, so the textures of every area the game keeps loaded never
counted as idle and their memory only grew. Marked shaders now store into a buffer with one word per pool slot. Every
four submissions that recorded work, the replay copies it out and clears it, and the texture side notes when each
texture was last sampled. Under VRAM pressure or low commit, once idle textures are gone, live textures that come back
intact and were not sampled for 1800 presents go too. Their keys are held, so the pool refresh keeps the stand-in until
a shader samples that slot again, and the next refresh makes the texture again.
Fragment shaders that neither discard nor write depth, sample mask, stencil reference or memory keep early depth tests
(EarlyFragmentTests), so the marks do not take away early depth rejection. Alpha to coverage leaves the dynamic state
while marks are on, so each pipeline knows whether early tests are safe. Disuse eviction waits until every module that
indexes the pool was marked and two reads arrived. NX2NAVI_TEX_DISUSE=0 (tex_disuse = 0) turns it off.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 1a4c896150a748f9275c3024e96af3055db94af4)
Tested in the 0.6.1 test build (RTX 5090, Kokiri Forest), switching the frame rate changes off in turn: the foliage
flickered with split_idle on alone and not with it off, with render target image reuse on either way. split_idle sends
a submission's first part to the GPU at the first pass boundary after a draw or dispatch while the GPU is idle,
without the usual draw and time minimums; why parts cut that early make the foliage flicker is not known yet.
The settings file's two stacked descriptions of the split settings, left by the merge, are one again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A device lost seen on RTX 4090, 5090 and 5070 Ti cards in 0.5.0 and 0.6.0 (#177, #61, and a 0.6.0 test run): a
"read invalid" at the start of a 128x128 scene capture pass, re-rendered at area loads. destroy_tex purged the views of
the image and deferred the image, then freed its record at once, on whichever thread released the texture. The replay
keeps &e->pub as a render target for the rest of a command buffer (vk_exec.c resolve_target) and DLSS for the frame:
a pass that began again (each indirect mesh draw ends the pass) read the freed record and cached a view of the dead
image, which nothing removed. Vulkan gives a destroyed image's handle to a later image, which then got that view, and
the pass loaded its attachments from freed memory.
- destroy_tex marks the record gone and frees it through the deferred destruction (vk_defer_free, a host kind stamped
with the submission being recorded, as the image is).
- A gone texture's attachment view is made per command buffer (vk_exec.c view_of) or deferred with it
(vk_exec_target_view, for DLSS and the contact shadows), never cached.
- A new image drops any view still cached under its handle, and the log counts it ("cached views of a destroyed image").
No rendering changes: the views and images live exactly as long as before, the records slightly longer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the local fixes package (fix 11), measured in Kokiri Forest at 4K on an RTX 4090 over PCIe 3.0: 40.5 to 62.6 fps
native and 45.6 to 74.9 fps with DLSS Quality, the mirror copies from 6.5 to 0.3 ms of GPU time a frame.
- Pages a VRAM mirror wrote back into guest memory are not copied up into it again: the mirror sync reads page stamps
of their own that a mirror's write-back leaves alone (vk_gpu_written_back, decided at record time by
vk_mem_mirrored). This replaces the lookup table NX2NAVI_WB_NO_REUP had, keeps its name, and is now on by default.
- The game's flushes of mirrored memory are not copied where the write watch covers the region, as every page the game
wrote there is stamped already (47-57 MiB a frame); NX2NAVI_FLUSH_UPLOAD=1 copies them as before.
- Written storage ranges up to NX2NAVI_WB_MERGE_MAX (now 32 MiB) are merged in 4 MiB pieces, so a range is no longer
bound to what one dispatch covers; the snapshot pool keeps its budget-scaled size with 128 slots.
- A submission's first part goes to the GPU at the first pass boundary while the GPU is idle (split_idle), long passes
are split too (split_in_pass), and the part's time limit reads the clock less often.
- A render target that needs an image of its own takes the image of a released one with the same parameters, released
two or more frames before (tex_reuse); memory requirements are kept per parameter set; a released entry another
descriptor's entry replaced at its address is kept and taken back.
- The memory check reads a page's write-watch bit and stamp under the lock that keeps them together.
- The launcher reads the gamepad only while its own window is in front, and does not start a second copy of the game.
- make_game.py says why a Python package did not load, not only that it is missing.
The fix's per-pass GPU timing is left out: NX2NAVI_GPU_PASSES does the same in main.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The target history is now four-way set-associative and counts a target as recent when one of the last four frames
rendered it, so the scene passes, probe layers and alternating effect mips no longer read as never rendered and wait
for their pipelines one at a time when a new area is entered. NX2NAVI_ONESHOT_BUDGET=1 makes one-shot draws share a
wait budget per presented frame (100 ms for large passes, 300 ms for small ones) instead of waiting up to
ONESHOT_WAIT_MS each; it is off by default until play shows no target left empty that way.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lock_time (NX2NAVI_LOCK_TIME) makes the replay threads time each lock they have to wait for, kept per asking site (platform.h plat_lock_timing, plat_mutex_lock_at and plat_rmutex_lock_at carry the site), and the frame-time summary lists the sites with the most wait, the recorder's full-ring and drain waits, and the texture lock's holds over 1 ms with the code that took it. replay_time now also keeps each frame's replay ops: a frame over 25 ms charged to the replay says how many draws, binds and transfers it replayed, their time, its slowest op, and how much of the replay's wall time the thread was on the CPU. NX2NAVI_PROFILE_REPLAY=1 samples only the replay thread without call chains, and the profiler now follows the thread that replays instead of the one that started the back end.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Performance: draws reuse the bindings that did not change and record only the dynamic states that differ from what
the command buffer holds; small uploads come from a staging arena; finished submissions are held briefly and handed to
the queue in batches; the recorder thread can keep a CPU core of its own; ASTC block-linear textures are deswizzled on
the GPU; dynamic resolution steps the upscaler mode to hold a frame-time target. Each has a setting in nx2navi.ini
(bind_reuse, dyn_delta, stg_arena, submit_batch, recorder_core, recorder_prio, astc_gpu_deswizzle, dyn_res) and the
staging cache now frees what it holds at shutdown.
Capture: NX2NAVI_DUMP=key saves the frame after each press of Insert, with mesh draws listed by task, mesh and fragment
shader, depth test and face culling, and mesh_prims.txt giving each mesh draw's primitive, clipping and fragment counts.
All of it is off unless NX2NAVI_DUMP is set.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The event order (vk_exec.c) holds a submission at a wait whose signal has not been replayed. In Kokiri Forest dialogue several presents a second came with the frame's whole submission held (4 held submissions, nothing drawn since the last present): the present showed the picture from before, a flash of the whole picture, and the frame was drawn afterwards in the present thread's empty submission, next to the following frame. With the event order off (NX2NAVI_EVENT_ORDER=0) the same spot had no such frame in 3600.
present_held (vk_backend.c, synchronous replay only): at a present with held work, what can go on runs first in a submission of its own; if work is still held, the present waits and goes out after the submission that replays the last of it, at the next present, or after 8 submissions. present_held = 0 (NX2NAVI_PRESENT_HELD=0) turns it off.
held_sync (vk_exec.c vk_exec_record): a submission without commands of its own that resumes held work now gets the VRAM mirror sync before and the write-back after, and a uniform copy, like any submission. Before, that work read mirrors that had missed the CPU's latest writes, and its GPU writes were dropped from the write-back list by the next submission's vk_mem_mirror_begin. held_sync = 0 (NX2NAVI_HELD_SYNC=0) turns it off.
Not yet measured in play.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>