Copy the pages the CPU wrote into a VRAM mirror when their part is handed to the queue, behind NX2NAVI_MIRROR_CAPTURE_KB.

The prologue of a part copies the pages the CPU wrote since a mirror's last sync from guest memory into the mirror when
the GPU runs it, a frame or more after the game's submit when the GPU is behind: the GPU then reads what the game
wrote there meanwhile, values meant for a later frame. With NX2NAVI_MIRROR_CAPTURE_KB=n the CPU copies up to n KiB of
those pages per part into a host buffer of the submission when the part is handed over, and the prologue copies them
from there. A pool keeps the GPU-time copy while a write-back of its mirror into guest memory is still on the GPU (its
fence, now kept per pool), when it has a twin (the device address table's writes reach guest memory unrecorded), and
for a range with a page that cannot be read without a fault (memory.c mem_copy_readable, under the freeze lock, so a
lazily armed range stays with the GPU). With the default of every mirror twinned (NX2NAVI_GPA_TWIN_MAX_KB=0) it
copies nothing; it is a test switch for the replay's late reads, default 0.

Measured on an RTX 5090 with NX2NAVI_GPA_TWIN_MAX_KB=64 (mirrors of pools over 64 KiB without a twin), 75 s runs
with warm caches, NX2NAVI_MIRROR_CAPTURE_KB 0 against 8192: Kakariko 0.95 MiB a frame copied in 149 ranges for
0.21 ms of the replay thread, 116.7 against 118.4 fps (interval median 6.26 against 6.46 ms); Hyrule Field walk
1.38 MiB in 224 ranges, 0.37 ms, 110.2 against 107.2 fps, 1% low 83.3 against 78.1. 0.08 MiB a frame stayed with the
GPU-time copy. A first version that took the freeze lock and asked the system for each range's pages cost 2.9 to 4.3
ms a frame. A Kakariko run with NX2NAVI_REPLAY_ASYNC=1 and the copy on played through without errors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
chrg6
2026-11-11 19:37:20 -05:00
parent 74b7449518
commit 10dd325dbe
4 changed files with 144 additions and 6 deletions

View File

@@ -483,6 +483,34 @@ int mem_lazy_armed(uint64_t a, uint64_t n) {
return ok;
}
// Copies each of n ranges of guest memory, src[i] = {address, bytes}, to dst[i] when every page of it can be read
// without a fault; ok[i] says which were. An armed range, a probed region or a page the game made inaccessible is left
// out instead of faulting on the caller's thread (vk_memory.c mirror_capture runs inside a submission, where a fetch
// could not be sent). frz_lock is held over all of them, so nothing is armed or probed meanwhile; a range in regions
// with neither (lzr, prb: both change under that lock, with the pages' protection) is copied without asking the system
// for its pages' state. Returns how many were copied.
int mem_copy_readable(uint8_t *const *dst, const uint64_t (*src)[2], int n, uint8_t *ok) {
int got = 0;
plat_mutex_lock(&frz_lock);
for (int i = 0; i < n; i++) {
uint64_t a = src[i][0], len = src[i][1];
ok[i] = 0;
if (!len || a >= RT_ADDRESS_LIMIT || a + len > RT_ADDRESS_LIMIT) continue;
int quick = 1, can = 1;
for (uint64_t r = a >> FRZ_SHIFT; quick && r <= (a + len - 1) >> FRZ_SHIFT; r++)
quick = !__atomic_load_n(&lzr[r], __ATOMIC_RELAXED) && !__atomic_load_n(&prb[r], __ATOMIC_RELAXED);
for (uint64_t x = a, e; !quick && can && x < a + len; x = e) {
int st = plat_mem_state(x, a + len, &e);
can = (st == PLAT_PROT_RW || st == PLAT_PROT_READ) && e > x;
}
if (!can) continue;
memcpy(dst[i], (const void *)(uintptr_t)a, len);
ok[i] = 1, got++;
}
plat_mutex_unlock(&frz_lock);
return got;
}
// Releases what is armed in [a, a + n): before the buffers a fetch would read go away.
void mem_lazy_release(uint64_t a, uint64_t n) {
if (!__atomic_load_n(&lz_on, __ATOMIC_RELAXED) || !n) return;