From 58fad9128d1336ea6516c3908e62909282237f35 Mon Sep 17 00:00:00 2001 From: chrg6 Date: Fri, 13 Nov 2026 19:10:28 -0500 Subject: [PATCH] Give a synchronous submission of a recording copy that another thread's submission still replays a clone of its own for its deferred binds, instead of rewriting the copy under that replay. nvnQueueSubmitCommands resolves a copy's deferred binds at the submit call: the handles the game's arrays hold now are written into the recording copy (nvn_api_cmd.c resolve_deferred). With the replay on the submitting thread, a second guest thread that submitted the same copy while the first one replayed it, or while the first copied it for a held continuation (vk_exec.c cont_new), rewrote those handles under that replay: the rest of it bound the next frame's textures and samplers, and a held continuation could be copied torn (F-2-03's part left after the private copies). Now each copy counts the synchronous submissions between their prepare and the end of their replay (copy_prefix.busy, which shares the 8 bytes pins had, so the prefix keeps its 56 bytes). nvn_submit_prepare takes copy_lock shared once for all handles: the first submission in flight resolves the copy in place, as before; one that finds another in flight, unless the copy is known to hold no deferred binds, resolves a clone made under that lock, with its texture pins counted, and nvn_submit_done frees it after the replay, as the asynchronous path does for a copy a queued submission still reads. A copy the game releases while a submission of it is in flight becomes a zombie and is freed at the next sweep after it ends (release_copies, zombie_sweep). The bind loop moved unchanged into runtime/nvn_submit_guard.h (nvn_resolve_binds) so a check can build it; it reports a corrupt record or a call past 16 deep to resolve_deferred, which ends the run with the messages it had, and keeps the first six changes' trace line. Being code the front end carries under its MIT header, the new header keeps that line and THIRD_PARTY_NOTICES.md's row names it. A clone is logged the first four times and at each power of two ("submits the recording ... while another thread's submission of it is replayed"). No timing change and no wait: the clone path runs only when two threads have one copy in flight at once. Not reproduced before the change; nobody knows yet whether the game ever submits one copy from two threads at once (the new line will say). Checks that passed: the new tests/submit_clone_check (four threads submitting two copies 200,000 times each while their arrays change: no replay saw its handles change, both paths ran, the counts ended at 0; the unguarded 0.10.1 shape through the same harness: about 450,000 of 800,000 replays saw them change; nvn_resolve_binds step by step), copy_sweep_check (its stand-in prefix follows the new layout), pass_cc_check on every runtime file, no new -Wall -Wextra warnings in nvn_api_cmd.c and nvn_api_core.c. Co-Authored-By: Claude Opus 5.5 --- THIRD_PARTY_NOTICES.md | 2 +- docs/building.md | 7 ++ runtime/nvn_api.h | 5 +- runtime/nvn_api_cmd.c | 130 ++++++++++++++---------- runtime/nvn_api_core.c | 17 +++- runtime/nvn_submit_guard.h | 69 +++++++++++++ tests/copy_sweep_check.py | 2 +- tests/submit_clone_check.c | 190 ++++++++++++++++++++++++++++++++++++ tests/submit_clone_check.py | 59 +++++++++++ 9 files changed, 426 insertions(+), 55 deletions(-) create mode 100644 runtime/nvn_submit_guard.h create mode 100644 tests/submit_clone_check.c create mode 100644 tests/submit_clone_check.py diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index 1e7a164a..ed3ecc46 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -16,7 +16,7 @@ Code copied or adapted from another project is listed here with its source, the | oot-mod (Ocarina of Time Switch 2 ultrawide mod): the matrix-update sites of sead's perspective projection and the 0.01 tolerance around 16:9, the 2D canvas site with the full-screen layout projection test and the screen vtables of the capture backgrounds, the movie layer viewport site, their instruction words and what each hook does there (rewritten without branches for the generator's patch sites) | https://github.com/craftycorvid/oot-mod @ `c5ccd1cd57e8219aea3be1e758221795fa5b7ec8` (`build.py` hooks), Copyright (c) 2026 Ivan0xFF | MIT | `recomp/patches-v111.txt` (sets wide_aspect, wide_hud, wide_movie), `runtime/widescreen.c` | | MacNativeGames NX2navi virtual-core preemption: asynchronous preemption of an owner in top-level game code, priority inheritance through guest mutexes, the SDK's preemption-disable protocol (TLR+0x100/+0x102, SVC 0x36), the core-wait diagnostics | https://github.com/ly05010419/MacNativeGames @ `33e8fa83052ea1ae09a3583137008e9797e12950` (`runtime/threads.c`, commits 8bf0893 and 19730cd), Copyright (c) 2026 MacNativeGames contributors | MIT | `runtime/kernel.c` (SuspendThread and a register-saving stub replace SIGUSR2), `runtime/hostfn.c`, `runtime/svc.c` (host depth) | | MacNativeGames RomFS index builder (`NX2NRFS1` layout) | https://github.com/ly05010419/MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`scripts/prepare_romfs_backend.py`), Copyright (c) 2026 MacNativeGames contributors | MIT | `tools/romfs_index.py` | -| d2r-mobile host thread environment (TLR, TLS, thread object), the origin of MacNativeGames' thread layer; d2r-mobile NVN host object layouts, command record format and NVN front end, as adapted for NX2navi in MacNativeGames' `d2r_gfx` copy | https://github.com/jhaizhou-ops/d2r-mobile @ `53e7c5bb9ca3ea53e2160005be8fbe920cd9cb53` (`dm/host/thread.c`, `dm/host/hle_thread.c`); the NVN layouts and front end via https://github.com/ly05010419/MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`third_party/d2r_gfx/gfx/nvn/nvn_obj.h`, the two types of `nvn_tex.h`, `hle_nvn.c`, `nx2navi_nvn_ext.c`, `nvn_impl.h`, `nvn_hmap.h`, `nvn_pool_index.h`, the request checks and mapping semantics of `nx2navi_vpool.c`, the generation stamps of `nvn_dirty.c`, texture mirroring and texel copies of `nvn_mtl.m`; licence in `third_party/d2r_gfx/LICENSE`), Copyright (c) 2026 jhaizhou | MIT | `runtime/kernel.c`; `runtime/nvn_obj.h`; `runtime/nvn_api.h`, `runtime/nvn_api_core.c`, `runtime/nvn_hmap.h`, `runtime/nvn_pool_index.h`; `runtime/nvn_api_obj.c` (also `nvn_pace.h`); `runtime/nvn_api_cmd.c`, `runtime/nvn_ext.c`; `runtime/vk_memory.c`, `runtime/vk_texture.c`; `runtime/vk_sync.c` (side effects written after GPU completion, `hle_nvn.c`); `runtime/vk_pipeline.c` (pipeline key and state mapping of `nvn_mtl.m` `pso_key_fill` / `pso_desc`, its `vfmt` vertex formats and the missing-varying fix-up of `msl_varyings`); `runtime/vk_exec.c`, `runtime/vk_exec.h` (op loop, pure-state ops and state defaults of `nvn_mtl.m` `mtl_exec` / `state_op` / `state_reset`); `runtime/vk_exec_xfer.c` (copy, clear, event, counter and conditional-render semantics of `nvn_mtl.m` `copy_t2b` / `copy_b2t` / `copy_t2t` / `clear_tex` / `cond_eval`); `runtime/vk_exec_draw.c` (bind ops, draw and dispatch flow, vertex streams of `nvn_mtl.m` `state_op` / `bind_vertex_streams` / `draw_inner` / `dispatch`) | +| d2r-mobile host thread environment (TLR, TLS, thread object), the origin of MacNativeGames' thread layer; d2r-mobile NVN host object layouts, command record format and NVN front end, as adapted for NX2navi in MacNativeGames' `d2r_gfx` copy | https://github.com/jhaizhou-ops/d2r-mobile @ `53e7c5bb9ca3ea53e2160005be8fbe920cd9cb53` (`dm/host/thread.c`, `dm/host/hle_thread.c`); the NVN layouts and front end via https://github.com/ly05010419/MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`third_party/d2r_gfx/gfx/nvn/nvn_obj.h`, the two types of `nvn_tex.h`, `hle_nvn.c`, `nx2navi_nvn_ext.c`, `nvn_impl.h`, `nvn_hmap.h`, `nvn_pool_index.h`, the request checks and mapping semantics of `nx2navi_vpool.c`, the generation stamps of `nvn_dirty.c`, texture mirroring and texel copies of `nvn_mtl.m`; licence in `third_party/d2r_gfx/LICENSE`), Copyright (c) 2026 jhaizhou | MIT | `runtime/kernel.c`; `runtime/nvn_obj.h`; `runtime/nvn_api.h`, `runtime/nvn_api_core.c`, `runtime/nvn_hmap.h`, `runtime/nvn_pool_index.h`; `runtime/nvn_api_obj.c` (also `nvn_pace.h`); `runtime/nvn_api_cmd.c` (also `nvn_submit_guard.h`), `runtime/nvn_ext.c`; `runtime/vk_memory.c`, `runtime/vk_texture.c`; `runtime/vk_sync.c` (side effects written after GPU completion, `hle_nvn.c`); `runtime/vk_pipeline.c` (pipeline key and state mapping of `nvn_mtl.m` `pso_key_fill` / `pso_desc`, its `vfmt` vertex formats and the missing-varying fix-up of `msl_varyings`); `runtime/vk_exec.c`, `runtime/vk_exec.h` (op loop, pure-state ops and state defaults of `nvn_mtl.m` `mtl_exec` / `state_op` / `state_reset`); `runtime/vk_exec_xfer.c` (copy, clear, event, counter and conditional-render semantics of `nvn_mtl.m` `copy_t2b` / `copy_b2t` / `copy_t2t` / `clear_tex` / `cond_eval`); `runtime/vk_exec_draw.c` (bind ops, draw and dispatch flow, vertex streams of `nvn_mtl.m` `state_op` / `bind_vertex_streams` / `draw_inner` / `dispatch`) | | nxnative NVN front end (d2r-mobile's successor): texture view depth-stencil mode bits, stencil-view rule, `Downsample` and indirect draw and dispatch records | https://github.com/seedds/nxnative @ `d5b380b3a80495609c2e751bb994911a638935ad` (`dm/gfx/nvn/hle_nvn.c`, `nvn_obj.h`, `nvn_mtl.m`), Copyright (c) 2026 jhaizhou | MIT | `runtime/nvn_obj.h`, `runtime/nvn_ext.c`, `runtime/nvn_api_cmd.c`, `runtime/vk_exec_draw.c` | | nxnative spin wait search and gate: loops that only another thread can end, polls that take a spin lock, the gate's wait and hand-off rule; MacNativeGames' NX2navi port of the gate | https://github.com/seedds/nxnative @ `6d349844880eb0527be77d7e0e4b8d999c0ae2f4` (`dm/tools/patch_nso.py` "Spin waits", `dm/tools/tests/test_spin_waits.py`, `dm/host/hle_thread.c` dm_spin_wait), Copyright (c) 2026 seedds; https://github.com/ly05010419/MacNativeGames @ `a038e6be7608e6f4eba6fc0381026cebf24ef4a3` (`runtime/threads.c` runtime_spin_wait), Copyright (c) 2026 MacNativeGames contributors | MIT | `tools/spin_waits.py`, `tests/spin_waits_test.py`, `runtime/kernel.c` (rt_spin_gate) | | NVN type definitions reconstructed from Darksiders II DWARF information, with MacNativeGames' NX2navi size annotations | h1k421, via https://github.com/ly05010419/MacNativeGames @ `b75be8cfa704df3701e86130c8ced268e79d6b11` (`third_party/d2r_gfx/gfx/nvn/include/nvn.h`), Copyright (c) 2020 h1k421 | MIT | `runtime/nvn.h` (unmodified) | diff --git a/docs/building.md b/docs/building.md index 7c94894a..b2108b6d 100644 --- a/docs/building.md +++ b/docs/building.md @@ -781,6 +781,13 @@ replaying threads, some replays holding the lock 40 ms, over 20,000 frames; ever order, never during a replay and before any replay that took the lock after its present, and no present waits for a replay. No GPU, no game files. +A change to a synchronous submission's deferred binds (`runtime/nvn_submit_guard.h`, `runtime/nvn_api_cmd.c` +`nvn_submit_prepare`, `nvn_submit_done`, `resolve_deferred`) passes `python tests\submit_clone_check.py`, which builds the +header with the platform layer: four threads submit two recording copies while another rewrites the arrays their +deferred binds read, and no replay sees its handles change while both the in-place and the clone path run; then the +bind resolution step by step (changed and unchanged arrays, binds it leaves alone, calls 16 deep and not 17, a corrupt +record). No GPU, no game files. + A change to how `runtime/memory.c` `mem_copy_readable` reads the lazy heap (`NX2NAVI_HEAP_LAZY=1`) passes `python tests\heap_lazy_copy_check.py`, which builds `memory.c` with the Windows platform layer and its fault handler: every copy returns, a heap page nothing touched yet reads zero and stays reserved, and pages past the heap's end are diff --git a/runtime/nvn_api.h b/runtime/nvn_api.h index 97ce5eee..42524c7a 100644 --- a/runtime/nvn_api.h +++ b/runtime/nvn_api.h @@ -60,7 +60,10 @@ void nvn_guest_written(uint64_t addr, uint64_t size); // the host or the ga // NX2NAVI_TRACE=perf: MemoryPoolFlushMappedRange [0] and BufferFlushMappedRange [1] calls [0] and bytes [1] by the pool's // CPU class (0 CPU_CACHED, 1 CPU_UNCACHED, 2 other) since the previous call. void nvn_flush_stats(uint64_t out[2][3][2]); -void nvn_submit_prepare(const uint64_t *handles, int n); // weak no-op; the command-buffer slice reads deferred binds +// Synchronous replay: the command-buffer slice reads the deferred binds and writes the handles to replay to out (a +// clone where another thread's submission of the recording is in flight); weak: a copy of handles. +void nvn_submit_prepare(const uint64_t *handles, uint64_t *out, int n); +void nvn_submit_done(const uint64_t *handles, const uint64_t *out, int n); // after the replay of out; weak no-op // Asynchronous replay: as nvn_submit_prepare, writing the handles to replay to out (a clone where a still queued // submission uses the same recording with other deferred binds) void nvn_submit_prepare_async(const uint64_t *handles, uint64_t *out, int n); diff --git a/runtime/nvn_api_cmd.c b/runtime/nvn_api_cmd.c index e7828ed3..8d759eed 100644 --- a/runtime/nvn_api_cmd.c +++ b/runtime/nvn_api_cmd.c @@ -20,6 +20,7 @@ #include "host.h" #include "nvn_api.h" #include "nvn_hmap.h" +#include "nvn_submit_guard.h" #include "platform/platform.h" #define CMD_MAGIC MAGIC('C', 'M', 'D', 'B') @@ -409,10 +410,13 @@ void nvn_pins_stats(uint64_t *live, uint64_t *freed) { // adding control memory again releases the copies recorded there without walking every live copy. 4 KiB pages cost // about 500 lookups per AddControlMemory under the exclusive lock (1.4.3 field: 53 calls a frame, 2 MiB ranges, 3.7 // copies per page), and the recording threads waited on it. -// deferred: 0 unknown, 1 none, 2 some; pins: texture pins its ops name, counted in their refs (pin_refs); used_min: the -// minute of its last use (copy_age) -typedef struct { uint64_t magic, bytes, made_at, used_at, deferred, pins, used_min; } copy_prefix; +// deferred: 0 unknown, 1 none, 2 some; used_min: the minute of its last use (copy_age); pins: texture pins its ops name, +// counted in their refs (pin_refs); busy: synchronous submissions of it between their prepare and the end of their +// replay (nvn_submit_prepare, nvn_submit_done; atomic, see nvn_submit_guard.h). +typedef struct { uint64_t magic, bytes, made_at, used_at, deferred, used_min; uint32_t pins, busy; } copy_prefix; _Static_assert(sizeof(copy_prefix) % 8 == 0, "records stay 8-byte aligned"); +// a session held over 200,000 copies (the note at COPY_AGE_MINS): busy shares the 8 bytes pins had +_Static_assert(sizeof(copy_prefix) == 56, "copy_prefix keeps its size"); typedef struct { uintptr_t lo, hi; uint8_t *copy; } cmd_copy; typedef struct { cmd_copy *v; uint32_t n, cap; } copy_bucket; static PlatMutex copy_lock = PLAT_MUTEX_INIT; // guards the two maps, the counters and every prefix @@ -444,7 +448,8 @@ static void copy_age(copy_prefix *pf, uint64_t min) { // copy_lock held, share if (was != min) age_add(was, pf, -1), age_add(min, pf, 1); } // Asynchronous replay: copies released by the game, and per-submission clones, that a queued submission still reads. -// Freed once gfx_replayed_submits() reaches their used_at. Empty with synchronous replay. +// Freed once gfx_replayed_submits() reaches their used_at. Synchronous replay: copies the game released while another +// of its threads' submission of them was still in flight (busy), freed at the next sweep after it ended. static copy_prefix **zombies; static uint32_t nzombies, cap_zombies; @@ -481,7 +486,7 @@ static void doom_free(copy_prefix *list) { // copy_lock not held static copy_prefix *zombie_sweep(uint64_t replayed, copy_prefix *doomed) { // copy_lock held; returns the doom chain for (uint32_t i = 0; i < nzombies;) { - if (zombies[i]->used_at <= replayed) { + if (zombies[i]->used_at <= replayed && !__atomic_load_n(&zombies[i]->busy, __ATOMIC_ACQUIRE)) { doomed = doom(doomed, zombies[i]); zombies[i] = zombies[--nzombies]; } else { @@ -546,7 +551,9 @@ static uint8_t *flatten_recording(const uint8_t *p, size_t n) { break; } } - *(copy_prefix *)out = (copy_prefix){.magic = COPY_MAGIC, .bytes = n, .pins = pins_named}; + if (pins_named > UINT32_MAX) rt_fatal("nvn: recording at %p names %llu texture pins", (const void *)p, + (unsigned long long)pins_named); + *(copy_prefix *)out = (copy_prefix){.magic = COPY_MAGIC, .bytes = n, .pins = (uint32_t)pins_named}; return out + sizeof(copy_prefix); } @@ -769,7 +776,8 @@ static void release_copies(uintptr_t lo, uintptr_t hi) { copy_bytes -= pf->bytes; age_add(pf->used_min, pf, -1); pf->magic = 0; - if (pf->used_at > replayed) zombie_add(pf); // a queued submission still reads it + // a queued submission still reads it, or another thread's synchronous one is in flight + if (pf->used_at > replayed || __atomic_load_n(&pf->busy, __ATOMIC_ACQUIRE)) zombie_add(pf); else doomed = doom(doomed, pf); b->v[i] = b->v[--b->n]; } @@ -1417,50 +1425,74 @@ NVN_API(nvnCommandBufferSetPatchSize) { (void)c; } // ---------- submit // What the console reads at the submit call is read here, on the game thread: every handle must be a live recording, // and deferred binds take the handles the game's arrays hold now (the game fills its tables for the next frame as soon -// as the submit returns). The result is written into the recording copy. -static uint64_t def_reads, def_changed, n_clones; +// as the submit returns). The result is written into the recording copy, or into a clone of it (nvn_submit_guard.h). +static uint64_t def_reads, def_changed, n_clones, n_sync_clones; -static int resolve_deferred(const uint8_t *p, int depth) { // returns the number of deferred binds - int nd = 0; - for (;;) { - op_hdr h; - memcpy(&h, p, sizeof h); - if (h.size < 8 || (h.size & 7) || h.op >= OP_COUNT_) - rt_fatal("nvn: corrupt recording at %p while resolving deferred binds (op %u size %u)", (const void *)p, - h.op, h.size); - if (h.op == OP_END) return nd; - if (h.op == OP_JUMP) { - memcpy(&p, p + 8, 8); - continue; - } - if (h.op == OP_CALL) { - uint64_t t; - memcpy(&t, p + 8, 8); - if (depth >= 16) rt_fatal("nvn: deferred bind resolution: call depth over 16"); - nd += resolve_deferred((const uint8_t *)(uintptr_t)t, depth + 1); - } else if (h.op == OP_BIND_TEX || h.op == OP_BIND_IMG || h.op == OP_BIND_SAMP) { - p_bind_handles *b = (p_bind_handles *)(uintptr_t)(p + 8); - if (b->src && b->n > 0 && b->n <= 32) { - nd++; - uint64_t cur[32]; - memcpy(cur, (const void *)(uintptr_t)b->src, 8 * (size_t)b->n); - __atomic_fetch_add(&def_reads, 1, __ATOMIC_RELAXED); - if (memcmp(cur, b->h, 8 * (size_t)b->n)) { - if (__atomic_fetch_add(&def_changed, 1, __ATOMIC_RELAXED) < 6) - NVN_LOG("deferred bind read new handles at submit: stage %d slot %d, %#llx at record -> %#llx", - b->stage, b->first, (unsigned long long)b->h[0], (unsigned long long)cur[0]); - memcpy(b->h, cur, 8 * (size_t)b->n); - } - } - } - p += h.size; - } +static void note_first_changes(const p_bind_handles *b, const uint64_t *now, uint64_t nth) { + if (nth < 6) + NVN_LOG("deferred bind read new handles at submit: stage %d slot %d, %#llx at record -> %#llx", b->stage, + b->first, (unsigned long long)b->h[0], (unsigned long long)now[0]); } -void nvn_submit_prepare(const uint64_t *handles, int n) { - uint64_t now = __atomic_add_fetch(&nsubmits, 1, __ATOMIC_RELAXED); - for (int i = 0; i < n; i++) copy_touch(handles[i], now, "QueueSubmitCommands"); - for (int i = 0; i < n; i++) resolve_deferred((const uint8_t *)(uintptr_t)handles[i], 0); +static int resolve_deferred(const uint8_t *p) { // returns the number of deferred binds + const uint8_t *bad = NULL; + int nd = nvn_resolve_binds(p, 0, &def_reads, &def_changed, note_first_changes, &bad); + if (nd >= 0) return nd; + op_hdr h; + memcpy(&h, bad, sizeof h); + if (h.op == OP_CALL && h.size >= 8 && !(h.size & 7)) rt_fatal("nvn: deferred bind resolution: call depth over 16"); + rt_fatal("nvn: corrupt recording at %p while resolving deferred binds (op %u size %u)", (const void *)bad, h.op, + h.size); +} + +// Synchronous replay. Each copy counts the submissions between this prepare and their nvn_submit_done (busy): the first +// in flight resolves the copy in place; one another thread makes while that is replayed resolves a clone of its own +// (nvn_submit_guard.h), unless the copy is known to hold no deferred binds. out: the handles to replay. +void nvn_submit_prepare(const uint64_t *handles, uint64_t *out, int n) { + uint64_t now = __atomic_add_fetch(&nsubmits, 1, __ATOMIC_RELAXED), min = copy_minute(), k = 0, cloned = 0; + plat_mutex_lock_shared(©_lock); // keeps every copy found here alive while it is cloned + for (int i = 0; i < n; i++) { + copy_prefix *pf = copy_find(handles[i]); + if (!pf) { + plat_mutex_unlock_shared(©_lock); + rt_fatal("nvn: QueueSubmitCommands: command handle %#llx is not a live recording (released or never recorded)", + (unsigned long long)handles[i]); + } + __atomic_store_n(&pf->used_at, now, __ATOMIC_RELAXED); + copy_age(pf, min); + out[i] = handles[i]; + if (nvn_copy_enter(&pf->busy) || __atomic_load_n(&pf->deferred, __ATOMIC_RELAXED) == 1) continue; + size_t sz = sizeof *pf + pf->bytes; + copy_prefix *cl = malloc(sz); + if (!cl) { + plat_mutex_unlock_shared(©_lock); + rt_fatal("nvn: out of memory (recording clone, %zu bytes)", sz); + } + memcpy(cl, pf, sz); + cl->magic = 0, cl->busy = 0; // not a live handle: the game never sees it + if (cl->pins) pin_refs((const uint8_t *)(cl + 1), 1); + nvn_copy_leave(&pf->busy); + out[i] = (uintptr_t)(cl + 1); + cloned = handles[i], k = __atomic_add_fetch(&n_sync_clones, 1, __ATOMIC_RELAXED); + } + plat_mutex_unlock_shared(©_lock); + for (int i = 0; i < n; i++) { + int nd = resolve_deferred((const uint8_t *)(uintptr_t)out[i]); + if (out[i] == handles[i]) + __atomic_store_n(&((copy_prefix *)(uintptr_t)out[i] - 1)->deferred, (uint64_t)(nd ? 2 : 1), __ATOMIC_RELAXED); + } + if (k && (k <= 4 || !(k & (k - 1)))) + rt_log("nvn: thread %u submits the recording %#llx while another thread's submission of it is replayed: its " + "deferred binds go into a copy of its own (%llu so far)", plat_thread_id(), (unsigned long long)cloned, + (unsigned long long)k); +} + +// After the replay of what nvn_submit_prepare handed out: clones are freed, copies resolved in place leave. +void nvn_submit_done(const uint64_t *handles, const uint64_t *out, int n) { + for (int i = 0; i < n; i++) { + if (out[i] != handles[i]) copy_free((copy_prefix *)(uintptr_t)out[i] - 1); + else nvn_copy_leave(&((copy_prefix *)(uintptr_t)handles[i] - 1)->busy); + } } // Asynchronous replay. A copy a queued submission still reads keeps that submission's deferred binds: when it holds @@ -1498,7 +1530,7 @@ void nvn_submit_prepare_async(const uint64_t *handles, uint64_t *out, int n) { plat_mutex_unlock(©_lock); doom_free(doomed); for (int i = 0; i < n; i++) { - int nd = resolve_deferred((const uint8_t *)(uintptr_t)out[i], 0); + int nd = resolve_deferred((const uint8_t *)(uintptr_t)out[i]); if (out[i] == handles[i]) ((copy_prefix *)(uintptr_t)out[i] - 1)->deferred = nd ? 2 : 1; } if (now % 7200 == 0) diff --git a/runtime/nvn_api_core.c b/runtime/nvn_api_core.c index 79553e29..d0ed2d92 100644 --- a/runtime/nvn_api_core.c +++ b/runtime/nvn_api_core.c @@ -64,7 +64,12 @@ static void ov_memcpy(Cpu *c) { } HOST_OVERRIDE("memcpy", ov_memcpy); -__attribute__((weak)) void nvn_submit_prepare(const uint64_t *handles, int n) { (void)handles, (void)n; } +__attribute__((weak)) void nvn_submit_prepare(const uint64_t *handles, uint64_t *out, int n) { + memcpy(out, handles, sizeof *out * (size_t)n); +} +__attribute__((weak)) void nvn_submit_done(const uint64_t *handles, const uint64_t *out, int n) { + (void)handles, (void)out, (void)n; +} __attribute__((weak)) void nvn_submit_prepare_async(const uint64_t *handles, uint64_t *out, int n) { memcpy(out, handles, sizeof *out * (size_t)n); } @@ -762,10 +767,16 @@ NVN_API(nvnQueueSubmitCommands) { plat_mutex_unlock(&submit_lock); if (hs != local) free(hs); } else { - nvn_submit_prepare(h, n); + // Another of the game's threads may submit the same recording meanwhile: a second submission in flight gets a + // clone of its own to resolve and replay (nvn_api_cmd.c nvn_submit_prepare), freed after (nvn_submit_done). + uint64_t local[32], *hs = n <= 32 ? local : malloc(sizeof *hs * (size_t)n); + if (!hs) rt_fatal("nvn: out of memory (submit of %d handles)", n); + nvn_submit_prepare(h, hs, n); gfx_submit_queue = (uintptr_t)q; - f = gfx_backend()->submit(h, n); + f = gfx_backend()->submit(hs, n); gfx_submit_queue = 0; + nvn_submit_done(h, hs, n); + if (hs != local) free(hs); } uint64_t prev = __atomic_load_n(&last_fence, __ATOMIC_RELAXED); while (f > prev && !__atomic_compare_exchange_n(&last_fence, &prev, f, 1, __ATOMIC_RELEASE, __ATOMIC_RELAXED)) {} diff --git a/runtime/nvn_submit_guard.h b/runtime/nvn_submit_guard.h new file mode 100644 index 00000000..1169fe60 --- /dev/null +++ b/runtime/nvn_submit_guard.h @@ -0,0 +1,69 @@ +// Contains code Copyright (c) 2026 jhaizhou (MIT; see THIRD_PARTY_NOTICES.md). +// The deferred binds of a recording copy at a synchronous submission, and what keeps a second submission of the same +// copy, made by another guest thread while the first is replayed, from rewriting it (nvn_api_cmd.c +// nvn_submit_prepare, nvn_submit_done). Header only, so that tests/submit_clone_check.c checks it without a device. +// +// A deferred bind takes the handles the game's array holds at the submit call, and they are written into the recording +// copy (nvn_resolve_binds). With the replay on the submitting thread, a second thread that submitted the same copy +// while the first one replayed it, or copied it for a held continuation (vk_exec.c cont_new), rewrote the handles that +// replay was reading: textures and samplers of the next frame halfway through it. So each copy counts the submissions +// between their prepare and the end of their replay (busy): the first one resolves in place, as before; one that finds +// another in flight resolves a clone of its own, freed after its replay, as the asynchronous path does for a copy a +// queued submission still reads. A clone taken while the first thread still resolves in place may hold a mix of old and +// new handles in its binds' arrays only, and its own resolve then writes every deferred array again: it ends whole. +#pragma once +#include +#include + +#include "nvn_obj.h" + +// 1: no other submission has the copy in flight; resolve it in place and call nvn_copy_leave after the replay. 0: +// another one has; take a clone and call nvn_copy_leave once the clone is made. +static inline int nvn_copy_enter(uint32_t *busy) { return __atomic_fetch_add(busy, 1, __ATOMIC_ACQ_REL) == 0; } +static inline void nvn_copy_leave(uint32_t *busy) { __atomic_fetch_sub(busy, 1, __ATOMIC_RELEASE); } + +// Told of a bind whose handles changed, before they are written: b holds the handles at record, now the game's, nth +// how many changed binds came before this one (the changed count). +typedef void (*NvnBindNote)(const p_bind_handles *b, const uint64_t *now, uint64_t nth); + +// Deferred binds of the recording at p take the handles the game's arrays hold now: OP_BIND_TEX, OP_BIND_IMG and +// OP_BIND_SAMP with src set and 0 < n <= 32, through OP_JUMP and OP_CALL (16 calls deep at most). reads and changed +// count the binds read and those whose handles changed (atomic adds; either may be NULL); note may be NULL. Returns +// how many deferred binds the recording has, or -1 with *bad at the record that stopped it: a corrupt one, or a call +// past the depth limit. +static inline int nvn_resolve_binds(const uint8_t *p, int depth, uint64_t *reads, uint64_t *changed, NvnBindNote note, + const uint8_t **bad) { + int nd = 0; + for (;;) { + op_hdr h; + memcpy(&h, p, sizeof h); + if (h.size < 8 || (h.size & 7) || h.op >= OP_COUNT_) return *bad = p, -1; + if (h.op == OP_END) return nd; + if (h.op == OP_JUMP) { + memcpy(&p, p + 8, 8); + continue; + } + if (h.op == OP_CALL) { + uint64_t t; + memcpy(&t, p + 8, 8); + if (depth >= 16) return *bad = p, -1; + int k = nvn_resolve_binds((const uint8_t *)(uintptr_t)t, depth + 1, reads, changed, note, bad); + if (k < 0) return k; + nd += k; + } else if (h.op == OP_BIND_TEX || h.op == OP_BIND_IMG || h.op == OP_BIND_SAMP) { + p_bind_handles *b = (p_bind_handles *)(uintptr_t)(p + 8); + if (b->src && b->n > 0 && b->n <= 32) { + nd++; + uint64_t cur[32]; + memcpy(cur, (const void *)(uintptr_t)b->src, 8 * (size_t)b->n); + if (reads) __atomic_fetch_add(reads, 1, __ATOMIC_RELAXED); + if (memcmp(cur, b->h, 8 * (size_t)b->n)) { + uint64_t nth = changed ? __atomic_fetch_add(changed, 1, __ATOMIC_RELAXED) : 0; + if (note) note(b, cur, nth); + memcpy(b->h, cur, 8 * (size_t)b->n); + } + } + } + p += h.size; + } +} diff --git a/tests/copy_sweep_check.py b/tests/copy_sweep_check.py index b9147c36..30cb21f3 100644 --- a/tests/copy_sweep_check.py +++ b/tests/copy_sweep_check.py @@ -42,7 +42,7 @@ static void plat_mutex_lock_shared(PlatMutex *m) { (void)m; } static void plat_mutex_unlock_shared(PlatMutex *m) { (void)m; } static int64_t plat_add64(volatile int64_t *p, int64_t v) { return __atomic_add_fetch(p, v, __ATOMIC_SEQ_CST); } static int cfg_int(const char *name) { (void)name; return 1; } -typedef struct { uint64_t magic, bytes, made_at, used_at, deferred, pins, used_min; } copy_prefix; +typedef struct { uint64_t magic, bytes, made_at, used_at, deferred, used_min; uint32_t pins, busy; } copy_prefix; static PlatMutex copy_lock = PLAT_MUTEX_INIT; #define COPY_IDLE_MINS 5 #define NCOPY 6000 diff --git a/tests/submit_clone_check.c b/tests/submit_clone_check.c new file mode 100644 index 00000000..747d11ad --- /dev/null +++ b/tests/submit_clone_check.c @@ -0,0 +1,190 @@ +// tests/submit_clone_check.py builds this: runtime/nvn_submit_guard.h, a synchronous submission's deferred binds when +// another guest thread may be replaying the same recording copy. Four threads submit two hand-built copies while a +// "game" thread rewrites the handle arrays their deferred binds read: a replay must see its copy's handles unchanged +// from its resolve to its end, the in-flight count must be back to 0, and both the in-place and the clone path must +// run. Then nvn_resolve_binds step by step. "old" resolves in place without the guard (0.10.1) and only prints what it +// sees, to show that the harness catches the race. +#include +#include +#include + +#include "platform/platform.h" +#include "nvn_submit_guard.h" + +#define COPIES 2 +#define BINDS 6 +#define THREADS 4 +#define ROUNDS 200000 + +static int fails; +#define CHECK(c, ...) do { if (!(c)) { printf("submit_clone_check: FAIL: " __VA_ARGS__); printf("\n"); fails++; } } while (0) + +static int old_mode; +static uint64_t tables[COPIES][BINDS][32]; +static uint8_t *copy[COPIES]; +static size_t copy_bytes; +static uint32_t busy[COPIES]; +static volatile int32_t stop; +static uint64_t torn, clones, in_place, reads, changed; + +static uint8_t *bind_rec(uint8_t *q, uint32_t op, int first, int n, const uint64_t *src) { + *(op_hdr *)q = (op_hdr){op, (uint32_t)(8 + sizeof(p_bind_handles))}; + p_bind_handles *h = (p_bind_handles *)(q + 8); + memset(h, 0, sizeof *h); + h->stage = 4, h->first = first, h->n = n, h->src = (uint64_t)(uintptr_t)src; + return q + 8 + sizeof(p_bind_handles); +} + +static uint8_t *make_copy(int c) { + size_t sz = 8 + BINDS * (8 + sizeof(p_bind_handles)) + 8; + uint8_t *p = calloc(1, sz), *q = p; + *(op_hdr *)q = (op_hdr){OP_BEGIN, 8}, q += 8; + for (int b = 0; b < BINDS; b++) q = bind_rec(q, OP_BIND_TEX, b * 4, 32, tables[c][b]); + *(op_hdr *)q = (op_hdr){OP_END, 8}; + copy_bytes = sz; + return p; +} + +static void game(void *arg) { + (void)arg; + for (uint64_t gen = 1; !__atomic_load_n(&stop, __ATOMIC_ACQUIRE); gen++) + for (int c = 0; c < COPIES; c++) + for (int b = 0; b < BINDS; b++) + for (int i = 0; i < 32; i++) __atomic_store_n(&tables[c][b][i], gen, __ATOMIC_RELAXED); +} + +static void snap(const uint8_t *p, uint64_t out[BINDS][32]) { + const uint8_t *q = p + 8; + for (int b = 0; b < BINDS; b++, q += 8 + sizeof(p_bind_handles)) + for (int i = 0; i < 32; i++) + out[b][i] = __atomic_load_n(&((const p_bind_handles *)(q + 8))->h[i], __ATOMIC_RELAXED); +} + +// nvn_api_cmd.c nvn_submit_prepare, the replay, nvn_submit_done +static void submitter(void *arg) { + uint64_t r = (uint64_t)(uintptr_t)arg * 0x2545F4914F6CDD1Dull + 7; + static uint64_t a[THREADS][BINDS][32], b2[THREADS][BINDS][32]; + int me = (int)(uintptr_t)arg - 1; + for (int k = 0; k < ROUNDS; k++) { + r ^= r << 13, r ^= r >> 7, r ^= r << 17; + int c = (int)(r % COPIES); + uint8_t *use = copy[c], *clone = NULL; + int alone = old_mode ? 1 : nvn_copy_enter(&busy[c]); + if (!alone) { + clone = malloc(copy_bytes); + memcpy(clone, copy[c], copy_bytes); + nvn_copy_leave(&busy[c]); + use = clone; + __atomic_add_fetch(&clones, 1, __ATOMIC_RELAXED); + } else { + __atomic_add_fetch(&in_place, 1, __ATOMIC_RELAXED); + } + const uint8_t *bad = NULL; + nvn_resolve_binds(use, 0, &reads, &changed, NULL, &bad); + snap(use, a[me]); // the replay reads the binds at its start ... + for (volatile int s = 0; s < 300; s++) {} + snap(use, b2[me]); // ... and at its end (or a held continuation copies them) + if (memcmp(a[me], b2[me], sizeof a[me])) __atomic_add_fetch(&torn, 1, __ATOMIC_RELAXED); + if (clone) free(clone); + else if (!old_mode) nvn_copy_leave(&busy[c]); + } +} + +static void stress(void) { + for (int c = 0; c < COPIES; c++) copy[c] = make_copy(c); + PlatThread *g = plat_thread_new(game, NULL, "game"), *t[THREADS]; + for (int i = 0; i < THREADS; i++) t[i] = plat_thread_new(submitter, (void *)(uintptr_t)(i + 1), "submit"); + for (int i = 0; i < THREADS; i++) plat_thread_join(t[i]); + __atomic_store_n(&stop, 1, __ATOMIC_RELEASE); + plat_thread_join(g); + printf("submit_clone_check: %s: %d submissions, %llu in place, %llu clones; %llu deferred binds read, %llu changed; " + "%llu replays saw their handles change under them\n", old_mode ? "old (prints only)" : "new", + THREADS * ROUNDS, (unsigned long long)in_place, (unsigned long long)clones, (unsigned long long)reads, + (unsigned long long)changed, (unsigned long long)torn); + if (old_mode) return; + for (int c = 0; c < COPIES; c++) CHECK(!busy[c], "copy %d left with %u submissions in flight", c, busy[c]); + CHECK(!torn, "%llu replays saw their handles change under them", (unsigned long long)torn); + CHECK(in_place && clones, "a path never ran (%llu in place, %llu clones)", (unsigned long long)in_place, + (unsigned long long)clones); +} + +// ---- nvn_resolve_binds, one thread +static uint64_t noted, noted_nth[8]; +static void note(const p_bind_handles *b, const uint64_t *now, uint64_t nth) { + if (noted < 8) noted_nth[noted] = nth; + noted++; + if (b->h[0] == now[0]) printf("submit_clone_check: FAIL: a note came after the handles were written\n"), fails++; +} + +static void steps(void) { + static uint64_t arr[4][32], called_arr[32]; + static uint8_t buf[2048], callee[512], chain[18][16]; + for (int i = 0; i < 32; i++) arr[0][i] = 100 + i, arr[1][i] = 200 + i, called_arr[i] = 300 + i; + uint8_t *q = buf; + *(op_hdr *)q = (op_hdr){OP_BEGIN, 8}, q += 8; + uint8_t *changed_rec = q; + q = bind_rec(q, OP_BIND_TEX, 0, 4, arr[0]); // its handles differ from the game's: read and written + uint8_t *same_rec = q; + q = bind_rec(q, OP_BIND_SAMP, 0, 4, arr[1]); // made equal below: read, not counted as changed + uint8_t *nosrc = q; + q = bind_rec(q, OP_BIND_IMG, 0, 4, NULL); // immediate: untouched + uint8_t *zero = q; + q = bind_rec(q, OP_BIND_TEX, 0, 0, arr[0]); // n 0: untouched + uint8_t *big = q; + q = bind_rec(q, OP_BIND_TEX, 0, 33, arr[0]); // n over 32: untouched + *(op_hdr *)q = (op_hdr){OP_CALL, 16}, memcpy(q + 8, &(uint64_t){(uint64_t)(uintptr_t)callee}, 8), q += 16; + *(op_hdr *)q = (op_hdr){OP_END, 8}; + uint8_t *c = callee; + *(op_hdr *)c = (op_hdr){OP_BEGIN, 8}, c += 8; + uint8_t *called_rec = c; + c = bind_rec(c, OP_BIND_TEX, 8, 2, called_arr); // reached through the call + *(op_hdr *)c = (op_hdr){OP_END, 8}; + memcpy(((p_bind_handles *)(same_rec + 8))->h, arr[1], 4 * 8); + for (int i = 0; i < 32; i++) ((p_bind_handles *)(nosrc + 8))->h[i] = 7, ((p_bind_handles *)(zero + 8))->h[i] = 7; + ((p_bind_handles *)(big + 8))->h[0] = 7; + uint64_t rd = 0, ch = 5; + const uint8_t *bad = NULL; + int nd = nvn_resolve_binds(buf, 0, &rd, &ch, note, &bad); + const p_bind_handles *bc = (const p_bind_handles *)(changed_rec + 8), *bs = (const p_bind_handles *)(same_rec + 8); + const p_bind_handles *bn = (const p_bind_handles *)(nosrc + 8), *bz = (const p_bind_handles *)(zero + 8); + const p_bind_handles *bb = (const p_bind_handles *)(big + 8), *bk = (const p_bind_handles *)(called_rec + 8); + CHECK(nd == 3 && rd == 3, "%d deferred binds, %llu read; 3 and 3 wanted", nd, (unsigned long long)rd); + CHECK(!memcmp(bc->h, arr[0], 4 * 8) && bc->h[4] == 0, "a changed array was not copied in, or more than n"); + CHECK(!memcmp(bs->h, arr[1], 4 * 8), "an unchanged array was changed"); + CHECK(bn->h[0] == 7 && bz->h[0] == 7 && bb->h[0] == 7, "a bind without src, with n 0 or n over 32 was written"); + CHECK(bk->h[0] == 300 && bk->h[1] == 301 && bk->h[2] == 0, "the bind behind OP_CALL was not resolved"); + CHECK(ch == 7 && noted == 2 && noted_nth[0] == 5 && noted_nth[1] == 6, "changed %llu, noted %llu (%llu, %llu): two " + "changes counted from 5 wanted", (unsigned long long)ch, (unsigned long long)noted, + (unsigned long long)noted_nth[0], (unsigned long long)noted_nth[1]); + rd = 0, noted = 0; + nd = nvn_resolve_binds(buf, 0, &rd, NULL, note, &bad); + CHECK(nd == 3 && rd == 3 && noted == 0, "a second resolve with nothing changed noted %llu", (unsigned long long)noted); + // 17 calls deep: the 17th call stops it + for (int i = 0; i < 17; i++) { + *(op_hdr *)chain[i] = (op_hdr){OP_CALL, 16}; + memcpy(chain[i] + 8, &(uint64_t){(uint64_t)(uintptr_t)chain[i + 1]}, 8); + } + *(op_hdr *)chain[17] = (op_hdr){OP_END, 8}; + bad = NULL; + CHECK(nvn_resolve_binds(chain[0], 0, NULL, NULL, NULL, &bad) == -1 && bad == chain[16], + "a call 17 deep was followed (bad %p, chain[16] %p)", (const void *)bad, (const void *)chain[16]); + *(op_hdr *)chain[16] = (op_hdr){OP_END, 8}; + CHECK(nvn_resolve_binds(chain[0], 0, NULL, NULL, NULL, &bad) == 0, "calls 16 deep were refused"); + uint8_t corrupt[32] = {0}; + *(op_hdr *)corrupt = (op_hdr){OP_BEGIN, 8}; + *(op_hdr *)(corrupt + 8) = (op_hdr){OP_BIND_TEX, 12}; // a size not a multiple of 8 + bad = NULL; + CHECK(nvn_resolve_binds(corrupt, 0, NULL, NULL, NULL, &bad) == -1 && bad == corrupt + 8, + "a corrupt record was not reported where it is"); + *(op_hdr *)(corrupt + 8) = (op_hdr){OP_COUNT_, 8}; + CHECK(nvn_resolve_binds(corrupt, 0, NULL, NULL, NULL, &bad) == -1 && bad == corrupt + 8, "an unknown op passed"); +} + +int main(int argc, char **argv) { + old_mode = argc > 1 && !strcmp(argv[1], "old"); + if (!old_mode) steps(); + stress(); + if (old_mode) return 0; + printf("submit_clone_check: %s\n", fails ? "FAILED" : "passed"); + return fails ? 1 : 0; +} diff --git a/tests/submit_clone_check.py b/tests/submit_clone_check.py new file mode 100644 index 00000000..ff3788a2 --- /dev/null +++ b/tests/submit_clone_check.py @@ -0,0 +1,59 @@ +"""Build and run tests/submit_clone_check.c: runtime/nvn_submit_guard.h, a synchronous submission's deferred binds when +another guest thread may be replaying the same recording copy (nvn_api_cmd.c nvn_submit_prepare, nvn_submit_done). +Four threads submit two hand-built copies 200,000 times each while a "game" thread rewrites the arrays their deferred +binds read: no replay sees its copy's handles change between its resolve and its end, the in-flight counts end at 0, +and both the in-place and the clone path run. nvn_resolve_binds step by step: a changed array is copied in and counted, +an unchanged one is not counted, binds without src, with n 0 or over 32 stay untouched, OP_CALL is followed 16 deep and +refused at 17, and a corrupt record is reported where it is. Then the unguarded 0.10.1 shape runs through the same +harness and only prints what it saw, to show that the harness catches the race. + +No GPU, no game files, any Python. + +Usage: submit_clone_check.py +""" +import os +import shutil +import subprocess +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +MSYS_BIN = Path(r'C:\msys64\ucrt64\bin') + + +def compiler(): + """$CC, else MSYS2 UCRT64's clang, else clang on PATH.""" + if os.environ.get('CC'): + return os.environ['CC'] + if os.name == 'nt' and (MSYS_BIN / 'clang.exe').exists(): + return str(MSYS_BIN / 'clang.exe') + return shutil.which('clang') or 'clang' + + +def main(): + work = ROOT / 'build' / 'test' / 'submit_clone' + work.mkdir(parents=True, exist_ok=True) + win = os.name == 'nt' + exe = work / ('submit_clone_check.exe' if win else 'submit_clone_check') + env = dict(os.environ) + if win and MSYS_BIN.exists(): + env['PATH'] = str(MSYS_BIN) + os.pathsep + env.get('PATH', '') + rt = ROOT / 'runtime' + plat = [rt / 'platform' / ('win32.c' if win else 'linux.c'), rt / 'platform' / 'common.c'] + libs = ['-lbcrypt', '-lws2_32', '-lpsapi', '-lwinmm', '-ldbghelp', '-lshell32', '-lole32', '-luser32', '-ladvapi32'] if win \ + else ['-lpthread'] + subprocess.check_call([compiler(), '-std=gnu11', '-O1', '-g', '-Wall', '-Wno-unused-function', '-fno-strict-aliasing', + '-march=x86-64-v3', '-I', str(rt), str(ROOT / 'tests' / 'submit_clone_check.c'), + *map(str, plat), '-o', str(exe), '-lm', *libs], env=env) + try: + rc = subprocess.run([str(exe)], env=env, timeout=300).returncode + if rc == 0: + subprocess.run([str(exe), 'old'], env=env, timeout=300) + return rc + except subprocess.TimeoutExpired: + print('submit_clone_check: FAIL: the check did not end within 300 s') + return 1 + + +if __name__ == '__main__': + sys.exit(main())