diff --git a/docs/architecture.md b/docs/architecture.md index f07624cd..6704149b 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -83,6 +83,12 @@ keeps per-core data without locks and relies on that, so the host models it (`ru - Uniform snapshot. The game rewrites per-object uniform blocks soon after it submits the frame that reads them; at the start of a submission's replay every uniform range it binds from CPU-written pools is copied, and the draws bind the copy (`vk_usnap.c`). +- Event order. Ocarina's queues hand work to each other with events, and its main queue waits each frame for a + signal the game gives from the CPU once it has written the frame's data. The replay stops a submission at a wait + whose signal has not been replayed and lets it go on after the signal (`vk_exec.c`, "event order across the game's + queues"); a CPU signal counts for a submission only when it came before the copies above were taken, and a + submission it comes too late for goes on at once in one of its own (`vk_exec_xfer.c`, "events the CPU signals"). + `NX2NAVI_EVO_TRACE=1` logs the order every frame was replayed in. - Driver data. The NVN driver keeps a per-stage constant bank (c0). The back end fills it as the driver does: storage and uniform tables, push constants and, for NVN 2 compute, the dispatch's group counts in bytes 0-0xb. - Storage buffers the shaders reach through pointer tables in memory are bound at the address the table holds diff --git a/docs/maxwell.md b/docs/maxwell.md index c51f3a32..8c0a4b01 100644 --- a/docs/maxwell.md +++ b/docs/maxwell.md @@ -88,6 +88,7 @@ Source shorthands: | 4.5 | **GPU-written index buffers** (8-bit indices written earlier in the same submission). | No `uint8` index type without `VK_EXT_index_type_uint8`; CPU widening reads stale bytes. | Widening on the GPU avoids it. Ours: CPU widening at record time; device-local byte indices skipped (check 9). | — | | 4.6 | **Compute shared memory**: 48 KiB per block on GM20B. | AMD 64 KiB. | Ours: `QueryComputeSharedMemorySize` from job (`Program.cs`). | OGD `clb1c0.h` | | 4.7 | **Virtual memory pools** map physical pool pages; the same bytes reachable at two GPU addresses. | Two `VkBuffer`s over one allocation need alias-aware barriers. | — | — | +| 4.8 | **Events gate queues**: a command buffer's `WaitEvent` holds its queue until a `SignalEvent` of another queue, or `nvnEventSignal` on the CPU, gives the event the value; the GPU reads what the commands behind the wait use only from then on. Ocarina's worker queues wait for the main queue's signals and the other way round, and the main queue waits each frame for a signal the CPU gives once the game has written the frame's data, before, after or during the main queue's submit call. | One queue in submission order, and the replay copies what a submission reads (VRAM mirror sync, uniform snapshot) when its recording starts. | A wait reached before its signal stops its submission, which goes on when the signal has been replayed (`vk_exec.c`, "event order across the game's queues"). A CPU signal counts for a submission only when it came before those copies; one that comes during the replay stops the submission, and it goes on at once in a submission of its own (`vk_exec_xfer.c`, "events the CPU signals"). Passing the wait on it drew single frames from copies taken before the game had finished writing: the scene's shadowing missing or the scene black, the interface as it was. | — | | 4.8 | **Command list calls**: a `CallCommands` / `CallList` is not a GPU subroutine. A call is stored as a control entry; at submit the CPU walks the entries and recurses into the called list, emitting gpfifo entries that point at the command words, which the GPU fetches later (prefetch can be switched off per entry). So list structure is fixed at submit, command words are read when fetched. | n/a | Ours: `OP_CALL` is walked at replay, `CallCommands` inlines the recording at record time (`vk_exec.c` `walk`, `nvn_api_cmd.c` `inline_recording`). Equivalent for control flow; deferred bind handles are the data read late (2.4). | — | | 4.9 | **The texture pool is read as the GPU goes** (bindless `TEX.B` handles index it), and the GPU runs a submission right after it is submitted, so it sees the pool as of about the submit; Ocarina registers the next submission's textures right after submitting. | A descriptor a pending command buffer may read must not be updated (update-after-bind allows only unused ones), and on RADV a view of memory allocated after a submit is not in that submit's page tables. | A replay part handed to the queue early (`NX2NAVI_SPLIT_DRAWS`) kept reading the set later parts wrote: GPU page fault (`gfxhub`, TCP read) on the 128×128 probe pass of Kokiri Forest's first world frame, about 40 s in, nearly every run on RX 9070 XT. Ours: one pool set per submission from a ring, sealed when a part goes; the next sync that has entries to write takes another set, and no part is split off while every other set belongs to the same submission (`vk_exec_draw.c` `vk_exec_pool_split`, `pool_take_set`). | — | diff --git a/runtime/cfg.c b/runtime/cfg.c index e7582b95..51620278 100644 --- a/runtime/cfg.c +++ b/runtime/cfg.c @@ -62,7 +62,7 @@ static const char *const known[] = { "SVC_SLOW_MS", "SHOTS_FULL", "SPEED_PROBE", "SPIN", "SPIN_HANDOFF_US", "SPLIT_DRAWS", "SPLIT_IDLE", "SPLIT_IN_PASS", "SPLIT_US", "SPV_MARK_TEST", "SPV_OVERRIDE", "STICK_DEADZONE", "STICK_SENS", "SWKBD_ANSWER", - "SET_AT", "SWKBD_TEST", "TAA", "TEST_PAD", "TEST_PAD_AT", "TEST_PREPARE_SHADERS", "TEX_ALIAS", "TEX_DISUSE", "TEX_GEN_UNDO", "TEX_MEMO", "TEX_VRAM_PRESSURE", "TEX_RELEASED_MB", "TEX_REUSE", "TEX_COMMIT_FREE_MB", "TEX_PROC_COMMIT_MB", "TEX_EVICT", "TEX_COMMIT_EVICT", "MIRROR_EVICT", "VRAM_BUDGET_MB", "TIMER_1MS", "TRACE", "TXD_TRANSPOSE", "UBO_SNAPSHOT", "UPSCALER", "VK_DYN_A2C", "VK_DYN_VERTEX", "VK_NAN_PRESERVE", "VK_NOOPT", "VK_NO_FLOAT64", + "SET_AT", "SWKBD_TEST", "TAA", "TEST_PAD", "TEST_PAD_AT", "TEST_PREPARE_SHADERS", "TEX_ALIAS", "TEX_DISUSE", "TEX_GEN_UNDO", "TEX_MEMO", "TEX_VRAM_PRESSURE", "TEX_RELEASED_MB", "TEX_REUSE", "TEX_COMMIT_FREE_MB", "TEX_PROC_COMMIT_MB", "TEX_EVICT", "TEX_COMMIT_EVICT", "MIRROR_EVICT", "VRAM_BUDGET_MB", "TIMER_1MS", "TRACE", "TXD_TRANSPOSE", "UBO_SNAPSHOT", "EVO_TRACE", "EVENT_AT_SYNC", "UPSCALER", "VK_DYN_A2C", "VK_DYN_VERTEX", "VK_NAN_PRESERVE", "VK_NOOPT", "VK_NO_FLOAT64", "VK_PRESENT", "VK_RECONVERGE", "VK_RECORDER", "VK_SSBO_ALIGN", "VK_SUBGROUP", "VK_SUBGROUP_PIN", "VK_VALIDATION", "VOLUME", "VRAM_HEADROOM", "VS_OUTS", "WALK_KEY", "WALK_SCALE", "WB_EXACT", "WB_FENCE_LIMIT", "WB_MERGE", "WB_NO_REUP", "WB_MERGE_MAX", "WB_PROBE", "WB_SPLIT", "WB_LAZY", "WB_LAZY_TOUCH", "WINDOW_SIZE", "XLATE", "XLATE_ID", "XLATE_KEEP", "XLATE_PROCS", "ZSTD_CHECK", "ZSTD_NATIVE", diff --git a/runtime/nvn_api.h b/runtime/nvn_api.h index 6438ee92..dbc6e367 100644 --- a/runtime/nvn_api.h +++ b/runtime/nvn_api.h @@ -56,6 +56,7 @@ void nvn_submit_prepare(const uint64_t *handles, int n); // weak no-op; the co void nvn_submit_prepare_async(const uint64_t *handles, uint64_t *out, int n); int gfx_replay_async(void); // 1: the back end replays submissions on its own thread uint64_t gfx_replayed_submits(void); // newest submit number (nvn_submit_count) replayed +void gfx_event_signalling(const void *value, uint32_t to); // nvnEventSignal is about to write `to` into the event's value #define NVN_HANDLE_TEX(h) ((int)((h) & 0xfffff)) // combined texture handle = texture id | sampler id << 20 #define NVN_HANDLE_SAMP(h) ((int)(((h) >> 20) & 0xfff)) extern GfxStats gfx_counters; // behind gfx_stats(); add with __atomic_fetch_add. frames: diff --git a/runtime/nvn_api_core.c b/runtime/nvn_api_core.c index 73f5d213..2c98ef8c 100644 --- a/runtime/nvn_api_core.c +++ b/runtime/nvn_api_core.c @@ -69,6 +69,7 @@ __attribute__((weak)) void nvn_submit_prepare_async(const uint64_t *handles, uin } __attribute__((weak)) int gfx_replay_async(void) { return 0; } __attribute__((weak)) uint64_t gfx_replayed_submits(void) { return nvn_submit_count(); } +__attribute__((weak)) void gfx_event_signalling(const void *value, uint32_t to) { (void)value, (void)to; } // ---------- pools: GPU window (section 3.1) and the address index static PlatMutex pool_lock = PLAT_MUTEX_INIT; // guards gva and index @@ -540,7 +541,10 @@ NVN_API(nvnEventGetValue) { ret_x(c, *((const ev_t *)arg_p(c, 0))->val); } NVN_API(nvnEventSignal) { const ev_t *e = arg_p(c, 0); uint32_t v = (uint32_t)arg_x(c, 2); - if ((uint32_t)arg_x(c, 1) == NVN_EVENT_SIGNAL_MODE_ADD) *e->val += v; + int add = (uint32_t)arg_x(c, 1) == NVN_EVENT_SIGNAL_MODE_ADD; + // before the write: the replay tells a signal that came after a submission's copies from one before them + gfx_event_signalling((const void *)e->val, add ? *e->val + v : v); + if (add) *e->val += v; else *e->val = v; } diff --git a/runtime/vk_backend.c b/runtime/vk_backend.c index 37d6bec7..cbd08768 100644 --- a/runtime/vk_backend.c +++ b/runtime/vk_backend.c @@ -308,6 +308,12 @@ static int vk_be_exec_wait(uint64_t value, uint64_t timeout_ns) { // The front end's view (nvn_api_core.c, nvn_api_cmd.c): replay is asynchronous only on this back end. int gfx_replay_async(void) { return gfx_backend() == &gfx_vulkan_backend && vk_replay_async(); } +// The front end's nvnEventSignal, before it writes (vk_exec_xfer.c "events the CPU signals"). +void gfx_event_signalling(const void *value, uint32_t to) { + vk_exec_event_cpu(value); + vk_exec_evt('c', value, to); +} + // The newest submit number (nvn_submit_count) whose replay is done: recording copies it used may be freed. uint64_t gfx_replayed_submits(void) { if (!gfx_replay_async()) return nvn_submit_count(); @@ -382,6 +388,10 @@ static uint64_t submit_sync(const uint64_t *handles, int n) { vk_present_drain(); vk_texture_new_submit(); uint64_t f = vk_sync_submit(handles, n); + // The submission stopped at a wait whose signal the CPU gave while it was being replayed (vk_exec_xfer.c "events + // the CPU signals"): it goes on now, in a submission of its own with copies taken after the signal, and that + // submission's value is the one the game's fence stands for. + if (n && vk_exec_event_at_sync() && vk_exec_held() && vk_exec_held_ready()) f = submit_sync(NULL, 0); if (PH.on && n) { if (!vk_exec_held()) ph_flush(&PH.after); else if (vk_exec_submit_seq - PH.since >= PH_MAX) ph_flush(&PH.capped); @@ -528,6 +538,7 @@ static void vk_be_present(const tex_t *t, const int crop[4], uint64_t at_ns) { if (!gfx_replay_async()) { uint64_t frame = __atomic_add_fetch(&gfx_counters.frames, 1, __ATOMIC_RELAXED); vk_dlssg_submitted(frame); + vk_exec_evt_frame(frame); perf_present(frame); // before vk_exec_frame_log, which resets the frame's exec counts the flight recorder reads uint64_t queued = kern_now_ns(); if (present_held_on()) { // present_held diff --git a/runtime/vk_exec.c b/runtime/vk_exec.c index 1e3824d5..ce15cf6c 100644 --- a/runtime/vk_exec.c +++ b/runtime/vk_exec.c @@ -976,6 +976,82 @@ int vk_exec_held(void) { return evo_pending; } static const op_hdr *walk(VkExec *ex, const op_hdr *h, int depth); +// ---- event order trace (NX2NAVI_EVO_TRACE=1, diagnostics) +// The order in which a frame's submissions, waits, signals and held parts were replayed, as a string of tokens: +// |. a submission of n command buffers on that queue; p it lines up behind held work of its queue; m a +// wait on event e that holds; u one that does not; h the submission stops there; s= a replayed signal; r +// held work of that queue goes on; w it stops at a wait again; d it is through; F it went on without its signal; +// c= the CPU signals an event (nvnEventSignal, on whatever thread, where the call came between the others); L +// a wait that holds only by a CPU signal that came after the submission took its copies (vk_exec_xfer.c "events the +// CPU signals"). Events and queues are numbered in the order they first appear; a value of 16 or more is written n, +// as a counter would make every frame's string its own. Every distinct string is logged once with its number, and the +// numbers of the frames are logged sixty to a line, so the frames that were replayed in another order than usual can +// be set against the pictures of the flight recorder. +#define EVT_KINDS 512 +static struct { + int on; + PlatMutex lock; + char buf[2048]; + int n, over; + const void *ev[64]; + int nev; + char *kind[EVT_KINDS]; + int nkind; + uint64_t line_first; + int line_n; + char line[512]; + int line_len; +} ET = {.on = -1, .lock = PLAT_MUTEX_INIT}; + +static int evt_on(void) { + if (ET.on < 0) ET.on = rt_knob("EVO_TRACE", "0"); + return ET.on; +} + +void vk_exec_evt(char what, const void *event, uint32_t value) { + if (!evt_on()) return; + char t[40]; + int e = -1; + plat_mutex_lock(&ET.lock); + if (event) { + for (e = 0; e < ET.nev && ET.ev[e] != event; e++) {} + if (e == ET.nev && ET.nev < 64) ET.ev[ET.nev++] = event; + } + if (what == 's' || what == 'c') snprintf(t, sizeof t, value < 16 ? "%c%d=%u " : "%c%d=n ", what, e, value); + else if (what == '|') snprintf(t, sizeof t, "|%d.%u ", e, value); + else if (event) snprintf(t, sizeof t, "%c%d ", what, e); + else snprintf(t, sizeof t, "%c ", what); + for (const char *s = t; *s; s++) { + if (ET.n >= (int)sizeof ET.buf - 1) { ET.over = 1; break; } + ET.buf[ET.n++] = *s; + } + plat_mutex_unlock(&ET.lock); +} + +void vk_exec_evt_frame(uint64_t frame) { + if (!evt_on()) return; + plat_mutex_lock(&ET.lock); + ET.buf[ET.n] = 0; + int k = 0; + while (k < ET.nkind && strcmp(ET.kind[k], ET.buf)) k++; + if (k == ET.nkind && ET.nkind < EVT_KINDS) { + size_t len = strlen(ET.buf) + 1; + if ((ET.kind[k] = malloc(len))) { + memcpy(ET.kind[k], ET.buf, len); + ET.nkind++; + rt_log("evo: order %d first at frame %llu%s: %s", k, (unsigned long long)frame, ET.over ? " (cut)" : "", ET.buf); + } + } + if (!ET.line_n) ET.line_first = frame, ET.line_len = 0; + ET.line_len += snprintf(ET.line + ET.line_len, sizeof ET.line - (size_t)ET.line_len, " %d", k); + if (++ET.line_n == 60 || ET.line_len > (int)sizeof ET.line - 16) { + rt_log("evo: frames %llu-%llu:%s", (unsigned long long)ET.line_first, (unsigned long long)frame, ET.line); + ET.line_n = 0; + } + ET.n = 0, ET.over = 0; + plat_mutex_unlock(&ET.lock); +} + static int evo_enabled(void) { if (evo_on < 0) { const char *e = cfg_get("NX2NAVI_EVENT_ORDER"); @@ -1069,6 +1145,7 @@ static int held_sync(void) { static int cont_walk(VkExec *ex, Cont *c) { if (c->at && !vk_exec_event_met(nvn_op_payload(c->at))) { evo_forced++; + vk_exec_evt('F', NULL, 0); static int logged; if (logged++ < 8) rt_log("vk: event order: a held submission goes on after %llu submissions without its signal", @@ -1108,7 +1185,9 @@ static void evo_resume(VkExec *ex) { evo_in = 1, evo_queue = evq[s].q; for (;;) { evo_resumed++; + vk_exec_evt('r', (const void *)evq[s].q, 0); if (cont_walk(ex, c)) { // waits again: the queue's state stays with it + vk_exec_evt('w', NULL, 0); vk_exec_pass_end(ex); if (!c->st && !(c->st = malloc(ST_HI - ST_LO))) rt_fatal("vk: out of memory (queue state)"); memcpy(c->st, (uint8_t *)ex + ST_LO, ST_HI - ST_LO); @@ -1118,6 +1197,7 @@ static void evo_resume(VkExec *ex) { evq[s].head = c->next; // done: the queue's next submission starts with the state this one left if (!evq[s].head) evq[s].tail = NULL; evo_pending--; + vk_exec_evt('d', NULL, 0); cont_free(c); c = evq[s].head; if (!c) break; @@ -1213,12 +1293,15 @@ static const op_hdr *walk(VkExec *ex, const op_hdr *h, int depth) { case OP_CLEAR_BUF: vk_exec_op_xfer(ex, h->op, a); break; - case OP_WAIT_EVENT: - if (!vk_exec_wait_event(a) && depth == 0 && evo_queue) { + case OP_WAIT_EVENT: { + int met = vk_exec_wait_event(a); + vk_exec_evt(met ? 'm' : 'u', (const void *)((const p_wait_event *)a)->val, 0); + if (!met && depth == 0 && evo_queue) { vk_exec_pass_end(ex); return h; } break; + } default: rt_fatal("vk: op %s has no replay handler", nvn_op_name(h->op)); } if (vk_exec_signalled && depth == 0 && !evo_in) evo_resume(ex); @@ -1296,6 +1379,8 @@ int vk_exec_record(VkCommandBuffer cb, const uint64_t *handles, int n, uint64_t int evo = evo_enabled() && gfx_submit_queue; int slot = evo ? evq_slot(gfx_submit_queue) : -1; if (evo && slot < 0) evo = 0; + vk_exec_evt('|', (const void *)gfx_submit_queue, (uint32_t)n); + vk_exec_event_sync(1); // before the mirror sync and the uniform copy: a CPU signal after this point came after them int synced = n || (evo_pending && held_sync() && vk_exec_held_ready()); // held_sync if (synced) { vk_mem_mirror_begin(); @@ -1317,7 +1402,7 @@ int vk_exec_record(VkCommandBuffer cb, const uint64_t *handles, int n, uint64_t if (!handles[i]) rt_fatal("vk: command handle %d of a submission is null", i); if (evo_pending) evo_resume(ex); // signals the CPU gave since, and holds past CONT_MAX if (evo && evq[slot].head) { // the queue waits: this submission lines up behind it - if (n) cont_push(slot, cont_new(handles, n, 0, NULL)), evo_held++; + if (n) cont_push(slot, cont_new(handles, n, 0, NULL)), evo_held++, vk_exec_evt('p', NULL, 0); } else { evo_queue = evo ? gfx_submit_queue : 0; for (int i = 0; i < n; i++) { @@ -1325,12 +1410,14 @@ int vk_exec_record(VkCommandBuffer cb, const uint64_t *handles, int n, uint64_t if (!stop) continue; cont_push(slot, cont_new(handles, n, i, stop)); // the rest of this submission waits for its signal evo_held++; + vk_exec_evt('h', NULL, 0); break; } evo_queue = 0; } vk_exec_pass_end(ex); x_here = 0; + vk_exec_event_sync(0); if (ex->recorded) vk_exec_barrier(ex), vk_exec_pool_marks(ex); // later submissions see this one's writes // The write-backs in a part of their own, after one that signals the executed timeline (vk_sync.c exec_tl): the // wait after a frame's first submission then ends when the submission's passes are through, not its write-backs. @@ -1353,9 +1440,11 @@ int vk_exec_record(VkCommandBuffer cb, const uint64_t *handles, int n, uint64_t if ((rt_trace & RT_TRACE_PERF) && n_subs >= 240) { rt_log("perf: replay command buffer parts per submission %.2f (%.2f first parts sent while the GPU was idle, %.2f " "splits inside a pass); event order: held %llu, resumed %llu (%llu without their signal), switches %llu, " - "held now %d", (double)(n_subs + n_parts) / (double)n_subs, (double)n_idle_parts / (double)n_subs, + "held now %d; waits a CPU signal met only after the submission's copies were taken: %llu", + (double)(n_subs + n_parts) / (double)n_subs, (double)n_idle_parts / (double)n_subs, (double)n_pass_splits / (double)n_subs, (unsigned long long)evo_held, (unsigned long long)evo_resumed, - (unsigned long long)evo_forced, (unsigned long long)evo_switches, evo_pending); + (unsigned long long)evo_forced, (unsigned long long)evo_switches, evo_pending, + (unsigned long long)vk_exec_event_late_take()); evo_held = evo_resumed = evo_forced = evo_switches = 0; n_subs = n_parts = n_idle_parts = n_pass_splits = 0; } diff --git a/runtime/vk_exec.h b/runtime/vk_exec.h index b551705d..6dbbfee6 100644 --- a/runtime/vk_exec.h +++ b/runtime/vk_exec.h @@ -231,10 +231,18 @@ void vk_exec_op_xfer(VkExec *ex, uint32_t op, const void *payload); int vk_exec_wait_event(const void *payload); int vk_exec_event_met(const void *payload); // the same test without counting extern int vk_exec_signalled; // set by every replayed SignalEvent; vk_exec.c clears it +// Events the CPU signals (vk_exec_xfer.c "events the CPU signals"). +void vk_exec_event_cpu(const void *event); // any thread: nvnEventSignal is about to write the event's value +void vk_exec_event_sync(int begin); // vk_exec_record: its mirror sync and uniform copy are taken now / it ends +int vk_exec_event_at_sync(void); // NX2NAVI_EVENT_AT_SYNC +uint64_t vk_exec_event_late_take(void); // waits only a CPU signal after the copies would have passed, since the last call // Since the last call, which resets them: waits, waits reached before their signal, those of them a later replayed // signal met, and the sum of how many submissions later it came. void vk_exec_event_stats(uint64_t *waits, uint64_t *unmet, uint64_t *late, uint64_t *late_subs); extern uint64_t vk_exec_submit_seq; // replay thread: submissions replayed so far +// NX2NAVI_EVO_TRACE=1 (diagnostics, vk_exec.c "event order trace"): one token per step of the replay's event order. +void vk_exec_evt(char what, const void *event, uint32_t value); +void vk_exec_evt_frame(uint64_t frame); // the game's present call: the frame's tokens are closed and logged int vk_exec_held(void); // replay thread: held submissions waiting for a signal (event order) int vk_exec_held_ready(void); // ... and one of them can go on now (vk_backend.c present_held) // vk_dlaa.c: once DLSS has scaled the scene this frame, passes larger than the scene's light target vk_dlaa_swap_img diff --git a/runtime/vk_exec_xfer.c b/runtime/vk_exec_xfer.c index f30dcce5..f5cd3a6c 100644 --- a/runtime/vk_exec_xfer.c +++ b/runtime/vk_exec_xfer.c @@ -366,11 +366,99 @@ static int ev_holds(uint32_t have, uint32_t mode, uint32_t want) { return mode == NVN_EVENT_WAIT_MODE_GEQUAL_WRAP ? (int32_t)(have - want) >= 0 : have == want; } +// ---- events the CPU signals +// Ocarina gates its main queue on the CPU: each frame the game sets one of two events to 0 (they take turns frame by +// frame), submits the worker queues' command buffers and the main queue's, whose first command waits for the event +// to be 1, and signals 1 from the CPU (nvnEventSignal) once it has finished writing what the frame's commands read. +// The signal comes before the main queue's submit call, after it, or while that call is replaying. On the console +// the GPU starts the queue at the signal and reads memory from then on. +// Here a submission's recording begins with the copies of what it reads: the VRAM mirrors' sync and the uniform +// snapshot (vk_exec_record), and the wait is tested when the walk reaches it, some hundred microseconds later. A +// signal that came in between passed the wait, and the frame was drawn from copies taken before the game had +// finished writing: single frames with the scene's shadowing missing or the whole scene black, the interface as it +// was, most of them when other work keeps the CPU busy, since the game's threads then finish later and the signal +// falls into the replay more often. +// So a wait is tested against the value the event had when the submission took its copies: every nvnEventSignal is +// numbered and keeps the value it replaced (vk_exec_event_cpu, before the write), vk_exec_record notes the number at +// its start (vk_exec_event_sync), and a signal with a later number does not count for that submission. The wait then +// stops the submission like any wait reached before its signal, and vk_backend.c submit_sync lets it go on at once +// in a submission of its own, whose copies are taken after the signal. +// NX2NAVI_EVENT_AT_SYNC=0: the value in memory when the walk reaches the wait, as before. +#define CPU_EV 16 +#define CPU_EV_HIST 8 +static struct { const void *val; uint64_t seq[CPU_EV_HIST]; uint32_t before[CPU_EV_HIST]; unsigned n; } cpu_ev[CPU_EV]; +static int cpu_ev_n; // cpu_ev_lock +static uint64_t cpu_ev_seq; // cpu_ev_lock: nvnEventSignal calls so far +static uint64_t ev_sync_seq = UINT64_MAX; // cpu_ev_lock: cpu_ev_seq at the recording's start; UINT64_MAX outside one +static PlatMutex cpu_ev_lock = PLAT_MUTEX_INIT; +static uint64_t ev_after_sync; // replay: waits only a signal after the copies would have passed + +int vk_exec_event_at_sync(void) { + static int on = -1; + if (on < 0) on = rt_knob("EVENT_AT_SYNC", "1"); + return on; +} + +void vk_exec_event_cpu(const void *event) { + plat_mutex_lock(&cpu_ev_lock); + int i = 0; + while (i < cpu_ev_n && cpu_ev[i].val != event) i++; + if (i == cpu_ev_n && cpu_ev_n < CPU_EV) cpu_ev[cpu_ev_n++].val = event, cpu_ev[i].n = 0; + cpu_ev_seq++; + if (i < cpu_ev_n) { + unsigned k = cpu_ev[i].n++ % CPU_EV_HIST; + cpu_ev[i].seq[k] = cpu_ev_seq, cpu_ev[i].before[k] = __atomic_load_n((const volatile uint32_t *)event, __ATOMIC_ACQUIRE); + } + plat_mutex_unlock(&cpu_ev_lock); +} + +void vk_exec_event_sync(int begin) { + plat_mutex_lock(&cpu_ev_lock); + ev_sync_seq = begin ? cpu_ev_seq : UINT64_MAX; + plat_mutex_unlock(&cpu_ev_lock); +} + +uint64_t vk_exec_event_late_take(void) { + uint64_t n = ev_after_sync; + ev_after_sync = 0; + return n; +} + +// The event's value in memory, and in *at_sync the value it had when the submission being recorded took its copies +// (the same outside a recording, and for an event the CPU has not signalled since). +static uint32_t ev_mem(volatile uint32_t *val, uint32_t *at_sync) { + uint32_t mem = __atomic_load_n(val, __ATOMIC_ACQUIRE); + *at_sync = mem; + plat_mutex_lock(&cpu_ev_lock); + if (ev_sync_seq != UINT64_MAX) + for (int i = 0; i < cpu_ev_n; i++) { + if (cpu_ev[i].val != (const void *)val) continue; + uint64_t first = UINT64_MAX; + unsigned n = cpu_ev[i].n < CPU_EV_HIST ? cpu_ev[i].n : CPU_EV_HIST; + for (unsigned k = 0; k < n; k++) // the earliest signal after the copies replaced the value they saw + if (cpu_ev[i].seq[k] > ev_sync_seq && cpu_ev[i].seq[k] < first) first = cpu_ev[i].seq[k], *at_sync = cpu_ev[i].before[k]; + break; + } + plat_mutex_unlock(&cpu_ev_lock); + return mem; +} + +// Whether the wait holds. *late: it holds only by a CPU signal that came after the submission's copies. +static int ev_met(const p_wait_event *p, uint32_t *mem, const uint32_t **seen, int *late) { + uint32_t at_sync; + *mem = ev_mem(p->val, &at_sync); + *seen = ev_slot(p->val, 0); + int by_seen = *seen && ev_holds(**seen, p->mode, p->value); + int now = ev_holds(*mem, p->mode, p->value) || by_seen, then = ev_holds(at_sync, p->mode, p->value) || by_seen; + *late = now && !then; + return vk_exec_event_at_sync() ? then : now; +} + int vk_exec_event_met(const void *payload) { - const p_wait_event *p = payload; - const uint32_t *seen = ev_slot(p->val, 0); - return ev_holds(__atomic_load_n(p->val, __ATOMIC_ACQUIRE), p->mode, p->value) || - (seen && ev_holds(*seen, p->mode, p->value)); + uint32_t mem; + const uint32_t *seen; + int late; + return ev_met(payload, &mem, &seen, &late); } int vk_exec_signalled; @@ -378,9 +466,15 @@ int vk_exec_signalled; int vk_exec_wait_event(const void *payload) { const p_wait_event *p = payload; ev_waits++; - uint32_t mem = __atomic_load_n(p->val, __ATOMIC_ACQUIRE); - const uint32_t *seen = ev_slot(p->val, 0); - if (ev_holds(mem, p->mode, p->value) || (seen && ev_holds(*seen, p->mode, p->value))) return 1; + uint32_t mem; + const uint32_t *seen; + int late; + int met = ev_met(p, &mem, &seen, &late); + if (late) { + ev_after_sync++; + vk_exec_evt('L', (const void *)p->val, 0); + } + if (met) return 1; ev_unmet++; if (ev_open_n == EV_OPEN) memmove(ev_open, ev_open + 1, sizeof ev_open - sizeof *ev_open), ev_open_n--; ev_open[ev_open_n].val = p->val, ev_open[ev_open_n].mode = p->mode, ev_open[ev_open_n].value = p->value; @@ -400,6 +494,7 @@ void vk_exec_event_stats(uint64_t *waits, uint64_t *unmet, uint64_t *late, uint6 static void signal_event(VkExec *ex, const p_signal_event *p) { uint32_t *seen = ev_slot(p->val, 1); *seen = p->mode == NVN_EVENT_SIGNAL_MODE_ADD ? *seen + p->value : p->value; + vk_exec_evt('s', (const void *)p->val, *seen); vk_exec_signalled = 1; for (int i = 0; i < ev_open_n;) if (ev_open[i].val == p->val && ev_holds(*seen, ev_open[i].mode, ev_open[i].value)) {