Merge main's tail_early, replay_small and Testing rows into the 0.10.0 line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts:
#	runtime/cfg.c
#	runtime/vk_exec_xfer.c
This commit is contained in:
chrg6
2026-11-13 00:26:53 -05:00
17 changed files with 398 additions and 31 deletions

29
docs/releases/v0.9.3.md Normal file
View File

@@ -0,0 +1,29 @@
## Faster
- **Experimental: the next frame no longer waits for the end of the last one's graphics work** (`tail_early = 1` in
nx2navi.ini, off by default; contributed by epicmeme546). The game waits each frame until the graphics card has
finished the frame before. The last of that work is the upscaler (FSR 4, FSR 3.1 or DLSS) and the interface drawn
after it, and nothing the game reads comes out of it. With the setting on, the copies of what the card wrote for the
game go before that last stretch and the game goes on once they are there, so the card finishes the picture while
the next frame is already being prepared. The card runs the same work as before. On an RX 7900 XT with a Ryzen 7
5800X3D at 3440x1440, FSR 4 Native AA, Hyrule Field: 50.7 instead of 42.5 fps standing and 56.0 instead of 45.9 along
a route; with `replay_workers = 3` as well, 63.5 instead of 49.4 standing. With FSR 3.1, whose pass is shorter, 51.8
instead of 47.8 standing. Its pictures matched standing, along the route and through warps to five areas, but it is
new: leave it off to play, and report what you see if you try it. It does nothing without one of those upscalers,
and nothing yet on NVIDIA and Intel cards, where the lazy write-back is on.
- **Experimental: the replay workers take more of a frame off the game's render thread** (`replay_pipe` and
`replay_small` in nx2navi.ini, both 0 by default, both with `replay_workers`; contributed by epicmeme546). With
`replay_pipe = N`, 1 to 4, the thread goes on past up to N groups of draws its workers still record. With
`replay_small = N` as well, a group of N draws or more that is too small to share out goes whole to one worker. On
the same PC standing in Kokiri Forest, with `tail_early = 1`: 60.5 fps with 3 workers alone, 62.2 with 5 workers and
`replay_pipe = 4`, 63.2 with `replay_small = 16` added, against 45.4 with all four settings off. The pictures matched
there and through warps to five areas.
## Changed
- **The draw replay on several CPU cores is off by default again** (`replay_workers = 0`). With three workers on by
default in 0.9.1 and 0.9.2 some players' games ended with a lost graphics device; the cause is not found yet. The
settings menu has it under Game, Testing, as *CPU helper threads (experimental)*: 0 to 8, after a restart.
- **The other experimental performance settings have rows there too**, each a switch: *Helpers work ahead* (on is
`replay_pipe = 4`), *Helpers take small jobs* (on is `replay_small = 16`) and *Start next frame early*
(`tail_early`). A row whose setting does nothing without another one is greyed out and says which it needs.

View File

@@ -71,11 +71,29 @@
# of latency, and can show a dark or flickering frame now and then: the other thread reads vertex and buffer memory # of latency, and can show a dark or flickering frame now and then: the other thread reads vertex and buffer memory
# the game has sometimes rewritten by then. # the game has sometimes rewritten by then.
# replay_async = 0 # replay_async = 0
# Record the large groups of draws of a submission on this many replay worker threads beside the submitting one, 0 to # EXPERIMENTAL. Record the large groups of draws of a submission on this many replay worker threads beside the
# 16. Default: 3 (Hyrule Field standing on an RX 7900 XT: 51.7 instead of 43.6 fps). 0: the replay records everything # submitting one, 0 to 16. Default: 0, the replay records everything on one thread. 3 gave 51.7 instead of 43.6 fps
# on one thread, as before; try it if a picture looks wrong. With 1 a worker records each long run of draws while the # standing in Hyrule Field on an RX 7900 XT, and more than 3 gave nothing more; some players' games ended with a lost
# submitting thread waits for it: no faster, it checks that the pictures stay the same. # graphics device with it on, so it is off unless you turn it on (also in the menu: Game, Testing, CPU helper threads).
# replay_workers = 3 # With 1 a worker records each long run of draws while the submitting thread waits for it: no faster, it checks that
# the pictures stay the same.
# replay_workers = 0
# EXPERIMENTAL. 1 lets the game start on its next frame while the graphics card still runs the end of the last one: the
# upscaler (FSR 4, FSR 3.1 or DLSS) and the interface drawn after it. The graphics card runs the same work, with the
# copies of what it wrote for the game moved before that end, and the game's waits end once those copies and the
# results it reads are there. A frame whose end hands the game anything else runs as before. Default: 0. Standing in
# Hyrule Field on an RX 7900 XT with FSR 4: 50.7 instead of 42.5 fps, and 63.5 instead of 49.4 with replay_workers = 3.
# Does nothing without one of those upscalers, nor with the lazy write-back (the default on NVIDIA and Intel cards).
# Also in the menu: Game, Testing, Start next frame early.
# tail_early = 0
# EXPERIMENTAL, with replay_workers. replay_pipe = N (1 to 4, default 0): the submitting thread goes on past up to N
# groups its workers still record. replay_small = N (default 0, needs replay_pipe): a group of N draws or more that is
# too small to share out goes whole to one worker. Kokiri Forest standing on an RX 7900 XT with FSR 4, tail_early = 1,
# replay_workers = 5, replay_pipe = 4 and replay_small = 16: 63 instead of 45 fps with all of them off. Also in the
# menu: Game, Testing, as switches: Helpers work ahead (on is replay_pipe = 4) and Helpers take small jobs (on is
# replay_small = 16).
# replay_pipe = 0
# replay_small = 0
# EXPERIMENTAL, THE GAME CRASHES WITH 1. 1 lifts the virtual core gate: guest threads run without the console's limit # EXPERIMENTAL, THE GAME CRASHES WITH 1. 1 lifts the virtual core gate: guest threads run without the console's limit
# of one thread per core, and the game keeps per-core data without locks. With 1 the game crashes or hangs, # of one thread per core, and the game keeps per-core data without locks. With 1 the game crashes or hangs,
# usually at start. Keep 0 to play. # usually at start. Keep 0 to play.

View File

@@ -65,7 +65,7 @@ static const char *const known[] = {
"POOL_FAST", "POOL_GROUP", "POOL_GROUP_CHECK", "POOL_PRESNAP", "POOL_PRESNAP_RUNS", "POOL_FAST", "POOL_GROUP", "POOL_GROUP_CHECK", "POOL_PRESNAP", "POOL_PRESNAP_RUNS",
"SUBMIT_FINISH_GPU", "SVC_WAITS", "SNAP_WIDEN", "SUBMIT_FINISH_RESULTS", "SUBMIT_FINISH_KINDS", "FOLIAGE", "MIRROR_STAGE", "FLUSH_SKIP_CLEAN", "MIRROR_HOT", "MESH_EMULATION", "MESH_EMU_MB", "MESH_EMU_TASKS", "MESH_EMU_BATCH", "GUEST_OPT", "LOG_TEX_NEW", "DUMP_KEY", "VEC_INDEX_FIX", "PASS_TIMES", "SUBMIT_FINISH_GPU", "SVC_WAITS", "SNAP_WIDEN", "SUBMIT_FINISH_RESULTS", "SUBMIT_FINISH_KINDS", "FOLIAGE", "MIRROR_STAGE", "FLUSH_SKIP_CLEAN", "MIRROR_HOT", "MESH_EMULATION", "MESH_EMU_MB", "MESH_EMU_TASKS", "MESH_EMU_BATCH", "GUEST_OPT", "LOG_TEX_NEW", "DUMP_KEY", "VEC_INDEX_FIX", "PASS_TIMES",
"PROFILE_MAIN", "PROFILE_RENDER", "PROMPTS", "PUSH_SKIP", "PUSH_STAGE", "RDOC", "RDOC_DIR", "RDOC_DLL", "REALTIME", "REC_END", "REEXEC", "PROFILE_MAIN", "PROFILE_RENDER", "PROMPTS", "PUSH_SKIP", "PUSH_STAGE", "RDOC", "RDOC_DIR", "RDOC_DLL", "REALTIME", "REC_END", "REEXEC",
"REFLEX", "LOW_LATENCY", "RENDER_SCALE", "REPLAY_ASYNC", "REPLAY_PIPE", "REPLAY_RUNS", "REPLAY_WORKERS", "RTSS", "RTSS_ENUM", "SCENE_SHARE", "REFLEX", "LOW_LATENCY", "RENDER_SCALE", "REPLAY_ASYNC", "REPLAY_PIPE", "REPLAY_RUNS", "REPLAY_SMALL", "TAIL_EARLY", "REPLAY_WORKERS", "RTSS", "RTSS_ENUM", "SCENE_SHARE",
"PRESNAP_PLACE", "PRESNAP_PLACE",
"REPLAY_CHECK", "RESOLVE_CHECK", "RESTART_WAIT", "ROMFS", "ROMFS_INDEX", "ROMFS_MMAP", "ROMFS_MODS", "ROMFS_PATCHES", "REPLAY_CHECK", "RESOLVE_CHECK", "RESTART_WAIT", "ROMFS", "ROMFS_INDEX", "ROMFS_MMAP", "ROMFS_MODS", "ROMFS_PATCHES",
"ROMFS_PROOF_DIR", "ROMFS_VERIFY", "RSTICK_DEADZONE", "RUMBLE", "RUMBLE_CURVE", "RUMBLE_HD", "RUMBLE_HD_MS", "RUMBLE_STRENGTH", "RUMBLE_TRIGGERS", "SHADER_CACHE", "ROMFS_PROOF_DIR", "ROMFS_VERIFY", "RSTICK_DEADZONE", "RUMBLE", "RUMBLE_CURVE", "RUMBLE_HD", "RUMBLE_HD_MS", "RUMBLE_STRENGTH", "RUMBLE_TRIGGERS", "SHADER_CACHE",
@@ -135,7 +135,7 @@ static const struct { const char *key, *section; } sections[] = {
{"BIND_RIGHT", "input"}, {"BIND_DOWN", "input"}, {"BIND_LSTICK_LEFT", "input"}, {"BIND_LSTICK_UP", "input"}, {"BIND_RIGHT", "input"}, {"BIND_DOWN", "input"}, {"BIND_LSTICK_LEFT", "input"}, {"BIND_LSTICK_UP", "input"},
{"BIND_LSTICK_RIGHT", "input"}, {"BIND_LSTICK_DOWN", "input"}, {"BIND_RSTICK_LEFT", "input"}, {"BIND_LSTICK_RIGHT", "input"}, {"BIND_LSTICK_DOWN", "input"}, {"BIND_RSTICK_LEFT", "input"},
{"BIND_RSTICK_UP", "input"}, {"BIND_RSTICK_RIGHT", "input"}, {"BIND_RSTICK_DOWN", "input"}, {"BIND_SHAKE", "input"}, {"BIND_RSTICK_UP", "input"}, {"BIND_RSTICK_RIGHT", "input"}, {"BIND_RSTICK_DOWN", "input"}, {"BIND_SHAKE", "input"},
{"ASYNC_PREEMPT", "performance"}, {"REPLAY_ASYNC", "performance"}, {"REPLAY_WORKERS", "performance"}, {"REPLAY_PIPE", "performance"}, {"ASYNC_PREEMPT", "performance"}, {"REPLAY_ASYNC", "performance"}, {"REPLAY_WORKERS", "performance"}, {"REPLAY_PIPE", "performance"}, {"TAIL_EARLY", "performance"}, {"REPLAY_SMALL", "performance"},
{"FREE_CORES", "performance"}, {"FREE_CORES", "performance"},
{"DYN_RES", "performance"}, {"DYN_RES_TARGET", "performance"}, {"DYN_RES_STEP", "performance"}, {"DYN_RES", "performance"}, {"DYN_RES_TARGET", "performance"}, {"DYN_RES_STEP", "performance"},
{"RECORDER_CORE", "performance"}, {"RECORDER_PRIO", "performance"}, {"BIND_REUSE", "performance"}, {"RECORDER_CORE", "performance"}, {"RECORDER_PRIO", "performance"}, {"BIND_REUSE", "performance"},
@@ -284,6 +284,8 @@ static const CfgMeta metas[] = {
// Advanced: scheduling // Advanced: scheduling
{"ASYNC_PREEMPT", "1", "0|1|2", RESTART}, {"REPLAY_ASYNC", "0", "0|1", RESTART}, {"ASYNC_PREEMPT", "1", "0|1|2", RESTART}, {"REPLAY_ASYNC", "0", "0|1", RESTART},
{"REPLAY_WORKERS", "0", "0..16", RESTART}, {"REPLAY_PIPE", "0", "0..4", RESTART}, {"REPLAY_WORKERS", "0", "0..16", RESTART}, {"REPLAY_PIPE", "0", "0..4", RESTART},
{"TAIL_EARLY", "0", "0|1", RESTART}, // vk_exec.c "the early tail"
{"REPLAY_SMALL", "0", "0..1024", RESTART}, // vk_exec.c group_start
{"FREE_CORES", "0", "0|1", RESTART}, {"TIMER_1MS", "1", "0|1", RESTART}, {"FREE_CORES", "0", "0|1", RESTART}, {"TIMER_1MS", "1", "0|1", RESTART},
// dynamic resolution (dynres.c): the scene follows the GPU time toward DYN_RES_TARGET milliseconds // dynamic resolution (dynres.c): the scene follows the GPU time toward DYN_RES_TARGET milliseconds
{"DYN_RES", "0", "0|1", LIVE}, {"DYN_RES_TARGET", "16.7", "1..100", LIVE}, {"DYN_RES_STEP", "45", "8..600", LIVE}, {"DYN_RES", "0", "0|1", LIVE}, {"DYN_RES_TARGET", "16.7", "1..100", LIVE}, {"DYN_RES_STEP", "45", "8..600", LIVE},

View File

@@ -164,6 +164,10 @@ typedef struct {
// As fence_wait, but only until the GPU has executed the submission's own commands: its write-backs to guest // As fence_wait, but only until the GPU has executed the submission's own commands: its write-backs to guest
// memory and its fence's side effects may follow (vk_sync.c exec_tl). The same as fence_wait where there is no split. // memory and its fence's side effects may follow (vk_sync.c exec_tl). The same as fence_wait where there is no split.
int (*exec_wait)(uint64_t value, uint64_t timeout_ns); int (*exec_wait)(uint64_t value, uint64_t timeout_ns);
// The game's own wait for a fence (nvnSyncWait): fence_wait, or where the back end knows the rest of the value's
// GPU work reads nothing the game can change and hands it nothing, until its results are there (vk_sync.c
// vk_sync_game_wait).
int (*game_wait)(uint64_t value, uint64_t timeout_ns);
uint64_t (*fence_done)(void); // highest value whose side effects are written uint64_t (*fence_done)(void); // highest value whose side effects are written
// Queues the present and returns at once; the host present goes out at at_ns (kern_now_ns clock), at once when that // Queues the present and returns at once; the host present goes out at at_ns (kern_now_ns clock), at once when that
// has passed or at_ns is 0. // has passed or at_ns is 0.

View File

@@ -147,6 +147,7 @@ const GfxBackend gfx_null_backend = {
.submit = null_submit, .submit = null_submit,
.fence_wait = null_fence_wait, .fence_wait = null_fence_wait,
.exec_wait = null_fence_wait, .exec_wait = null_fence_wait,
.game_wait = null_fence_wait,
.fence_done = null_fence_done, .fence_done = null_fence_done,
.present = null_present, .present = null_present,
.finish = null_finish, .finish = null_finish,

View File

@@ -577,7 +577,7 @@ NVN_API(nvnSyncWait) {
int ok; int ok;
{ {
NVN_BLOCKING(); NVN_BLOCKING();
ok = b->fence_wait(s->fence, timeout); ok = b->game_wait(s->fence, timeout);
} }
if (ok) b->cpu_synced(s->fence); if (ok) b->cpu_synced(s->fence);
ret_x(c, ok ? NVN_SYNC_WAIT_RESULT_CONDITION_SATISFIED : NVN_SYNC_WAIT_RESULT_TIMEOUT_EXPIRED); ret_x(c, ok ? NVN_SYNC_WAIT_RESULT_CONDITION_SATISFIED : NVN_SYNC_WAIT_RESULT_TIMEOUT_EXPIRED);

View File

@@ -1447,8 +1447,17 @@ static void tab_game(float scale) {
rows_saves(); rows_saves();
// Testing: experimental rows, then the debug menu where the game version has it // Testing: experimental rows, then the debug menu where the game version has it
opt_group_text("Testing"); opt_group_text("Testing");
// vk_exec.c rw_init reads it at start: the row notes the restart and the bar's Restart does it // vk_exec.c rw_init and tail_enabled read these at start: each row notes the restart and the bar's Restart does
// it. A row whose setting does nothing without another one (rw_init) is greyed out until that one is set. The two
// rows under the workers are switches: on writes the value measured best (4 groups at once, groups from 16 draws),
// and any other value set in nx2navi.ini shows as on and stays.
opt_slider("NX2NAVI_REPLAY_WORKERS", UI_MENU_REPLAY_WORKERS, 0, 8, UI_MENU_REPLAY_WORKERS_DESC, 1, "%.0f"); opt_slider("NX2NAVI_REPLAY_WORKERS", UI_MENU_REPLAY_WORKERS, 0, 8, UI_MENU_REPLAY_WORKERS_DESC, 1, "%.0f");
const bool workers = cfg_int("NX2NAVI_REPLAY_WORKERS") > 0, pipe = workers && cfg_int("NX2NAVI_REPLAY_PIPE") > 0;
if (!workers) opt_disable_next(UI_MENU_NEEDS_WORKERS);
opt_toggle_value("NX2NAVI_REPLAY_PIPE", UI_MENU_REPLAY_PIPE, UI_MENU_REPLAY_PIPE_DESC, "4", "0");
if (!pipe) opt_disable_next(workers ? UI_MENU_NEEDS_PIPE : UI_MENU_NEEDS_WORKERS);
opt_toggle_value("NX2NAVI_REPLAY_SMALL", UI_MENU_REPLAY_SMALL, UI_MENU_REPLAY_SMALL_DESC, "16", "0");
opt_toggle("NX2NAVI_TAIL_EARLY", UI_MENU_TAIL_EARLY, UI_MENU_TAIL_EARLY_DESC);
// The debug menu, a tool for testing with English text of its own: off unless switched on here // The debug menu, a tool for testing with English text of its own: off unless switched on here
if (dbg_supported()) { if (dbg_supported()) {
const bool on = cfg_bool("NX2NAVI_DEBUG_MENU"); const bool on = cfg_bool("NX2NAVI_DEBUG_MENU");

View File

@@ -506,11 +506,23 @@ static const char *const text_en[UI_TEXT_COUNT] = {
"where the card has none, keeps video memory within what the driver reports free, waits less for the GPU and " "where the card has none, keeps video memory within what the driver reports free, waits less for the GPU and "
"prepares frames on a thread of their own. Measured on a Radeon RX 580 only: it can show a dark or flickering " "prepares frames on a thread of their own. Measured on a Radeon RX 580 only: it can show a dark or flickering "
"frame now and then, so leave it off on a newer card.", "frame now and then, so leave it off on a newer card.",
[UI_MENU_REPLAY_WORKERS] = "Replay workers (experimental)", [UI_MENU_REPLAY_WORKERS] = "CPU helper threads (experimental)",
[UI_MENU_REPLAY_WORKERS_DESC] = "CPU threads that help translate each frame's draws for the GPU, beside the game's own " [UI_MENU_REPLAY_WORKERS_DESC] = "Uses more of your CPU's cores to prepare each frame for the graphics card. 0 turns it "
"thread. 3 gave about 20% more frames a second in Hyrule Field on an 8-core CPU; more " "off. 3 gave about 20% more frames a second on an 8-core CPU, and 5 a little more "
"than 3 gave nothing more. Some players' games crashed with them on: 0 (off) is the " "with the settings below. Some players' games crashed with this on: set it back to "
"safe choice. Takes effect after a restart.", "0 if yours does.",
[UI_MENU_REPLAY_PIPE] = "Helpers work ahead (experimental)",
[UI_MENU_REPLAY_PIPE_DESC] = "The game carries on with the rest of the frame while the helper threads are still "
"busy, instead of waiting for them. A small gain: 1 to 2% more frames a second.",
[UI_MENU_REPLAY_SMALL] = "Helpers take small jobs (experimental)",
[UI_MENU_REPLAY_SMALL_DESC] = "Gives the helper threads the smaller batches of drawing work too, not only the large "
"ones. A small gain: about 2% more frames a second.",
[UI_MENU_TAIL_EARLY] = "Start next frame early (experimental)",
[UI_MENU_TAIL_EARLY_DESC] = "The game starts on its next frame while the graphics card is still finishing the "
"current one. The biggest gain of these settings: 19 to 29% more frames a second with "
"FSR 4 on a Radeon RX 7900 XT, the only card it was measured on. It needs TAA, DLSS, "
"FSR 3.1 or FSR 4, and for now an AMD graphics card. Without them it changes nothing.",
[UI_MENU_NEEDS_WORKERS] = "needs CPU helper threads", [UI_MENU_NEEDS_PIPE] = "needs Helpers work ahead",
[UI_MENU_DISCORD_RPC] = "Discord Rich Presence", [UI_MENU_DISCORD_RPC] = "Discord Rich Presence",
[UI_MENU_DISCORD_RPC_DESC] = "Shows \"Playing NX2navi\" with the play time in your Discord status while the " [UI_MENU_DISCORD_RPC_DESC] = "Shows \"Playing NX2navi\" with the play time in your Discord status while the "
"game runs. discord_app_id and discord_image in nx2navi.ini choose the application.", "game runs. discord_app_id and discord_image in nx2navi.ini choose the application.",

View File

@@ -175,7 +175,10 @@ typedef enum {
// Menu tab // Menu tab
UI_MENU_MENU_LANG, UI_MENU_MENU_LANG_DESC, UI_MENU_MENU_LANG_AUTO, UI_MENU_MENU_LANG, UI_MENU_MENU_LANG_DESC, UI_MENU_MENU_LANG_AUTO,
UI_MENU_DISCORD_RPC, UI_MENU_DISCORD_RPC_DESC, UI_MENU_LOW_END_GPU, UI_MENU_LOW_END_GPU_DESC, UI_MENU_DISCORD_RPC, UI_MENU_DISCORD_RPC_DESC, UI_MENU_LOW_END_GPU, UI_MENU_LOW_END_GPU_DESC,
UI_MENU_REPLAY_WORKERS, UI_MENU_REPLAY_WORKERS_DESC, // the Game tab, English only // the Game tab's Testing rows (ui/menu.cpp tab_game), English only
UI_MENU_REPLAY_WORKERS, UI_MENU_REPLAY_WORKERS_DESC, UI_MENU_REPLAY_PIPE, UI_MENU_REPLAY_PIPE_DESC,
UI_MENU_REPLAY_SMALL, UI_MENU_REPLAY_SMALL_DESC, UI_MENU_TAIL_EARLY, UI_MENU_TAIL_EARLY_DESC,
UI_MENU_NEEDS_WORKERS, UI_MENU_NEEDS_PIPE, // why a row is greyed out
// Language names, each in its own language and the same in every row (game language and menu language lists) // Language names, each in its own language and the same in every row (game language and menu language lists)
UI_MENU_LANGNAME_EN, UI_MENU_LANGNAME_JA, UI_MENU_LANGNAME_FR, UI_MENU_LANGNAME_DE, UI_MENU_LANGNAME_IT, UI_MENU_LANGNAME_EN, UI_MENU_LANGNAME_JA, UI_MENU_LANGNAME_FR, UI_MENU_LANGNAME_DE, UI_MENU_LANGNAME_IT,

View File

@@ -405,6 +405,23 @@ int vk_mem_take_live(void);
// for the newest such submission up to the one it waits for. // for the newest such submission up to the one it waits for.
enum { VK_RES_EVENT = 1, VK_RES_COUNTER = 2, VK_RES_OQ = 4, VK_RES_WRITE = 8, VK_RES_WRITEBACK = 16, VK_RES_KINDS = 5 }; enum { VK_RES_EVENT = 1, VK_RES_COUNTER = 2, VK_RES_OQ = 4, VK_RES_WRITE = 8, VK_RES_WRITEBACK = 16, VK_RES_KINDS = 5 };
void vk_exec_note_results(int kind); // VK_RES_* void vk_exec_note_results(int kind); // VK_RES_*
// The early tail (vk_exec.c "the early tail", NX2NAVI_TAIL_EARLY). vk_exec_tail: the replay is recording a submission's
// tail. vk_sync.c: the tail's command buffer set aside for the write-backs' one (hold), then the part with the
// write-backs handed over with the executed timeline's signal and the tail put behind it (release); whether a part can
// take that order now (ok); the game's wait for a fence (game_wait); for a value whose tail the GPU still runs, about
// how long that takes (tail_pending_ns, else 0).
struct VkExec;
void vk_exec_tail_begin(struct VkExec *ex);
int vk_exec_tail(void);
void vk_exec_tail_ub(uint64_t fence, uint64_t addr, const void *data, uint32_t size); // vk_exec_xfer.c
int vk_sync_tail_ok(void);
VkCommandBuffer vk_sync_tail_hold(void);
VkCommandBuffer vk_sync_tail_release(void);
int vk_sync_game_wait(uint64_t value, uint64_t timeout_ns);
uint64_t vk_sync_tail_pending_ns(uint64_t value);
int vk_mem_live_swap(int v); // vk_memory.c: the note of a late read of guest memory, exchanged
uint64_t vk_mem_gpu_wrote_count(void); // vk_memory.c: GPU writes of pool memory noted so far
int vk_mem_wb_lazy(void); // vk_memory.c: the lazy write-back runs
int vk_exec_take_results(void); // the kinds noted since the last take int vk_exec_take_results(void); // the kinds noted since the last take
void vk_mem_lazy_part_end(void); // lazy write-back: a part ends (before its prologue), and it is handed over void vk_mem_lazy_part_end(void); // lazy write-back: a part ends (before its prologue), and it is handed over
void vk_mem_lazy_handed(void); void vk_mem_lazy_handed(void);

View File

@@ -936,6 +936,7 @@ const GfxBackend gfx_vulkan_backend = {
.submit = vk_be_submit, .submit = vk_be_submit,
.fence_wait = vk_sync_fence_wait, .fence_wait = vk_sync_fence_wait,
.exec_wait = vk_be_exec_wait, .exec_wait = vk_be_exec_wait,
.game_wait = vk_sync_game_wait,
.fence_done = vk_sync_fence_done, .fence_done = vk_sync_fence_done,
.present = vk_be_present, .present = vk_be_present,
.frame_start = vk_be_frame_start, .frame_start = vk_be_frame_start,

View File

@@ -2170,6 +2170,7 @@ static void evaluate(VkExec *ex, const VkExecTarget *dst) {
const VkExecTarget *t = &D.color, *z = &D.depth; const VkExecTarget *t = &D.color, *z = &D.depth;
if (!dst) dst = t; if (!dst) dst = t;
uint32_t w = t->width, h = t->height, ow = dst->width, oh = dst->height; uint32_t w = t->width, h = t->height, ow = dst->width, oh = dst->height;
vk_exec_tail_begin(ex); // NX2NAVI_TAIL_EARLY: from here on the frame's tail, in a command buffer of its own
VkCommandBuffer cb = ex->cb; VkCommandBuffer cb = ex->cb;
vk_exec_barrier(ex); // what drew the scene is done (no pass is open here) vk_exec_barrier(ex); // what drew the scene is done (no pass is open here)
vk_gpu_gap_note(VK_GAP_UPSCALER, (uint64_t)ow * oh); vk_gpu_gap_note(VK_GAP_UPSCALER, (uint64_t)ow * oh);

View File

@@ -1122,6 +1122,59 @@ static int split_n = -1, split_idle, split_in_pass, split_draws, split_at, part_
static uint64_t split_ns, part_t0; static uint64_t split_ns, part_t0;
static uint64_t n_parts, n_subs, n_idle_parts, n_pass_splits, n_cuts; static uint64_t n_parts, n_subs, n_idle_parts, n_pass_splits, n_cuts;
// ---- the early tail (NX2NAVI_TAIL_EARLY, default 0)
// Standing in Hyrule Field with FSR 4 (RX 7900 XT, 2026-11-13) a frame's main submission ends with one part of 4.7 ms
// of GPU time, the temporal upscaler and the interface drawn after it, and one of 1.4 ms, the write-backs of what the
// shaders wrote into VRAM mirrors. The game waits for all of it before it writes the next frame's data (the wait after
// a frame's first submission, nvn_api_core.c, and its own fence waits), about 7 ms in which the replay thread has
// nothing to record. What the game reads of the submission is done before the upscaler runs: the occlusion and
// counter reports, the event values, and the write-backs' bytes, which only came last. After it the stream holds two
// timestamp reports, and inline uniform updates for the interface's draws (64 bytes into one mirrored block, some
// thirty times), and nothing else the CPU sees.
// So from the upscaler on (vk_dlaa.c evaluate) the replay records the tail into a command buffer of its own
// (vk_sync_cut), without splits or parallel groups. An inline uniform update of a mirrored block in it takes no
// write-back: the GPU copies the bytes into the mirror in stream order as ever, and guest memory gets them from the
// CPU with the submission's results (vk_exec_draw.c op_update_ub, vk_exec_xfer.c vk_exec_tail_ub). At the submission's
// end, when the tail turned out to hand the CPU nothing but timestamps (vk_exec_tail_results), to write no other pool
// memory (vk_mem_gpu_wrote_count) and to bind no memory the GPU reads from the game's own pages (vk_mem_live_swap: a
// pool without a mirror, a device address table),
// the write-backs are recorded into a command buffer that goes to the queue before the tail's, with the executed
// timeline's signal (vk_sync_tail_hold, vk_sync_tail_release): the GPU runs the same commands, the write-backs before
// the tail. The mirror copies the tail's first uses queued go with that part too, so they read the game's pages before
// the signal. The game's waits then end at that signal and the results the game reads (vk_sync.c vk_sync_exec_wait,
// vk_sync_game_wait), and the GPU runs the tail while the game and the replay are at the next frame. A timestamp
// reported in the tail is written then with the time the tail is expected to end at (vk_exec_xfer.c counter_cb), as
// the game measures the frame's GPU time with it. A tail that fails a test goes as before, the write-backs after it.
// Measured there, 150 s runs, each pair in one batch: 63.5 against 49.4 fps with three replay workers; without them
// 50.7 against 42.5 standing and 56.0 against 45.9 along the field route, and with FSR 3.1, whose tail the GPU runs
// for 1.2 ms where FSR 4's takes 3.5, 51.8 against 47.8 and 54.9 against 52.3. All but a few of the tails passed the
// tests there; in area loads and some places a tail hands the CPU results and goes as before. The pictures matched
// those without it standing, along the route and through warps to five areas. Not measured with the lazy write-back
// (NVIDIA's and Intel's default), where a tail's uniform updates are noted as writes and no tail goes early.
// Not with shadow storage, emulated mesh shaders, a frame dump or a pass capture, nor inside a chunk or a continuation.
static struct {
int on; // the submission's tail is being recorded
int live0; // vk_mem_live_swap's note from before it
uint64_t wrote0; // vk_mem_gpu_wrote_count when it began
// since the last perf line: tails; those that went early; those refused by results, writes, late reads, other
uint64_t n, early, why[4];
} TL;
int vk_exec_tail_results(void); // vk_exec_xfer.c
void vk_sync_tail_stats(uint64_t *tails, uint64_t *waits, double *ms);
int vk_gpu_passes_capturing(void);
int vk_mesh_shader_native(void); // vk_device.c
int vk_exec_tail(void) { return TL.on; }
static int tail_enabled(void) {
static int on = -1;
if (on < 0) {
on = cfg_bool("NX2NAVI_TAIL_EARLY");
if (on)
rt_log("vk: the write-backs go before a frame's tail, and the game's waits end there (NX2NAVI_TAIL_EARLY)");
}
return on;
}
int vk_exec_split_now(VkExec *ex) { int vk_exec_split_now(VkExec *ex) {
int in_pass = ex->in_pass; int in_pass = ex->in_pass;
if (in_pass) vk_exec_pass_end(ex); if (in_pass) vk_exec_pass_end(ex);
@@ -1153,7 +1206,7 @@ static void maybe_split(VkExec *ex) {
split_idle ? "; a first part at once while the GPU is idle" : ""); split_idle ? "; a first part at once while the GPU is idle" : "");
split_at = split_n ? split_n : INT_MAX; split_at = split_n ? split_n : INT_MAX;
} }
if (!split_n) return; if (!split_n || TL.on) return; // the tail stays one command buffer (the early tail)
if (Q.drained < Q.out) { // groups the submitting thread went on past go into the part first, once done if (Q.drained < Q.out) { // groups the submitting thread went on past go into the part first, once done
if (__atomic_load_n(&Q.done, __ATOMIC_ACQUIRE) < Q.out) return; // split_draws stays: a later op asks again if (__atomic_load_n(&Q.done, __ATOMIC_ACQUIRE) < Q.out) return; // split_draws stays: a later op asks again
group_drain(ex, PW_SPLIT); group_drain(ex, PW_SPLIT);
@@ -1784,7 +1837,7 @@ static void op_one(VkExec *ex, const op_hdr *h) {
} }
// ---- replay workers (NX2NAVI_REPLAY_WORKERS, default 0: the replay records every op on the submitting thread; the // ---- replay workers (NX2NAVI_REPLAY_WORKERS, default 0: the replay records every op on the submitting thread; the
// Game tab's Replay workers row sets it) // Game tab's CPU helper threads row sets it)
// 3 workers: on an RX 7900 XT with a Ryzen 7 5800X3D, Hyrule Field standing went from 41.2 to 50.4 fps (2026-11-12, one // 3 workers: on an RX 7900 XT with a Ryzen 7 5800X3D, Hyrule Field standing went from 41.2 to 50.4 fps (2026-11-12, one
// batch), with the same pictures there and in Kokiri Forest, Kakariko, Goron City, a graveyard house and Gerudo // batch), with the same pictures there and in Kokiri Forest, Kakariko, Goron City, a graveyard house and Gerudo
// Valley; 5 or 7 workers measured the same as 3. Default 0 again after players' games crashed with 3 on by default // Valley; 5 or 7 workers measured the same as 3. Default 0 again after players' games crashed with 3 on by default
@@ -1877,6 +1930,8 @@ static const char *const pw_name[PW_N] = {"next group", "op", "pass hooks", "tur
static struct { static struct {
int n; // workers; 0 off, -1 not read yet int n; // workers; 0 off, -1 not read yet
int pipe; // NX2NAVI_REPLAY_PIPE: the groups out at once at most, 0 none int pipe; // NX2NAVI_REPLAY_PIPE: the groups out at once at most, 0 none
int small; // NX2NAVI_REPLAY_SMALL: the fewest draws of a group one worker takes whole
uint64_t small_groups, small_draws; // such groups and their draws since the last perf line
void *help_dc; // the submitting thread's draw context for chunks it takes while it waits void *help_dc; // the submitting thread's draw context for chunks it takes while it waits
RwWorker *w[RW_MAX]; RwWorker *w[RW_MAX];
const op_hdr *skip_to[CALL_DEPTH + 1]; // per call depth: the walk read ahead to this op, a group too small const op_hdr *skip_to[CALL_DEPTH + 1]; // per call depth: the walk read ahead to this op, a group too small
@@ -2008,6 +2063,10 @@ static void rw_init(void) {
int pipe = cfg_int("NX2NAVI_REPLAY_PIPE"); // the groups out at once int pipe = cfg_int("NX2NAVI_REPLAY_PIPE"); // the groups out at once
RW.pipe = RW.n <= 0 || pipe <= 0 ? 0 : pipe > LIVE_GROUPS ? LIVE_GROUPS : pipe; RW.pipe = RW.n <= 0 || pipe <= 0 ? 0 : pipe > LIVE_GROUPS ? LIVE_GROUPS : pipe;
if (RW.pipe) RW.help_dc = vk_exec_draw_ctx_new(); if (RW.pipe) RW.help_dc = vk_exec_draw_ctx_new();
RW.small = RW.pipe ? cfg_int("NX2NAVI_REPLAY_SMALL") : 0;
if (RW.small > 0)
rt_log("vk: a group of %d draws or more goes whole to one replay worker (NX2NAVI_REPLAY_SMALL)", RW.small);
else RW.small = 0;
if (RW.pipe) if (RW.pipe)
rt_log("vk: groups of %d draws or more are recorded by %d replay worker%s while the submitting thread goes on " rt_log("vk: groups of %d draws or more are recorded by %d replay worker%s while the submitting thread goes on "
"past them, %d at once at most (NX2NAVI_REPLAY_WORKERS, NX2NAVI_REPLAY_PIPE)", 2 * CHUNK_DRAWS, RW.n, "past them, %d at once at most (NX2NAVI_REPLAY_WORKERS, NX2NAVI_REPLAY_PIPE)", 2 * CHUNK_DRAWS, RW.n,
@@ -2319,7 +2378,14 @@ static const op_hdr *group_start(VkExec *ex, const op_hdr *h, int depth) {
int n = (int)(draws / CHUNK_DRAWS); int n = (int)(draws / CHUNK_DRAWS);
if (n > RW.n + 1) n = RW.n + 1; if (n > RW.n + 1) n = RW.n + 1;
if (n > GROUP_CHUNKS) n = GROUP_CHUNKS; if (n > GROUP_CHUNKS) n = GROUP_CHUNKS;
if (n < 2 || vk_dump_armed || !vk_sync_append_room(n)) { // NX2NAVI_REPLAY_SMALL=N (default 0, with NX2NAVI_REPLAY_PIPE): a group of N draws or more that is too small to cut
// goes whole to one worker, and the submitting thread goes on with what follows it without taking the chunk
// itself. Kokiri Forest standing, 5 workers, 4 groups out, the early tail on (2026-11-13): about 700 draws a frame
// were in such groups, 3.6 ms of the submitting thread; with 16 the replay took 10.2 instead of 11.0 ms a frame
// (63.2 against 62.2 fps: the game then waits longer for the GPU, which is busy 83% of the frame there).
int small = RW.pipe && RW.small && n < 2 && draws >= (uint32_t)RW.small;
if (small) n = 1;
if (n < 1 + !small || vk_dump_armed || TL.on || !vk_sync_append_room(n)) {
RW.skip_to[depth] = end, RW.serial_draws += draws, RW.small_end[why] += draws; RW.skip_to[depth] = end, RW.serial_draws += draws, RW.small_end[why] += draws;
return h; return h;
} }
@@ -2393,7 +2459,8 @@ static const op_hdr *group_start(VkExec *ex, const op_hdr *h, int depth) {
// the submitting thread takes chunks too, as the workers wake up, and goes on once none is left to take: the // the submitting thread takes chunks too, as the workers wake up, and goes on once none is left to take: the
// GPU waits for the group's part until its last chunk is done // GPU waits for the group's part until its last chunk is done
rw_behind = 1; rw_behind = 1;
group_take(RW.n); if (small) RW.small_groups++, RW.small_draws += draws; // a worker takes it: this thread goes on at once
else group_take(RW.n);
uint64_t to = kern_now_ns(); uint64_t to = kern_now_ns();
RW.front_ns += tf - t0, RW.own_ns += to - tf, RW.wait_ns += to - t0; RW.front_ns += tf - t0, RW.own_ns += to - tf, RW.wait_ns += to - t0;
return end; return end;
@@ -2511,6 +2578,25 @@ static void rec_claim(void) {
} }
} }
// The tail begins here (the early tail): what was recorded so far waits in the chain, the tail gets a command buffer of
// its own. Not in a chunk or a continuation, nor with groups still out: the cut needs every piece of the part.
void vk_exec_tail_begin(VkExec *ex) {
if (!tail_enabled() || TL.on || ex != x_cur || rw_chunk || evo_in) return;
if (Q.drained != Q.out) group_drain(ex, PW_OP); // the cut needs every piece of the part
if (!vk_sync_wb_split() || !vk_mesh_shader_native() || vk_dump_armed || vk_gpu_passes_capturing()) return;
if (!vk_sync_tail_ok()) return;
if (ex->in_pass) vk_exec_pass_end(ex);
vk_exec_barrier(ex); // the command buffer so far ends behind a full barrier, every image back in GENERAL
VkCommandBuffer cb = vk_sync_cut();
if (!cb) return;
ex->cb = cb;
ex->dirty = VK_EXEC_DIRTY_ALL;
vk_exec_draw_new_cb();
n_cuts++;
TL.on = 1, TL.live0 = vk_mem_live_swap(0), TL.wrote0 = vk_mem_gpu_wrote_count();
vk_exec_tail_results();
}
int vk_exec_record(VkCommandBuffer cb, const uint64_t *handles, int n, uint64_t fence) { int vk_exec_record(VkCommandBuffer cb, const uint64_t *handles, int n, uint64_t fence) {
if (vk_device_is_lost()) return 0; // the run is ending (vk_device.c vk_device_lost_check): nothing is recorded if (vk_device_is_lost()) return 0; // the run is ending (vk_device.c vk_device_lost_check): nothing is recorded
uint64_t in_ns = kern_now_ns(); uint64_t in_ns = kern_now_ns();
@@ -2528,6 +2614,7 @@ int vk_exec_record(VkCommandBuffer cb, const uint64_t *handles, int n, uint64_t
nvn_pins_sweep(nvn_submit_count(), keep, 9); nvn_pins_sweep(nvn_submit_count(), keep, 9);
ex->cb = cb; ex->cb = cb;
ex->fence = fence; ex->fence = fence;
TL.on = 0;
ex->recorded = 0; ex->recorded = 0;
ex->in_pass = 0; ex->in_pass = 0;
ex->mark = NULL; ex->mark = NULL;
@@ -2594,12 +2681,47 @@ int vk_exec_record(VkCommandBuffer cb, const uint64_t *handles, int n, uint64_t
// NX2NAVI_WB_SPLIT: the write-backs in a part of their own, after one that signals the executed timeline // NX2NAVI_WB_SPLIT: the write-backs in a part of their own, after one that signals the executed timeline
// (vk_sync.c exec_tl): the wait after a frame's first submission then ends when the submission's passes are // (vk_sync.c exec_tl): the wait after a frame's first submission then ends when the submission's passes are
// through, not its write-backs. Off by default: vk_sync.c says what else that wait then leaves out. // through, not its write-backs. Off by default: vk_sync.c says what else that wait then leaves out.
if (synced && ex->recorded && vk_sync_wb_split() && vk_mem_mirror_wb_pending() && vk_exec_pool_split(ex->fence)) { int tail = 0; // the early tail: the write-backs go before the tail, with the executed timeline's signal
if (TL.on) {
int live = vk_mem_live_swap(0), res = vk_exec_tail_results();
vk_mem_live_swap(live | TL.live0); // the submission's own note, as it would have been
int wrote = vk_mem_gpu_wrote_count() != TL.wrote0;
int other = !synced || !ex->recorded || vk_dump_armed || vk_gpu_passes_capturing() || !vk_sync_tail_ok();
TL.on = 0, TL.n++;
if (res) TL.why[0]++;
else if (wrote) TL.why[1]++;
else if (live) TL.why[2]++;
else if (other || !vk_exec_pool_split(ex->fence)) TL.why[3]++;
else tail = 1, TL.early++;
if (tail) {
ex->cb = vk_sync_tail_hold();
ex->dirty = VK_EXEC_DIRTY_ALL;
vk_exec_draw_new_cb();
}
}
if (!tail && synced && ex->recorded && vk_sync_wb_split() && vk_mem_mirror_wb_pending() &&
vk_exec_pool_split(ex->fence)) {
ex->cb = vk_sync_split_exec(); ex->cb = vk_sync_split_exec();
ex->dirty = VK_EXEC_DIRTY_ALL; ex->dirty = VK_EXEC_DIRTY_ALL;
vk_exec_draw_new_cb(); vk_exec_draw_new_cb();
} }
if (synced) ex->recorded += vk_mem_mirror_end(ex->cb, fence), vk_usnap_end(fence); // VRAM mirror bytes the GPU wrote go back to guest memory if (synced) ex->recorded += vk_mem_mirror_end(ex->cb, fence), vk_usnap_end(fence); // VRAM mirror bytes the GPU wrote go back to guest memory
if (tail) {
ex->cb = vk_sync_tail_release();
part_start = ex->recorded; // the last command buffer holds nothing: the tail is in the chain before it
}
if ((rt_trace & RT_TRACE_PERF) && TL.n >= 600) {
uint64_t tails, waits;
double ms;
vk_sync_tail_stats(&tails, &waits, &ms);
rt_log("perf: early tail: %llu tails, %llu with the write-backs before them (refused: %llu handed the CPU "
"results, %llu wrote pool memory, %llu read guest memory late, %llu other); %llu waits of the game "
"ended before their tail, which the GPU runs for about %.2f ms",
(unsigned long long)TL.n, (unsigned long long)TL.early,
(unsigned long long)TL.why[0], (unsigned long long)TL.why[1], (unsigned long long)TL.why[2],
(unsigned long long)TL.why[3], (unsigned long long)waits, ms);
TL.n = TL.early = 0, memset(TL.why, 0, sizeof TL.why);
}
views_release(ex, fence); views_release(ex, fence);
ex->cb = VK_NULL_HANDLE; ex->cb = VK_NULL_HANDLE;
if (n) n_subs++; if (n) n_subs++;
@@ -2696,8 +2818,13 @@ int vk_exec_record(VkCommandBuffer cb, const uint64_t *handles, int n, uint64_t
memset(RW.pw_n, 0, sizeof RW.pw_n), memset(RW.pw_ns, 0, sizeof RW.pw_ns); memset(RW.pw_n, 0, sizeof RW.pw_n), memset(RW.pw_ns, 0, sizeof RW.pw_ns);
memset(RW.pd_op, 0, sizeof RW.pd_op); memset(RW.pd_op, 0, sizeof RW.pd_op);
RW.behind_ns = 0; RW.behind_ns = 0;
if (RW.small)
rt_log("perf: replay workers: groups one worker took whole (NX2NAVI_REPLAY_SMALL): %.2f per "
"submission, %.1f draws each", (double)RW.small_groups / (double)n_subs,
RW.small_groups ? (double)RW.small_draws / (double)RW.small_groups : 0.0);
} }
} }
RW.small_groups = RW.small_draws = 0;
RW.groups = RW.group_draws = RW.chunks = RW.serial_draws = RW.wait_ns = 0; RW.groups = RW.group_draws = RW.chunks = RW.serial_draws = RW.wait_ns = 0;
evo_held = evo_resumed = evo_forced = evo_switches = 0; evo_held = evo_resumed = evo_forced = evo_switches = 0;
n_subs = n_parts = n_idle_parts = n_pass_splits = n_cuts = 0; n_subs = n_parts = n_idle_parts = n_pass_splits = n_cuts = 0;

View File

@@ -7371,7 +7371,14 @@ static void op_update_ub(VkExec *ex, const p_update_ub *u) {
VK_MEM_PICK(sp); VK_MEM_PICK(sp);
VkBufferCopy cp = {o, sp.offset, u->size}; VkBufferCopy cp = {o, sp.offset, u->size};
vkCmdCopyBuffer(ex->cb, b, sp.buffer, 1, &cp); vkCmdCopyBuffer(ex->cb, b, sp.buffer, 1, &cp);
vk_mem_gpu_wrote(&sp, u->size); // In a submission's tail (vk_exec.c "the early tail") an update of a mirrored block reaches guest memory with the
// results the game reads, from the bytes here, and takes no write-back: one after the tail would land when the
// game is already writing the next frame's data. Not with the lazy write-back, where the block's pages may be
// armed and a store into them would wait for the GPU (memory.c lazy_fault): the update is noted as ever there,
// and the tail goes after the write-backs as before.
if (vk_exec_tail() && !vk_mem_wb_lazy() && vk_mem_mirrored(u->dst, u->size))
vk_exec_tail_ub(ex->fence, u->dst, u->data, u->size);
else vk_mem_gpu_wrote(&sp, u->size);
vk_usnap_drop(u->dst, u->size); vk_usnap_drop(u->dst, u->size);
ex->recorded++; ex->recorded++;
vk_mem_count(sp.cls, VK_MUSE_GPU_WRITE, u->size); vk_mem_count(sp.cls, VK_MUSE_GPU_WRITE, u->size);

View File

@@ -67,12 +67,32 @@ typedef struct { uint64_t addr, size; int back; } Range; // back: written into
// for the newest such submission). Write-backs of the mirrors note it in vk_memory.c; the lazily written ones fetch on // for the newest such submission). Write-backs of the mirrors note it in vk_memory.c; the lazily written ones fetch on
// the CPU's first access. // the CPU's first access.
static volatile int32_t ex_results; // VK_RES_* bits static volatile int32_t ex_results; // VK_RES_* bits
static int tail_results; // those noted while the submission's tail records (vk_exec.c "the early tail")
int vk_exec_tail_results(void) { int r = tail_results; tail_results = 0; return r; }
void vk_exec_note_results(int kind) { void vk_exec_note_results(int kind) {
if (vk_exec_tail()) tail_results |= kind;
__atomic_fetch_or(&ex_results, kind, __ATOMIC_RELAXED); __atomic_fetch_or(&ex_results, kind, __ATOMIC_RELAXED);
vk_gpu_passes_result(kind); // NX2NAVI_GPU_PASSES: where in the submission it is done vk_gpu_passes_result(kind); // NX2NAVI_GPU_PASSES: where in the submission it is done
} }
int vk_exec_take_results(void) { return __atomic_exchange_n(&ex_results, 0, __ATOMIC_RELAXED); } int vk_exec_take_results(void) { return __atomic_exchange_n(&ex_results, 0, __ATOMIC_RELAXED); }
// An inline uniform update in a submission's tail (vk_exec_draw.c op_update_ub, vk_exec.c "the early tail"): the GPU
// copies the bytes into the pool's mirror where the stream has the update, and guest memory gets them here, with the
// results the game reads, in place of a write-back.
typedef struct { uint64_t addr; uint32_t size; uint8_t data[]; } UbWrite;
static void ub_write_cb(void *arg) {
UbWrite *w = arg;
memcpy((void *)(uintptr_t)w->addr, w->data, w->size);
free(w);
}
void vk_exec_tail_ub(uint64_t fence, uint64_t addr, const void *data, uint32_t size) {
UbWrite *w = malloc(sizeof *w + size);
if (!w) rt_fatal("vk: out of memory");
w->addr = addr, w->size = size;
memcpy(w->data, data, size);
vk_completion_add_result(fence, ub_write_cb, w);
}
static void guest_written_cb(void *arg) { static void guest_written_cb(void *arg) {
Range *r = arg; Range *r = arg;
if (r->back) vk_gpu_written_back(r->addr, r->size); if (r->back) vk_gpu_written_back(r->addr, r->size);
@@ -114,8 +134,9 @@ static void event_cb(void *arg) {
// NVNcounterData = {counter, timestamp}; timestamps are Tegra 19.2 MHz ticks (the base of GetSystemTick). // NVNcounterData = {counter, timestamp}; timestamps are Tegra 19.2 MHz ticks (the base of GetSystemTick).
// keep: the 4-byte quarters of the record the write leaves as they are (bit n: bytes 4n to 4n + 3), 0 none. // keep: the 4-byte quarters of the record the write leaves as they are (bit n: bytes 4n to 4n + 3), 0 none.
static void write_counter(uint64_t addr, uint64_t counter, unsigned keep) { // ahead_ns: the timestamp is that much past now (a tail timestamp the GPU has not reached yet, counter_cb), else 0.
uint64_t rec[2] = {counter, (uint64_t)((unsigned __int128)kern_now_ns() * 12 / 625)}; static void write_counter(uint64_t addr, uint64_t counter, unsigned keep, uint64_t ahead_ns) {
uint64_t rec[2] = {counter, (uint64_t)((unsigned __int128)(kern_now_ns() + ahead_ns) * 12 / 625)};
if (!keep) { if (!keep) {
memcpy((void *)(uintptr_t)addr, rec, sizeof rec); memcpy((void *)(uintptr_t)addr, rec, sizeof rec);
return; return;
@@ -127,14 +148,16 @@ static void write_counter(uint64_t addr, uint64_t counter, unsigned keep) {
// Reports other than samples passed (timestamps; the statistics counters are not modelled) write a counter of 0. The // Reports other than samples passed (timestamps; the statistics counters are not modelled) write a counter of 0. The
// counter word matters: Ocarina writes {1, 0} into a record from the CPU, reports TIMESTAMP_TOP into it and gates // counter word matters: Ocarina writes {1, 0} into a record from the CPU, reports TIMESTAMP_TOP into it and gates
// clears on it differing from a zero record (mode 1), clears that must not run. // clears on it differing from a zero record (mode 1), clears that must not run.
static void report_write(uint64_t addr, uint64_t seq, uint64_t counter); // below, with the pending table static void report_write(uint64_t addr, uint64_t seq, uint64_t counter, uint64_t ahead_ns); // below, with the pending table
typedef struct { uint64_t addr, seq; } CounterWrite; // tail: the submission's value for a timestamp reported in its tail (vk_exec.c "the early tail"), else 0
typedef struct { uint64_t addr, seq, tail; } CounterWrite;
static void counter_cb(void *arg) { static void counter_cb(void *arg) {
CounterWrite *w = arg; CounterWrite *w = arg;
if (vk_sync_effect_late()) __atomic_fetch_add(&late_counters, 1, __ATOMIC_RELAXED); if (vk_sync_effect_late()) __atomic_fetch_add(&late_counters, 1, __ATOMIC_RELAXED);
report_write(w->addr, w->seq, 0); // A timestamp of the tail, written while the GPU still runs the tail: the time the tail is expected to end at
report_write(w->addr, w->seq, 0, w->tail ? vk_sync_tail_pending_ns(w->tail) : 0);
free(w); free(w);
} }
@@ -308,7 +331,7 @@ static unsigned clear_quarters(const Pending *e) { return e->clr_seq >= e->seq ?
// Completion thread, or the thread whose wait runs the results: report seq to addr goes into guest memory, but for // Completion thread, or the thread whose wait runs the results: report seq to addr goes into guest memory, but for
// the quarters a small clear replayed after it has filled. // the quarters a small clear replayed after it has filled.
static void report_write(uint64_t addr, uint64_t seq, uint64_t counter) { static void report_write(uint64_t addr, uint64_t seq, uint64_t counter, uint64_t ahead_ns) {
int lk = __atomic_load_n(&clr_any, __ATOMIC_ACQUIRE); int lk = __atomic_load_n(&clr_any, __ATOMIC_ACQUIRE);
unsigned keep = 0; unsigned keep = 0;
if (lk) { if (lk) {
@@ -317,7 +340,7 @@ static void report_write(uint64_t addr, uint64_t seq, uint64_t counter) {
if (e && seq <= e->clr_seq) keep = e->clr_mask; if (e && seq <= e->clr_seq) keep = e->clr_mask;
if (keep) __atomic_fetch_add(&CS.kept, 1, __ATOMIC_RELAXED); if (keep) __atomic_fetch_add(&CS.kept, 1, __ATOMIC_RELAXED);
} }
write_counter(addr, counter, keep); write_counter(addr, counter, keep, ahead_ns);
if (lk) plat_mutex_unlock(&clr_lock); if (lk) plat_mutex_unlock(&clr_lock);
pending_written(addr, seq); pending_written(addr, seq);
} }
@@ -644,7 +667,7 @@ static void oq_cb(void *arg) {
if (s->lost && !oq_count) oq_count = 1; if (s->lost && !oq_count) oq_count = 1;
if (s->addr) { if (s->addr) {
if (vk_sync_effect_late()) __atomic_fetch_add(&late_counts, 1, __ATOMIC_RELAXED); if (vk_sync_effect_late()) __atomic_fetch_add(&late_counts, 1, __ATOMIC_RELAXED);
report_write(s->addr, s->seq, oq_count); report_write(s->addr, s->seq, oq_count, 0);
} }
} }
free(s->q), free(s); free(s->q), free(s);
@@ -893,10 +916,13 @@ static void report_counter(VkExec *ex, const p_report_counter *p) {
clear_note_report(p->addr); clear_note_report(p->addr);
if (samples) oq_report(ex, p->addr); if (samples) oq_report(ex, p->addr);
else { else {
// a timestamp in the tail is the one result a tail may hand the CPU: its time is written ahead (counter_cb)
int ts_tail = vk_exec_tail() && p->type == NVN_COUNTER_TYPE_TIMESTAMP, keep = tail_results;
vk_exec_note_results(VK_RES_COUNTER); vk_exec_note_results(VK_RES_COUNTER);
if (ts_tail) tail_results = keep;
CounterWrite *a = malloc(sizeof *a); CounterWrite *a = malloc(sizeof *a);
if (!a) rt_fatal("vk: out of memory"); if (!a) rt_fatal("vk: out of memory");
*a = (CounterWrite){p->addr, report_seq}; *a = (CounterWrite){p->addr, report_seq, ts_tail ? ex->fence : 0};
vk_completion_add_result(ex->fence, counter_cb, a); vk_completion_add_result(ex->fence, counter_cb, a);
} }
Pending *e = pending_slot(p->addr, 1); Pending *e = pending_slot(p->addr, 1);

View File

@@ -3712,7 +3712,11 @@ static uint64_t wb_skip(void) {
} }
static int64_t m_wb_skipped; // bytes wb_skip kept in mirrors since the last summary static int64_t m_wb_skipped; // bytes wb_skip kept in mirrors since the last summary
static volatile uint64_t n_gpu_wrote; // vk_mem_gpu_wrote_count
uint64_t vk_mem_gpu_wrote_count(void) { return __atomic_load_n(&n_gpu_wrote, __ATOMIC_RELAXED); }
static void gpu_wrote(const VkMemSpan *sp, uint64_t size, int shader) { static void gpu_wrote(const VkMemSpan *sp, uint64_t size, int shader) {
__atomic_fetch_add(&n_gpu_wrote, 1, __ATOMIC_RELAXED);
PoolRec *r = sp->pool; PoolRec *r = sp->pool;
if (!r || r->mstate != MS_ON || sp->buffer != r->mbuf || !size || !wb_lazy()) { if (!r || r->mstate != MS_ON || sp->buffer != r->mbuf || !size || !wb_lazy()) {
uint64_t skip = r && r->mstate == MS_ON && sp->buffer == r->mbuf ? wb_skip() : 0; uint64_t skip = r && r->mstate == MS_ON && sp->buffer == r->mbuf ? wb_skip() : 0;
@@ -4087,6 +4091,9 @@ static uint64_t capture_cap(void) {
} }
void vk_mem_note_live(void) { __atomic_store_n(&mirror_live, 1, __ATOMIC_RELAXED); } void vk_mem_note_live(void) { __atomic_store_n(&mirror_live, 1, __ATOMIC_RELAXED); }
int vk_mem_live_swap(int v) { return __atomic_exchange_n(&mirror_live, v, __ATOMIC_RELAXED); }
static int wb_lazy(void);
int vk_mem_wb_lazy(void) { return wb_lazy(); } // the lazy write-back runs (vk_exec_draw.c op_update_ub)
int vk_mem_take_live(void) { int vk_mem_take_live(void) {
int live = __atomic_exchange_n(&mirror_live, 0, __ATOMIC_RELAXED); int live = __atomic_exchange_n(&mirror_live, 0, __ATOMIC_RELAXED);

View File

@@ -63,6 +63,16 @@ static uint64_t live_last_of[LIVE_RING], live_last;
// per value (index & LIVE_RING - 1): the newest value up to it whose completion hands results to the CPU (vk_exec_take_results) // per value (index & LIVE_RING - 1): the newest value up to it whose completion hands results to the CPU (vk_exec_take_results)
static uint64_t res_of[LIVE_RING], res_last; static uint64_t res_of[LIVE_RING], res_last;
static uint8_t res_kinds[LIVE_RING]; // per value: its VK_RES_* kinds static uint8_t res_kinds[LIVE_RING]; // per value: its VK_RES_* kinds
// The early tail (vk_exec.c "the early tail"), per value under st_lock: early_of holds the value when its part with
// the write-backs signalled the executed timeline before its tail; empty_of when the submission handed the GPU nothing
// and the CPU no results (a submission held at its first event wait: Ocarina's first of a frame in the field).
static uint64_t early_of[LIVE_RING], empty_of[LIVE_RING];
static VkCommandBuffer tail_cb; // sub_lock: the tail's command buffer between hold and release
// sub_lock: this submission's tail went behind its write-backs; the parts of it handed over before the last
static int tail_early, sub_parts;
static volatile uint64_t tail_ns; // about how long the GPU takes for a tail (a moving average)
static volatile uint64_t tail_t0, tail_val; // the first early return for tail_val, for tail_ns
static volatile uint64_t tail_n, tail_waits; // since the last vk_sync_tail_stats: early tails; waits they ended
static int res_mask = 0x1f; // NX2NAVI_SUBMIT_FINISH_KINDS: the kinds the wait counts static int res_mask = 0x1f; // NX2NAVI_SUBMIT_FINISH_KINDS: the kinds the wait counts
static int stop; static int stop;
static Pending *queue; // side effects in the order they were added static Pending *queue; // side effects in the order they were added
@@ -920,6 +930,12 @@ static void completion_main(void *arg) {
// finished submissions held (vk_mem_collect: upload batches, destroyed images, views and buffers, each // finished submissions held (vk_mem_collect: upload batches, destroyed images, views and buffers, each
// checked against the timeline itself) is nothing they read, and it takes cmd_lock, which the upload // checked against the timeline itself) is nothing they read, and it takes cmd_lock, which the upload
// path holds, so done waited behind uploads. // path holds, so done waited behind uploads.
uint64_t tv = __atomic_load_n(&tail_val, __ATOMIC_ACQUIRE);
if (tv && reached >= tv) { // the tail a wait went ahead of is through: how long it took from there
uint64_t d = kern_now_ns() - __atomic_load_n(&tail_t0, __ATOMIC_RELAXED), a = tail_ns;
if (d < 50000000ull) __atomic_store_n(&tail_ns, a ? (a * 7 + d) / 8 : d, __ATOMIC_RELAXED);
__atomic_compare_exchange_n(&tail_val, &tv, 0, 0, __ATOMIC_RELEASE, __ATOMIC_RELAXED);
}
uint64_t was = done; uint64_t was = done;
__atomic_store_n(&done, reached, __ATOMIC_RELEASE); __atomic_store_n(&done, reached, __ATOMIC_RELEASE);
plat_cond_broadcast(&cv_done); plat_cond_broadcast(&cv_done);
@@ -1197,6 +1213,63 @@ VkCommandBuffer vk_sync_worker_begin(int w) {
void vk_sync_worker_end(VkCommandBuffer cb) { vkEndCommandBuffer(cb); } void vk_sync_worker_end(VkCommandBuffer cb) { vkEndCommandBuffer(cb); }
static VkCommandBuffer split_part(int exec); static VkCommandBuffer split_part(int exec);
// ---- the early tail (vk_exec.c "the early tail"). The replay cut the part where the tail began (vk_sync_cut), so the
// chain holds what came before and the command buffer being recorded is the tail's. vk_sync_tail_hold sets that one
// aside, ended and timed as a part of its own, and begins the one the write-backs are recorded into;
// vk_sync_tail_release hands the chain and the write-backs to the queue with the executed timeline's signal
// (split_part), and puts the tail into the chain for the submission's last part, which signals the timeline. The queue
// runs them in that order, so the GPU's work is what it was with the write-backs moved before the tail.
int vk_sync_tail_ok(void) { return !pre_cb && !tail_cb && exec_tl && exec_sig < cur_fence && nchain + 2 <= CHAIN_MAX; }
VkCommandBuffer vk_sync_tail_hold(void) {
VkCtx *c = vk_ctx();
VkCommandBuffer cb = slots[cur_slot].cb;
vk_gpu_passes_part(cb, 1);
ts_end(cb, cur_fence);
if (!rec_end_cb(cb, 0, 0)) vkEndCommandBuffer(cb);
tail_cb = cb;
CmdSlot *s = take_slot(c, cur_fence); // the tail keeps its slot until the value is reached
VkCommandBufferBeginInfo bi = {.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO,
.flags = VK_COMMAND_BUFFER_USAGE_ONE_TIME_SUBMIT_BIT};
vkBeginCommandBuffer(s->cb, &bi);
ts_begin(s->cb);
rec_begin(s->cb, cur_fence);
vk_gpu_passes_part(s->cb, 0);
return s->cb;
}
VkCommandBuffer vk_sync_tail_release(void) {
VkCommandBuffer cb = split_part(1);
chain[nchain++] = tail_cb;
tail_cb = VK_NULL_HANDLE;
tail_early = 1;
return cb;
}
// The newest value up to `value` whose GPU work a wait of the game is for: the empty submissions after it are passed
// over. *early: that one's write-backs went before its tail, so the executed timeline reaches it there.
static uint64_t wait_target(uint64_t value, int *early) {
plat_mutex_lock(&st_lock);
uint64_t t = value;
while (t > done && empty_of[t & (LIVE_RING - 1)] == t) t--;
*early = t > done && early_of[t & (LIVE_RING - 1)] == t;
plat_mutex_unlock(&st_lock);
return t;
}
uint64_t vk_sync_tail_pending_ns(uint64_t value) {
plat_mutex_lock(&st_lock);
int pending = value > done && early_of[value & (LIVE_RING - 1)] == value;
plat_mutex_unlock(&st_lock);
return pending ? __atomic_load_n(&tail_ns, __ATOMIC_RELAXED) : 0;
}
void vk_sync_tail_stats(uint64_t *tails, uint64_t *waits, double *ms) {
*tails = __atomic_exchange_n(&tail_n, 0, __ATOMIC_RELAXED);
*waits = __atomic_exchange_n(&tail_waits, 0, __ATOMIC_RELAXED);
*ms = (double)__atomic_load_n(&tail_ns, __ATOMIC_RELAXED) / 1e6;
}
VkCommandBuffer vk_sync_split(void) { return split_part(0); } VkCommandBuffer vk_sync_split(void) { return split_part(0); }
// The part that ends here is the last with the submission's own commands: it signals the executed timeline. // The part that ends here is the last with the submission's own commands: it signals the executed timeline.
VkCommandBuffer vk_sync_split_exec(void) { return split_part(1); } VkCommandBuffer vk_sync_split_exec(void) { return split_part(1); }
@@ -1218,6 +1291,7 @@ static VkCommandBuffer split_part(int exec) {
si.signalSemaphoreInfoCount = 1, si.pSignalSemaphoreInfos = &esig, exec_sig = cur_fence; si.signalSemaphoreInfoCount = 1, si.pSignalSemaphoreInfos = &esig, exec_sig = cur_fence;
ts_submitted(cbi, ncb); ts_submitted(cbi, ncb);
queue_submit(c, &si, 0); // held or handed to the queue (submit batching) queue_submit(c, &si, 0); // held or handed to the queue (submit batching)
sub_parts++;
vk_mem_lazy_handed(); vk_mem_lazy_handed();
vk_mem_note_submit(cur_fence); // objects destroyed from here on may be in use by the part just submitted vk_mem_note_submit(cur_fence); // objects destroyed from here on may be in use by the part just submitted
CmdSlot *s = take_slot(c, cur_fence); // the submitted part keeps its slot until the value is reached CmdSlot *s = take_slot(c, cur_fence); // the submitted part keeps its slot until the value is reached
@@ -1259,6 +1333,7 @@ static uint64_t submit(const uint64_t *handles, int n, const OwnCommands *own) {
vk_mem_dirty_submit(); // NX2NAVI_TRACE=perf: what a mirror would upload before this submit vk_mem_dirty_submit(); // NX2NAVI_TRACE=perf: what a mirror would upload before this submit
uint64_t fence = __atomic_load_n(&issued, __ATOMIC_ACQUIRE) + 1; uint64_t fence = __atomic_load_n(&issued, __ATOMIC_ACQUIRE) + 1;
cur_fence = fence; cur_fence = fence;
tail_early = sub_parts = 0;
CmdSlot *s = take_slot(c, fence); CmdSlot *s = take_slot(c, fence);
VkCommandBufferBeginInfo bi = {.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO, VkCommandBufferBeginInfo bi = {.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO,
.flags = VK_COMMAND_BUFFER_USAGE_ONE_TIME_SUBMIT_BIT}; .flags = VK_COMMAND_BUFFER_USAGE_ONE_TIME_SUBMIT_BIT};
@@ -1318,6 +1393,9 @@ static uint64_t submit(const uint64_t *handles, int n, const OwnCommands *own) {
if (live) live_last = fence; if (live) live_last = fence;
live_last_of[fence & (LIVE_RING - 1)] = live_last; live_last_of[fence & (LIVE_RING - 1)] = live_last;
res_kinds[fence & (LIVE_RING - 1)] = (uint8_t)res; res_kinds[fence & (LIVE_RING - 1)] = (uint8_t)res;
early_of[fence & (LIVE_RING - 1)] = tail_early ? fence : 0;
empty_of[fence & (LIVE_RING - 1)] = !own && !sent && !sub_parts && !res ? fence : 0;
if (tail_early) __atomic_fetch_add(&tail_n, 1, __ATOMIC_RELAXED);
if (res & res_mask) res_last = fence; if (res & res_mask) res_last = fence;
res_of[fence & (LIVE_RING - 1)] = res_last; res_of[fence & (LIVE_RING - 1)] = res_last;
__atomic_store_n(&issued, fence, __ATOMIC_RELEASE); __atomic_store_n(&issued, fence, __ATOMIC_RELEASE);
@@ -1530,8 +1608,13 @@ int vk_sync_exec_wait(uint64_t value, uint64_t timeout_ns) {
if (vk_device_is_lost()) return __atomic_load_n(&done, __ATOMIC_ACQUIRE) >= value; if (vk_device_is_lost()) return __atomic_load_n(&done, __ATOMIC_ACQUIRE) >= value;
if (!vk_sync_wb_split()) return vk_sync_fence_wait(value, timeout_ns); if (!vk_sync_wb_split()) return vk_sync_fence_wait(value, timeout_ns);
if (__atomic_load_n(&done, __ATOMIC_ACQUIRE) >= value) return 1; if (__atomic_load_n(&done, __ATOMIC_ACQUIRE) >= value) return 1;
// The early tail: a value whose GPU work is that of an earlier submission with its write-backs before its tail (the
// submissions since handed the GPU nothing) is reached where that one's executed signal is, its tail still to run.
int early = 0;
uint64_t asked = value, target = wb_results ? wait_target(value, &early) : value;
if (early) value = target;
uint64_t p0 = perf_begin(); uint64_t p0 = perf_begin();
hold_release(value, 0, 0); // a held submission the waiter needs goes to the queue now hold_release(asked, 0, 0); // a held submission the waiter needs goes to the queue now
VkCtx *c = vk_ctx(); VkCtx *c = vk_ctx();
VkSemaphoreWaitInfo wi = {.sType = VK_STRUCTURE_TYPE_SEMAPHORE_WAIT_INFO, .semaphoreCount = 1, VkSemaphoreWaitInfo wi = {.sType = VK_STRUCTURE_TYPE_SEMAPHORE_WAIT_INFO, .semaphoreCount = 1,
.pSemaphores = &exec_tl, .pValues = &value}; .pSemaphores = &exec_tl, .pValues = &value};
@@ -1545,6 +1628,13 @@ int vk_sync_exec_wait(uint64_t value, uint64_t timeout_ns) {
if (r == VK_SUCCESS && wb_results) results_run(value); if (r == VK_SUCCESS && wb_results) results_run(value);
perf_end(PERF_FENCE, p0); perf_end(PERF_FENCE, p0);
perf_gpu_wait(p0); perf_gpu_wait(p0);
if (r == VK_SUCCESS && early && __atomic_load_n(&done, __ATOMIC_ACQUIRE) < value) {
__atomic_fetch_add(&tail_waits, 1, __ATOMIC_RELAXED);
uint64_t none = 0; // the first wait back for this tail starts its clock (the completion thread ends it)
if (__atomic_load_n(&tail_val, __ATOMIC_ACQUIRE) != value &&
__atomic_compare_exchange_n(&tail_val, &none, value, 0, __ATOMIC_RELEASE, __ATOMIC_RELAXED))
__atomic_store_n(&tail_t0, kern_now_ns(), __ATOMIC_RELAXED);
}
if (r == VK_SUCCESS) { if (r == VK_SUCCESS) {
__atomic_fetch_add(&exw_n, 1, __ATOMIC_RELAXED); __atomic_fetch_add(&exw_n, 1, __ATOMIC_RELAXED);
if (__atomic_load_n(&done, __ATOMIC_ACQUIRE) < value) { if (__atomic_load_n(&done, __ATOMIC_ACQUIRE) < value) {
@@ -1556,6 +1646,19 @@ int vk_sync_exec_wait(uint64_t value, uint64_t timeout_ns) {
return r == VK_SUCCESS; return r == VK_SUCCESS;
} }
// The game's own wait for a fence (nvn_api_core.c nvnSyncWait): the fence's wait, or with the early tail the wait for
// the results of the submission the value's GPU work belongs to (vk_sync_exec_wait), when that one's tail is all the
// GPU still has of it: the tail reads nothing the game can change and hands it nothing.
int vk_sync_game_wait(uint64_t value, uint64_t timeout_ns) {
vk_sync_start();
if (vk_sync_wb_split() && wb_results && __atomic_load_n(&done, __ATOMIC_ACQUIRE) < value) {
int early = 0;
wait_target(value, &early);
if (early) return vk_sync_exec_wait(value, timeout_ns);
}
return vk_sync_fence_wait(value, timeout_ns);
}
uint64_t vk_sync_fence_done(void) { return __atomic_load_n(&done, __ATOMIC_ACQUIRE); } uint64_t vk_sync_fence_done(void) { return __atomic_load_n(&done, __ATOMIC_ACQUIRE); }
uint64_t vk_sync_issued(void) { return __atomic_load_n(&issued, __ATOMIC_ACQUIRE); } uint64_t vk_sync_issued(void) { return __atomic_load_n(&issued, __ATOMIC_ACQUIRE); }