diff --git a/docs/development.md b/docs/development.md index cd4eb4e..77a79a8 100644 --- a/docs/development.md +++ b/docs/development.md @@ -141,12 +141,13 @@ case-sensitive, and an unknown one is ignored. | `RX_VK_HANG_DUMP=` | Every thread's stack to the file once no frame was shown for `RX_VK_HANG_SEC` seconds (20) (Windows) | | `RX_VK_NOOPT` (set) | Pipelines built without the driver's optimisations | | `RX_VK_SKIP_PROG=,...` | Leaves out the draws of these programs (default `53509044401d78d2`; `-` leaves out none) | -| `RX_VK_PIPE_SKIP=0` | A draw whose pipeline is not ready waits for it, instead of being left out when its targets hold an earlier frame | +| `RX_VK_PIPE_SKIP=0`, `1` or `2` | Default/`1`: wait for shader translation; a pending pipeline can be skipped only for attachment-only draws whose targets hold an earlier frame. `0`: wait for every pipeline too. `2`: restore the old skip for untranslated draws with persistent targets; this can omit writes before the shader is known. Translated memory-writing draws still wait | | `RX_VK_PRECOMPILE=0` or `1` | The shader preparation before the title off or on (on in builds made with `RX_RELEASE=1`, as the builders' are) | | `RX_VK_NO_PROGRAM_CACHE` (set) | Neither reads nor writes `hookshot_programs.bin` and the pipeline notes | | `RX_VK_MIRROR=0`, `RX_VK_MIRROR_MB=` | No VRAM mirrors of the game's memory pools (on for discrete cards); their budget | | `RX_VK_VRAM_MB=` | The texture cache's budget (default 70% of what the driver allows, at most 6 GB) | | `RX_VK_COMPACT=1`, `RX_VK_COMPACT_PCT=<50..90>`, `RX_VK_COMPACT_MB=` | VRAM compaction (off by default): the textures that are not render targets in a pool of their own, moved out of sparsely used blocks a few a present so that the blocks go back to the driver, and the texture budget holding the driver's count at that share of what it allows (85); the free space that starts a run (512; 192 while the count is over 90%) | +| `RX_VK_SKIP_COMPRESSED_OVER_RT=1` | A compressed (ASTC, BC, ETC2/EAC) texture made over storage render targets used is not uploaded, as before: bosses and new enemies all blue the first time | | `RX_VK_NO_WRITEBACK` (set) | GPU writes to mirrored pools are not copied back to guest memory (an experiment, for A/B) | | `RX_VK_WB_DIFF=0` | Write-backs copy every byte again instead of only what changed against a VRAM shadow (the diff is on by default; v2.0.9 and v2.0.10 had it off, issue 1161) | | `RX_VK_DLSS=0/1/2`, `RX_VK_DLSS_PRESET=`, `RX_VK_DLSS_LOG=1`, `RX_VK_REFLEX`, `RX_VK_REFLEX_LOG=1`, `RX_VK_ANTILAG=0/1` | NVIDIA DLSS and Reflex, and AMD Anti-Lag 2, for one run (`=-` on Reflex or Anti-Lag 2: its extensions not even enabled): the readme's NVIDIA DLSS section. `RX_VK_DLSS_TRACE=` logs one frame's large draws every that many frames, `RX_VK_DLSS_DEBUG=motion` shows the motion vectors, `=alternate` the game's own upscale every other frame | diff --git a/docs/tested-configurations.md b/docs/tested-configurations.md index 6e65747..2de1863 100644 --- a/docs/tested-configurations.md +++ b/docs/tested-configurations.md @@ -16,6 +16,9 @@ otherwise. |---|---|---|---|---|---|---|---| | 2026-11-10 | host `v2.0.11-438-gcd544a7` (source `8bdd7a54`); renderer main `8bac8aa` + #1502 + the host-clock capture change | Windows 11 Home N, 26300 | RX 9070 XT 16 GB, 32.0.32015.12012 | Ryzen 7 5700X3D, 64 GB | 2560x720 window, 32:9 camera, serial, FG off, dense capture, core validation; fresh prepared shader bank | Boot, title, save load; gameplay not reached | 96 s: 1,476 consecutive original capture records; all 1,461 records covered by the flushed native trace align one-to-one with native returns. Movement window rejected: no filled gameplay interval. No new validation IDs; existing `02251`, and two `08737` errors during full preparation. No gameplay speed or performance claim | | 2026-11-10 | v2.0.11-447-gcbe49eb, sampler admission branch (main 8bac8aa + sampler checks + staging fixture repair) | Windows 11, windowed | RTX 5070 Ti, 621.01 | Ryzen 7 9800X3D, 32 GB | 3440x1440 picture, native upscaling, fixed FG 2x, 1 MB recording streams; serial and parallel replay (2 workers, batch 1) | Kokiri Forest, inside Link's house; copied checkpoint | Both runs rendered the scene, maximized/restored (client 1444x720 to 1440x829 and back), and exited normally. Six representative PNGs decoded at 3440x1440; original checkpoint hashes unchanged. Healthy-path validation only; no FPS comparison or native OOM recovery claim | +| 2026-11-10 | host `0f5d423a` (API13 diagnostic backport), renderer main `dadecd53` | Windows 11 | RX 9070 XT 16 GB, 32.0.32015.12012 | Ryzen 7 5700X3D, 64 GB | 32:9, serial, FG off, GPU profiling/validation off; menu-only CPU program bank plus fixed warmed application pipeline caches; sparse screenshots | Hyrule Field, first entry, control first | 96 s; reached gameplay, terrain checked, no fatal error, original saves unchanged. 557 readiness waits totaling 30.725 s; longest native-present interval 32.568 s. Empty 40-60 s window rejected. Driver/OS caches uncontrolled; no FPS claim | +| 2026-11-10 | same host, renderer `d39e988c` (#1573) | as above | as above | as above | identical nine-file starting cache hashes and settings | Hyrule Field, first entry, candidate second | 96 s; terrain/grass/castle checked, no fatal error, original saves unchanged. 815 readiness waits totaling 3.120 s; longest native-present interval 2.114 s. CPU decision tests do not prove GPU writes; no speedup claim | +| 2026-11-10 | same host, renderer main `dadecd53` | as above | as above | as above | identical nine-file starting cache hashes and settings | Hyrule Field, first entry, control repeated third | 96 s; terrain checked, no fatal error, original saves unchanged. 557 readiness waits totaling 0.923 s; longest native-present interval 2.128 s. Unchanged control also lost its long freeze: do not attribute the first/second difference to #1573 | | 2026-11-10 | main b2bb214 (+ #1447 #1451 #1520 #1660 #1619 #1448 #1654 #1612 #1600, unmerged then) | Ubuntu 26.04.1, KDE Plasma 6.6.6 Wayland | RTX 4070 Ti 12 GB, 595.99.02 | Ryzen 7 9800X3D, 32 GB | 3840x2160 output, DLSS Quality, no mods | Hyrule Castle road (child) | 48-50 fps. Same spot at 2560x1440 output: 56.3 fps. No device lost, saves written | | 2026-11-10 | main b2bb214 + #1459 (Dear ImGui menu) | as above | as above | as above | as above, windowed | Title, Hyrule Castle road | F1 menu, help box, prompts, the game's font and the FPS badge all correct; 60 fps | | 2026-11-10 | main b2bb214 | as above | as above | as above | HDR on | start-up | KDE offers an HDR10 surface (`vk: HDR: an HDR10 surface (format 58)`) | diff --git a/gpu/vk/vk_dlss.cpp b/gpu/vk/vk_dlss.cpp index 0abd2e3..72409b5 100644 --- a/gpu/vk/vk_dlss.cpp +++ b/gpu/vk/vk_dlss.cpp @@ -339,6 +339,7 @@ int prepare(VkCommandBuffer cmd, uint32_t rw, uint32_t rh, uint32_t ow, uint32_t // (LDR colour, already tonemapped: no exposure; motion at the render size, not jittered; depth not reversed) cp.InFeatureCreateFlags = NVSDK_NGX_DLSS_Feature_Flags_MVLowRes; const auto t0 = std::chrono::steady_clock::now(); + ::mkwvk::rec::boundary(::mkwvk::rec::B_DIRECT); // (NGX records into cmd itself: what this thread recorded through vk_work.h reaches cmd first) const NVSDK_NGX_Result r = NGX_VULKAN_CREATE_DLSS_EXT(cmd, 1, 1, &s.feature, s.params, &cp); const double made_ms = std::chrono::duration(std::chrono::steady_clock::now() - t0).count(); note_work(); // (NGX's own commands, which vk_work.h does not see: the next full_barrier is not left out) @@ -371,6 +372,7 @@ bool evaluate(VkCommandBuffer cmd, const TemporalFrame& f) { ep.InRenderSubrectDimensions = {f.render_w, f.render_h}; ep.InReset = f.reset ? 1 : 0; ep.InMVScaleX = ep.InMVScaleY = 1.0f; + ::mkwvk::rec::boundary(::mkwvk::rec::B_DIRECT); // (NGX records into cmd itself: what this thread recorded through vk_work.h reaches cmd first) const NVSDK_NGX_Result r = NGX_VULKAN_EVALUATE_DLSS_EXT(cmd, s.feature, s.params, &ep); note_work(); // (as at its creation: the barrier after it, and the game's next one, are recorded) if (NVSDK_NGX_FAILED(r)) { diff --git a/gpu/vk/vk_draw.cpp b/gpu/vk/vk_draw.cpp index 8d6faab..ac7855f 100644 --- a/gpu/vk/vk_draw.cpp +++ b/gpu/vk/vk_draw.cpp @@ -5832,15 +5832,13 @@ void draw(VkCommandBuffer cmd, NvnState& s, const DrawArgs& d) { if (skip) return; } } - // Not ready (translating, or its pipeline building): left out of the frame when its targets hold an earlier - // frame's picture already, which shows for a frame without it; waited for otherwise (a pass drawn once, a target - // made this frame), as totk-pc-recomp's TOTK_PIPE_SKIP rule. RX_VK_PIPE_WAIT=1: always waited for. - // (waited for by default: a draw left out showed as missing geometry in races, where many programs are new; - // RX_VK_PIPE_SKIP=1: left out when its targets hold an earlier frame's picture, as before) - // (hookshot: left out by default. Waiting, Ocarina froze 0.2-3 s at each area's first new programs, the most reported stutter; - // RX_VK_PIPE_SKIP=0: waited for, as mkw-recomp needs) + // Wait for translation before reading its stages. A pending pipeline may be left out only when no stage writes + // global memory or storage buffers/images and its targets hold an earlier frame's picture. RX_VK_PIPE_SKIP=0: + // pipelines are always waited for too. + // Waiting on unknown translations preserves writes but can restore first-visit pauses (previously 0.2-3 s). + // RX_VK_PIPE_SKIP=2 restores the old translation skip for persistent targets; after translation, memory-writing pipelines wait. static const bool always_wait = getenv("RX_VK_PIPE_SKIP") && atoi(getenv("RX_VK_PIPE_SKIP")) == 0; - const bool may_skip = !always_wait && targets_persistent(at); + static const bool legacy_translation_skip = getenv("RX_VK_PIPE_SKIP") && atoi(getenv("RX_VK_PIPE_SKIP")) == 2; auto wait_for = [&](const std::atomic& ready) { PROF_EXACT(); const auto t0 = std::chrono::steady_clock::now(); @@ -5849,7 +5847,7 @@ void draw(VkCommandBuffer cmd, NvnState& s, const DrawArgs& d) { st_wait_ms += std::chrono::duration(std::chrono::steady_clock::now() - t0).count(); }; if (!cp->ready.load(std::memory_order_acquire)) { - if (may_skip) return void(st_pending_skips++); + if (legacy_translation_skip && targets_persistent(at)) return void(st_pending_skips++); wait_for(cp->ready); } if (exposure_log_on() && cp) exposure_note(s, *cp); @@ -5880,6 +5878,19 @@ void draw(VkCommandBuffer cmd, NvnState& s, const DrawArgs& d) { pe = graphics_pipeline(p, s, at, topo, types); } if (!pe->ready.load(std::memory_order_acquire)) { + bool may_skip = !always_wait && targets_persistent(at); + if (may_skip) + for (const Stage& st : p.stages) { + const auto& info = st.info; + const auto written = [](const auto& ds) { + return std::any_of(ds.begin(), ds.end(), [](const auto& d) { return d.is_written; }); + }; + if (info.stores_global_memory || written(info.storage_buffers_descriptors) || + written(info.image_buffer_descriptors) || written(info.image_descriptors)) { + may_skip = false; + break; + } + } if (may_skip) return void(st_pending_skips++); wait_for(pe->ready); } diff --git a/gpu/vk/vk_fsr4.cpp b/gpu/vk/vk_fsr4.cpp index 1096152..f76b977 100644 --- a/gpu/vk/vk_fsr4.cpp +++ b/gpu/vk/vk_fsr4.cpp @@ -412,6 +412,7 @@ bool evaluate(VkCommandBuffer cmd, const TemporalFrame& f) { d.cameraNear = f.near_z, d.cameraFar = f.far_z, d.cameraFovAngleVertical = f.fov_y; d.viewSpaceToMetersFactor = 1.0f; d.flags = FFX_UPSCALE_FLAG_NON_LINEAR_COLOR_SRGB; + ::mkwvk::rec::boundary(::mkwvk::rec::B_DIRECT); // (ffxDispatch records into cmd itself: what this thread recorded through vk_work.h reaches cmd first) const ffxReturnCode_t r = l.dispatch(&s.context->context, &d.header); note_work(); // (AMD's dispatches: full_barrier must not take the next barrier for a repeat of the one before them) diff --git a/gpu/vk/vk_gprof.cpp b/gpu/vk/vk_gprof.cpp index c5531f9..37a2bd3 100644 --- a/gpu/vk/vk_gprof.cpp +++ b/gpu/vk/vk_gprof.cpp @@ -7,6 +7,7 @@ #include #include "vk_gprof.h" +#include "vk_work.h" // (after the others: its timestamps go through RX_VK_RECORDER with the commands around them) namespace mkwvk { namespace { diff --git a/gpu/vk/vk_internal.h b/gpu/vk/vk_internal.h index acc5e15..c2d7902 100644 --- a/gpu/vk/vk_internal.h +++ b/gpu/vk/vk_internal.h @@ -112,6 +112,8 @@ void memory_table_begin(); // (hookshot: a recording's p bool memory_table_flush(); uint64_t memory_guest_pages(); // (hookshot: the device address of guest memory's page table, 0: none) void memory_sync(); // (at each submit: pools the game made or let go) +extern std::atomic memsync_lock_wait_ns; // (hookshot, diagnostics: time memory_sync waited for its two locks; the slow-submit line) +extern std::atomic memsync_part_ns[3]; // (hookshot, diagnostics: submit_memory_work's memory_sync, memory_take_capture, memory_collect_written) BufferRef gpu_buffer(uint64_t gpu, uint64_t size); // (empty when no pool holds the range) /// A range as a binding of the replay sees it: the pool's VRAM mirror when it has one (vk_memory.cpp), else the /// imported pool. written: the GPU may write it (copied back to guest memory at the submission's end). @@ -197,6 +199,7 @@ uint64_t gpu_pool_start(uint64_t gpu); // (EvilOOT) and where it begins bool create_buffer_cached(uint64_t size, VkBufferUsageFlags usage, VkBuffer& buffer, VmaAllocation& allocation, void** mapped, bool dedicated = false); // (vk_vma.cpp: cached host memory, mapped) void flush_mapped(VmaAllocation allocation, uint64_t size); +bool host_cached(VmaAllocation allocation); // (its memory type has VK_MEMORY_PROPERTY_HOST_CACHED_BIT) void flush_mapped_range(VmaAllocation allocation, uint64_t offset, uint64_t size); void invalidate_mapped(VmaAllocation allocation, uint64_t size); // (the CPU is to read what the GPU wrote: a no-op in coherent memory) void memory_stamp_gpu(uint64_t guest, uint64_t size); // (a GPU write: memory_stamp, and the pages noted) @@ -245,6 +248,8 @@ bool has_stencil(VkFormat f); // device memory (vk_vma.cpp, Vulkan Memory Allocator) bool vma_init(PFN_vkGetInstanceProcAddr gipa); +extern uint64_t test_card_ballast; // (vk_vma.cpp, RX_VK_TEST_CARD_MB: 0 outside that lab test) +void vma_blocks_ahead_check(); // (vk_vma.cpp: the blocks made ahead given back while VRAM is short; every 600 presents) /// (textures: from the textures' own pool, the one image_compact_* moves allocations in: vk_texture.cpp's compaction) bool create_image(const VkImageCreateInfo& ci, VkImage& image, VmaAllocation& allocation, bool dedicated = false, bool target = false, bool textures = false); diff --git a/gpu/vk/vk_jobs.cpp b/gpu/vk/vk_jobs.cpp index 0ca5cc9..7fc8587 100644 --- a/gpu/vk/vk_jobs.cpp +++ b/gpu/vk/vk_jobs.cpp @@ -59,7 +59,10 @@ void worker() { } void start() { - const unsigned n = std::clamp(std::thread::hardware_concurrency() / 3, 2u, 4u); + // (hookshot: with 16 hardware threads or more, 6 workers; RX_VK_WORKERS=: n, 1 to 16) + const unsigned hw = std::thread::hardware_concurrency(); + unsigned n = hw >= 16 ? 6u : std::clamp(hw / 3, 2u, 4u); + if (const char* e = getenv("RX_VK_WORKERS")) n = (unsigned)std::clamp(atoi(e), 1, 16); for (unsigned i = 0; i < n; i++) std::thread(worker).detach(); started = true; logf("vk: %u background workers (shader translation, pipelines)", n); diff --git a/gpu/vk/vk_memory.cpp b/gpu/vk/vk_memory.cpp index fc46aad..4e6bdc2 100644 --- a/gpu/vk/vk_memory.cpp +++ b/gpu/vk/vk_memory.cpp @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -67,6 +68,13 @@ struct Pool { uint64_t direct_write_epoch = 0; // (the part that last wrote it in guest memory, before it had a mirror) uint64_t retired_value = 0; // (retired: the submission that may still use it) bool was_mirrored = false; // (its mirror was given back: its pages stay in the captures, for the next one) + // (hookshot) the GPU wrote it past what is copied back (memory_note_gpu_write: a bare entry's guessed range, the pointer + // table's, a wide binding's): that is in its mirror alone, so the mirror is never given back (mirrors_give_back) + bool gpu_only = false; + // (and the GPU wrote it through a binding of its mirror, copied back or not: after a save reload, the mirrors of such pools + // given back and made again left cyan in play and over the Start menu (RX 5500M, gone with RX_VK_MIRROR_KEEP=1); kept too) + bool gpu_written = false; + bool inherit_lost = false; // (retired, and the pool that took it in made no mirror in time to copy its GPU contents) // (EvilOOT) retired with its mirror on: the pool that took it in copies the GPU's contents from it when its own mirror is // made (mirror_make), and until then the mirror stays (mirrors_give_back) bool inherit_pending = false; @@ -116,6 +124,9 @@ Pool* last_pool = nullptr; // (the last pool_of hit: draws in a row b extern Pool* pool_cache[4096]; void retire_pool(std::map::iterator it); Pool* pool_lookup(uint64_t guest, uint64_t size); +// (Triforce, 11-12) why the import a binding needs is made on its own thread and not in the background (import_in_background, +// gpu_buffer_bind), for the import's log line; nullptr: not a binding's, or not asked +thread_local const char* tl_sync_why = nullptr; struct SrwMutex { // (a slim reader/writer lock: half std::mutex's cost, taken for every buffer a draw binds) SRWLOCK l = SRWLOCK_INIT; void lock() { AcquireSRWLockExclusive(&l); } @@ -696,9 +707,17 @@ bool memory_init() { } /// Refresh the guest's pool table (no Vulkan work: pools are imported as they are used) +std::atomic memsync_lock_wait_ns{0}; +std::atomic memsync_part_ns[3]; void memory_sync() { static std::mutex sync_mu; // (called from every thread that touches memory) + const auto w0 = std::chrono::steady_clock::now(); std::lock_guard sync_lk(sync_mu); + struct Waited { // (diagnostics: only the lock waits, added up; nothing else changes) + std::chrono::steady_clock::time_point t; + ~Waited() {} + } waited{w0}; + memsync_lock_wait_ns += std::chrono::duration_cast(std::chrono::steady_clock::now() - waited.t).count(); const uint32_t g = host->maps_generation(); if (g == generation) return; generation = g; @@ -714,7 +733,9 @@ void memory_sync() { next.push_back(p); } std::sort(next.begin(), next.end(), [](const GuestPool& a, const GuestPool& b) { return a.lo < b.lo; }); + const auto w1 = std::chrono::steady_clock::now(); std::lock_guard lk(pools_mu); + memsync_lock_wait_ns += std::chrono::duration_cast(std::chrono::steady_clock::now() - w1).count(); guest_pools.swap(next); guest_pools_gen.fetch_add(1, std::memory_order_relaxed); last_pool = nullptr; @@ -975,7 +996,65 @@ struct Capture { size_t nbytes = 0; std::vector late; // (indices into pages: copied again by the replay once late_value is done) uint64_t late_value = 0; + // (hookshot) "shadow staging": the texture pages of this capture, copied from guest memory into ordinary cached RAM too, for + // take_capture's texture shadow to read instead of the buffer above (often write-combined here: CPU reads ~0.3 GB/s). + // sidx[i] = the staging slot of pages[i] (4 KB a slot), or -1: that page is read from the buffer. Empty sidx: no staging. + std::unique_ptr stage; + size_t stage_bytes = 0; + std::vector sidx; + bool buf_cached = false; // (its capture buffer's memory is HOST_CACHED: for the staging log line) }; +// (the staging blocks, kept for the next capture: a fresh 20 MB block is ~5000 page faults on the game's thread) +std::mutex stage_mu; // (a leaf lock) +std::vector, size_t>> stage_free; +size_t stage_free_bytes = 0; +// (Triforce: upstream's two switch helpers (gpu/vk/compat/hookshot_env.h), with the same rule, for the code ported from there. +// This tree has no such header yet, and a file added under compat/ would change the shaders' stamp, so every folder would +// prepare its shaders again for it. A value says off when it is "", "0", "false", "off" or "no", any case) +bool env_value_off(const char* v) { + while (*v == ' ' || *v == '\t') v++; + std::string s(v); + while (!s.empty() && (s.back() == ' ' || s.back() == '\t' || s.back() == '\r' || s.back() == '\n')) s.pop_back(); + for (char& c : s) c = (char)(c | 0x20); + return s.empty() || s == "0" || s == "false" || s == "off" || s == "no"; +} +bool env_on(const char* name) { + const char* v = getenv(name); + return v && !env_value_off(v); +} +bool stage_on() { // (RX_VK_SHADOW_STAGE=0 / off / false / no: the shadow reads the capture buffer, as before) + static const bool on = [] { const char* e = getenv("RX_VK_SHADOW_STAGE"); return !e || !env_value_off(e); }(); // (on unless an off value: env_value_off) + return on; +} +bool stage_get(Capture& c, size_t need) { // (a block of at least need bytes into c: a kept one, else a new one) + { + std::lock_guard lk(stage_mu); + size_t best = stage_free.size(); + for (size_t i = 0; i < stage_free.size(); i++) + if (stage_free[i].second >= need && (best == stage_free.size() || stage_free[i].second < stage_free[best].second)) best = i; + if (best < stage_free.size()) { + c.stage = std::move(stage_free[best].first), c.stage_bytes = stage_free[best].second; + stage_free_bytes -= c.stage_bytes; + stage_free.erase(stage_free.begin() + best); + return true; + } + } + const size_t sz = (need + (1u << 20) - 1) & ~size_t((1u << 20) - 1); // (rounded to 1 MB: blocks of near sizes fit each other) + c.stage.reset(new (std::nothrow) uint8_t[sz]); + c.stage_bytes = c.stage ? sz : 0; + return c.stage != nullptr; +} +void stage_put(Capture& c) { // (the block back to the kept ones, at most four and 96 MB; else freed) + c.sidx.clear(); + if (!c.stage) return; + std::lock_guard lk(stage_mu); + static const bool low_spec = env_on("RX_VK_LOW_SPEC"); // (RAM the tight resource there: no block kept) + if (!low_spec && stage_free.size() < 4 && stage_free_bytes + c.stage_bytes <= (96ull << 20)) { + stage_free_bytes += c.stage_bytes; + stage_free.emplace_back(std::move(c.stage), c.stage_bytes); + } + c.stage.reset(), c.stage_bytes = 0; +} std::mutex cap_mu; std::deque> captures; std::vector> spare; // (taken ones, their buffers kept) @@ -1022,6 +1101,7 @@ bool sync_live() { static const bool v = getenv("RX_VK_SYNC_LIVE") != nullptr; return v; } +std::atomic cap_shutting_down{false}; // (set under cap_free_mu when the process starts to exit: after it no worker touches cap_free) std::mutex cap_free_mu; // (the submitting threads take buffers from cap_free, the replay gives them back) uint64_t cap_free_bytes = 0; // (cap_free_mu) // (the spare buffers: 12 and, on a card of 6 GB or less, 32 MB of them; RX_VK_CAP_SPARE_MB=: that many, 0: as before. @@ -1034,9 +1114,14 @@ uint64_t cap_spare_max() { }(); return v; } -void cap_buf_free(VkBuffer buffer, VmaAllocation alloc, void* map, uint64_t size) { +void cap_buf_free(VkBuffer buffer, VmaAllocation alloc, void* map, uint64_t size, bool from_worker = false) { std::lock_guard lk(cap_free_mu); - if (cap_free.size() < 12 && size <= (64ull << 20) && cap_free_bytes + size <= cap_spare_max()) { + if (from_worker && cap_shutting_down.load()) return; + // (hookshot, RX 7800 XT: buffers up to 256 MB are kept for the next capture, not only up to 64 MB: a new one costs ~100 ms of + // vkAllocateMemory on AMD's driver. RX_VK_CAPTURE_KEEP_OLD=1: as before) + static const bool keep_old = getenv("RX_VK_CAPTURE_KEEP_OLD") != nullptr; + if (cap_free.size() < 12 && size <= (keep_old ? 64ull << 20 : 256ull << 20) && + cap_free_bytes + size <= std::min(cap_spare_max(), keep_old ? ~0ull : 768ull << 20)) { cap_free.push_back({buffer, alloc, map, size}); cap_free_bytes += size; } else { @@ -1055,10 +1140,86 @@ uint64_t cap_buf_size(uint64_t need) { static const uint64_t min = (getenv("RX_VK_CAP_MIN_MB") ? (uint64_t)std::max(1, atoi(getenv("RX_VK_CAP_MIN_MB"))) : 2u) << 20; uint64_t s = min; while (s < need && s < (16ull << 20)) s <<= 1; - return s >= need ? s : (need + (1u << 20) - 1) & ~((1ull << 20) - 1); + if (s >= need) return s; + // (hookshot: above 16 MB in classes of 32, 64, 128 and 256 MB, so that a buffer given back fits the next capture) + static const bool keep_old = getenv("RX_VK_CAPTURE_KEEP_OLD") != nullptr; + if (!keep_old) + for (const uint64_t cm : {32, 64, 128, 192, 256}) // (stutter: 192 added, a 130-158 MB capture took the 256 MB class) + if ((uint64_t)cm << 20 >= need) return (uint64_t)cm << 20; + return (need + (1u << 20) - 1) & ~((1ull << 20) - 1); +} +// (hookshot, RX_VK_CAP_PREWARM, on by default since the stutter fix; =0: off) The big capture buffers made ahead, on a worker thread. A capture of a scene load holds +// 16 to 256 MB of the pages the game wrote (texture storage), and its buffer was made on the game's thread, inside capture_make: +// AMD's driver commits and clears a host-visible memory object when it is allocated, ~0.8 ms a MB, and the log's isolated spikes +// ("memory sync ... capture 66.8", "capture 107.7", "capture 22.6", "capture 17.4": 5 in a session, each at a load, none in steady +// play) are that, in proportion to the class (128, 64 and 32 MB classes, and 128, 64 MB). The spares only come back from a capture +// retired past the 24 kept ones, of whatever size that one was, so a big class is rarely there when a load asks for it. With this +// a spare of each class of 32, 64 and 128 MB is made in the background once, and again as soon as one is taken. +bool cap_prewarm_on() { + // (stutter: on by default now; RX_VK_CAP_PREWARM=0 turns it off. The log at the scene loads: "capture ... the buffer of 32/64/256 MB 13-41 ms") + static const bool v = [] { + const char* e = getenv("RX_VK_CAP_PREWARM"); + const bool on = !e || atoi(e) != 0; + logf("vk: capture buffer prewarm (32, 64, 128 MB spares made on a worker) %s (RX_VK_CAP_PREWARM)", on ? "on" : "off"); + return on; + }(); + return v; +} +std::atomic cap_prewarm_busy{0}; +std::atomic cap_prewarm_class_busy[4]{}; // (32, 64, 128 and 192 MB spares being made) +void cap_exit_note() { + std::lock_guard lk(cap_free_mu); // (a worker inside cap_buf_free finishes first; one after this sees the flag under the same mutex) + cap_shutting_down = true; +} +int cap_prewarm_index(uint64_t size) { return size <= (32ull << 20) ? 0 : size <= (64ull << 20) ? 1 : size <= (128ull << 20) ? 2 : size <= (192ull << 20) ? 3 : -1; } +void cap_spare_make(uint64_t size, int index) { + VkBuffer b = VK_NULL_HANDLE; + VmaAllocation a = nullptr; + void* m = nullptr; + const auto t0 = std::chrono::steady_clock::now(); + if (!cap_shutting_down.load() && create_buffer_cached(size, VK_BUFFER_USAGE_TRANSFER_SRC_BIT, b, a, &m, !getenv("RX_VK_CAP_BLOCKS")) && m) { + static std::atomic told{0}; + if (told++ < 8) + logf("vk: a %llu MB capture buffer made ahead in %.1f ms, off the game's thread (RX_VK_CAP_PREWARM)", (unsigned long long)(size >> 20), + std::chrono::duration(std::chrono::steady_clock::now() - t0).count()); + // (a spare, within the same limits as any other: past them it is let go. Not once the process is exiting: the list and the device + // may be going, and the memory goes with the process) + cap_buf_free(b, a, m, size, true); + } + cap_prewarm_class_busy[index]--; + cap_prewarm_busy--; +} +/// A spare of at least `size` (32, 64 or 128 MB classes only) is there, or one of its class is being made (cap_free_mu held) +void cap_prewarm_class(uint64_t size) { + if (!cap_prewarm_on()) return; + const int index = cap_prewarm_index(size); + if (index < 0 || size > cap_spare_max()) return; // (not the 256 MB class, and not past what spares may hold) + static const uint64_t kCls[4] = {32, 64, 128, 192}; + const uint64_t cls = kCls[index]; + const uint64_t bytes = cls << 20; + for (const FreeBuf& f : cap_free) + if (f.size >= bytes) return; + if (cap_prewarm_class_busy[index].load() > 0 || cap_prewarm_busy.load() >= 3) return; // (three at a time: each commits its memory) + static const bool exit_hook = [] { return std::atexit(cap_exit_note) == 0; }(); + (void)exit_hook; + cap_prewarm_busy++; + cap_prewarm_class_busy[index]++; + try { + std::thread(cap_spare_make, bytes, index).detach(); + } catch (...) { // (no thread: no spare, the capture goes on) + cap_prewarm_class_busy[index]--; + cap_prewarm_busy--; + } } bool cap_buf_get(uint64_t need, VkBuffer& buffer, VmaAllocation& alloc, void*& map, uint64_t& size) { std::lock_guard lk(cap_free_mu); + if (cap_prewarm_on()) { + static bool first = true; + if (first) { // (the first capture: the classes a load asks for) + first = false; + for (uint64_t c : {32ull << 20, 64ull << 20, 128ull << 20, 192ull << 20}) cap_prewarm_class(c); + } + } size_t best = cap_free.size(); for (size_t i = 0; i < cap_free.size(); i++) if (cap_free[i].size >= need && (best == cap_free.size() || cap_free[i].size < cap_free[best].size)) best = i; @@ -1256,13 +1417,19 @@ void shadow_drop_range(uint64_t lo, uint64_t hi) { // (shadow_mu held) /// (the spans unused for 1800 presents dropped with their shadow pages; the rest published for the capture) void publish_tex_spans(uint64_t frame) { std::lock_guard lk(shadow_mu); - for (auto it = tex_spans.begin(); it != tex_spans.end();) { - if (frame > it->second.second + 1800) { - shadow_drop_range(it->first, it->second.first); - it = tex_spans.erase(it); - tex_spans_changed = true; - } else { - ++it; + // (a span ages by the present: looked over once a present, not at each of its ~20 submissions, which walked + // the ~300 spans every time, 1 ms a frame on a Steam Deck) + static uint64_t swept = ~0ull; + if (frame != swept) { + swept = frame; + for (auto it = tex_spans.begin(); it != tex_spans.end();) { + if (frame > it->second.second + 1800) { + shadow_drop_range(it->first, it->second.first); + it = tex_spans.erase(it); + tex_spans_changed = true; + } else { + ++it; + } } } if (!tex_spans_changed) return; @@ -1680,6 +1847,8 @@ struct CopyPart { uint8_t* ok; const CopyPair* pairs = nullptr; int* left = nullptr; // (the caller's parts not done yet (copy_helpers.mu): a caller waits for its own, not another's too) + uint8_t* stage = nullptr; // (guest copies only: the pages with sidx[i] >= 0 are copied into stage + sidx[i] * 4096 first, and out from there) + const int32_t* sidx = nullptr; }; void copy_part(const CopyPart& w) { if (w.pairs) { @@ -1693,12 +1862,46 @@ void copy_part(const CopyPart& w) { if (ga + 4096 > gsize) { i++; continue; } int64_t j = i + 1; while (j < w.i1 && w.pages[j] == w.pages[j - 1] + 1 && ((uint64_t)w.pages[j] << kPageShift) + 4096 <= gsize) j++; - if (guarded_copy(w.out + (uint64_t)i * 4096, base + ga, (SIZE_T)(j - i) * 4096)) { - std::memset(w.ok + i, 1, (size_t)(j - i)); + // (a run in out alone; a run of staged pages: guest memory into the staging first, then out from the staging, so the two + // hold the same bytes even if the game writes the page meanwhile; a page that fails is ok = 0 in both and not used) + auto plain = [&](int64_t a, int64_t b) { + if (guarded_copy(w.out + (uint64_t)a * 4096, base + ((uint64_t)w.pages[a] << kPageShift), (SIZE_T)(b - a) * 4096)) { + std::memset(w.ok + a, 1, (size_t)(b - a)); + } else { + for (int64_t q = a; q < b; q++) + w.ok[q] = guarded_copy(w.out + (uint64_t)q * 4096, base + ((uint64_t)w.pages[q] << kPageShift), 4096); + copy_fence(); + } + }; + auto staged = [&](int64_t a, int64_t b) { + uint8_t* const st = w.stage + (uint64_t)w.sidx[a] * 4096; + if (guarded_copy(st, base + ((uint64_t)w.pages[a] << kPageShift), (SIZE_T)(b - a) * 4096)) { + std::memcpy(w.out + (uint64_t)a * 4096, st, (size_t)(b - a) * 4096); + std::memset(w.ok + a, 1, (size_t)(b - a)); + } else { + for (int64_t q = a; q < b; q++) { + uint8_t* const sq = w.stage + (uint64_t)w.sidx[q] * 4096; + const bool good = guarded_copy(sq, base + ((uint64_t)w.pages[q] << kPageShift), 4096); + if (good) std::memcpy(w.out + (uint64_t)q * 4096, sq, 4096); + w.ok[q] = good; + } + copy_fence(); + } + }; + if (!w.sidx) { + plain(i, j); } else { - for (int64_t q = i; q < j; q++) - w.ok[q] = guarded_copy(w.out + (uint64_t)q * 4096, base + ((uint64_t)w.pages[q] << kPageShift), 4096); - copy_fence(); + for (int64_t q = i; q < j;) { // (split at the staged / not staged edges; staged slots are consecutive for adjacent pages) + int64_t r = q + 1; + if (w.sidx[q] >= 0) { + while (r < j && w.sidx[r] == w.sidx[r - 1] + 1) r++; + staged(q, r); + } else { + while (r < j && w.sidx[r] < 0) r++; + plain(q, r); + } + q = r; + } } i = j; } @@ -1736,7 +1939,7 @@ int capture_copy_helpers() { // (copy_pages and copy_pairs share one pool of h /// queued before them (copy_pairs runs with pools_mu and shadow_mu held, copy_pages' guest copies can take milliseconds) void copy_split(CopyPart whole, int64_t n) { const int helpers = capture_copy_helpers(); - if (!helpers || n < 128) return copy_part({whole.pages, 0, n, whole.out, whole.ok, whole.pairs}); + if (!helpers || n < 128) { whole.i0 = 0, whole.i1 = n; return copy_part(whole); } const int k = helpers + 1; int left = k - 1; auto part = [&](int i) { @@ -1767,13 +1970,17 @@ void copy_split(CopyPart whole, int64_t n) { } copy_helpers.done.wait(lk, [&] { return left == 0; }); } -void copy_pages(const uint32_t* pages, int64_t n, uint8_t* out, uint8_t* ok) { // (one capture at a time: submit_order_mu) - copy_split({pages, 0, n, out, ok}, n); +void copy_pages(const uint32_t* pages, int64_t n, uint8_t* out, uint8_t* ok, uint8_t* stage = nullptr, const int32_t* sidx = nullptr) { // (one capture at a time: submit_order_mu) + CopyPart w{pages, 0, n, out, ok}; + w.stage = stage, w.sidx = sidx; + copy_split(w, n); } /// (hookshot) n 4 KB copies on the capture's copy helpers too: the texture shadow's (take_capture), thousands of pages a submit /// for OoT's game thread, which copied them one after another void copy_pairs(const CopyPair* pairs, int64_t n) { - copy_split({nullptr, 0, n, nullptr, nullptr, pairs}, n); + CopyPart w{nullptr, 0, n, nullptr, nullptr}; + w.pairs = pairs; + copy_split(w, n); } std::unique_ptr capture_make(); @@ -1797,12 +2004,14 @@ std::unique_ptr capture_make() { if (!c) c = std::make_unique(); c->pages.clear(), c->copied.clear(), c->slots.clear(), c->nbytes = 0, c->late.clear(), c->late_value = 0; if (c->map) cap_buf_free(c->buffer, c->alloc, c->map, c->bsize), c->buffer = VK_NULL_HANDLE, c->map = nullptr; + stage_put(*c); // (only the pools mirrored in VRAM, the ones the mirrors are filled from, and only their part of the write marks: // every written page was taken at first (hundreds of MB a submit where the game streams its world), and the whole // collection, over every GPU range, is ~20 ms: 4 of them a frame on the game's thread made it crawl. The replay's // collection takes the rest, as before) // (the list the replay publishes when it makes mirrors: pools_mu here waited ~40 ms a submit for the replay) static thread_local std::vector> mirrored; + static thread_local std::vector> texr; // (tex_cap_ranges, sorted and merged: the staged pages) { // (and the texture storage the replay uploads from: see "texture storage in the capture"; and the small pools) std::lock_guard lk(cap_ranges_mu); captures_made++; @@ -1821,6 +2030,14 @@ std::unique_ptr capture_make() { else mirrored.push_back(r); } + texr = tex_cap_ranges; + std::sort(texr.begin(), texr.end()); + size_t m = 0; + for (const auto& r : texr) { + if (m && r.first <= texr[m - 1].second) texr[m - 1].second = std::max(texr[m - 1].second, r.second); + else texr[m++] = r; + } + texr.resize(m); } } captures_seen.store(true, std::memory_order_relaxed); @@ -1876,7 +2093,27 @@ std::unique_ptr capture_make() { cap_buf_get((uint64_t)n * 4096, c->buffer, c->alloc, c->map, c->bsize)) { static thread_local std::vector ok; ok.assign((size_t)n, 0); - copy_pages(pages, n, (uint8_t*)c->map, ok.data()); + // (shadow staging: the pages in the texture ranges are also copied into c->stage, from guest memory first and the buffer + // from there. Not the late pages: take_capture copies those again into the buffer after the GPU wait, so the buffer is + // where their newer bytes are, and they stay unstaged (sidx -1: the shadow reads them from the buffer; a few a take)) + // (only where the capture buffer is not cached memory: a cached one the shadow reads as fast as the staging, and the + // extra copy would be the game thread's for nothing) + c->buf_cached = host_cached(c->alloc); + if (stage_on() && !texr.empty() && !c->buf_cached) { + c->sidx.assign((size_t)n, -1); + int32_t tn = 0; + size_t lj = 0; + for (int64_t i = 0; i < n; i++) { + while (lj < c->late.size() && c->late[lj] < (uint32_t)i) lj++; + if (lj < c->late.size() && c->late[lj] == (uint32_t)i) continue; + const uint64_t g = (uint64_t)pages[i] << kPageShift; + auto it = std::upper_bound(texr.begin(), texr.end(), std::make_pair(g, ~uint64_t(0))); + if (it != texr.begin() && g < std::prev(it)->second) c->sidx[(size_t)i] = tn++; + } + if (!tn || !stage_get(*c, (size_t)tn * 4096)) c->sidx.clear(); + } + copy_pages(pages, n, (uint8_t*)c->map, ok.data(), c->sidx.empty() ? nullptr : c->stage.get(), + c->sidx.empty() ? nullptr : c->sidx.data()); for (int64_t i = 0; i < n; i++) if (ok[i]) c->copied.push_back(pages[i]), c->slots.push_back((uint32_t)i); c->nbytes = (size_t)n * 4096; @@ -2331,6 +2568,8 @@ bool take_capture(std::unique_ptr c, uint64_t value, uint64_t completed } } bool swept = false; + static int64_t st_takes, st_from_stage, st_from_map, st_cached; // (the shadow copies by source and the takes whose buffer was cached, logged every 600 takes; shadow_mu) + int64_t from_stage = 0, from_map = 0; static thread_local std::vector pairs; // (the copies, made together once the slots are known) pairs.clear(); size_t i = 0; @@ -2356,10 +2595,20 @@ bool take_capture(std::unique_ptr c, uint64_t value, uint64_t completed shadow_n++; } slot.seq = shadow_seq; - pairs.push_back({slot.b.get(), (const uint8_t*)c->map + (uint64_t)c->slots[i] * 4096}); + // (the staging, ordinary RAM, when it holds the page; else the capture buffer, maybe write-combined) + const int32_t sx = c->sidx.empty() ? -1 : c->sidx[c->slots[i]]; + if (sx >= 0) pairs.push_back({slot.b.get(), (const uint8_t*)c->stage.get() + (uint64_t)sx * 4096}), from_stage++; + else pairs.push_back({slot.b.get(), (const uint8_t*)c->map + (uint64_t)c->slots[i] * 4096}), from_map++; } // (shadow_mu still held: no slot taken here goes before its copy; the sweep above drops only older ones) if (!pairs.empty()) copy_pairs(pairs.data(), (int64_t)pairs.size()); + st_from_stage += from_stage, st_from_map += from_map, st_cached += c->buf_cached ? 1 : 0; + if (++st_takes % 600 == 0) { + logf("vk: shadow staging (600 takes): %lld shadow copies from staging, %lld from the capture buffer; capture buffers: %lld " + "host-cached, %lld not (write-combined: staged)%s", (long long)st_from_stage, (long long)st_from_map, + (long long)st_cached, (long long)(600 - st_cached), stage_on() ? "" : " (RX_VK_SHADOW_STAGE=0)"); + st_from_stage = st_from_map = 0, st_cached = 0; + } for (; i < c->copied.size(); i++) // (the shadow full: the copied pages not replaced lose their old copies) if (in_tex_merged((uint64_t)c->copied[i] << 12)) shadow_erase(c->copied[i]); } @@ -2392,6 +2641,7 @@ bool take_capture(std::unique_ptr c, uint64_t value, uint64_t completed if (e.gen) cap_entry_set(page, CapEntry{0, 0}); } cap_view_bump(); + stage_put(*c); // (the shadow's copies from it are done: copy_pairs returned) if (c->map) cap_buf_free(c->buffer, c->alloc, c->map, c->bsize), c->buffer = VK_NULL_HANDLE, c->map = nullptr; { std::lock_guard lk(cap_mu); @@ -2714,9 +2964,9 @@ Pool* pool_lookup(uint64_t guest, uint64_t size) { static int n, logged; total_ms += ms, n++; if ((ms >= 2.0 && logged++ < 200) || n % 256 == 0) - logf("vk: pool %#llx+%#llx imported in %.1f ms as %#llx..%#llx (%d imports, %.0f ms in all)", + logf("vk: pool %#llx+%#llx imported in %.1f ms as %#llx..%#llx (%d imports, %.0f ms in all)%s%s", (unsigned long long)range.lo, (unsigned long long)(range.hi - range.lo), ms, (unsigned long long)lo, - (unsigned long long)hi, n, total_ms); + (unsigned long long)hi, n, total_ms, tl_sync_why ? "; not in the background: " : "", tl_sync_why ? tl_sync_why : ""); } if (!ok) { if (!import_cooling && failures++ < 16) { @@ -2731,7 +2981,8 @@ Pool* pool_lookup(uint64_t guest, uint64_t size) { } for (uint64_t k : overlapped) { // (the pools it took in) auto o = imported.find(k); - if (o != imported.end()) retire_pool(o); + if (o != imported.end()) // (their mirrors' contents come with them) + p.gpu_only |= o->second.gpu_only, p.gpu_written |= o->second.gpu_written, retire_pool(o); } imp = imported.emplace(p.lo, p).first; guest_pages_set(p.lo, p.hi, p.address); @@ -2787,7 +3038,8 @@ bool mirror_on() { /// under way (sent now or before) and nothing covers it yet; false: the synchronous path as before (covered, outside the game's /// pools, or next to an import that would have to grow). RX_VK_ASYNC_IMPORT=1 (opt-in: background imports slowed AMD before) (pools_mu held) int64_t st_async_sent, st_async_copies; -bool import_in_background(uint64_t guest, uint64_t size) { +bool import_in_background(uint64_t guest, uint64_t size, const char** why) { + *why = "outside the game's pools"; if (guest_pools.empty() || !size || !dev.external_memory_host) return false; bg_drain(); auto it = std::upper_bound(guest_pools.begin(), guest_pools.end(), guest, [](uint64_t a, const GuestPool& p) { return a < p.lo; }); @@ -2798,7 +3050,7 @@ bool import_in_background(uint64_t guest, uint64_t size) { if (!chunk || guest + size > UINT64_MAX - chunk) return false; if (auto c = imported.upper_bound(guest); c != imported.begin() && std::prev(c)->second.lo <= guest && std::prev(c)->second.hi >= guest + size) - return false; // (covered) + return *why = nullptr, false; // (covered) const uint64_t lo = guest / chunk * chunk, hi = (guest + size + chunk - 1) / chunk * chunk; for (auto o = imported.lower_bound(hi); o != imported.begin();) { // (an import there: the synchronous path grows it) --o; @@ -2806,12 +3058,13 @@ bool import_in_background(uint64_t guest, uint64_t size) { if (o->second.hi + (1ull << 30) < lo) break; continue; } - return false; + return *why = "an import beside it grows", false; } + *why = nullptr; std::lock_guard lk(bg_mu); for (const uint64_t x : bg_inflight) if (x >= lo && x < hi) return true; // (under way) - if (bg_inflight.size() >= 16) return false; + if (bg_inflight.size() >= 16) return *why = "16 imports under way", false; bg_start(); bg_inflight.push_back(lo); bg_queue.push_back({lo, hi}); @@ -2826,7 +3079,7 @@ uint64_t device_vram_bytes() { uint64_t vram = 0; for (uint32_t h = 0; h < dev.memory.memoryHeapCount; h++) if (dev.memory.memoryHeaps[h].flags & VK_MEMORY_HEAP_DEVICE_LOCAL_BIT) vram = std::max(vram, (uint64_t)dev.memory.memoryHeaps[h].size); - return vram; + return vram - std::min(vram, test_card_ballast); // (RX_VK_TEST_CARD_MB, a lab tool: the card plays a smaller one) } bool preset_caps_memory() { static const uint64_t nominal_gb[] = {0, 4, 6, 8, 12, 16}; @@ -2839,9 +3092,7 @@ uint64_t mirror_budget() { // (a quarter of the card, at most the 3 GB it was for all: on a 6 GB laptop those and the textures were past what the card has) static const uint64_t b = [] { if (const char* e = getenv("RX_VK_MIRROR_MB")) return (uint64_t)atoi(e) << 20; - uint64_t vram = 0; - for (uint32_t h = 0; h < dev.memory.memoryHeapCount; h++) - if (dev.memory.memoryHeaps[h].flags & VK_MEMORY_HEAP_DEVICE_LOCAL_BIT) vram = std::max(vram, (uint64_t)dev.memory.memoryHeaps[h].size); + const uint64_t vram = device_vram_bytes(); // (cards of 6 GB or less only: on 8 GB ones, a 3070 or a 4060, the game's memory read over PCIe instead left the GPU at a few // percent and the frame stuttering) if (vram && vram <= (6ull << 30) + (512ull << 20)) return vram / 8; @@ -3425,6 +3676,12 @@ void memory_note_gpu_write(uint64_t gpu, uint64_t size) { } std::lock_guard lk(pools_mu); rb_hazard_locked(guest, guest + size); + // (and the pools it reaches keep their mirrors: imports are at most 1 GB, so none further down reaches guest) + for (auto it = imported.upper_bound(guest + size - 1); it != imported.begin();) { + --it; + if (it->second.hi > guest) it->second.gpu_only = true; + else if (it->second.hi + (1ull << 30) < guest) break; + } } void memory_rb_recording_begun() { std::lock_guard lk(pools_mu); @@ -3774,20 +4031,38 @@ BufferRef gpu_buffer_bind(uint64_t gpu, uint64_t size, bool written) { } ParLock plk; // (before pools_mu, always: a mirror sync records into the shared upload command buffer) std::lock_guard lk(pools_mu); - { // (hookshot, smoothness: import_in_background above. Reads up to 4 MB; not in a parallel replay's segment) - static const bool async_import = [] { const char* e = getenv("RX_VK_ASYNC_IMPORT"); return e && atoi(e) != 0; }(); - if (async_import && !written && !par && size <= (4u << 20) && guest < (1ull << 40) && - !(last_pool && guest >= last_pool->lo && guest + size <= last_pool->hi) && import_in_background(guest, size)) { - uint64_t avail = 0; - const uint8_t* src = host->gpu_ptr(gpu, &avail); - if (src && avail >= size) - if (const BufferRef r = stream_copy(src, size)) { - st_async_copies++; - return r; - } + { // (hookshot, smoothness: import_in_background above. Not in a parallel replay's segment) + // (Triforce, 11-12) On by default on NVIDIA cards (RX_VK_ASYNC_IMPORT=0: off; =1: on, any card; the other makes as before, + // off: background imports slowed an AMD card's own). A player's session on an RTX 4070 SUPER had 68 imports on the replay + // thread in 3.3 minutes, 964 ms in all, all of it the driver's page locking (vkAllocateMemory with the host pointer) while + // the game's loaders keep the address space busy; most were for a binding of 64 to 144 bytes, up to nine in one dispatch + // (194 ms). A tour of ten areas in the lab: 110 imports of 1,102 ms on the replay thread without, 75 of 685 ms with it, + // the picture's flicker counters at 0 in both. And reads up to RX_VK_ASYNC_IMPORT_MB (16; it was 4): the storage of a + // 2560 x 1440 texture bound as a buffer is 15 MB. What is still imported on its own thread says why in its log line + static const bool async_import = [] { + if (const char* e = getenv("RX_VK_ASYNC_IMPORT")) return atoi(e) != 0; + return dev.props.vendorID == 0x10DE; + }(); + static const uint64_t async_max = + (uint64_t)(getenv("RX_VK_ASYNC_IMPORT_MB") ? std::clamp(atoi(getenv("RX_VK_ASYNC_IMPORT_MB")), 1, 64) : 16) << 20; + tl_sync_why = nullptr; + if (async_import && guest < (1ull << 40) && !(last_pool && guest >= last_pool->lo && guest + size <= last_pool->hi)) { + const char* why = written ? "the GPU writes it" : par ? "a parallel replay" : size > async_max ? "a large read" : nullptr; + if (!why && import_in_background(guest, size, &why)) { + uint64_t avail = 0; + const uint8_t* src = host->gpu_ptr(gpu, &avail); + if (src && avail >= size) + if (const BufferRef r = stream_copy(src, size)) { + st_async_copies++; + return r; + } + why = "no room in the recording's stream"; + } + tl_sync_why = why; } } Pool* p = pool_of(guest, size); + tl_sync_why = nullptr; if (!p) return {}; tl = {p, pools_gen.load(std::memory_order_relaxed)}; if (p->last_value < mem_rec_value) p->last_value = mem_rec_value; @@ -3854,6 +4129,7 @@ BufferRef gpu_buffer_bind(uint64_t gpu, uint64_t size, bool written) { } if (written) { if (tl_from_compute) p->compute_writes = true; + p->gpu_written = true; // (its mirror kept: mirrors_give_back) // (hookshot, 11-10) RX_VK_WB_MAX_KB=: bindings larger than that do not queue a write-back. OoT writes // tens of MB of culling lists and per-frame buffers through the GPU that the CPU never reads; copying them // back was most of a submit's memory sync (issue 384: 68 MB a frame, 7 ms of the GPU's time, and @@ -3990,24 +4266,45 @@ void mirror_free(Pool& p) { void mirrors_give_back(uint64_t want_bytes) { // (pools_mu held) static const bool keep = getenv("RX_VK_MIRROR_KEEP") != nullptr; // (A/B: mirrors kept, as before) if (keep) return; + // (the mirrors still given back, why, and what the GPU did to them: the first 200, to see which went before a picture fault) + static int logged_frees = 0; + const auto told = [](const Pool& p, const char* why) { + if (logged_frees++ < 200) + logf("vk: mirror of pool %#llx (%llu MB) given back: %s, %llu submissions since its sync%s%s", (unsigned long long)p.lo, + (unsigned long long)((p.hi - p.lo) >> 20), why, (unsigned long long)(mirror_epoch - p.synced_epoch), + p.gpu_only ? ", GPU-only data" : "", p.gpu_written ? ", GPU-written" : ""); + }; for (auto& nh : retired_pools) { Pool& p = nh.mapped(); // (EvilOOT: kept while the pool that took it in has not made its mirror, for 600 submissions at most) - if (p.inherit_pending && mirror_epoch - p.retired_epoch > kIdleEpochs) p.inherit_pending = false; - if (p.mstate == Pool::MIRROR_ON && p.retired_value <= mem_completed && !p.inherit_pending) mirror_free(p); + if (p.inherit_pending && mirror_epoch - p.retired_epoch > kIdleEpochs) p.inherit_pending = false, p.inherit_lost = true; + if (p.mstate == Pool::MIRROR_ON && p.retired_value <= mem_completed && !p.inherit_pending) { + told(p, p.inherit_lost ? "retired, not inherited in time" : "retired"); + mirror_free(p); + } } + // (hookshot) not a pool the GPU wrote past what is copied back (gpu_only), nor one it wrote through a binding (gpu_written): made + // again, its mirror came from guest memory without that, and the GPU went on from stale data. Paused 1-2 minutes in the background, the idle pools' mirrors all went, + // and the game came back with cyan for a few seconds (RX 5500M; gone with RX_VK_MIRROR_KEEP=1). A pool refused a mirror for + // the budget meanwhile is read in system memory, slower, not wrong std::vector idle; + uint64_t kept_bytes = 0; for (auto& [lo, p] : imported) - if (p.mstate == Pool::MIRROR_ON && mirror_epoch - p.synced_epoch > kIdleEpochs && p.last_value <= mem_completed) - idle.push_back(&p); + if (p.mstate == Pool::MIRROR_ON && mirror_epoch - p.synced_epoch > kIdleEpochs && p.last_value <= mem_completed) { + if (p.gpu_only || p.gpu_written) kept_bytes += p.hi - p.lo; + else idle.push_back(&p); + } std::sort(idle.begin(), idle.end(), [](const Pool* a, const Pool* b) { return a->synced_epoch < b->synced_epoch; }); - for (Pool* p : idle) - if (mirror_epoch - p->synced_epoch > kLongIdle || mirror_bytes + want_bytes > mirror_budget()) mirror_free(*p); + for (Pool* p : idle) { + const bool long_idle = mirror_epoch - p->synced_epoch > kLongIdle; + if (long_idle || mirror_bytes + want_bytes > mirror_budget()) told(*p, long_idle ? "idle" : "idle, for room"), mirror_free(*p); + } static int64_t logged_at; if (st_mirrors_freed / 16 != logged_at) { logged_at = st_mirrors_freed / 16; - logf("vk: %lld mirrors given back (%llu MB in all); %llu MB of mirrors now", (long long)st_mirrors_freed, - (unsigned long long)(st_mirror_bytes_freed >> 20), (unsigned long long)(mirror_bytes >> 20)); + logf("vk: %lld mirrors given back (%llu MB in all); %llu MB of mirrors now, %llu MB of them idle and kept for what the GPU " + "wrote there alone", (long long)st_mirrors_freed, (unsigned long long)(st_mirror_bytes_freed >> 20), + (unsigned long long)(mirror_bytes >> 20), (unsigned long long)(kept_bytes >> 20)); } } diff --git a/gpu/vk/vk_menu.cpp b/gpu/vk/vk_menu.cpp index 72cfcbf..48faece 100644 --- a/gpu/vk/vk_menu.cpp +++ b/gpu/vk/vk_menu.cpp @@ -44,7 +44,6 @@ #endif #include "vk_ui.h" #include "vk_menu_keys.h" // (which gamepad buttons confirm and go back: those A and B are bound to) // (how it looks: Dear ImGui, vk_imgui.h) -#include "vk_work.h" // (after the others: counts the commands that do GPU work, for full_barrier) #include "vk_dlss.h" #include "vk_fsr3.h" // (the Upscaler row: whether AMD FSR 3.1 runs on this GPU) #include "vk_fsr3fi.h" // (the Frame generation row: whether FSR 3.1 makes the frame) @@ -54,6 +53,7 @@ #include "vk_antilag.h" #include "upscaler_rows.h" #include "menu_restart.h" +#include "vk_work.h" // (after the others: counts the commands that do GPU work, for full_barrier) #ifndef HOOKSHOT_VERSION // (nx2/build.py gives it to this file: nx2/version.py) #define HOOKSHOT_VERSION "unknown" #endif @@ -3240,6 +3240,7 @@ void menu_record(VkCommandBuffer cmd, uint32_t index, uint32_t w, uint32_t h) { if (prebake_needed(w, h) && ui::begin(w, h)) { ui::Ctx c = menu_ctx(w, h); prebake(c); + ::mkwvk::rec::boundary(::mkwvk::rec::B_DIRECT); // (Dear ImGui records into cmd directly, its backend through its own pointers: what this thread recorded through vk_work.h reaches cmd first) ui::record(cmd, index, w, h); return; } @@ -3263,6 +3264,7 @@ void menu_record(VkCommandBuffer cmd, uint32_t index, uint32_t w, uint32_t h) { else if (game_tab) draw_game_tab(c); if (fps) draw_fps(c, open || game_tab); if (note) draw_notice(c); + ::mkwvk::rec::boundary(::mkwvk::rec::B_DIRECT); // (Dear ImGui records into cmd directly, its backend through its own pointers: what this thread recorded through vk_work.h reaches cmd first) ui::record(cmd, index, w, h); } @@ -3278,6 +3280,7 @@ void menu_record_image(VkCommandBuffer cmd, VkImage image, VkImageView view, uin else if (game_tab) draw_game_tab(c); if (fps) draw_fps(c, open || game_tab); if (note) draw_notice(c); + ::mkwvk::rec::boundary(::mkwvk::rec::B_DIRECT); // (Dear ImGui records into cmd directly, its backend through its own pointers: what this thread recorded through vk_work.h reaches cmd first) ui::record_to(cmd, image, view, w, h); } diff --git a/gpu/vk/vk_recorder.h b/gpu/vk/vk_recorder.h new file mode 100644 index 0000000..6ef10d5 --- /dev/null +++ b/gpu/vk/vk_recorder.h @@ -0,0 +1,1134 @@ +// (hookshot) RX_VK_RECORDER: the command buffer calls as packets, step 1 of a recorder thread (the "sync" phase). +// +// Every vkCmd* call vk_work.h routes here is copied into a packet: its value arguments, and every array and struct it points to (the +// descriptor writes and the image/buffer/texel infos they point to, push constant bytes, vertex buffer arrays, the rendering info and +// its attachments, the barriers, the copy regions, the dynamic state arrays), so a packet owns all it needs and no longer reads the +// caller's memory. A packet is run by one decoder (execute: the packet's thunk calls the real ::vkx function with the copies). +// +// RX_VK_RECORDER unset / anything else: off, the calls go straight to the driver as before (one relaxed load and a branch a call) +// RX_VK_RECORDER=sync: each call packed, then run at once on the same thread, in the same order: the same calls with the same +// values, made from the packet's copies (the proof that the copies are complete and exact) +// RX_VK_RECORDER=defer: (step 1.5, serial replay only) the packets held in the thread's arena and run at drain(), in order, on the +// same thread; the caller reuses its memory in between, as it will once a recorder thread runs them. Drained +// before every call of RX_REC_BOUNDARIES (vk_work.h: command buffer begin/end/reset, submits, presents, waits, +// query and fault reads, every destruction), when vk_replay.cpp's replay_mu is let go (DrainingMutex), before +// a fallback, at the soft byte budget, and when the arena has no memory (then the call is made directly). +// Refused (off, logged once) with RX_VK_PARALLEL>=2; while par_active, packets run at once. +// RX_VK_RECORDER_POISON=1: the arena filled with 0xCD after each drain. tests/test_recorder_sync.py +// RX_VK_RECORDER=thread: (step 2, serial replay only) the thread that holds vk_replay.cpp's replay_mu is the producer: its packets +// go into one bounded ring (Recorder, below) and a recorder thread runs them, in order, while the replay goes +// on. Before every call of RX_REC_BOUNDARIES on the producer (but those of unordered()), and when it lets +// replay_mu go, it waits until +// the recorder has run everything it put in the ring: outside a replay_mu hold the recorder is idle, and no +// command buffer, pool or queue call of the replay's is made while its packets are in flight. Threads without +// replay_mu make their calls directly (after the recorder has run what was published, should it be busy: +// counted and logged once). The ring full: the producer waits (backpressure); a packet bigger than the room +// it is given, or no ring: the producer waits until the recorder is idle, then makes the call directly. +// Refused (off, logged once) with RX_VK_PARALLEL>=2. RX_VK_RECORDER_RING_MB (default 8), +// RX_VK_RECORDER_POISON=1 (each packet filled with 0xCD once run), RX_VK_RECORDER_STATS=0 (no 10 s stats +// line), RX_VK_RECORDER_SPIN_US (the recorder's spin with nothing to run, before it sleeps: default adaptive 20-200) +// +// A packet: [Packet header: thunk, next, op, size][the call's closure: its value arguments and the arena pointers of its copies], +// preceded in the arena by the copied arrays. The arena is the recording thread's own (thread_local: the parallel replay's workers each +// have theirs) and in sync mode is rewound after each packet runs, so it stays at one packet's size. A call whose structs carry a +// pNext chain (none does today) or a descriptor type the packer does not know is not packed: what is held is drained, then the call +// is made directly (counted in Arena::fallbacks). +// +// Include through vk_work.h, not by itself: the names below are the wrappers its macros call. +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#if defined(__x86_64__) || defined(_M_X64) || defined(__i386__) || defined(_M_IX86) +#include +#endif + +#include "vk_loader.h" + +namespace mkwvk { +void logf(const char* fmt, ...); // (vk_main.cpp) +extern std::atomic par_active; // (vk_replay.cpp: a parallel replay's segments are being recorded) +} // namespace mkwvk + +// The calls the recorder carries. V: value arguments only (handles, numbers, enums), one generic packer; A: arrays or structs behind +// pointers, a packer of their own below. vk_work.h has a macro for each (the test checks the two lists agree). +#define RX_REC_CALLS(V, A) \ + A(vkCmdPipelineBarrier2) A(vkCmdBeginRendering) V(vkCmdEndRendering) \ + V(vkCmdDraw) V(vkCmdDrawIndexed) V(vkCmdDrawIndirect) V(vkCmdDrawIndexedIndirect) V(vkCmdDrawIndirectCount) \ + V(vkCmdDrawIndexedIndirectCount) V(vkCmdDrawMeshTasksEXT) V(vkCmdDrawMeshTasksIndirectEXT) V(vkCmdDrawMeshTasksIndirectCountEXT) \ + V(vkCmdDispatch) V(vkCmdDispatchIndirect) \ + A(vkCmdCopyBuffer) A(vkCmdCopyBufferToImage) A(vkCmdCopyImageToBuffer) A(vkCmdCopyImage) A(vkCmdBlitImage) A(vkCmdResolveImage) \ + A(vkCmdClearColorImage) A(vkCmdClearDepthStencilImage) A(vkCmdClearAttachments) V(vkCmdFillBuffer) A(vkCmdUpdateBuffer) \ + V(vkCmdBeginQuery) V(vkCmdEndQuery) \ + V(vkCmdBindPipeline) A(vkCmdBindDescriptorSets) V(vkCmdBindIndexBuffer) A(vkCmdBindVertexBuffers2) A(vkCmdPushConstants) \ + A(vkCmdPushDescriptorSetKHR) \ + A(vkCmdSetViewport) A(vkCmdSetScissor) A(vkCmdSetViewportWithCount) A(vkCmdSetScissorWithCount) A(vkCmdSetBlendConstants) \ + V(vkCmdSetStencilReference) V(vkCmdSetStencilCompareMask) V(vkCmdSetStencilWriteMask) V(vkCmdSetStencilOp) V(vkCmdSetDepthBias) \ + V(vkCmdSetDepthBiasEnable) V(vkCmdSetLineWidth) V(vkCmdSetDepthBoundsTestEnable) V(vkCmdSetCullMode) V(vkCmdSetFrontFace) \ + V(vkCmdSetPrimitiveTopology) V(vkCmdSetPrimitiveRestartEnable) V(vkCmdSetDepthTestEnable) V(vkCmdSetDepthWriteEnable) \ + V(vkCmdSetDepthCompareOp) V(vkCmdSetStencilTestEnable) V(vkCmdSetRasterizerDiscardEnable) V(vkCmdSetPolygonModeEXT) \ + V(vkCmdSetDepthClampEnableEXT) V(vkCmdSetAlphaToCoverageEnableEXT) \ + A(vkCmdSetColorBlendEnableEXT) A(vkCmdSetColorBlendEquationEXT) A(vkCmdSetColorWriteMaskEXT) \ + V(vkCmdWriteBufferMarkerAMD) A(vkCmdSetCheckpointNV) V(vkCmdWriteTimestamp2) V(vkCmdResetQueryPool) + +// (DEFER) The calls that drain the thread's held packets first (vk_work.h has a macro for each: the test checks the two lists agree, +// and that every file making one of these calls includes vk_work.h or is on its list of known exceptions). K: a ::vkx function, +// M: VMA's (global) +#define RX_REC_BOUNDARIES(K, M) \ + K(vkBeginCommandBuffer) K(vkEndCommandBuffer) K(vkResetCommandPool) K(vkDestroyCommandPool) \ + K(vkQueueSubmit2) K(vkQueueWaitIdle) K(vkQueueBindSparse) K(vkQueuePresentKHR) K(vkAcquireNextImageKHR) K(vkDeviceWaitIdle) \ + K(vkWaitForFences) K(vkResetFences) K(vkWaitSemaphores) K(vkGetSemaphoreCounterValue) \ + K(vkGetQueryPoolResults) K(vkResetQueryPool) K(vkGetQueueCheckpointDataNV) K(vkGetDeviceFaultInfoEXT) K(vkResetDescriptorPool) \ + K(vkDestroyBuffer) K(vkDestroyImage) K(vkDestroyImageView) K(vkDestroyBufferView) K(vkDestroySampler) K(vkDestroyPipeline) \ + K(vkDestroyPipelineLayout) K(vkDestroyDescriptorSetLayout) K(vkDestroyDescriptorPool) K(vkDestroyQueryPool) \ + K(vkDestroyShaderModule) K(vkDestroyPipelineCache) K(vkDestroyFence) K(vkDestroySemaphore) K(vkDestroySwapchainKHR) \ + K(vkDestroySurfaceKHR) K(vkDestroyDevice) K(vkDestroyInstance) K(vkFreeMemory) \ + M(vmaDestroyBuffer) M(vmaDestroyImage) M(vmaInvalidateAllocation) M(vmaFreeMemory) + +namespace mkwvk::rec { + +enum Op : uint16_t { +#define RX_REC_OP(f) OP_##f, + RX_REC_CALLS(RX_REC_OP, RX_REC_OP) +#undef RX_REC_OP + OP_COUNT +}; +inline const char* op_name(uint32_t op) { + static const char* const names[] = { +#define RX_REC_NAME(f) #f, + RX_REC_CALLS(RX_REC_NAME, RX_REC_NAME) +#undef RX_REC_NAME + }; + return op < OP_COUNT ? names[op] : "?"; +} + +/// (THREAD) where the producer waited for the recorder: each boundary call, then the recorder's own reasons +enum Bound : uint16_t { +#define RX_REC_BOUND(f) B_##f, + RX_REC_BOUNDARIES(RX_REC_BOUND, RX_REC_BOUND) +#undef RX_REC_BOUND + B_REPLAY_UNLOCK, // replay_mu let go + B_FALLBACK, // a call not packed (pNext chain, unknown descriptor type) + B_NOMEM, // a packet bigger than the ring's room for it, or no ring + B_PAR, // a parallel replay's segments being recorded (par_active) + B_RING_FULL, // backpressure: the ring had no room + B_FOREIGN, // a thread without replay_mu made a call while the recorder was busy + B_DIRECT, // code about to record into a command buffer directly, outside vk_work.h (Dear ImGui: vk_menu.cpp) + B_COUNT +}; +inline const char* bound_name(uint32_t b) { + static const char* const names[] = { +#define RX_REC_BNAME(f) #f, + RX_REC_BOUNDARIES(RX_REC_BNAME, RX_REC_BNAME) +#undef RX_REC_BNAME + "replay_mu unlock", "fallback", "no ring room", "par_active", "ring full", "off the replay lock", "direct recording"}; + static_assert(sizeof names / sizeof *names == B_COUNT); + return b < B_COUNT ? names[b] : "?"; +} + +enum Mode : int { OFF = 0, SYNC = 1, DEFER = 2, THREAD = 3, PASS = 4 /* (an encoder's own: the call made directly) */ }; +inline std::atomic g_mode{-1}; +inline std::atomic g_poison{false}; // (RX_VK_RECORDER_POISON=1: the arena filled with 0xCD after each drain) +/// (DEFER) the bytes held before the next packet drains them first (soft), and the arena's most (hard: past it a call is not +/// packed: what is held is drained, then the call made directly). Tests change them; the renderer does not +inline size_t g_soft_budget = size_t(8) << 20, g_hard_budget = size_t(64) << 20; +/// (THREAD) the ring's bytes (a power of two; RX_VK_RECORDER_RING_MB) and the room each packet is given (a vkCmdUpdateBuffer of +/// 64 KB fits; a bigger packet is made directly, after the recorder is idle). Read when the recorder starts: tests change them +inline size_t g_ring_bytes = size_t(8) << 20, g_ring_room = size_t(80) << 10; +/// (THREAD) the bytes of packets the producer writes before it publishes them (W): fewer stores to the line the recorder polls. +/// Everything is published before any wait, so a boundary still waits for every packet before it. Tests change it +inline size_t g_publish_bytes = 1024; +/// (THREAD) how long the recorder spins with nothing to run before it sleeps. Spinning keeps it awake through the replay's short gaps +/// (no wake-up call on the replay thread, no wake-up delay at its next wait); sleeping gives the core back. 0 (the default): +/// adaptive, between 20 and 200 us: back to 200 when a sleep ended within 200 us (a gap the spin would have covered: the replay paid +/// a wake-up for it), halved when a sleep lasted longer (a long gap: the spin was wasted). RX_VK_RECORDER_SPIN_US=: fixed. +/// Read when the recorder starts: tests change it +inline uint64_t g_spin_ns = 0; +constexpr uint64_t kSpinLo = 20000, kSpinHi = 200000; +inline bool g_stats_log = true; // (RX_VK_RECORDER_STATS=0: no stats line every 10 s) + +/// RX_VK_PARALLEL as vk_replay.cpp's par_threads() reads it: 2 or more, segments of one command buffer recorded on several threads +inline bool parallel_configured() { + const char* p = getenv("RX_VK_PARALLEL"); + return p && atoi(p) >= 2; +} +/// RX_VK_RECORDER: sync; defer, thread (serial only: refused, off, with RX_VK_PARALLEL); anything else, or unset, off +inline int mode_from_env(bool* refused) { + const char* e = getenv("RX_VK_RECORDER"); + *refused = false; + if (!e) return OFF; + if (!strcmp(e, "sync")) return SYNC; + if (!strcmp(e, "defer") || !strcmp(e, "thread")) { + if (!parallel_configured()) return e[0] == 'd' ? DEFER : THREAD; + *refused = true; + return OFF; + } + return OFF; +} +inline int mode_init() { + bool refused; + const int m = mode_from_env(&refused); + const char* p = getenv("RX_VK_RECORDER_POISON"); + g_poison.store((m == DEFER || m == THREAD) && p && atoi(p) != 0, std::memory_order_relaxed); + if (m == THREAD) { + if (const char* mb = getenv("RX_VK_RECORDER_RING_MB")) + if (atoi(mb) > 0) g_ring_bytes = size_t(std::min(atoi(mb), 1024)) << 20; + if (const char* s = getenv("RX_VK_RECORDER_STATS")) g_stats_log = atoi(s) != 0; + if (const char* s = getenv("RX_VK_RECORDER_SPIN_US")) g_spin_ns = (uint64_t)std::max(1, atoi(s)) * 1000; + } + int was = -1; + if (!g_mode.compare_exchange_strong(was, m, std::memory_order_relaxed)) return was; // (another thread read it first) + if (refused) // (once: the thread that set the mode) + ::mkwvk::logf("vk: RX_VK_RECORDER=%s refused: RX_VK_PARALLEL=%s records a command buffer on several threads (deferred " + "packets are the recording thread's own); the recorder is off", + getenv("RX_VK_RECORDER"), getenv("RX_VK_PARALLEL")); + else if (m == DEFER) + ::mkwvk::logf("vk: RX_VK_RECORDER=defer: vkCmd* packed and held, drained in order at each command buffer begin, end and " + "reset, submit, wait, readback, destruction and replay lock release (serial replay only)%s", + g_poison.load(std::memory_order_relaxed) ? "; the arena poisoned after each drain" : ""); + else if (m == THREAD) + ::mkwvk::logf("vk: RX_VK_RECORDER=thread: vkCmd* packed on the replay thread and run in order by a recorder thread; the " + "replay waits for it at each command buffer begin, end and reset, submit, wait, readback, destruction and " + "replay lock release (serial replay only); ring %zu KB, spin %s%llu us%s", + g_ring_bytes >> 10, g_spin_ns ? "" : "adaptive 20-", (unsigned long long)(g_spin_ns ? g_spin_ns / 1000 : 200), + g_poison.load(std::memory_order_relaxed) ? "; packets poisoned once run" : ""); + return m; +} +inline int mode() { + const int m = g_mode.load(std::memory_order_relaxed); + return m >= 0 ? m : mode_init(); +} +inline bool on() { return mode() != OFF; } + +struct alignas(16) Packet { + void (*run)(const Packet*); // the decoder for this packet's call + Packet* next; // (DEFER) the next held packet + uint32_t op; // Op, for diagnostics + uint32_t size; // bytes of the closure after the header +}; + +/// The thread's packets. Chunks in a fixed table, each from the non-throwing aligned new: alloc never throws and never loses a +/// block. It returns null when the table is full, the hard budget would be passed, or the system has no memory; the encoder then +/// drains what is held (in order) and makes the call directly +struct Arena { + static constexpr size_t kChunk = 1u << 20; + static constexpr size_t kMaxChunks = 64; + struct Chunk { + uint8_t* p; + size_t cap; + }; + struct Mark { + size_t ci, used, held; + }; + Chunk chunks[kMaxChunks] = {}; + size_t nchunks = 0, capacity = 0; // (the chunks made, and their bytes) + size_t ci = 0, used = 0; // the chunk being filled and its bytes used + size_t held = 0; // (DEFER) bytes encoded since the last drain + Packet* head = nullptr; // (DEFER) the packets not run yet, in order + Packet** tail = &head; + uint64_t packets = 0, bytes = 0, fallbacks = 0, failures = 0, budget_drains = 0; + + void* alloc(size_t n, size_t align) noexcept { + for (;;) { + if (ci < nchunks) { + const size_t at = (used + align - 1) & ~(align - 1); + if (at + n <= chunks[ci].cap) { + used = at + n; + bytes += n, held += n; + return chunks[ci].p + at; + } + if (ci + 1 < nchunks || used) { // (the next chunk, or a new one; a chunk too small even empty is passed) + ci++, used = 0; + continue; + } + } + const size_t cap = n + align > kChunk ? n + align : kChunk; + if (nchunks == kMaxChunks || cap > g_hard_budget || capacity > g_hard_budget - cap) return nullptr; + void* p = ::operator new(cap, std::align_val_t(64), std::nothrow); + if (!p) return nullptr; + chunks[nchunks++] = {static_cast(p), cap}; + capacity += cap; + ci = nchunks - 1, used = 0; + } + } + Mark mark() const { return {ci, used, held}; } + void rewind(Mark m) { ci = m.ci, used = m.used, held = m.held; } + void poison() { // (the bytes used since the last drain: the chunks before ci whole, ci up to used) + for (size_t i = 0; i < nchunks && i <= ci; i++) std::memset(chunks[i].p, 0xCD, i < ci ? chunks[i].cap : used); + } + ~Arena(); +}; +inline thread_local Arena t_arena; + +/// The decoder: runs one packet's call with its copies +inline void execute(const Packet* p) { p->run(p); } + +/// (DEFER) runs every held packet, in order, and empties the arena +inline void drain() { + Arena& a = t_arena; + const Packet* p = a.head; + a.head = nullptr, a.tail = &a.head; + while (p) { + const Packet* next = p->next; + execute(p); + p = next; + } + if (g_poison.load(std::memory_order_relaxed)) a.poison(); + a.ci = 0, a.used = 0, a.held = 0; +} +/// (a thread that ends with packets held, which the drains leave none of: they are run, never dropped) +inline Arena::~Arena() { + if (head) drain(); + for (size_t i = 0; i < nchunks; i++) ::operator delete(chunks[i].p, std::align_val_t(64)); +} + +// ---- (THREAD) the ring and the recorder thread ---- +// One ring, one recorder thread (started at the first packet, joined by stop()). The producer is whichever thread holds replay_mu +// (DrainingMutex sets t_prod): one at a time, handed on by the mutex, so the ring has one producer and one consumer. Byte counters: +// W, what the producer published; R, what the recorder has run (every packet before R has reached the driver). A packet is +// [Packet: run, next = its end, op, size = its closure's offset][copied arrays][closure], contiguous, 16-aligned; one never wraps: +// the producer gives each packet g_ring_room contiguous bytes (a wrap marker, run = null, ends the ring's tail when less is left, +// and a tail too small for a header is passed over by both sides). The memory never moves: the packets' pointers into it hold +inline void cpu_pause() { +#if defined(__x86_64__) || defined(_M_X64) || defined(__i386__) || defined(_M_IX86) + _mm_pause(); +#endif +} +inline uint64_t now_ns() { + return (uint64_t)std::chrono::duration_cast(std::chrono::steady_clock::now().time_since_epoch()).count(); +} +/// (set by vk_replay.cpp: the recorder thread's name and priority, which need windows.h) +inline void (*g_thread_start_hook)() = nullptr; +#ifdef RX_REC_TEST_HOOKS +inline void (*g_test_idle_locked)() = nullptr; // (tests: called in idle() with sleep_mu held) +#endif +/// The recorder thread cannot go on (an exception out of a driver call): logged, written to stderr and the process ended, as +/// vk_replay.cpp's replay_fatal does (exit code 3). Never swallowed: a packet not run is a command lost +[[noreturn]] inline void recorder_fatal(const char* why) { + ::mkwvk::logf("vk: FATAL: RX_VK_RECORDER=thread: the recorder thread: %s", why); + std::fprintf(stderr, "vk: FATAL: RX_VK_RECORDER=thread: the recorder thread: %s\n", why); + std::fflush(stderr); + std::_Exit(3); +} + +struct Recorder { + alignas(64) std::atomic W{0}; // bytes published (the producer) + alignas(64) std::atomic R{0}; // bytes run (the recorder) + alignas(64) std::atomic sleeping{false}; + std::atomic stop_req{false}; + std::atomic ready{false}; // ring made, thread running + std::atomic broken{false}; // no ring or no thread: the calls made directly + std::atomic logged_direct{false}, logged_foreign{false}; + uint8_t* base = nullptr; + size_t cap = 0, mask = 0, room = 0; + // (the producer's: its cursor, what of it is published (W), R as it last read it, and how much it writes before publishing) + alignas(64) uint64_t w = 0, w_pub = 0, r_seen = 0, batch = 0; + uint64_t spin_limit = 0; // (the recorder's: g_spin_ns when it started) + // (stats: written by the producer or the recorder, read by the recorder's stats line; never on a packet's path) + std::atomic wait_n[B_COUNT]{}, wait_ns[B_COUNT]{}, wait_max[B_COUNT]{}; + std::atomic spin_ns{0}, sleeps{0}, short_sleeps{0}; // (the recorder idle: time spun, sleeps, those under 200 us) + std::atomic packets{0}, busy_ns{0}, starts{0}, direct{0}; // (direct: packets that did not fit, made directly) + std::mutex life_mu, sleep_mu; + std::condition_variable sleep_cv; + std::thread th; + + bool start(); + void stop(); + void main(); + void loop(); + bool idle(uint64_t r, uint64_t& seen, uint64_t& last_log); + void log_stats(double secs); + + void wake() { + { std::lock_guard lk(sleep_mu); } + sleep_cv.notify_one(); + } + void kick() { // (the recorder asleep with work published: a wake-up the packet's cheap check may have missed) + std::atomic_thread_fence(std::memory_order_seq_cst); + if (sleeping.load(std::memory_order_relaxed)) wake(); + } + template + void spin(C done) { + kick(); + for (uint32_t i = 0; !done(); i++) { + if (i < 4096) cpu_pause(); + else { + std::this_thread::yield(); + if ((i & 255) == 0) kick(); + } + } + } + bool idle_at(uint64_t target) const { return R.load(std::memory_order_acquire) >= target; } + void note_wait(Bound k, uint64_t ns) { // (a wait's count, time and longest, for the stats line) + wait_n[k].fetch_add(1, std::memory_order_relaxed), wait_ns[k].fetch_add(ns, std::memory_order_relaxed); + uint64_t m = wait_max[k].load(std::memory_order_relaxed); + while (ns > m && !wait_max[k].compare_exchange_weak(m, ns, std::memory_order_relaxed)) {} + } + /// (the producer) what it has written, published + void flush() { + if (w == w_pub) return; + W.store(w, std::memory_order_release); + w_pub = w; + if (sleeping.load(std::memory_order_relaxed)) [[unlikely]] + wake(); + } + /// (the producer) everything it wrote has reached the driver + void wait_idle(Bound k) { + flush(); + const uint64_t target = w; + if (idle_at(target)) return; + const uint64_t t0 = now_ns(); + spin([&] { return idle_at(target); }); + note_wait(k, now_ns() - t0); + } + /// (a thread without replay_mu) what was published so far has reached the driver: its direct call comes after it. Idle in + /// the replay's design (replay_mu's unlock waits); otherwise counted, logged once + void foreign_wait() { + const uint64_t target = W.load(std::memory_order_acquire); + if (idle_at(target)) return; + if (!logged_foreign.exchange(true, std::memory_order_relaxed)) + ::mkwvk::logf("vk: RX_VK_RECORDER=thread: a command buffer or recorded call on a thread without the replay lock while the " + "recorder was busy: made after the recorder caught up (counted in the stats line; logged once)"); + const uint64_t t0 = now_ns(); + spin([&] { return idle_at(target); }); + note_wait(B_FOREIGN, now_ns() - t0); + } + /// (the producer) `need` contiguous-or-not free bytes: waits for the recorder (backpressure) when the ring has less + void wait_room(uint64_t need) { + if (cap - (w - r_seen) >= need) return; + r_seen = R.load(std::memory_order_acquire); + if (cap - (w - r_seen) >= need) return; + flush(); // (what the recorder is to make room by running) + const uint64_t t0 = now_ns(); + spin([&] { return cap - (w - (r_seen = R.load(std::memory_order_acquire))) >= need; }); + note_wait(B_RING_FULL, now_ns() - t0); + } + /// (the producer, an encoder starting) the next packet's place, with `room` contiguous bytes: its header, and the end of what it + /// may use. Never more than `room`, even when the ring has more free: whether a packet is made directly depends on its size + /// alone, not on how far the recorder had got (tests/test_recorder_sync.py: the oversize runs) + Packet* reserve(uint8_t*& lim) { + size_t off = (size_t)(w & mask), tail = cap - off; + if (tail < room) { // (the ring's tail: passed over, the packet at its start) + wait_room(tail + room); + if (tail >= sizeof(Packet)) { + Packet* mk = reinterpret_cast(base + off); + mk->run = nullptr, mk->next = nullptr, mk->op = 0, mk->size = 0; + } + w += tail; // (published with the next packet) + off = 0, tail = cap; + } else { + wait_room(room); + } + lim = base + off + room; // (free and contiguous: tail >= room, and wait_room() made room bytes free past w) + return reinterpret_cast(base + off); + } + void publish(const Packet* p, const uint8_t* end) { + w += (uint64_t)(end - reinterpret_cast(p)); + // (the recorder asleep: this packet now, and the wake-up with it, so that it runs while the replay goes on, not at its + // next wait) + if (w - w_pub >= batch || sleeping.load(std::memory_order_relaxed)) flush(); + } + // (no destructor: the recorder lives as long as the process, below) +}; +/// The recorder, for the process's lifetime: made once, never destroyed, as vk_jobs' detached workers. Windows ends every other +/// thread before a DLL's global destructors run (ExitProcess: the window's close, svcExitProcess, the updater, main's return, +/// hos_fatal), possibly while the recorder holds sleep_mu in idle(); a destructor that woke (sleep_mu) and joined it would then wait +/// for a mutex a dead thread owns. No global destructor touches its mutexes or thread: at exit nothing needs it (what is in the +/// ring is commands never to be submitted; it writes nothing else). stop() is the explicit teardown (vkDestroyDevice's boundary). +/// tests/test_recorder_sync.py: recorder_exit_test.cpp +inline Recorder& g_rec = *new Recorder; +/// (THREAD) the ring, while this thread holds replay_mu; null otherwise +inline constinit thread_local Recorder* t_prod = nullptr; + +inline bool Recorder::start() { // (the producer, its first packet) + std::lock_guard lk(life_mu); + if (ready.load(std::memory_order_relaxed)) return true; + if (broken.load(std::memory_order_relaxed)) return false; + if (!base) { + size_t c = 4096; + while (c < g_ring_bytes && c < (size_t(1) << 30)) c <<= 1; + base = static_cast(::operator new(c, std::align_val_t(64), std::nothrow)); + if (!base) { + broken.store(true); + ::mkwvk::logf("vk: RX_VK_RECORDER=thread: no memory for the recorder's ring (%zu KB): every call made directly", c >> 10); + return false; + } + cap = c, mask = c - 1; + } + room = std::max(std::min(g_ring_room, cap / 4) & ~size_t(15), 2 * sizeof(Packet)); + // (the counters go on from where a stopped recorder left them, never back to 0: a foreign_wait() that read W just before + // a restart still sees R reach it. Anything a stopped recorder left unrun is dropped, as a fresh ring did; offsets are & mask) + const uint64_t at = W.load(std::memory_order_acquire); + R.store(at, std::memory_order_release), w = w_pub = r_seen = at, stop_req.store(false); + batch = std::min(g_publish_bytes, room); + spin_limit = 0; // (set by the recorder: g_spin_ns, or adaptive) + try { + th = std::thread([this] { main(); }); + } catch (...) { + broken.store(true); + ::mkwvk::logf("vk: RX_VK_RECORDER=thread: the recorder thread could not be started: every call made directly"); + return false; + } + starts.fetch_add(1); + ready.store(true); + return true; +} +/// Teardown: everything published run, the thread joined, the ring let go (made again at the next packet). Any thread, with no +/// producer encoding at the time (the renderer's teardown, or the producer itself: vkDestroyDevice's boundary) +inline void Recorder::stop() { + std::lock_guard lk(life_mu); + if (!th.joinable()) return; + if (t_prod == this) flush(); // (the producer's own stop: what it wrote, published first) + const uint64_t target = W.load(std::memory_order_acquire); + spin([&] { return idle_at(target); }); + stop_req.store(true); + wake(); + th.join(); + ready.store(false); + if (g_stats_log) log_stats(0); + ::operator delete(base, std::align_val_t(64)); + base = nullptr, cap = 0; +} +inline void Recorder::main() { + if (g_thread_start_hook) g_thread_start_hook(); + try { + loop(); + } catch (const std::exception& e) { + recorder_fatal(e.what()); + } catch (...) { + recorder_fatal("an exception out of a driver call"); + } +} +/// (the recorder, nothing published to run) spins a while, then sleeps until woken (or 2 ms); false: stop asked and nothing left +inline bool Recorder::idle(uint64_t r, uint64_t& seen, uint64_t& last_log) { + const uint64_t t0 = now_ns(); + if (!spin_limit) spin_limit = g_spin_ns ? g_spin_ns : kSpinHi; // (fixed, or adaptive from the top) + for (uint32_t i = 0;; i++) { + if ((seen = W.load(std::memory_order_acquire)) != r || stop_req.load(std::memory_order_acquire)) { + spin_ns.fetch_add(now_ns() - t0, std::memory_order_relaxed); + return seen != r || W.load(std::memory_order_acquire) != r; + } + cpu_pause(); + if ((i & 15) == 15 && now_ns() - t0 > spin_limit) break; // (spin_limit: a replay's gaps between calls, spun through) + } + spin_ns.fetch_add(now_ns() - t0, std::memory_order_relaxed); + sleeps.fetch_add(1, std::memory_order_relaxed); + if (g_stats_log && t0 - last_log > 10000000000ull) { + if (last_log) log_stats((double)(t0 - last_log) / 1e9); + last_log = t0; + } + const uint64_t s0 = now_ns(); + { + std::unique_lock lk(sleep_mu); +#ifdef RX_REC_TEST_HOOKS + if (g_test_idle_locked) g_test_idle_locked(); +#endif + sleeping.store(true, std::memory_order_seq_cst); + if (W.load(std::memory_order_seq_cst) == r && !stop_req.load()) sleep_cv.wait_for(lk, std::chrono::milliseconds(2)); + sleeping.store(false, std::memory_order_relaxed); + } + seen = W.load(std::memory_order_acquire); + const uint64_t slept = now_ns() - s0; + if (slept < kSpinHi) short_sleeps.fetch_add(1, std::memory_order_relaxed); + if (!g_spin_ns) { // (adaptive: a short sleep means a gap the longest spin covers; a long one, a spin to shorten) + if (slept < kSpinHi && seen != r) spin_limit = kSpinHi; + else if (slept >= kSpinHi) spin_limit = std::max(kSpinLo, spin_limit / 2); + } + return true; +} +inline void Recorder::loop() { + uint64_t r = R.load(std::memory_order_relaxed), seen = r, last_log = now_ns(), n = 0, busy0 = 0; + const bool poison = g_poison.load(std::memory_order_relaxed); + for (;;) { + if (r == seen && (seen = W.load(std::memory_order_acquire)) == r) { + if (busy0) busy_ns.fetch_add(now_ns() - busy0, std::memory_order_relaxed), busy0 = 0; + if (n) packets.fetch_add(n, std::memory_order_relaxed), n = 0; + if (!idle(r, seen, last_log)) return; + continue; + } + if (!busy0) busy0 = now_ns(); + const size_t off = (size_t)(r & mask), tail = cap - off; + const Packet* p = reinterpret_cast(base + off); + if (tail < sizeof(Packet) || !p->run) { // (the ring's tail: the next packet is at its start) + r += tail; + R.store(r, std::memory_order_release); + continue; + } + p->run(p); + const size_t len = (size_t)(reinterpret_cast(p->next) - reinterpret_cast(p)); + if (poison) std::memset(base + off, 0xCD, len); + r += len, n++; + R.store(r, std::memory_order_release); + } +} +inline void Recorder::log_stats(double secs) { + uint64_t n[B_COUNT], ns[B_COUNT], mx[B_COUNT], tot_n = 0, tot_ns = 0, max_ns = 0; + for (int k = 0; k < B_COUNT; k++) + n[k] = wait_n[k].exchange(0, std::memory_order_relaxed), ns[k] = wait_ns[k].exchange(0, std::memory_order_relaxed), + mx[k] = wait_max[k].exchange(0, std::memory_order_relaxed), tot_n += n[k], tot_ns += ns[k], max_ns = std::max(max_ns, mx[k]); + const uint64_t spun = spin_ns.exchange(0, std::memory_order_relaxed), slept = sleeps.exchange(0, std::memory_order_relaxed); + const uint64_t short_slept = short_sleeps.exchange(0, std::memory_order_relaxed); + const uint64_t pk = packets.exchange(0, std::memory_order_relaxed), busy = busy_ns.exchange(0, std::memory_order_relaxed); + const uint64_t dr = direct.exchange(0, std::memory_order_relaxed); + if (!pk && !tot_n && !dr) return; + char top[512]; + size_t at = 0; + top[0] = 0; + for (int shown = 0; shown < 6; shown++) { // (the six it waited longest at) + int best = -1; + for (int k = 0; k < B_COUNT; k++) + if (n[k] && (best < 0 || ns[k] > ns[best])) best = k; + if (best < 0) break; + const int c = std::snprintf(top + at, sizeof top - at, "%s %s %llu/%.2f ms (max %.2f)", shown ? "," : ";", bound_name(best), + (unsigned long long)n[best], ns[best] / 1e6, mx[best] / 1e6); + if (c < 0 || (size_t)c >= sizeof top - at) break; + at += (size_t)c; + n[best] = 0; + } + ::mkwvk::logf("vk: recorder: %llu packets run in %.1f s (busy %.2f s, spun idle %.2f s, slept %llu times, %llu of them under 200 " + "us), %llu calls made directly (no room); the replay waited for it %llu times, %.2f ms in all, the longest %.2f ms%s", + (unsigned long long)pk, secs, busy / 1e9, spun / 1e9, (unsigned long long)slept, (unsigned long long)short_slept, + (unsigned long long)dr, (unsigned long long)tot_n, tot_ns / 1e6, max_ns / 1e6, top); +} + +/// (THREAD) The boundary calls the producer makes without waiting for the recorder: only a call that touches no command buffer or +/// pool and can read or change nothing a command still in the ring could do. The ring holds vkCmd* recordings only, and every +/// submission (vkQueueSubmit2, vkQueueBindSparse) waits for it: what has been submitted when such a call runs is what it would be +/// with the recorder off. Each case's reason is beside it; +/// tests/test_recorder_sync.py checks each listed call runs with packets in flight and every other one waits, and that adding +/// vkQueueSubmit2, vkDestroyBuffer or vkEndCommandBuffer here fails it +constexpr bool unordered(Bound k) { + switch (k) { + case B_vkGetSemaphoreCounterValue: return true; // (a timeline's value: signalled only by submissions, each made after the ring is empty) + case B_vkWaitSemaphores: return true; // (a timeline's value: signalled only by submissions, each made after the ring is empty, or by + // the driver itself: vk_reflex.cpp's latency-sleep semaphore, which no recorded command signals) + case B_vkGetQueueCheckpointDataNV: return true; // (what the device ran of submitted work, after a loss) + case B_vkGetDeviceFaultInfoEXT: return true; // (what the device ran of submitted work, after a loss) + default: return false; // (unordered: the end of the list) + } +} + +/// Before a call that begins, ends or resets command buffers, submits, presents or waits, reads back what the GPU did, or lets go of +/// what recorded commands use (vk_work.h's second list), and when the replay's lock is let go: the thread's held packets reach the +/// driver first. DEFER: drained; THREAD: on the producer, waits until the recorder has run all it published (vkDestroyDevice and +/// vkDestroyInstance: the recorder stopped and joined first); otherwise a relaxed load and a branch +inline void boundary(Bound k) { + const int m = g_mode.load(std::memory_order_relaxed); + if (m == DEFER) { + drain(); + } else if (m == THREAD) { + if (k == B_vkDestroyDevice || k == B_vkDestroyInstance) g_rec.stop(); + else if (unordered(k)) return; + else if (Recorder* g = t_prod) g->wait_idle(k); + } +} + +/// A std::mutex whose unlock drains the thread's held packets first (vk_replay.cpp's replay_mu): what a thread recorded into the +/// replay's command buffers reaches the driver before another thread may record into them. THREAD: the holder is the ring's +/// producer; the unlock waits for the recorder to run all it published, so the recorder is idle whenever no thread holds it +struct DrainingMutex { + std::mutex m; + void lock() { + m.lock(); + enter(); + } + bool try_lock() { + if (!m.try_lock()) return false; + enter(); + return true; + } + void unlock() { + boundary(B_REPLAY_UNLOCK); + t_prod = nullptr; + m.unlock(); + } + static void enter() { + if (mode() == THREAD) t_prod = &g_rec; + } +}; + +/// (for tests) a mode, or back; what is held is run first (THREAD: what was published) +inline void set_mode(int m) { + drain(); + if (Recorder* g = t_prod) g->flush(); + if (g_rec.ready.load()) { + const uint64_t target = g_rec.W.load(std::memory_order_acquire); + g_rec.spin([&] { return g_rec.idle_at(target); }); + } + g_mode.store(m, std::memory_order_relaxed); +} + +/// THREAD: the recorder has run what this thread may depend on (the producer: all it published; another thread: what was published) +inline void settle(Bound k) { + if (Recorder* g = t_prod) g->wait_idle(k); + else g_rec.foreign_wait(); +} + +/// A call that is not packed (a pNext chain, an unknown descriptor type): what is held goes first, then the call itself +template +inline void fallback(const F& direct) { + if (g_mode.load(std::memory_order_relaxed) == THREAD) settle(B_FALLBACK); + else drain(); + t_arena.fallbacks++; + direct(); +} + +template +void thunk(const Packet* p) { + (*reinterpret_cast(p + 1))(); +} +template +void thunk_at(const Packet* p) { // (THREAD: the closure at its offset in the packet, after the copied arrays) + (*reinterpret_cast(reinterpret_cast(p) + p->size))(); +} + +/// One packet being encoded: its copies, then emit() with the closure that makes the call and the direct call. Sync: run at emit, the +/// arena rewound when the encoder goes out of scope. Defer: held; drained first when the held bytes reached the soft budget, and +/// made at once (after a drain) while a parallel replay records (par_active: never held then). An allocation that fails (the hard +/// budget, the table, the system) makes emit drain what is held, in order, then make the direct call: no call lost or reordered +/// +/// THREAD, on the producer (the replay lock's holder): the packet is written straight into the ring (header first, then the copies, +/// then the closure) and published at emit; no room for it (bigger than the room a packet is given) or no ring: the producer waits +/// until the recorder has run everything published, then makes the call directly. On another thread, or while par_active: the +/// call made directly (PASS), after what was published has run +inline uint8_t* align_up(uint8_t* p, size_t a) { return reinterpret_cast(((uintptr_t)p + a - 1) & ~(uintptr_t)(a - 1)); } +struct Enc { + int md = mode(); + Arena* a = nullptr; // (SYNC, DEFER) + Arena::Mark m{}; + bool failed = false; + Recorder* g = nullptr; // (THREAD) the ring, its packet's header, where the copies go and how far they may + Packet* hdr = nullptr; + uint8_t *cur = nullptr, *lim = nullptr; + + Enc() { + if (md == THREAD) { + begin_thread(); + return; + } + a = &t_arena; + if (md == DEFER) { + if (par_active.load(std::memory_order_relaxed)) { + drain(); + md = SYNC; + } else if (a->held >= g_soft_budget) { + drain(); + a->budget_drains++; + } + } + m = a->mark(); + } + void begin_thread() { + g = t_prod; + if (!g || par_active.load(std::memory_order_relaxed)) [[unlikely]] { + settle(B_PAR); + md = PASS, failed = true; + return; + } + if (!g->ready.load(std::memory_order_relaxed) && !g->start()) [[unlikely]] { + md = PASS, failed = true; + return; + } + hdr = g->reserve(lim); + cur = reinterpret_cast(hdr + 1); + } + template + T* alloc(size_t n) { + if (failed) return nullptr; + if (md == THREAD) { + uint8_t* at = align_up(cur, alignof(T) < 8 ? 8 : alignof(T)); + if (at > lim || (size_t)(lim - at) < sizeof(T) * n) { + failed = true; + return nullptr; + } + cur = at + sizeof(T) * n; + return reinterpret_cast(at); + } + T* p = static_cast(a->alloc(sizeof(T) * n, alignof(T) < 8 ? 8 : alignof(T))); + if (!p) failed = true; + return p; + } + template + const T* dup(const T* p, size_t n) { + if (!p || !n) return nullptr; + T* d = alloc(n); + if (!d) return nullptr; + std::memcpy(static_cast(d), p, sizeof(T) * n); + return d; + } + const void* dup_bytes(const void* p, size_t n) { return dup(static_cast(p), n); } + + template + void emit(Op op, const F& f, const D& direct) { + static_assert(std::is_trivially_copyable_v && std::is_trivially_destructible_v, "a packet holds values only"); + static_assert(alignof(F) <= alignof(Packet), "a closure aligned beyond the header"); + if (md == THREAD || md == PASS) { + emit_thread(op, f, direct); + return; + } + Packet* p = failed ? nullptr : static_cast(a->alloc(sizeof(Packet) + sizeof(F), alignof(Packet))); + if (!p) { + const size_t held = a->held; + if (md == DEFER) drain(); // (what is held first, in order; this call's partial copies let go with it) + else a->rewind(m); + if (a->failures++ == 0) + ::mkwvk::logf("vk: RX_VK_RECORDER: no arena memory for %s (%zu bytes held, %zu in chunks, hard budget %zu): the " + "held packets drained, the call made directly (logged once a thread)", + op_name(op), held, a->capacity, g_hard_budget); + direct(); + return; + } + p->run = &thunk, p->next = nullptr, p->op = op, p->size = (uint32_t)sizeof(F); + new (static_cast(p + 1)) F(f); + a->packets++; + if (md == SYNC) { + execute(p); + } else { + *a->tail = p; + a->tail = &p->next; + } + } + template + void emit_thread(Op op, const F& f, const D& direct) { + uint8_t* at = failed ? nullptr : align_up(cur, alignof(Packet)); + if (md == PASS) { + direct(); + return; + } + if (!at || at > lim || (size_t)(lim - at) < sizeof(F)) [[unlikely]] { // (nothing published: the recorder idle, then direct) + g->wait_idle(B_NOMEM); + g->direct.fetch_add(1, std::memory_order_relaxed); + if (!g->logged_direct.exchange(true, std::memory_order_relaxed)) + ::mkwvk::logf("vk: RX_VK_RECORDER=thread: %s did not fit in the room a packet is given (%zu bytes): made directly once " + "the recorder had run what was before it (logged once)", + op_name(op), g->room); + direct(); + return; + } + new (static_cast(at)) F(f); + hdr->run = &thunk_at, hdr->op = op, hdr->size = (uint32_t)(at - reinterpret_cast(hdr)); + uint8_t* end = align_up(at + sizeof(F), alignof(Packet)); + hdr->next = reinterpret_cast(end); + g->publish(hdr, end); + } + ~Enc() { + if (md == SYNC) a->rewind(m); + } +}; + +// ---- The wrapper each vk_work.h macro calls: the driver's function when the recorder is off (the branch is all the off path costs: the +// packer is a function of its own, out of line, so the caller's code stays as it was), else the call's packer +#if defined(__GNUC__) || defined(__clang__) +#define RX_REC_PACKER __attribute__((noinline)) inline +#else +#define RX_REC_PACKER __declspec(noinline) inline +#endif +template >> +struct Route; +template +struct Route { + static void call(P... p) { + if (mode() == OFF) [[likely]] + return F(p...); + PK(p...); + } +}; + +// ---- V: value arguments only. The closure holds them as the call's own parameter types (converted at the wrapper, as the direct call +// converts them). A pointer to const data here would be a pointer kept past the call: refused at compile time (handles are pointers to +// non-const opaque types and pass) +template >> +struct Value; +template +struct Value { + static_assert(((!std::is_pointer_v

|| !std::is_const_v>) && ...), + "a call with a pointer to data needs a packer of its own"); + RX_REC_PACKER static void pack(P... p) { + Enc e; + const auto call = [=] { F(p...); }; + e.emit(OP, call, call); + } +}; +#define RX_REC_VALUE(f) inline constexpr auto& rx_##f = Route<::vkx::f, Value<::vkx::f, OP_##f>::pack>::call; +#define RX_REC_SKIP(f) +RX_REC_CALLS(RX_REC_VALUE, RX_REC_SKIP) +#undef RX_REC_VALUE +#undef RX_REC_SKIP + +// ---- A: the calls with arrays and structs behind pointers. Each copies what the driver reads, by the counts the driver reads by +template +inline bool chained(const T* p, uint32_t n) { + for (uint32_t i = 0; i < n; i++) + if (p[i].pNext) return true; + return false; +} + +RX_REC_PACKER void pk_vkCmdPipelineBarrier2(VkCommandBuffer cb, const VkDependencyInfo* di) { + if (di->pNext || chained(di->pMemoryBarriers, di->memoryBarrierCount) || + chained(di->pBufferMemoryBarriers, di->bufferMemoryBarrierCount) || + chained(di->pImageMemoryBarriers, di->imageMemoryBarrierCount)) + return fallback([&] { ::vkx::vkCmdPipelineBarrier2(cb, di); }); + Enc e; + VkDependencyInfo c = *di; + c.pMemoryBarriers = e.dup(di->pMemoryBarriers, di->memoryBarrierCount); + c.pBufferMemoryBarriers = e.dup(di->pBufferMemoryBarriers, di->bufferMemoryBarrierCount); + c.pImageMemoryBarriers = e.dup(di->pImageMemoryBarriers, di->imageMemoryBarrierCount); + e.emit(OP_vkCmdPipelineBarrier2, [=] { ::vkx::vkCmdPipelineBarrier2(cb, &c); }, [&] { ::vkx::vkCmdPipelineBarrier2(cb, di); }); +} +inline constexpr auto& rx_vkCmdPipelineBarrier2 = Route<::vkx::vkCmdPipelineBarrier2, pk_vkCmdPipelineBarrier2>::call; + +RX_REC_PACKER void pk_vkCmdBeginRendering(VkCommandBuffer cb, const VkRenderingInfo* ri) { + if (ri->pNext || chained(ri->pColorAttachments, ri->colorAttachmentCount) || (ri->pDepthAttachment && ri->pDepthAttachment->pNext) || + (ri->pStencilAttachment && ri->pStencilAttachment->pNext)) + return fallback([&] { ::vkx::vkCmdBeginRendering(cb, ri); }); + Enc e; + VkRenderingInfo c = *ri; + c.pColorAttachments = e.dup(ri->pColorAttachments, ri->colorAttachmentCount); + c.pDepthAttachment = e.dup(ri->pDepthAttachment, 1); + c.pStencilAttachment = e.dup(ri->pStencilAttachment, 1); + e.emit(OP_vkCmdBeginRendering, [=] { ::vkx::vkCmdBeginRendering(cb, &c); }, [&] { ::vkx::vkCmdBeginRendering(cb, ri); }); +} +inline constexpr auto& rx_vkCmdBeginRendering = Route<::vkx::vkCmdBeginRendering, pk_vkCmdBeginRendering>::call; + +RX_REC_PACKER void pk_vkCmdCopyBuffer(VkCommandBuffer cb, VkBuffer src, VkBuffer dst, uint32_t n, const VkBufferCopy* r) { + Enc e; + const VkBufferCopy* c = e.dup(r, n); + e.emit(OP_vkCmdCopyBuffer, [=] { ::vkx::vkCmdCopyBuffer(cb, src, dst, n, c); }, [&] { ::vkx::vkCmdCopyBuffer(cb, src, dst, n, r); }); +} +inline constexpr auto& rx_vkCmdCopyBuffer = Route<::vkx::vkCmdCopyBuffer, pk_vkCmdCopyBuffer>::call; + +RX_REC_PACKER void pk_vkCmdCopyBufferToImage(VkCommandBuffer cb, VkBuffer src, VkImage dst, VkImageLayout layout, uint32_t n, + const VkBufferImageCopy* r) { + Enc e; + const VkBufferImageCopy* c = e.dup(r, n); + e.emit(OP_vkCmdCopyBufferToImage, [=] { ::vkx::vkCmdCopyBufferToImage(cb, src, dst, layout, n, c); }, + [&] { ::vkx::vkCmdCopyBufferToImage(cb, src, dst, layout, n, r); }); +} +inline constexpr auto& rx_vkCmdCopyBufferToImage = Route<::vkx::vkCmdCopyBufferToImage, pk_vkCmdCopyBufferToImage>::call; + +RX_REC_PACKER void pk_vkCmdCopyImageToBuffer(VkCommandBuffer cb, VkImage src, VkImageLayout layout, VkBuffer dst, uint32_t n, + const VkBufferImageCopy* r) { + Enc e; + const VkBufferImageCopy* c = e.dup(r, n); + e.emit(OP_vkCmdCopyImageToBuffer, [=] { ::vkx::vkCmdCopyImageToBuffer(cb, src, layout, dst, n, c); }, + [&] { ::vkx::vkCmdCopyImageToBuffer(cb, src, layout, dst, n, r); }); +} +inline constexpr auto& rx_vkCmdCopyImageToBuffer = Route<::vkx::vkCmdCopyImageToBuffer, pk_vkCmdCopyImageToBuffer>::call; + +RX_REC_PACKER void pk_vkCmdCopyImage(VkCommandBuffer cb, VkImage src, VkImageLayout sl, VkImage dst, VkImageLayout dl, uint32_t n, + const VkImageCopy* r) { + Enc e; + const VkImageCopy* c = e.dup(r, n); + e.emit(OP_vkCmdCopyImage, [=] { ::vkx::vkCmdCopyImage(cb, src, sl, dst, dl, n, c); }, + [&] { ::vkx::vkCmdCopyImage(cb, src, sl, dst, dl, n, r); }); +} +inline constexpr auto& rx_vkCmdCopyImage = Route<::vkx::vkCmdCopyImage, pk_vkCmdCopyImage>::call; + +RX_REC_PACKER void pk_vkCmdBlitImage(VkCommandBuffer cb, VkImage src, VkImageLayout sl, VkImage dst, VkImageLayout dl, uint32_t n, + const VkImageBlit* r, VkFilter filter) { + Enc e; + const VkImageBlit* c = e.dup(r, n); + e.emit(OP_vkCmdBlitImage, [=] { ::vkx::vkCmdBlitImage(cb, src, sl, dst, dl, n, c, filter); }, + [&] { ::vkx::vkCmdBlitImage(cb, src, sl, dst, dl, n, r, filter); }); +} +inline constexpr auto& rx_vkCmdBlitImage = Route<::vkx::vkCmdBlitImage, pk_vkCmdBlitImage>::call; + +RX_REC_PACKER void pk_vkCmdResolveImage(VkCommandBuffer cb, VkImage src, VkImageLayout sl, VkImage dst, VkImageLayout dl, uint32_t n, + const VkImageResolve* r) { + Enc e; + const VkImageResolve* c = e.dup(r, n); + e.emit(OP_vkCmdResolveImage, [=] { ::vkx::vkCmdResolveImage(cb, src, sl, dst, dl, n, c); }, + [&] { ::vkx::vkCmdResolveImage(cb, src, sl, dst, dl, n, r); }); +} +inline constexpr auto& rx_vkCmdResolveImage = Route<::vkx::vkCmdResolveImage, pk_vkCmdResolveImage>::call; + +RX_REC_PACKER void pk_vkCmdClearColorImage(VkCommandBuffer cb, VkImage image, VkImageLayout layout, const VkClearColorValue* color, uint32_t n, + const VkImageSubresourceRange* ranges) { + Enc e; + const VkClearColorValue* col = e.dup(color, 1); + const VkImageSubresourceRange* c = e.dup(ranges, n); + e.emit(OP_vkCmdClearColorImage, [=] { ::vkx::vkCmdClearColorImage(cb, image, layout, col, n, c); }, + [&] { ::vkx::vkCmdClearColorImage(cb, image, layout, color, n, ranges); }); +} +inline constexpr auto& rx_vkCmdClearColorImage = Route<::vkx::vkCmdClearColorImage, pk_vkCmdClearColorImage>::call; + +RX_REC_PACKER void pk_vkCmdClearDepthStencilImage(VkCommandBuffer cb, VkImage image, VkImageLayout layout, const VkClearDepthStencilValue* ds, + uint32_t n, const VkImageSubresourceRange* ranges) { + Enc e; + const VkClearDepthStencilValue* v = e.dup(ds, 1); + const VkImageSubresourceRange* c = e.dup(ranges, n); + e.emit(OP_vkCmdClearDepthStencilImage, [=] { ::vkx::vkCmdClearDepthStencilImage(cb, image, layout, v, n, c); }, + [&] { ::vkx::vkCmdClearDepthStencilImage(cb, image, layout, ds, n, ranges); }); +} +inline constexpr auto& rx_vkCmdClearDepthStencilImage = Route<::vkx::vkCmdClearDepthStencilImage, pk_vkCmdClearDepthStencilImage>::call; + +RX_REC_PACKER void pk_vkCmdClearAttachments(VkCommandBuffer cb, uint32_t na, const VkClearAttachment* att, uint32_t nr, const VkClearRect* rects) { + Enc e; + const VkClearAttachment* a = e.dup(att, na); + const VkClearRect* r = e.dup(rects, nr); + e.emit(OP_vkCmdClearAttachments, [=] { ::vkx::vkCmdClearAttachments(cb, na, a, nr, r); }, + [&] { ::vkx::vkCmdClearAttachments(cb, na, att, nr, rects); }); +} +inline constexpr auto& rx_vkCmdClearAttachments = Route<::vkx::vkCmdClearAttachments, pk_vkCmdClearAttachments>::call; + +RX_REC_PACKER void pk_vkCmdUpdateBuffer(VkCommandBuffer cb, VkBuffer dst, VkDeviceSize offset, VkDeviceSize size, const void* data) { + Enc e; + const void* d = e.dup_bytes(data, (size_t)size); + e.emit(OP_vkCmdUpdateBuffer, [=] { ::vkx::vkCmdUpdateBuffer(cb, dst, offset, size, d); }, + [&] { ::vkx::vkCmdUpdateBuffer(cb, dst, offset, size, data); }); +} +inline constexpr auto& rx_vkCmdUpdateBuffer = Route<::vkx::vkCmdUpdateBuffer, pk_vkCmdUpdateBuffer>::call; + +RX_REC_PACKER void pk_vkCmdBindDescriptorSets(VkCommandBuffer cb, VkPipelineBindPoint bp, VkPipelineLayout layout, uint32_t first, uint32_t n, + const VkDescriptorSet* sets, uint32_t nd, const uint32_t* dyn) { + Enc e; + const VkDescriptorSet* s = e.dup(sets, n); + const uint32_t* d = e.dup(dyn, nd); + e.emit(OP_vkCmdBindDescriptorSets, [=] { ::vkx::vkCmdBindDescriptorSets(cb, bp, layout, first, n, s, nd, d); }, + [&] { ::vkx::vkCmdBindDescriptorSets(cb, bp, layout, first, n, sets, nd, dyn); }); +} +inline constexpr auto& rx_vkCmdBindDescriptorSets = Route<::vkx::vkCmdBindDescriptorSets, pk_vkCmdBindDescriptorSets>::call; + +RX_REC_PACKER void pk_vkCmdBindVertexBuffers2(VkCommandBuffer cb, uint32_t first, uint32_t n, const VkBuffer* buffers, const VkDeviceSize* offsets, + const VkDeviceSize* sizes, const VkDeviceSize* strides) { + Enc e; + const VkBuffer* b = e.dup(buffers, n); + const VkDeviceSize* o = e.dup(offsets, n); + const VkDeviceSize* s = e.dup(sizes, n); // (null stays null: the whole buffer) + const VkDeviceSize* st = e.dup(strides, n); // (null stays null: the pipeline's strides) + e.emit(OP_vkCmdBindVertexBuffers2, [=] { ::vkx::vkCmdBindVertexBuffers2(cb, first, n, b, o, s, st); }, + [&] { ::vkx::vkCmdBindVertexBuffers2(cb, first, n, buffers, offsets, sizes, strides); }); +} +inline constexpr auto& rx_vkCmdBindVertexBuffers2 = Route<::vkx::vkCmdBindVertexBuffers2, pk_vkCmdBindVertexBuffers2>::call; + +RX_REC_PACKER void pk_vkCmdPushConstants(VkCommandBuffer cb, VkPipelineLayout layout, VkShaderStageFlags stages, uint32_t offset, uint32_t size, + const void* values) { + Enc e; + const void* v = e.dup_bytes(values, size); + e.emit(OP_vkCmdPushConstants, [=] { ::vkx::vkCmdPushConstants(cb, layout, stages, offset, size, v); }, + [&] { ::vkx::vkCmdPushConstants(cb, layout, stages, offset, size, values); }); +} +inline constexpr auto& rx_vkCmdPushConstants = Route<::vkx::vkCmdPushConstants, pk_vkCmdPushConstants>::call; + +/// Which of a write's three arrays the driver reads, by its type (the other two are ignored, and left null in the copy); 0: a type +/// the packer does not know (inline uniform blocks, acceleration structures: their data is in pNext) +inline int write_kind(VkDescriptorType t) { + switch (t) { + case VK_DESCRIPTOR_TYPE_SAMPLER: + case VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER: + case VK_DESCRIPTOR_TYPE_SAMPLED_IMAGE: + case VK_DESCRIPTOR_TYPE_STORAGE_IMAGE: + case VK_DESCRIPTOR_TYPE_INPUT_ATTACHMENT: return 1; + case VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER: + case VK_DESCRIPTOR_TYPE_STORAGE_BUFFER: + case VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER_DYNAMIC: + case VK_DESCRIPTOR_TYPE_STORAGE_BUFFER_DYNAMIC: return 2; + case VK_DESCRIPTOR_TYPE_UNIFORM_TEXEL_BUFFER: + case VK_DESCRIPTOR_TYPE_STORAGE_TEXEL_BUFFER: return 3; + default: return 0; + } +} + +RX_REC_PACKER void pk_vkCmdPushDescriptorSetKHR(VkCommandBuffer cb, VkPipelineBindPoint bp, VkPipelineLayout layout, uint32_t set, uint32_t n, + const VkWriteDescriptorSet* w) { + for (uint32_t i = 0; i < n; i++) + if (w[i].pNext || !write_kind(w[i].descriptorType)) + return fallback([&] { ::vkx::vkCmdPushDescriptorSetKHR(cb, bp, layout, set, n, w); }); + Enc e; + VkWriteDescriptorSet* c = n ? e.alloc(n) : nullptr; + for (uint32_t i = 0; c && i < n; i++) { // (no memory: emit makes the direct call) + c[i] = w[i]; + c[i].pImageInfo = nullptr, c[i].pBufferInfo = nullptr, c[i].pTexelBufferView = nullptr; + switch (write_kind(w[i].descriptorType)) { + case 1: c[i].pImageInfo = e.dup(w[i].pImageInfo, w[i].descriptorCount); break; + case 2: c[i].pBufferInfo = e.dup(w[i].pBufferInfo, w[i].descriptorCount); break; + default: c[i].pTexelBufferView = e.dup(w[i].pTexelBufferView, w[i].descriptorCount); break; + } + } + e.emit(OP_vkCmdPushDescriptorSetKHR, [=] { ::vkx::vkCmdPushDescriptorSetKHR(cb, bp, layout, set, n, c); }, + [&] { ::vkx::vkCmdPushDescriptorSetKHR(cb, bp, layout, set, n, w); }); +} +inline constexpr auto& rx_vkCmdPushDescriptorSetKHR = Route<::vkx::vkCmdPushDescriptorSetKHR, pk_vkCmdPushDescriptorSetKHR>::call; + +RX_REC_PACKER void pk_vkCmdSetViewport(VkCommandBuffer cb, uint32_t first, uint32_t n, const VkViewport* v) { + Enc e; + const VkViewport* c = e.dup(v, n); + e.emit(OP_vkCmdSetViewport, [=] { ::vkx::vkCmdSetViewport(cb, first, n, c); }, [&] { ::vkx::vkCmdSetViewport(cb, first, n, v); }); +} +inline constexpr auto& rx_vkCmdSetViewport = Route<::vkx::vkCmdSetViewport, pk_vkCmdSetViewport>::call; + +RX_REC_PACKER void pk_vkCmdSetScissor(VkCommandBuffer cb, uint32_t first, uint32_t n, const VkRect2D* r) { + Enc e; + const VkRect2D* c = e.dup(r, n); + e.emit(OP_vkCmdSetScissor, [=] { ::vkx::vkCmdSetScissor(cb, first, n, c); }, [&] { ::vkx::vkCmdSetScissor(cb, first, n, r); }); +} +inline constexpr auto& rx_vkCmdSetScissor = Route<::vkx::vkCmdSetScissor, pk_vkCmdSetScissor>::call; + +RX_REC_PACKER void pk_vkCmdSetViewportWithCount(VkCommandBuffer cb, uint32_t n, const VkViewport* v) { + Enc e; + const VkViewport* c = e.dup(v, n); + e.emit(OP_vkCmdSetViewportWithCount, [=] { ::vkx::vkCmdSetViewportWithCount(cb, n, c); }, + [&] { ::vkx::vkCmdSetViewportWithCount(cb, n, v); }); +} +inline constexpr auto& rx_vkCmdSetViewportWithCount = Route<::vkx::vkCmdSetViewportWithCount, pk_vkCmdSetViewportWithCount>::call; + +RX_REC_PACKER void pk_vkCmdSetScissorWithCount(VkCommandBuffer cb, uint32_t n, const VkRect2D* r) { + Enc e; + const VkRect2D* c = e.dup(r, n); + e.emit(OP_vkCmdSetScissorWithCount, [=] { ::vkx::vkCmdSetScissorWithCount(cb, n, c); }, + [&] { ::vkx::vkCmdSetScissorWithCount(cb, n, r); }); +} +inline constexpr auto& rx_vkCmdSetScissorWithCount = Route<::vkx::vkCmdSetScissorWithCount, pk_vkCmdSetScissorWithCount>::call; + +RX_REC_PACKER void pk_vkCmdSetBlendConstants(VkCommandBuffer cb, const float blend[4]) { + Enc e; + const float* c = e.dup(blend, 4); + e.emit(OP_vkCmdSetBlendConstants, [=] { ::vkx::vkCmdSetBlendConstants(cb, c); }, [&] { ::vkx::vkCmdSetBlendConstants(cb, blend); }); +} +inline constexpr auto& rx_vkCmdSetBlendConstants = Route<::vkx::vkCmdSetBlendConstants, pk_vkCmdSetBlendConstants>::call; + +RX_REC_PACKER void pk_vkCmdSetColorBlendEnableEXT(VkCommandBuffer cb, uint32_t first, uint32_t n, const VkBool32* en) { + Enc e; + const VkBool32* c = e.dup(en, n); + e.emit(OP_vkCmdSetColorBlendEnableEXT, [=] { ::vkx::vkCmdSetColorBlendEnableEXT(cb, first, n, c); }, + [&] { ::vkx::vkCmdSetColorBlendEnableEXT(cb, first, n, en); }); +} +inline constexpr auto& rx_vkCmdSetColorBlendEnableEXT = Route<::vkx::vkCmdSetColorBlendEnableEXT, pk_vkCmdSetColorBlendEnableEXT>::call; + +RX_REC_PACKER void pk_vkCmdSetColorBlendEquationEXT(VkCommandBuffer cb, uint32_t first, uint32_t n, const VkColorBlendEquationEXT* eq) { + Enc e; + const VkColorBlendEquationEXT* c = e.dup(eq, n); + e.emit(OP_vkCmdSetColorBlendEquationEXT, [=] { ::vkx::vkCmdSetColorBlendEquationEXT(cb, first, n, c); }, + [&] { ::vkx::vkCmdSetColorBlendEquationEXT(cb, first, n, eq); }); +} +inline constexpr auto& rx_vkCmdSetColorBlendEquationEXT = Route<::vkx::vkCmdSetColorBlendEquationEXT, pk_vkCmdSetColorBlendEquationEXT>::call; + +RX_REC_PACKER void pk_vkCmdSetColorWriteMaskEXT(VkCommandBuffer cb, uint32_t first, uint32_t n, const VkColorComponentFlags* wm) { + Enc e; + const VkColorComponentFlags* c = e.dup(wm, n); + e.emit(OP_vkCmdSetColorWriteMaskEXT, [=] { ::vkx::vkCmdSetColorWriteMaskEXT(cb, first, n, c); }, + [&] { ::vkx::vkCmdSetColorWriteMaskEXT(cb, first, n, wm); }); +} +inline constexpr auto& rx_vkCmdSetColorWriteMaskEXT = Route<::vkx::vkCmdSetColorWriteMaskEXT, pk_vkCmdSetColorWriteMaskEXT>::call; + +/// (the marker is an opaque value the driver hands back, never read through: kept as the pointer value) +RX_REC_PACKER void pk_vkCmdSetCheckpointNV(VkCommandBuffer cb, const void* marker) { + Enc e; + e.emit(OP_vkCmdSetCheckpointNV, [=] { ::vkx::vkCmdSetCheckpointNV(cb, marker); }, [&] { ::vkx::vkCmdSetCheckpointNV(cb, marker); }); +} +inline constexpr auto& rx_vkCmdSetCheckpointNV = Route<::vkx::vkCmdSetCheckpointNV, pk_vkCmdSetCheckpointNV>::call; + +} // namespace mkwvk::rec diff --git a/gpu/vk/vk_reflex.cpp b/gpu/vk/vk_reflex.cpp index 46f5072..5c4e4b0 100644 --- a/gpu/vk/vk_reflex.cpp +++ b/gpu/vk/vk_reflex.cpp @@ -26,6 +26,7 @@ #include "vk_framegen.h" #include "vk_menu.h" +#include "vk_work.h" // (last: RX_VK_RECORDER=defer drains before the latency-sleep wait; =thread does not need to, the driver signals it) namespace mkwvk { diff --git a/gpu/vk/vk_replay.cpp b/gpu/vk/vk_replay.cpp index ce7bb83..3e58ba7 100644 --- a/gpu/vk/vk_replay.cpp +++ b/gpu/vk/vk_replay.cpp @@ -78,7 +78,25 @@ std::unordered_map q_last; // queue -> its last submission std::map value_queue; // a submission's value -> its queue (sync_order_mu) std::atomic pushed{0}; std::atomic q_values{0}; // the fence values handed out (a submit or a present takes one each) -std::mutex replay_mu; // one replay at a time (the game submits from several threads) +::mkwvk::rec::DrainingMutex replay_mu; // one replay at a time (the game submits from several threads; RX_VK_RECORDER=defer: what + // a thread recorded is drained before another may take it) +// (RX_VK_RECORDER=thread: the recorder thread named, and above normal priority as par_worker_main's threads) +// RX_VK_RECORDER_PIN (experiment): next = the physical core below the reserved one (the replay thread's: hos_reserved_cpus), +// sibling = the reserved core itself (shares it with the replay thread), or a hex mask; unset: not pinned +const bool rec_thread_hook = (::mkwvk::rec::g_thread_start_hook = [] { + SetThreadDescription(GetCurrentThread(), L"mkw_vk recorder"); + SetThreadPriority(GetCurrentThread(), THREAD_PRIORITY_ABOVE_NORMAL); + const char* pin = getenv("RX_VK_RECORDER_PIN"); + if (!pin || !*pin) return; + const uint64_t reserved = host && host->reserved_cpus ? host->reserved_cpus() : 0; + uint64_t m = 0; + if (!strcmp(pin, "next")) m = reserved ? reserved >> __builtin_popcountll(reserved) : 0; + else if (!strcmp(pin, "sibling")) m = reserved; + else m = strtoull(pin, nullptr, 16); + const bool ok = m && SetThreadAffinityMask(GetCurrentThread(), (DWORD_PTR)m) != 0; + ::mkwvk::logf("vk: RX_VK_RECORDER_PIN=%s: the recorder %s %#llx (reserved %#llx)", pin, ok ? "pinned to" : "NOT pinned, mask", + (unsigned long long)m, (unsigned long long)reserved); +}, true); // (RX_VK_BATCH: see queue_flush_locked) VkSemaphore batch_tl = VK_NULL_HANDLE; uint64_t batch_next = 0, batch_flushed = 0; @@ -1723,8 +1741,13 @@ void bench_frame(int64_t presents) { struct SlowWatch { std::chrono::steady_clock::time_point t0 = std::chrono::steady_clock::now(); int64_t prof0[PROF_COUNT]; + int64_t ms0[4]; // (hookshot: memory sync's parts and the lock wait, at the start) const char* what; - explicit SlowWatch(const char* w) : what(w) { std::memcpy(prof0, prof_ns, sizeof prof0); } + explicit SlowWatch(const char* w) : what(w) { + std::memcpy(prof0, prof_ns, sizeof prof0); + for (int i = 0; i < 3; i++) ms0[i] = memsync_part_ns[i].load(std::memory_order_relaxed); + ms0[3] = memsync_lock_wait_ns.load(std::memory_order_relaxed); + } ~SlowWatch() { const double ms = std::chrono::duration(std::chrono::steady_clock::now() - t0).count(); static int logged; @@ -1739,6 +1762,15 @@ struct SlowWatch { const double d = (prof_ns[i] - prof0[i]) / 1e6; if (d >= 1.0) k += snprintf(line + k, sizeof line - k, " %s %.1f", names[i], d); } + { // (hookshot: which part of "memory sync" it was: the pools' refresh [and its lock waits], the capture, the written pages collected. + // Counted across threads, so a part another thread spent meanwhile shows too) + const double a = (memsync_part_ns[0].load(std::memory_order_relaxed) - ms0[0]) / 1e6, + b = (memsync_part_ns[1].load(std::memory_order_relaxed) - ms0[1]) / 1e6, + c = (memsync_part_ns[2].load(std::memory_order_relaxed) - ms0[2]) / 1e6, + w = (memsync_lock_wait_ns.load(std::memory_order_relaxed) - ms0[3]) / 1e6; + if (a + b + c >= 1.0 || w >= 1.0) + k += snprintf(line + k, sizeof line - k, " [memory sync: maps %.1f (lock waits %.1f), capture %.1f, collect %.1f]", a, w, b, c); + } logf("%s", line); } }; @@ -1749,6 +1781,8 @@ void submit_memory_work(uint64_t value) { PROF(PROF_MEMSYNC); const auto m0 = std::chrono::steady_clock::now(); memory_sync(); + const auto m1a = std::chrono::steady_clock::now(); + memsync_part_ns[0] += std::chrono::duration_cast(m1a - m0).count(); // (the capture takes only the mirrored pools' pages, the collection the rest: skipping it whenever a capture was // taken, which with the replay thread is at every submit, left every other CPU write uncollected. Textures never // saw them: MK World's font cache kept its first glyphs (the menus lost letters), CPU-streamed terrain went stale) @@ -1757,6 +1791,8 @@ void submit_memory_work(uint64_t value) { const auto m2 = std::chrono::steady_clock::now(); memory_collect_written(); const auto m3 = std::chrono::steady_clock::now(); + memsync_part_ns[1] += std::chrono::duration_cast(m2 - m1).count(); + memsync_part_ns[2] += std::chrono::duration_cast(m3 - m2).count(); { // (every 600 submits: what the memory work before a submit's recording cost) static double a, b, c; static int64_t k; diff --git a/gpu/vk/vk_texture.cpp b/gpu/vk/vk_texture.cpp index 266afa8..ff32d54 100644 --- a/gpu/vk/vk_texture.cpp +++ b/gpu/vk/vk_texture.cpp @@ -1289,16 +1289,19 @@ void watch_tex(const Texture* tex, const char* what) { } /// Whether [gpu, gpu + n) is storage the GPU's images rendered last (covered by ranges render targets took), and the /// generation from which the CPU's writes would count (0: not the GPU's) +/// (hookshot) The ranges can overlap (texture_rendered drops only the ones inside a new range), so the bytes they cover are +/// counted once each: summed, two overlapping ranges counted their overlap twice and a range with a gap the GPU never drew +/// read as covered, and a texture over it was never uploaded uint32_t gpu_owned_since(uint64_t gpu, uint64_t n) { - uint64_t covered = 0; + uint64_t covered = 0, reach = gpu; // (reach: the end of what is counted so far; the ranges come in address order) uint32_t since = UINT32_MAX; auto it = gpu_owned.upper_bound(gpu); if (it != gpu_owned.begin()) --it; for (; it != gpu_owned.end() && it->first < gpu + n; ++it) { const uint64_t a = std::max(it->first, gpu), b = std::min(it->second.first, gpu + n); if (b <= a) continue; - covered += b - a; since = std::min(since, it->second.second); + if (b > std::max(a, reach)) covered += b - std::max(a, reach), reach = b; } return covered >= n && since != UINT32_MAX ? std::max(since, 1u) : 0; } @@ -1359,6 +1362,76 @@ bool upload(Texture& tex, const Ranges* dirty) { } return ok; } +/// (hookshot) A big texture whose whole storage is zero: the image cleared to zero on the GPU, not unswizzled on the CPU. All-zero +/// bits are 0 in every format this path takes (not ASTC or BC, not one widened on the way), so the clear is what the unswizzle +/// would have made. The game's shadow map (12288 x 12288 D16, 288 MB, its storage fresh and zero) took 73 ms of the replay +/// thread at a scene change; the check reads the storage once, on several threads, and stops at the first byte that is not 0. +/// RX_VK_ZERO_UPLOAD=0: off +bool upload_all_zero(Texture& tex, uint64_t total, const Ranges* dirty, bool widened_format) { + static const bool off = getenv("RX_VK_ZERO_UPLOAD") && atoi(getenv("RX_VK_ZERO_UPLOAD")) == 0; + if (off || total < (32ull << 20) || tex.astc || tex.bc || tex.sparse || widened_format || !tex.image || + tex.type == VK_IMAGE_TYPE_3D || is_bc(tex.format) || is_astc(tex.format)) + return false; + if (dirty) { // (only an upload of the whole storage: a part of it says nothing of the rest) + uint64_t covered = 0; + for (const auto& r : *dirty) covered += std::min(r.second, total) - std::min(r.first, total); + if (covered < total) return false; + } + // (live memory, as the GPU unswizzle reads it: not a storage the capture kept as it was at its submit, which the CPU path reads) + if (tex.gpu >= (1ull << 40) || memory_has_shadow(tex.gpu, total)) return false; + uint64_t avail = 0; + const uint8_t* p = host->gpu_ptr(tex.gpu, &avail); + if (!p || avail < total) return false; + const auto t0 = std::chrono::steady_clock::now(); + std::atomic found{false}; + const uint32_t nthreads = std::clamp(std::thread::hardware_concurrency() / 2, 1u, 8u); + const uint64_t per = ((total / nthreads) + 4095) & ~4095ull; + auto part = [&](uint32_t t) { + const uint64_t a = std::min(total, (uint64_t)t * per), b = std::min(total, a + per); + for (uint64_t at = a; at < b && !found.load(std::memory_order_relaxed); at += 65536) { + const uint64_t end = std::min(b, at + 65536); + uint64_t acc = 0; + uint64_t k = at; + for (; k + 8 <= end; k += 8) { + uint64_t w; + std::memcpy(&w, p + k, 8); + acc |= w; + } + for (; k < end; k++) acc |= p[k]; + if (acc) found.store(true, std::memory_order_relaxed); + } + }; + if (nthreads > 1) { + std::vector th; + for (uint32_t t = 1; t < nthreads; t++) th.emplace_back(part, t); + part(0); + for (auto& x : th) x.join(); + } else { + part(0); + } + const double ms = std::chrono::duration(std::chrono::steady_clock::now() - t0).count(); + static std::atomic told{0}; + if (found.load()) { + if (told.fetch_add(1) < 4) + logf("vk: upload: %ux%u NVN %#x (%llu MB) is not all zero (checked in %.1f ms): unswizzled as before", tex.width, + tex.height, tex.nvn_format, (unsigned long long)(total >> 20), ms); + return false; + } + texture_resident_flush(&tex); + checkpoint_mark(upload_cmd(), tex.gpu, 'u'); + const VkImageSubresourceRange range{tex.aspect, 0, VK_REMAINING_MIP_LEVELS, 0, VK_REMAINING_ARRAY_LAYERS}; + if (tex.aspect & (VK_IMAGE_ASPECT_DEPTH_BIT | VK_IMAGE_ASPECT_STENCIL_BIT)) { + const VkClearDepthStencilValue z{0.0f, 0}; + vkCmdClearDepthStencilImage(upload_cmd(), tex.image, VK_IMAGE_LAYOUT_GENERAL, &z, 1, &range); + } else { + const VkClearColorValue z{}; + vkCmdClearColorImage(upload_cmd(), tex.image, VK_IMAGE_LAYOUT_GENERAL, &z, 1, &range); + } + if (told.fetch_add(1) < 4) + logf("vk: upload: %ux%u NVN %#x (%llu MB) is all zero: cleared on the GPU (checked in %.1f ms)", tex.width, tex.height, + tex.nvn_format, (unsigned long long)(total >> 20), ms); + return true; +} bool upload_inner(Texture& tex, const Ranges* dirty) { PROF(PROF_UPLOAD); if (!tex.gpu || tex.samples != VK_SAMPLE_COUNT_1_BIT) return false; @@ -1392,6 +1465,7 @@ bool upload_inner(Texture& tex, const Ranges* dirty) { const bool linear = nvn_tex_is_linear(&tex.d); int widen_from = 0; widened(tex.d.format ? vk_format(tex.nvn_format).format : tex.format, &widen_from); + if (upload_all_zero(tex, total, dirty, widen_from != 0)) return true; const int src_bpb = tex.bpb; const int out_bpb = tex.astc ? 4 : widen_from ? (widen_from == 3 ? 4 : widen_from == 6 ? 8 : 16) : src_bpb; const uint32_t copy_layers = tex.type == VK_IMAGE_TYPE_3D ? 1 : tex.layers; @@ -2358,6 +2432,7 @@ bool upload_defer(const Texture& tex) { } // namespace Texture* texture_touch(Texture* tex, bool for_sampling) { tex->last_frame = frame_no; + tex->let_go = 0; // (in use: texture_sweep) tex->last_value = rec_value; if (!for_sampling) return tex; if (upload_defer(*tex)) return nullptr; @@ -2446,7 +2521,22 @@ Texture* texture_touch(Texture* tex, bool for_sampling) { // was unswizzled from that memory: 55 MB for the 3072 x 3072 x 3 shadow map, frame after frame. Not "storage // the write watch never saw written": the game fills memory before it makes a pool of it.) static const bool upload_all = getenv("RX_VK_UPLOAD_ALL") != nullptr; // (A/B: such storage uploaded too) - uint32_t owned = !tex->uploaded && !upload_all ? gpu_owned_since(tex->gpu, tex->guest_size) : 0; + // (from Shakoquo/Mako ff479e988d) never for a block-compressed texture (ASTC, BC, ETC): the GPU cannot render into one, so + // its texels come from memory whatever render targets used that storage before. Skipped, a boss or a new enemy met for + // the first time (its textures loaded into the transient heaps' memory) was drawn with an empty texture, all blue, for + // the whole cutscene. RX_VK_SKIP_COMPRESSED_OVER_RT=1: as before + // (Triforce: this tree has no env_on yet; the switch is on when set to anything but 0) + static const bool skip_compressed = getenv("RX_VK_SKIP_COMPRESSED_OVER_RT") && atoi(getenv("RX_VK_SKIP_COMPRESSED_OVER_RT")) != 0; + const bool compressed = tex->astc || tex->bc || + (tex->format >= VK_FORMAT_BC1_RGB_UNORM_BLOCK && tex->format <= VK_FORMAT_ASTC_12x12_SRGB_BLOCK); + uint32_t owned = !tex->uploaded && !upload_all && (skip_compressed || !compressed) ? gpu_owned_since(tex->gpu, tex->guest_size) : 0; + if (!tex->uploaded && !upload_all && !skip_compressed && compressed) { + static std::atomic told{0}; + if (told.load(std::memory_order_relaxed) < 30 && gpu_owned_since(tex->gpu, tex->guest_size) && told++ < 30) + logf("vk: texture #%u %#llx %ux%u x%u NVN %#x (compressed) over storage render targets took: uploaded from the game's " + "memory all the same", tex->id, (unsigned long long)tex->gpu, tex->width, tex->height, tex->layers, + tex->nvn_format); + } if (owned && !remade_skip_old()) { // (one evicted that came from memory: uploaded from it again. Above) std::lock_guard lk(from_memory_mu); auto e = evicted_from_memory.find(evicted_key(tex->gpu, tex->width, tex->height, tex->nvn_format)); @@ -3264,6 +3354,7 @@ void texture_written(Texture* tex, const char* why) { bool texture_touch_quick(Texture* tex, bool for_sampling) { tex->last_frame = frame_no; // (the same values whichever thread stores them) + tex->let_go = 0; tex->last_value = rec_value; if (!for_sampling) return true; if (tex->gpu_written && !tex->released && !tex->only_cleared) return true; @@ -3558,10 +3649,11 @@ VkBufferView texel_buffer_view(const nvn_tex* t, const nvn_view* v, bool storage return bv; } -// ---- (hookshot) VRAM compacted (RX_VK_COMPACT=1; off by default) ---- +// ---- (hookshot) VRAM compacted (on by default since the hold lets go of the dead render targets first and of long-unused +// textures only; RX_VK_COMPACT=0: off, as before) ---- // An evicted texture's memory stays inside its 128 MB block, and the driver counts the block whole: after some play 0.6 to // 1.6 GB of what the driver counts is such free space (3 to 4 GB after the cache was emptied), and evicting more gives -// none of it back. With RX_VK_COMPACT=1 the textures that are not render targets have a pool of their own (vk_vma.cpp; the targets have theirs) and, when RX_VK_COMPACT_MB (512) +// none of it back. Compacted, the textures that are not render targets have a pool of their own (vk_vma.cpp; the targets have theirs) and, when RX_VK_COMPACT_MB (512) // or more of it is free space, VMA's defragmentation moves textures out of the sparsely used blocks, a few a pass: a new // image at the new place, the texels copied on the GPU in the present's upload commands (every draw of the frame is in an // earlier recording), the texture's views made again when next asked for, and the old image destroyed once the GPU is past @@ -4099,7 +4191,7 @@ uint64_t preset_texture_budget(uint64_t preset_mb, bool preset_applies, bool vra uint64_t dead_sweep_next = 0; // (the present of the next pass for the dead targets between texture_frame's looks; 0: none) void texture_sweep(uint64_t completed, bool sweep, uint64_t budget, uint64_t evict_to, bool known, uint64_t used, uint64_t allowed, - bool dead_only); + bool dead_only, uint64_t hold_to, uint64_t rule_to); void texture_frame(uint64_t completed) { upload_frame_report(); frame_upload_ns.store(0, std::memory_order_relaxed); @@ -4222,9 +4314,10 @@ void texture_frame(uint64_t completed) { // (Triforce, 11-11) between the looks, a pass for the dead transient targets alone while the game makes them fast // (dead_sweep_next, set by texture_sweep): see there if (dead_sweep_next && frame_no >= dead_sweep_next) - texture_sweep(completed, true, ~0ull, 0, last_known, last_used, last_allowed, true); + texture_sweep(completed, true, ~0ull, 0, last_known, last_used, last_allowed, true, 0, UINT64_MAX); // (no hold in this pass) return; } + vma_blocks_ahead_check(); // (Triforce: the blocks made ahead go back while VRAM is short, vk_vma.cpp) uint64_t used = 0, allowed = 0, block_free = 0; const bool known = vram_usage(&used, &allowed, &block_free); last_known = known, last_used = used, last_allowed = allowed; @@ -4264,6 +4357,7 @@ void texture_frame(uint64_t completed) { // out. RX_VK_BUDGET_BY_BLOCKS=1: as before) static const bool by_blocks = getenv("RX_VK_BUDGET_BY_BLOCKS") != nullptr; uint64_t evict_to = 0; // (hookshot, RX_VK_COMPACT: what an eviction goes down to when the compaction's hold set the budget) + uint64_t hold_to = 0; // (and what the hold's own line asks for, when the reserve's lower line set evict_to) // (hookshot, RX_VK_COMPACT) compacted, only the textures' pool's free space comes off the count: vram_compact.h uint64_t reclaim = block_free; if (compact_on()) { @@ -4277,6 +4371,10 @@ void texture_frame(uint64_t completed) { const uint64_t over = pressure - allowed / 10 * 9; budget = std::min(budget, total_bytes > over ? total_bytes - over : 0); } + // (hookshot, RX_VK_COMPACT: what the budget's size and the 90% rule ask on their own, before the hold's lines; the 97% + // rule's joins it below. The hold's pass lets a texture unused for the eviction's own time go for that part, as a pass + // does with the compaction off: vram_compact::hold_target) + uint64_t rule_budget = budget; // (hookshot, RX_VK_COMPACT) and the count is held at RX_VK_COMPACT_PCT% of the allowance (85; 50 to 90), as it will be once a // run has given the textures' free space back: within ~0.8 GB of what the driver allows the GPU's frame went from ~10 ms // to 13-20 ms with the shadows in place, the driver itself short. At 85 a five-minute walk across Hyrule Field at DLAA @@ -4290,6 +4388,11 @@ void texture_frame(uint64_t completed) { // was crossed on five passes in a row and the cache went from 4,589 to 2,312 MB (0.875^5) in two seconds, the // textures made again at 65-90 ms a frame) if (held < budget) budget = evict_to = held; + hold_to = evict_to; + // (and a reserve under that line, for the textures long unused: vram_compact.h. The pass below takes the others only + // for the hold's own line) + const uint64_t reserve = vram_compact::held_budget(budget, total_bytes, used, reclaim, allowed, vram_compact::reserve_pct(pct)); + if (reserve < budget) budget = evict_to = reserve; } // (hookshot) not while VMA's blocks hold an eighth of the allowance free: new images take that space, the driver's count does // not grow, and what this evicts frees space inside the blocks that the count keeps. Ten frames later it squeezed again: a @@ -4301,8 +4404,14 @@ void texture_frame(uint64_t completed) { if (known && !by_blocks && used > allowed / 100 * 97 && !room_in_blocks) { const uint64_t over = used - allowed / 100 * 97; budget = std::min(budget, total_bytes > over ? total_bytes - over : 0); + rule_budget = std::min(rule_budget, total_bytes > over ? total_bytes - over : 0); } - if (budget < evict_to) evict_to = 0; // (hookshot, RX_VK_COMPACT: the driver's limit asks for more: 7/8 of that, as before) + const uint64_t rule_to = vram_compact::rule_target(total_bytes, rule_budget); + // (hookshot, RX_VK_COMPACT: the driver's limit asks for more than the reserve does: a pass over the budget, to 7/8 of it, as + // before. Rare once the reserve is set: its line is 17 points under the limit's and counts the textures' free space out, so + // this takes more than that share of the allowance free in their pool. Otherwise the pass is the hold's, and what the + // limit and the 90% rule ask for goes by rule_to) + if (budget < evict_to) evict_to = hold_to = 0; if (frame_no % 600 == 0) { memory_trial_tick(); // (Triforce: the memory handling on trial on this card, vk_memory.cpp) vma_log_heaps(); @@ -4387,7 +4496,7 @@ void texture_frame(uint64_t completed) { logf("vk: verify %ux%u: %d textures, %d changed since uploaded, %d of them unnoticed", verify_dim().first, verify_dim().second, checked, changed, missed); } - texture_sweep(completed, total_bytes <= budget, budget, evict_to, known, used, allowed, false); + texture_sweep(completed, total_bytes <= budget, budget, evict_to, known, used, allowed, false, hold_to, rule_to); } /// (texture_frame) What goes: over the budget the least recently used, and the render targets drawn over since. @@ -4400,7 +4509,7 @@ void texture_frame(uint64_t completed) { /// pass is the next present, at one in three presents three presents on, else the next look as before: the dead wait 4-6 /// presents, not 13. RX_VK_DEAD_QUICK=0: only at the looks, as before void texture_sweep(uint64_t completed, bool sweep, uint64_t budget, uint64_t evict_to, bool known, uint64_t used, uint64_t allowed, - bool dead_only) { + bool dead_only, uint64_t hold_to, uint64_t rule_to) { static const bool no_evict = getenv("RX_VK_NO_EVICT") != nullptr; // (an experiment: nothing evicted) // (hookshot) under the budget, every 600 presents: the render targets drawn over since (below), unused for 120 // (hookshot-amd) and every 60 presents for the render targets drawn over by one at another start (below): OoT makes a frame's @@ -4522,14 +4631,55 @@ void texture_sweep(uint64_t completed, bool sweep, uint64_t budget, uint64_t evi struct Evict { Texture* tex; bool flush, shifted; }; std::vector evict; uint64_t pending_bytes = 0; // (chosen below, their bytes not yet off total_bytes) + // (hookshot, RX_VK_COMPACT) a pass the compaction's hold asked for (evict_to) lets go of the dead render targets as a sweep + // does, all of them, and of live textures only for the excess left after those, and only of ones long unused: vram_compact.h's + // hold_met and hold_target. So it does not stop at the first texture past the excess: what the dead targets hold is counted first + const bool by_hold = evict_to != 0 && !sweep; + uint64_t hold_dead = 0, hold_live = 0; // (what the dead targets hold; the live textures chosen so far) + if (by_hold) + for (auto& [last, tex] : order) { + if (frame_no - last < need_dead) break; + if (tex->last_value > completed) continue; + const bool shifted = drawn_over_shifted(tex); + if (frame_no - last < need_unused && !shifted) continue; + if (shifted || drawn_over(tex)) hold_dead += tex->bytes + tex->pad; + } + // (Triforce, 11-12) What the game has let go goes first while VRAM is short. At an area's arrival the textures of the area + // left stayed until a rule asked for room, which was after the new area's had come in: those took new blocks, the old ones + // then left holes in theirs, and for seconds the driver counted both. A lab arrival in Hyrule Field from Kakariko: VRAM + // 10,993 to 11,249 MB of 10,468 to 10,663 allowed with 1.3 GB free inside the blocks, and each image made meanwhile took + // 93 to 124 ms (three frames of 97 to 134 ms). A texture uploaded from memory whose texture the game finalized + // RX_VK_LET_GO_AFTER presents ago (30) and that nothing has used since goes at the next pass while VRAM is over + // RX_VK_LET_GO_PCT% of what the driver allows (75; 0: off, as before), whatever its age: its room is there when the next + // area's textures come. One the game makes again is uploaded from memory again, as after any eviction. + // Only when the game has let go of much at once, RX_VK_LET_GO_MB and more (256: an area left behind): at the Market the + // game finalizes and makes again a few backdrop textures all the time, and with every one of those going at once the + // uploads there went from 1.4-1.7 GB to 3.6 GB in 95 seconds. (Not "or whenever VRAM is over 92%" either: a card that + // sits there for good, 5.6 GB allowed, had more frames over 40 ms with that than without, 559 against 369 in a tour.) + // And only where the driver allows 9 GB and more, unless RX_VK_LET_GO_PCT is set: at 5.6 GB allowed three tours with this + // and the blocks made ahead had 559-896 frames over 40 ms against 369-614 without (vk_vma.cpp) + static const bool let_go_set = getenv("RX_VK_LET_GO_PCT") != nullptr; + static const uint64_t let_go_pct = let_go_set ? (uint64_t)std::clamp(atoi(getenv("RX_VK_LET_GO_PCT")), 0, 100) : 75; + static const uint64_t let_go_after = getenv("RX_VK_LET_GO_AFTER") ? (uint64_t)std::max(1, atoi(getenv("RX_VK_LET_GO_AFTER"))) : 30; + static const uint64_t let_go_mass_min = (uint64_t)(getenv("RX_VK_LET_GO_MB") ? std::max(0, atoi(getenv("RX_VK_LET_GO_MB"))) : 256) << 20; + bool let_go_short = let_go_pct && known && used > allowed / 100 * let_go_pct && (let_go_set || allowed >= (9ull << 30)); + if (let_go_short) { + uint64_t mass = 0; + for (auto& [last, tex] : order) + if (tex->let_go && !tex->gpu_written && frame_no - tex->let_go >= let_go_after) mass += tex->bytes + tex->pad; + let_go_short = mass >= let_go_mass_min; + } + static uint64_t let_go_n, let_go_bytes, let_go_logged_at; for (auto& [last, tex] : order) { // (total_bytes can have drifted below the textures' sum (destroy_later clamps it): the chosen ones' bytes past it mean the // budget is met, not a wrapped difference that never is) - if ((!sweep && (pending_bytes >= total_bytes || total_bytes - pending_bytes <= (evict_to ? evict_to : budget / 8 * 7))) || + if ((!sweep && !by_hold && (pending_bytes >= total_bytes || total_bytes - pending_bytes <= budget / 8 * 7)) || frame_no - last < need_dead) break; const bool shifted = drawn_over_shifted(tex); - if (frame_no - last < need_unused && !shifted) continue; // (younger: only a dead transient target goes) + const bool gone_early = let_go_short && tex->let_go && !tex->gpu_written && + frame_no - tex->let_go >= let_go_after; + if (frame_no - last < need_unused && !shifted && !gone_early) continue; // (younger: only a dead transient target goes) if (tex->last_value > completed) continue; // (nor what the GPU wrote and the game still has: the image is its only copy, and one made again over that // storage started empty or from stale memory: a sky or course texture drawn once, garbage from then on. @@ -4543,6 +4693,13 @@ void texture_sweep(uint64_t completed, bool sweep, uint64_t budget, uint64_t evi static const bool keep_over_only = getenv("RX_VK_KEEP_RENDERED_OVER") != nullptr; const uint64_t keep_rendered = keep_rendered::presents(known, used, allowed, keep_over_only); const bool overwritten = shifted || drawn_over(tex); // (above) + // (hookshot, RX_VK_COMPACT: the hold's pass, above. A live texture stays once the excess over its line is covered: the + // reserve's line for one long unused, the hold's own for one unused ten seconds, and what the other rules ask (the 90% + // rule, the budget's size, the driver's limit) for one unused as long as an eviction asks with the compaction off) + if (by_hold && !overwritten && !gone_early) { + const uint64_t to = vram_compact::hold_target(frame_no - last, need_unused, rule_to, hold_to, evict_to); + if (to == UINT64_MAX || vram_compact::hold_met(total_bytes, hold_live, hold_dead, to)) continue; + } if (finalized_kept && keep_finalized(tex)) { // (above. Oldest first: beyond the cap the least recently used go) if (finalized_kept <= keep_finalized_max) continue; finalized_kept -= tex->bytes; @@ -4553,7 +4710,8 @@ void texture_sweep(uint64_t completed, bool sweep, uint64_t budget, uint64_t evi tex->height, tex->nvn_format, (unsigned long long)(frame_no - last)); } if (!overwritten && tex->gpu_written && !tex->released && !evict_rendered && frame_no - last < keep_rendered) continue; - if (sweep && !overwritten) continue; + if (sweep && !overwritten && !gone_early) continue; + if (gone_early) let_go_n++, let_go_bytes += tex->bytes + tex->pad; // (hookshot) nor a live texture the GPU drew whose texels cannot be written back into its memory (ASTC kept as BC3, sparse, // past the write-back's size, ...): its image is the only copy, and one made again over that storage is not uploaded (the // memory counts as the GPU's), so it showed whatever its new memory held: the rainbow/garbled textures that spread over play @@ -4589,6 +4747,7 @@ void texture_sweep(uint64_t completed, bool sweep, uint64_t budget, uint64_t evi static const bool no_flush = getenv("RX_VK_EVICT_NO_FLUSH") != nullptr; evict.push_back({tex, !no_flush && !overwritten && tex->gpu_written && !tex->released && flushable(*tex), shifted}); pending_bytes += tex->bytes + tex->pad; + if (by_hold && !overwritten) hold_live += tex->bytes + tex->pad; } { // (the write-backs: batches of up to 64 textures or ~512 MB of staging memory, each one submission and one wait) std::vector batch; @@ -4647,6 +4806,12 @@ void texture_sweep(uint64_t completed, bool sweep, uint64_t budget, uint64_t evi } const size_t n = order.size() - textures.size(); if (n && !dead_only) logf("vk: %zu textures retired (%zu left, %llu MB)", n, textures.size(), (unsigned long long)(total_bytes >> 20)); + if (let_go_n / 256 != let_go_logged_at) { + let_go_logged_at = let_go_n / 256; + logf("vk: textures the game had let go, gone before their age while VRAM is over %llu%% of what the driver allows: %llu so " + "far (%llu MB; RX_VK_LET_GO_PCT=0: off)", (unsigned long long)let_go_pct, (unsigned long long)let_go_n, + (unsigned long long)(let_go_bytes >> 20)); + } // (the next pass for the dead targets: by how fast this one found them coming) if (dead_quick && dead_targets) { if (shifted_now >= pass_presents) dead_sweep_next = frame_no + 1; @@ -4696,6 +4861,7 @@ void texture_log_guest(const Texture* t) { void texture_released(const nvn_tex* t) { if (!t) return; auto [a, b] = by_gpu.equal_range(t->gpu); + for (auto it = a; it != b; ++it) it->second->let_go = std::max(frame_no, 1); // (texture_sweep: first to go while VRAM is short) for (auto it = a; it != b; ++it) if (it->second->gpu_written && !it->second->released) { it->second->released = true, it->second->release_gen = memory_generation(); diff --git a/gpu/vk/vk_texture.h b/gpu/vk/vk_texture.h index 30bbb7e..6b2bdf3 100644 --- a/gpu/vk/vk_texture.h +++ b/gpu/vk/vk_texture.h @@ -64,6 +64,8 @@ struct Texture { bool only_cleared = false; // (hookshot) all the GPU did was clear it: what the CPU writes after still counts (a font cache's glyphs) uint32_t gpu_gen = 0; // (the memory generation when the GPU last wrote it: CPU writes after it are taken) bool released = false; // its texture was finalized after the GPU wrote it: CPU writes count again + uint64_t let_go = 0; // (Triforce) the present at which the game finalized a texture over it, 0 once it is used again: + // one uploaded from memory goes first while VRAM is short (texture_sweep) uint32_t release_gen = 0; uint64_t gpu_wrote_seen = 0; // (the newest finished GPU write into its storage it was uploaded after) uint64_t last_written = 0; // (the frame the GPU last wrote it: a render target, each frame it is used) diff --git a/gpu/vk/vk_vma.cpp b/gpu/vk/vk_vma.cpp index 00ba81c..9457ceb 100644 --- a/gpu/vk/vk_vma.cpp +++ b/gpu/vk/vk_vma.cpp @@ -8,12 +8,19 @@ #include #include #include +#include #include +#include +#include +#include #include +#include "vk_work.h" // (last: RX_VK_RECORDER=defer drains before the destructions and the invalidate below) + namespace mkwvk { VmaAllocator vma; +uint64_t test_card_ballast = 0; // (RX_VK_TEST_CARD_MB, a lab tool: the VRAM taken so that the card plays a smaller one; vma_init) bool vma_init(PFN_vkGetInstanceProcAddr gipa) { VmaVulkanFunctions f{}; @@ -33,8 +40,21 @@ bool vma_init(PFN_vkGetInstanceProcAddr gipa) { // of their own. RX_VK_TEST_ALLOWED_MB only changes the number the budget is told; this fills the card for real, as a card // with less VRAM or another program's use does: the driver's own behaviour at its limit (allocations of 110-170 ms in a // player's log) only shows then - if (const char* e = getenv("RX_VK_TEST_BALLAST_MB")) { - const uint64_t want = (uint64_t)std::max(0, atoi(e)) << 20, chunk = 256ull << 20; + // (Triforce, 11-12, a lab tool) RX_VK_TEST_CARD_MB=: a card of n MB played on a larger one. The rest of the card is taken + // as ballast, and neither what the process uses nor what the driver allows counts it (vram_usage, device_vram_bytes), so + // every rule that goes by the card's size or by the share used sees the small card, and the driver is as short as there. + // With RX_VK_TEST_BALLAST_MB alone the rules still saw the large card (the textures' budget 70% of it, the mirrors' 3 GB) + const char* card = getenv("RX_VK_TEST_CARD_MB"); + uint64_t card_ballast = 0; + if (card) { + uint64_t heap = 0; + for (uint32_t h = 0; h < dev.memory.memoryHeapCount; h++) + if (dev.memory.memoryHeaps[h].flags & VK_MEMORY_HEAP_DEVICE_LOCAL_BIT) heap = std::max(heap, (uint64_t)dev.memory.memoryHeaps[h].size); + const uint64_t n = (uint64_t)std::max(0, atoi(card)) << 20; + card_ballast = heap > n ? heap - n : 0; + } + if (const char* e = card ? card : getenv("RX_VK_TEST_BALLAST_MB")) { + const uint64_t want = card ? card_ballast : (uint64_t)std::max(0, atoi(e)) << 20, chunk = 256ull << 20; uint64_t got = 0; for (; got < want; got += chunk) { VkBufferCreateInfo bi{VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO}; @@ -48,7 +68,9 @@ bool vma_init(PFN_vkGetInstanceProcAddr gipa) { VmaAllocation a = VK_NULL_HANDLE; if (vmaCreateBuffer(vma, &bi, &ai, &b, &a, nullptr) != VK_SUCCESS) break; // (kept: never freed) } - logf("vk: lab: %llu MB of VRAM taken as ballast (RX_VK_TEST_BALLAST_MB)", (unsigned long long)(got >> 20)); + if (card) test_card_ballast = got; + logf("vk: lab: %llu MB of VRAM taken as ballast (%s)", (unsigned long long)(got >> 20), + card ? "RX_VK_TEST_CARD_MB: not counted, the card plays a smaller one" : "RX_VK_TEST_BALLAST_MB"); } return true; } @@ -70,6 +92,12 @@ const uint64_t kImageBlock = (getenv("RX_VK_IMAGE_BLOCK_MB") ? (uint64_t)std::ma VmaPool target_image_pool = VK_NULL_HANDLE; bool target_pool_tried = false; size_t target_floor_blocks = 0; +// (Triforce, 11-12) RX_VK_TARGET_BLOCK_MB=, 16 and up: the size of the render targets' blocks alone (the textures' and the +// other images' stay at RX_VK_IMAGE_BLOCK_MB) +uint64_t target_block() { + static const uint64_t b = getenv("RX_VK_TARGET_BLOCK_MB") ? (uint64_t)std::max(16, atoi(getenv("RX_VK_TARGET_BLOCK_MB"))) << 20 : kImageBlock; + return b; +} bool target_floor_kept() { return target_floor_blocks != 0; } VmaPool images_pool(); VmaPool target_images_pool() { @@ -91,7 +119,7 @@ VmaPool target_images_pool() { if (vmaFindMemoryTypeIndexForImageInfo(vma, &rep, &ai, &type) != VK_SUCCESS) return VK_NULL_HANDLE; VmaPoolCreateInfo pi{}; pi.memoryTypeIndex = type; - pi.blockSize = kImageBlock; + pi.blockSize = target_block(); // (Triforce, 11-11) A floor of blocks the pool keeps for the session (RX_VK_TARGET_FLOOR_MB; 0: none, as before). While the // camera moves the game makes its transient targets at new sizes all the time (a player's session: up to 6733 new images in // 600 presents, 11 a frame), their blocks emptied and filled again; VMA keeps one empty block and frees the rest, and at the @@ -99,7 +127,7 @@ VmaPool target_images_pool() { // driver, 110-170 ms each there. That session had ten frames of 115-180 ms in a row from it, VRAM climbing 9.5 -> 10.8 GB // of 10.5 allowed. Below the floor VMA never gives a block back, so nothing is asked of the driver in play static const uint64_t floor_mb = getenv("RX_VK_TARGET_FLOOR_MB") ? (uint64_t)std::max(0, atoi(getenv("RX_VK_TARGET_FLOOR_MB"))) : 0; - pi.minBlockCount = (size_t)((floor_mb << 20) / kImageBlock); + pi.minBlockCount = (size_t)((floor_mb << 20) / target_block()); target_floor_blocks = pi.minBlockCount; if (vmaCreatePool(vma, &pi, &target_image_pool) != VK_SUCCESS) { pi.minBlockCount = 0; // (no room for the floor now: the pool as before) @@ -107,7 +135,7 @@ VmaPool target_images_pool() { if (vmaCreatePool(vma, &pi, &target_image_pool) != VK_SUCCESS) target_image_pool = VK_NULL_HANDLE; } logf("vk: render targets' images from blocks of their own (%llu MB, memory type %u; %zu blocks kept for the session)%s", - (unsigned long long)(kImageBlock >> 20), type, pi.minBlockCount, target_image_pool ? "" : ": not made"); + (unsigned long long)(target_block() >> 20), type, pi.minBlockCount, target_image_pool ? "" : ": not made"); return target_image_pool; } @@ -135,7 +163,7 @@ VmaPool images_pool() { return image_pool; } -// (hookshot) The sampled textures' own pool, when they are compacted (RX_VK_COMPACT=1): VMA's defragmentation asks to move any +// (hookshot) The sampled textures' own pool, when they are compacted (unless RX_VK_COMPACT=0): VMA's defragmentation asks to move any // allocation of the pool it works on, and only vk_texture.cpp's images can be moved (it knows what they hold and who uses // them) or kept from being freed while a pass is open. The render targets' images stay in their blocks above; DLSS's, the // frame interpolation's and the scratch ones in the images' pool @@ -270,6 +298,119 @@ bool create_image(const VkImageCreateInfo& ci, VkImage& image, VmaAllocation& al } return ok; } +// ---- (Triforce, 11-12) a block made ahead ---- +// A pool's new block is a vkAllocateMemory on the thread that asked for the image, the replay thread: 37 to 55 ms each for the +// textures' pool in a player's session on an RTX 4070 SUPER (four of them inside the frames of an area's arrival, VRAM at 26 to +// 54% of what the driver allows), 18 to 63 ms for a mirror's piece, and 110 to 170 ms near the driver's limit. VMA keeps one +// empty block for the next image, but lets it go at the next free of anything else in the pool while it is the last block, so +// with images coming and going there is seldom one. Here each pool holds a block of its own in reserve: an allocation of a whole +// block, made on a thread of its own, that nothing uses. An image (or a piece) first asks the pool for room in the blocks it +// has (VMA_ALLOCATION_CREATE_NEVER_ALLOCATE); with none, the reserve is let go, which leaves its block empty, the image goes +// there, and the thread makes the next reserve. No reserve is made that would put VRAM over 88% of what the driver allows, and +// the ones held go back over 93% (vma_blocks_ahead_check, every 600 presents): a card that is short keeps nothing aside. What +// is held is left out of vram_usage's count: the rules that go by the share used (the textures' budget, the compaction's +// hold) see the VRAM as it is without the reserves, which go the moment anything needs them. +// Only where the driver allows the process 9 GB and more (cards of more than 8 GB): a card played at 5.6 GB allowed +// (RX_VK_TEST_CARD_MB=7168, three tours against two) had 559-896 frames over 40 ms with this and the let-go rule against +// 369-614 without, and flat textures on screen for 6.1-7.4 s against 3.9-4.3: there every block held aside is missed. +// RX_VK_BLOCK_AHEAD=0: off, as before; =1: on whatever the driver allows +namespace { +struct Ahead { + std::atomic held{nullptr}; + std::atomic asked{false}; + uint64_t block = 0; + const char* what = ""; +}; +Ahead ahead_targets, ahead_textures, ahead_images, ahead_pieces; +uint64_t ahead_held_bytes() { + uint64_t n = 0; + for (const Ahead* a : {&ahead_targets, &ahead_textures, &ahead_images, &ahead_pieces}) + if (a->held.load(std::memory_order_relaxed)) n += a->block; + return n; +} +std::mutex ahead_mu; +std::condition_variable ahead_cv; +std::deque> ahead_queue; +std::atomic st_ahead_made{0}, st_ahead_used{0}, st_ahead_none{0}, st_ahead_back{0}; +int ahead_mode() { // (0 off, 1 on, 2 by what the driver allows: the default) + static const int m = getenv("RX_VK_BLOCK_AHEAD") ? (atoi(getenv("RX_VK_BLOCK_AHEAD")) != 0 ? 1 : 0) : 2; + return m; +} +bool ahead_on() { return ahead_mode() != 0; } +constexpr uint64_t kAheadMinAllowed = 9ull << 30; +void ahead_main() { + for (;;) { + std::pair job; + { + std::unique_lock lk(ahead_mu); + ahead_cv.wait(lk, [] { return !ahead_queue.empty(); }); + job = ahead_queue.front(); + ahead_queue.pop_front(); + } + Ahead& a = *job.second; + uint64_t used = 0, allowed = 0; + const bool known = vram_usage(&used, &allowed, nullptr); + used += ahead_held_bytes(); // (vram_usage leaves them out) + if (a.held.load() || (known && used + a.block > allowed / 100 * 88) || // (one there, or the card has no room to keep one, + (ahead_mode() == 2 && (!known || allowed < kAheadMinAllowed))) { // or is a small one) + a.asked = false; + continue; + } + VkMemoryRequirements req{}; + req.size = a.block, req.alignment = 4096, req.memoryTypeBits = ~0u; + VmaAllocationCreateInfo ai{}; + ai.pool = job.first; + if (dev.memory_budget) ai.flags = VMA_ALLOCATION_CREATE_WITHIN_BUDGET_BIT; + VmaAllocation r = VK_NULL_HANDLE; + const auto t0 = std::chrono::steady_clock::now(); + if (vmaAllocateMemory(vma, &req, &ai, &r, nullptr) == VK_SUCCESS) { + const double ms = std::chrono::duration(std::chrono::steady_clock::now() - t0).count(); + a.held = r; + const int64_t n = ++st_ahead_made; + if (n <= 6 || n % 32 == 0) + logf("vk: a %llu MB block made ahead for %s in %.1f ms, off the replay thread (%lld made, %lld used, %lld times none " + "was there, %lld given back; RX_VK_BLOCK_AHEAD=0: off)", (unsigned long long)(a.block >> 20), a.what, ms, + (long long)n, (long long)st_ahead_used.load(), (long long)st_ahead_none.load(), (long long)st_ahead_back.load()); + } + a.asked = false; + } +} +void ahead_ask(VmaPool pool, Ahead& a) { + if (a.held.load() || a.asked.exchange(true)) return; + std::lock_guard lk(ahead_mu); + static bool started; + if (!started) { + started = true; + std::thread(ahead_main).detach(); + } + ahead_queue.push_back({pool, &a}); + ahead_cv.notify_one(); +} +/// The pool's block made ahead let go, so that the allocation that found no room goes into it; false: none was there +bool ahead_take(VmaPool pool, Ahead& a) { + VmaAllocation r = compact_in_pass && pool == texture_pool ? VK_NULL_HANDLE : a.held.exchange(nullptr); // (an open pass may list it) + if (r) vmaFreeMemory(vma, r), st_ahead_used++; + else st_ahead_none++; + ahead_ask(pool, a); + return r != VK_NULL_HANDLE; +} +Ahead* ahead_of(VmaPool pool) { + if (!ahead_on() || !pool) return nullptr; + Ahead* a = pool == texture_pool ? &ahead_textures : pool == target_image_pool ? &ahead_targets : pool == image_pool ? &ahead_images : nullptr; + if (a && !a->block) + a->block = a == &ahead_targets ? target_block() : kImageBlock, a->what = a == &ahead_textures ? "the textures" : a == &ahead_targets ? "the render targets" : "the other images"; + return a; +} +} // namespace +void vma_blocks_ahead_check() { + if (!ahead_on()) return; + uint64_t used = 0, allowed = 0; + if (!ahead_held_bytes() || !vram_usage(&used, &allowed, nullptr) || used + ahead_held_bytes() <= allowed / 100 * 93) return; + for (Ahead* a : {&ahead_targets, &ahead_textures, &ahead_images, &ahead_pieces}) { + if (a == &ahead_textures && compact_in_pass) continue; + if (VmaAllocation r = a->held.exchange(nullptr)) vmaFreeMemory(vma, r), st_ahead_back++; + } +} bool create_image_inner(const VkImageCreateInfo& ci, VkImage& image, VmaAllocation& allocation, bool dedicated, bool target, bool textures) { VmaAllocationCreateInfo ai{}; ai.usage = VMA_MEMORY_USAGE_AUTO_PREFER_DEVICE; @@ -289,6 +430,15 @@ bool create_image_inner(const VkImageCreateInfo& ci, VkImage& image, VmaAllocati VmaAllocationCreateInfo pai = ai; pai.pool = pool; pai.usage = VMA_MEMORY_USAGE_UNKNOWN; + if (Ahead* a = ahead_of(pool)) { // (Triforce: in the blocks the pool has; else in the one made ahead: above) + VmaAllocationCreateInfo nai = pai; + nai.flags |= VMA_ALLOCATION_CREATE_NEVER_ALLOCATE_BIT; + if (vmaCreateImage(vma, &ci, &nai, &image, &allocation, nullptr) == VK_SUCCESS) { + ahead_ask(pool, *a); + return true; + } + if (ahead_take(pool, *a) && vmaCreateImage(vma, &ci, &nai, &image, &allocation, nullptr) == VK_SUCCESS) return true; + } if (vmaCreateImage(vma, &ci, &pai, &image, &allocation, nullptr) == VK_SUCCESS) return true; create_image_path = 1; // (the pool refused it: the allocator's own blocks, or a memory object of its own) } @@ -345,6 +495,11 @@ bool create_buffer_cached(uint64_t size, VkBufferUsageFlags usage, VkBuffer& buf if (mapped) *mapped = info.pMappedData; return true; } +bool host_cached(VmaAllocation allocation) { + VkMemoryPropertyFlags f = 0; + if (allocation) vmaGetAllocationMemoryProperties(vma, allocation, &f); + return (f & VK_MEMORY_PROPERTY_HOST_CACHED_BIT) != 0; +} void flush_mapped(VmaAllocation allocation, uint64_t size) { if (size) vmaFlushAllocation(vma, allocation, 0, size); } @@ -379,7 +534,36 @@ bool vma_piece_alloc(uint64_t size, uint64_t alignment, uint32_t type_bits, VmaA if (dev.memory_budget) ai.flags = VMA_ALLOCATION_CREATE_WITHIN_BUDGET_BIT; VmaAllocationInfo info{}; const auto t0 = std::chrono::steady_clock::now(); - const bool ok = vmaAllocateMemory(vma, &req, &ai, &allocation, &info) == VK_SUCCESS; + // (Triforce, 11-12) from a pool of their own with a block made ahead (above), where one could be made for this memory type; + // a piece larger than half a block, or one the pool has no room for and no block ahead, as before + static VmaPool piece_pool = VK_NULL_HANDLE; + static uint32_t piece_type = UINT32_MAX; + static bool piece_pool_tried = false; + constexpr uint64_t kPieceBlock = 64ull << 20; + if (!piece_pool_tried && ahead_on()) { + piece_pool_tried = true; + if (vmaFindMemoryTypeIndex(vma, type_bits, &ai, &piece_type) == VK_SUCCESS) { + VmaPoolCreateInfo pi{}; + pi.memoryTypeIndex = piece_type; + pi.blockSize = kPieceBlock; + if (vmaCreatePool(vma, &pi, &piece_pool) != VK_SUCCESS) piece_pool = VK_NULL_HANDLE; + } + ahead_pieces.block = kPieceBlock, ahead_pieces.what = "the mirrors' pieces"; + } + bool ok = false; + if (piece_pool && piece_type < 32 && (type_bits >> piece_type & 1) && size <= kPieceBlock / 2) { + VmaAllocationCreateInfo nai = ai; + nai.pool = piece_pool; + nai.flags |= VMA_ALLOCATION_CREATE_NEVER_ALLOCATE_BIT; + ok = vmaAllocateMemory(vma, &req, &nai, &allocation, &info) == VK_SUCCESS; + if (ok) ahead_ask(piece_pool, ahead_pieces); + else if (ahead_take(piece_pool, ahead_pieces)) ok = vmaAllocateMemory(vma, &req, &nai, &allocation, &info) == VK_SUCCESS; + if (!ok) { // (no block ahead: the pool takes a new one now, as the allocator's own blocks did) + nai.flags &= ~(VmaAllocationCreateFlags)VMA_ALLOCATION_CREATE_NEVER_ALLOCATE_BIT; + ok = vmaAllocateMemory(vma, &req, &nai, &allocation, &info) == VK_SUCCESS; + } + } + if (!ok) ok = vmaAllocateMemory(vma, &req, &ai, &allocation, &info) == VK_SUCCESS; { // (Triforce, 11-11: one that takes over 3 ms is named, 40 times: at the driver's limit an image's new block took // 110-170 ms, and submissions beside those were 130-150 ms of "memory sync" with nothing said of them) const double ms = std::chrono::duration(std::chrono::steady_clock::now() - t0).count(); @@ -439,6 +623,8 @@ bool vram_usage(uint64_t* used, uint64_t* budget, uint64_t* block_free) { // (a test: RX_VK_TEST_ALLOWED_MB= reports what the driver allows as at most n MB, so a 16 GB card plays the eviction and // keep-rendered paths an 8 GB card takes when its VRAM fills (the rainbow textures of issue 993 were never seen here otherwise)) static const uint64_t test_allowed = getenv("RX_VK_TEST_ALLOWED_MB") ? (uint64_t)atoi(getenv("RX_VK_TEST_ALLOWED_MB")) << 20 : 0; + if (test_card_ballast) u -= std::min(u, test_card_ballast), t -= std::min(t, test_card_ballast); // (RX_VK_TEST_CARD_MB) + u -= std::min(u, ahead_held_bytes()); // (Triforce: the blocks made ahead are nobody's until something needs them) if (test_allowed && t > test_allowed) t = test_allowed; *used = u, *budget = t; if (block_free) *block_free = f; diff --git a/gpu/vk/vk_work.h b/gpu/vk/vk_work.h index 215efd7..e9e31f3 100644 --- a/gpu/vk/vk_work.h +++ b/gpu/vk/vk_work.h @@ -7,6 +7,7 @@ #include #include "vk_loader.h" +#include "vk_recorder.h" // (RX_VK_RECORDER: each macro below calls its packer, which calls the driver directly when the recorder is off) namespace mkwvk { extern std::atomic work_epoch; // (vk_texture.cpp) @@ -17,30 +18,119 @@ inline void note_work() { } } // namespace mkwvk -#define vkCmdPipelineBarrier2(...) (::mkwvk::note_work(), ::vkx::vkCmdPipelineBarrier2(__VA_ARGS__)) -#define vkCmdBeginRendering(...) (::mkwvk::note_work(), ::vkx::vkCmdBeginRendering(__VA_ARGS__)) -#define vkCmdEndRendering(...) (::mkwvk::note_work(), ::vkx::vkCmdEndRendering(__VA_ARGS__)) -#define vkCmdDraw(...) (::mkwvk::note_work(), ::vkx::vkCmdDraw(__VA_ARGS__)) -#define vkCmdDrawIndexed(...) (::mkwvk::note_work(), ::vkx::vkCmdDrawIndexed(__VA_ARGS__)) -#define vkCmdDrawIndirect(...) (::mkwvk::note_work(), ::vkx::vkCmdDrawIndirect(__VA_ARGS__)) -#define vkCmdDrawIndexedIndirect(...) (::mkwvk::note_work(), ::vkx::vkCmdDrawIndexedIndirect(__VA_ARGS__)) -#define vkCmdDrawIndirectCount(...) (::mkwvk::note_work(), ::vkx::vkCmdDrawIndirectCount(__VA_ARGS__)) -#define vkCmdDrawIndexedIndirectCount(...) (::mkwvk::note_work(), ::vkx::vkCmdDrawIndexedIndirectCount(__VA_ARGS__)) -#define vkCmdDrawMeshTasksEXT(...) (::mkwvk::note_work(), ::vkx::vkCmdDrawMeshTasksEXT(__VA_ARGS__)) -#define vkCmdDrawMeshTasksIndirectEXT(...) (::mkwvk::note_work(), ::vkx::vkCmdDrawMeshTasksIndirectEXT(__VA_ARGS__)) -#define vkCmdDrawMeshTasksIndirectCountEXT(...) (::mkwvk::note_work(), ::vkx::vkCmdDrawMeshTasksIndirectCountEXT(__VA_ARGS__)) -#define vkCmdDispatch(...) (::mkwvk::note_work(), ::vkx::vkCmdDispatch(__VA_ARGS__)) -#define vkCmdDispatchIndirect(...) (::mkwvk::note_work(), ::vkx::vkCmdDispatchIndirect(__VA_ARGS__)) -#define vkCmdCopyBuffer(...) (::mkwvk::note_work(), ::vkx::vkCmdCopyBuffer(__VA_ARGS__)) -#define vkCmdCopyBufferToImage(...) (::mkwvk::note_work(), ::vkx::vkCmdCopyBufferToImage(__VA_ARGS__)) -#define vkCmdCopyImageToBuffer(...) (::mkwvk::note_work(), ::vkx::vkCmdCopyImageToBuffer(__VA_ARGS__)) -#define vkCmdCopyImage(...) (::mkwvk::note_work(), ::vkx::vkCmdCopyImage(__VA_ARGS__)) -#define vkCmdBlitImage(...) (::mkwvk::note_work(), ::vkx::vkCmdBlitImage(__VA_ARGS__)) -#define vkCmdResolveImage(...) (::mkwvk::note_work(), ::vkx::vkCmdResolveImage(__VA_ARGS__)) -#define vkCmdClearColorImage(...) (::mkwvk::note_work(), ::vkx::vkCmdClearColorImage(__VA_ARGS__)) -#define vkCmdClearDepthStencilImage(...) (::mkwvk::note_work(), ::vkx::vkCmdClearDepthStencilImage(__VA_ARGS__)) -#define vkCmdClearAttachments(...) (::mkwvk::note_work(), ::vkx::vkCmdClearAttachments(__VA_ARGS__)) -#define vkCmdFillBuffer(...) (::mkwvk::note_work(), ::vkx::vkCmdFillBuffer(__VA_ARGS__)) -#define vkCmdUpdateBuffer(...) (::mkwvk::note_work(), ::vkx::vkCmdUpdateBuffer(__VA_ARGS__)) -#define vkCmdBeginQuery(...) (::mkwvk::note_work(), ::vkx::vkCmdBeginQuery(__VA_ARGS__)) -#define vkCmdEndQuery(...) (::mkwvk::note_work(), ::vkx::vkCmdEndQuery(__VA_ARGS__)) +#define vkCmdPipelineBarrier2(...) (::mkwvk::note_work(), ::mkwvk::rec::rx_vkCmdPipelineBarrier2(__VA_ARGS__)) +#define vkCmdBeginRendering(...) (::mkwvk::note_work(), ::mkwvk::rec::rx_vkCmdBeginRendering(__VA_ARGS__)) +#define vkCmdEndRendering(...) (::mkwvk::note_work(), ::mkwvk::rec::rx_vkCmdEndRendering(__VA_ARGS__)) +#define vkCmdDraw(...) (::mkwvk::note_work(), ::mkwvk::rec::rx_vkCmdDraw(__VA_ARGS__)) +#define vkCmdDrawIndexed(...) (::mkwvk::note_work(), ::mkwvk::rec::rx_vkCmdDrawIndexed(__VA_ARGS__)) +#define vkCmdDrawIndirect(...) (::mkwvk::note_work(), ::mkwvk::rec::rx_vkCmdDrawIndirect(__VA_ARGS__)) +#define vkCmdDrawIndexedIndirect(...) (::mkwvk::note_work(), ::mkwvk::rec::rx_vkCmdDrawIndexedIndirect(__VA_ARGS__)) +#define vkCmdDrawIndirectCount(...) (::mkwvk::note_work(), ::mkwvk::rec::rx_vkCmdDrawIndirectCount(__VA_ARGS__)) +#define vkCmdDrawIndexedIndirectCount(...) (::mkwvk::note_work(), ::mkwvk::rec::rx_vkCmdDrawIndexedIndirectCount(__VA_ARGS__)) +#define vkCmdDrawMeshTasksEXT(...) (::mkwvk::note_work(), ::mkwvk::rec::rx_vkCmdDrawMeshTasksEXT(__VA_ARGS__)) +#define vkCmdDrawMeshTasksIndirectEXT(...) (::mkwvk::note_work(), ::mkwvk::rec::rx_vkCmdDrawMeshTasksIndirectEXT(__VA_ARGS__)) +#define vkCmdDrawMeshTasksIndirectCountEXT(...) (::mkwvk::note_work(), ::mkwvk::rec::rx_vkCmdDrawMeshTasksIndirectCountEXT(__VA_ARGS__)) +#define vkCmdDispatch(...) (::mkwvk::note_work(), ::mkwvk::rec::rx_vkCmdDispatch(__VA_ARGS__)) +#define vkCmdDispatchIndirect(...) (::mkwvk::note_work(), ::mkwvk::rec::rx_vkCmdDispatchIndirect(__VA_ARGS__)) +#define vkCmdCopyBuffer(...) (::mkwvk::note_work(), ::mkwvk::rec::rx_vkCmdCopyBuffer(__VA_ARGS__)) +#define vkCmdCopyBufferToImage(...) (::mkwvk::note_work(), ::mkwvk::rec::rx_vkCmdCopyBufferToImage(__VA_ARGS__)) +#define vkCmdCopyImageToBuffer(...) (::mkwvk::note_work(), ::mkwvk::rec::rx_vkCmdCopyImageToBuffer(__VA_ARGS__)) +#define vkCmdCopyImage(...) (::mkwvk::note_work(), ::mkwvk::rec::rx_vkCmdCopyImage(__VA_ARGS__)) +#define vkCmdBlitImage(...) (::mkwvk::note_work(), ::mkwvk::rec::rx_vkCmdBlitImage(__VA_ARGS__)) +#define vkCmdResolveImage(...) (::mkwvk::note_work(), ::mkwvk::rec::rx_vkCmdResolveImage(__VA_ARGS__)) +#define vkCmdClearColorImage(...) (::mkwvk::note_work(), ::mkwvk::rec::rx_vkCmdClearColorImage(__VA_ARGS__)) +#define vkCmdClearDepthStencilImage(...) (::mkwvk::note_work(), ::mkwvk::rec::rx_vkCmdClearDepthStencilImage(__VA_ARGS__)) +#define vkCmdClearAttachments(...) (::mkwvk::note_work(), ::mkwvk::rec::rx_vkCmdClearAttachments(__VA_ARGS__)) +#define vkCmdFillBuffer(...) (::mkwvk::note_work(), ::mkwvk::rec::rx_vkCmdFillBuffer(__VA_ARGS__)) +#define vkCmdUpdateBuffer(...) (::mkwvk::note_work(), ::mkwvk::rec::rx_vkCmdUpdateBuffer(__VA_ARGS__)) +#define vkCmdBeginQuery(...) (::mkwvk::note_work(), ::mkwvk::rec::rx_vkCmdBeginQuery(__VA_ARGS__)) +#define vkCmdEndQuery(...) (::mkwvk::note_work(), ::mkwvk::rec::rx_vkCmdEndQuery(__VA_ARGS__)) + +// (RX_VK_RECORDER) the calls that set state, bind or mark: not work (no count), but in the command buffer in order with the work, so +// through the recorder too +#define vkCmdBindPipeline(...) (::mkwvk::rec::rx_vkCmdBindPipeline(__VA_ARGS__)) +#define vkCmdBindDescriptorSets(...) (::mkwvk::rec::rx_vkCmdBindDescriptorSets(__VA_ARGS__)) +#define vkCmdBindIndexBuffer(...) (::mkwvk::rec::rx_vkCmdBindIndexBuffer(__VA_ARGS__)) +#define vkCmdBindVertexBuffers2(...) (::mkwvk::rec::rx_vkCmdBindVertexBuffers2(__VA_ARGS__)) +#define vkCmdPushConstants(...) (::mkwvk::rec::rx_vkCmdPushConstants(__VA_ARGS__)) +#define vkCmdPushDescriptorSetKHR(...) (::mkwvk::rec::rx_vkCmdPushDescriptorSetKHR(__VA_ARGS__)) +#define vkCmdSetViewport(...) (::mkwvk::rec::rx_vkCmdSetViewport(__VA_ARGS__)) +#define vkCmdSetScissor(...) (::mkwvk::rec::rx_vkCmdSetScissor(__VA_ARGS__)) +#define vkCmdSetViewportWithCount(...) (::mkwvk::rec::rx_vkCmdSetViewportWithCount(__VA_ARGS__)) +#define vkCmdSetScissorWithCount(...) (::mkwvk::rec::rx_vkCmdSetScissorWithCount(__VA_ARGS__)) +#define vkCmdSetBlendConstants(...) (::mkwvk::rec::rx_vkCmdSetBlendConstants(__VA_ARGS__)) +#define vkCmdSetStencilReference(...) (::mkwvk::rec::rx_vkCmdSetStencilReference(__VA_ARGS__)) +#define vkCmdSetStencilCompareMask(...) (::mkwvk::rec::rx_vkCmdSetStencilCompareMask(__VA_ARGS__)) +#define vkCmdSetStencilWriteMask(...) (::mkwvk::rec::rx_vkCmdSetStencilWriteMask(__VA_ARGS__)) +#define vkCmdSetStencilOp(...) (::mkwvk::rec::rx_vkCmdSetStencilOp(__VA_ARGS__)) +#define vkCmdSetDepthBias(...) (::mkwvk::rec::rx_vkCmdSetDepthBias(__VA_ARGS__)) +#define vkCmdSetDepthBiasEnable(...) (::mkwvk::rec::rx_vkCmdSetDepthBiasEnable(__VA_ARGS__)) +#define vkCmdSetLineWidth(...) (::mkwvk::rec::rx_vkCmdSetLineWidth(__VA_ARGS__)) +#define vkCmdSetDepthBoundsTestEnable(...) (::mkwvk::rec::rx_vkCmdSetDepthBoundsTestEnable(__VA_ARGS__)) +#define vkCmdSetCullMode(...) (::mkwvk::rec::rx_vkCmdSetCullMode(__VA_ARGS__)) +#define vkCmdSetFrontFace(...) (::mkwvk::rec::rx_vkCmdSetFrontFace(__VA_ARGS__)) +#define vkCmdSetPrimitiveTopology(...) (::mkwvk::rec::rx_vkCmdSetPrimitiveTopology(__VA_ARGS__)) +#define vkCmdSetPrimitiveRestartEnable(...) (::mkwvk::rec::rx_vkCmdSetPrimitiveRestartEnable(__VA_ARGS__)) +#define vkCmdSetDepthTestEnable(...) (::mkwvk::rec::rx_vkCmdSetDepthTestEnable(__VA_ARGS__)) +#define vkCmdSetDepthWriteEnable(...) (::mkwvk::rec::rx_vkCmdSetDepthWriteEnable(__VA_ARGS__)) +#define vkCmdSetDepthCompareOp(...) (::mkwvk::rec::rx_vkCmdSetDepthCompareOp(__VA_ARGS__)) +#define vkCmdSetStencilTestEnable(...) (::mkwvk::rec::rx_vkCmdSetStencilTestEnable(__VA_ARGS__)) +#define vkCmdSetRasterizerDiscardEnable(...) (::mkwvk::rec::rx_vkCmdSetRasterizerDiscardEnable(__VA_ARGS__)) +#define vkCmdSetPolygonModeEXT(...) (::mkwvk::rec::rx_vkCmdSetPolygonModeEXT(__VA_ARGS__)) +#define vkCmdSetDepthClampEnableEXT(...) (::mkwvk::rec::rx_vkCmdSetDepthClampEnableEXT(__VA_ARGS__)) +#define vkCmdSetAlphaToCoverageEnableEXT(...) (::mkwvk::rec::rx_vkCmdSetAlphaToCoverageEnableEXT(__VA_ARGS__)) +#define vkCmdSetColorBlendEnableEXT(...) (::mkwvk::rec::rx_vkCmdSetColorBlendEnableEXT(__VA_ARGS__)) +#define vkCmdSetColorBlendEquationEXT(...) (::mkwvk::rec::rx_vkCmdSetColorBlendEquationEXT(__VA_ARGS__)) +#define vkCmdSetColorWriteMaskEXT(...) (::mkwvk::rec::rx_vkCmdSetColorWriteMaskEXT(__VA_ARGS__)) +#define vkCmdWriteBufferMarkerAMD(...) (::mkwvk::rec::rx_vkCmdWriteBufferMarkerAMD(__VA_ARGS__)) +#define vkCmdSetCheckpointNV(...) (::mkwvk::rec::rx_vkCmdSetCheckpointNV(__VA_ARGS__)) +#define vkCmdWriteTimestamp2(...) (::mkwvk::rec::rx_vkCmdWriteTimestamp2(__VA_ARGS__)) +#define vkCmdResetQueryPool(...) (::mkwvk::rec::rx_vkCmdResetQueryPool(__VA_ARGS__)) + +// (RX_VK_RECORDER=defer) the calls that begin, end or reset command buffers, submit, present or wait, read back what the GPU did, +// or let go of what recorded commands use: the thread's held packets reach the driver first (rec::boundary(): a relaxed load and +// a branch unless deferred; RX_VK_RECORDER=thread: the replay lock's holder waits for the recorder thread to run all it put in its ring; +// the kind is for the recorder's stats line). RX_REC_BOUNDARIES in vk_recorder.h lists them +#define vkBeginCommandBuffer(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkBeginCommandBuffer), ::vkx::vkBeginCommandBuffer(__VA_ARGS__)) +#define vkEndCommandBuffer(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkEndCommandBuffer), ::vkx::vkEndCommandBuffer(__VA_ARGS__)) +#define vkResetCommandPool(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkResetCommandPool), ::vkx::vkResetCommandPool(__VA_ARGS__)) +#define vkDestroyCommandPool(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkDestroyCommandPool), ::vkx::vkDestroyCommandPool(__VA_ARGS__)) +#define vkQueueSubmit2(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkQueueSubmit2), ::vkx::vkQueueSubmit2(__VA_ARGS__)) +#define vkQueueWaitIdle(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkQueueWaitIdle), ::vkx::vkQueueWaitIdle(__VA_ARGS__)) +#define vkQueueBindSparse(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkQueueBindSparse), ::vkx::vkQueueBindSparse(__VA_ARGS__)) +#define vkQueuePresentKHR(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkQueuePresentKHR), ::vkx::vkQueuePresentKHR(__VA_ARGS__)) +#define vkAcquireNextImageKHR(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkAcquireNextImageKHR), ::vkx::vkAcquireNextImageKHR(__VA_ARGS__)) +#define vkDeviceWaitIdle(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkDeviceWaitIdle), ::vkx::vkDeviceWaitIdle(__VA_ARGS__)) +#define vkWaitForFences(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkWaitForFences), ::vkx::vkWaitForFences(__VA_ARGS__)) +#define vkResetFences(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkResetFences), ::vkx::vkResetFences(__VA_ARGS__)) +#define vkWaitSemaphores(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkWaitSemaphores), ::vkx::vkWaitSemaphores(__VA_ARGS__)) +#define vkGetSemaphoreCounterValue(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkGetSemaphoreCounterValue), ::vkx::vkGetSemaphoreCounterValue(__VA_ARGS__)) +#define vkGetQueryPoolResults(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkGetQueryPoolResults), ::vkx::vkGetQueryPoolResults(__VA_ARGS__)) +#define vkResetQueryPool(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkResetQueryPool), ::vkx::vkResetQueryPool(__VA_ARGS__)) +#define vkGetQueueCheckpointDataNV(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkGetQueueCheckpointDataNV), ::vkx::vkGetQueueCheckpointDataNV(__VA_ARGS__)) +#define vkGetDeviceFaultInfoEXT(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkGetDeviceFaultInfoEXT), ::vkx::vkGetDeviceFaultInfoEXT(__VA_ARGS__)) +#define vkResetDescriptorPool(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkResetDescriptorPool), ::vkx::vkResetDescriptorPool(__VA_ARGS__)) +#define vkDestroyBuffer(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkDestroyBuffer), ::vkx::vkDestroyBuffer(__VA_ARGS__)) +#define vkDestroyImage(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkDestroyImage), ::vkx::vkDestroyImage(__VA_ARGS__)) +#define vkDestroyImageView(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkDestroyImageView), ::vkx::vkDestroyImageView(__VA_ARGS__)) +#define vkDestroyBufferView(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkDestroyBufferView), ::vkx::vkDestroyBufferView(__VA_ARGS__)) +#define vkDestroySampler(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkDestroySampler), ::vkx::vkDestroySampler(__VA_ARGS__)) +#define vkDestroyPipeline(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkDestroyPipeline), ::vkx::vkDestroyPipeline(__VA_ARGS__)) +#define vkDestroyPipelineLayout(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkDestroyPipelineLayout), ::vkx::vkDestroyPipelineLayout(__VA_ARGS__)) +#define vkDestroyDescriptorSetLayout(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkDestroyDescriptorSetLayout), ::vkx::vkDestroyDescriptorSetLayout(__VA_ARGS__)) +#define vkDestroyDescriptorPool(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkDestroyDescriptorPool), ::vkx::vkDestroyDescriptorPool(__VA_ARGS__)) +#define vkDestroyQueryPool(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkDestroyQueryPool), ::vkx::vkDestroyQueryPool(__VA_ARGS__)) +#define vkDestroyShaderModule(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkDestroyShaderModule), ::vkx::vkDestroyShaderModule(__VA_ARGS__)) +#define vkDestroyPipelineCache(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkDestroyPipelineCache), ::vkx::vkDestroyPipelineCache(__VA_ARGS__)) +#define vkDestroyFence(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkDestroyFence), ::vkx::vkDestroyFence(__VA_ARGS__)) +#define vkDestroySemaphore(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkDestroySemaphore), ::vkx::vkDestroySemaphore(__VA_ARGS__)) +#define vkDestroySwapchainKHR(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkDestroySwapchainKHR), ::vkx::vkDestroySwapchainKHR(__VA_ARGS__)) +#define vkDestroySurfaceKHR(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkDestroySurfaceKHR), ::vkx::vkDestroySurfaceKHR(__VA_ARGS__)) +#define vkDestroyDevice(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkDestroyDevice), ::vkx::vkDestroyDevice(__VA_ARGS__)) +#define vkDestroyInstance(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkDestroyInstance), ::vkx::vkDestroyInstance(__VA_ARGS__)) +#define vkFreeMemory(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vkFreeMemory), ::vkx::vkFreeMemory(__VA_ARGS__)) +#define vmaDestroyBuffer(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vmaDestroyBuffer), ::vmaDestroyBuffer(__VA_ARGS__)) +#define vmaDestroyImage(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vmaDestroyImage), ::vmaDestroyImage(__VA_ARGS__)) +#define vmaInvalidateAllocation(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vmaInvalidateAllocation), ::vmaInvalidateAllocation(__VA_ARGS__)) +// (Triforce: the memory of a mirror's piece, which recorded copies and bindings use, and the blocks made ahead: vk_vma.cpp) +#define vmaFreeMemory(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_vmaFreeMemory), ::vmaFreeMemory(__VA_ARGS__)) diff --git a/gpu/vk/vram_compact.h b/gpu/vk/vram_compact.h index cd51e85..1c10188 100644 --- a/gpu/vk/vram_compact.h +++ b/gpu/vk/vram_compact.h @@ -1,4 +1,4 @@ -// (hookshot) The compaction's rules (RX_VK_COMPACT=1; vk_texture.cpp's compact_step and the texture budget), apart from Vulkan so +// (hookshot) The compaction's rules (on unless RX_VK_COMPACT=0; vk_texture.cpp's compact_step and the texture budget), apart from Vulkan so // that tests/test_vram_compact.py can run them. // // The driver counts VMA's blocks whole. An evicted texture's memory stays inside its 128 MB block, so the count keeps what the @@ -10,9 +10,14 @@ #include #include +#include namespace vram_compact { +/// Whether the textures are compacted, by RX_VK_COMPACT as the environment has it: on when it is not set, off for 0 (or empty). +/// It was opt-in (RX_VK_COMPACT=1) while its hold could let go of textures in use; that is what hold_keeps and hold_met are for +inline bool enabled(const char* env) { return !env || atoi(env) != 0; } + /// The driver's count over 90% of what it allows: a run starts for less free space, and the next one sooner inline bool pressed(uint64_t used, uint64_t allowed) { return allowed && used > allowed / 10 * 9; } @@ -38,4 +43,54 @@ inline uint64_t held_budget(uint64_t budget, uint64_t total, uint64_t used, uint return std::min(budget, total > over ? total - over : 0); } +/// What the hold lets go of has gone unused for this many presents (ten seconds at 60 fps). The eviction's own 60 is right for a +/// budget that is a size: over it, something has to go. The hold's line is crossed for as long as the rest of the game's VRAM +/// leaves the scene's textures no room under it, and then every texture out of view for a second went and came back when the +/// camera turned: with the count held under a 7 GB allowance, a turning camera in Hyrule Field at DLAA let go of 2.3-2.6 GB of live +/// textures every 600 presents and uploaded 6.8 GB in 50 seconds, against 1.4 GB with the compaction off +inline constexpr uint64_t kHoldUnused = 600; + +/// Whether the hold's pass keeps a texture no render target has drawn over: one used in the last kHoldUnused presents +inline bool hold_keeps(uint64_t unused) { return unused < kHoldUnused; } + +/// The reserve: a second line, kReservePoints points of the allowance under the hold's own (80 for 85). Over it the hold lets go +/// of textures unused for kReserveUnused presents (30 seconds at 60 fps), down to that line. With one line the cache sat at it, +/// and whatever came in next went over: in a walk across Hyrule Field at DLAA with 768 MB of the card held by something else the +/// count was over 85% of the allowance for 47-68% of the time and over 87% for 7-26% (peaks of 91-93%), and in play with a +/// recorder running the GPU's frame went from 11 to 15-23 ms for ten seconds each time it reached 87-90%. With the reserve: +/// over 85% for 5%, over 87% for 0-1%, peaks of 87-88%, no more uploads +inline constexpr uint64_t kReservePoints = 5; +inline constexpr uint64_t kReserveUnused = 1800; +inline uint64_t reserve_pct(uint64_t pct) { return pct > kReservePoints ? pct - kReservePoints : 0; } + +/// What the rules that are not the hold's (the budget's size, the 90% rule, the driver's limit at 97%) ask an eviction to go down +/// to on their own: 7/8 of their budget once the textures hold more than it, as a pass over the budget cuts with the compaction +/// off; UINT64_MAX: nothing +inline uint64_t rule_target(uint64_t total, uint64_t rule_budget) { return total > rule_budget ? rule_budget / 8 * 7 : UINT64_MAX; } + +/// What a live texture unused for that many presents may be let go down to in the hold's pass, the lowest of the lines that +/// apply to it; UINT64_MAX: it stays. Three tiers: +/// - unused for need_unused (the eviction's own 60 presents): what the other rules ask for (rule_to). Over 90% of the allowance +/// or over the budget's size something has to go now, and the hold's patience is not theirs: with the hold's two thresholds +/// alone, a pass the 90% rule asked for took nothing used in the last ten seconds, and the reserve's lower budget made every +/// such pass a hold's pass, on the cards whose count sits past 90% +/// - unused for kHoldUnused: the hold's own line too (hold_to, 0 while it is not crossed) +/// - unused for kReserveUnused: and the reserve's (reserve_to) +inline uint64_t hold_target(uint64_t unused, uint64_t need_unused, uint64_t rule_to, uint64_t hold_to, uint64_t reserve_to) { + uint64_t to = unused >= need_unused ? rule_to : UINT64_MAX; + if (!hold_keeps(unused) && hold_to) to = std::min(to, hold_to); + if (unused >= kReserveUnused) to = std::min(to, reserve_to); + return to; +} + +/// Whether the hold's pass is done with live textures: those it has chosen (live) and the dead render targets it lets go in any +/// case (dead) bring what the textures hold (total) down to evict_to. The dead ones count first: they are the youngest of the +/// textures, and a pass that stopped at the first texture past the excess, oldest first, never reached them, so while the hold +/// was on, live textures went in their place and the dead targets piled up (0.9 to 3.5 GB of them in 120 presents as a new +/// stretch of Hyrule Field came in, the count over what the driver allows, frames of 60-128 ms for three seconds) +inline bool hold_met(uint64_t total, uint64_t live, uint64_t dead, uint64_t evict_to) { + const uint64_t going = live + dead; + return going >= total || total - going <= evict_to; +} + } // namespace vram_compact diff --git a/host/nn_audio.c b/host/nn_audio.c index f05d318..e16a049 100644 --- a/host/nn_audio.c +++ b/host/nn_audio.c @@ -601,6 +601,27 @@ static void au_device_name(unsigned device, char *out, size_t size) { #endif } static uint64_t au_ms(void) { return (uint64_t)GetTickCount64(); } +/* (hookshot) Which output Windows has as its default now (the waveOut mapper's preferred device, its number and name): + a key that changes when the default does; 0 when unknown. A device opened as the default stays on that device when + Windows moves its default (a Bluetooth headset gone and back), so the output section asks every second */ +#ifndef DRVM_MAPPER_PREFERRED_GET +#define DRVM_MAPPER_PREFERRED_GET (0x2000 + 21) /* (mmddk.h: DRVM_MAPPER + 21) */ +#endif +static uint32_t au_default_key(void) { +#ifdef _WIN32 + DWORD id = (DWORD)-1, flags = 0; + if (waveOutMessage((HWAVEOUT)(UINT_PTR)WAVE_MAPPER, DRVM_MAPPER_PREFERRED_GET, (DWORD_PTR)&id, (DWORD_PTR)&flags) != + MMSYSERR_NOERROR || id == (DWORD)-1) + return 0; + char name[64]; + au_device_name(id, name, sizeof name); + uint32_t h = 2166136261u ^ id; + for (const char *p = name; *p; p++) h = (h ^ (uint8_t)*p) * 16777619u; + return h ? h : 1; +#else + return 0; /* (linux: SDL follows the default device itself) */ +#endif +} /* HOS_AUDIO_MUTE=1 (test runs): the device opened and fed as for a player, at volume 0 */ static void au_opened(HWAVEOUT w) { #ifdef _WIN32 @@ -735,6 +756,15 @@ static void output_error(const char *operation, MMRESULT result) { static int out_next; /* the next candidate: 0 = HOS_AUDIO_DEVICE, 1 = the default, 2.. = device n-2 */ static uint64_t out_opened, out_stall_since; static unsigned out_played, out_peak, out_reported; +/* (hookshot) A device gone (a Bluetooth headset off, the audio service restarted: "no driver", "invalid handle" from the + next call) is closed and the default opened again; none opening, the default is tried every second until one does. + Opened as the default, the output follows Windows': its key (au_default_key) looked at every second, a change reopens + there. Before, a call that failed this way was only logged, the same block tried again for good, and the game had no + sound for the rest of the run, on any device (RX 5500M laptop, a WH-CH720N headset) */ +static int out_retry, out_reopens; +static uint64_t out_retry_at, out_default_seen; +static uint32_t out_default_key; /* (0: not opened as the default, or Windows did not say which it is) */ +static int au_device_gone(MMRESULT r) { return r == 2 || r == 5 || r == 6; } /* BADDEVICEID, INVALHANDLE, NODRIVER */ static int output_try(int candidate, const WAVEFORMATEX *f) { const char *want = getenv("HOS_AUDIO_DEVICE"); unsigned device; @@ -749,6 +779,8 @@ static int output_try(int candidate, const WAVEFORMATEX *f) { return 0; } au_opened(wout); + out_default_key = device == WAVE_MAPPER ? au_default_key() : 0; + out_default_seen = au_ms(); char name[64]; if (device == WAVE_MAPPER) snprintf(name, sizeof name, "the default device"); else au_device_name(device, name, sizeof name); @@ -790,6 +822,7 @@ static void output_open(void) { channels = 2; } out_channels = channels; + out_retry = !wout, out_retry_at = au_ms(); /* (none opened: tried again in a second, output_frame) */ if (!wout) return; for (int i = 0; i < NBLOCKS; i++) { memset(&hdr[i], 0, sizeof hdr[i]); @@ -802,18 +835,30 @@ static void output_open(void) { out_stall_since = 0; out_played = out_peak = out_reported = 0; } -static void output_reopen(void) { - static int reopens; - out_stall_since = 0; - if (++reopens > 6) return; - hos_log("audio: the output device took no sound for 1.5 s: trying the next one"); +static void output_close(void) { /* (what fails here is not looked at: a device gone takes none of these calls) */ waveOutReset(wout); for (int i = 0; i < NBLOCKS; i++) if (hdr[i].dwFlags & WHDR_PREPARED) waveOutUnprepareHeader(wout, &hdr[i], sizeof hdr[i]); waveOutClose(wout); wout = NULL; +} +static void output_reopen(void) { /* (a device that takes nothing: the next candidate, 6 times; again after 30 s of sound) */ + out_stall_since = 0; + if (++out_reopens > 6) return; + hos_log("audio: the output device took no sound for 1.5 s: trying the next one"); + output_close(); output_open(); } +static void output_open_default(void) { /* (HOS_AUDIO_DEVICE first when it is set, as at the start) */ + const char *want = getenv("HOS_AUDIO_DEVICE"); + out_next = want && *want ? 0 : 1; + output_open(); +} +static void output_move(const char *why) { + hos_log("audio: %s: the output opened again on the default device", why); + output_close(); + output_open_default(); +} static void dump_finish(void) { if (!dump) return; uint8_t h[44]; @@ -846,7 +891,19 @@ static void output_frame(const int16_t *pcm) { fseek(dump, at, SEEK_SET); } } - if (!wout) return; + if (!wout) { + if (!out_retry || au_ms() - out_retry_at < 1000) return; + output_open_default(); /* (the service back, a device plugged in) */ + if (!wout) return; + } + if (out_default_key && au_ms() - out_default_seen >= 1000) { + out_default_seen = au_ms(); + const uint32_t now = au_default_key(); + if (now && now != out_default_key) { + output_move("Windows' default output changed"); + if (!wout) return; + } + } for (int i = 0; i < FRAME * out_channels; i++) { const unsigned a = (unsigned)(pcm[i] < 0 ? -pcm[i] : pcm[i]); if (a > out_peak) out_peak = a; @@ -869,6 +926,7 @@ static void output_frame(const int16_t *pcm) { const MMRESULT result = waveOutUnprepareHeader(wout, h, sizeof *h); if (result != MMSYSERR_NOERROR) { output_error("waveOutUnprepareHeader", result); + if (au_device_gone(result)) output_move("the output device went away"); return; /* the driver has not released the header: do not change its samples or flags */ } } @@ -881,10 +939,17 @@ static void output_frame(const int16_t *pcm) { if (result == MMSYSERR_NOERROR) { operation = "waveOutWrite"; result = waveOutWrite(wout, h, sizeof *h); - if (result == MMSYSERR_NOERROR) out_played++; + if (result == MMSYSERR_NOERROR) { + out_played++; + if (out_reopens && au_ms() - out_opened > 30000) out_reopens = 0; /* (sound again: the stall rule's 6 anew) */ + } } if (result != MMSYSERR_NOERROR) { output_error(operation, result); + if (au_device_gone(result)) { + output_move("the output device went away"); + return; + } /* A failed submission has no completion to set DONE. Reuse only a host-owned block; a prepared header is unprepared on its next turn. Without this, returning to this slot drops every later audio frame. */ if (!(h->dwFlags & WHDR_INQUEUE)) h->dwFlags |= WHDR_DONE; diff --git a/host/nn_fs.c b/host/nn_fs.c index 25840fa..573b008 100644 --- a/host/nn_fs.c +++ b/host/nn_fs.c @@ -592,8 +592,12 @@ HLE(_ZN2nn2fs13OpenDirectoryEPNS0_15DirectoryHandleEPKci) { G64(ARG(0)) = new_handle(dh); RET(0); } +/* (hookshot) A directory's listing is read, counted and closed under one lock: a CloseDirectory on another guest thread freed + the handle (and its find handle) while a ReadDirectory or GetDirectoryEntryCount still walked it */ +static SRWLOCK dir_lock = SRWLOCK_INIT; /* ReadDirectory(long* count, DirectoryEntry* entries, DirectoryHandle, long max) */ HLE(_ZN2nn2fs13ReadDirectoryEPlPNS0_14DirectoryEntryENS0_15DirectoryHandleEl) { + AcquireSRWLockExclusive(&dir_lock); Handle *dh = get_handle(c, ARG(2)); int64_t n = 0, max = (int64_t)ARG(3); while (n < max && !dh->find_done) { @@ -620,11 +624,13 @@ HLE(_ZN2nn2fs13ReadDirectoryEPlPNS0_14DirectoryEntryENS0_15DirectoryHandleEl) { G64(e + 0x308) = x->size; n++; } + ReleaseSRWLockExclusive(&dir_lock); G64(ARG(0)) = (uint64_t)n; RET(0); } /* (hookshot) GetDirectoryEntryCount(long* count, DirectoryHandle): every entry the open mode lists, read or not */ HLE(_ZN2nn2fs22GetDirectoryEntryCountEPlNS0_15DirectoryHandleE) { + AcquireSRWLockExclusive(&dir_lock); Handle *dh = get_handle(c, ARG(1)); int64_t n = 0; WIN32_FIND_DATAA fd; @@ -640,13 +646,16 @@ HLE(_ZN2nn2fs22GetDirectoryEntryCountEPlNS0_15DirectoryHandleE) { } for (int i = 0; i < dh->nextras; i++) n += (dh->extras[i].is_dir && (dh->is_dir & 1)) || (!dh->extras[i].is_dir && (dh->is_dir & 2)); + ReleaseSRWLockExclusive(&dir_lock); G64(ARG(0)) = (uint64_t)n; RET(0); } HLE(_ZN2nn2fs14CloseDirectoryENS0_15DirectoryHandleE) { + AcquireSRWLockExclusive(&dir_lock); Handle *dh = get_handle(c, ARG(0)); if (dh->find != INVALID_HANDLE_VALUE) FindClose(dh->find); free_handle(ARG(0)); + ReleaseSRWLockExclusive(&dir_lock); free(dh->extras); free(dh); } diff --git a/host/nn_vi.c b/host/nn_vi.c index 5feeadd..a1d2b37 100644 --- a/host/nn_vi.c +++ b/host/nn_vi.c @@ -278,6 +278,21 @@ uint32_t hos_frame_whole(uint64_t at, uint32_t amount_bits) { return whole; } volatile uint64_t hos_res_module = 0; /* (f_831f3200's object, whose +0xe5c is the 3D's scale: nx2/oot_patches.py) */ +/* Quantize the time step to the engine's integer nominal rate. Keep debt from a short frame even when the minimum + step is 1/60: discarding it made alternating short/long frames advance more game time than elapsed wall time. + Keep at most four nominal frames of debt so unpaced fast presents cannot leave a long recovery. Steps beyond + 1/15 s are still dropped, rather than caught up after a long stall. */ +static int realtime_rate(double ratio, int maximum, double *carry) { + ratio += *carry; + if (ratio > 4.0) ratio = 4.0; + const double least = 60.0 / maximum; + const double bounded = ratio < least ? least : ratio; + int fps = (int)(60.0 / bounded + 0.5); + fps = fps < 15 ? 15 : fps > maximum ? maximum : fps; + *carry = ratio - 60.0 / fps; + if (*carry < -4.0) *carry = -4.0; + return fps; +} static void realtime_step(int64_t now) { int rx_option(int id, int otherwise); /* (Resolution 100% from the start once came out black (the dynamic block's fields unset), so the dynamic path ran 30 @@ -415,19 +430,17 @@ static void realtime_step(int64_t now) { __atomic_store_n(&hos_frame_seconds_bits, 0, __ATOMIC_RELAXED); } const int top = timed ? 60 : fps_max(); - double ratio = sum / count * 60.0 + carry; - const double least = 60.0 / top; - ratio = ratio < least ? least : ratio > 4.0 ? 4.0 : ratio; - int fps = (int)(60.0 / ratio + 0.5); - fps = fps < 15 ? 15 : fps > top ? top : fps; + const double ratio = sum / count * 60.0; int *slot = (int *)GP(mod + 0x2c8); if (*slot < 15 || *slot > fps_max()) { static int logged; if (!logged++) hos_log("vi: the graphics module's frame rate reads %d: no real-time step", *slot); return; } - carry = ratio - 60.0 / fps; - carry = carry > 0.5 ? 0.5 : carry < -0.5 ? -0.5 : carry; + /* The timed hook already supplies the actual frame time. Its nominal rate tracks this frame without debt. */ + double frame_carry = timed ? 0 : carry; + const int fps = realtime_rate(ratio, top, &frame_carry); + carry = timed ? 0 : frame_carry; *slot = fps; if (!(steps++ % 3000)) hos_log("vi: real-time step on: ratio %.3f, game's frame rate %d", ratio, fps); } diff --git a/tests/audio_output_errors_test.c b/tests/audio_output_errors_test.c index 5fb9b8d..33ef54c 100644 --- a/tests/audio_output_errors_test.c +++ b/tests/audio_output_errors_test.c @@ -26,6 +26,8 @@ static int prepare_calls, write_calls, accepted, unprepare_calls, open_calls, cl static unsigned last_device, last_channels; static uint64_t now_ms; static int fail_prepare, fail_write, fail_unprepare, fail_open, fail_open6; +static int gone_write_at, gone_unprepare, opens_fail; /* (the device gone: "no driver" 6, "invalid handle" 5) */ +static uint32_t default_key = 11; /* (Windows' default output, as au_default_key says) */ static int queued[10]; static WAVEHDR *known[10]; static char last_log[512]; @@ -52,6 +54,7 @@ static MMRESULT test_open(HWAVEOUT *out, unsigned device, const WAVEFORMATEX *f, CHECK(f->nBlockAlign == 2 * f->nChannels && f->nAvgBytesPerSec == 48000 * f->nBlockAlign); last_channels = f->nChannels; open_calls++; + if (opens_fail) return 6; /* (MMSYSERR_NODRIVER: the audio service down) */ if (fail_open || (fail_open6 && f->nChannels == 6)) return 7; /* (MMSYSERR_NOMEM; a device without 5.1: 32) */ *out = (void *)(uintptr_t)1; return 0; @@ -69,6 +72,7 @@ static MMRESULT test_write(HWAVEOUT out, WAVEHDR *h, unsigned size) { const int i = slot(h); CHECK(!queued[i]); write_calls++; + if (gone_write_at && write_calls == gone_write_at) return 6; /* (MMSYSERR_NODRIVER) */ if (!(h->dwFlags & WHDR_PREPARED)) return 34; /* WAVERR_UNPREPARED: no completion follows. */ if (fail_write && write_calls == 1) return 7; /* MMSYSERR_NOMEM: no buffer was queued. */ queued[i] = 1; @@ -80,6 +84,7 @@ static MMRESULT test_unprepare(HWAVEOUT out, WAVEHDR *h, unsigned size) { (void)out; (void)size; CHECK(!queued[slot(h)]); unprepare_calls++; + if (gone_unprepare) return 5; /* (MMSYSERR_INVALHANDLE, every call until the handle is closed) */ if (fail_unprepare && unprepare_calls == 1) return 33; /* WAVERR_STILLPLAYING: do not reuse. */ h->dwFlags &= ~WHDR_PREPARED; return 0; @@ -93,11 +98,12 @@ static MMRESULT test_reset(HWAVEOUT out) { } return 0; } -static MMRESULT test_close(HWAVEOUT out) { (void)out; close_calls++; return 0; } +static MMRESULT test_close(HWAVEOUT out) { (void)out; close_calls++; gone_unprepare = 0; return 0; } /* (the platform helpers nn_audio.c defines before its output section) */ static unsigned au_device_count(void) { return 2; } static void au_device_name(unsigned device, char *out, size_t size) { snprintf(out, size, "test device %u", device); } static uint64_t au_ms(void) { return now_ms; } +static uint32_t au_default_key(void) { return default_key; } static void au_opened(HWAVEOUT w) { (void)w; } static int speakers = 2; /* (the game's speakers: 6 in the "surround" and "fallback" cases) */ static int au_speakers(void) { return speakers; } @@ -158,6 +164,52 @@ int main(int argc, char **argv) { CHECK(accepted == 20 && prepare_calls == 20 && write_calls == 20 && !logs); CHECK(blocks[9][0] == 38 * 31 && blocks[9][FRAME * 6] == 39 * 31); CHECK(blocks[9][FRAME * 6 - 1] == (int16_t)(38 * 31 + FRAME * 6 - 1)); + } else if (!strcmp(scenario, "lost")) { + gone_write_at = 25; /* a Bluetooth headset off: its 25th block, "no driver" */ + frames(60, 1); + CHECK(reset_calls == 1 && close_calls == 1 && open_calls == 2 && last_device == WAVE_MAPPER); + CHECK(accepted >= 28); /* the default took the blocks after it */ + } else if (!strcmp(scenario, "gone")) { + frames(20, 1); + gone_unprepare = 1; /* the audio service restarted: every call on the old handle "invalid" */ + frames(2, 1); + CHECK(close_calls == 1 && open_calls == 2 && last_device == WAVE_MAPPER); + const int before = accepted; + frames(10, 1); + CHECK(accepted == before + 5); /* (before, the same block tried again for good: no sound for the run) */ + } else if (!strcmp(scenario, "service")) { + frames(10, 1); + opens_fail = 1; /* the service down: no device opens */ + gone_write_at = write_calls + 1; + frames(2, 1); + CHECK(close_calls == 1 && !wout && open_calls == 4); /* (the first open, then the default and both devices) */ + frames(5, 1); + CHECK(open_calls == 4); /* not again within the second */ + now_ms += 1000; + frames(1, 1); + CHECK(open_calls == 7 && !wout); /* a second later: all three again, the service still down */ + opens_fail = 0; + now_ms += 1000; + frames(1, 1); + CHECK(wout && open_calls == 8 && last_device == WAVE_MAPPER); /* the service back: the default first */ + const int before = accepted; + frames(4, 1); + CHECK(accepted == before + 2); + } else if (!strcmp(scenario, "default")) { + frames(10, 1); + default_key = 12; /* Windows moves its default (the headset back) */ + now_ms += 500; + frames(1, 1); + CHECK(open_calls == 1); /* looked at once a second */ + now_ms += 600; + frames(1, 1); + CHECK(close_calls == 1 && open_calls == 2 && last_device == WAVE_MAPPER && out_default_key == 12); + now_ms += 2000; + frames(2, 1); + CHECK(open_calls == 2); /* the same default: not opened again */ + const int before = accepted; + frames(4, 1); + CHECK(accepted == before + 2); } else if (!strcmp(scenario, "stall")) { frames(20, 0); /* every block queued, none ever completes: the device takes nothing */ now_ms += 1000; diff --git a/tests/recorder_exit_test.cpp b/tests/recorder_exit_test.cpp new file mode 100644 index 0000000..21aa826 --- /dev/null +++ b/tests/recorder_exit_test.cpp @@ -0,0 +1,92 @@ +// RX_VK_RECORDER=thread at process exit, on Windows (tests/test_recorder_sync.py builds and runs it; skipped elsewhere). +// +// Built twice: with RX_EXIT_DLL, a DLL holding the real vk_work.h / vk_recorder.h (as hookshot_vk.dll does) and a fake vkCmdDraw; +// without, an exe that loads it, has the recorder parked in idle() holding sleep_mu (the test hook RX_REC_TEST_HOOKS), and calls +// ExitProcess, as the window's close (host/display.c) and svcExitProcess do. Windows ends the recorder thread first, then runs the +// DLL's global destructors. A destructor that takes sleep_mu, now owned by a dead thread, either hangs (MinGW-w64 GCC and +// clang with libstdc++; measured on Windows 11) or has the process ended on the spot, the DLL's other destructors skipped +// (llvm-mingw's libc++, clang for the MSVC target). So: the exe must end with code 0 within the watchdog's 20 s (it ends only +// this child), and the sentinel below, destroyed after the recorder's global, must print its line. +#ifdef RX_EXIT_DLL +#define RX_REC_TEST_HOOKS 1 +#include +#include +#include +#include +#include +#include + +extern "C" __declspec(dllimport) void* __stdcall GetStdHandle(unsigned long); +extern "C" __declspec(dllimport) int __stdcall WriteFile(void*, const void*, unsigned long, unsigned long*, void*); +/// (made before the recorder's global, so destroyed after it: its line shows the DLL's destructors all ran. The recorder's +/// global is an inline variable, whose dynamic initialization is only partially ordered against this file's statics: with +/// llvm-mingw it was made first, so this line printed before the old destructor ended the process and the broken variant +/// passed. init_priority makes this object first on every compiler that builds the test) +struct Sentinel { + ~Sentinel() { + static const char line[] = "DLL detach: the global destructors ran to the end\n"; + unsigned long n; + WriteFile(GetStdHandle((unsigned long)-11), line, sizeof line - 1, &n, nullptr); + } +}; +static Sentinel g_sentinel __attribute__((init_priority(101))); + +#include "vk_work.h" + +namespace mkwvk { +std::atomic work_epoch{0}; +std::atomic par_active{false}; +void logf(const char* fmt, ...) { + va_list a; + va_start(a, fmt); + std::vfprintf(stdout, fmt, a); + va_end(a); + std::fputc('\n', stdout); +} +} // namespace mkwvk + +static std::atomic g_draws{0}; +static std::atomic g_parked{false}; +static void VKAPI_CALL fake_vkCmdDraw(VkCommandBuffer, uint32_t, uint32_t, uint32_t, uint32_t) { g_draws++; } +namespace vkx { +PFN_vkCmdDraw vkCmdDraw = fake_vkCmdDraw; +} // namespace vkx + +/// The recorder started by a packet, run, then parked for good inside idle() with sleep_mu held; 1 once it is +extern "C" __declspec(dllexport) int park_recorder() { + using namespace mkwvk; + rec::g_stats_log = false; + rec::g_test_idle_locked = [] { + g_parked = true; + for (;;) std::this_thread::sleep_for(std::chrono::seconds(1)); + }; + rec::set_mode(rec::THREAD); + static rec::DrainingMutex mu; + { + std::lock_guard lk(mu); + vkCmdDraw((VkCommandBuffer)(uintptr_t)0x100, 3, 1, 0, 0); + } // (the unlock: the draw has run) + for (int i = 0; i < 5000 && !g_parked; i++) std::this_thread::sleep_for(std::chrono::milliseconds(1)); + return g_parked && g_draws == 1 && rec::g_rec.th.joinable(); +} +#else +#include +extern "C" { +__declspec(dllimport) void* __stdcall LoadLibraryA(const char*); +__declspec(dllimport) void* __stdcall GetProcAddress(void*, const char*); +__declspec(dllimport) void __stdcall ExitProcess(unsigned int); +} +int main(int argc, char** argv) { + if (argc < 2) return 2; + void* m = LoadLibraryA(argv[1]); + auto park = m ? reinterpret_cast(GetProcAddress(m, "park_recorder")) : nullptr; + if (!park) { + std::printf("no fixture DLL\n"); + return 2; + } + const int ok = park(); + std::printf("recorder parked in idle() holding sleep_mu: %d; ExitProcess\n", ok); + std::fflush(stdout); + ExitProcess(ok ? 0 : 2); +} +#endif diff --git a/tests/recorder_sync_test.cpp b/tests/recorder_sync_test.cpp new file mode 100644 index 0000000..aeae19a --- /dev/null +++ b/tests/recorder_sync_test.cpp @@ -0,0 +1,1760 @@ +// RX_VK_RECORDER's packets against a fake Vulkan (tests/test_recorder_sync.py builds it with the real vk_work.h and vk_recorder.h). +// Every wrapped vkCmd* goes to a fake that logs the call with its arguments fully dereferenced, the way a driver reads them (by the +// counts and descriptor types, the pointers themselves never), so two logs are equal exactly when the driver was handed the same +// values. A frame of representative calls is made the way the renderer makes them (thread-local descriptor writes reset at every draw, +// push constants and vertex arrays on the stack, a rendering info with attachments on the stack, barriers, a reused copy vector, +// dynamic state arrays), and with `mutate` the caller poisons (0xCD) all of that right after each call, as the next draw reuses it. +// +// recorder_sync_test the checks; exits 1 on the first difference +// recorder_sync_test --bench N the packet cost per draw on the fake (fakes that do nothing), ns, median of 5 +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#ifdef _WIN32 +#include +#endif + +#include "vk_work.h" + +#ifdef _WIN32 +extern "C" __declspec(dllimport) unsigned int __stdcall SetErrorMode(unsigned int); // (a fault exits, without a dialog) +extern "C" __declspec(dllimport) int __stdcall GetThreadTimes(void*, void*, void*, void*, void*); // (--idle: the recorder's CPU) +#endif + +namespace mkwvk { +std::atomic work_epoch{0}; +std::atomic par_active{false}; +void logf(const char* fmt, ...) { // (the renderer's log: printed, for test_recorder_sync.py to count) + char b[1024]; + va_list a; + va_start(a, fmt); + std::vsnprintf(b, sizeof b, fmt, a); + va_end(a); + std::printf("LOG: %s\n", b); +} +} // namespace mkwvk + +// ---- the arena's aligned blocks: counted (a block lost is a leak), and the Nth one made to fail (g_fail_nth: an allocation failure; +// the throwing form throws std::bad_alloc as the system's does) +static std::atomic g_aligned_live{0}, g_aligned_calls{0}, g_fail_nth{0}; +static bool aligned_fails() { return g_fail_nth && ++g_aligned_calls == g_fail_nth; } +static void* aligned_get(size_t n, std::align_val_t al) { +#ifdef _WIN32 + void* p = _aligned_malloc(n, (size_t)al); +#else + void* p = nullptr; + if (posix_memalign(&p, (size_t)al, n)) p = nullptr; +#endif + if (p) g_aligned_live++; + return p; +} +static void aligned_put(void* p) { + if (!p) return; + g_aligned_live--; +#ifdef _WIN32 + _aligned_free(p); +#else + free(p); +#endif +} +void* operator new(size_t n, std::align_val_t al) { + void* p = aligned_fails() ? nullptr : aligned_get(n, al); + if (!p) throw std::bad_alloc(); + return p; +} +void* operator new[](size_t n, std::align_val_t al) { return operator new(n, al); } +void* operator new(size_t n, std::align_val_t al, const std::nothrow_t&) noexcept { return aligned_fails() ? nullptr : aligned_get(n, al); } +void* operator new[](size_t n, std::align_val_t al, const std::nothrow_t&) noexcept { return aligned_fails() ? nullptr : aligned_get(n, al); } +void operator delete(void* p, std::align_val_t) noexcept { aligned_put(p); } +void operator delete[](void* p, std::align_val_t) noexcept { aligned_put(p); } +void operator delete(void* p, size_t, std::align_val_t) noexcept { aligned_put(p); } +void operator delete[](void* p, size_t, std::align_val_t) noexcept { aligned_put(p); } +void operator delete(void* p, std::align_val_t, const std::nothrow_t&) noexcept { aligned_put(p); } +void operator delete[](void* p, std::align_val_t, const std::nothrow_t&) noexcept { aligned_put(p); } +using namespace mkwvk; + +// ---- the fake driver's log (RX_VK_RECORDER=thread: the recorder thread and the producer both call the fake: one lock for the log, +// and a command buffer's in-use flag held across each call, so two calls on one command buffer at once are seen, not serialized) +struct Call { + std::string op, bytes; +}; +static std::vector g_log; +static std::mutex g_log_mu; +static bool g_quiet = false; // (bench: the fakes only count) +static uint64_t g_calls = 0; +static int g_spin_ns = 0; // (bench --overlap: the time a quiet fake vkCmd* takes, as the driver's encoding does) +static std::atomic g_delay_ns{0}; // (thread runs: each logged vkCmd* takes this long, so the recorder lags the producer) +static std::atomic g_gate{false}; // (B_DIRECT control: vkCmd* on other threads wait until it is let go) +// (status reads: vkCmd* on other threads also wait until this steady-clock time, so the recorder cannot catch up before the read +// however the runner schedules the threads; a read that waits for the recorder is let through when it passes) +static std::atomic g_hold_until_ns{0}; +static bool held_until() { + const int64_t until = g_hold_until_ns.load(); + return until && std::chrono::duration_cast(std::chrono::steady_clock::now().time_since_epoch()).count() < until; +} +// (thread runs of the ordering scenarios: hold() before each point where the producer must wait for the recorder (a recording's +// end, a submit while another command buffer records, the replay lock let go, a pool reset) holds the recorder for this long, so its +// commands are still in flight there whatever the runner's load: a producer that does not wait is caught every time, not only when +// the 1 us a call happened to keep the recorder behind (the flakes of the broken variants on CI). 0: no hold (the reference runs) +static int g_hold_ms = 0; +static void hold() { + if (!g_hold_ms) return; + g_hold_until_ns = (int64_t)std::chrono::duration_cast(std::chrono::steady_clock::now().time_since_epoch()).count() + + (int64_t)g_hold_ms * 1000000; +} +static bool g_hash_only = false; // (stress: each call folded into g_hash and dropped) +static uint64_t g_hash = 0xcbf29ce484222325ull; +static std::thread::id g_main_id; +static std::atomic g_cmd_off_main{0}, g_cmd_calls{0}; // (vkCmd* the fake was given, and those on another thread) +static std::atomic g_throw{false}; // (--throw: the next vkCmdDraw throws, as a driver call could) +static void spin_for(int ns) { + const auto t0 = std::chrono::steady_clock::now(); + while (std::chrono::steady_clock::now() - t0 < std::chrono::nanoseconds(ns)) {} +} +static void fold(const Call& c) { + for (const std::string* str : {&c.op, &c.bytes}) + for (unsigned char ch : *str) g_hash = (g_hash ^ ch) * 0x100000001b3ull; + g_hash = (g_hash ^ 0xFF) * 0x100000001b3ull; +} + +static void begin(const char* op) { g_log.push_back({op, {}}); } +template +static void put(const T& v) { + static_assert(std::is_arithmetic_v || std::is_enum_v, "fields one by one: no padding, no pointers"); + g_log.back().bytes.append(reinterpret_cast(&v), sizeof v); +} +template +static void handle(T h) { + put((uint64_t)(uintptr_t)h); +} +static void raw(const void* p, size_t n) { g_log.back().bytes.append(static_cast(p), n); } +static void tag(const char* s) { g_log.back().bytes.append(s).push_back(0); } +// (a pointer or count the caller has since poisoned, which a missed copy hands over: logged as such, never followed) +static bool readable(const void* p, uint64_t n) { + if ((uintptr_t)p == (uintptr_t)0xCDCDCDCDCDCDCDCDull || n > 4096) { + tag("UNREADABLE"); + return false; + } + return true; +} +template +static void each(const T* p, uint64_t n, F f) { + put((uint32_t)n); + if (!n || !readable(p, n)) return; + for (uint64_t i = 0; i < n; i++) f(p[i]); +} +template +static void opt(const T* p, F f) { // (an optional struct: there or not, then what it holds) + put((uint8_t)(p != nullptr)); + if (p && readable(p, 1)) f(*p); +} + +static void L(const VkImageSubresourceRange& r) { put(r.aspectMask), put(r.baseMipLevel), put(r.levelCount), put(r.baseArrayLayer), put(r.layerCount); } +static void L(const VkImageSubresourceLayers& r) { put(r.aspectMask), put(r.mipLevel), put(r.baseArrayLayer), put(r.layerCount); } +static void L(const VkOffset3D& o) { put(o.x), put(o.y), put(o.z); } +static void L(const VkExtent3D& e) { put(e.width), put(e.height), put(e.depth); } +static void L(const VkRect2D& r) { put(r.offset.x), put(r.offset.y), put(r.extent.width), put(r.extent.height); } +static void L(const VkViewport& v) { put(v.x), put(v.y), put(v.width), put(v.height), put(v.minDepth), put(v.maxDepth); } +static void L(const VkClearValue& v) { raw(&v, sizeof v); } // (a union of 16 bytes, no padding) +static void L(const VkClearColorValue& v) { raw(&v, sizeof v); } +static void L(const VkClearDepthStencilValue& v) { put(v.depth), put(v.stencil); } +static void L(const VkMemoryBarrier2& b) { + put(b.sType), put((uint8_t)!!b.pNext), put(b.srcStageMask), put(b.srcAccessMask), put(b.dstStageMask), put(b.dstAccessMask); +} +static void L(const VkBufferMemoryBarrier2& b) { + put(b.sType), put((uint8_t)!!b.pNext), put(b.srcStageMask), put(b.srcAccessMask), put(b.dstStageMask), put(b.dstAccessMask); + put(b.srcQueueFamilyIndex), put(b.dstQueueFamilyIndex), handle(b.buffer), put(b.offset), put(b.size); +} +static void L(const VkImageMemoryBarrier2& b) { + put(b.sType), put((uint8_t)!!b.pNext), put(b.srcStageMask), put(b.srcAccessMask), put(b.dstStageMask), put(b.dstAccessMask); + put(b.oldLayout), put(b.newLayout), put(b.srcQueueFamilyIndex), put(b.dstQueueFamilyIndex), handle(b.image), L(b.subresourceRange); +} +static void L(const VkRenderingAttachmentInfo& a) { + put(a.sType), put((uint8_t)!!a.pNext), handle(a.imageView), put(a.imageLayout), put(a.resolveMode), handle(a.resolveImageView); + put(a.resolveImageLayout), put(a.loadOp), put(a.storeOp), L(a.clearValue); +} +static void L(const VkBufferCopy& c) { put(c.srcOffset), put(c.dstOffset), put(c.size); } +static void L(const VkBufferImageCopy& c) { + put(c.bufferOffset), put(c.bufferRowLength), put(c.bufferImageHeight), L(c.imageSubresource), L(c.imageOffset), L(c.imageExtent); +} +static void L(const VkImageCopy& c) { L(c.srcSubresource), L(c.srcOffset), L(c.dstSubresource), L(c.dstOffset), L(c.extent); } +static void L(const VkImageResolve& c) { L(c.srcSubresource), L(c.srcOffset), L(c.dstSubresource), L(c.dstOffset), L(c.extent); } +static void L(const VkImageBlit& b) { + L(b.srcSubresource), L(b.srcOffsets[0]), L(b.srcOffsets[1]), L(b.dstSubresource), L(b.dstOffsets[0]), L(b.dstOffsets[1]); +} +static void L(const VkClearAttachment& a) { put(a.aspectMask), put(a.colorAttachment), L(a.clearValue); } +static void L(const VkClearRect& r) { L(r.rect), put(r.baseArrayLayer), put(r.layerCount); } +static void L(const VkColorBlendEquationEXT& e) { + put(e.srcColorBlendFactor), put(e.dstColorBlendFactor), put(e.colorBlendOp), put(e.srcAlphaBlendFactor), put(e.dstAlphaBlendFactor), + put(e.alphaBlendOp); +} +static void L(const VkWriteDescriptorSet& w) { // (as a driver reads a push: by type, one of the three arrays) + put(w.sType), put((uint8_t)!!w.pNext), handle(w.dstSet), put(w.dstBinding), put(w.dstArrayElement), put(w.descriptorType); + switch (rec::write_kind(w.descriptorType)) { + case 1: each(w.pImageInfo, w.descriptorCount, [](const VkDescriptorImageInfo& i) { handle(i.sampler), handle(i.imageView), put(i.imageLayout); }); break; + case 2: each(w.pBufferInfo, w.descriptorCount, [](const VkDescriptorBufferInfo& b) { handle(b.buffer), put(b.offset), put(b.range); }); break; + case 3: each(w.pTexelBufferView, w.descriptorCount, [](VkBufferView v) { handle(v); }); break; + default: put(w.descriptorCount); break; // (not read here: the packer leaves these to the direct call) + } +} +template +static void A(const T* p, uint64_t n) { + each(p, n, [](const T& v) { L(v); }); +} + +#define QUIET \ + if (g_quiet) { g_calls++; if (g_spin_ns) spin_for(g_spin_ns); return; } +// ---- the fake lifecycle (g_track): each command buffer's state as a driver tracks it. A vkCmd* on one that is not recording, a begin +// of one recording, an end of one not recording or a submit of one not executable is a violation; the boundaries are logged as calls +// too, so two logs also compare where the begins, ends, resets, submits, waits, reads and destructions fall among the commands. +// Always: a call on a command buffer (or a reset of its pool, a submit of it) while another call on it is under way is a violation +enum CbState { INITIAL, RECORDING, EXECUTABLE }; +static bool g_track = false; +static std::map g_state; +static std::map g_pool_of; // (command buffer -> its pool) +static std::mutex g_viol_mu; +static int g_violations; +static std::string g_violation; +void violation(const char* op, const void* cb, const char* what) { + std::lock_guard lk(g_viol_mu); + if (g_violations++) return; + char b[240]; + std::snprintf(b, sizeof b, "%s on command buffer %#llx %s (driver call %zu)", op, (unsigned long long)(uintptr_t)cb, what, g_log.size()); + g_violation = b; +} +static CbState& state(const void* cb) { return g_state[(uint64_t)(uintptr_t)cb]; } +static void track(const void* cb, const char* op) { + if (g_track && state(cb) != RECORDING) violation(op, cb, "not recording"); +} +// ---- the fake GPU (g_track): a command buffer's recorded effects (timestamps, checkpoints, fills) take effect when it is submitted, +// as on a device: then the timeline steps and the fence is signalled. The status reads below (fences, semaphores, checkpoints, fault +// data, memory invalidation) see only that, and are logged apart (g_reads), unordered against the vkCmd* stream; the ordered stream +// keeps every other call. Destroyed buffers are remembered: a recorded command naming one is a violation +struct Fx { + int kind; // 1 timestamp (a: pool << 16 | query), 2 checkpoint (a: marker), 3 fill (a: buffer, b: data) + uint64_t a, b; +}; +static std::map> g_cb_fx; +static std::map g_query, g_mem; +static std::set g_fences_done, g_destroyed; +static uint64_t g_timeline = 0, g_checkpoint = 0; +static std::vector g_reads; // (each status read: the call, what it saw) +static std::map g_reads_overlapped; // (status reads made while the producer had packets not yet run) +static void gpu_reset() { + g_cb_fx.clear(), g_query.clear(), g_mem.clear(), g_fences_done.clear(), g_destroyed.clear(); + g_timeline = 0, g_checkpoint = 0, g_reads.clear(), g_reads_overlapped.clear(); +} +static void fx(const void* cb, int kind, uint64_t a, uint64_t b = 0) { + if (g_track) g_cb_fx[(uint64_t)(uintptr_t)cb].push_back({kind, a, b}); +} +void violation(const char* op, const void* cb, const char* what); +static void alive(const void* cb, const char* op, VkBuffer b) { // (a recorded command naming a destroyed buffer) + if (g_track && g_destroyed.count((uint64_t)(uintptr_t)b)) violation(op, cb, "names a destroyed buffer"); +} +static std::atomic g_inuse[256]; +static std::atomic& inuse(uint64_t cb) { return g_inuse[(cb >> 8) & 255]; } // (the tests' command buffers: 0x100, 0x200, 0x300) +/// One fake call: its command buffer marked in use for the call (a second call on it meanwhile is a violation), a vkCmd* slowed by +/// g_delay_ns, then the log's lock +struct Guard { + const void* cb; + bool held = false; + std::unique_lock lk; + Guard(const void* c, const char* op, bool cmd) : cb(c) { + while (cmd && std::this_thread::get_id() != g_main_id && (g_gate.load() || held_until())) std::this_thread::yield(); // (before the in-use mark) + if (cb) { + if (inuse((uint64_t)(uintptr_t)cb).exchange(true)) violation(op, cb, "in use by another thread (no external synchronization)"); + else held = true; + } + if (cmd) { + g_cmd_calls++; + if (std::this_thread::get_id() != g_main_id) g_cmd_off_main++; + if (const int d = g_delay_ns.load(std::memory_order_relaxed)) spin_for(d); + } + lk = std::unique_lock(g_log_mu); + } + ~Guard() { + if (g_hash_only && !g_log.empty()) fold(g_log.back()), g_log.pop_back(); + lk.unlock(); + if (held) inuse((uint64_t)(uintptr_t)cb).store(false); + } +}; +/// (a pool reset or a submit: its command buffers must not be in use meanwhile) +static void not_in_use(const void* cb, const char* op) { + if (inuse((uint64_t)(uintptr_t)cb).load()) violation(op, cb, "in use by another thread (no external synchronization)"); +} + +#define F(name, params, ...) \ + static void VKAPI_CALL fake_##name params { QUIET Guard g_(cb, #name, true); begin(#name); track(cb, #name); __VA_ARGS__ } + +F(vkCmdPipelineBarrier2, (VkCommandBuffer cb, const VkDependencyInfo* d), handle(cb); put(d->sType), put((uint8_t)!!d->pNext), put(d->dependencyFlags); + A(d->pMemoryBarriers, d->memoryBarrierCount), A(d->pBufferMemoryBarriers, d->bufferMemoryBarrierCount); + A(d->pImageMemoryBarriers, d->imageMemoryBarrierCount);) +F(vkCmdBeginRendering, (VkCommandBuffer cb, const VkRenderingInfo* r), handle(cb); put(r->sType), put((uint8_t)!!r->pNext), put(r->flags); + L(r->renderArea), put(r->layerCount), put(r->viewMask); A(r->pColorAttachments, r->colorAttachmentCount); + opt(r->pDepthAttachment, [](const VkRenderingAttachmentInfo& a) { L(a); }); opt(r->pStencilAttachment, [](const VkRenderingAttachmentInfo& a) { L(a); });) +F(vkCmdEndRendering, (VkCommandBuffer cb), handle(cb);) +F(vkCmdDraw, (VkCommandBuffer cb, uint32_t a, uint32_t b, uint32_t c, uint32_t d), handle(cb), put(a), put(b), put(c), put(d); + if (g_throw.exchange(false)) throw std::runtime_error("a driver call threw (test)");) +F(vkCmdDrawIndexed, (VkCommandBuffer cb, uint32_t a, uint32_t b, uint32_t c, int32_t d, uint32_t e), handle(cb), put(a), put(b), put(c), put(d), put(e);) +F(vkCmdDrawIndirect, (VkCommandBuffer cb, VkBuffer b, VkDeviceSize o, uint32_t n, uint32_t s), handle(cb), handle(b), put(o), put(n), put(s);) +F(vkCmdDrawIndexedIndirect, (VkCommandBuffer cb, VkBuffer b, VkDeviceSize o, uint32_t n, uint32_t s), handle(cb), handle(b), put(o), put(n), put(s);) +F(vkCmdDrawIndirectCount, (VkCommandBuffer cb, VkBuffer b, VkDeviceSize o, VkBuffer cbuf, VkDeviceSize co, uint32_t m, uint32_t s), + handle(cb), handle(b), put(o), handle(cbuf), put(co), put(m), put(s);) +F(vkCmdDrawIndexedIndirectCount, (VkCommandBuffer cb, VkBuffer b, VkDeviceSize o, VkBuffer cbuf, VkDeviceSize co, uint32_t m, uint32_t s), + handle(cb), handle(b), put(o), handle(cbuf), put(co), put(m), put(s);) +F(vkCmdDrawMeshTasksEXT, (VkCommandBuffer cb, uint32_t x, uint32_t y, uint32_t z), handle(cb), put(x), put(y), put(z);) +F(vkCmdDrawMeshTasksIndirectEXT, (VkCommandBuffer cb, VkBuffer b, VkDeviceSize o, uint32_t n, uint32_t s), handle(cb), handle(b), put(o), put(n), put(s);) +F(vkCmdDrawMeshTasksIndirectCountEXT, (VkCommandBuffer cb, VkBuffer b, VkDeviceSize o, VkBuffer cbuf, VkDeviceSize co, uint32_t m, uint32_t s), + handle(cb), handle(b), put(o), handle(cbuf), put(co), put(m), put(s);) +F(vkCmdDispatch, (VkCommandBuffer cb, uint32_t x, uint32_t y, uint32_t z), handle(cb), put(x), put(y), put(z);) +F(vkCmdDispatchIndirect, (VkCommandBuffer cb, VkBuffer b, VkDeviceSize o), handle(cb), handle(b), put(o);) +F(vkCmdCopyBuffer, (VkCommandBuffer cb, VkBuffer s, VkBuffer d, uint32_t n, const VkBufferCopy* r), handle(cb), handle(s), handle(d); A(r, n); + alive(cb, "vkCmdCopyBuffer", s), alive(cb, "vkCmdCopyBuffer", d);) +F(vkCmdCopyBufferToImage, (VkCommandBuffer cb, VkBuffer s, VkImage d, VkImageLayout l, uint32_t n, const VkBufferImageCopy* r), + handle(cb), handle(s), handle(d), put(l); A(r, n);) +F(vkCmdCopyImageToBuffer, (VkCommandBuffer cb, VkImage s, VkImageLayout l, VkBuffer d, uint32_t n, const VkBufferImageCopy* r), + handle(cb), handle(s), put(l), handle(d); A(r, n);) +F(vkCmdCopyImage, (VkCommandBuffer cb, VkImage s, VkImageLayout sl, VkImage d, VkImageLayout dl, uint32_t n, const VkImageCopy* r), + handle(cb), handle(s), put(sl), handle(d), put(dl); A(r, n);) +F(vkCmdBlitImage, (VkCommandBuffer cb, VkImage s, VkImageLayout sl, VkImage d, VkImageLayout dl, uint32_t n, const VkImageBlit* r, VkFilter f), + handle(cb), handle(s), put(sl), handle(d), put(dl); A(r, n); put(f);) +F(vkCmdResolveImage, (VkCommandBuffer cb, VkImage s, VkImageLayout sl, VkImage d, VkImageLayout dl, uint32_t n, const VkImageResolve* r), + handle(cb), handle(s), put(sl), handle(d), put(dl); A(r, n);) +F(vkCmdClearColorImage, (VkCommandBuffer cb, VkImage i, VkImageLayout l, const VkClearColorValue* c, uint32_t n, const VkImageSubresourceRange* r), + handle(cb), handle(i), put(l); opt(c, [](const VkClearColorValue& v) { L(v); }); A(r, n);) +F(vkCmdClearDepthStencilImage, (VkCommandBuffer cb, VkImage i, VkImageLayout l, const VkClearDepthStencilValue* c, uint32_t n, + const VkImageSubresourceRange* r), handle(cb), handle(i), put(l); opt(c, [](const VkClearDepthStencilValue& v) { L(v); }); A(r, n);) +F(vkCmdClearAttachments, (VkCommandBuffer cb, uint32_t na, const VkClearAttachment* a, uint32_t nr, const VkClearRect* r), + handle(cb); A(a, na); A(r, nr);) +F(vkCmdFillBuffer, (VkCommandBuffer cb, VkBuffer b, VkDeviceSize o, VkDeviceSize s, uint32_t d), handle(cb), handle(b), put(o), put(s), put(d); + alive(cb, "vkCmdFillBuffer", b); fx(cb, 3, (uint64_t)(uintptr_t)b, d);) +F(vkCmdUpdateBuffer, (VkCommandBuffer cb, VkBuffer b, VkDeviceSize o, VkDeviceSize s, const void* d), handle(cb), handle(b), put(o), put(s); + if (readable(d, s / 16)) raw(d, (size_t)s);) +F(vkCmdBeginQuery, (VkCommandBuffer cb, VkQueryPool p, uint32_t q, VkQueryControlFlags f), handle(cb), handle(p), put(q), put(f);) +F(vkCmdEndQuery, (VkCommandBuffer cb, VkQueryPool p, uint32_t q), handle(cb), handle(p), put(q);) +F(vkCmdBindPipeline, (VkCommandBuffer cb, VkPipelineBindPoint bp, VkPipeline p), handle(cb), put(bp), handle(p);) +F(vkCmdBindDescriptorSets, (VkCommandBuffer cb, VkPipelineBindPoint bp, VkPipelineLayout l, uint32_t first, uint32_t n, const VkDescriptorSet* s, + uint32_t nd, const uint32_t* d), handle(cb), put(bp), handle(l), put(first); each(s, n, [](VkDescriptorSet v) { handle(v); }); + each(d, nd, [](uint32_t v) { put(v); });) +F(vkCmdBindIndexBuffer, (VkCommandBuffer cb, VkBuffer b, VkDeviceSize o, VkIndexType t), handle(cb), handle(b), put(o), put(t);) +F(vkCmdBindVertexBuffers2, (VkCommandBuffer cb, uint32_t first, uint32_t n, const VkBuffer* b, const VkDeviceSize* o, const VkDeviceSize* s, + const VkDeviceSize* st), handle(cb), put(first); each(b, n, [](VkBuffer v) { handle(v); }); each(o, n, [](VkDeviceSize v) { put(v); }); + put((uint8_t)!!s); if (s) each(s, n, [](VkDeviceSize v) { put(v); }); put((uint8_t)!!st); if (st) each(st, n, [](VkDeviceSize v) { put(v); });) +F(vkCmdPushConstants, (VkCommandBuffer cb, VkPipelineLayout l, VkShaderStageFlags f, uint32_t o, uint32_t s, const void* v), + handle(cb), handle(l), put(f), put(o), put(s); if (readable(v, s / 16)) raw(v, s);) +F(vkCmdPushDescriptorSetKHR, (VkCommandBuffer cb, VkPipelineBindPoint bp, VkPipelineLayout l, uint32_t set, uint32_t n, const VkWriteDescriptorSet* w), + handle(cb), put(bp), handle(l), put(set); A(w, n);) +F(vkCmdSetViewport, (VkCommandBuffer cb, uint32_t first, uint32_t n, const VkViewport* v), handle(cb), put(first); A(v, n);) +F(vkCmdSetScissor, (VkCommandBuffer cb, uint32_t first, uint32_t n, const VkRect2D* r), handle(cb), put(first); A(r, n);) +F(vkCmdSetViewportWithCount, (VkCommandBuffer cb, uint32_t n, const VkViewport* v), handle(cb); A(v, n);) +F(vkCmdSetScissorWithCount, (VkCommandBuffer cb, uint32_t n, const VkRect2D* r), handle(cb); A(r, n);) +F(vkCmdSetBlendConstants, (VkCommandBuffer cb, const float b[4]), handle(cb); each(b, 4, [](float v) { put(v); });) +F(vkCmdSetStencilReference, (VkCommandBuffer cb, VkStencilFaceFlags f, uint32_t v), handle(cb), put(f), put(v);) +F(vkCmdSetStencilCompareMask, (VkCommandBuffer cb, VkStencilFaceFlags f, uint32_t v), handle(cb), put(f), put(v);) +F(vkCmdSetStencilWriteMask, (VkCommandBuffer cb, VkStencilFaceFlags f, uint32_t v), handle(cb), put(f), put(v);) +F(vkCmdSetStencilOp, (VkCommandBuffer cb, VkStencilFaceFlags f, VkStencilOp a, VkStencilOp b, VkStencilOp c, VkCompareOp d), + handle(cb), put(f), put(a), put(b), put(c), put(d);) +F(vkCmdSetDepthBias, (VkCommandBuffer cb, float a, float b, float c), handle(cb), put(a), put(b), put(c);) +F(vkCmdSetDepthBiasEnable, (VkCommandBuffer cb, VkBool32 v), handle(cb), put(v);) +F(vkCmdSetLineWidth, (VkCommandBuffer cb, float v), handle(cb), put(v);) +F(vkCmdSetDepthBoundsTestEnable, (VkCommandBuffer cb, VkBool32 v), handle(cb), put(v);) +F(vkCmdSetCullMode, (VkCommandBuffer cb, VkCullModeFlags v), handle(cb), put(v);) +F(vkCmdSetFrontFace, (VkCommandBuffer cb, VkFrontFace v), handle(cb), put(v);) +F(vkCmdSetPrimitiveTopology, (VkCommandBuffer cb, VkPrimitiveTopology v), handle(cb), put(v);) +F(vkCmdSetPrimitiveRestartEnable, (VkCommandBuffer cb, VkBool32 v), handle(cb), put(v);) +F(vkCmdSetDepthTestEnable, (VkCommandBuffer cb, VkBool32 v), handle(cb), put(v);) +F(vkCmdSetDepthWriteEnable, (VkCommandBuffer cb, VkBool32 v), handle(cb), put(v);) +F(vkCmdSetDepthCompareOp, (VkCommandBuffer cb, VkCompareOp v), handle(cb), put(v);) +F(vkCmdSetStencilTestEnable, (VkCommandBuffer cb, VkBool32 v), handle(cb), put(v);) +F(vkCmdSetRasterizerDiscardEnable, (VkCommandBuffer cb, VkBool32 v), handle(cb), put(v);) +F(vkCmdSetPolygonModeEXT, (VkCommandBuffer cb, VkPolygonMode v), handle(cb), put(v);) +F(vkCmdSetDepthClampEnableEXT, (VkCommandBuffer cb, VkBool32 v), handle(cb), put(v);) +F(vkCmdSetAlphaToCoverageEnableEXT, (VkCommandBuffer cb, VkBool32 v), handle(cb), put(v);) +F(vkCmdSetColorBlendEnableEXT, (VkCommandBuffer cb, uint32_t first, uint32_t n, const VkBool32* v), handle(cb), put(first); + each(v, n, [](VkBool32 x) { put(x); });) +F(vkCmdSetColorBlendEquationEXT, (VkCommandBuffer cb, uint32_t first, uint32_t n, const VkColorBlendEquationEXT* v), handle(cb), put(first); A(v, n);) +F(vkCmdSetColorWriteMaskEXT, (VkCommandBuffer cb, uint32_t first, uint32_t n, const VkColorComponentFlags* v), handle(cb), put(first); + each(v, n, [](VkColorComponentFlags x) { put(x); });) +F(vkCmdWriteBufferMarkerAMD, (VkCommandBuffer cb, VkPipelineStageFlagBits s, VkBuffer b, VkDeviceSize o, uint32_t m), + handle(cb), put(s), handle(b), put(o), put(m);) +F(vkCmdSetCheckpointNV, (VkCommandBuffer cb, const void* m), handle(cb), handle(m); fx(cb, 2, (uint64_t)(uintptr_t)m);) // (an opaque value: the pointer is the datum) +F(vkCmdWriteTimestamp2, (VkCommandBuffer cb, VkPipelineStageFlags2 s, VkQueryPool p, uint32_t q), handle(cb), put(s), handle(p), put(q); + fx(cb, 1, (uint64_t)(uintptr_t)p << 16 | q);) +F(vkCmdResetQueryPool, (VkCommandBuffer cb, VkQueryPool p, uint32_t f, uint32_t n), handle(cb), handle(p), put(f), put(n);) + +static VkResult VKAPI_CALL fake_vkBeginCommandBuffer(VkCommandBuffer cb, const VkCommandBufferBeginInfo* bi) { + Guard g_(cb, "vkBeginCommandBuffer", false); + begin("vkBeginCommandBuffer"), handle(cb), put(bi->flags); + g_cb_fx.erase((uint64_t)(uintptr_t)cb); + if (g_track && state(cb) == RECORDING) violation("vkBeginCommandBuffer", cb, "already recording"); + state(cb) = RECORDING; + return VK_SUCCESS; +} +static VkResult VKAPI_CALL fake_vkEndCommandBuffer(VkCommandBuffer cb) { + Guard g_(cb, "vkEndCommandBuffer", false); + begin("vkEndCommandBuffer"), handle(cb); + track(cb, "vkEndCommandBuffer"); + state(cb) = EXECUTABLE; + return VK_SUCCESS; +} +static VkResult VKAPI_CALL fake_vkResetCommandPool(VkDevice d, VkCommandPool pool, VkCommandPoolResetFlags f) { + Guard g_(nullptr, "vkResetCommandPool", false); + begin("vkResetCommandPool"), handle(d), handle(pool), put(f); + for (const auto& [cb, of] : g_pool_of) + if (of == (uint64_t)(uintptr_t)pool) not_in_use((const void*)(uintptr_t)cb, "vkResetCommandPool"), g_state[cb] = INITIAL, g_cb_fx.erase(cb); + return VK_SUCCESS; +} +static VkResult VKAPI_CALL fake_vkQueueSubmit2(VkQueue q, uint32_t n, const VkSubmitInfo2* si, VkFence fence) { + Guard g_(nullptr, "vkQueueSubmit2", false); + begin("vkQueueSubmit2"), handle(q); + for (uint32_t i = 0; i < n; i++) + for (uint32_t c = 0; c < si[i].commandBufferInfoCount; c++) { + const VkCommandBuffer cb = si[i].pCommandBufferInfos[c].commandBuffer; + handle(cb); + not_in_use(cb, "vkQueueSubmit2"); + if (g_track && state(cb) != EXECUTABLE) violation("vkQueueSubmit2", cb, "not executable"); + for (const Fx& e : g_cb_fx[(uint64_t)(uintptr_t)cb]) // (the fake GPU runs it: its effects, now) + if (e.kind == 1) g_query[e.a] = g_timeline + 1; + else if (e.kind == 2) g_checkpoint = e.a; + else g_mem[e.a] = e.b; + } + handle(fence); + g_timeline++; + if (fence) g_fences_done.insert((uint64_t)(uintptr_t)fence); + return VK_SUCCESS; +} +/// A status read: logged apart from the ordered stream, with what it saw; counted when it ran with packets not yet run +static void read_seen(const char* op, uint64_t v) { + g_reads.push_back(std::hash()(op)), g_reads.push_back(v); + if (rec::mode() == rec::THREAD && rec::t_prod && rec::g_rec.w > rec::g_rec.R.load()) g_reads_overlapped[op]++; +} +static VkResult VKAPI_CALL fake_vkWaitForFences(VkDevice, uint32_t n, const VkFence* f, VkBool32, uint64_t) { + Guard g_(nullptr, "vkWaitForFences", false); + bool done = true; + for (uint32_t i = 0; i < n; i++) done = done && g_fences_done.count((uint64_t)(uintptr_t)f[i]); + read_seen("vkWaitForFences", done); + return done ? VK_SUCCESS : VK_TIMEOUT; +} +static VkResult VKAPI_CALL fake_vkGetSemaphoreCounterValue(VkDevice, VkSemaphore, uint64_t* v) { + Guard g_(nullptr, "vkGetSemaphoreCounterValue", false); + *v = g_timeline; + read_seen("vkGetSemaphoreCounterValue", *v); + return VK_SUCCESS; +} +static VkResult VKAPI_CALL fake_vkWaitSemaphores(VkDevice, const VkSemaphoreWaitInfo* wi, uint64_t) { + Guard g_(nullptr, "vkWaitSemaphores", false); + const bool done = g_timeline >= wi->pValues[0]; + read_seen("vkWaitSemaphores", done); + return done ? VK_SUCCESS : VK_TIMEOUT; +} +static void VKAPI_CALL fake_vkGetQueueCheckpointDataNV(VkQueue, uint32_t* n, VkCheckpointDataNV* cd) { + Guard g_(nullptr, "vkGetQueueCheckpointDataNV", false); + if (cd && *n) cd[0].pCheckpointMarker = (void*)(uintptr_t)g_checkpoint, *n = 1; + read_seen("vkGetQueueCheckpointDataNV", g_checkpoint); +} +static VkResult VKAPI_CALL fake_vkGetDeviceFaultInfoEXT(VkDevice, VkDeviceFaultCountsEXT* c, VkDeviceFaultInfoEXT*) { + Guard g_(nullptr, "vkGetDeviceFaultInfoEXT", false); + c->addressInfoCount = 0, c->vendorInfoCount = 0, c->vendorBinarySize = g_timeline; // (what the device ran: the submits) + read_seen("vkGetDeviceFaultInfoEXT", g_timeline); + return VK_SUCCESS; +} +// (VMA's, faked: the allocation is named by the buffer the GPU wrote; what the host then sees) +struct VmaAllocator_T; +struct VmaAllocation_T; +typedef VmaAllocator_T* VmaAllocator; +typedef VmaAllocation_T* VmaAllocation; +VkResult (vmaInvalidateAllocation)(VmaAllocator, VmaAllocation a, VkDeviceSize, VkDeviceSize) { + Guard g_(nullptr, "vmaInvalidateAllocation", false); + const uint64_t v = g_mem.count((uint64_t)(uintptr_t)a) ? g_mem[(uint64_t)(uintptr_t)a] : 0; + read_seen("vmaInvalidateAllocation", v); + return VK_SUCCESS; +} +static VkResult VKAPI_CALL fake_vkGetQueryPoolResults(VkDevice d, VkQueryPool p, uint32_t first, uint32_t n, size_t size, void* data, + VkDeviceSize stride, VkQueryResultFlags flags) { + Guard g_(nullptr, "vkGetQueryPoolResults", false); + begin("vkGetQueryPoolResults"), handle(d), handle(p), put(first), put(n), put(stride), put(flags); + std::memset(data, 0, size); + for (uint32_t i = 0; i < n && (i + 1) * stride <= size; i++) { // (the timestamps the submitted command buffers wrote) + const auto it = g_query.find((uint64_t)(uintptr_t)p << 16 | (first + i)); + const uint64_t v = it == g_query.end() ? 0 : it->second; + std::memcpy(static_cast(data) + i * stride, &v, 8), put(v); + } + return VK_SUCCESS; +} +static void VKAPI_CALL fake_vkDestroyBuffer(VkDevice d, VkBuffer b, const VkAllocationCallbacks*) { + Guard g_(nullptr, "vkDestroyBuffer", false); + begin("vkDestroyBuffer"), handle(d), handle(b); + g_destroyed.insert((uint64_t)(uintptr_t)b); +} +static void VKAPI_CALL fake_vkDestroyDevice(VkDevice d, const VkAllocationCallbacks*) { + Guard g_(nullptr, "vkDestroyDevice", false); + begin("vkDestroyDevice"), handle(d); +} + +namespace vkx { +#define RX_FAKE(f) PFN_##f f = fake_##f; +RX_REC_CALLS(RX_FAKE, RX_FAKE) +RX_FAKE(vkBeginCommandBuffer) RX_FAKE(vkEndCommandBuffer) RX_FAKE(vkResetCommandPool) RX_FAKE(vkQueueSubmit2) RX_FAKE(vkWaitForFences) +RX_FAKE(vkGetQueryPoolResults) RX_FAKE(vkDestroyBuffer) RX_FAKE(vkDestroyDevice) RX_FAKE(vkGetSemaphoreCounterValue) +RX_FAKE(vkWaitSemaphores) RX_FAKE(vkGetQueueCheckpointDataNV) RX_FAKE(vkGetDeviceFaultInfoEXT) +#undef RX_FAKE +} // namespace vkx + +// ---- the renderer's shapes of argument storage +template +static T H(uint64_t v) { + return (T)(uintptr_t)v; +} +template +static void poison(T* p, size_t n = 1) { + std::memset((void*)p, 0xCD, sizeof(T) * n); +} +template +static void poison(std::vector& v) { // (the next draw's reset: the storage stays, its contents change) + if (!v.empty()) poison(v.data(), v.size()); + v.clear(); +} + +static bool g_mutate = false; +constexpr uint32_t kPushSize = 96; // (vk_draw.cpp's: MESH_FALLBACK_PUSH_END) + +struct DescriptorWrites { // (vk_draw.cpp's thread-local dw: reset at every build, the writes point into the info vectors) + std::vector writes; + std::vector bufs; + std::vector imgs; + std::vector views; + void reset() { + writes.clear(), bufs.clear(), imgs.clear(), views.clear(); + bufs.reserve(64), imgs.reserve(64), views.reserve(64); + } + void after_call() { + if (g_mutate) poison(writes), poison(bufs), poison(imgs), poison(views); + } +}; +static thread_local DescriptorWrites dw; + +static VkWriteDescriptorSet write(uint32_t binding, VkDescriptorType t, uint32_t count) { + VkWriteDescriptorSet w{VK_STRUCTURE_TYPE_WRITE_DESCRIPTOR_SET}; + w.dstBinding = binding, w.descriptorType = t, w.descriptorCount = count; + return w; +} +static void build_graphics(int d) { + dw.reset(); + for (uint32_t i = 0; i < 3; i++) { + dw.bufs.push_back({H(0xB000 + d * 16 + i), 256ull * i + d, 64ull + i}); + VkWriteDescriptorSet w = write(i, i == 2 ? VK_DESCRIPTOR_TYPE_STORAGE_BUFFER : VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER, 1); + w.pBufferInfo = &dw.bufs.back(); + if (i == 1) w.pImageInfo = (const VkDescriptorImageInfo*)(uintptr_t)0xCDCDCDCDCDCDCDCDull; // (ignored for a buffer) + dw.writes.push_back(w); + } + for (uint32_t i = 0; i < 4; i++) { + dw.imgs.push_back({H(0x5000 + i), H(0x6000 + d * 16 + i), VK_IMAGE_LAYOUT_GENERAL}); + VkWriteDescriptorSet w = write(3 + i, VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER, 1); + w.pImageInfo = &dw.imgs.back(); + dw.writes.push_back(w); + } + const size_t at = dw.imgs.size(); // (an array binding: two sampled images) + dw.imgs.push_back({VK_NULL_HANDLE, H(0x6100 + d), VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL}); + dw.imgs.push_back({VK_NULL_HANDLE, H(0x6200 + d), VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL}); + VkWriteDescriptorSet arr = write(7, VK_DESCRIPTOR_TYPE_SAMPLED_IMAGE, 2); + arr.dstArrayElement = 0, arr.pImageInfo = &dw.imgs[at]; + dw.writes.push_back(arr); + dw.views.push_back(H(0x7000 + d)); + VkWriteDescriptorSet tb = write(8, VK_DESCRIPTOR_TYPE_UNIFORM_TEXEL_BUFFER, 1); + tb.pTexelBufferView = &dw.views.back(); + dw.writes.push_back(tb); +} + +static void push_constants(VkCommandBuffer cmd, VkPipelineLayout layout, int d) { + uint8_t data[kPushSize]{}; + for (uint32_t i = 0; i < kPushSize; i++) data[i] = (uint8_t)(i * 7 + d); + vkCmdPushConstants(cmd, layout, VK_SHADER_STAGE_ALL_GRAPHICS, 0, kPushSize, data); + if (g_mutate) poison(data, kPushSize); + const uint64_t firsts = 0x1000ull + d; // (a mesh draw's firsts: 8 bytes at their offset) + uint64_t f = firsts; + vkCmdPushConstants(cmd, layout, VK_SHADER_STAGE_ALL_GRAPHICS, 80, 8, &f); + if (g_mutate) poison(&f); +} + +struct DynState { // (vk_draw.cpp's DynState: the arrays the dynamic state calls point into) + VkViewport vp; + VkRect2D sc; + float blend[4]; + VkBool32 en[8]; + VkColorBlendEquationEXT eq[8]; + VkColorComponentFlags wm[8]; +}; +static void dynamic_state(VkCommandBuffer cmd, int d) { + DynState s{}; + s.vp = {0, (float)d, 1280, 720, 0, 1}; + s.sc = {{0, d}, {1280, 720}}; + for (int i = 0; i < 4; i++) s.blend[i] = 0.25f * i + d; + for (uint32_t i = 0; i < 8; i++) { + s.en[i] = (i + d) & 1; + s.eq[i] = {VK_BLEND_FACTOR_SRC_ALPHA, VK_BLEND_FACTOR_ONE_MINUS_SRC_ALPHA, VK_BLEND_OP_ADD, VK_BLEND_FACTOR_ONE, VK_BLEND_FACTOR_ZERO, + (VkBlendOp)(i % 3)}; + s.wm[i] = 0xF >> (i & 3); + } + vkCmdSetViewportWithCount(cmd, 1, &s.vp); + vkCmdSetScissorWithCount(cmd, 1, &s.sc); + vkCmdSetBlendConstants(cmd, s.blend); + vkCmdSetStencilReference(cmd, VK_STENCIL_FACE_FRONT_BIT, 1 + d); + vkCmdSetStencilReference(cmd, VK_STENCIL_FACE_BACK_BIT, 2 + d); + vkCmdSetStencilCompareMask(cmd, VK_STENCIL_FACE_FRONT_BIT, 0xFF); + vkCmdSetStencilWriteMask(cmd, VK_STENCIL_FACE_BACK_BIT, 0x0F); + vkCmdSetStencilOp(cmd, VK_STENCIL_FACE_FRONT_BIT, VK_STENCIL_OP_KEEP, VK_STENCIL_OP_REPLACE, VK_STENCIL_OP_INCREMENT_AND_CLAMP, + VK_COMPARE_OP_LESS_OR_EQUAL); + vkCmdSetDepthBias(cmd, 1.5f, 0.0f, -2.0f + d); + vkCmdSetDepthBiasEnable(cmd, VK_TRUE); + if (d == 0) vkCmdSetLineWidth(cmd, 1.0f), vkCmdSetDepthBoundsTestEnable(cmd, VK_FALSE); + vkCmdSetCullMode(cmd, VK_CULL_MODE_BACK_BIT); + vkCmdSetFrontFace(cmd, VK_FRONT_FACE_COUNTER_CLOCKWISE); + vkCmdSetPrimitiveTopology(cmd, VK_PRIMITIVE_TOPOLOGY_TRIANGLE_LIST); + vkCmdSetPrimitiveRestartEnable(cmd, VK_FALSE); + vkCmdSetDepthTestEnable(cmd, VK_TRUE); + vkCmdSetDepthWriteEnable(cmd, d & 1); + vkCmdSetDepthCompareOp(cmd, VK_COMPARE_OP_GREATER); + vkCmdSetStencilTestEnable(cmd, VK_FALSE); + vkCmdSetRasterizerDiscardEnable(cmd, VK_FALSE); + vkCmdSetPolygonModeEXT(cmd, VK_POLYGON_MODE_FILL); + vkCmdSetDepthClampEnableEXT(cmd, VK_FALSE); + vkCmdSetAlphaToCoverageEnableEXT(cmd, d == 2); + vkCmdSetColorBlendEnableEXT(cmd, 0, 3, s.en); + vkCmdSetColorBlendEquationEXT(cmd, 0, 3, s.eq); + vkCmdSetColorWriteMaskEXT(cmd, 0, 3, s.wm); + if (g_mutate) poison(&s); +} + +static void vertex_streams(VkCommandBuffer cmd, int d) { + VkBuffer vbufs[16]; + VkDeviceSize voffs[16], vsizes[16], vstrides[16]; + for (uint32_t i = 0; i < 16; i++) vbufs[i] = H(0x9000 + i), voffs[i] = 64ull * i + d, vsizes[i] = 4096 - i, vstrides[i] = 12 + 4 * (i % 3); + vkCmdBindVertexBuffers2(cmd, 0, 16, vbufs, voffs, vsizes, vstrides); + if (g_mutate) poison(vbufs, 16), poison(voffs, 16), poison(vsizes, 16), poison(vstrides, 16); + VkBuffer one = H(0x9100 + d); + VkDeviceSize off = 32; + vkCmdBindVertexBuffers2(cmd, 3, 1, &one, &off, nullptr, nullptr); // (sizes and strides left out: null stays null) + if (g_mutate) poison(&one), poison(&off); +} + +static void begin_pass(VkCommandBuffer cmd, int pass) { // (vk_draw.cpp's begin_pass: ca[8] and da on the stack) + VkRenderingAttachmentInfo ca[8]{}, da{VK_STRUCTURE_TYPE_RENDERING_ATTACHMENT_INFO}; + for (uint32_t i = 0; i < 8; i++) { + ca[i].sType = VK_STRUCTURE_TYPE_RENDERING_ATTACHMENT_INFO; + ca[i].imageView = H(0xA000 + pass * 16 + i); + ca[i].imageLayout = VK_IMAGE_LAYOUT_GENERAL; + ca[i].loadOp = i ? VK_ATTACHMENT_LOAD_OP_LOAD : VK_ATTACHMENT_LOAD_OP_CLEAR; + ca[i].storeOp = VK_ATTACHMENT_STORE_OP_STORE; + ca[i].clearValue.color.float32[0] = 0.5f + i, ca[i].clearValue.color.float32[3] = 1.0f; + } + da.imageView = H(0xA0FF + pass), da.imageLayout = VK_IMAGE_LAYOUT_GENERAL, da.loadOp = VK_ATTACHMENT_LOAD_OP_CLEAR; + da.storeOp = VK_ATTACHMENT_STORE_OP_STORE, da.clearValue.depthStencil = {0.0f, 7}; + VkRenderingInfo ri{VK_STRUCTURE_TYPE_RENDERING_INFO}; + ri.renderArea = {{0, 0}, {1280u + pass, 720}}; + ri.layerCount = 1; + ri.colorAttachmentCount = 3; + ri.pColorAttachments = ca; + ri.pDepthAttachment = &da; + ri.pStencilAttachment = pass ? &da : nullptr; // (the same struct for both, or no stencil) + vkCmdBeginRendering(cmd, &ri); + if (g_mutate) poison(ca, 8), poison(&da), poison(&ri); +} + +static void barrier(VkCommandBuffer cmd, int k) { // (full_barrier / texture transitions: barriers on the stack) + VkMemoryBarrier2 mb{VK_STRUCTURE_TYPE_MEMORY_BARRIER_2}; + mb.srcStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, mb.srcAccessMask = VK_ACCESS_2_MEMORY_WRITE_BIT; + mb.dstStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, mb.dstAccessMask = VK_ACCESS_2_MEMORY_READ_BIT | k; + VkBufferMemoryBarrier2 bb[2]{{VK_STRUCTURE_TYPE_BUFFER_MEMORY_BARRIER_2}, {VK_STRUCTURE_TYPE_BUFFER_MEMORY_BARRIER_2}}; + for (int i = 0; i < 2; i++) bb[i].buffer = H(0xC000 + i), bb[i].offset = 16ull * k, bb[i].size = VK_WHOLE_SIZE, bb[i].srcQueueFamilyIndex = bb[i].dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED; + VkImageMemoryBarrier2 ib[3]{}; + for (int i = 0; i < 3; i++) { + ib[i].sType = VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER_2; + ib[i].oldLayout = VK_IMAGE_LAYOUT_UNDEFINED, ib[i].newLayout = VK_IMAGE_LAYOUT_GENERAL; + ib[i].image = H(0xD000 + i + k); + ib[i].subresourceRange = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 1 + (uint32_t)i, 0, VK_REMAINING_ARRAY_LAYERS}; + } + VkDependencyInfo di{VK_STRUCTURE_TYPE_DEPENDENCY_INFO}; + di.memoryBarrierCount = 1, di.pMemoryBarriers = &mb; + di.bufferMemoryBarrierCount = 2, di.pBufferMemoryBarriers = bb; + di.imageMemoryBarrierCount = k ? 3 : 0, di.pImageMemoryBarriers = ib; // (a count of 0 with a pointer: not read) + vkCmdPipelineBarrier2(cmd, &di); + if (g_mutate) poison(&mb), poison(bb, 2), poison(ib, 3), poison(&di); +} + +static thread_local std::vector copies; // (vk_memory.cpp's mirror copies: a reused thread-local vector) +static void copy_runs(VkCommandBuffer up, int k) { + copies.clear(); + for (int i = 0; i < 5 - k; i++) copies.push_back({4096ull * i, 8192ull * i + k, 256ull + i}); + vkCmdCopyBuffer(up, H(0xE000), H(0xE001 + k), (uint32_t)copies.size(), copies.data()); + if (g_mutate) poison(copies); +} + +static void frame() { + const VkCommandBuffer cmd = H(0x100), up = H(0x200); // (the recording's and its upload one) + const VkPipelineLayout layout = H(0x300); + vkCmdResetQueryPool(up, H(0x400), 0, 64); + vkCmdWriteTimestamp2(cmd, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, H(0x400), 0); + copy_runs(up, 0); + copy_runs(up, 2); + { + uint8_t data[64]; + for (int i = 0; i < 64; i++) data[i] = (uint8_t)(255 - i); + vkCmdUpdateBuffer(up, H(0xE100), 128, sizeof data, data); + if (g_mutate) poison(data, 64); + } + vkCmdFillBuffer(cmd, H(0xE200), 0, VK_WHOLE_SIZE, 0); + barrier(cmd, 0); + for (int pass = 0; pass < 2; pass++) { + begin_pass(cmd, pass); + for (int d = 0; d < 3; d++) { // (draw(): pipeline, descriptors, push constants, dynamic state, streams, the draw) + const int at = pass * 3 + d; + vkCmdWriteBufferMarkerAMD(cmd, VK_PIPELINE_STAGE_TOP_OF_PIPE_BIT, H(0xF000), 8ull * at, 100 + at); + vkCmdSetCheckpointNV(cmd, (void*)(uintptr_t)(at + 1)); + vkCmdBindPipeline(cmd, VK_PIPELINE_BIND_POINT_GRAPHICS, H(0x800 + at)); + build_graphics(at); + vkCmdPushDescriptorSetKHR(cmd, VK_PIPELINE_BIND_POINT_GRAPHICS, layout, 0, (uint32_t)dw.writes.size(), dw.writes.data()); + { // (bind_descriptors' delta: the changed writes copied into another thread-local vector) + static thread_local std::vector changed; + changed.clear(); + changed.push_back(dw.writes[0]), changed.push_back(dw.writes[4]), changed.push_back(dw.writes[8]); + vkCmdPushDescriptorSetKHR(cmd, VK_PIPELINE_BIND_POINT_GRAPHICS, layout, 0, (uint32_t)changed.size(), changed.data()); + if (g_mutate) poison(changed); + } + dw.after_call(); + push_constants(cmd, layout, at); + dynamic_state(cmd, at); + vertex_streams(cmd, at); + vkCmdBindIndexBuffer(cmd, H(0x9F00), 4ull * at, d ? VK_INDEX_TYPE_UINT16 : VK_INDEX_TYPE_UINT32); + vkCmdBeginQuery(cmd, H(0x401), at, 0); + switch ((pass * 3 + d) % 6) { + case 0: vkCmdDrawIndexed(cmd, 36, 1, 0, -4, 0); vkCmdDraw(cmd, 3, 1, 0, 0); break; + case 1: vkCmdDrawIndirect(cmd, H(0x9E00), 16, 4, 16); vkCmdDrawIndexedIndirect(cmd, H(0x9E00), 64, 2, 20); break; + case 2: vkCmdDrawIndirectCount(cmd, H(0x9E00), 0, H(0x9E01), 4, 8, 16); + vkCmdDrawIndexedIndirectCount(cmd, H(0x9E00), 0, H(0x9E01), 8, 8, 20); break; + case 3: vkCmdDrawMeshTasksEXT(cmd, 4, 1, 1); break; + case 4: vkCmdDrawMeshTasksIndirectEXT(cmd, H(0x9E02), 12, 3, 12); break; + default: vkCmdDrawMeshTasksIndirectCountEXT(cmd, H(0x9E02), 0, H(0x9E01), 12, 5, 12); break; + } + vkCmdEndQuery(cmd, H(0x401), at); + } + { // (a clear inside the pass: a rect and an attachment on the stack; the shader clear's own viewport and scissor) + VkClearAttachment ca{VK_IMAGE_ASPECT_COLOR_BIT, 1, {}}; + ca.clearValue.color.uint32[2] = 0xABCD + pass; + VkClearRect cr{{{0, 0}, {64, 32}}, 0, 1}; + vkCmdClearAttachments(cmd, 1, &ca, 1, &cr); + if (g_mutate) poison(&ca), poison(&cr); + VkViewport v{0, 0, 64, 32, 0, 1}; + VkRect2D rect{{0, 0}, {64, 32}}; + vkCmdSetViewport(cmd, 0, 1, &v); + vkCmdSetScissor(cmd, 0, 1, &rect); + if (g_mutate) poison(&v), poison(&rect); + } + vkCmdEndRendering(cmd); + barrier(cmd, 1 + pass); + } + { // (a compute dispatch: storage image, storage buffer, texel buffer; push constants; descriptor sets with dynamic offsets) + vkCmdBindPipeline(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, H(0x900)); + VkDescriptorImageInfo ii{VK_NULL_HANDLE, H(0x6500), VK_IMAGE_LAYOUT_GENERAL}; + VkDescriptorBufferInfo bi{H(0xB500), 0, VK_WHOLE_SIZE}; + VkBufferView tv = H(0x7500); + VkWriteDescriptorSet w[3] = {write(0, VK_DESCRIPTOR_TYPE_STORAGE_IMAGE, 1), write(1, VK_DESCRIPTOR_TYPE_STORAGE_BUFFER, 1), + write(2, VK_DESCRIPTOR_TYPE_STORAGE_TEXEL_BUFFER, 1)}; + w[0].pImageInfo = &ii, w[1].pBufferInfo = &bi, w[2].pTexelBufferView = &tv; + vkCmdPushDescriptorSetKHR(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, layout, 0, 3, w); + if (g_mutate) poison(&ii), poison(&bi), poison(&tv), poison(w, 3); + uint32_t size[3] = {640, 480, 1}; + vkCmdPushConstants(cmd, layout, VK_SHADER_STAGE_COMPUTE_BIT, 0, 12, size); + if (g_mutate) poison(size, 3); + vkCmdDispatch(cmd, 80, 60, 1); + vkCmdDispatchIndirect(cmd, H(0x9D00), 48); + VkDescriptorSet sets[2] = {H(0x1100), H(0x1101)}; + uint32_t dyn[3] = {256, 512, 768}; + vkCmdBindDescriptorSets(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, layout, 0, 2, sets, 3, dyn); + if (g_mutate) poison(dyn, 3); + vkCmdBindDescriptorSets(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, layout, 2, 1, &sets[1], 0, dyn); // (no dynamic offsets: not read) + if (g_mutate) poison(sets, 2); + } + { // (texture work: uploads, copies, blits, resolves, clears; regions and values on the stack) + VkBufferImageCopy bic[2]{}; + for (int i = 0; i < 2; i++) bic[i] = {1024ull * i, 0, 0, {VK_IMAGE_ASPECT_COLOR_BIT, (uint32_t)i, 0, 1}, {i, 0, 0}, {64u >> i, 64u >> i, 1}}; + vkCmdCopyBufferToImage(up, H(0xE300), H(0xD100), VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, 2, bic); + vkCmdCopyImageToBuffer(cmd, H(0xD101), VK_IMAGE_LAYOUT_GENERAL, H(0xE301), 1, &bic[1]); + if (g_mutate) poison(bic, 2); + VkImageCopy ic{{VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1}, {1, 2, 0}, {VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1}, {3, 4, 0}, {16, 16, 1}}; + vkCmdCopyImage(cmd, H(0xD102), VK_IMAGE_LAYOUT_GENERAL, H(0xD103), VK_IMAGE_LAYOUT_GENERAL, 1, &ic); + if (g_mutate) poison(&ic); + VkImageBlit bl{{VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1}, {{0, 0, 0}, {128, 64, 1}}, {VK_IMAGE_ASPECT_COLOR_BIT, 1, 0, 1}, {{0, 0, 0}, {64, 32, 1}}}; + vkCmdBlitImage(cmd, H(0xD104), VK_IMAGE_LAYOUT_GENERAL, H(0xD105), VK_IMAGE_LAYOUT_GENERAL, 1, &bl, VK_FILTER_LINEAR); + if (g_mutate) poison(&bl); + VkImageResolve rs{{VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1}, {}, {VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1}, {}, {1280, 720, 1}}; + vkCmdResolveImage(cmd, H(0xD106), VK_IMAGE_LAYOUT_GENERAL, H(0xD107), VK_IMAGE_LAYOUT_GENERAL, 1, &rs); + if (g_mutate) poison(&rs); + VkClearColorValue col{}; + col.float32[0] = 0.1f, col.float32[1] = 0.2f, col.float32[2] = 0.3f, col.float32[3] = 1.0f; + VkImageSubresourceRange ranges[2] = {{VK_IMAGE_ASPECT_COLOR_BIT, 0, 1, 0, 1}, {VK_IMAGE_ASPECT_COLOR_BIT, 1, 2, 0, 6}}; + vkCmdClearColorImage(cmd, H(0xD108), VK_IMAGE_LAYOUT_GENERAL, &col, 2, ranges); + if (g_mutate) poison(&col); + VkClearDepthStencilValue ds{1.0f, 0x55}; + VkImageSubresourceRange dr{VK_IMAGE_ASPECT_DEPTH_BIT | VK_IMAGE_ASPECT_STENCIL_BIT, 0, 1, 0, 1}; + vkCmdClearDepthStencilImage(cmd, H(0xD109), VK_IMAGE_LAYOUT_GENERAL, &ds, 1, &dr); + if (g_mutate) poison(&ds), poison(&dr), poison(ranges, 2); + } + vkCmdWriteTimestamp2(cmd, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, H(0x400), 1); +} + +// (a call the packer does not take: a pNext chain, a descriptor type it does not know. Held packets go first, then the direct call) +static void fallbacks() { + const VkCommandBuffer cmd = H(0x100); + vkCmdDraw(cmd, 1, 2, 3, 4); + VkMemoryBarrier2 next{VK_STRUCTURE_TYPE_MEMORY_BARRIER_2}; + VkDependencyInfo di{VK_STRUCTURE_TYPE_DEPENDENCY_INFO}; + di.pNext = &next; + vkCmdPipelineBarrier2(cmd, &di); + vkCmdDraw(cmd, 5, 6, 7, 8); + VkWriteDescriptorSet w = write(0, VK_DESCRIPTOR_TYPE_INLINE_UNIFORM_BLOCK, 16); + vkCmdPushDescriptorSetKHR(cmd, VK_PIPELINE_BIND_POINT_GRAPHICS, H(0x300), 0, 1, &w); + vkCmdDispatch(cmd, 1, 1, 1); +} + +// ---- RX_VK_RECORDER=defer's lifecycle: the recording's two command buffers (its own and the upload one, in pools of their own, as +// vk_replay.cpp's begin_recording / submit_recording), reset, begun, recorded into interleaved, ended, submitted, waited for, twice; +// a query read and a destruction after. Every held command must reach the fake while its command buffer records, none after its end +static const VkDevice kDev = H(0x50); +static const VkCommandPool kPool = H(0x10), kUpPool = H(0x20); +static const VkCommandBuffer kCmd = H(0x100), kUp = H(0x200); +static void lifecycle_reset() { + g_state.clear(), g_violations = 0, g_violation.clear(); + gpu_reset(); + g_pool_of = {{0x100, 0x10}, {0x200, 0x20}, {0x300, 0x60}}; +} +static void recording_begin() { + vkResetCommandPool(kDev, kPool, 0); + vkResetCommandPool(kDev, kUpPool, 0); + VkCommandBufferBeginInfo bi{VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO}; + bi.flags = VK_COMMAND_BUFFER_USAGE_ONE_TIME_SUBMIT_BIT; + vkBeginCommandBuffer(kCmd, &bi); + vkBeginCommandBuffer(kUp, &bi); +} +static void recording_end(int k) { + hold(); // (the recording's commands in flight at its end) + vkEndCommandBuffer(kUp); + vkEndCommandBuffer(kCmd); + VkCommandBufferSubmitInfo cbi[2]{{VK_STRUCTURE_TYPE_COMMAND_BUFFER_SUBMIT_INFO}, {VK_STRUCTURE_TYPE_COMMAND_BUFFER_SUBMIT_INFO}}; + cbi[0].commandBuffer = kUp, cbi[1].commandBuffer = kCmd; + VkSubmitInfo2 si{VK_STRUCTURE_TYPE_SUBMIT_INFO_2}; + si.commandBufferInfoCount = 2, si.pCommandBufferInfos = cbi; + const VkFence fence = H(0x30 + k); + vkQueueSubmit2(H(0x40), 1, &si, fence); + if (g_mutate) poison(cbi, 2), poison(&si); + vkWaitForFences(kDev, 1, &fence, VK_TRUE, UINT64_MAX); +} +static void lifecycle() { + for (int k = 0; k < 2; k++) { + recording_begin(); + frame(); + fallbacks(); // (FIFO around the calls not packed) + frame(); + recording_end(k); + } + uint64_t q[2]; + vkGetQueryPoolResults(kDev, H(0x400), 0, 2, sizeof q, q, 8, VK_QUERY_RESULT_64_BIT); + vkDestroyBuffer(kDev, H(0xE000), nullptr); +} +/// (one recording of 40 frames: past 1 MB of packets held, for the budgets) +static void big() { + recording_begin(); + for (int i = 0; i < 40; i++) frame(); + recording_end(7); +} +/// A recording handed between two threads under the replay's lock (vk_replay.cpp's replay_mu, a DrainingMutex): one begins it and +/// records, lets the lock go and stays alive; the other records more, ends and submits it +static rec::DrainingMutex g_replay_mu; +static void handoff() { + std::atomic step{0}; + std::thread other([&] { + { + std::lock_guard lk(g_replay_mu); + recording_begin(); + hold(); // (its commands in flight when it lets the lock go) + frame(); + } + step = 1; + while (step.load() != 2) std::this_thread::yield(); + }); + while (step.load() != 1) std::this_thread::yield(); + { + std::lock_guard lk(g_replay_mu); + frame(); + recording_end(9); + } + step = 2; + other.join(); +} + +// ---- RX_VK_RECORDER=thread's own (each also run direct, for the reference log) +static const VkCommandPool kPoolB = H(0x60); +static const VkCommandBuffer kCmdB = H(0x300); +static void draws_into(VkCommandBuffer cb, int n, int k) { // (draws with push constants on the stack) + for (int i = 0; i < n; i++) { + uint32_t pc[4] = {(uint32_t)i, (uint32_t)k, 0xABCD, 7}; + vkCmdPushConstants(cb, H(0x300), VK_SHADER_STAGE_VERTEX_BIT, 0, sizeof pc, pc); + if (g_mutate) poison(pc, 4); + vkCmdDraw(cb, 3, 1, (uint32_t)i, (uint32_t)k); + } +} +static void submit(const VkCommandBuffer* cbs, uint32_t n, VkFence fence) { + VkCommandBufferSubmitInfo cbi[2]{{VK_STRUCTURE_TYPE_COMMAND_BUFFER_SUBMIT_INFO}, {VK_STRUCTURE_TYPE_COMMAND_BUFFER_SUBMIT_INFO}}; + for (uint32_t i = 0; i < n; i++) cbi[i].commandBuffer = cbs[i]; + VkSubmitInfo2 si{VK_STRUCTURE_TYPE_SUBMIT_INFO_2}; + si.commandBufferInfoCount = n, si.pCommandBufferInfos = cbi; + vkQueueSubmit2(H(0x40), 1, &si, fence); + if (g_mutate) poison(cbi, 2), poison(&si); + vkWaitForFences(kDev, 1, &fence, VK_TRUE, UINT64_MAX); +} +/// Two recordings at once: A ended and submitted while B, still recording, has commands in flight (A's submit must not pass them) +static void two_recordings() { + recording_begin(); + vkResetCommandPool(kDev, kPoolB, 0); + VkCommandBufferBeginInfo bi{VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO}; + vkBeginCommandBuffer(kCmdB, &bi); + frame(); + draws_into(kCmdB, 40, 0); + vkEndCommandBuffer(kUp); + vkEndCommandBuffer(kCmd); + hold(); // (B's commands in flight at A's submit) + draws_into(kCmdB, 40, 1); + const VkCommandBuffer a[2] = {kUp, kCmd}; + submit(a, 2, H(0x31)); + hold(); + draws_into(kCmdB, 40, 2); + vkEndCommandBuffer(kCmdB); + submit(&kCmdB, 1, H(0x32)); +} +/// A recording given up: its upload pool reset while recording (valid: nothing of it is pending), begun again, recorded, ended +static void reset_while_recording() { + recording_begin(); + copy_runs(kUp, 0); + frame(); + hold(); // (its commands in flight at the pool's reset) + copy_runs(kUp, 2); + vkResetCommandPool(kDev, kUpPool, 0); + VkCommandBufferBeginInfo bi{VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO}; + vkBeginCommandBuffer(kUp, &bi); + copy_runs(kUp, 1); + recording_end(5); +} +/// The replay lock let go by a thread that recorded, then another thread, without the lock, ends and submits the recording: the +/// recorder must be idle once the lock is let go +static void unlock_then_end() { + std::atomic step{0}; + std::thread other([&] { + { + std::lock_guard lk(g_replay_mu); + recording_begin(); + hold(); // (its commands in flight when it lets the lock go) + frame(); + } + step = 1; + while (step.load() != 2) std::this_thread::yield(); + }); + while (step.load() != 1) std::this_thread::yield(); + recording_end(11); + step = 2; + other.join(); +} +/// A packet bigger than the room it is given (with the 2 KB room of the run): made directly, in its place among the others +static void oversize() { + recording_begin(); + draws_into(kCmd, 20, 0); + std::vector big(4096); + for (size_t i = 0; i < big.size(); i++) big[i] = (uint8_t)(i * 13); + vkCmdUpdateBuffer(kUp, H(0xE100), 0, big.size(), big.data()); + if (g_mutate) poison(big.data(), big.size()); + draws_into(kCmd, 20, 1); + recording_end(6); +} +/// The same packet first, the ring empty (the recorder just started, at the ring's start): the ring has room for it, the packet's +/// room has not, so it is still made directly (the room alone decides, not how far the recorder has got) +static void oversize_first() { + recording_begin(); + std::vector big(4096); + for (size_t i = 0; i < big.size(); i++) big[i] = (uint8_t)(i * 7); + vkCmdUpdateBuffer(kUp, H(0xE200), 0, big.size(), big.data()); + if (g_mutate) poison(big.data(), big.size()); + draws_into(kCmd, 20, 2); + recording_end(7); +} +/// Teardown with work queued: g_at_stop runs with frames in flight (the recorder stopped and joined), then more is recorded (the +/// recorder started again), then vkDestroyDevice (stopped again: its boundary) +static void (*g_at_stop)() = nullptr; +static void teardown() { + recording_begin(); + for (int i = 0; i < 4; i++) frame(); + g_at_stop(); + frame(); + recording_end(12); + vkDestroyDevice(kDev, nullptr); +} +/// The status reads (the calls the producer makes without waiting: unordered()) while a recording's commands are in flight, and after +/// its submit: each must see what the direct calls see, the submitted work and nothing recorded since +static void status_reads_at(VkFence prev, bool recording) { + // (recording: a few commands put in flight before each read, so that one read's wait cannot empty the ring for the next. The + // recorder is held for 50 ms from there: a read made without waiting finds them still in flight on a loaded runner too, and + // one that waits is let through once the hold ends) + const auto busy = [&] { + if (!recording) return; + g_hold_until_ns = (int64_t)std::chrono::duration_cast( + std::chrono::steady_clock::now().time_since_epoch()).count() + 50000000; + draws_into(kCmd, 3, 7); + }; + uint64_t v = 0; + busy(); + vkGetSemaphoreCounterValue(kDev, H(0x70), &v); + const VkSemaphore sem = H(0x70); + const uint64_t want = v + 1; // (not reached until the next submit) + VkSemaphoreWaitInfo wi{VK_STRUCTURE_TYPE_SEMAPHORE_WAIT_INFO}; + wi.semaphoreCount = 1, wi.pSemaphores = &sem, wi.pValues = &want; + busy(); + vkWaitSemaphores(kDev, &wi, 0); + wi.pValues = &v; + busy(); + vkWaitSemaphores(kDev, &wi, 0); + busy(); + vkWaitForFences(kDev, 1, &prev, VK_TRUE, 0); + busy(); + vmaInvalidateAllocation(nullptr, (VmaAllocation)(uintptr_t)0xE400, 0, 16); + uint32_t n = 1; + VkCheckpointDataNV cd{VK_STRUCTURE_TYPE_CHECKPOINT_DATA_NV}; + busy(); + vkGetQueueCheckpointDataNV(H(0x40), &n, &cd); + VkDeviceFaultCountsEXT fc{VK_STRUCTURE_TYPE_DEVICE_FAULT_COUNTS_EXT}; + busy(); + vkGetDeviceFaultInfoEXT(kDev, &fc, nullptr); + g_hold_until_ns = 0; +} +static void status_reads() { + for (int k = 0; k < 3; k++) { + recording_begin(); + vkCmdFillBuffer(kCmd, H(0xE400), 0, 16, 0x100 + k); + vkCmdSetCheckpointNV(kCmd, (void*)(uintptr_t)(0x200 + k)); + frame(); + { // (a query read mid-recording, the frame's commands in flight: kept ordered, so in the stream after them) + uint64_t q[2]; + vkGetQueryPoolResults(kDev, H(0x400), 0, 2, sizeof q, q, 8, VK_QUERY_RESULT_64_BIT); + } + frame(); + status_reads_at(H(0x30 + 20 + k - 1), true); + vkCmdFillBuffer(kCmd, H(0xE400), 0, 16, 0x180 + k); + vkCmdSetCheckpointNV(kCmd, (void*)(uintptr_t)(0x280 + k)); + status_reads_at(H(0x30 + 20 + k), true); + recording_end(20 + k); + status_reads_at(H(0x30 + 20 + k), false); + uint64_t q[2]; + vkGetQueryPoolResults(kDev, H(0x400), 0, 2, sizeof q, q, 8, VK_QUERY_RESULT_64_BIT); + } +} +/// A buffer destroyed while a recording that names it is open (valid: that recording is reset, never submitted); every command +/// naming it must have reached the driver before the destruction +static void destroy_recorded() { + recording_begin(); + frame(); + copy_runs(kUp, 0); // (names 0xE001: in flight behind the frame when it is destroyed) + vkDestroyBuffer(kDev, H(0xE001), nullptr); + vkResetCommandPool(kDev, kUpPool, 0); + VkCommandBufferBeginInfo bi{VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO}; + vkBeginCommandBuffer(kUp, &bi); + copy_runs(kUp, 2); + recording_end(14); +} +static int g_stress_frames = 0; +static void stress() { + recording_begin(); + for (int i = 0; i < g_stress_frames; i++) frame(); + recording_end(13); +} +/// --throw: a driver call throws on the recorder thread: the process must end loudly (exit code 3, "FATAL"), never go on +static int throw_test() { + rec::set_mode(rec::THREAD); + std::lock_guard lk(g_replay_mu); + vkCmdDraw(kCmd, 1, 1, 0, 0); + g_throw = true; + vkCmdDraw(kCmd, 2, 1, 0, 0); + vkCmdDraw(kCmd, 3, 1, 0, 0); + rec::boundary(rec::B_vkEndCommandBuffer); + std::printf("the recorder's exception was not reported\n"); + return 0; +} + +// ---- checks +static int failures = 0; +static std::vector run(void (*scenario)(), int mode, bool mutate) { + rec::set_mode(mode); + g_mutate = mutate; + g_log.clear(); + if (mode == rec::THREAD) { // (the producer: the replay lock's holder; its unlock waits for the recorder) + std::lock_guard lk(g_replay_mu); + scenario(); + } else { + scenario(); + } + if (mode == rec::DEFER && !g_log.empty() && scenario == frame) { + std::printf("FAIL deferred: %zu calls reached the driver before the drain\n", g_log.size()); + failures++; + } + rec::drain(); + rec::set_mode(rec::OFF); + return g_log; +} +static void same(const char* what, const std::vector& ref, const std::vector& got) { + for (size_t i = 0; i < std::max(ref.size(), got.size()); i++) { + if (i < ref.size() && i < got.size() && ref[i].op == got[i].op && ref[i].bytes == got[i].bytes) continue; + size_t at = 0; + if (i < ref.size() && i < got.size()) + while (at < ref[i].bytes.size() && at < got[i].bytes.size() && ref[i].bytes[at] == got[i].bytes[at]) at++; + std::printf("MISMATCH %s: call %zu of %zu/%zu: expected %s, got %s (bytes differ from %zu)\n", what, i, ref.size(), got.size(), + i < ref.size() ? ref[i].op.c_str() : "(end)", i < got.size() ? got[i].op.c_str() : "(end)", at); + failures++; + return; + } + size_t bytes = 0; + for (const Call& c : got) bytes += c.bytes.size(); + std::printf("PASS %s: %zu calls, %zu argument bytes identical\n", what, got.size(), bytes); +} + +/// A scenario with the lifecycle tracked: its log, the violations, and the arena's counters for the run (own_thread: on a thread of +/// its own, an arena that starts empty: its aligned blocks are counted against its chunks) +struct Out { + std::vector log; + std::vector reads; + std::map overlapped; + int violations = 0; + std::string violation; + bool escaped = false; + uint64_t failures = 0, budget_drains = 0, fallbacks = 0; + size_t capacity = 0; + long leaked = 0; +}; +static Out run_tracked(void (*scenario)(), int mode, bool mutate, bool own_thread = false, bool lock = true) { + Out o; + const auto body = [&] { + rec::Arena& a = rec::t_arena; + const uint64_t f0 = a.failures, b0 = a.budget_drains, fb0 = a.fallbacks; + const long live0 = g_aligned_live.load() - (long)a.nchunks; + rec::set_mode(mode); + g_mutate = mutate, g_track = true; + lifecycle_reset(); + g_log.clear(); + try { + if (mode == rec::THREAD && lock) { + std::lock_guard lk(g_replay_mu); + scenario(); + } else { + scenario(); + } + } catch (const std::bad_alloc&) { + o.escaped = true; + } + rec::drain(); + g_track = false; + o.log = g_log, o.violations = g_violations, o.violation = g_violation, o.reads = g_reads, o.overlapped = g_reads_overlapped; + o.failures = a.failures - f0, o.budget_drains = a.budget_drains - b0, o.fallbacks = a.fallbacks - fb0, o.capacity = a.capacity; + if (own_thread) o.leaked = g_aligned_live.load() - live0 - (long)a.nchunks; + }; + if (own_thread) { + std::thread t(body); + t.join(); + } else { + body(); + } + rec::set_mode(rec::OFF); + return o; +} +static void check_run(const char* what, const std::vector& ref, const Out& o, const std::vector* ref_reads = nullptr) { + if (ref_reads && *ref_reads != o.reads) + std::printf("MISMATCH %s: the status reads (fences, semaphores, checkpoints, fault data, invalidated memory) saw other " + "values: %zu/%zu\n", what, ref_reads->size(), o.reads.size()), failures++; + if (o.escaped) std::printf("FAIL %s: an allocation failure escaped a vkCmd* (the call lost)\n", what), failures++; + if (o.violations) std::printf("FAIL %s: %d lifecycle violation(s), the first: %s\n", what, o.violations, o.violation.c_str()), failures++; + same(what, ref, o.log); +} + +static double now_ns() { return (double)std::chrono::duration_cast(std::chrono::steady_clock::now().time_since_epoch()).count(); } + +// ---- bench: a typical draw (~4 calls: the descriptor delta, push constants, streams, the draw) and a cold one (the whole state) +static void draw_typical(VkCommandBuffer cmd, VkPipelineLayout layout, int d) { + static thread_local std::vector changed; + changed.clear(); + changed.push_back(dw.writes[0]), changed.push_back(dw.writes[3]), changed.push_back(dw.writes[4]); + vkCmdPushDescriptorSetKHR(cmd, VK_PIPELINE_BIND_POINT_GRAPHICS, layout, 0, 3, changed.data()); + uint8_t data[kPushSize]; + std::memset(data, d, sizeof data); + vkCmdPushConstants(cmd, layout, VK_SHADER_STAGE_ALL_GRAPHICS, 0, kPushSize, data); + VkBuffer vb[16]; + VkDeviceSize vo[16], vs[16], vst[16]; + for (int i = 0; i < 16; i++) vb[i] = H(0x9000 + i), vo[i] = i + d, vs[i] = 4096, vst[i] = 16; + vkCmdBindVertexBuffers2(cmd, 0, 16, vb, vo, vs, vst); + vkCmdDrawIndexed(cmd, 36, 1, 0, d & 7, 0); +} +static void draw_cold(VkCommandBuffer cmd, VkPipelineLayout layout, int d) { + vkCmdBindPipeline(cmd, VK_PIPELINE_BIND_POINT_GRAPHICS, H(0x800 + (d & 3))); + vkCmdPushDescriptorSetKHR(cmd, VK_PIPELINE_BIND_POINT_GRAPHICS, layout, 0, (uint32_t)dw.writes.size(), dw.writes.data()); + DynState s{}; + s.vp = {0, 0, 1280, 720, 0, 1}, s.sc = {{0, 0}, {1280, 720}}, s.en[0] = d & 1, s.wm[0] = 0xF; + vkCmdSetViewportWithCount(cmd, 1, &s.vp); + vkCmdSetScissorWithCount(cmd, 1, &s.sc); + vkCmdSetCullMode(cmd, VK_CULL_MODE_NONE); + vkCmdSetDepthTestEnable(cmd, VK_TRUE); + vkCmdSetDepthWriteEnable(cmd, d & 1); + vkCmdSetDepthCompareOp(cmd, VK_COMPARE_OP_GREATER); + vkCmdSetColorBlendEnableEXT(cmd, 0, 1, s.en); + vkCmdSetColorBlendEquationEXT(cmd, 0, 1, s.eq); + vkCmdSetColorWriteMaskEXT(cmd, 0, 1, s.wm); + vkCmdBindIndexBuffer(cmd, H(0x9F00), 0, VK_INDEX_TYPE_UINT16); + draw_typical(cmd, layout, d); +} +static void draw_typical_direct(VkCommandBuffer cmd, VkPipelineLayout layout, int d) { // (no wrapper: the driver's functions, as before) + static thread_local std::vector changed; + changed.clear(); + changed.push_back(dw.writes[0]), changed.push_back(dw.writes[3]), changed.push_back(dw.writes[4]); + (::vkx::vkCmdPushDescriptorSetKHR)(cmd, VK_PIPELINE_BIND_POINT_GRAPHICS, layout, 0, 3, changed.data()); + uint8_t data[kPushSize]; + std::memset(data, d, sizeof data); + (::vkx::vkCmdPushConstants)(cmd, layout, VK_SHADER_STAGE_ALL_GRAPHICS, 0, kPushSize, data); + VkBuffer vb[16]; + VkDeviceSize vo[16], vs[16], vst[16]; + for (int i = 0; i < 16; i++) vb[i] = H(0x9000 + i), vo[i] = i + d, vs[i] = 4096, vst[i] = 16; + (::vkx::vkCmdBindVertexBuffers2)(cmd, 0, 16, vb, vo, vs, vst); + (::vkx::vkCmdDrawIndexed)(cmd, 36, 1, 0, d & 7, 0); +} +static int bench(int n) { + if (const char* e = getenv("BENCH_PUBLISH_BYTES")) rec::g_publish_bytes = (size_t)atoll(e); // (the publication batch to time) + g_quiet = true; + build_graphics(0); + const VkCommandBuffer cmd = H(0x100); + const VkPipelineLayout layout = H(0x300); + struct Shape { + const char* name; + void (*draw)(VkCommandBuffer, VkPipelineLayout, int); + void (*direct)(VkCommandBuffer, VkPipelineLayout, int); + int calls; + } shapes[] = {{"typical draw (push descriptors x3 writes, push constants 96 B, vertex buffers x16, draw)", draw_typical, + draw_typical_direct, 4}, + {"cold draw (+ pipeline, 9 writes, 9 dynamic states, index buffer)", draw_cold, nullptr, 16}}; + rec::Arena& a = rec::t_arena; + for (const Shape& sh : shapes) { + // (modes: 0 direct (no wrapper), 1 off (the wrapper's branch), 2 sync, 3 deferred: encode and drain timed apart, 4 thread: + // the producer's encode and publish, and its wait at the boundary, timed apart; the recorder's busy time a packet) + double med[7] = {}; + double bytes_per_packet = 0, rec_ns = 0; + for (int m = 0; m < 5; m++) { + if (m == 0 && !sh.direct) continue; + std::vector reps[3]; + for (int rep = 0; rep < 5; rep++) { + rec::set_mode(m <= 1 ? rec::OFF : m == 2 ? rec::SYNC : m == 3 ? rec::DEFER : rec::THREAD); + const uint64_t p0 = a.packets, b0 = a.bytes; + const uint64_t rp0 = rec::g_rec.packets.load(), rb0 = rec::g_rec.busy_ns.load(); + double enc = 0, drn = 0; + { + std::unique_lock lk(g_replay_mu, std::defer_lock); + if (m == 4) lk.lock(); + for (int i = 0; i < n; i += 256) { // (deferred: drained every 256 draws, about a recording's draws; thread: waited) + const double t0 = now_ns(); + for (int j = 0; j < 256; j++) (m == 0 ? sh.direct : sh.draw)(cmd, layout, i + j); + const double t1 = now_ns(); + if (m == 4) rec::boundary(rec::B_vkEndCommandBuffer); + else rec::drain(); + enc += t1 - t0, drn += now_ns() - t1; + } + } + reps[0].push_back(enc / n), reps[1].push_back(drn / n); + if (m == 3) bytes_per_packet = (double)(a.bytes - b0) / (double)(a.packets - p0); + if (m == 4) { + rec::set_mode(rec::OFF); // (the recorder idle: its counters are in) + reps[2].push_back((double)(rec::g_rec.busy_ns.load() - rb0) / (double)(rec::g_rec.packets.load() - rp0)); + } + } + for (auto& r : reps) std::sort(r.begin(), r.end()); + med[m] = reps[0][2]; + if (m == 3) med[5] = reps[1][2]; + if (m == 4) med[6] = reps[1][2], rec_ns = reps[2][2]; + } + std::printf("BENCH %s, %d calls: ", sh.name, sh.calls); + if (sh.direct) std::printf("direct %.1f ns, ", med[0]); + std::printf("off %.1f ns, sync %.1f ns (+%.1f over off, +%.1f a call), deferred encode %.1f ns + drain %.1f ns; " + "%.0f bytes a packet (median of 5, %d draws)\n", + med[1], med[2], med[2] - med[1], (med[2] - med[1]) / sh.calls, med[3], med[5], bytes_per_packet, n); + std::printf("BENCH thread: the replay thread %.1f ns a draw (+%.1f over off, %.1f a call), its wait at the boundary every 256 " + "draws %.1f ns a draw; the recorder %.1f ns a packet busy (%.1f M packets/s)\n", + med[4], med[4] - med[1], med[4] / sh.calls, med[6], rec_ns, rec_ns > 0 ? 1e3 / rec_ns : 0.0); + } + rec::set_mode(rec::OFF); + std::printf("BENCH %llu fake calls, arena %zu chunk(s)\n", (unsigned long long)g_calls, a.nchunks); + g_quiet = false; + return 0; +} + +/// --overlap F: the critical thread's time a frame, off vs thread, with a driver that takes D ns a vkCmd* (4 calls a draw), translation +/// of W ns a draw, 2000 draws a frame, and every K draws a recording's end (the producer waits for the recorder) plus 30 us that stays +/// on the replay thread in both (vkQueueSubmit2, the next recording's begin) +static void spin_ns_busy(int ns) { spin_for(ns); } +static int overlap(int frames) { + g_quiet = true; + build_graphics(0); + const VkCommandBuffer cmd = H(0x100); + const VkPipelineLayout layout = H(0x300); + const int draws = 2000, D = 300, W = 4000; + g_spin_ns = D; + std::printf("OVERLAP %d draws a frame, driver %d ns a call (%.2f ms a frame), translation %d ns a draw, 30 us a recording end " + "(median of 5, %d frames)\n", draws, D, draws * 4 * D / 1e6, W, frames); + for (int K : {120, 30, 8, 1}) { + double med[2]; + uint64_t waits = 0; + double waited = 0; + for (int m = 0; m < 2; m++) { + std::vector reps; + for (int rep = 0; rep < 5; rep++) { + rec::set_mode(m ? rec::THREAD : rec::OFF); + const uint64_t n0 = rec::g_rec.wait_n[rec::B_vkEndCommandBuffer].load(), ns0 = rec::g_rec.wait_ns[rec::B_vkEndCommandBuffer].load(); + const double t0 = now_ns(); + for (int f = 0; f < frames; f++) { + std::lock_guard lk(g_replay_mu); + for (int d = 0; d < draws; d++) { + spin_ns_busy(W); + draw_typical(cmd, layout, d); + if ((d + 1) % K == 0 || d + 1 == draws) { + rec::boundary(rec::B_vkEndCommandBuffer); + spin_ns_busy(30000); + } + } + } + reps.push_back((now_ns() - t0) / frames / 1e6); + if (m) waits = rec::g_rec.wait_n[rec::B_vkEndCommandBuffer].load() - n0, waited = (double)(rec::g_rec.wait_ns[rec::B_vkEndCommandBuffer].load() - ns0) / frames / 1e6; + } + std::sort(reps.begin(), reps.end()); + med[m] = reps[2]; + } + std::printf("OVERLAP every %4d draws (%4d waits a frame): off %.2f ms, thread %.2f ms a frame: %.2f ms saved of %.2f ms of " + "driver time (%.0f%%); waited %.3f ms a frame, %llu of the last run's waits found the recorder busy\n", + K, (draws + K - 1) / K, med[0], med[1], med[0] - med[1], draws * 4 * D / 1e6, 100 * (med[0] - med[1]) / (draws * 4 * D / 1e6), + waited, (unsigned long long)waits); + } + rec::set_mode(rec::OFF); + g_spin_ns = 0; + g_quiet = false; + return 0; +} + +/// --idle N: the recorder's idle spin (RX_VK_RECORDER_SPIN_US) against its CPU and the replay thread's waits. (a) bursts of one +/// typical draw after a gap of G us of translation, each followed by a boundary: the replay thread's wait there (it includes the +/// recorder's wake-up when it slept in the gap) and its encode time (it includes the wake-up call); (b) replay-like frames: 2000 +/// draws of 4 us of translation, a recording end every 120, a 5 ms gap between frames, the fake driver 300 ns a call: the +/// recorder's CPU (its thread's times, and its own spun/busy count) against the wall time, and the replay thread's frame time +// (native_handle() is a Windows HANDLE with MSVC's and libc++'s std::thread, an integer pthread id with MinGW's +// libstdc++ (winpthreads): the thread's CPU only where it is a HANDLE. A template, so the other branch is not compiled) +template static bool thread_cpu_ticks(H h, uint64_t* kernel, uint64_t* user) { +#ifdef _WIN32 + if constexpr (std::is_pointer_v) { + uint64_t c, e; + return GetThreadTimes(h, &c, &e, kernel, user) != 0; + } +#endif + (void)h, (void)kernel, (void)user; + return false; +} +static double recorder_cpu_s() { + uint64_t k, u; + if (rec::g_rec.th.joinable() && thread_cpu_ticks(rec::g_rec.th.native_handle(), &k, &u)) return (double)(k + u) / 1e7; + return -1; // (not measured) +} +static int idle_bench(int iters) { + g_quiet = true; + build_graphics(0); + rec::g_stats_log = false; + const VkCommandBuffer cmd = H(0x100); + const VkPipelineLayout layout = H(0x300); + for (const uint64_t spin_us : {200ull, 20ull, 0ull}) { // (0: adaptive, the default) + rec::g_rec.stop(); + rec::g_spin_ns = spin_us * 1000; + rec::set_mode(rec::THREAD); + { // (start it, idle) + std::lock_guard lk(g_replay_mu); + draw_typical(cmd, layout, 0); + } + for (const int gap : {2, 10, 30, 100, 1000}) { + std::vector waits, encs; + const uint64_t sl0 = rec::g_rec.sleeps.load(), sp0 = rec::g_rec.spin_ns.load(), bu0 = rec::g_rec.busy_ns.load(); + const int n = std::max(iters, 300000 / gap); // (at least ~0.3 s) + const double c0 = recorder_cpu_s(), t0 = now_ns(); + { + std::lock_guard lk(g_replay_mu); + for (int i = 0; i < n; i++) { + spin_for(gap * 1000); + const double a = now_ns(); + draw_typical(cmd, layout, i); + const double b = now_ns(); + rec::boundary(rec::B_vkEndCommandBuffer); + encs.push_back(b - a), waits.push_back(now_ns() - b); + } + } + const double wall = (now_ns() - t0) / 1e9, cpu = recorder_cpu_s() - c0; + std::sort(waits.begin(), waits.end()), std::sort(encs.begin(), encs.end()); + double wsum = 0, esum = 0; + for (double w : waits) wsum += w; + for (double e : encs) esum += e; + std::printf("IDLE spin %s us, gap %4d us: boundary wait mean %.2f us, p99 %.2f, max %.1f; encode (4 calls) mean %.0f ns, " + "p99 %.0f; recorder slept %.2f times a burst, CPU %.2f of a core (its own count; thread times %.2f)\n", + spin_us ? std::to_string(spin_us).c_str() : "adaptive", gap, wsum / n / 1e3, waits[n * 99 / 100] / 1e3, + waits.back() / 1e3, esum / n, encs[n * 99 / 100], (double)(rec::g_rec.sleeps.load() - sl0) / n, + (double)(rec::g_rec.spin_ns.load() - sp0 + rec::g_rec.busy_ns.load() - bu0) / 1e9 / wall, cpu / wall); + } + { // (b) frames + const int frames = 30, draws = 2000; + g_spin_ns = 300; + const uint64_t sp0 = rec::g_rec.spin_ns.load(), bu0 = rec::g_rec.busy_ns.load(), sl0 = rec::g_rec.sleeps.load(); + const uint64_t wn0 = rec::g_rec.wait_n[rec::B_vkEndCommandBuffer].load(), wt0 = rec::g_rec.wait_ns[rec::B_vkEndCommandBuffer].load(); + rec::g_rec.wait_max[rec::B_vkEndCommandBuffer] = 0; + const double c0 = recorder_cpu_s(), t0 = now_ns(); + double replay = 0; + for (int f = 0; f < frames; f++) { + const double a = now_ns(); + { + std::lock_guard lk(g_replay_mu); + for (int d = 0; d < draws; d++) { + spin_for(d % 50 == 49 ? 64000 : 4000); // (and a 60 us stretch of translation every 50 draws) + draw_typical(cmd, layout, d); + if ((d + 1) % 120 == 0 || d + 1 == draws) rec::boundary(rec::B_vkEndCommandBuffer); + } + } + replay += now_ns() - a; + std::this_thread::sleep_for(std::chrono::milliseconds(5)); + } + g_spin_ns = 0; + const double wall = (now_ns() - t0) / 1e9, cpu = recorder_cpu_s() - c0; + std::printf("IDLE spin %s us, frames (2000 draws x 4 us, 60 us more every 50, an end every 120, 5 ms between): replay %.2f ms a frame, waited " + "%.3f ms a frame (max %.1f us); recorder CPU %.2f of a core by thread times (busy %.2f, spun %.2f s of %.2f s), " + "slept %llu times a frame\n", + spin_us ? std::to_string(spin_us).c_str() : "adaptive", replay / frames / 1e6, + (double)(rec::g_rec.wait_ns[rec::B_vkEndCommandBuffer].load() - wt0) / frames / 1e6, + rec::g_rec.wait_max[rec::B_vkEndCommandBuffer].load() / 1e3, cpu / wall, + (double)(rec::g_rec.busy_ns.load() - bu0) / 1e9, (double)(rec::g_rec.spin_ns.load() - sp0) / 1e9, wall, + (unsigned long long)(rec::g_rec.sleeps.load() - sl0) / frames); + (void)wn0; + } + rec::set_mode(rec::OFF); + } + rec::g_rec.stop(); + g_quiet = false; + return 0; +} + +/// RX_VK_RECORDER=thread: the recorder thread's checks (also run alone, --only thread, for the broken variants) +static void thread_checks(const std::vector& ref) { + rec::g_stats_log = false; + const bool was_poison = rec::g_poison.load(); + rec::g_poison = true; // (each packet filled with 0xCD once run: a pointer kept into a run packet would show) + // the frame, every vkCmd* run by the recorder thread + { + const uint64_t c0 = g_cmd_calls, off0 = g_cmd_off_main; + same("thread (packed, run by the recorder thread in order), caller memory poisoned", ref, run(frame, rec::THREAD, true)); + const uint64_t c = g_cmd_calls - c0, off = g_cmd_off_main - off0; + if (c != ref.size() || off != c) std::printf("FAIL thread: %llu of %llu vkCmd* on the recorder thread\n", (unsigned long long)off, (unsigned long long)c), failures++; + else std::printf("PASS thread: all %llu vkCmd* made on the recorder thread\n", (unsigned long long)off); + } + g_delay_ns = 1000; // (the recorder behind the producer: a missed wait shows as a call out of place or on a busy command buffer) + const Out lref = run_tracked(lifecycle, rec::OFF, false); + const uint64_t fb0 = rec::t_arena.fallbacks; + g_hold_ms = 50; // (and held at each point the producer must wait: hold()) + const Out lt = run_tracked(lifecycle, rec::THREAD, true); + g_hold_ms = 0, g_hold_until_ns = 0; + check_run("thread lifecycle (2 command buffers interleaved; reset, begin, end, submit, wait x2; query read; destruction; caller " + "memory poisoned; driver 1 us a call)", lref.log, lt); + if (rec::t_arena.fallbacks - fb0 != 4) std::printf("FAIL thread lifecycle: %llu fallbacks, 4 expected\n", (unsigned long long)(rec::t_arena.fallbacks - fb0)), failures++; + // (each thread run with the recorder held at the points the producer must wait: hold(); the direct references without) + const auto held = [](void (*scenario)(), bool lock) { + g_hold_ms = 50; + Out o = run_tracked(scenario, rec::THREAD, true, false, lock); + g_hold_ms = 0, g_hold_until_ns = 0; + return o; + }; + check_run("thread handoff (a recording across two threads under the replay lock)", run_tracked(handoff, rec::OFF, false).log, + held(handoff, false)); + check_run("thread: a submit while another command buffer records (its commands in flight)", run_tracked(two_recordings, rec::OFF, false).log, + held(two_recordings, true)); + check_run("thread: a pool reset while its command buffer records (re-begun)", run_tracked(reset_while_recording, rec::OFF, false).log, + held(reset_while_recording, true)); + check_run("thread: the replay lock let go, then the recording ended on another thread without it", + run_tracked(unlock_then_end, rec::OFF, false).log, held(unlock_then_end, false)); + { // (the status reads made without waiting: the same values as direct, each made while packets were in flight) + const Out sref = run_tracked(status_reads, rec::OFF, false); + const Out st = run_tracked(status_reads, rec::THREAD, true); + check_run("thread: status reads mid-recording (semaphore value and wait, fence wait, invalidate, checkpoints, fault info) " + "without waiting for the recorder", sref.log, st, &sref.reads); + // (each made without waiting exactly when rec::unordered() lists it: then with packets in flight; else always after them) + const std::pair calls[] = { + {"vkGetSemaphoreCounterValue", rec::B_vkGetSemaphoreCounterValue}, {"vkWaitSemaphores", rec::B_vkWaitSemaphores}, + {"vkWaitForFences", rec::B_vkWaitForFences}, {"vmaInvalidateAllocation", rec::B_vmaInvalidateAllocation}, + {"vkGetQueueCheckpointDataNV", rec::B_vkGetQueueCheckpointDataNV}, {"vkGetDeviceFaultInfoEXT", rec::B_vkGetDeviceFaultInfoEXT}}; + std::string ran, waited; + for (const auto& [c, b] : calls) { + const auto it = st.overlapped.find(c); + const uint64_t n = it == st.overlapped.end() ? 0 : it->second; + if (rec::unordered(b) && !n) + std::printf("FAIL thread status reads: %s (in unordered()) always waited for the recorder\n", c), failures++; + if (!rec::unordered(b) && n) + std::printf("FAIL thread status reads: %s made %llu times with packets in flight: it must wait\n", c, (unsigned long long)n), failures++; + std::string& to = rec::unordered(b) ? ran : waited; + to += std::string(to.empty() ? "" : ", ") + c + (rec::unordered(b) ? " " + std::to_string(n) : std::string()); + } + if (sref.reads.size() < 60) std::printf("FAIL thread status reads: only %zu values read\n", sref.reads.size() / 2), failures++; + std::printf("PASS thread status reads: %zu values identical to direct; made with packets in flight: %s; waited: %s\n", + st.reads.size() / 2, ran.empty() ? "(none)" : ran.c_str(), waited.empty() ? "(none)" : waited.c_str()); + check_run("thread: a buffer destroyed while a recording names it (the commands naming it run first)", + run_tracked(destroy_recorded, rec::OFF, false).log, run_tracked(destroy_recorded, rec::THREAD, true)); + } + { // (teardown with work queued: stopped, joined, started again; vkDestroyDevice stops it too) + static size_t at_stop; + static bool joined_at_stop; + g_at_stop = [] { at_stop = g_log.size(); }; + const Out tref = run_tracked(teardown, rec::OFF, false); + const size_t ref_at = at_stop; + g_at_stop = [] { + rec::g_rec.stop(); + std::lock_guard lk(g_log_mu); + at_stop = g_log.size(), joined_at_stop = !rec::g_rec.th.joinable(); + if (!joined_at_stop) std::printf("FAIL thread teardown: stop() returned with the recorder thread not joined\n"), failures++; + }; + const uint64_t want_starts = rec::g_rec.ready.load() ? 1 : 2; // (running: started again after the stop; else at the first packet too) + const uint64_t starts0 = rec::g_rec.starts.load(); + const Out t = run_tracked(teardown, rec::THREAD, true); + check_run("thread teardown (stopped with 4 frames queued, started again, stopped at vkDestroyDevice)", tref.log, t); + const uint64_t starts = rec::g_rec.starts.load() - starts0; + if (at_stop != ref_at || !joined_at_stop || rec::g_rec.th.joinable() || starts != want_starts) + std::printf("FAIL thread teardown: %zu of %zu calls in when stop() returned, joined %d, joinable after vkDestroyDevice %d, " + "started %llu times (%llu expected)\n", at_stop, ref_at, (int)joined_at_stop, (int)rec::g_rec.th.joinable(), + (unsigned long long)starts, (unsigned long long)want_starts), failures++; + else std::printf("PASS thread teardown: all %zu queued calls run when stop() returned, the thread joined; started again lazily; " + "joined at vkDestroyDevice\n", at_stop); + } + { // (backpressure: an 8 KB ring, 2 KB of room a packet) + rec::g_rec.stop(); + const size_t bytes = rec::g_ring_bytes, room = rec::g_ring_room; + rec::g_ring_bytes = 8192, rec::g_ring_room = 2048; + const uint64_t full0 = rec::g_rec.wait_n[rec::B_RING_FULL].load(), d0 = rec::g_rec.direct.load(); + check_run("thread, an 8 KB ring (the producer waits for room)", lref.log, run_tracked(lifecycle, rec::THREAD, true)); + const uint64_t full = rec::g_rec.wait_n[rec::B_RING_FULL].load() - full0; + if (!full) std::printf("FAIL thread 8 KB ring: the producer never waited for room\n"), failures++; + else std::printf("PASS thread 8 KB ring: the producer waited for room %llu times\n", (unsigned long long)full); + const Out oref = run_tracked(oversize, rec::OFF, false); + check_run("thread, a packet bigger than its room (4 KB vkCmdUpdateBuffer, 2 KB room): made directly in order", oref.log, + run_tracked(oversize, rec::THREAD, true)); + if (rec::g_rec.direct.load() - d0 != 1) std::printf("FAIL thread oversize: %llu calls made directly, 1 expected\n", (unsigned long long)(rec::g_rec.direct.load() - d0)), failures++; + // (again with the ring empty: stopped, its counters moved on to the ring's start, so the 8 KB free ahead of the first + // packet would hold it; the 2 KB room must still decide) + rec::g_rec.stop(); + const uint64_t at = (rec::g_rec.W.load() + 8191) & ~uint64_t(8191); + rec::g_rec.W.store(at), rec::g_rec.R.store(at); + const uint64_t d1 = rec::g_rec.direct.load(); + check_run("thread, the same packet first with the ring empty: made directly in order", + run_tracked(oversize_first, rec::OFF, false).log, run_tracked(oversize_first, rec::THREAD, true)); + if (rec::g_rec.direct.load() - d1 != 1) + std::printf("FAIL thread oversize, ring empty: %llu calls made directly, 1 expected\n", + (unsigned long long)(rec::g_rec.direct.load() - d1)), failures++; + rec::g_rec.stop(); + rec::g_ring_bytes = bytes, rec::g_ring_room = room; + } + { // (no memory for the ring: every call made directly, logged once) + rec::g_rec.stop(); + g_aligned_calls = 0, g_fail_nth = 1; + const Out o = run_tracked(lifecycle, rec::THREAD, true); + g_fail_nth = 0; + check_run("thread, no memory for the ring (every call made directly)", lref.log, o); + if (!rec::g_rec.broken.load()) std::printf("FAIL thread no ring: the recorder ran\n"), failures++; + rec::g_rec.broken = false; + } + { // (a parallel replay records: each packet made at once, after what was published) + par_active = true; + const std::vector got = run(frame, rec::THREAD, true); + par_active = false; + same("thread while par_active (each call made directly)", ref, got); + } + g_delay_ns = 0; + { // (stress: >= 1e6 packets through a 16 KB ring, twice; each run's stream hashed, against the direct one's) + rec::g_rec.stop(); + const size_t bytes = rec::g_ring_bytes, room = rec::g_ring_room; + rec::g_ring_bytes = 16384, rec::g_ring_room = 4096; + g_stress_frames = (int)(1000000 / ref.size()) + 1; + uint64_t h[3]; + uint64_t calls = 0, full = 0; + for (int i = 0; i < 3; i++) { + g_hash_only = true, g_hash = 0xcbf29ce484222325ull; + const uint64_t c0 = g_cmd_off_main, f0 = rec::g_rec.wait_n[rec::B_RING_FULL].load(); + const Out o = run_tracked(stress, i ? rec::THREAD : rec::OFF, i != 0); + g_hash_only = false; + h[i] = g_hash; + if (o.violations) std::printf("FAIL thread stress: %s\n", o.violation.c_str()), failures++; + if (i) calls += g_cmd_off_main - c0, full += rec::g_rec.wait_n[rec::B_RING_FULL].load() - f0; + } + if (h[1] != h[0] || h[2] != h[0] || calls < 2000000) + std::printf("FAIL thread stress: hashes %016llx (direct) %016llx %016llx, %llu calls on the recorder\n", (unsigned long long)h[0], + (unsigned long long)h[1], (unsigned long long)h[2], (unsigned long long)calls), failures++; + else std::printf("PASS thread stress: 2 runs of %llu packets through a 16 KB ring (%llu waits for room), each stream identical " + "to the direct one (hash %016llx)\n", (unsigned long long)(calls / 2), (unsigned long long)full, (unsigned long long)h[0]); + rec::g_rec.stop(); + rec::g_ring_bytes = bytes, rec::g_ring_room = room; + } + rec::g_poison = was_poison; +} + +int main(int argc, char** argv) { +#ifdef _WIN32 + SetErrorMode(0x0002 | 0x0001); // (SEM_NOGPFAULTERRORBOX | SEM_FAILCRITICALERRORS) +#endif + std::setvbuf(stdout, nullptr, _IONBF, 0); // (a broken variant may end the process: what it printed before is kept) + // (the switch as the renderer reads it: first use, from the environment) + std::printf("env mode %d\n", rec::mode()); + const int env_mode = rec::mode(); + g_main_id = std::this_thread::get_id(); + if (argc > 2 && !std::strcmp(argv[1], "--bench")) return bench(std::atoi(argv[2])); + if (argc > 2 && !std::strcmp(argv[1], "--overlap")) return overlap(std::atoi(argv[2])); + if (argc > 2 && !std::strcmp(argv[1], "--idle")) return idle_bench(std::atoi(argv[2])); + if (argc > 1 && !std::strcmp(argv[1], "--throw")) return throw_test(); + const bool only_thread = argc > 2 && !std::strcmp(argv[1], "--only") && !std::strcmp(argv[2], "thread"); + + const std::vector ref = run(frame, rec::OFF, false); + if (only_thread) { + thread_checks(ref); + std::printf(failures ? "FAILED (%d)\n" : "ALL PASS\n", failures); + return failures ? 1 : 0; + } + same("direct, caller memory reused after each call", ref, run(frame, rec::OFF, true)); + if (env_mode != rec::OFF) same("RX_VK_RECORDER from the environment, caller memory reused", ref, run(frame, env_mode, true)); + rec::t_arena.packets = 0; + same("sync (packed, run at once), caller memory reused", ref, run(frame, rec::SYNC, true)); + if (rec::t_arena.packets != ref.size()) std::printf("FAIL sync: %llu packets for %zu calls\n", (unsigned long long)rec::t_arena.packets, ref.size()), failures++; + if (rec::t_arena.ci || rec::t_arena.used) std::printf("FAIL sync: the arena not rewound\n"), failures++; + same("deferred (packed, run at the drain), caller memory poisoned in between", ref, run(frame, rec::DEFER, true)); + if (rec::t_arena.fallbacks) std::printf("FAIL: %llu calls not packed\n", (unsigned long long)rec::t_arena.fallbacks), failures++; + + const std::vector fref = run(fallbacks, rec::OFF, false); + same("fallback order (pNext chain, inline uniform block), deferred", fref, run(fallbacks, rec::DEFER, false)); + if (rec::t_arena.fallbacks != 2) std::printf("FAIL: %llu fallbacks, 2 expected\n", (unsigned long long)rec::t_arena.fallbacks), failures++; + + // ---- defer (step 1.5): the lifecycle, the replay lock, the budgets, an allocation failure, a parallel replay + const Out lref = run_tracked(lifecycle, rec::OFF, false); + if (lref.violations) std::printf("FAIL the lifecycle scenario itself: %s\n", lref.violation.c_str()), failures++; + const Out ld = run_tracked(lifecycle, rec::DEFER, true); + check_run("defer lifecycle (2 command buffers interleaved; reset, begin, end, submit, wait x2; query read; destruction; caller memory " + "poisoned)", lref.log, ld); + if (ld.fallbacks != 4) std::printf("FAIL defer lifecycle: %llu fallbacks, 4 expected\n", (unsigned long long)ld.fallbacks), failures++; + const Out href = run_tracked(handoff, rec::OFF, false); + check_run("defer handoff (a recording across two threads under the replay lock)", href.log, run_tracked(handoff, rec::DEFER, true)); + { + const size_t soft = rec::g_soft_budget; + rec::g_soft_budget = 2048; + const Out o = run_tracked(lifecycle, rec::DEFER, true); + rec::g_soft_budget = soft; + check_run("defer soft budget 2 KB (held packets drained first past it)", lref.log, o); + if (!o.budget_drains) std::printf("FAIL defer soft budget: no drain at the budget\n"), failures++; + else std::printf("PASS defer soft budget: %llu drains at the budget\n", (unsigned long long)o.budget_drains); + } + const Out bref = run_tracked(big, rec::OFF, false); + { + const size_t hard = rec::g_hard_budget; + rec::g_hard_budget = size_t(1) << 20; + const Out o = run_tracked(big, rec::DEFER, true, true); + rec::g_hard_budget = hard; + check_run("defer hard budget 1 MB (past it: what is held drained, the call made directly)", bref.log, o); + if (!o.failures || o.capacity > (size_t(1) << 20)) + std::printf("FAIL defer hard budget: %llu calls past it, %zu bytes of chunks\n", (unsigned long long)o.failures, o.capacity), failures++; + else std::printf("PASS defer hard budget: %llu calls made directly, %zu bytes of chunks\n", (unsigned long long)o.failures, o.capacity); + } + { + const long live = g_aligned_live.load(); + g_aligned_calls = 0, g_fail_nth = 2; + const Out o = run_tracked(big, rec::DEFER, true, true); + g_fail_nth = 0; + check_run("defer allocation failure (the second chunk's allocation fails)", bref.log, o); + if (o.failures != 1 || o.leaked || g_aligned_live.load() != live) + std::printf("FAIL defer allocation failure: %llu failures (1 expected), %ld blocks lost, %ld left after the thread\n", + (unsigned long long)o.failures, o.leaked, g_aligned_live.load() - live), failures++; + else std::printf("PASS defer allocation failure: held packets drained, the call made directly, no block lost\n"); + } + { // (a parallel replay records: never held, each packet run at once) + par_active = true; + rec::set_mode(rec::DEFER); + g_mutate = true, g_log.clear(); + frame(); + const bool at_once = g_log.size() == ref.size() && !rec::t_arena.head; + rec::drain(); + const std::vector got = g_log; + rec::set_mode(rec::OFF); + par_active = false; + if (!at_once) std::printf("FAIL defer while par_active: packets held\n"), failures++; + same("defer while par_active (each packet run at once)", ref, got); + } + + thread_checks(ref); + + { // (B_DIRECT: code that records into the command buffer directly, as Dear ImGui's backend does through its own pointers, + // after packed calls: with the boundary the driver sees packed, direct, packed in order; without it (the control) not) + const auto order = [](int mode, bool bounded) { + const VkCommandBuffer cmd = H(0x300); + rec::set_mode(mode); + g_log.clear(); + // (thread: with the boundary, each packed call takes 0.3 ms so a missing wait would show; without it (the + // control), the recorder is held at the fake until the direct call is made, so the wrong order is certain) + if (mode == rec::THREAD) bounded ? (void)(g_delay_ns = 300000) : (void)(g_gate = true); + { + std::lock_guard lk(g_replay_mu); + vkCmdSetLineWidth(cmd, 1.0f); + vkCmdSetLineWidth(cmd, 2.0f); + if (bounded) rec::boundary(rec::B_DIRECT); + fake_vkCmdSetLineWidth(cmd, 9.0f); // (opaque: not through vk_work.h) + g_gate = false; + vkCmdSetLineWidth(cmd, 3.0f); + } + rec::drain(); + g_delay_ns = 0; + std::string seq; + for (const Call& c : g_log) { + float v; + std::memcpy(&v, c.bytes.data() + c.bytes.size() - sizeof v, sizeof v); + seq += (char)('0' + (int)v); + } + rec::set_mode(rec::OFF); + return seq; + }; + for (const int mode : {rec::DEFER, rec::THREAD}) { + const char* name = mode == rec::DEFER ? "defer" : "thread"; + const std::string with = order(mode, true), without = order(mode, false); + if (with != "1293") std::printf("FAIL %s B_DIRECT: the driver saw %s, 1293 expected\n", name, with.c_str()), failures++; + else if (without == "1293") std::printf("FAIL %s B_DIRECT control: without the boundary the order was still 1293\n", name), failures++; + else std::printf("PASS %s B_DIRECT: packed, direct, packed reach the driver in order (%s; without the boundary %s)\n", name, + with.c_str(), without.c_str()); + } + } + + std::set seen; + for (const Call& c : ref) seen.insert(c.op); + int missing = 0; + for (uint32_t op = 0; op < rec::OP_COUNT; op++) + if (!seen.count(rec::op_name(op))) std::printf("NOT COVERED %s\n", rec::op_name(op)), missing++; + if (missing) failures++; + else std::printf("PASS coverage: all %d packed calls made by the frame\n", (int)rec::OP_COUNT); + std::printf(failures ? "FAILED (%d)\n" : "ALL PASS\n", failures); + return failures ? 1 : 0; +} diff --git a/tests/test_audio_output_errors.py b/tests/test_audio_output_errors.py index 6169d0b..ada75d2 100644 --- a/tests/test_audio_output_errors.py +++ b/tests/test_audio_output_errors.py @@ -11,7 +11,8 @@ import sys import tempfile ROOT = Path(__file__).resolve().parents[1] -CASES = ("normal", "busy", "prepare", "write", "unprepare", "open", "bounded", "stall", "surround", "fallback") +CASES = ("normal", "busy", "prepare", "write", "unprepare", "open", "bounded", "stall", "surround", "fallback", + "lost", "gone", "service", "default") def main(): diff --git a/tests/test_capture_copy_nt.py b/tests/test_capture_copy_nt.py index 0636a5d..4828cb7 100644 --- a/tests/test_capture_copy_nt.py +++ b/tests/test_capture_copy_nt.py @@ -39,9 +39,13 @@ def main(): if "capture_nt() ? copy_nt : std::memcpy" not in source: raise SystemExit("the guarded copy does not use copy_nt") part = function(source, "void copy_part(const CopyPart& w)") - retry = part[part.index("for (int64_t q = i; q < j; q++)"):] - if "copy_fence();" not in retry[:retry.index("i = j;")]: - raise SystemExit("copy_part does not fence after its page-by-page retry") + # (the page-by-page retries: copy_part's plain and staged copies (the shadow staging), each fenced after its loop) + retries = [i for i in range(len(part)) if part.startswith("for (int64_t q = a; q < b; q++)", i)] + if len(retries) < 2: + raise SystemExit("copy_part's page-by-page retries not found") + for at in retries: + if "copy_fence();" not in part[at:part.index("};", at)]: + raise SystemExit("copy_part does not fence after its page-by-page retry") if platform.machine().lower() not in ("amd64", "x86_64", "x64"): print("SKIP: not an x86-64 host") return diff --git a/tests/test_compressed_over_rt.py b/tests/test_compressed_over_rt.py new file mode 100644 index 0000000..c1fa360 --- /dev/null +++ b/tests/test_compressed_over_rt.py @@ -0,0 +1,144 @@ +"""A block-compressed texture over storage render targets used is uploaded from memory (gpu/vk/vk_texture.cpp texture_touch). + +texture_touch skips the upload of a new texture whose storage the GPU rendered into since the CPU last wrote it: the image +already holds what the GPU drew there. The GPU cannot render into a compressed format, so for ASTC, BC and ETC textures +that storage only means render targets used it before; skipped, a boss or a new enemy whose textures were loaded there was +drawn with an empty texture, all blue, for its first cutscene (from Shakoquo/Mako ff479e988d). Cuts texture_touch's decision +out of the source as it is and checks it for each kind of format, with and without RX_VK_SKIP_COMPRESSED_OVER_RT=1 (with +gpu_owned_since stubbed: the decision alone). Then gpu_owned_since itself, cut out the same way: overlapping render-target +ranges count their bytes once, so memory with a gap the GPU never drew is not taken as the GPU's. + +python -I -B tests/test_compressed_over_rt.py [--cxx g++] +""" +import argparse +import os +import shutil +import subprocess +import tempfile +from pathlib import Path + +RUN_ALL_ARGS = ["--cxx", "{cxx}"] + +ROOT = Path(__file__).resolve().parents[1] +ENV_ON_H = ROOT / "gpu" / "vk" / "compat" / "hookshot_env.h" + +FIXTURE = r'''#include +#include +#include +#include +#include +#include +enum VkFormat { VK_FORMAT_R8G8B8A8_UNORM = 37, VK_FORMAT_BC1_RGB_UNORM_BLOCK = 131, VK_FORMAT_BC7_SRGB_BLOCK = 146, + VK_FORMAT_ETC2_R8G8B8_UNORM_BLOCK = 147, VK_FORMAT_ASTC_4x4_UNORM_BLOCK = 157, + VK_FORMAT_ASTC_12x12_SRGB_BLOCK = 184, VK_FORMAT_D32_SFLOAT = 126 }; +struct Texture { bool uploaded = false, astc = false, bc = false; VkFormat format = VK_FORMAT_R8G8B8A8_UNORM; + uint64_t gpu = 0x1000, guest_size = 0x4000; uint32_t id = 1, width = 64, height = 64, layers = 1; int nvn_format = 0; }; +static int logs; +static void logf(const char *, ...) { logs++; } +static uint32_t gpu_owned_since(uint64_t, uint64_t) { return 7; } /* (render targets took this storage) */ +static uint32_t decide(Texture *tex) { + static const bool upload_all = false; +DECISION + return owned; +} +#define CHECK(x) do { if (!(x)) { std::fprintf(stderr, "FAIL:%d %s\n", __LINE__, #x); return 1; } } while (0) +int main(int argc, char **) { + const bool old_rule = argc > 1; /* (RX_VK_SKIP_COMPRESSED_OVER_RT=1) */ + Texture rgba, bc, bc_kept, astc_decoded, astc_raw, etc2, depth; + bc.format = VK_FORMAT_BC1_RGB_UNORM_BLOCK; + bc_kept.bc = true; /* (ASTC decoded and kept as BC3) */ + astc_decoded.astc = true; /* (ASTC decoded into RGBA8: its image format says nothing) */ + astc_raw.format = VK_FORMAT_ASTC_12x12_SRGB_BLOCK; + etc2.format = VK_FORMAT_ETC2_R8G8B8_UNORM_BLOCK; + depth.format = VK_FORMAT_D32_SFLOAT; + CHECK(decide(&rgba) == 7 && decide(&depth) == 7); /* (formats the GPU renders: skipped as before) */ + for (Texture *t : {&bc, &bc_kept, &astc_decoded, &astc_raw, &etc2}) CHECK(decide(t) == (old_rule ? 7u : 0u)); + Texture done = bc; + done.uploaded = true; + CHECK(decide(&done) == 0); + CHECK(old_rule ? logs == 0 : logs > 0); + std::printf("PASS: %s\n", old_rule ? "RX_VK_SKIP_COMPRESSED_OVER_RT=1: compressed textures skipped as before" + : "compressed textures over render targets' storage uploaded, others skipped as before"); + return 0; +} +''' + + +# (gpu_owned_since as it is in the source, over a gpu_owned of the test's own: overlapping ranges count their bytes once) +OWNED = r'''#include +#include +#include +#include +static std::map> gpu_owned; +FUNCTION +#define CHECK(x) do { if (!(x)) { std::fprintf(stderr, "FAIL:%d %s\n", __LINE__, #x); return 1; } } while (0) +static void set(std::initializer_list> r) { + gpu_owned.clear(); + for (auto& x : r) gpu_owned[x.first] = {x.second, 5}; +} +int main() { + set({{0x0, 0x3000}, {0x2000, 0x4000}}); /* (overlapping, with [0x4000, 0x5000) never drawn) */ + CHECK(gpu_owned_since(0x1000, 0x4000) == 0); + set({{0x0, 0x3000}, {0x2000, 0x6000}}); /* (overlapping, covering it) */ + CHECK(gpu_owned_since(0x1000, 0x4000) == 5); + set({{0x1000, 0x5000}}); + CHECK(gpu_owned_since(0x1000, 0x4000) == 5); + set({{0x1000, 0x2000}, {0x3000, 0x5000}}); /* (a gap) */ + CHECK(gpu_owned_since(0x1000, 0x4000) == 0); + set({}); + CHECK(gpu_owned_since(0x1000, 0x4000) == 0); + std::printf("PASS: gpu_owned_since counts overlapping render-target ranges once: a gap never drawn is not the GPU's\n"); + return 0; +} +''' + + +def main(): + p = argparse.ArgumentParser(description=__doc__) + p.add_argument("--cxx", default="g++") + a = p.parse_args() + src = (ROOT / "gpu" / "vk" / "vk_texture.cpp").read_text(encoding="utf-8") + body = src[src.index("Texture* texture_touch(Texture* tex, bool for_sampling) {"):] + start = body.index(" static const bool skip_compressed") + end = body.index("\n", body.index(" uint32_t owned = ", start)) + decision = body[start:end] + assert "gpu_owned_since(tex->gpu, tex->guest_size)" in decision, decision + # (the log after it is in the block: up to its closing brace) + tail = body.index("\n }\n", end) + len("\n }\n") + decision = body[start:tail] + cxx = shutil.which(a.cxx) or shutil.which("g++") or shutil.which("clang++") + if not cxx: + print("SKIP: no C++ compiler") + return + with tempfile.TemporaryDirectory(prefix="compressed-rt-") as tmp: + tmp = Path(tmp) + (tmp / "f.cpp").write_text(FIXTURE.replace("DECISION", decision), encoding="utf-8") + exe = tmp / ("f.exe" if os.name == "nt" else "f") + # (env_on from the renderer's own header; _CRT_SECURE_NO_WARNINGS: Windows' C runtime marks getenv deprecated, and + # -Werror made that an error with clang on Windows) + # (Triforce: this tree has no hookshot_env.h; the switch is read with getenv and atoi here, so stands in) + inc = ["-include", str(ENV_ON_H)] if ENV_ON_H.exists() else ["-include", "cstdlib"] + subprocess.run([cxx, "-std=c++17", "-D_CRT_SECURE_NO_WARNINGS", *inc, "-Wall", + "-Wno-unused-function", "-Werror", str(tmp / "f.cpp"), "-o", str(exe)], + check=True) + fn = src[src.index("uint32_t gpu_owned_since(uint64_t gpu, uint64_t n) {"):] + fn = fn[:fn.index("\n}\n") + 3] + (tmp / "owned.cpp").write_text(OWNED.replace("FUNCTION", fn), encoding="utf-8") + owned = tmp / ("owned.exe" if os.name == "nt" else "owned") + subprocess.run([cxx, "-std=c++17", "-Wall", "-Werror", str(tmp / "owned.cpp"), "-o", str(owned)], check=True) + r = subprocess.run([str(owned)], capture_output=True, text=True) + print(r.stdout.strip()) + if r.returncode: + raise SystemExit("FAIL:\n" + r.stdout + r.stderr) + for extra in ({}, {"RX_VK_SKIP_COMPRESSED_OVER_RT": "1"}): + env = {k: v for k, v in os.environ.items() if k != "RX_VK_SKIP_COMPRESSED_OVER_RT"} + env.update(extra) + args = [str(exe)] + (["old"] if extra else []) + r = subprocess.run(args, env=env, capture_output=True, text=True) + print(r.stdout.strip()) + if r.returncode: + raise SystemExit("FAIL:\n" + r.stdout + r.stderr) + + +if __name__ == "__main__": + main() diff --git a/tests/test_draw_side_effects.py b/tests/test_draw_side_effects.py new file mode 100644 index 0000000..d34426c --- /dev/null +++ b/tests/test_draw_side_effects.py @@ -0,0 +1,167 @@ +"""CPU-only draw readiness decision check: inline fake worker completion, no Vulkan/shader execution. + +The real draw decision block is extracted intact. Publication and final resource writes are CPU witnesses, +not a claim about GPU output, timing, or a reproduced game failure. +""" +import argparse +import os +from pathlib import Path +import subprocess +import tempfile + +RUN_ALL_ARGS = ["--cxx", "{cxx}"] + +ROOT = Path(__file__).resolve().parents[1] +FIXTURE = r""" +#include +#include +#include +#include +#include +#include +#define PROF_EXACT() ((void)0) +#define PROF_HOT(x) ((void)0) +#define SKIP_HERE() return +using VkPipeline = int; +struct Desc { bool is_written = false; }; +struct Info { + bool stores_global_memory = false; + std::vector storage_buffers_descriptors, image_buffer_descriptors, image_descriptors; +}; +struct Stage { Info info; }; +int scans, early_reads, translation_waits, pipeline_waits, draws, output, st_waits, st_pending_skips; +double st_wait_ms; +// Instrument only fixture iteration: reject inspecting ordinary stages before acquire-ready publication. +struct Stages { + std::atomic* ready; + std::vector values; + auto begin() const { + if (!ready->load(std::memory_order_acquire)) ++early_reads; + ++scans; + return values.begin(); + } + auto end() const { return values.end(); } +}; +struct Program { + std::atomic ready{false}; + bool failed = false, fallback = false, fallback_task = false; + int layout = 1, fallback_topology = 0; + Stages stages{&ready, {}}; +}; +struct PipeEntry { std::atomic ready{false}; VkPipeline pipeline = 1; }; +struct Targets { bool persistent = true; }; +struct NvnState {}; +struct ParLock {}; +struct DrawArgs { bool indirect = true; int groups[3]{1, 1, 1}; int first_task = 0; }; +int fallback_task_groups(const Program&) { std::abort(); } +void draw(int, NvnState&, const DrawArgs&) { std::abort(); } +Program* current; +PipeEntry pipe_entry; +int effect; +Stage shader_stage(int kind) { + Stage s; + if (kind == 1) s.info.stores_global_memory = true; + if (kind == 2) s.info.storage_buffers_descriptors.push_back({true}); + if (kind == 3) s.info.image_buffer_descriptors.push_back({true}); + if (kind == 4) s.info.image_descriptors.push_back({true}); + // Nonempty read-only descriptors must still qualify for the color-only control. + if (kind == 0) s.info.storage_buffers_descriptors.push_back({false}); + return s; +} +void publish() { + // Put the write in a later stage, not stages[0]. + current->stages.values = {Stage{}, shader_stage(effect)}; + current->ready.store(true, std::memory_order_release); +} +void jobs_wait(const std::atomic& ready) { + if (&ready == ¤t->ready) { ++translation_waits; publish(); } + else { ++pipeline_waits; pipe_entry.ready.store(true, std::memory_order_release); } +} +bool targets_persistent(const Targets& at) { return at.persistent; } +bool exposure_log_on() { return false; } +void exposure_note(NvnState&, const Program&) {} +void ensure_layout(Program&) {} +// These cases use non-fallback programs, whose topology stays unchanged. +int pipeline_topology(const Program&, int drawn) { return drawn; } +PipeEntry* graphics_pipeline(const Program&, NvnState&, const Targets&, int, const unsigned char*) { return &pipe_entry; } +void decision(const Program* cp, const Targets& at) { + NvnState s; DrawArgs d; int cmd = 0, topo = 0; unsigned char types[8]{}; +@BLOCK@ + // Fake draw/consumer witness only: not actual GPU shader execution or resource binding. + ++draws; + if (effect) output = 42; +} +int failures; +void check(bool ok, const char* label) { if (!ok) { ++failures; std::printf("FAIL %s\n", label); } } +void scenario(int kind, bool translated, bool pipeline_hot, bool persistent) { + scans = early_reads = translation_waits = pipeline_waits = draws = output = st_pending_skips = 0; + Program p; current = &p; effect = kind; + if (translated) publish(); + pipe_entry.ready.store(pipeline_hot, std::memory_order_release); + const bool force = std::getenv("RX_VK_PIPE_SKIP") && std::atoi(std::getenv("RX_VK_PIPE_SKIP")) == 0; + const bool legacy = std::getenv("RX_VK_PIPE_SKIP") && std::atoi(std::getenv("RX_VK_PIPE_SKIP")) == 2; + const bool skip_unknown = legacy && !translated && persistent; + const bool should_draw = !skip_unknown && (pipeline_hot || force || !persistent || kind != 0); + decision(&p, {persistent}); + char label[120]; + std::snprintf(label, sizeof label, "kind%d translated%d hot%d persistent%d force%d", kind, translated, pipeline_hot, persistent, force); + const int before = failures; + check(draws == int(should_draw), label); + if (kind && should_draw) check(output == 42, "producer write absent at fake consumer"); + check(!early_reads, "metadata inspected before publication"); + check(translation_waits == int(!translated && !skip_unknown), "translation wait mode"); + if (pipeline_hot || force || !persistent || skip_unknown) check(scans == 0, "unnecessary classification"); + check(pipeline_waits == int(!pipeline_hot && should_draw), "pending pipeline must publish before draw"); + std::printf("%s %s draws%d skips%d scans%d twait%d pwait%d\n", failures == before ? "PASS" : "RED", label, + draws, st_pending_skips, scans, translation_waits, pipeline_waits); +} +int main() { + for (int kind = 0; kind <= 4; ++kind) + for (bool translated : {false, true}) + for (bool hot : {false, true}) scenario(kind, translated, hot, true); + scenario(1, true, false, false); + scenario(1, false, false, false); + scenario(0, false, false, false); + std::printf("%d failures; fake CPU decision check only\n", failures); + return failures ? 1 : 0; +} +""" + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--cxx", default="g++") + parser.add_argument("--source", type=Path, default=ROOT) + args = parser.parse_args() + src = (args.source / "gpu/vk/vk_draw.cpp").read_text(encoding="utf-8") + try: + start = src.index(" static const bool always_wait =", src.index("void draw(VkCommandBuffer")) + end = src.index(" cur_program = &p;", start) + except ValueError: + raise SystemExit("draw readiness block changed; update this test's extraction anchors") from None + with tempfile.TemporaryDirectory(prefix="draw-side-effects-") as tmp: + cpp, exe = Path(tmp) / "test.cpp", Path(tmp) / "test" + cpp.write_text(FIXTURE.replace("@BLOCK@", src[start:end]), encoding="utf-8") + subprocess.run([args.cxx, "-std=c++20", "-Wall", "-Wextra", "-Wno-unused-variable", + str(cpp), "-o", str(exe)], check=True, timeout=60) + failed = False + for mode in (None, "0", "1", "2"): + env = os.environ.copy() + env.pop("RX_VK_PIPE_SKIP", None) + if mode is not None: + env["RX_VK_PIPE_SKIP"] = mode + result = subprocess.run([str(exe)], env=env, capture_output=True, text=True, timeout=10) + lines = result.stdout.splitlines() + complete = "0 failures; fake CPU decision check only" in lines and sum(l.startswith("PASS kind") for l in lines) == 23 + ok = result.returncode == 0 and complete + print(f"RX_VK_PIPE_SKIP={mode if mode is not None else 'unset'}: {'PASS' if ok else 'FAIL'}") + if not ok: + failed = True + print("fixture failed or did not finish all 23 cases\n" + result.stdout + result.stderr) + if failed: + raise SystemExit(1) + print("PASS: unknown translations wait by default; mode 2 restores their old skip; translated writers wait; hot draws do not scan stages") + + +if __name__ == "__main__": + main() diff --git a/tests/test_fps_overlay_order.py b/tests/test_fps_overlay_order.py index ec36c4a..c826fe6 100644 --- a/tests/test_fps_overlay_order.py +++ b/tests/test_fps_overlay_order.py @@ -49,6 +49,10 @@ bool begin(uint32_t, uint32_t) { calls += "b"; return can_draw; } void idle() { calls += "i"; } void record(VkCommandBuffer cmd, uint32_t index, uint32_t w, uint32_t h) { CHECK(cmd == 1 && index == 2 && w == 640 && h == 360); calls += "r"; } } +namespace mkwvk::rec { // (RX_VK_RECORDER: the boundary before each Dear ImGui recording, logged as "d") +enum Bound { B_DIRECT }; +void boundary(Bound) { calls += "d"; } +} ui::Ctx ui_ctx(uint32_t, uint32_t) { return {}; } void prebake(const ui::Ctx&) { calls += "p"; } bool resized = false; @@ -69,7 +73,7 @@ int main() { CHECK(calls == "i"); calls.clear(), resized = true; menu_record(1, 2, 640, 360); - CHECK(calls == (can ? "bpr" : "bi")); + CHECK(calls == (can ? "bpdr" : "bi")); resized = false; continue; } @@ -78,7 +82,7 @@ int main() { if (startup) want += "s"; if (menu) want += "m"; if (fps && !startup) want += menu ? "F" : "f"; - want += "r"; + want += "dr"; // (the recorder boundary right before the recording) CHECK(calls == want); } std::printf("PASS: one ImGui frame and recording for whatever shows, in order, nothing while nothing shows\n"); diff --git a/tests/test_frame_rate_cap_step.py b/tests/test_frame_rate_cap_step.py index 386c47d..99f0983 100644 --- a/tests/test_frame_rate_cap_step.py +++ b/tests/test_frame_rate_cap_step.py @@ -93,14 +93,20 @@ int main(int argc, char **argv) { hos_step_probe = strcmp(mode, "no hook") ? 0x2000 : 0; /* (the lifted code sets it when the hook is in the build) */ realtime_step(now); if (!strcmp(mode, "cap 120")) { - CHECK(near(frames(10, 1.0 / 120), 1.0 / 120) && *rate == 60 && *limit == 120); /* (the rate stays the engine's) */ + CHECK(near(frames(72000, 1.0 / 120), 1.0 / 120) && *rate == 60 && *limit == 120); /* (the rate stays the engine's) */ CHECK(near(lifted_step(60.0f, (float)*rate), 0.5) && near(lifted_step(30.0f, (float)*rate), 0.25)); /* (half a 60th a frame) */ - CHECK(near(frames(10, 1.0 / 40), 1.0 / 40) && near(lifted_step(60.0f, (float)*rate), 1.5)); + CHECK(near(frames(10, 1.0 / 40), 1.0 / 40) && *rate == 40 && near(lifted_step(60.0f, (float)*rate), 1.5)); CHECK(near(frames(10, 1.0 / 1000), 1.0 / 240)); /* (the shortest step) */ CHECK(near(frames(10, 0.2), 4.0 / 60)); /* (the longest, as the rate's 15) */ } else if (!strcmp(mode, "cap 60")) { - CHECK(frames(10, 1.0 / 120) == 0.0f && *rate == 60 && *limit == 60); - CHECK(frames(10, 1.0 / 40) == 0.0f && *rate == 40 && lifted_step(60.0f, (float)*rate) == 1.5f); /* (the game's own divide) */ + CHECK(frames(72000, 1.0 / 120) == 0.0f && *rate == 60 && *limit == 60); + /* A long unpaced stretch must repay only bounded debt, including the five-frame smoothing transition. */ + int recovered = 0; + for (int i = 0; i < 24; i++) { + CHECK(frames(1, 1.0 / 40) == 0.0f && *rate >= 15 && *rate <= 60); + if (*rate == 40) { recovered = 1; break; } + } + CHECK(recovered && lifted_step(60.0f, (float)*rate) == 1.5f); } else { /* (HOS_FPS_STEP=0, or a lift without the hook: the rate up to the cap, as before) */ CHECK(frames(10, 1.0 / 120) == 0.0f && *rate == 120 && *limit == 120); CHECK(lifted_step(60.0f, 60.0f) == 1.0f); @@ -160,7 +166,7 @@ def main(): "a second run left one divide's hook from the first") vi = (ROOT / "host" / "nn_vi.c").read_text(encoding="utf-8") - host = "\n".join(block(vi, m) for m in ("static float text_scale(", "static int fps_max(void) {", "static void realtime_step(")) + host = "\n".join(block(vi, m) for m in ("static float text_scale(", "static int fps_max(void) {", "static int realtime_rate(", "static void realtime_step(")) fixture = FIXTURE.replace("@HOST@", host).replace("@HOOK@", "\n".join(oot_patches.STEP_HOOK.split("\n")[:2])) with tempfile.TemporaryDirectory(prefix="cap-step-") as tmp: c = Path(tmp) / "fixture.c" diff --git a/tests/test_import_windows.py b/tests/test_import_windows.py new file mode 100644 index 0000000..ccb020f --- /dev/null +++ b/tests/test_import_windows.py @@ -0,0 +1,140 @@ +"""Run actual pool lookup/range guards/prefetch queueing with fake GPU/background dependencies. + +No Vulkan, background import thread, SDK or game data is used. + +python tests/test_import_windows.py [--cxx clang++] + +(Triforce: this is the test to run. nx2/test_import_windows.py stays as v0.3.2 had it, byte for byte: every .py under nx2/ is +part of the lift key (make_game.lift_key), so a change there makes each player's update translate the whole game again.) +""" +import argparse,os +from pathlib import Path +import subprocess,tempfile + +RUN_ALL_ARGS = ["--cxx", "g++"] + +ROOT=Path(__file__).resolve().parents[1] +def block(text,marker): + start=text.index(marker);end=text.index('{',start)+1;depth=1 + while depth: + depth+=(text[end]=='{')-(text[end]=='}');end+=1 + return text[start:end] + +def main(): + p=argparse.ArgumentParser(description=__doc__);p.add_argument('--cxx',default='clang++');a=p.parse_args() + source=(ROOT/'gpu/vk/vk_memory.cpp').read_text(encoding='utf-8') + guard=source[source.index('bool import_pool(uint64_t lo, uint64_t hi, Pool& p) {'):source.index(' p.lo = lo;',source.index('bool import_pool('))] + lookup=block(source,'Pool* pool_lookup(uint64_t guest, uint64_t size) {') + chunking=source[source.index('uint64_t import_chunk() {'):source.index('Pool* pool_lookup(uint64_t guest, uint64_t size) {')] + cache=block(source,'Pool* pool_of(uint64_t guest, uint64_t size) {') + prefetch=block(source,'void prefetch_chunks(uint64_t lo, uint64_t hi, uint64_t chunk) {') + fixture='''#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#define CHECK(x) do { if(!(x)){std::fprintf(stderr,"FAIL:%d %s\\n",__LINE__,#x);std::exit(1);} }while(0) +struct Pool { uint64_t lo=0,hi=0,range_lo=0,range_hi=0,address=0; bool prefetched=false,gpu_only=false,gpu_written=false; }; +struct GuestPool { uint64_t lo,hi; }; +std::vector guest_pools; +std::map imported; +Pool *last_pool=nullptr,*pool_cache[4096]{}; +uint64_t max_import_size=128ull<<20; +int failures=0,imports=0,retires=0,bg_starts=0; +int64_t st_bg_waits=0,st_prefetch_used=0,st_prefetch_sent=0,st_prefetch_dropped=0; +thread_local const char* tl_sync_why=nullptr; // (Triforce: why a binding's import ran on its own thread, for the log line) +struct { bool external_memory_host=true; } dev; +std::mutex bg_mu;std::condition_variable bg_done_cv,bg_cv;std::vector bg_inflight; +std::deque> bg_queue; +void bg_start() { bg_starts++; } // Recording spy: no thread or GPU import is started. +void logf(const char*,...){} void bg_drain(){} +void guest_pages_set(uint64_t,uint64_t,uint64_t){} +void retire_pool(std::map::iterator it) { imported.erase(it);last_pool=nullptr;std::fill(std::begin(pool_cache),std::end(pool_cache),nullptr);retires++; } +enum { VK_SUCCESS = 0 }; +thread_local bool import_cooling = false; +thread_local int import_result = 0; +bool import_on_hold(uint64_t, uint64_t) { import_cooling = false; return false; } +void import_remember_failure(uint64_t, uint64_t) {} +'''+guard+''' + p.lo=lo;p.hi=hi;imports++;return true; +} +'''+prefetch+'\n'+chunking+'\n'+lookup+'\n'+cache+''' +int check_prefetch(int mode) { + const uint64_t base=0x1000000000ull,chunk=16ull<<20; + guest_pools.push_back({base,base+8*chunk}); + dev.external_memory_host=mode!=2; + Pool seed;seed.lo=base+3*chunk;seed.hi=base+4*chunk;seed.prefetched=true; + auto &ready=imported.emplace(seed.lo,seed).first->second; + CHECK(pool_lookup(seed.lo+64,64)==&ready && !ready.prefetched && st_prefetch_used==1 && imports==0); + if(mode==1) { + CHECK(bg_queue.size()==2 && bg_starts==2 && st_prefetch_sent==2); + CHECK(bg_queue[0].first==base+4*chunk && bg_queue[0].second==base+5*chunk); + CHECK(bg_queue[1].first==base+2*chunk && bg_queue[1].second==base+3*chunk); + } else CHECK(bg_queue.empty() && bg_starts==0 && st_prefetch_sent==0); + CHECK(pool_lookup(seed.lo+128,64)==&ready && st_prefetch_used==1 && imports==0); + Pool plain;plain.lo=base+6*chunk;plain.hi=base+7*chunk; + auto &unprefetched=imported.emplace(plain.lo,plain).first->second; + CHECK(pool_lookup(plain.lo+64,64)==&unprefetched && !unprefetched.prefetched && st_prefetch_used==1 && imports==0); + CHECK(bg_starts==(mode==1?2:0) && st_prefetch_sent==(mode==1?2:0)); + auto demand=pool_lookup(base+64,64); + CHECK(demand && !demand->prefetched && imports==1 && st_prefetch_used==1); + CHECK(bg_queue.size()==(mode==1?3u:0u) && bg_starts==(mode==1?3:0) && st_prefetch_sent==(mode==1?3:0)); + CHECK(bg_inflight.size()==bg_queue.size() && st_prefetch_dropped==0 && st_bg_waits==0); + std::printf("PASS: prefetched first use, reuse, unprefetched lookup, neighbour queueing (mode %d)\\n",mode); + return 0; +} +int main(int argc,char **argv) { + if(argc>1) return check_prefetch(std::atoi(argv[1])); + const uint64_t base=0x1000000000ull,extent=0x99200000ull; + guest_pools.push_back({base,base+extent}); + const auto chunk=uint64_t(std::atoi(std::getenv("RX_VK_IMPORT_CHUNK_MB")))<<20; + if(!chunk) { + CHECK(!pool_lookup(base+1234,64) && imports==0); + max_import_size=4ull<<30; + auto p=pool_lookup(base+1234,64);CHECK(p && p->lo==base && p->hi==base+extent); + } else { + const uint64_t address=base+1234,expected=address/chunk*chunk; + auto p=pool_lookup(address,64);CHECK(p && p->lo==expected && p->hi==expected+chunk && imports==1); + CHECK(pool_lookup(address+100,64)==p && imports==1); + p=pool_lookup(expected+4*chunk+123,64);CHECK(p && p->lo==expected+4*chunk && p->hi==expected+5*chunk && imports==2); + p=pool_lookup(expected+chunk-16,64);CHECK(p && p->lo==expected && p->hi==expected+2*chunk && retires==1); + if(chunk==(16ull<<20)) { + // a game pool across a chunk's end: its first binding imports both chunks (span_game_pools) + const uint64_t far=base+0x40000000ull,far2=base+0x50000000ull; + guest_pools.push_back({far+chunk-4096,far+chunk+4096}); + guest_pools.push_back({far2+chunk-4096,far2+chunk+4096}); + p=pool_lookup(far+chunk-64,32);CHECK(p && p->lo==far && p->hi==far+2*chunk); + CHECK(pool_lookup(far+chunk+64,32)==p); // (the binding across the end: no import again) + // a spanned import that fails (past the device's limit here) is made again without the span + const uint64_t limit=max_import_size;max_import_size=chunk; + p=pool_lookup(far2+chunk-64,32);CHECK(p && p->lo==far2 && p->hi==far2+chunk); + max_import_size=limit; + } + } + const int before=imports; + CHECK(!pool_lookup(base+extent+1,1)); + CHECK(!pool_lookup(base+extent-1,2)); + CHECK(!pool_lookup(base,UINT64_MAX)); + CHECK(!pool_lookup(base,0)); + CHECK(!pool_of(base,UINT64_MAX) && !pool_of(base,0) && imports==before); + guest_pools.push_back({UINT64_MAX-1024,UINT64_MAX}); + if(chunk) CHECK(!pool_lookup(UINT64_MAX-16,8) && imports==before); + Pool rejected;CHECK(!import_pool(base,base,rejected) && !import_pool(base+1,base,rejected)); + CHECK(!import_pool(base,base+max_import_size+1,rejected)); + std::printf("PASS: request windows, reuse, overlap retirement, range/overflow guards, import limits (chunk %llu)\\n",(unsigned long long)chunk); +} +''' + work=ROOT/'work';work.mkdir(exist_ok=True) + with tempfile.TemporaryDirectory(prefix='import-windows-',dir=work) as temp: + directory=Path(temp).resolve();directory.relative_to(work.resolve());cpp=directory/'fixture.cpp';exe=directory/'fixture.exe';cpp.write_text(fixture,encoding='utf-8') + subprocess.run([a.cxx,'-std=c++20','-O2','-D_CRT_SECURE_NO_WARNINGS',str(cpp),'-o',str(exe)],check=True) + for chunk in ('16','3','0'): + subprocess.run([str(exe)],env=dict(os.environ,RX_VK_IMPORT_CHUNK_MB=chunk,RX_VK_PREFETCH_CHUNKS='0'),check=True) + for mode in ('1','0','2'): + subprocess.run([str(exe),mode],env=dict(os.environ,RX_VK_IMPORT_CHUNK_MB='16',RX_VK_PREFETCH_CHUNKS='0' if mode=='0' else '1'),check=True) +if __name__=='__main__':main() diff --git a/tests/test_mirror_gpu_only.py b/tests/test_mirror_gpu_only.py new file mode 100644 index 0000000..772c037 --- /dev/null +++ b/tests/test_mirror_gpu_only.py @@ -0,0 +1,46 @@ +"""A pool the GPU wrote keeps its VRAM mirror (gpu/vk/vk_memory.cpp mirrors_give_back). + +An idle mirror is given back after kIdleEpochs submissions (at once past kLongIdle, or when a wanted one does not fit), and +made again from guest memory at the pool's next binding. What a shader wrote through a bare entry, the pointer table or a +wide binding is never copied back (memory_note_gpu_write): it was in the old mirror alone, and the GPU went on from stale +data. Paused 1-2 minutes in the background, the idle pools' mirrors all went, and the game came back with cyan for a few +seconds (RX 5500M; gone with RX_VK_MIRROR_KEEP=1). Reads the source: memory_note_gpu_write marks the pools it reaches, the +idle give-back leaves those, and a pool that takes others in keeps their mark. +python -I -B tests/test_mirror_gpu_only.py +""" +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] + + +def body(src, signature): + """the text of the function that starts with signature, up to its closing brace at column 0""" + start = src.index(signature) + return src[start:src.index("\n}\n", start)] + + +def main(): + mem = (ROOT / "gpu" / "vk" / "vk_memory.cpp").read_text(encoding="utf-8") + if "bool gpu_only = false;" not in mem[mem.index("struct Pool {"):mem.index("};", mem.index("struct Pool {"))]: + raise SystemExit("Pool has no gpu_only mark: nothing tells which mirrors hold what guest memory never gets") + if ".gpu_only = true;" not in body(mem, "void memory_note_gpu_write(uint64_t gpu, uint64_t size) {"): + raise SystemExit("memory_note_gpu_write does not mark the pools it reaches: their mirrors go with the GPU's data") + give = body(mem, "void mirrors_give_back(uint64_t want_bytes) {") + idle = give[give.index("std::vector idle;"):give.index("std::sort(idle.begin()")] + if "if (p.gpu_only || p.gpu_written)" not in idle or "else idle.push_back(&p);" not in idle: + raise SystemExit("mirrors_give_back gives back a gpu_only pool's idle mirror: made again from guest memory, the GPU's " + "data in it is lost (cyan for seconds after a pause in the background)") + bind = body(mem, "BufferRef gpu_buffer_bind(uint64_t gpu, uint64_t size, bool written) {") + if "p->gpu_written = true;" not in bind[bind.rindex("if (written) {"):]: + raise SystemExit("gpu_buffer_bind does not mark a pool the GPU writes through its mirror: after a save reload its mirror " + "goes and comes back without what the GPU made of it (cyan in play and over the Start menu)") + lookup = body(mem, "Pool* pool_lookup(uint64_t guest, uint64_t size) {") + took_in = lookup[lookup.index("for (uint64_t k : overlapped) { // (the pools it took in)"):] + merged = took_in[:took_in.index("imported.emplace(p.lo, p)")] + if "p.gpu_only |= o->second.gpu_only" not in merged or "p.gpu_written |= o->second.gpu_written" not in merged: + raise SystemExit("a pool that takes others in does not keep their gpu_only and gpu_written marks") + print("PASS: mirrors the GPU wrote past the write-back are kept") + + +if __name__ == "__main__": + main() diff --git a/tests/test_realtime_rate.py b/tests/test_realtime_rate.py new file mode 100644 index 0000000..3abafb8 --- /dev/null +++ b/tests/test_realtime_rate.py @@ -0,0 +1,115 @@ +"""Frame-time jitter must not make the real-time game clock run fast.""" +import argparse +import subprocess +import tempfile +from pathlib import Path + +RUN_ALL_ARGS = ["--cc", "gcc"] + +ROOT = Path(__file__).resolve().parents[1] + + +def block(text, marker): + start = text.index(marker) + end = text.index("{", start) + 1 + depth = 1 + while depth: + depth += (text[end] == "{") - (text[end] == "}") + end += 1 + return text[start:end] + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--cc", default="gcc") + args = parser.parse_args() + source = (ROOT / "host/nn_vi.c").read_text(encoding="utf-8") + if "realtime_rate(" not in block(source, "static void realtime_step("): + raise SystemExit("the present path does not use the tested clock quantizer") + fixture = r''' +#include +#include +#include +#define CHECK(x) do { if (!(x)) { fprintf(stderr,"FAIL:%d %s\n",__LINE__,#x); exit(1); } } while(0) +''' + fixture += block(source, "static int realtime_rate(") + fixture += r''' +int main(void) { + double carry = 0, elapsed = 0, game = 0; + // The old clamp erased the short frame's debt and sped this clock up by ~8%. + for (int i = 0; i < 6000; i++) { + double dt = i % 2 ? 1.0 / 30.0 - 0.014 : 0.014; + int fps = realtime_rate(dt * 60, 60, &carry); + CHECK(fps >= 15 && fps <= 60); + elapsed += dt; game += 1.0 / fps; + } + CHECK(fabs(game - elapsed) <= 1.0 / 60 + 1e-8); + // Integer-rate rounding retains elapsed time at a sustained non-integer rate. + carry = elapsed = game = 0; + for (int i = 0; i < 6000; i++) { + double dt = 1.0 / 47.3; + int fps = realtime_rate(dt * 60, 60, &carry); + elapsed += dt; game += 1.0 / fps; + } + CHECK(fabs(game - elapsed) < 0.001); + // A slow frame followed by shorter ones does not leave extra game time behind. + carry = elapsed = game = 0; + for (int i = 0; i < 6000; i++) { + double dt = i % 4 == 0 ? 1.0 / 30.0 : 1.0 / 90.0; + int fps = realtime_rate(dt * 60, 60, &carry); + elapsed += dt; game += 1.0 / fps; + } + CHECK(fabs(game - elapsed) <= 1.0 / 60 + 1e-8); + // High display caps use the same debt accounting and do not force a 60 Hz step. + carry = 0; + CHECK(realtime_rate(0.5, 120, &carry) == 120); + CHECK(fabs(carry) < 1e-12); + // The default five-frame window must also preserve time for both jitter patterns. + for (int pattern = 0; pattern < 2; pattern++) { + double samples[5] = {0}; + int count = 0; + carry = elapsed = game = 0; + for (int i = 0; i < 6000; i++) { + const double dt = pattern ? (i % 4 == 0 ? 1.0 / 30 : 1.0 / 90) + : (i % 2 ? 1.0 / 30 - 0.014 : 0.014); + samples[i % 5] = dt; + if (count < 5) count++; + double sum = 0; + for (int j = 0; j < count; j++) sum += samples[j]; + const int fps = realtime_rate(sum / count * 60, 60, &carry); + CHECK(fps >= 15 && fps <= 60 && carry >= -4.0); + elapsed += dt; game += 1.0 / fps; + } + CHECK(fabs(game - elapsed) <= 1.0 / 30 + 1e-8); + printf("default window pattern %d: elapsed %.9f s, game %.9f s\n", pattern, elapsed, game); + } + // Ten minutes of unpaced 120 Hz presents at a 60 cap must not accumulate minutes of debt. + carry = 0; + for (int i = 0; i < 72000; i++) { + CHECK(realtime_rate(0.5, 60, &carry) == 60); + CHECK(carry >= -4.0); + } + CHECK(carry == -4.0); + int recovered = 0; + for (int i = 0; i < 10; i++) { + if (realtime_rate(1.5, 60, &carry) == 40) { recovered = 1; break; } + } + CHECK(recovered); + // A long stall drops excess time instead of scheduling an oversized catch-up. + carry = 0; + CHECK(realtime_rate(12.0, 60, &carry) == 15); + CHECK(fabs(carry) < 1e-12); + CHECK(realtime_rate(1.0, 60, &carry) == 60); + puts("PASS: jitter preserves elapsed time; slow and high-rate steps stay bounded"); +} +''' + with tempfile.TemporaryDirectory(prefix="realtime-rate-") as tmp: + c = Path(tmp) / "fixture.c" + exe = Path(tmp) / "fixture" + c.write_text(fixture, encoding="utf-8") + subprocess.run([args.cc, "-std=c11", "-O2", "-Wall", "-Wextra", "-Werror", str(c), "-lm", "-o", str(exe)], check=True) + subprocess.run([str(exe)], check=True) + + +if __name__ == "__main__": + main() diff --git a/tests/test_recorder_sync.py b/tests/test_recorder_sync.py new file mode 100644 index 0000000..b5a3bdc --- /dev/null +++ b/tests/test_recorder_sync.py @@ -0,0 +1,503 @@ +"""RX_VK_RECORDER=sync: every vkCmd* the renderer makes through vk_work.h is packed with deep copies of its arguments, and the packed +calls reach the driver exactly as the direct ones do. No game, no GPU, no Vulkan library. + +python -I -B tests/test_recorder_sync.py --headers [--cxx clang++] [--bench N] + +1. The sources: vk_work.h has a macro for each call vk_recorder.h packs and no other, each macro calls its packer, every vkCmd* call + in a file that includes vk_work.h is one of them, and the files that make vkCmd* calls without it are the known ones (their calls + do not go through the recorder: a recorder thread would reorder them). +2. tests/recorder_sync_test.cpp, built with the real vk_work.h and vk_recorder.h and a fake Vulkan that logs every call with its + arguments dereferenced: a frame of the renderer's calls, direct vs packed (sync) vs packed and run later (deferred) while the + caller poisons its argument memory after each call, must log the same bytes; all packed calls covered; the fallback order kept. + Run with RX_VK_RECORDER=sync and without, so the switch is read as the renderer reads it. + RX_VK_RECORDER=defer (step 1.5), against the fake's command buffer lifecycle: two command buffers recorded into interleaved, + reset/begun/ended/submitted/waited for twice, a query read and a destruction, the caller's memory poisoned; a recording handed + between two threads under the replay lock; a 2 KB soft budget; a 1 MB hard budget; the second chunk's allocation failing; a + parallel replay running (par_active). Each must log what the direct calls log, no command outside its command buffer's recording, + no exception out of a vkCmd*, no block lost. Run with RX_VK_RECORDER=defer (mode 2, logged once) and with RX_VK_PARALLEL=4 too + (refused: off, logged once). + RX_VK_RECORDER=thread (step 2): the same frame and lifecycle with every vkCmd* run by a recorder thread, the fake driver slowed + (1 us a call) so the recorder lags, and in the ordering scenarios the recorder held (50 ms) at each point the producer must wait + for it (a recording's end, a submit while another command buffer records, the replay lock let go, a pool reset), so a missing + wait is caught on a loaded runner too, each fake command buffer flagged in use for each call (two calls at once on one are a + violation), the caller's memory poisoned and each packet poisoned once run; a handoff under the lock; a submit while another + command buffer's commands are in flight; a pool reset while recording; the lock let go, then the recording ended on another + thread; an 8 KB ring (backpressure); a packet bigger than its room (made directly, in order; again first in an empty ring); no + memory for the ring; teardown with work queued (stop, join, lazy restart, vkDestroyDevice); par_active; >= 1e6 packets through a + 16 KB ring, twice, hashed against the direct stream. --throw: a driver call that throws on the recorder thread ends the process + (exit code 3, FATAL). Run with RX_VK_RECORDER=thread (mode 3, logged once) and with RX_VK_PARALLEL=4 (refused). +3. The test has teeth: vk_recorder.h with a copy left shallow (all of them, or only the descriptor writes' image infos), with the + fallback's drain left out, with no drain before vkEndCommandBuffer, with the replay lock let go without a drain, with a throwing + allocation (exception safety broken), with the failure's drain left out, or with the parallel refusal left out, must fail it. + Thread (each run with --only thread, so the recorder thread's checks alone catch it): no wait before vkEndCommandBuffer, before + vkQueueSubmit2, at the replay lock's release, before vkResetCommandPool; no backpressure (the ring overwritten); no join at + teardown; a packet bigger than its room written into the ring when the ring has more free; the recorder's exception swallowed + (--throw). +4. (Windows) the process ends cleanly with the recorder parked in idle() holding its lock: tests/recorder_exit_test.cpp, a DLL with + the real headers and an exe that calls ExitProcess (as the window's close does); exit code 0 within 20 s (a watchdog ends only + that child) and the DLL's last global destructor run. Broken: the old global recorder with its destructor (wake, join). +--bench N: the packet cost per draw on the fake (fakes that do nothing), ns, median of 5; and the recorder thread's. +--overlap F: the replay thread's time a frame, off vs thread, with a fake driver that takes time and drains every K draws. +""" +import argparse +import os +from pathlib import Path +import re +import shutil +import subprocess +import sys +import tempfile + +RUN_ALL_ARGS = ["--cxx", "{cxx}", "--headers", "{vulkan_headers}"] + +ROOT = Path(__file__).resolve().parents[1] +VK = ROOT / "gpu" / "vk" +# (files that call vkCmd* without vk_work.h, and why that is right) +UNWRAPPED_OK = { + "vk_coopmat.cpp": "the cooperative-matrix probe at start: its own command pool, buffer, submit and fence wait", + "vk_imgui.cpp": "Dear ImGui's overlays: vk_menu.cpp calls boundary(B_DIRECT) before each ui::record/record_to (checked below)", + "vk_main.cpp": "Triforce's start-up screen as pictures (RX_VK_STARTUP_SHOTS, a lab switch): a command buffer of its own, before any replay", +} +# (files that make a boundary call (RX_REC_BOUNDARIES: begin/end/reset, submit, wait, read, destroy) without vk_work.h, and why no packet +# can be held then: RX_VK_RECORDER=defer drains only through vk_work.h's macros and the replay lock) +BOUNDARY_OK = { + "vk_coopmat.cpp": "the start-up probe: before any replay, its own vkCmd* direct (nothing held)", + "vk_main.cpp": "create_swapchain: after device_wait_idle_locked() (vkDeviceWaitIdle through vk_work.h in vk_replay.cpp: drained)", + "vk_shader.cpp": "program_cache_borrow's and a dropped try's own modules and layouts, which no recorded command uses", + "vk_imgui.cpp": "its own font/logo texture views, which only its direct recordings use", +} +# (the names a boundary has: a new one in a vk_work.h file must be in RX_REC_BOUNDARIES) +BOUNDARY_NAME = re.compile(r"\b(vk(?:Begin|End|Reset)\w*|vkQueue\w+|vkWait\w+|vkDeviceWaitIdle|vkDestroy\w+|vkFree\w+|" + r"vkGetQueryPoolResults|vkGetQueueCheckpointDataNV|vkGetDeviceFaultInfoEXT|vkGetSemaphoreCounterValue|" + r"vkAcquireNextImageKHR|vma(?:Destroy|Free|Invalidate)\w*)\s*\(") + + +def code(text): + """the text without // and /* */ comments (string literals in these files hold no comment markers)""" + return re.sub(r"//[^\n]*", "", re.sub(r"/\*.*?\*/", "", text, flags=re.S)) + + +def recorder_calls(text): + m = re.search(r"#define RX_REC_CALLS\(V, A\)(.*?)\n\s*\nnamespace", text, re.S) + if not m: + raise SystemExit("RX_REC_CALLS not found in vk_recorder.h") + return re.findall(r"\b[VA]\((vkCmd\w+)\)", m.group(1)) + + +B_DIRECT_STMT = "::mkwvk::rec::boundary(::mkwvk::rec::B_DIRECT);" + + +C_COMMENT_OR_STRING = re.compile(r'//[^\n]*|/\*.*?\*/|"(?:\\.|[^"\\\n])*"|\'(?:\\.|[^\'\\\n])*\'', re.S) +UI_RECORD_CALL = re.compile(r"\bui\s*::\s*record(?:_to)?\s*\(") +NGX_RECORD_CALL = re.compile(r"\bNGX_VULKAN_\w+_EXT\s*\(") # (NVIDIA NGX's create/evaluate: its own commands into cmd) +FFX_RECORD_CALL = re.compile(r"\bl\s*\.\s*dispatch\s*\(") # (AMD FidelityFX's ffxDispatch, through its loaded pointer: its own commands into cmd) +OPAQUE_RECORD_CALL = re.compile(UI_RECORD_CALL.pattern + "|" + NGX_RECORD_CALL.pattern + "|" + FFX_RECORD_CALL.pattern) +B_DIRECT_BEFORE = re.compile(r"::\s*mkwvk\s*::\s*rec\s*::\s*boundary\s*\(\s*::\s*mkwvk\s*::\s*rec\s*::\s*B_DIRECT\s*\)\s*;\Z") + + +def blank_comments(text): + return C_COMMENT_OR_STRING.sub(lambda m: m.group(0) if m.group(0)[0] in "\"'" else re.sub(r"[^\n]", " ", m.group(0)), text) + + +def statement_before(code, at): + """the code up to the end of the statement before the one containing `at` (its ';', '{' or '}')""" + return code[:max(code.rfind(";", 0, at), code.rfind("{", 0, at), code.rfind("}", 0, at)) + 1].rstrip() + + +def direct_unbounded(text, fname="vk_menu.cpp", call=UI_RECORD_CALL): + """Recordings into a command buffer by code vk_work.h does not see (Dear ImGui's ui::record/record_to in vk_menu.cpp, + NVIDIA NGX's NGX_VULKAN_*_EXT in vk_dlss.cpp; any spacing, split over lines) whose statement before is not the + B_DIRECT boundary. Comments are blanked first (newlines kept): a boundary in a // or /* */ comment does not count""" + code = blank_comments(text) + sites = list(call.finditer(code)) + if not sites: + return ["%s: no recording call found (the B_DIRECT check needs updating)" % fname] + return ["%s:%d: a direct recording without boundary(B_DIRECT) as the statement just before" % (fname, code.count("\n", 0, m.start()) + 1) + for m in sites if not B_DIRECT_BEFORE.search(statement_before(code, m.start()))] + + +def opaque_unregistered(text, fname): + """note_work() written out by hand marks commands vk_work.h did not see: each must follow a call this test knows + (OPAQUE_RECORD_CALL), so a new library recording into the command buffer has to be added here, with its boundary""" + code = blank_comments(text) + bad = [] + for m in re.finditer(r"(?::call;" % (f, f, f) in rec + if not re.search(r"\bV\(%s\)" % f, rec) and not written: + problems.append("vk_recorder.h: no packer for %s" % f) + # (defer) the boundaries: RX_REC_BOUNDARIES and vk_work.h's drain-first macros agree + m = re.search(r"#define RX_REC_BOUNDARIES\(K, M\)(.*?)\n\s*\n", rec, re.S) + if not m: + raise SystemExit("RX_REC_BOUNDARIES not found in vk_recorder.h") + bounds = dict((f, k) for k, f in re.findall(r"\b([KM])\((\w+)\)", m.group(1))) + bmacros = dict(re.findall(r"#define (\w+)\(\.\.\.\) \(::mkwvk::rec::boundary\(::mkwvk::rec::B_\w+\), (::vkx::|::)\w+\(__VA_ARGS__\)\)", work)) + for f, q in bmacros.items(): + if "#define %s(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_%s), %s%s(__VA_ARGS__))" % (f, f, q, f) not in work: + problems.append("vk_work.h: %s's macro calls another function" % f) + if set(bmacros) != set(bounds) or any(bmacros[f] != ("::vkx::" if k == "K" else "::") for f, k in bounds.items() if f in bmacros): + problems.append("vk_work.h's boundary macros and RX_REC_BOUNDARIES differ: only in vk_work.h %s, only in vk_recorder.h %s" + % (sorted(set(bmacros) - set(bounds)), sorted(set(bounds) - set(bmacros)))) + unwrapped = {} + unbounded = {} + routed = 0 + for path in sorted(VK.glob("*.cpp")) + sorted(VK.glob("*.h")): + if path.name in ("vk_recorder.h", "vk_work.h", "vk_loader.h"): + continue + text = code(path.read_text(encoding="utf-8", errors="replace")) + used = sorted(set(re.findall(r"\b(vkCmd\w+)\s*\(", text))) + bused = sorted(set(BOUNDARY_NAME.findall(text))) + if path.suffix == ".h": # (a header's calls would be changed or not by the file including it: none make any) + if used or bused: + problems.append("%s (a header) calls %s: move them into a .cpp that includes vk_work.h" % (path.name, used + bused)) + continue + if '#include "vk_work.h"' in text: + routed += 1 + bad = [c for c in used if c not in macros] + if bad: + problems.append("%s calls %s, which vk_work.h does not route through the recorder" % (path.name, bad)) + bad = [c for c in bused if c not in bounds] + if bad: + problems.append("%s calls %s, which vk_work.h does not drain before (RX_REC_BOUNDARIES)" % (path.name, bad)) + last = text.rfind("#include") + if text.rfind('#include "vk_work.h"') != last: + problems.append("%s: vk_work.h is not its last include" % path.name) + else: + if used: + unwrapped[path.name] = used + if bused: + unbounded[path.name] = bused + for name, used in unwrapped.items(): + why = UNWRAPPED_OK.get(name) + print("not through the recorder: %s %s (%s)" % (name, ", ".join(used), why or "NOT EXPECTED")) + if not why: + problems.append("%s calls %s without vk_work.h: not through the recorder" % (name, used)) + for name in set(UNWRAPPED_OK) - set(unwrapped): + print("(%s: now through the recorder, or no vkCmd* calls; its entry in UNWRAPPED_OK can go)" % name) + for name, used in unbounded.items(): + why = BOUNDARY_OK.get(name) + print("boundaries without a drain: %s %s (%s)" % (name, ", ".join(used), why or "NOT EXPECTED")) + if not why: + problems.append("%s calls %s without vk_work.h: RX_VK_RECORDER=defer would not drain before them" % (name, used)) + for name in set(BOUNDARY_OK) - set(unbounded): + print("(%s: no boundary calls without vk_work.h now; its entry in BOUNDARY_OK can go)" % name) + # (vk_imgui.cpp's entries above hold only if every Dear ImGui recording starts after a B_DIRECT boundary) + menu = (VK / "vk_menu.cpp").read_text(encoding="utf-8") + problems += direct_unbounded(menu) + # (controls: each way of losing the last boundary must be caught: a line comment, a block comment over lines, + # removal with the call spaced `ui::record_to (`, removal with the call split over lines) + last = menu.rindex(B_DIRECT_STMT) + gone = menu[:last] + menu[last + len(B_DIRECT_STMT):] + call = gone.index("ui::record_to(", last) + for what, broken in (("in a line comment", menu[:last] + "// " + menu[last:]), + ("in a block comment over lines", menu[:last] + "/*\n" + B_DIRECT_STMT + "\n*/" + gone[last:]), + ("removed, call spaced", gone[:call] + "ui::record_to (" + gone[call + 14:]), + ("removed, call split over lines", gone[:call] + "ui::record_to\n (" + gone[call + 14:])): + if not direct_unbounded(broken): + problems.append("the B_DIRECT check missed a boundary " + what) + # (NVIDIA NGX records into cmd itself as well: each create/evaluate after a B_DIRECT boundary; control: one removed) + dlss = (VK / "vk_dlss.cpp").read_text(encoding="utf-8") + problems += direct_unbounded(dlss, "vk_dlss.cpp", NGX_RECORD_CALL) + if B_DIRECT_STMT in dlss and not direct_unbounded(dlss.replace(B_DIRECT_STMT, "", 1), "vk_dlss.cpp", NGX_RECORD_CALL): + problems.append("the B_DIRECT check missed an NGX recording without its boundary") + # (AMD FSR 4 records into cmd through ffxDispatch: after a B_DIRECT boundary; control: that boundary removed) + fsr4 = (VK / "vk_fsr4.cpp").read_text(encoding="utf-8") + problems += direct_unbounded(fsr4, "vk_fsr4.cpp", FFX_RECORD_CALL) + if not FFX_RECORD_CALL.search(blank_comments(fsr4)): + problems.append("vk_fsr4.cpp: no ffxDispatch call (l.dispatch) found: update FFX_RECORD_CALL") + elif not direct_unbounded(fsr4.replace(B_DIRECT_STMT, "", 1), "vk_fsr4.cpp", FFX_RECORD_CALL): + problems.append("the B_DIRECT check missed an ffxDispatch recording without its boundary") + # (and any other recording vk_work.h does not see must be registered: every hand-written note_work() follows a known one) + for path in sorted(VK.glob("*.cpp")): + problems += opaque_unregistered(path.read_text(encoding="utf-8"), path.name) + if not opaque_unregistered("void f() {\n some_sdk_record(cmd);\n note_work();\n}\n", "control.cpp"): + problems.append("the note_work() registry missed an unknown recording") + if problems: + raise SystemExit("\n".join(problems)) + print("PASS sources: %d calls packed and %d boundaries drained, each with its vk_work.h macro; every call in the %d vk_work.h " + "files routed; no header calls either" % (len(calls), len(bounds), routed)) + return calls + + +def test_loader(): + """vk_loader.h without windows.h and the Win32 surface: the same declarations, on any system""" + text = (VK / "vk_loader.h").read_text(encoding="utf-8") + text = text.replace("#ifdef _WIN32\n#define VK_USE_PLATFORM_WIN32_KHR", "#if 0\n#define VK_USE_PLATFORM_WIN32_KHR", 1) + text = text.replace("#include \n", "", 1) + # (its windows.h switches go too: unused without windows.h, and MinGW's libstdc++ defines NOMINMAX itself (-Werror)) + text = text.replace("#define WIN32_LEAN_AND_MEAN\n#define NOMINMAX\n", "", 1) + if "windows.h" in text or "#if 0" not in text or "#define NOMINMAX" in text: + raise SystemExit("vk_loader.h's shape changed: update test_loader()") + return text + + +def build(cxx, headers, out, recorder_text, name, extra=(), work_text=None): + d = out / name + d.mkdir() + (d / "vk_loader.h").write_text(test_loader(), encoding="utf-8") + (d / "vk_work.h").write_text(work_text or (VK / "vk_work.h").read_text(encoding="utf-8"), encoding="utf-8") + (d / "vk_recorder.h").write_text(recorder_text, encoding="utf-8") + exe = d / "recorder_sync_test.exe" + cmd = [cxx, "-std=c++20", "-O2", "-Wall", "-Wextra", "-Werror", "-Wno-unused-function", "-Wno-missing-field-initializers", "-DVK_NO_PROTOTYPES", + "-D_CRT_SECURE_NO_WARNINGS", + *extra, "-I" + str(d), "-I" + str(headers), str(ROOT / "tests" / "recorder_sync_test.cpp"), "-o", str(exe)] + r = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) + if r.returncode: + raise SystemExit("%s: the build failed:\n%s" % (name, r.stdout)) + return exe + + +def run(exe, env_mode, *args, parallel=None, timeout=600): + env = dict(os.environ) + for k in ("RX_VK_RECORDER", "RX_VK_PARALLEL", "RX_VK_RECORDER_POISON", "RX_VK_RECORDER_RING_MB", "RX_VK_RECORDER_STATS"): + env.pop(k, None) + if env_mode: + env["RX_VK_RECORDER"] = env_mode + if parallel: + env["RX_VK_PARALLEL"] = parallel + try: + return subprocess.run([str(exe), *args], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, env=env, timeout=timeout) + except subprocess.TimeoutExpired as e: # (a hang: reported as exit code None) + out = e.stdout.decode(errors="replace") if isinstance(e.stdout, bytes) else (e.stdout or "") + return subprocess.CompletedProcess(e.cmd, None, out + "\nTIMEOUT after %d s" % timeout) + + +# (the environment runs: RX_VK_RECORDER, RX_VK_PARALLEL; the mode the renderer reads; the log lines it must write, once) +ENV_RUNS = ( + ("sync", None, "env mode 1", ()), + (None, None, "env mode 0", ()), + ("defer", None, "env mode 2", ("LOG: vk: RX_VK_RECORDER=defer:",)), + ("defer", "1", "env mode 2", ("LOG: vk: RX_VK_RECORDER=defer:",)), + ("defer", "4", "env mode 0", ("LOG: vk: RX_VK_RECORDER=defer refused: RX_VK_PARALLEL=4",)), + ("thread", None, "env mode 3", ("LOG: vk: RX_VK_RECORDER=thread: vkCmd* packed",)), + ("thread", "1", "env mode 3", ("LOG: vk: RX_VK_RECORDER=thread: vkCmd* packed",)), + ("thread", "4", "env mode 0", ("LOG: vk: RX_VK_RECORDER=thread refused: RX_VK_PARALLEL=4",)), +) + + +def env_run_ok(exe, env_mode, parallel, want, logs, show=True): + r = run(exe, env_mode, parallel=parallel) + if show: + print("---- RX_VK_RECORDER=%s RX_VK_PARALLEL=%s\n%s" % (env_mode or "(unset)", parallel or "(unset)", r.stdout.rstrip())) + lines = r.stdout.splitlines() + ok = r.returncode == 0 and want in lines and "ALL PASS" in lines + for l in logs: # (exactly once) + ok = ok and sum(1 for x in lines if x.startswith(l)) == 1 + if not logs: + ok = ok and not any(x.startswith(("LOG: vk: RX_VK_RECORDER=defer", "LOG: vk: RX_VK_RECORDER=thread: vkCmd* packed", + "LOG: vk: RX_VK_RECORDER=thread refused")) for x in lines) + return ok, r + + +UNORDERED_END = " default: return false; // (unordered: the end of the list)" + + +def unordered_calls(rec): + """the calls rec::unordered() lists (made by the producer without waiting for the recorder)""" + m = re.search(r"constexpr bool unordered\(Bound k\) \{(.*?)\n\}", rec, re.S) + if not m or UNORDERED_END not in m.group(1): + raise SystemExit("rec::unordered() not found, or its shape changed") + return re.findall(r"^ case B_(\w+): return true;", m.group(1), re.M) + + +def unordered_too(rec, f): + """vk_recorder.h with f among the calls the producer makes without waiting for the recorder""" + return broken(rec, UNORDERED_END, " case B_%s: return true;\n%s" % (f, UNORDERED_END), "unordered") + + +def no_wait(work_h, f): + """vk_work.h with f's boundary macro calling the driver at once (no drain, no wait)""" + return broken(work_h, "#define %s(...) (::mkwvk::rec::boundary(::mkwvk::rec::B_%s), ::vkx::%s(__VA_ARGS__))" % (f, f, f), + "#define %s(...) (::vkx::%s(__VA_ARGS__))" % (f, f), f) + + +EXIT_LINES = ("recorder parked in idle() holding sleep_mu: 1; ExitProcess", "DLL detach: the global destructors ran to the end") + + +def exit_test(cxx, headers, out, recorder_text, name): + """(Windows) recorder_exit_test.cpp's DLL and exe built with this vk_recorder.h; (ok, what it printed); None elsewhere""" + if os.name != "nt": + return None + d = out / name + d.mkdir() + (d / "vk_loader.h").write_text(test_loader(), encoding="utf-8") + (d / "vk_work.h").write_text((VK / "vk_work.h").read_text(encoding="utf-8"), encoding="utf-8") + (d / "vk_recorder.h").write_text(recorder_text, encoding="utf-8") + src = str(ROOT / "tests" / "recorder_exit_test.cpp") + flags = ["-std=c++20", "-O2", "-Wall", "-Wextra", "-Werror", "-DVK_NO_PROTOTYPES", "-D_CRT_SECURE_NO_WARNINGS"] + for cmd in ([cxx, *flags, "-shared", "-DRX_EXIT_DLL", "-I" + str(d), "-I" + str(headers), src, "-o", str(d / "rx_exit.dll")], + [cxx, *flags, src, "-o", str(d / "rx_exit.exe")]): + r = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) + if r.returncode: + raise SystemExit("%s: the exit fixture's build failed:\n%s" % (name, r.stdout)) + try: # (the watchdog: only this child is ended) + r = subprocess.run([str(d / "rx_exit.exe"), str(d / "rx_exit.dll")], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, + text=True, timeout=20) + except subprocess.TimeoutExpired: + return False, "TIMEOUT: the process did not end within 20 s of ExitProcess" + lines = r.stdout.splitlines() + return r.returncode == 0 and all(l in lines for l in EXIT_LINES), "exit code %s: %s" % (r.returncode, " | ".join(lines)) + + +def broken(text, old, new, what): + if text.count(old) != 1: + raise SystemExit("vk_recorder.h's shape changed (%s): update the broken variants" % what) + return text.replace(old, new) + + +def main(): + p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + p.add_argument("--cxx", default="clang++") + p.add_argument("--headers", required=True, help="the Vulkan headers' include folder") + p.add_argument("--bench", type=int, default=0, help="also time the packets: N draws a run") + p.add_argument("--overlap", type=int, default=0, help="also model the overlap: F frames a run") + a = p.parse_args() + check_sources() + + work = ROOT / "work" + work.mkdir(exist_ok=True) + out = Path(tempfile.mkdtemp(prefix="recorder-sync-", dir=work)) + rec = (VK / "vk_recorder.h").read_text(encoding="utf-8") + exe = build(a.cxx, a.headers, out, rec, "real") + failed = False + for env_mode, parallel, want, logs in ENV_RUNS: + if not env_run_ok(exe, env_mode, parallel, want, logs)[0]: + failed = True + if failed: + raise SystemExit("FAIL: the packed calls differ from the direct ones, or the mode or its log line is wrong") + ex = exit_test(a.cxx, a.headers, out, rec, "exit") + if ex is None: + print("---- exit with the recorder parked: SKIP (not Windows)") + else: + print("---- exit with the recorder parked in idle() holding its lock, then ExitProcess: %s" % ex[1]) + if not ex[0]: + raise SystemExit("FAIL: the process does not end cleanly with the recorder parked") + old = broken(broken(rec, "inline Recorder& g_rec = *new Recorder;", "inline Recorder g_rec;", "g_rec"), + " // (no destructor: the recorder lives as long as the process, below)\n", + " ~Recorder() {\n if (th.joinable()) {\n stop_req.store(true);\n wake();\n th.join();\n" + " }\n if (base) ::operator delete(base, std::align_val_t(64));\n }\n", "dtor") + ob = exit_test(a.cxx, a.headers, out, old, "exit_broken") + print("---- broken: the old global recorder with its destructor (wake, join) -> %s\n %s" % ("caught" if not ob[0] else "NOT CAUGHT", ob[1])) + if ob[0]: + raise SystemExit("FAIL: the exit test passes the old destructor") + r = run(exe, None, "--throw", timeout=60) # (the recorder thread's exception: loud, exit code 3) + print("---- --throw: exit code %s\n%s" % (r.returncode, r.stdout.rstrip())) + if r.returncode != 3 or "FATAL: RX_VK_RECORDER=thread" not in r.stdout: + raise SystemExit("FAIL: an exception on the recorder thread did not end the process loudly") + + work_h = (VK / "vk_work.h").read_text(encoding="utf-8") + # (each: what is broken, vk_recorder.h, vk_work.h, the environment it runs in) + variants = [ + ("shallow copies (dup hands the caller's pointer on)", + broken(rec, " std::memcpy(static_cast(d), p, sizeof(T) * n);\n return d;\n", + " (void)d;\n return p;\n", "dup"), work_h, ("sync", None)), + ("descriptor writes' image infos not copied", + broken(rec, "c[i].pImageInfo = e.dup(w[i].pImageInfo, w[i].descriptorCount);", + "c[i].pImageInfo = w[i].pImageInfo;", "push descriptors"), work_h, ("sync", None)), + ("fallback without the drain (order)", + broken(rec, " else drain();\n t_arena.fallbacks++;", " t_arena.fallbacks++;", "fallback"), work_h, ("sync", None)), + ("defer: no drain before vkEndCommandBuffer", + rec, no_wait(work_h, "vkEndCommandBuffer"), (None, None)), + ("defer: the replay lock let go without a drain", + broken(rec, " boundary(B_REPLAY_UNLOCK);\n t_prod = nullptr;", " t_prod = nullptr;", "unlock"), work_h, (None, None)), + ("defer: exception safety broken (a throwing chunk allocation)", + broken(broken(rec, "::operator new(cap, std::align_val_t(64), std::nothrow)", "::operator new(cap, std::align_val_t(64))", "new"), + "void* alloc(size_t n, size_t align) noexcept {", "void* alloc(size_t n, size_t align) {", "noexcept"), work_h, (None, None)), + ("defer: an allocation failure's call made before the held ones (no drain)", + broken(rec, " if (md == DEFER) drain(); // (what is held first", " if (0) drain(); // (what is held first", + "failure drain"), work_h, (None, None)), + ("defer: the parallel refusal left out (RX_VK_PARALLEL=4)", + broken(rec, " if (!parallel_configured()) return e[0] == 'd' ? DEFER : THREAD;", " return e[0] == 'd' ? DEFER : THREAD;", + "parallel"), work_h, ("defer", "4")), + # (the recorder thread's: run with --only thread) + ("thread: no wait before vkEndCommandBuffer", rec, no_wait(work_h, "vkEndCommandBuffer"), ("--only", None)), + ("thread: no wait before vkQueueSubmit2", rec, no_wait(work_h, "vkQueueSubmit2"), ("--only", None)), + ("thread: the replay lock let go without waiting for the recorder", + broken(rec, " boundary(B_REPLAY_UNLOCK);\n t_prod = nullptr;", " t_prod = nullptr;", "unlock"), work_h, ("--only", None)), + ("thread: a command pool reset without waiting for the recorder", rec, no_wait(work_h, "vkResetCommandPool"), ("--only", None)), + ("thread: no backpressure (the ring overwritten)", + broken(broken(rec, " void wait_room(uint64_t need) {\n", " void wait_room(uint64_t need) {\n if (need) return;\n", "room"), + " lim = base + off + room;", " lim = base + off + tail;", "lim"), work_h, + ("--only", None)), + ("thread: a packet bigger than its room written into the ring when it has more free (the room not the limit)", + broken(rec, " lim = base + off + room;", " lim = base + off + std::min(tail, cap - (w - r_seen));", + "room limit"), work_h, ("--only", None)), + ("thread: no join at teardown", broken(rec, " th.join();\n ready.store(false);", " ready.store(false);", "join"), work_h, + ("--only", None)), + ("thread: the recorder thread's exception swallowed", + broken(rec, " recorder_fatal(e.what());", " (void)e;", "fatal"), work_h, ("--throw", None)), + # (rec::unordered(): a call that does depend on the ring added to it must fail a check) + ("thread: vkQueueSubmit2 made without waiting (added to unordered())", unordered_too(rec, "vkQueueSubmit2"), work_h, ("--only", None)), + ("thread: direct recording (Dear ImGui) without waiting (B_DIRECT added to unordered())", unordered_too(rec, "DIRECT"), work_h, + (None, None)), + ("thread: vkDestroyBuffer made without waiting (added to unordered())", unordered_too(rec, "vkDestroyBuffer"), work_h, ("--only", None)), + ("thread: vkEndCommandBuffer made without waiting (added to unordered())", unordered_too(rec, "vkEndCommandBuffer"), work_h, ("--only", None)), + ("thread: vkGetQueryPoolResults made without waiting (added to unordered())", unordered_too(rec, "vkGetQueryPoolResults"), work_h, + ("--only", None)), + ] + [ # (the calls it lists made with a wait after all (boundary() ignoring unordered()): the check that each runs with packets + # in flight must fail; with none listed, nothing to check) + ("thread: the calls of unordered() (%s) waiting after all" % ", ".join(unordered_calls(rec)), + broken(rec, " else if (unordered(k)) return;\n", "", "skip"), work_h, ("--only", None)) + ][:1 if unordered_calls(rec) else 0] + [ + ("thread: the parallel refusal left out (RX_VK_PARALLEL=4)", + broken(rec, " if (!parallel_configured()) return e[0] == 'd' ? DEFER : THREAD;", " return e[0] == 'd' ? DEFER : THREAD;", + "parallel"), work_h, ("thread", "4")), + ] + for i, (what, text, wtext, (env_mode, parallel)) in enumerate(variants): + exe_b = build(a.cxx, a.headers, out, text, "broken%d" % i, work_text=wtext) + if parallel: + refused = [e for e in ENV_RUNS if e[0] == env_mode and e[1] == "4"][0] + ok, r = env_run_ok(exe_b, env_mode, parallel, "env mode 0", refused[3], show=False) + caught = not ok + elif env_mode == "--only": + r = run(exe_b, None, "--only", "thread", timeout=120) + caught = r.returncode != 0 + elif env_mode == "--throw": + r = run(exe_b, None, "--throw", timeout=30) + caught = not (r.returncode == 3 and "FATAL: RX_VK_RECORDER=thread" in r.stdout) + else: + r = run(exe_b, env_mode) + caught = r.returncode != 0 + lines = [l for l in r.stdout.splitlines() if l.startswith(("MISMATCH", "FAIL", "PASS sync", "PASS deferred", "env mode", "TIMEOUT", + "the recorder"))] + if not lines: + lines = ["(no output: exit code %s)" % r.returncode] + print("---- broken: %s -> %s (exit code %s)\n %s" % (what, "caught" if caught else "NOT CAUGHT", r.returncode, "\n ".join(lines[:6]))) + if not caught: + raise SystemExit("FAIL: a broken recorder (%s) passed" % what) + if a.bench or a.overlap: # (with the renderer's CPU flags, nx2/build.py's CPU: RX_MARCH or x86-64-v3) + fast = build(a.cxx, a.headers, out, rec, "bench", ["-march=" + os.environ.get("RX_MARCH", "x86-64-v3"), "-ffp-contract=off"]) + if a.bench: + r = run(fast, None, "--bench", str(a.bench)) + print(r.stdout.rstrip()) + if a.overlap: + r = run(fast, None, "--overlap", str(a.overlap)) + print(r.stdout.rstrip()) + shutil.rmtree(out, ignore_errors=True) + print("PASS recorder sync: identical streams, poisoned caller memory, coverage, fallback order; broken variants caught") + + +if __name__ == "__main__": + main() diff --git a/tests/test_tex_spans_once_a_present.py b/tests/test_tex_spans_once_a_present.py new file mode 100644 index 0000000..ba786bd --- /dev/null +++ b/tests/test_tex_spans_once_a_present.py @@ -0,0 +1,90 @@ +"""The texture spans' ageing (gpu/vk/vk_memory.cpp publish_tex_spans): a span unused for 1800 presents is dropped, and the +spans are looked over for that once a present, not at each of the present's submissions. A span noted between two +submissions of one present is still published by the next. + +python -I -B tests/test_tex_spans_once_a_present.py [--cxx clang++] +""" +import argparse +from pathlib import Path +import subprocess +import tempfile + +ROOT = Path(__file__).resolve().parents[1] + + +def block(text, marker): + start = text.index(marker) + end = text.index('{', start) + 1 + depth = 1 + while depth: + depth += (text[end] == '{') - (text[end] == '}') + end += 1 + return text[start:end] + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--cxx', default='g++') + parser.add_argument('--source', type=Path, default=ROOT / 'gpu/vk/vk_memory.cpp') + args = parser.parse_args() + publish = block(args.source.read_text(encoding='utf-8'), 'void publish_tex_spans(uint64_t frame) {') + fixture = r''' +#include +#include +#include +#include +#include +#include +#include +#include +#define CHECK(ok) do { if (!(ok)) { std::fprintf(stderr, "failed line %d\n", __LINE__); std::exit(1); } } while (0) +// (the spans, counting each time they are gone through from the start) +struct Spans : std::map> { + int walks = 0; + iterator begin() { walks++; return std::map>::begin(); } +}; +Spans tex_spans; +std::vector> tex_merged, tex_cap_ranges, dropped; +bool tex_spans_changed = false; +uint64_t tex_merged_bytes = 0, cap_ranges_ver = 0; +std::mutex shadow_mu, cap_ranges_mu; +void shadow_drop_range(uint64_t lo, uint64_t hi) { dropped.push_back({lo, hi}); } +''' + publish + r''' +int main() { + tex_spans.try_emplace(0x1000, 0x3000, 100); // (lo -> hi, the present it was last uploaded from) + tex_spans.try_emplace(0x8000, 0x9000, 100); + tex_spans_changed = true; + publish_tex_spans(100); // (the present's first submission: looked over, and published) + CHECK(tex_cap_ranges.size() == 2 && tex_merged_bytes == 0x3000 && cap_ranges_ver == 1 && !tex_spans_changed); + const int after_first = tex_spans.walks; + for (int i = 0; i < 19; i++) publish_tex_spans(100); // (its other submissions: nothing to do) + CHECK(tex_spans.walks == after_first && cap_ranges_ver == 1 && dropped.empty()); + tex_spans.try_emplace(0x20000, 0x21000, 100); // (a texture uploaded between two of them) + tex_spans_changed = true; + publish_tex_spans(100); + CHECK(tex_cap_ranges.size() == 3 && tex_merged_bytes == 0x4000 && cap_ranges_ver == 2 && dropped.empty()); + const int same_present = tex_spans.walks; + publish_tex_spans(101); // (the next present: looked over again, once) + publish_tex_spans(101); + CHECK(tex_spans.walks == same_present + 1 && cap_ranges_ver == 2); + tex_spans[0x8000].second = 1500; // (used again later than the others) + publish_tex_spans(1900); // (1800 presents after 100: kept, the bound is past 1800) + CHECK(dropped.empty() && tex_cap_ranges.size() == 3); + publish_tex_spans(1901); // (the spans last used at 100 go, with their shadow pages; the one used at 1500 stays) + CHECK(dropped.size() == 2 && dropped[0] == std::make_pair(uint64_t(0x1000), uint64_t(0x3000)) && + dropped[1] == std::make_pair(uint64_t(0x20000), uint64_t(0x21000))); + CHECK(tex_spans.size() == 1 && tex_cap_ranges.size() == 1 && tex_cap_ranges[0].first == 0x8000 && tex_merged_bytes == 0x1000 && + cap_ranges_ver == 3 && !tex_spans_changed); + std::puts("PASS: spans aged once a present, a span noted mid-present published, unused spans dropped after 1800"); +} +''' + with tempfile.TemporaryDirectory(prefix='tex-spans-') as tmp: + cpp = Path(tmp) / 'fixture.cpp' + exe = Path(tmp) / 'fixture.exe' + cpp.write_text(fixture, encoding='utf-8') + subprocess.run([args.cxx, '-std=c++20', '-Wall', '-Wextra', '-Werror', str(cpp), '-o', str(exe)], check=True) + subprocess.run([str(exe)], check=True, timeout=60) + + +if __name__ == '__main__': + main() diff --git a/tests/test_vram_compact.cpp b/tests/test_vram_compact.cpp index 872f457..f1cc120 100644 --- a/tests/test_vram_compact.cpp +++ b/tests/test_vram_compact.cpp @@ -9,6 +9,10 @@ int main() { using namespace vram_compact; const uint64_t MB = 1ull << 20; + // on unless RX_VK_COMPACT says 0: not set is on (the default), and 1 still is + CHECK(enabled(nullptr)); + CHECK(enabled("1") && enabled("2")); + CHECK(!enabled("0") && !enabled("")); // pressed: the driver's count over 90% of what it allows; never without an allowance CHECK(!pressed(9000 * MB, 10900 * MB)); CHECK(pressed(10023 * MB, 10900 * MB)); // (the uncompacted walk's count) @@ -31,6 +35,63 @@ int main() { CHECK(held_budget(budget, 300 * MB, 10800 * MB, 0, allowed, 85) == 0); // (the excess is more than the textures hold) CHECK(held_budget(budget, 5600 * MB, 10023 * MB, 215 * MB, 0, 85) == budget); // (no allowance known: left alone) CHECK(held_budget(budget, 5600 * MB, 100 * MB, 215 * MB, allowed, 85) == budget); // (more to give back than is used) + // the hold's pass keeps a live texture used in the last ten seconds: the eviction's own 60 presents is not enough for it + CHECK(kHoldUnused == 600); + CHECK(hold_keeps(0) && hold_keeps(60) && hold_keeps(599)); // (out of view for a second: a turning camera brings it back) + CHECK(!hold_keeps(600) && !hold_keeps(3600)); + // the reserve: a line five points under the hold's, for the textures unused thirty seconds + CHECK(reserve_pct(85) == 80 && reserve_pct(50) == 45 && reserve_pct(3) == 0); + CHECK(kReserveUnused == 1800 && kReserveUnused > kHoldUnused); + const uint64_t none = UINT64_MAX; // (the other rules ask nothing: the hold's two lines alone) + CHECK(hold_target(59, 60, none, 5000 * MB, 4500 * MB) == UINT64_MAX); // (used a second ago: stays, whatever the lines ask) + CHECK(hold_target(599, 60, none, 5000 * MB, 4500 * MB) == UINT64_MAX); + CHECK(hold_target(600, 60, none, 5000 * MB, 4500 * MB) == 5000 * MB); // (unused ten seconds: for the hold's own line) + CHECK(hold_target(1799, 60, none, 5000 * MB, 4500 * MB) == 5000 * MB); + CHECK(hold_target(1800, 60, none, 5000 * MB, 4500 * MB) == 4500 * MB); // (unused thirty: down to the reserve's line) + CHECK(hold_target(600, 60, none, 0, 4500 * MB) == UINT64_MAX); // (the hold's line not crossed: only the long unused go) + CHECK(hold_target(1800, 60, none, 0, 4500 * MB) == 4500 * MB); + // the other rules' part: over the budget's size, over 90% of the allowance or at the driver's limit, a texture unused for + // the eviction's own 60 presents goes for what those rules ask, as with the compaction off; the hold's patience is for its + // own lines + CHECK(rule_target(5600 * MB, 6144 * MB) == UINT64_MAX); // (under the budget: they ask nothing) + CHECK(rule_target(5600 * MB, 5600 * MB) == UINT64_MAX); + CHECK(rule_target(5600 * MB, 5300 * MB) == 5300 * MB / 8 * 7); // (over it: cut to 7/8 of it, as ever) + CHECK(hold_target(100, 60, 4637 * MB, 5000 * MB, 4500 * MB) == 4637 * MB); // (the count over 90%: one unused 100 presents still goes) + CHECK(hold_target(59, 60, 4637 * MB, 5000 * MB, 4500 * MB) == UINT64_MAX); // (one used in the last 60: never) + CHECK(hold_target(100, 120, 4637 * MB, 5000 * MB, 4500 * MB) == UINT64_MAX); // (nor before the pass's own time, whatever that is) + CHECK(hold_target(700, 60, 5200 * MB, 5000 * MB, 4500 * MB) == 5000 * MB); // (the lowest of the lines that apply) + CHECK(hold_target(700, 60, 4000 * MB, 5000 * MB, 4500 * MB) == 4000 * MB); + CHECK(hold_target(2000, 60, 4000 * MB, 5000 * MB, 4500 * MB) == 4000 * MB); + CHECK(hold_target(2000, 60, 5200 * MB, 5000 * MB, 4500 * MB) == 4500 * MB); + CHECK(hold_target(700, 60, 4000 * MB, 0, 4500 * MB) == 4000 * MB); // (the hold's line not crossed, the rules asking) + { // a count at 94% of a 7,000 MB allowance (6,600 MB used, 100 of it to give back; the textures hold 2,400): the 90% rule's + // budget is 2,200 MB and a pass of its own goes to 1,925; the hold and the reserve ask for less still, so the pass is the + // hold's. A texture unused 100 presents goes until the 90% rule's part is covered, and no further + const uint64_t a = 7000 * MB, total = 2400 * MB; + const uint64_t rule = total - (6500 - 6300) * MB; // (the 90% rule: the excess over 6,300 MB off the textures) + const uint64_t hold = held_budget(rule, total, 6600 * MB, 100 * MB, a, 85); + const uint64_t res = held_budget(hold, total, 6600 * MB, 100 * MB, a, reserve_pct(85)); + CHECK(hold == total - (6500 - 5950) * MB && res == total - (6500 - 5600) * MB); + const uint64_t to = hold_target(100, 60, rule_target(total, rule), hold, res); + CHECK(to == 2200 * MB / 8 * 7); + CHECK(!hold_met(total, 0, 0, to) && !hold_met(total, 400 * MB, 0, to) && hold_met(total, 475 * MB, 0, to)); + CHECK(hold_met(total, 100 * MB, 375 * MB, to)); // (the dead render targets count for it first, as for the other lines) + CHECK(hold_target(700, 60, rule_target(total, rule), hold, res) == hold); // (older ones go on to the hold's line) + CHECK(hold_target(1800, 60, rule_target(total, rule), hold, res) == res); // (and the long unused to the reserve's) + CHECK(hold_target(100, 60, rule_target(2100 * MB, rule), hold, res) == UINT64_MAX); // (under that budget: the hold's patience again) + } + // a count between the two lines: the reserve asks for less than the budget, the hold for nothing + CHECK(held_budget(budget, 5000 * MB, 9000 * MB, 200 * MB, allowed, 85) == budget); + CHECK(held_budget(budget, 5000 * MB, 9000 * MB, 200 * MB, allowed, reserve_pct(85)) == 5000 * MB - (9000 - 200 - 8720) * MB); + // and it is done with live textures once those chosen and the dead render targets together cover the excess + CHECK(!hold_met(5600 * MB, 0, 0, 5000 * MB)); // (600 MB over, nothing chosen yet) + CHECK(!hold_met(5600 * MB, 599 * MB, 0, 5000 * MB)); + CHECK(hold_met(5600 * MB, 600 * MB, 0, 5000 * MB)); + CHECK(hold_met(5600 * MB, 0, 600 * MB, 5000 * MB)); // (the dead targets alone cover it: no live texture goes) + CHECK(hold_met(5600 * MB, 0, 2500 * MB, 5000 * MB)); // (the walk's 2.5 GB of dead targets) + CHECK(!hold_met(5600 * MB, 100 * MB, 400 * MB, 5000 * MB)); // (100 MB still to go) + CHECK(hold_met(5600 * MB, 200 * MB, 400 * MB, 5000 * MB)); + CHECK(hold_met(300 * MB, 200 * MB, 200 * MB, 0)); // (more chosen than the textures hold, their sum having drifted) std::printf("PASS: the compaction's rules\n"); return 0; } diff --git a/tests/test_vram_compact.py b/tests/test_vram_compact.py index 3156745..91e2538 100644 --- a/tests/test_vram_compact.py +++ b/tests/test_vram_compact.py @@ -1,5 +1,5 @@ """Compile and run the VRAM compaction's rules (gpu/vk/vram_compact.h; no game/GPU required), and check that vk_texture.cpp and -vk_vma.cpp still use them: the textures in a pool of their own only with RX_VK_COMPACT, nothing of that pool freed while a pass is +vk_vma.cpp still use them: the textures in a pool of their own unless RX_VK_COMPACT=0, nothing of that pool freed while a pass is open, and the budget holding the driver's count with the textures' free space alone taken off. python -I -B tests/test_vram_compact.py [--cxx g++] @@ -31,8 +31,9 @@ def need(where, text, needles): def check_source(): tex = (ROOT / "gpu/vk/vk_texture.cpp").read_text(encoding="utf-8") vma = (ROOT / "gpu/vk/vk_vma.cpp").read_text(encoding="utf-8") - # (the compaction is opt-in) - need("vk_texture.cpp: compact_on()", block(tex, "bool compact_on()"), ['getenv("RX_VK_COMPACT")']) + # (Triforce: the switch decides when it is set; unset, the compaction is on where the memory handling is, NVIDIA cards and + # the other makes on trial: memory_trial_on. Upstream's rule, on unless RX_VK_COMPACT=0, is vram_compact::enabled, run below) + need("vk_texture.cpp: compact_on()", block(tex, "bool compact_on()"), ['getenv("RX_VK_COMPACT")', "memory_trial_on()"]) # (a texture that is not a render target's goes to the textures' pool only when they are compacted, in a branch before the # render targets' pool and blocks, which stay as they are) make = block(tex, "Texture* create(const Key& key") @@ -50,8 +51,31 @@ def check_source(): ["vram_compact::reclaimable(block_free, pool_free)", "vram_compact::pressed(used, allowed)", "vram_compact::held_budget(budget, total_bytes, used, reclaim, allowed, pct)", 'getenv("RX_VK_COMPACT_PCT")', # (the hold lets go of the excess alone, not of an eighth of the cache; the driver's limit still asks for 7/8) - "if (held < budget) budget = evict_to = held;", "if (budget < evict_to) evict_to = 0;", - "total_bytes - pending_bytes <= (evict_to ? evict_to : budget / 8 * 7)"]) + "if (held < budget) budget = evict_to = held;", + # (the hold's pass: every dead render target, and live textures only for the excess left and only long unused ones; + # it does not stop at the first texture past the excess, which kept it from the dead targets, the youngest) + # (and what the other rules ask on their own is kept from before the hold's lines, the 97% rule's with it, for the + # pass's third tier: a texture unused for the eviction's own time goes for that part) + "uint64_t rule_budget = budget;", "rule_budget = std::min(rule_budget, total_bytes > over ? total_bytes - over : 0);", + "const uint64_t rule_to = vram_compact::rule_target(total_bytes, rule_budget);", + # (the reserve: a lower line that sets the budget too, the hold's own line kept for the textures not long unused; + # the driver's limit, when it asks for more, clears both) + "hold_to = evict_to;", "vram_compact::reserve_pct(pct));", "if (reserve < budget) budget = evict_to = reserve;", + "if (budget < evict_to) evict_to = hold_to = 0;", + # (Triforce: the pass itself is texture_sweep here, split from texture_frame on 11-11 for the dead targets' own + # passes; the hold's two lines go to it as arguments) + "texture_sweep(completed, total_bytes <= budget, budget, evict_to, known, used, allowed, false, hold_to, rule_to);"]) + sweep_fn = block(tex[tex.index("void texture_frame(uint64_t completed)"):], "void texture_sweep(uint64_t completed") + need("vk_texture.cpp: texture_sweep()", sweep_fn, + ["const bool by_hold = evict_to != 0 && !sweep;", + "if (shifted || drawn_over(tex)) hold_dead += tex->bytes + tex->pad;", + "if ((!sweep && !by_hold && (pending_bytes >= total_bytes || total_bytes - pending_bytes <= budget / 8 * 7)) ||", + "const uint64_t to = vram_compact::hold_target(frame_no - last, need_unused, rule_to, hold_to, evict_to);", + "if (to == UINT64_MAX || vram_compact::hold_met(total_bytes, hold_live, hold_dead, to)) continue;", + "if (by_hold && !overwritten) hold_live += tex->bytes + tex->pad;"]) + # (and a live texture is judged by the hold only once it is known whether a render target drew over it) + if not sweep_fn.index("const bool overwritten = shifted || drawn_over(tex);") < sweep_fn.index("vram_compact::hold_target(frame_no - last"): + raise SystemExit("vk_texture.cpp: the hold's pass judges a texture before it knows whether it is a dead render target") need("vk_texture.cpp: compact_step()", block(tex, "void compact_step(uint64_t completed)"), ["if (!compact_on()) return;", "vram_compact::run_from(min_free, compact.pressed)", "vram_compact::rest(compact.pressed)", "t != present_source", "image_compact_ignore(i);", "destroy_gen++;"])