diff --git a/AUDIT.md b/AUDIT.md index a7346cea..16a58342 100644 --- a/AUDIT.md +++ b/AUDIT.md @@ -1075,7 +1075,7 @@ back. | 68 | Lazy write-back command buffer and fence | `lz_init_lock` | One | Never freed explicitly | | 69 | Upload batches (command buffers and fences: inflight and spare) | `cmd_lock` | Grow only | Reused; shutdown | | 70 | Staging cache | `cmd_lock` | `STG_CACHE` 24 buffers, `STG_CACHE_BYTES` 192 MiB | At batch retire | -| 71 | Staging VMA pool | — | `STG_BLOCK` 64 MiB blocks, `STG_MIN_BLOCKS` 2; requests ≤ 32 MiB | — | +| 71 | Staging VMA pool | — | `STG_BLOCK` 64 MiB blocks; `STG_MIN_BLOCKS` 4 kept on a discrete GPU (256 MiB of host memory), `STG_MIN_BLOCKS_UMA` 2 on an integrated one; requests ≤ 32 MiB; an open batch is submitted at `STG_FLUSH` 64 MiB of staging | — | | 72 | Staging arena | `cmd_lock` | `ARENA_CHUNK` 8 MiB, ≤ 2 MiB per request, `ARENA_KEEP` 8; `NX2NAVI_STG_ARENA` | At fence; shutdown | | 73 | Deferred destruction list `dead[]` (images, views, buffers, host frees) | `cmd_lock` | None found | `vk_mem_collect` on the completion thread once the stamped value is reached | | 74 | Headroom spare (half a VMA block per device-local type; block `NX2NAVI_VMA_BLOCK_MB` 64, 16..256) | "vk headroom" | 85 % of VMA's budget; `NX2NAVI_VRAM_HEADROOM=0` off | Replaced, dropped when there is no room or commit is low, at stop | diff --git a/docs/architecture.md b/docs/architecture.md index 3b7b6af5..2b77aef4 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -121,7 +121,10 @@ keeps per-core data without locks and relies on that, so the host models it (`ru prepared shader variants built into the driver's own cache when a sample says it lacks them (`vk_pipeline.c` `bf_decide`, `bf_wait`). `boot-warm-owed.txt` next to the caches lets a start go on with a warm that an earlier one began and did not end, three starts at most, and says when the warm for a GPU and driver is done - (`runtime/boot_warm.h`); `NX2NAVI_PREWARM_FULL=0` turns it off. + (`runtime/boot_warm.h`); it waits `NX2NAVI_PREWARM_FULL_S` seconds at most (600), after which the rest is built + beside the game, and `NX2NAVI_PREWARM_FULL=0` turns it off. A start that dies in the boot's builds leaves a + marker or a journal; the crash handler's note (`vk_pipeline_crash_note`) tells the next start whether that was + a crash, which skips the prebuild or the variants, or a start that was closed or stopped, which skips nothing. `NX2NAVI_PREWARM_WAIT_MS` caps the whole wait, after which the rest finishes in the background. - Translator updates: every cache key names the translator, so a new one has no translations. The cache carries the previous translator's (`NX2NAVI_SHADER_CARRY`, `runtime/gfx_shader.c` "Carry"): `shader-cache/identities.txt` diff --git a/docs/building.md b/docs/building.md index 42f0b5e0..d353e93b 100644 --- a/docs/building.md +++ b/docs/building.md @@ -580,7 +580,8 @@ as short. With the executable it also runs `--probe-memneed` (it prints the mach starts nothing) with `mem_need_mb` at 0, at 1 and at a value no machine has, and reads the warning line of the last. A change to when a start does the whole warm on the boot screen (`runtime/boot_warm.h`, `runtime/vk_pipeline.c` `bf_decide`) passes `python tests\boot_warm_check.py`, which builds the header alone: the warm after a driver change, -its continuation by a later start, the end of the tries, and no second warm once one is done. The warm itself shows +its continuation by a later start, the end of the tries, no second warm once one is done, and what a marker left by +a start that ended in the boot's builds means with and without the crash handler's note. The warm itself shows in `--probe-prewarm` on a copy of an install's caches whose `pipeline-cache.bin` has another driver's header (bytes 8 to 15 of the file are the vendor and device ids, the 16 after them the driver's cache id): the log names the whole warm, the compute pipelines on every core and the library warm's answer, and `boot-warm-owed.txt` ends as "done". diff --git a/nx2navi.ini.example b/nx2navi.ini.example index 04faf6e7..826bba77 100644 --- a/nx2navi.ini.example +++ b/nx2navi.ini.example @@ -245,6 +245,10 @@ # 1: every start waits until every known pipeline is built. # 0: never more than the 20 seconds, also after a driver update. # prewarm_full = auto +# The most seconds that first start after a driver update waits (0: no limit). When they are up the game starts and +# the rest is built in the background while you play, as on any other start. Closing the window while it waits is +# safe: the next start goes on with it. +# prewarm_full_s = 600 # Threads that translate known shaders at start (1 to 64; default: half the CPU threads, at most 16). # prewarm_xlate = 8 # After an update that brings a new shader translator, 1 keeps using the shader cache's translations from the earlier @@ -328,7 +332,8 @@ # video memory: 1 turns on the settings written for them and measured on a Radeon RX 580, for every key below that # this file does not set itself. Mesh shaders run as compute where the card has none (mesh_emulation), video memory # stays within what the driver reports free (vram_heap_limit = auto, vram_reserve_mb = 512), large texture upload -# batches go to the GPU early (upload_batch_mb = 256), game memory is copied for the GPU while a frame is recorded and +# batches go to the GPU early (upload_batch_mb = 256; every batch goes at 64 MiB of staging at the latest with or +# without it, so only a value below 64 acts), game memory is copied for the GPU while a frame is recorded and # the game waits less for the GPU (mirror_stage = 1, submit_finish_gpu = 0), frames are prepared on a thread of their # own (replay_async = 1, which can show a dark or flickering frame now and then), pools the game refills every frame # stay in system memory (mirror_hot = 25), and a texture read after another image was drawn over its memory gets @@ -545,7 +550,9 @@ # What DLSS works from: dlaa (the 3D scene at full size), quality, balanced, performance or ultra_performance (the # scene at 67, 58, 50 or 33 % of the output, scaled up by DLSS; these replace render_scale and sharpen texture filtering # to match). They apply only with dlaa = 1 and the DLSS model next to the executable; otherwise the scene follows -# render_scale. Changes while the game runs take effect at once. +# render_scale. Changes while the game runs take effect at once, with one limit: the game's memory for render targets +# is sized at the start for the mode set then, so dlaa chosen in the menu of a run that started with a smaller mode +# (or with dlaa = 0) renders at 75 % until the next start. # dlss = dlaa # AMD FSR in DLSS's place, on any GPU (Windows): with taa = fsr3 and dlaa = 1 the scene is scaled by AMD FSR 3.1 in the # mode dlss names (dlaa is FSR's Native AA). The menu's Upscaling row sets all of these. Changes while the game runs diff --git a/runtime/boot_warm.h b/runtime/boot_warm.h index dc65a9d7..106702ad 100644 --- a/runtime/boot_warm.h +++ b/runtime/boot_warm.h @@ -8,6 +8,12 @@ // goes to the play cache until that is folded in), and "done" is what keeps those starts ordinary ones. After // BOOT_WARM_TRIES starts that began the warm and did not end it the file stays as it is, and starts on that GPU and // driver are ordinary ones too. +// +// boot_end_decide: what a marker means that a start left behind while the boot built pipelines +// (/pipeline-prewarm-running.txt, " "; vk_pipeline.c +// pm_skipped). With the crash handler's note the driver crashed in those builds and the prebuild is skipped on that +// GPU and driver. Without it the start was closed or stopped, which only counts; BOOT_QUIET_ENDS such ends in a row +// skip the prebuild as well, for the driver that ends the process without a fault the handler sees. #pragma once #include @@ -37,3 +43,21 @@ static inline int boot_warm_decide(const char *text, const char *key, int foreig *tries = (int)n + 1; return BOOT_WARM_YES; } + +#define BOOT_QUIET_ENDS 3 + +// OTHER: the marker is another GPU's or driver's and means nothing here. SKIP: the prebuild is skipped on this GPU +// and driver from now on. COUNTED: this start builds again, and writes *quiet into its own marker. +enum { BOOT_END_OTHER, BOOT_END_SKIP, BOOT_END_COUNTED }; + +// text: the marker's content (a marker of 0.9.2 holds the GPU and driver alone). crashed: the crash handler's note was +// there. *quiet: the ends in a row without a note, this one counted; 0 with BOOT_END_OTHER. +static inline int boot_end_decide(const char *text, const char *key, int crashed, int *quiet) { + char was[64] = ""; + int n = 0; + if (sscanf(text, "%63s %d", was, &n) < 2 || n < 0) n = 0; + *quiet = 0; + if (strcmp(was, key)) return BOOT_END_OTHER; + *quiet = n + 1; + return crashed || n + 1 >= BOOT_QUIET_ENDS ? BOOT_END_SKIP : BOOT_END_COUNTED; +} diff --git a/runtime/cfg.c b/runtime/cfg.c index 873733ff..f25ce313 100644 --- a/runtime/cfg.c +++ b/runtime/cfg.c @@ -61,7 +61,7 @@ static const char *const known[] = { "MVOBJ_ALL", "MVOBJ_BANKS", "MVOBJ_EQUAL", "MVOBJ_EXACT", "MVOBJ_EXEC", "MVOBJ_NOC4", "MVOBJ_ONLY", "MVOBJ_TIMES", "MVPREV_CHECK", "MVPREV_GROW", "MVPREV_SKIP", "MVPREV_TRUST", "MV_CHECK", "MV_SURVEY", "PIPE_SEED_BOOT", "ONE_PER_CORE", "OPERATION_MODE", "OUTPUT_RES", "PAD_LAYOUT", "PACE_HZ", "PASS_CRUMBS", "PATCH_OUTPUT", "PATCH_RES_FIXED", "PATCH_SCENE", "PATCH_WIDE_ASPECT", "PATCH_WIDE_HUD", "PATCH_WIDE_MOVIE", "PATCH_WIDE_OUT", "PATCH_FREECAM", "PATCH_OCCLUSION_OFF", "PATCH_WIDE_SAVE", "PATCH_WIDE_SCENE", "PCSS", "PCSS_K", "PERF_OVERLAY", "PERF_SLOW_MS", "PIPELINE_CACHE", "PIPELINE_CACHE_MAX_MB", "PIPELINE_CACHE_PLAY", "PIPELINE_CACHE_PLAY_MB", "PIPE_BINARIES", "PIPE_BINARIES_MAX_MB", "PIPE_FEEDBACK", "PIPE_MEMO", "PIPE_PREBUILD", "PIPE_SEED", "PIPE_SKIP", "PIPE_SYNC", "PIPE_WAIT", "POWER_FULL", "TEX_SAME", "TEX_SAME_BIG", "TEX_FLAT3D", "TEX_KEEP_TAKEN", "TEX_STALE_KEPT", "PRESENT_HELD", "HELD_SYNC", - "PREPARE_PIPELINES", "PREWARM_FULL", "PREWARM_WAIT_MS", "PREWARM_XLATE", "PROBE_MOVIE", "PROBE_MOVIE_SHOT", "PROFILE", "PROFILE_CHAINS", "PROFILE_DIR", "PROFILE_FRAMES", "PROFILE_REPLAY", "PROFILE_SLOW", + "PREPARE_PIPELINES", "PREWARM_FULL", "PREWARM_FULL_S", "PREWARM_WAIT_MS", "PREWARM_XLATE", "PROBE_MOVIE", "PROBE_MOVIE_SHOT", "PROFILE", "PROFILE_CHAINS", "PROFILE_DIR", "PROFILE_FRAMES", "PROFILE_REPLAY", "PROFILE_SLOW", "POOL_FAST", "POOL_GROUP", "POOL_GROUP_CHECK", "POOL_PRESNAP", "POOL_PRESNAP_RUNS", "SUBMIT_FINISH_GPU", "SVC_WAITS", "SNAP_WIDEN", "SUBMIT_FINISH_RESULTS", "SUBMIT_FINISH_KINDS", "FOLIAGE", "MIRROR_STAGE", "FLUSH_SKIP_CLEAN", "MIRROR_HOT", "MESH_EMULATION", "MESH_EMU_MB", "MESH_EMU_TASKS", "MESH_EMU_BATCH", "GUEST_OPT", "LOG_TEX_NEW", "DUMP_KEY", "VEC_INDEX_FIX", "PASS_TIMES", "PROFILE_MAIN", "PROFILE_RENDER", "PROMPTS", "PUSH_SKIP", "PUSH_STAGE", "RDOC", "RDOC_DIR", "RDOC_DLL", "REALTIME", "REC_END", "REEXEC", @@ -340,7 +340,9 @@ static const CfgMeta metas[] = { {"FOLIAGE", "full", "full|reduced|off", RESTART}, // vk_exec_draw.c foliage_skip {"VRAM_HEAP_LIMIT", "0", NULL, RESTART}, // vk_device.c vram_heap_limits: auto, 0 or MiB {"VRAM_RESERVE_MB", "0", "0..4096", RESTART}, // vk_vma_budget.c: video memory kept free of VMA's allocations - {"UPLOAD_BATCH_MB", "0", NULL, RESTART}, // vk_memory.c: MiB of staging before a batch goes early, 0 no limit + // vk_memory.c vk_upload_end: MiB of staging before a batch goes early. 0: at 64 MiB, the staging pool's own limit + // (STG_FLUSH), which also holds for every value above it + {"UPLOAD_BATCH_MB", "0", NULL, RESTART}, {"MESH_EMULATION", "0", "0|1", RESTART}, {"MESH_EMU_MB", "256", "32..2048", RESTART}, {"MESH_EMU_TASKS", "16384", "256..262144", RESTART}, {"MESH_EMU_BATCH", "1", "0|1", RESTART}, {"VEC_INDEX_FIX", "0", "0|auto|1", RESTART}, {"GUEST_OPT", "1", "1|2", RESTART}, // tools/make_game.py: the next Rebuild the Game diff --git a/runtime/host.h b/runtime/host.h index 51b71972..275dffc2 100644 --- a/runtime/host.h +++ b/runtime/host.h @@ -359,11 +359,15 @@ uint64_t res_heap_extra_kib(void); // added to the HeapD int res_heapdef_build(const void *in, size_t n, void **out, size_t *out_n); // the HeapDef table (.byml.zs) with // the grown heaps, malloc'd; 0 = cannot (logged) void res_heapdef_served(void); // romfs.c serves the grown table +int res_heapdef_ok(void); // 1 once it does; 0 after fs_init: the game has the shipped heaps void wide_init(void); void wide_present(void); // each present: DLSS's render scale (res_fixed) // after rt_patches_init: widescreen.c, NX2NAVI_GAME_ASPECT int wide_available(void); // the generated C has the widescreen sets void wide_picture(unsigned *w, unsigned *h); // the picture the game draws (its scene is never larger) uint64_t wide_heap_extra_kib(void); // the texture heap's growth for the widescreen picture +// The most res_heap_extra_kib may be (memory.c): with more the application pool passes the heap region and the +// game's own SetHeapSize is refused. A whole number of 2 MiB. +uint64_t mem_heap_extra_max_kib(void); void res_init(void); // after rt_patches_init: the patch page and the sets float res_dlss_lod_bias(void); // NX2NAVI_DLSS upscaling: texture LOD bias (<= 0) int res_dlss_mode(const char *mode); // NX2NAVI_DLSS changed while running: new scene size @@ -456,6 +460,9 @@ int vk_pipeline_warm_shaders(void); // vk_pipeline.c: --w int vk_pipeline_prepare_pipelines(void); // vk_pipeline.c: --prepare-pipelines (the shared pipeline list) int vk_pipeline_probe_prewarm(void); // vk_pipeline.c: --probe-prewarm int vk_pipeline_probe_buildgate(void); // vk_pipeline.c: --probe-buildgate +// main.c crash_handler: a fault while the boot builds pipelines or stage libraries leaves a note, which tells the +// next start that the marker or journal it finds is a crash's and not a stopped start's (vk_pipeline.c pm_skipped). +void vk_pipeline_crash_note(void); int vk_ffx_probe_cache(void); // vk_ffx.cpp: --probe-ffx-cache // vk_device.c: the drivers' own shader caches beside the exe (MESA_SHADER_CACHE_DIR, on NVIDIA's Linux driver // __GL_SHADER_DISK_CACHE_PATH and _SIZE) in the environment, once; before any thread can load a Vulkan driver diff --git a/runtime/main.c b/runtime/main.c index 18901c62..c635c4f6 100644 --- a/runtime/main.c +++ b/runtime/main.c @@ -297,6 +297,7 @@ static void crash_handler(const PlatCrash *x) { mode_rc); plat_exit(mode_rc); } + vk_pipeline_crash_note(); // first: the next start tells a crash in the boot's builds from a stopped start by it rt_log_sync(); // the lines before the crash, then the report, each written as it is made char d[160]; uintptr_t rip = x->rip; diff --git a/runtime/memory.c b/runtime/memory.c index 6ce3cf29..6940c49c 100644 --- a/runtime/memory.c +++ b/runtime/memory.c @@ -1306,6 +1306,12 @@ static int inside(uint64_t a, uint64_t n, RtRange r) { return a >= r.base && n < static uint32_t map_gen; uint32_t mem_map_gen(void) { return __atomic_load_n(&map_gen, __ATOMIC_ACQUIRE); } +// The application pool is the title's own plus res_heap_extra_bytes (mem_init), and the game asks for a heap of +// nearly all of it: past HEAP_CAP the request below is refused, so this is the most that may be added. +uint64_t mem_heap_extra_max_kib(void) { + return HEAP_CAP > DEFAULT_POOL ? ((HEAP_CAP - DEFAULT_POOL) & ~0x1fffffull) >> 10 : 0; +} + // ---- 0x01 SetHeapSize uint32_t mem_set_heap_size(uint64_t size, uint64_t *addr) { uint64_t t0 = perf_mark_begin(); // NX2NAVI_TRACE=perf: the commit or decommit, the wait for the lock included diff --git a/runtime/nvn_api_core.c b/runtime/nvn_api_core.c index caad7a04..74f17a4a 100644 --- a/runtime/nvn_api_core.c +++ b/runtime/nvn_api_core.c @@ -236,17 +236,20 @@ NVN_API(nvnDeviceRegisterFastClearDepth) { ret_x(c, 0); } NVN_API(nvnDeviceGetImageHandle) { ret_x(c, (uint64_t)(int64_t)(int)arg_x(c, 1)); } // NVNtextureHandle nvnDeviceGetTextureHandle(const NVNdevice *d, int texture_id, int sampler_id): // texture id (20 bits) | sampler id << 20, the encoding the replay decodes (NVN_HANDLE_TEX / NVN_HANDLE_SAMP) -// An id out of range (-1: the game had no texture to give, as when its render-target heap is full) gets the handle of -// the last texture slot, which no game registers (its pool has 32,768), so a draw through it samples the stand-in as -// for any slot without a texture, and sampler 0. Ending the game here turned one missing texture into a crash. +// An id out of range (-1: the game had no texture to give, as when its render-target heap is full) gets texture +// slot 0 and sampler slot 0: the first 256 slots of both pools are reserved (nvnDeviceGetInteger 0x11 and 0x12 above) +// and no game registers into them, so slot 0 holds the stand-in. That holds for a draw that binds the handle, where +// handle 0 is "no texture" (vk_exec_draw.c), and for a shader that takes the handle out of a uniform buffer and +// indexes the pool arrays with it: slot 0 is inside their VK_POOL_TEX entries, which a slot past them is not. +// Ending the game here turned one missing texture into a crash. NVN_API(nvnDeviceGetTextureHandle) { int tid = (int)arg_x(c, 1), sid = (int)arg_x(c, 2); if (tid < 0 || tid >= (1 << 20) || sid < 0 || sid >= (1 << 12)) { static int said; if (said < 8 && ++said) - rt_log("nvn: texture handle asked for with ids out of range (texture %d, sampler %d): the handle of an " - "empty slot is given, which draws the stand-in", tid, sid); - if (tid < 0 || tid >= (1 << 20)) tid = (1 << 20) - 1; + rt_log("nvn: texture handle asked for with ids out of range (texture %d, sampler %d): the handle of the " + "reserved slot 0 is given, which draws the stand-in", tid, sid); + if (tid < 0 || tid >= (1 << 20)) tid = 0; if (sid < 0 || sid >= (1 << 12)) sid = 0; } ret_x(c, (uint64_t)tid | (uint64_t)sid << 20); diff --git a/runtime/resolution.c b/runtime/resolution.c index cb7541fd..3d5350b7 100644 --- a/runtime/resolution.c +++ b/runtime/resolution.c @@ -284,6 +284,7 @@ int res_heapdef_build(const void *in, size_t n, void **out, size_t *out_n) { } void res_heapdef_served(void) { R.heapdef_ok = 1; } +int res_heapdef_ok(void) { return R.heapdef_ok; } // ---- sun shadow map size: each frame the engine sizes its shadow cascades to the area they cover, down to about 400 near // the ground, which draws foliage shadows blocky, and each change of size moves the shadow texels, so leaf shadows jump. diff --git a/runtime/tex_probe.c b/runtime/tex_probe.c index 2f2d3110..11b6a88b 100644 --- a/runtime/tex_probe.c +++ b/runtime/tex_probe.c @@ -483,6 +483,76 @@ static int test_remade(uint8_t *guest) { return 0; } +// A pool in the device address table gets its VRAM mirror while a guest submission is recorded, as on a draw's first +// use of it as a buffer (vk_memory.c mirror_create). The table takes the mirror up by a store the GPU runs after the +// mirror's first copy (vk_gpa.c vk_gpa_flip_cmd), so nothing is changed when the mirror is made; read from the table's +// own memory once the submission is done, the pool's page reads the mirror and writes guest memory as well, and the +// mirror holds the pool's bytes. +static void mirror_use_record(VkCommandBuffer cb, void *arg) { + vk_texture_new_submit(); + vk_mem_mirror_begin(); + vk_mem_use(arg); + vk_mem_mirror_end(cb, vk_sync_issued() + 1); +} + +static int test_table_mirror(uint8_t *guest) { + VkCtx *c = vk_ctx(); + if (vk_mem_shadow_storage() || !vk_gpa_wanted()) { + printf("probe-vkmem: a new mirror in the device address table: skipped (%s)\n", + vk_mem_shadow_storage() ? "shadow storage, no mirrors" : "no device address table"); + return 0; + } + pool_t pool = {.magic = NVN_POOL_MAGIC, .flags = 0x24, .mem = guest + 0xD00000, .size = 0x10000, + .game_mem = guest + 0xD00000}; + volatile uint32_t *w = (volatile uint32_t *)(pool.mem + 0x3000); + uint64_t a = (uintptr_t)w, f0 = 0, t0 = 0, f1 = 0, t1 = 0, fm = 0, tm = 0; + *w = 0x600DF00Du; + GCHECK(vk_gpa_on() && vk_pool_init(&pool), "a CPU-visible pool in the device address table"); + VkBuffer sb; + VmaAllocation sa; + volatile uint32_t *s = vk_staging(0x1000, &sb, &sa); + VkMemSpan sp; + int bad = 1; +#define TM_CHECK(cond, ...) do { if (!(cond)) { printf("probe-vkmem: FAIL table mirror: "); printf(__VA_ARGS__); printf("\n"); goto done; } } while (0) + TM_CHECK(vk_mem_lookup(a, 4, &sp) && sp.buffer && sp.pool, "the pool has no buffer"); + VkBuffer imported = sp.buffer; + TM_CHECK(vk_gpa_probe(a, &f0, &t0) && f0 == t0 && f0 == vk_gpa_device_address(imported) + sp.offset, + "before a mirror the table gives %#llx / %#llx for the pool's page, expected its imported buffer twice", + (unsigned long long)f0, (unsigned long long)t0); + uint64_t fence = vk_sync_submit_own(mirror_use_record, &sp); + // read while the submission may still be in flight: the table as the CPU left it when the mirror was made + int mid = vk_gpa_probe(a, &fm, &tm); + TM_CHECK(vk_sync_fence_wait(fence, 5000000000ull), "fence timed out"); + if (sp.buffer == imported) { + printf("probe-vkmem: a new mirror in the device address table: skipped (the pool got no mirror)\n"); + bad = 0; + goto done; + } + TM_CHECK(vk_gpa_probe(a, &f1, &t1), "the pool's page left the table"); + if (vk_mem_gpa_late()) // else the table changes when the mirror is made, as up to 0.9.2 + TM_CHECK(mid && (fm == f0 || fm == f1) && tm == f0, "while the submission ran the table gave %#llx / %#llx: neither " + "the state before the store (%#llx twice) nor after it", (unsigned long long)fm, (unsigned long long)tm, + (unsigned long long)f0); + TM_CHECK(f1 == vk_gpa_device_address(sp.buffer) + sp.offset, "after the submission the page reads %#llx, expected the " + "mirror's %#llx", (unsigned long long)f1, (unsigned long long)(vk_gpa_device_address(sp.buffer) + sp.offset)); + TM_CHECK(t1 == f0, "after the submission the page's twin is %#llx, expected the imported buffer's %#llx", + (unsigned long long)t1, (unsigned long long)f0); + copy_wait(sp.buffer, sp.offset, sb, 0, 4); + vmaInvalidateAllocation(c->vma, sa, 0, VK_WHOLE_SIZE); + TM_CHECK(s[0] == 0x600DF00Du, "the mirror holds %#010x, expected the pool's 0x600df00d", s[0]); + g_gpu_checks += 4; + printf("probe-vkmem: a pool that gets its mirror in a recorded submission: the device address table names the mirror " + "after it, with guest memory as the twin (%s)\n", + vk_mem_gpa_late() ? "by a store in queue order, after the mirror's first copy" : "changed at once, NX2NAVI_GPA_LATE=0"); + bad = 0; +done: + vk_sync_finish(); + vmaDestroyBuffer(c->vma, sb, sa); + vk_pool_fini(&pool); +#undef TM_CHECK + return bad; +} + // test_table_sync, test_table_store: a guest submission of one part that records nothing. bound: its record binds the // device address table, as the replay of a draw whose shader reads through the table does (vk_exec_draw.c ssbo_info), // so its prologue brings every shadow of the table up to date (vk_memory.c vk_mem_shadow_sync). @@ -1366,7 +1436,7 @@ static int test_gpu(void) { vk_mem_stats(&st); GCHECK(!st.imported_pools && !st.physical_pools && !st.virtual_pools, "pools left after fini"); g_gpu_checks += 2; - return test_table_sync(h) || test_table_store(h) || test_remade(h); + return test_table_sync(h) || test_table_store(h) || test_remade(h) || test_table_mirror(h); } int probe_vkmem(void) { diff --git a/runtime/ui_text.c b/runtime/ui_text.c index 29aa15e8..047759b1 100644 --- a/runtime/ui_text.c +++ b/runtime/ui_text.c @@ -130,8 +130,9 @@ static const char *const text_en[UI_TEXT_COUNT] = { "driver does not run out. Off: no limit.", [UI_MENU_LOWEND_UPLOAD_BATCH] = "Upload batch limit", [UI_MENU_LOWEND_UPLOAD_BATCH_DESC] = "Texture uploads waiting in staging memory are sent to the GPU early once they " - "hold this much, so an area load cannot run video memory out.", - [UI_MENU_LOWEND_NO_LIMIT] = "No limit", + "hold this much, so an area load cannot run video memory out. Every batch is " + "sent at 64 MB at the latest, so none of these choices changes anything now.", + [UI_MENU_LOWEND_NO_LIMIT] = "64 MB", [UI_MENU_LOWEND_ASTC_KEEP] = "Keep the ASTC decode buffer", [UI_MENU_LOWEND_ASTC_KEEP_DESC] = "The buffer that GPU-decodes compressed textures is kept between upload batches " "instead of being allocated again for each one.", diff --git a/runtime/vk.h b/runtime/vk.h index 9321e31e..d65ba89f 100644 --- a/runtime/vk.h +++ b/runtime/vk.h @@ -192,7 +192,8 @@ void vk_pass_marks_format(const char *source, uint32_t started, uint32_t finishe void vk_pass_marks_report(void); // vk_device.c device_fault_report: the AMD words and the NVIDIA checkpoints, formatted // Draw marks (vk_crumbs.c, NX2NAVI_DRAW_CRUMBS=1 with VK_AMD_buffer_marker; off by default): a mark where the shaders // or the pipeline state change inside pass m, before the draw with index draw in that pass, so a lost device's report -// names the shaders of the draws the GPU had not finished. Replay thread only; nothing while vk_draw_marks_on is 0. +// names the shaders of the draws the GPU had not finished. From any thread that records draws (the replay and its +// workers): the mark takes its number atomically. Nothing while vk_draw_marks_on is 0. extern int vk_draw_marks_on; void vk_draw_mark(VkCommandBuffer cb, const VkPassMark *m, const GfxShader *vs, const GfxShader *fs, uint32_t draw); @@ -381,6 +382,7 @@ void vk_mvprev_frame(void); // each present, beside vk_dlaa_frame void vk_mem_mirror_begin(void); int vk_mem_mirror_end(VkCommandBuffer cb, uint64_t fence); int vk_mem_mirror_pending(void); +int vk_mem_gpa_late(void); // --probe-vkmem: NX2NAVI_GPA_LATE, the table takes a new mirror up in queue order void vk_mem_mirror_prologue(VkCommandBuffer cb); // Staged mirror uploads (vk_memory.c): the GPU of the submission being recorded reads guest memory itself (noted by // vk_mem_note_live); vk_mem_take_live returns that and starts the next submission's note. diff --git a/runtime/vk_crumbs.c b/runtime/vk_crumbs.c index 6cd57fbd..0ecbc715 100644 --- a/runtime/vk_crumbs.c +++ b/runtime/vk_crumbs.c @@ -149,16 +149,21 @@ static void pass_marks_init(VkCtx *c) { // which every draw had finished: the draw that hung was recorded after it, with its shaders or those of one of the few // marks up to the one in word 4. It costs two marker writes per change, a few thousand a frame, so it is for the // player who can reproduce a loss, not for everyone. +// The marks are numbered as they are recorded, by the replay thread and by its workers when a pass's chunks are +// recorded side by side (vk_exec.c): the number is taken atomically and each number has its own slot of the ring, so +// no two threads write one. The two marks the GPU names are exact either way. The numbers between them are in queue +// order only where one thread recorded the pass; with workers some of them belong to chunks that ran earlier, which +// the report says. #define VK_DRAW_MARKS 8192 // power of two typedef struct { uint32_t id, pass, draw; const GfxShader *vs, *fs; } DrawMark; static DrawMark dring[VK_DRAW_MARKS]; -static uint32_t dseq; // draw marks made; only the replay thread writes it +static volatile uint32_t dseq; // draw marks made; any recording thread takes the next number int vk_draw_marks_on; void vk_draw_mark(VkCommandBuffer cb, const VkPassMark *m, const GfxShader *vs, const GfxShader *fs, uint32_t draw) { if (!vk_draw_marks_on || !cb) return; - uint32_t id = ++dseq; - if (!id) id = ++dseq; // 0 means no mark + uint32_t id = __atomic_add_fetch(&dseq, 1, __ATOMIC_RELAXED); + if (!id) id = __atomic_add_fetch(&dseq, 1, __ATOMIC_RELAXED); // 0 means no mark dring[id & (VK_DRAW_MARKS - 1)] = (DrawMark){id, m ? m->id : 0, draw, vs, fs}; rec_pass_mark(cb, id, 2); } @@ -793,7 +798,9 @@ static void draw_marks_report(void) { uint32_t reached = words[4], finished = words[5]; char t[256]; rt_log("vk: draw marks (NX2NAVI_DRAW_CRUMBS, %u made): the GPU last reached draw mark %u, and every draw before draw " - "mark %u had finished; the draws it had not finished follow these marks:", (unsigned)dseq, (unsigned)reached, + "mark %u had finished; the draws it had not finished follow these marks. The marks are numbered as they were " + "recorded: with replay workers on, the numbers between the two are not all later in the queue, and only the " + "two marks named here are certain:", (unsigned)__atomic_load_n(&dseq, __ATOMIC_RELAXED), (unsigned)reached, (unsigned)finished); uint32_t n = reached >= finished ? reached - finished + 1 : 1; if (n > DRAW_MARK_LINES) n = DRAW_MARK_LINES; @@ -803,6 +810,10 @@ static void draw_marks_report(void) { } if (reached >= finished && reached - finished + 1 > n) rt_log("vk: and %u more up to draw mark %u", (unsigned)(reached - finished + 1 - n), (unsigned)reached); + if (reached < finished || reached - finished + 1 > n) { // the mark the GPU last reached, when the list left it out + describe_draw_mark(reached, t, sizeof t); + rt_log("vk: draw mark %u, the last the GPU reached: %s", (unsigned)reached, t); + } } void vk_pass_marks_report(void) { diff --git a/runtime/vk_gpa.c b/runtime/vk_gpa.c index 9daf0a14..8439b812 100644 --- a/runtime/vk_gpa.c +++ b/runtime/vk_gpa.c @@ -19,8 +19,11 @@ // writes (vk_exec_draw.c ssbo_info), so with the mirror alone the CPU never saw them, and a sync of a page the CPU wrote // copied the stale guest page over them. The game's compute passes write tree and foliage records this way, and a // record overwritten so drew a tree stretched into a column reaching the sky (Hyrule Field, Kakariko, Death Mountain). -// A record a page entry names is never written: a new mapping takes a new record, and a record goes back to the free -// list only once the GPU finished every submission issued while a page could still name it (vk_completion_add). Page +// A record a page entry names is never written by the CPU: a new mapping takes a new record, and a record goes back +// to the free list only once the GPU finished every submission issued while a page could still name it +// (vk_completion_add). The one write to a record in use is the GPU's own, in queue order, when a pool's new mirror +// takes its record over (vk_gpa_flip_cmd): what was queued before it reads the old addresses, what comes after it +// the new ones. Page // and L1 entries are single 32-bit stores, so a shader that reads the table while it changes sees the old mapping or // the new one, never half of each. The CPU keeps a copy (shadow) for its reads: the GPU copy is often device memory // behind the PCIe BAR, slow to read back. @@ -110,7 +113,8 @@ static int create(void) { G.free_recs = malloc(RECS * sizeof *G.free_recs); if (!G.cpu || !G.free_recs) return 0; VkBufferCreateInfo bi = {.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, .size = (VkDeviceSize)WORDS * 4, - .usage = VK_BUFFER_USAGE_STORAGE_BUFFER_BIT, .sharingMode = VK_SHARING_MODE_EXCLUSIVE}; + .usage = VK_BUFFER_USAGE_STORAGE_BUFFER_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT, // vk_gpa_flip_cmd + .sharingMode = VK_SHARING_MODE_EXCLUSIVE}; VmaAllocationCreateInfo ai = {.flags = VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT | VMA_ALLOCATION_CREATE_MAPPED_BIT, .usage = VMA_MEMORY_USAGE_AUTO_PREFER_DEVICE}; VmaAllocationInfo info; @@ -352,23 +356,39 @@ static uint32_t remap(uint64_t lo, uint64_t hi, uint32_t old_rec, VkBuffer buf, // the twin: the buffer the pages named before (the pool's imported buffer), so writes reach guest memory too. uint64_t twin = old_rec && twins_on() && keep_twin ? rec_first(old_rec) : dev - buf_lo; uint32_t r = dev ? rec_alloc(dev - buf_lo, twin) : 0; - if (!r) { plat_mutex_unlock(&gpa_lock); return old_rec; } + if (!r) { // no record left: the pages keep the old one, and the caller its number + static int said; + plat_mutex_unlock(&gpa_lock); + if (!said++) + rt_log("vk_gpa: WARNING: no record left for a buffer that takes over %llu pages (%u records in all): they " + "keep naming the buffer they named", (unsigned long long)(p1 > p0 ? p1 - p0 : 0), RECS - 1); + return old_rec; + } pages_move(p0, p1, old_rec, r); plat_mutex_unlock(&gpa_lock); if (old_rec) vk_completion_add(vk_sync_issued() + 1, rec_free_cb, (void *)(uintptr_t)old_rec); return r; } -uint32_t vk_gpa_remap_writes(uint64_t lo, uint64_t hi, uint32_t old_rec, VkBuffer buf, uint64_t buf_lo) { - if (!old_rec || !buf || !vk_gpa_on() || !twins_on()) return 0; +// Records into cb the store that hands record rec over to buf, a pool's new VRAM mirror: from that point of the queue +// on the pages that name rec read buf, and their twin is the buffer they read until then (keep_twin, with twins on) or +// buf again. A change of the table made on the CPU is seen at once by every shader still running, the ones queued +// before the mirror's first copy included, which would then read or write the mirror before anything had filled it; +// this store runs where the command stands, after that copy (vk_memory.c vk_mem_mirror_prologue). The caller puts +// barriers around it. No new record is taken, so there is none to run out of. The shadow takes the new words now: +// the CPU never reads a record's addresses back for a record handed over this way. 0 when nothing was recorded. +int vk_gpa_flip_cmd(VkCommandBuffer cb, uint32_t rec, VkBuffer buf, uint64_t buf_lo, int keep_twin) { + if (!rec || !buf || !vk_gpa_on()) return 0; uint64_t dev = vk_gpa_device_address(buf); - uint64_t p0 = (lo + 4095) >> PAGE_SHIFT, p1 = hi >> PAGE_SHIFT; + if (!dev) return 0; + uint32_t at = REC_WORD + rec * REC_WORDS, w[REC_WORDS]; plat_mutex_lock(&gpa_lock); - uint32_t r = dev ? rec_alloc(rec_first(old_rec), dev - buf_lo) : 0; - if (r) pages_move(p0, p1, old_rec, r); + uint64_t first = dev - buf_lo, twin = keep_twin && twins_on() ? rec_first(rec) : first; + w[0] = (uint32_t)first, w[1] = (uint32_t)(first >> 32), w[2] = (uint32_t)twin, w[3] = (uint32_t)(twin >> 32); + memcpy(G.cpu + at, w, sizeof w); plat_mutex_unlock(&gpa_lock); - if (r) vk_completion_add(vk_sync_issued() + 1, rec_free_cb, (void *)(uintptr_t)old_rec); - return r; + vkCmdUpdateBuffer(cb, G.buf, (VkDeviceSize)at * 4, sizeof w, w); + return 1; } uint32_t vk_gpa_remap(uint64_t lo, uint64_t hi, uint32_t old_rec, VkBuffer buf, uint64_t buf_lo) { @@ -379,6 +399,21 @@ uint32_t vk_gpa_remap_plain(uint64_t lo, uint64_t hi, uint32_t old_rec, VkBuffer return remap(lo, hi, old_rec, buf, buf_lo, 0); } +// --probe-vkmem: what the table's own memory, as the GPU has it, resolves addr to: the device address a shader reads +// there and the one it writes as well. 0 when no page maps addr. Slow (the table may be device memory), and only +// meaningful with no submission in flight. +int vk_gpa_probe(uint64_t addr, uint64_t *first, uint64_t *twin) { + if (G.state <= 0 || addr >> 40) return 0; + vmaInvalidateAllocation(vk_ctx()->vma, G.alloc, 0, VK_WHOLE_SIZE); + uint32_t blk = G.gpu[L1_WORD + (addr >> 24)]; + uint32_t rec = blk ? G.gpu[L2_WORD + blk * L2_N + ((addr >> PAGE_SHIFT) & (L2_N - 1))] : 0; + if (!rec) return 0; + const volatile uint32_t *r = G.gpu + REC_WORD + rec * REC_WORDS; + *first = ((uint64_t)r[0] | (uint64_t)r[1] << 32) + addr; + *twin = ((uint64_t)r[2] | (uint64_t)r[3] << 32) + addr; + return 1; +} + void vk_gpa_report(void) { if (G.state <= 0) return; plat_mutex_lock(&gpa_lock); diff --git a/runtime/vk_gpa.h b/runtime/vk_gpa.h index 29df805e..18d778e5 100644 --- a/runtime/vk_gpa.h +++ b/runtime/vk_gpa.h @@ -24,9 +24,12 @@ uint32_t vk_gpa_remap(uint64_t lo, uint64_t hi, uint32_t old_rec, VkBuffer buf, // The same without the twin: the pages name buf alone (a pool whose VRAM mirror is evicted goes back to its imported // buffer). uint32_t vk_gpa_remap_plain(uint64_t lo, uint64_t hi, uint32_t old_rec, VkBuffer buf, uint64_t buf_lo); -// The pages of [lo, hi) that name old_rec keep reading old_rec's buffer and get buf as their twin: shaders read the -// pool where they did and write buf too (a VRAM mirror until its first copy has run, vk_memory.c mirror_create). -// Returns the new record, or 0 when there is none to give or twins are off (old_rec then stays as it is). -uint32_t vk_gpa_remap_writes(uint64_t lo, uint64_t hi, uint32_t old_rec, VkBuffer buf, uint64_t buf_lo); +// Records into cb the store that hands record rec over to buf (a pool's new VRAM mirror) in queue order: what is +// queued after it reads buf through rec's pages, with the buffer they read before as the twin (keep_twin) or buf +// again; what was queued before it sees no change. 0 when nothing was recorded (no record, no table). +int vk_gpa_flip_cmd(VkCommandBuffer cb, uint32_t rec, VkBuffer buf, uint64_t buf_lo, int keep_twin); +// --probe-vkmem: the device addresses the table, read from its own memory, gives a shader for addr: the one it +// reads and the one it writes as well. 0 when no page maps addr. +int vk_gpa_probe(uint64_t addr, uint64_t *first, uint64_t *twin); void vk_gpa_report(void); void vk_gpa_shutdown(void); diff --git a/runtime/vk_memory.c b/runtime/vk_memory.c index 998ef77c..2465fbea 100644 --- a/runtime/vk_memory.c +++ b/runtime/vk_memory.c @@ -61,8 +61,8 @@ typedef struct PoolRec { // VRAM mirror (POOL_IMPORT, "VRAM mirror" below). mstate and mbuf change on the replay thread only. uint8_t mstate; // MS_NONE until the first buffer use, then MS_ON or MS_REFUSED uint32_t gpa_rec; // POOL_IMPORT: its record in the device address table (vk_gpa.c), 0 none - uint64_t gpa_at; // pool_lock: the submission after which the table reads the mirror (gpa_switch); - // 0 when it does already, or there is no mirror + uint8_t gpa_flip; // as pend: the next prologue records the store that hands its table record to + // the mirror (gpa_flip_cmds); 0 when the table names the mirror, or none exists struct GpaMap { uint64_t lo, hi; uint32_t rec; } *vmaps; // POOL_VIRT: its mapped ranges in the table int nvmaps, cvmaps; VkBuffer mbuf; // MS_ON: device-local copy of [import_lo, import_hi), same offsets as buf @@ -2758,50 +2758,58 @@ static int twin_wanted(const PoolRec *r) { return !twin_max || (int64_t)(r->import_hi - r->import_lo) <= twin_max; } -// ---- when the device address table starts reading a new mirror (NX2NAVI_GPA_LATE, default 1; 0 = at once, as before) -// A mirror is made while the replay records a submission, and its first copy of the pool is in that submission's -// prologue. The table is memory the shaders read as they run, so with its pages moved to the mirror at once, every -// part already handed to the queue (the submissions before, and the earlier parts of this one) read the mirror through -// the table before that copy had run: memory nothing had written yet, and on a card over its budget a block the -// driver had not made resident for those parts. A GTX 1650 4 GB lost its device that way at area loads in every -// session: six reports of a GPU read fault at byte 0 of a mirror made in the faulting submission, each the first byte -// of a new 2 MiB-aligned block, with mirrors being made over the budget; with the table kept empty the same routes ran -// without a loss. -// Now the pages keep reading the imported buffer and name the mirror as their twin (vk_gpa_remap_writes): what shaders -// write through the table goes to both, so the mirror misses nothing, and what they read is guest memory, as before -// the mirror existed. Once the GPU has finished the submission whose prologue holds the copy, the pages move to the -// mirror with the imported buffer as the twin (gpa_switch, at the start of a later submission). In between, a shader -// that reads through the table what another wrote through a buffer binding sees guest memory, which gets such a write -// at the write-back; data written through the table itself, as the game's foliage, instance and light records are, -// is read where it was written. -static int n_gpa_wait; // pool_lock: mirrors whose pages still read the imported buffer (PoolRec.gpa_at) -static uint64_t n_gpa_late; // replay: pages moved to their mirror after its first copy, so far +// ---- when the device address table takes up a new mirror (NX2NAVI_GPA_LATE, default 1; 0 = at once, as before) +// A mirror is made while the replay records a part of a submission, and its first copy of the pool is in that part's +// prologue. The table is memory the shaders read as they run, so a change of it made here, at record time, reaches +// every part already handed to the queue (the submissions before, and the earlier parts of this one), which run +// before that copy: they read the mirror through the table, memory nothing had written yet, and wrote into it, and on +// a card over its budget the block was one the driver had not made resident for them. A GTX 1650 4 GB lost its device +// that way at area loads in every session: six reports of a GPU read fault at byte 0 of a mirror made in the faulting +// submission, each the first byte of a new 2 MiB-aligned block, with mirrors being made over the budget; with the +// table kept empty the same routes ran without a loss. +// So the table is not touched here. The prologue that holds the mirror's first copy also holds, right after it, a +// store into the pool's record (vk_gpa_flip_cmd): the GPU runs it in its place in the queue, so the parts before it +// read and write guest memory through the table as if no mirror existed, and the parts from this one on read the +// mirror and write both. What the earlier parts wrote through the table is in guest memory when the first copy reads +// it there (a twinned pool's first copy is always the GPU's own copy from the imported buffer, mirror_capture), so +// the mirror misses none of it, and no shader reads through the table a state older than the one its place in the +// queue gives it. +static int n_gpa_flip; // as pend: mirrors whose record the next prologue hands over (PoolRec.gpa_flip) +static uint64_t n_gpa_late; // replay: records handed to their mirror in queue order, so far + +static int gpa_late(void); +int vk_mem_gpa_late(void) { return gpa_late(); } // --probe-vkmem static int gpa_late(void) { static int on = -1; if (on < 0) { on = rt_knob("GPA_LATE", "1") == 1; - rt_log("mem: the device address table reads a new VRAM mirror %s (NX2NAVI_GPA_LATE)", - on ? "once the mirror's first copy has run" : "at once, as before"); + rt_log("mem: the device address table takes up a new VRAM mirror %s (NX2NAVI_GPA_LATE)", + on ? "in queue order, after the mirror's first copy" : "at once, as before"); } return on; } -// Replay thread, at the start of a submission: the mirrors whose first copy the GPU has finished take the table's reads. -static void gpa_switch(void) { - if (!__atomic_load_n(&n_gpa_wait, __ATOMIC_RELAXED)) return; - uint64_t done = vk_sync_fence_done(); +static void mbarrier(VkCommandBuffer cb, VkPipelineStageFlags2 ss, VkAccessFlags2 sa, VkPipelineStageFlags2 ds, + VkAccessFlags2 da); + +// Prologue, after the copies into the mirrors: the stores that hand the new mirrors' records over. Everything queued +// before is done with the table before the stores run, and everything after sees them. +static void gpa_flip_cmds(VkCommandBuffer cb) { + if (!n_gpa_flip) return; + mbarrier(cb, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT, + VK_PIPELINE_STAGE_2_ALL_TRANSFER_BIT, VK_ACCESS_2_TRANSFER_WRITE_BIT); plat_mutex_lock(&pool_lock); - for (int i = 0; i < nm && n_gpa_wait; i++) { + for (int i = 0; i < nm && n_gpa_flip; i++) { PoolRec *r = mlist[i]; - if (!r->gpa_at || done < r->gpa_at) continue; - r->gpa_at = 0, n_gpa_wait--; - if (!r->gpa_rec || !r->mbuf) continue; - r->gpa_rec = r->twin ? vk_gpa_remap(r->import_lo, r->import_hi, r->gpa_rec, r->mbuf, r->import_lo) - : vk_gpa_remap_plain(r->import_lo, r->import_hi, r->gpa_rec, r->mbuf, r->import_lo); - n_gpa_late++; + if (!r->gpa_flip) continue; + r->gpa_flip = 0, n_gpa_flip--; + if (r->gpa_rec && r->mbuf && vk_gpa_flip_cmd(cb, r->gpa_rec, r->mbuf, r->import_lo, r->twin)) n_gpa_late++; } + n_gpa_flip = 0; plat_mutex_unlock(&pool_lock); + mbarrier(cb, VK_PIPELINE_STAGE_2_ALL_TRANSFER_BIT, VK_ACCESS_2_TRANSFER_WRITE_BIT, + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT); } // Replay thread: the pool's mirror on its first buffer use. @@ -2897,25 +2905,19 @@ static void mirror_create(PoolRec *r) { r->mstate = MS_ON; r->asked_room = 0; // The device address table names the mirror: the GPU's writes go there, and the sync copies the CPU's. With a - // twin they also go to the imported buffer; without one (twin_wanted) the record names the mirror twice. Reads - // move to the mirror once its first copy has run (gpa_late, below). + // twin they also go to the imported buffer; without one (twin_wanted) the record names the mirror twice. The + // table takes the mirror up in queue order, after its first copy (gpa_late, above). r->twin = (uint8_t)twin_wanted(r); if (!r->twin) { plat_inc64(&m_untwinned_n); plat_add64(&m_untwinned_bytes, (int64_t)n); } - if (r->gpa_rec) { + if (r->gpa_rec && gpa_late()) { + r->gpa_flip = 1, n_gpa_flip++; // vk_mem_mirror_prologue, with the copy range_add queues below + } else if (r->gpa_rec) { plat_mutex_lock(&pool_lock); - uint32_t late = gpa_late() ? vk_gpa_remap_writes(r->import_lo, r->import_hi, r->gpa_rec, r->mbuf, r->import_lo) : 0; - if (late) { - uint64_t f = vk_exec_recording_fence(); - r->gpa_rec = late; - r->gpa_at = f ? f : vk_sync_issued() + 1; // the submission being recorded: its prologue holds the copy - n_gpa_wait++; - } else { - r->gpa_rec = r->twin ? vk_gpa_remap(r->import_lo, r->import_hi, r->gpa_rec, r->mbuf, r->import_lo) - : vk_gpa_remap_plain(r->import_lo, r->import_hi, r->gpa_rec, r->mbuf, r->import_lo); - } + r->gpa_rec = r->twin ? vk_gpa_remap(r->import_lo, r->import_hi, r->gpa_rec, r->mbuf, r->import_lo) + : vk_gpa_remap_plain(r->import_lo, r->import_hi, r->gpa_rec, r->mbuf, r->import_lo); plat_mutex_unlock(&pool_lock); } range_add(&pend, &npend, &cpend, r, 0, n); @@ -2926,7 +2928,7 @@ static void mirror_create(PoolRec *r) { static void mirror_forget(PoolRec *r) { __atomic_add_fetch(&mirror_gen, 1, __ATOMIC_RELEASE); - if (r->gpa_at) r->gpa_at = 0, n_gpa_wait--; // the caller takes the table's pages off the mirror + if (r->gpa_flip) r->gpa_flip = 0, n_gpa_flip--; // gone before its first copy: its record stays as it is for (int i = 0; i < nm; i++) if (mlist[i] == r) { memmove(mlist + i, mlist + i + 1, (size_t)(nm - i - 1) * sizeof *mlist); @@ -3993,7 +3995,7 @@ static void hot_check(void) { } void vk_mem_mirror_begin(void) { - if (vk_mirror_on) gpa_switch(), hot_check(); + if (vk_mirror_on) hot_check(); if (!vk_mirror_on) { // import mode: the texture checks' snapshots still skip frozen regions if (!freeze_idle) return; plat_mutex_lock(&gen_lock); @@ -4039,7 +4041,7 @@ void vk_mem_mirror_begin(void) { slow_mem_call("mirror begin (its queries and freezes)", 0, ww_bytes - b0, q, qpc() - t0); } -int vk_mem_mirror_pending(void) { return npend || snl_copied < nsnl; } +int vk_mem_mirror_pending(void) { return npend || snl_copied < nsnl || n_gpa_flip; } // NX2NAVI_MIRROR_CAPTURE_KB: the pages the CPU wrote since a VRAM mirror's last sync are copied out of guest memory by // the CPU when the part that needs them is handed to the queue, up to this many KiB a part, into a host buffer the @@ -4163,7 +4165,7 @@ static void mirror_capture(VkCommandBuffer cb) { } void vk_mem_mirror_prologue(VkCommandBuffer cb) { - if (!npend && snl_copied == nsnl) return; + if (!npend && snl_copied == nsnl && !n_gpa_flip) return; mbarrier(cb, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_MEMORY_WRITE_BIT, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_READ_BIT | VK_ACCESS_2_TRANSFER_WRITE_BIT); mirror_capture(cb); @@ -4176,6 +4178,7 @@ void vk_mem_mirror_prologue(VkCommandBuffer cb) { } mbarrier(cb, VK_PIPELINE_STAGE_2_COPY_BIT, VK_ACCESS_2_TRANSFER_WRITE_BIT, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT); + gpa_flip_cmds(cb); // after the new mirrors' first copies npend = 0; mrep.prologues++; } @@ -4448,10 +4451,9 @@ static void mirror_log(uint64_t frame) { "%.2f, %.3f MiB, from %.1f flushes of %.3f MiB; flushed and not copied (write-watched) %.3f MiB%s", (unsigned long long)frame, a.page_bytes / f / M, a.flush_n / f, a.flush_bytes / f / M, a.flush_calls / f, a.flush_sum / f / M, a.flush_skipped / f / M, capl); - if (n_gpa_late || __atomic_load_n(&n_gpa_wait, __ATOMIC_RELAXED)) - rt_log("mem: mirror frame %llu: the device address table took up %llu mirrors after their first copy so far, %d " - "wait for theirs (NX2NAVI_GPA_LATE)", (unsigned long long)frame, (unsigned long long)n_gpa_late, - __atomic_load_n(&n_gpa_wait, __ATOMIC_RELAXED)); + if (n_gpa_late) + rt_log("mem: mirror frame %llu: the device address table took up %llu mirrors in queue order, after their first " + "copy, so far (NX2NAVI_GPA_LATE)", (unsigned long long)frame, (unsigned long long)n_gpa_late); } // After vram_log: every heap's usage as the driver reports it (VK_EXT_memory_budget) beside VMA's share of it, VMA by @@ -5095,7 +5097,8 @@ VkCommandBuffer vk_upload_begin(void) { // NX2NAVI_UPLOAD_BATCH_MB (256 with low_end_gpu = 1; 0, the default, = no limit of its own): once the open batch's // staging buffers hold that much, the outermost vk_upload_end submits it. It does so too once the batch's staging // reaches STG_FLUSH, so that an area load's staging stays inside the staging pool's blocks (the staging pool's -// comment). A save load made thousands of textures between two replay submissions, and every one of them went into +// comment): 0 therefore means STG_FLUSH (64 MiB), not "never", and a value above STG_FLUSH changes nothing, the +// low-end mode's 256 and the settings menu's 128 to 1024 included. A save load made thousands of textures between two replay submissions, and every one of them went into // one batch: about 1.5 GB of raw ASTC blocks plus the decoder's work in a single submission, // which AMD's 23.10 driver on an RX 580 refused with VK_ERROR_OUT_OF_DEVICE_MEMORY. Submitted early, a batch still // precedes the replay submission that uses its textures (the queue keeps order), and its staging is freed as it retires. @@ -5166,19 +5169,27 @@ void vk_mem_oom_report(const char *what, VkResult r) { // which it waits for in order (a wait covers all the queue's work submitted ahead of that batch, tens of ms on a heavy // frame, with cmd_lock held). Only when none of that makes room (the open batch's own staging, staging buffers their // callers hold, the blocks' fragmentation) does the pool grow by a block, and the log says so. -// A discrete GPU keeps STG_MIN_BLOCKS blocks; an integrated one, whose host memory is the machine's RAM and where low -// memory ended the game before (vk_mem_give_back), keeps STG_MIN_BLOCKS_UMA, and its loads wait for batches sooner. +// A discrete GPU keeps STG_MIN_BLOCKS blocks, 256 MiB of host memory for the whole run (128 MiB up to 0.9.2); an +// integrated one, whose host memory is the machine's RAM and where low memory ended the game before +// (vk_mem_give_back), keeps STG_MIN_BLOCKS_UMA, 128 MiB, and its loads wait for batches sooner. #define STG_BLOCK (64ull << 20) #define STG_MIN_BLOCKS 4 #define STG_MIN_BLOCKS_UMA 2 static VmaPool stg_pool; static int stg_pool_tried; static uint64_t stg_room_cache, stg_room_wait, stg_room_new; // cmd_lock: how stg_create found room (rt_log below) +static uint64_t stg_wait_ticks, stg_wait_max; // cmd_lock: qpc ticks stg_create waited for batches // vmaCreateBuffer of a staging buffer (*ai names stg_pool) inside the pool's blocks: when they are full, the kept // staging buffers and the arena's idle chunks go, then the oldest batch in flight is waited for and retired, until the // buffer fits. chunk: the buffer is an arena chunk, and VK_INCOMPLETE says that a retired batch gave chunks back to // arena_free, to take one of those instead. Without a batch to wait for (or once the device is lost) the pool grows. +// The wait runs with cmd_lock held, and cannot leave it: the caller is inside vk_upload_begin .. vk_upload_end with +// the open batch's command buffer in hand, and the staging buffer it asks for belongs to that batch, so letting go of +// the lock would let another thread submit the batch under it. Meanwhile a replay submission that wants to send its +// uploads (vk_upload_flush) waits for the lock; its own commands would queue behind the same batches, so what it +// loses is recording time, not GPU time. How long that is on a load is not measured anywhere yet: the waits are +// timed here and logged (the first four, then at every doubling of their count), so a played session's log says. static VkResult stg_create(VkCtx *c, const VkBufferCreateInfo *bi, const VmaAllocationCreateInfo *ai, VkBuffer *b, VmaAllocation *a, VmaAllocationInfo *info, int chunk) { VmaAllocationCreateInfo in = *ai; @@ -5196,13 +5207,23 @@ static VkResult stg_create(VkCtx *c, const VkBufferCreateInfo *bi, const VmaAllo } if (ninflight && !vk_device_is_lost()) { int before = ninflight; + uint64_t w0 = qpc(); VkResult w = vkWaitForFences(c->device, 1, &inflight[0].fence, VK_TRUE, UINT64_MAX); if (w != VK_SUCCESS) { vk_device_lost_check(w, "the wait for an upload batch"); rt_fatal("vk: the wait for an upload failed (%d)", (int)w); } + uint64_t took = qpc() - w0; + stg_wait_ticks += took; + if (took > stg_wait_max) stg_wait_max = took; retire(c, 0); stg_room_wait++; + if (stg_room_wait <= 4 || !(stg_room_wait & (stg_room_wait - 1))) + rt_log("vk: a staging buffer of %.1f MiB waited %.1f ms for an upload batch to finish, with the upload " + "lock held (%llu such waits so far, %.0f ms in all, the longest %.1f ms; %d batches still in " + "flight). A replay submission that sends its uploads waits for that lock", + bi->size / 1048576.0, qpc_ms(took), (unsigned long long)stg_room_wait, qpc_ms(stg_wait_ticks), + qpc_ms(stg_wait_max), ninflight); if (chunk && narena_free) { r = VK_INCOMPLETE; break; @@ -5562,7 +5583,7 @@ void vk_mem_shutdown(void) { plat_mutex_lock(&pool_lock); for (int i = 0; i < pool_index.n; i++) destroy_rec(c, pool_index.e[i].buf); pool_index.n = 0; - nm = npend = nwbl = nsnl = snl_copied = nswl = 0; // mirrors went with their pools + nm = npend = nwbl = nsnl = snl_copied = nswl = n_gpa_flip = 0; // mirrors went with their pools swl_marks = -1; plat_mutex_lock(&snap_pool_lock); for (int i = 0; i < nsnap_pool; i++) vmaDestroyBuffer(c->vma, snap_pool[i].buf, snap_pool[i].alloc); diff --git a/runtime/vk_pipeline.c b/runtime/vk_pipeline.c index 97583b1c..ee4f51dd 100644 --- a/runtime/vk_pipeline.c +++ b/runtime/vk_pipeline.c @@ -5526,6 +5526,7 @@ static struct { int on; // this start waits for all of it int lw; // lb_start left the library warm's start to bf_wait volatile int32_t game; // the wait is over: the warm ended, or the boot let the game start beside it + double t_end; // now_ms() at which the whole warm's time on the boot screen is up; 0 no limit char owed[PLAT_PATH_MAX * 2 + 48]; // boot-warm-owed.txt } BF; @@ -5849,9 +5850,11 @@ static void prewarm_run(void) { // (bf_decide) waits for all of them, with its count on the boot screen, as a start without the library does. int all = (full && !strcmp(full, "1")) || (BF.on && queued); if (P.gpl && all && BF.on && !(full && !strcmp(full, "1"))) - rt_log("vk_pipeline: prewarm: the start waits for all %u recorded pipelines and not %d s (the whole warm after a " - "driver change)", queued, (int)(PREWARM_PIPE_MS / 1000)); + rt_log("vk_pipeline: prewarm: the start waits for the %u recorded pipelines as long as the whole warm after a " + "driver change may take, and not %d s", queued, (int)(PREWARM_PIPE_MS / 1000)); K.until = P.gpl && !all ? now_ms() + PREWARM_PIPE_MS : 0; // before the builders start + // the whole warm's own limit: what it leaves goes to the pool's workers, as after PREWARM_PIPE_MS + if (P.gpl && BF.on && BF.t_end && !(full && !strcmp(full, "1"))) K.until = BF.t_end; PlatThread *t[64]; int nt = 0; if (queued) warm_builder((void *)1); // the first alone (warm_builder) @@ -5905,29 +5908,81 @@ static void prewarm_run(void) { // each one is built at its first draw instead, as with NX2NAVI_PIPE_PREBUILD=0. Another driver version, or deleting // the file, brings the prebuild back. Only the boot's own share is covered: a crash during play, where the pool also // builds what the boot left, says nothing about these builds. An orderly end of the process during the boot (the -// window closed, the guest's ExitProcess, a --seconds run) deletes the marker as well: only a process that died leaves it. +// window closed, the guest's ExitProcess, a --seconds run) deletes the marker as well. +// A marker left over is not yet a driver crash: a start stopped from outside (the task manager, a program that ends +// the game, the power) leaves one too, and since the whole warm (bf_decide) the boot builds for minutes. So the crash +// handler says which it was (vk_pipeline_crash_note: /boot-build-crash.txt, written while the boot's builds +// run): with that file the next start skips the prebuild as before; without it the end is taken for a stopped start +// and only counted, in the marker itself. BOOT_QUIET_ENDS such ends in a row skip the prebuild after all: a driver +// that takes the process down without a fault our handler sees would otherwise end every start (boot_warm.h +// boot_end_decide). static int cache_dir(char *dir, size_t n); static void gpu_key(char *out, size_t n); -static char PM_run[PLAT_PATH_MAX * 2 + 48]; +static char PM_run[PLAT_PATH_MAX * 2 + 48], PM_crash[PLAT_PATH_MAX * 2 + 48]; static volatile int32_t pm_on; +static volatile int32_t jr_boot_on; // the boot's journal of library builds is open (journal_open) +static int pm_quiet; // starts in a row before this one that ended in the boot's builds without a crash report + +static void crash_path_init(const char *dir) { + if (!PM_crash[0]) snprintf(PM_crash, sizeof PM_crash, "%s/boot-build-crash.txt", dir); +} + +// main.c crash_handler, before its report: a fault while the boot's pipeline builds or its journalled library builds +// run. The file is all the next start needs; nothing here allocates. +void vk_pipeline_crash_note(void) { + static const char text[] = "the process faulted while the boot built pipelines or stage libraries " + "(runtime/vk_pipeline.c vk_pipeline_crash_note)\n"; + if (!PM_crash[0] || (!__atomic_load_n(&pm_on, __ATOMIC_ACQUIRE) && !__atomic_load_n(&jr_boot_on, __ATOMIC_ACQUIRE))) + return; + plat_file_write(PM_crash, text, sizeof text - 1); +} + +// 1 when the last start left the crash handler's note; asked once a start, and the note goes. +static int crash_noted(void) { + static int was = -1; + if (was < 0) { + was = PM_crash[0] && plat_file_exists(PM_crash); + if (was) plat_file_delete(PM_crash); + } + return was; +} static int pm_skipped(void) { char dir[PLAT_PATH_MAX * 2], sp[sizeof PM_run], key[32]; if (!cache_dir(dir, sizeof dir)) return 0; snprintf(PM_run, sizeof PM_run, "%s/pipeline-prewarm-running.txt", dir); snprintf(sp, sizeof sp, "%s/pipeline-prewarm-skip.txt", dir); + crash_path_init(dir); gpu_key(key, sizeof key); uint64_t n = 0; - uint8_t *left = plat_file_read(PM_run, &n); + char *left = (char *)plat_file_read(PM_run, &n); if (left) { - char text[256]; - int len = snprintf(text, sizeof text, - "# The boot's pipeline prebuild did not finish on this GPU and driver (runtime/vk_pipeline.c " - "pm_skipped): starts skip it\n# while this line matches. Delete this file to build ahead of play " - "again.\n%s\n", key); - if (plat_file_write_atomic(sp, text, (size_t)len)) plat_file_delete(PM_run); - rt_log("vk_pipeline: the last start died while the boot built pipelines (%s was left); %s now skips that " - "prebuild on this GPU and driver (%s)", PM_run, sp, key); + char head[64]; + snprintf(head, sizeof head, "%.*s", (int)(n < sizeof head - 1 ? n : sizeof head - 1), left); + int crashed = crash_noted(), quiet = 0, what = boot_end_decide(head, key, crashed, &quiet); + if (what == BOOT_END_OTHER) { // another GPU or driver's builds + plat_file_delete(PM_run); + } else if (what == BOOT_END_SKIP) { + char text[256]; + int len = snprintf(text, sizeof text, + "# The boot's pipeline prebuild did not finish on this GPU and driver (runtime/vk_pipeline.c " + "pm_skipped): starts skip it\n# while this line matches. Delete this file to build ahead of play " + "again.\n%s\n", key); + if (plat_file_write_atomic(sp, text, (size_t)len)) plat_file_delete(PM_run); + if (crashed) + rt_log("vk_pipeline: the last start crashed while the boot built pipelines (%s was left, with the crash " + "handler's note); %s now skips that prebuild on this GPU and driver (%s)", PM_run, sp, key); + else + rt_log("vk_pipeline: %d starts in a row ended while the boot built pipelines, none with a crash report " + "(%s was left each time); %s now skips that prebuild on this GPU and driver (%s)", quiet, PM_run, + sp, key); + } else { + pm_quiet = quiet; + plat_file_delete(PM_run); + rt_log("vk_pipeline: the last start ended while the boot built pipelines and reported no crash: taken for a " + "start that was closed or stopped, and the pipelines are built again (%d in a row; at %d the prebuild " + "is skipped on this GPU and driver)", pm_quiet, BOOT_QUIET_ENDS); + } free(left); } uint8_t *skip = plat_file_read(sp, &n); @@ -5948,10 +6003,11 @@ static int pm_skipped(void) { static void pm_set(void) { if (!PM_run[0]) return; - char key[32]; + char key[32], text[64]; gpu_key(key, sizeof key); + int len = snprintf(text, sizeof text, "%s %d\n", key, pm_quiet); // straight to the disk: the file has to be there if the driver takes the process down a moment later - if (plat_file_write_atomic(PM_run, key, strlen(key))) plat_xchg32(&pm_on, 1); + if (plat_file_write_atomic(PM_run, text, (size_t)len)) plat_xchg32(&pm_on, 1); } void vk_pipeline_prewarm_marker_clear(void) { @@ -5979,10 +6035,16 @@ void vk_pipeline_prewarm_marker_clear(void) { // into it (cache_load), some starts later. After BOOT_WARM_TRIES starts that did not end the warm the file stays as // well, and starts on this GPU and driver are ordinary ones again. While the boot waits, the library builds write // the preparation's journal (journal_mark), so a variant the driver's compiler dies on is skipped from the next -// start on (skip_init) instead of ending every start. -// NX2NAVI_PREWARM_FULL=0 turns it off. NX2NAVI_PREWARM_WAIT_MS and a minute without progress end the wait as for any -// prewarm (boot_wait); what is left then runs as in play (vk_pipeline_prewarm_detach). Not for a --shot image, which -// no build timing may change and which nobody watches. +// start on (skip_init) instead of ending every start; a start that was only closed or stopped then blames no variant +// (the crash handler's note decides, as at pm_skipped). +// The wait has a limit, NX2NAVI_PREWARM_FULL_S seconds from here (default 600, 0 none): the 425 s above are a fast +// machine's, and a slow one would sit on the screen for half an hour. At the limit the game starts and what is left +// goes on as on any other start: the recorded pipelines on the pool's workers, the compute pipelines and the library +// warm on their idle threads (vk_pipeline_prewarm_detach). The file then says "done" as well: the boot screen had its +// turn, and the rest is the background's, as it always was. NX2NAVI_PREWARM_WAIT_MS and a minute without progress +// end the wait the same way, as for any prewarm (boot_wait). +// NX2NAVI_PREWARM_FULL=0 turns the whole warm off. Not for a --shot image, which no build timing may change and +// which nobody watches. static void bf_decide(void) { const char *full = cfg_get("NX2NAVI_PREWARM_FULL"); char dir[PLAT_PATH_MAX * 2], key[32], line[100]; @@ -6008,11 +6070,14 @@ static void bf_decide(void) { int len = snprintf(line, sizeof line, "%s %d\n", key, tries); if (!plat_file_write_atomic(BF.owed, line, (size_t)len)) rt_log("vk_pipeline: cannot write %s", BF.owed); BF.on = 1; + const char *se = cfg_get("NX2NAVI_PREWARM_FULL_S"); + double secs = se && *se ? atof(se) : 600; + BF.t_end = secs > 0 ? now_ms() + secs * 1000 : 0; rt_log("vk_pipeline: the whole warm on the boot screen (%s, start %d of %d at most): the start waits for the " - "recorded pipelines, the compute pipelines and the driver's cache of stage libraries. " - "NX2NAVI_PREWARM_FULL=0 leaves them to the background", + "recorded pipelines, the compute pipelines and the driver's cache of stage libraries, for %.0f s at most " + "(NX2NAVI_PREWARM_FULL_S; 0 = no limit). NX2NAVI_PREWARM_FULL=0 leaves them to the background", cache_foreign ? "the pipeline cache was written by another device or driver" : "an earlier start began it", - tries, BOOT_WARM_TRIES); + tries, BOOT_WARM_TRIES, secs > 0 ? secs : 0.0); } free(f); } @@ -6316,54 +6381,76 @@ static int hex_digit(int c) { return c >= '0' && c <= '9' ? c - '0' : c >= 'a' && c <= 'f' ? c - 'a' + 10 : c >= 'A' && c <= 'F' ? c - 'A' + 10 : -1; } -// A journal a crashed run left into the skip list, then the skip list's variants for this GPU and driver into SK. +// The variants a leftover journal j (jn bytes, of file jp) names join the skip list's text *s (*sn bytes; NULL +// without a list) for this GPU and driver, and the list's file sp is written again. 1 when the journal may go: +// nothing in it, or the list holds it now. +static int journal_fold(const uint8_t *j, uint64_t jn, const char *jp, const char *sp, const char *key, char **s, + uint64_t *sn) { + static const char head[] = + "# Shader variants the graphics driver was building when the process died during preparation\n" + "# (runtime/vk_pipeline.c skip_init): input id, stage, GPU vendor:device:driver version. Preparation skips\n" + "# them on that GPU and driver. Delete this file to build them again.\n"; + size_t cap = (*s ? *sn : sizeof head) + (jn / JR_SLOT + 1) * 128, len = 0; + char *out = malloc(cap); + int added = 0; + if (!out) return 1; + len = *s ? *sn : sizeof head - 1; + memcpy(out, *s ? *s : head, len); + if (len && out[len - 1] != '\n') out[len++] = '\n'; + for (uint64_t o = 0; o + 33 <= jn; o += JR_SLOT) { + int set = 0; + for (int b = 0; b < 32; b++) set |= j[o + b]; + if (!set) continue; + char hex[65]; + for (int b = 0; b < 32; b++) snprintf(hex + 2 * b, 3, "%02x", j[o + b]); + len += (size_t)snprintf(out + len, cap - len, "%s %s %s\n", hex, stage_name(j[o + 32]), key); + added++; + } + out[len] = 0; // parsed by skip_init as the list's text + if (added && plat_file_write_atomic(sp, out, len)) { + rt_log("vk_pipeline: the last shader preparation, or the last start's warm on the boot screen, died while " + "the driver built %d variants; %s now lists them and they are skipped ahead of play on this GPU and " + "driver (%s)", added, sp, key); + free(*s), *s = out, *sn = len; + return 1; + } + if (added) rt_log("vk_pipeline: cannot write %s; %s stays for the next run", sp, jp); + free(out); + return !added; +} + +// A journal a run that died left into the skip list, then the skip list's variants for this GPU and driver into SK. +// The preparation's journal (shader-prepare-journal.bin) always counts. The boot's (shader-warm-journal.bin, +// journal_open) counts only with the crash handler's note (crash_noted): without it the start was closed or stopped +// while it waited, and the variants it happened to be building are no one's fault. static void skip_init(const char *dir) { if (SK.loaded) return; SK.loaded = 1; - char jp[PLAT_PATH_MAX * 2 + 48], sp[PLAT_PATH_MAX * 2 + 48], key[32]; + char jp[PLAT_PATH_MAX * 2 + 48], bj[PLAT_PATH_MAX * 2 + 48], sp[PLAT_PATH_MAX * 2 + 48], key[32]; snprintf(jp, sizeof jp, "%s/shader-prepare-journal.bin", dir); + snprintf(bj, sizeof bj, "%s/shader-warm-journal.bin", dir); snprintf(sp, sizeof sp, "%s/shader-prepare-skip.txt", dir); + crash_path_init(dir); gpu_key(key, sizeof key); - uint64_t jn = 0, sn = 0; - uint8_t *j = plat_file_read(jp, &jn); + uint64_t jn = 0, bn = 0, sn = 0; + uint8_t *j = plat_file_read(jp, &jn), *b = plat_file_read(bj, &bn); char *s = (char *)plat_file_read(sp, &sn); - int keep_journal = 0; - if (j) { - static const char head[] = - "# Shader variants the graphics driver was building when the process died during preparation\n" - "# (runtime/vk_pipeline.c skip_init): input id, stage, GPU vendor:device:driver version. Preparation skips\n" - "# them on that GPU and driver. Delete this file to build them again.\n"; - size_t cap = (s ? sn : sizeof head) + (jn / JR_SLOT + 1) * 128, len = 0; - char *out = malloc(cap); - int added = 0; - if (out) { - len = s ? sn : sizeof head - 1; - memcpy(out, s ? s : head, len); - if (len && out[len - 1] != '\n') out[len++] = '\n'; - for (uint64_t o = 0; o + 33 <= jn; o += JR_SLOT) { - int set = 0; - for (int b = 0; b < 32; b++) set |= j[o + b]; - if (!set) continue; - char hex[65]; - for (int b = 0; b < 32; b++) snprintf(hex + 2 * b, 3, "%02x", j[o + b]); - len += (size_t)snprintf(out + len, cap - len, "%s %s %s\n", hex, stage_name(j[o + 32]), key); - added++; - } - out[len] = 0; // parsed below as the list's text + if (!j || journal_fold(j, jn, jp, sp, key, &s, &sn)) plat_file_delete(jp); + if (b) { + int marked = 0; + for (uint64_t o = 0; o + 33 <= bn; o += JR_SLOT) + for (int k = 0; k < 32; k++) + if (b[o + k]) { marked++; break; } + if (marked && !crash_noted()) { + rt_log("vk_pipeline: the last start ended during the library warm on the boot screen, with %d variants in " + "the driver's hands, and reported no crash: taken for a start that was closed or stopped; no variant " + "is skipped for it", marked); + plat_file_delete(bj); + } else if (journal_fold(b, bn, bj, sp, key, &s, &sn)) { + plat_file_delete(bj); } - if (added && plat_file_write_atomic(sp, out, len)) { - rt_log("vk_pipeline: the last shader preparation, or the last start's warm on the boot screen, died while " - "the driver built %d variants; %s now lists them and they are skipped ahead of play on this GPU and " - "driver (%s)", added, sp, key); - free(s), s = out, sn = len, out = NULL; - } else if (added) { - rt_log("vk_pipeline: cannot write %s; %s stays for the next run", sp, jp); - keep_journal = 1; - } - free(out); - free(j); } - if (!keep_journal) plat_file_delete(jp); + free(j), free(b); for (char *line = s, *next; line && *line; line = next) { next = strchr(line, '\n'); if (next) *next++ = 0; @@ -6371,10 +6458,10 @@ static void skip_init(const char *dir) { uint8_t iid[32]; int ok = line[0] != '#' && sscanf(line, "%64s %15s %31s", hex, st, k) == 3 && strlen(hex) == 64 && !strcmp(k, key); - for (int b = 0; ok && b < 32; b++) { - int hi = hex_digit(hex[2 * b]), lo = hex_digit(hex[2 * b + 1]); + for (int b2 = 0; ok && b2 < 32; b2++) { + int hi = hex_digit(hex[2 * b2]), lo = hex_digit(hex[2 * b2 + 1]); if (hi < 0 || lo < 0) ok = 0; - else iid[b] = (uint8_t)(hi << 4 | lo); + else iid[b2] = (uint8_t)(hi << 4 | lo); } uint8_t (*grown)[32] = ok ? realloc(SK.iid, (SK.n + 1) * sizeof *SK.iid) : NULL; if (grown) memcpy(grown[SK.n++], iid, 32), SK.iid = grown; @@ -6405,12 +6492,13 @@ static void journal_mark(int slot, const GfxWarmEnt *e) { plat_mutex_unlock(&SK.jlock); } -// The boot's journal, for the library builds the start waits for (bf_wait). skip_init has run (lb_start) and has -// dealt with a leftover one. +// The boot's journal, for the library builds the start waits for (bf_wait), in a file of its own: skip_init treats +// one left over differently from the preparation's. skip_init has run (lb_start) and has dealt with a leftover one. static void journal_open(void) { char dir[PLAT_PATH_MAX * 2]; if (!cache_dir(dir, sizeof dir)) return; - snprintf(SK.jpath, sizeof SK.jpath, "%s/shader-prepare-journal.bin", dir); + crash_path_init(dir); + snprintf(SK.jpath, sizeof SK.jpath, "%s/shader-warm-journal.bin", dir); PlatFile *j = plat_fs_create(SK.jpath, NULL); if (!j) { rt_log("vk_pipeline: cannot create %s: a driver crash in this start's library warm will not be recorded", SK.jpath); @@ -6420,6 +6508,7 @@ static void journal_open(void) { plat_mutex_lock(&SK.jlock); SK.journal = j; plat_mutex_unlock(&SK.jlock); + __atomic_store_n(&jr_boot_on, 1, __ATOMIC_RELEASE); } // Ends the boot's journal: when the library warm is done, when the game starts beside it (a crash in play says @@ -6428,6 +6517,7 @@ static void journal_open(void) { // flight write nothing afterwards (journal_mark). static void journal_close(void) { if (!SK.jpath[0]) return; + __atomic_store_n(&jr_boot_on, 0, __ATOMIC_RELEASE); plat_mutex_lock(&SK.jlock); PlatFile *j = SK.journal; SK.journal = NULL; @@ -7602,50 +7692,62 @@ static void lb_stop(void) { if (BG.th) plat_thread_join(BG.th), BG.th = NULL; // after its current build, if one runs } +// 1 once the whole warm's time on the boot screen is up (BF.t_end): the builders go on as on any other start. +static int bf_time_up(void) { + if (!BF.t_end || !bf_waiting() || now_ms() < BF.t_end) return 0; + vk_pipeline_prewarm_detach(); + return 1; +} + // The whole warm's wait (comment at bf_decide), on the prewarm thread after the recorded pipelines: the compute // pipelines lb_start has just started, then the library warm. The counts go to the boot screen -// (vk_pipeline_prewarm_progress). Should the boot stop waiting (vk_pipeline_prewarm_detach), this thread still sees -// the warm to its end, which the builders then reach as in play. +// (vk_pipeline_prewarm_progress). It ends with the warm, at the warm's time limit, or when the boot stops waiting +// (vk_pipeline_prewarm_detach); in the last two the builders go on beside the game. static void bf_wait(void) { double t0 = now_ms(); + int up = bf_time_up(); // the recorded pipelines took all of it if (LB.n && LB.threads) { __atomic_store_n(&pw_pipe_done, 0, __ATOMIC_RELAXED); __atomic_store_n(&pw_pipe_total, (int64_t)LB.n, __ATOMIC_RELAXED); __atomic_store_n(&pw_phase, 3, __ATOMIC_RELEASE); - while (__atomic_load_n(&LB.running, __ATOMIC_ACQUIRE) > 0 && !LB.stop) { + while (__atomic_load_n(&LB.running, __ATOMIC_ACQUIRE) > 0 && !LB.stop && bf_waiting()) { plat_sleep_ms(100); plat_mutex_lock_shared(&LB.stat); uint64_t d = LB.done; plat_mutex_unlock_shared(&LB.stat); __atomic_store_n(&pw_pipe_done, (int64_t)d, __ATOMIC_RELAXED); + up |= bf_time_up(); } } double t1 = now_ms(); int libs = 0; if (BF.lw && !LB.stop) { if (bf_waiting()) journal_open(); - lw_start(); + lw_start(); // beside the game when the wait is over: the check and the warm as on any other start if (LW.th[0]) { libs = 1; __atomic_store_n(&pw_pipe_done, 0, __ATOMIC_RELAXED); __atomic_store_n(&pw_pipe_total, (int64_t)LW.n, __ATOMIC_RELAXED); __atomic_store_n(&pw_phase, 4, __ATOMIC_RELEASE); - while ((!LW.checked || LW.running > 0) && !LW.stop) { + while ((!LW.checked || LW.running > 0) && !LW.stop && bf_waiting()) { plat_sleep_ms(100); __atomic_store_n(&pw_pipe_done, (int64_t)LW.done, __ATOMIC_RELAXED); - if (!bf_waiting()) journal_close(); // the game runs + up |= bf_time_up(); } } - journal_close(); + journal_close(); // the warm is over, or the game runs } - int stopped = LB.stop || LW.stop, beside = __atomic_load_n(&BF.game, __ATOMIC_ACQUIRE); - rt_log("vk_pipeline: the whole warm %s: %zu compute pipelines in %.0f ms, the driver's cache of stage libraries %s in " - "%.0f ms (%lld of %zu prepared variants built)%s", stopped ? "stopped with the process" : "is done", LB.n, - t1 - t0, !libs ? "not warmed (no prepared list, or the graphics pipeline library or NX2NAVI_LIB_WARM is off)" - : LW.done ? "warmed" : "left as it is (the check found it warm, or gave the warm up)", - now_ms() - t1, (long long)LW.done, LW.n, - beside ? "; the boot had stopped waiting, so the game ran beside the end of it" : ""); - if (!stopped) { // done for this GPU and driver (boot_warm.h) + int stopped = LB.stop || LW.stop; + int over = __atomic_load_n(&LB.running, __ATOMIC_ACQUIRE) <= 0 && (!libs || (LW.checked && LW.running <= 0)); + rt_log("vk_pipeline: the whole warm %s: the compute pipelines (%zu) took %.0f ms of the wait, the driver's cache of " + "stage libraries %.0f ms (%s; %lld of %zu prepared variants built so far)", + stopped ? "stopped with the process" : over ? "is done" + : up ? "goes on beside the game: its time on the boot screen is up (NX2NAVI_PREWARM_FULL_S)" + : "goes on beside the game: the boot stopped waiting", LB.n, t1 - t0, now_ms() - t1, + !libs ? "not warmed: no prepared list, or the graphics pipeline library or NX2NAVI_LIB_WARM is off" + : LW.done ? "being warmed" : "left as it is so far: the check found it warm, gave the warm up, or still runs", + (long long)LW.done, LW.n); + if (!stopped) { // the boot screen's turn is over for this GPU and driver (boot_warm.h) char key[32], line[64]; gpu_key(key, sizeof key); int len = snprintf(line, sizeof line, "%s done\n", key); diff --git a/runtime/widescreen.c b/runtime/widescreen.c index 2c976c33..14b95c7c 100644 --- a/runtime/widescreen.c +++ b/runtime/widescreen.c @@ -90,29 +90,48 @@ static unsigned picture_height(void) { static double dlss_share(void); double wide_dlss_share(void) { return dlss_share(); } // vk_dlaa.c: the mode's share, 0 off -static double heap_share(void); - -static double dlss_share(void) { - if (!vk_dlaa_wanted() || !cfg_bool("NX2NAVI_DLAA") || (!vk_dlaa_fsr3() && !vk_dlaa_available())) return 0; - const char *m = cfg_value("NX2NAVI_DLSS"); - double s = !strcmp(m, "quality") ? 2.0 / 3 : !strcmp(m, "balanced") ? 0.58 : !strcmp(m, "performance") ? 0.5 - : !strcmp(m, "ultra_performance") ? 1.0 / 3 : 1.0; - // Never more than the texture heap was made for at the start (heap_share): only an upscaler switched on in the - // menu of a run that began without one can ask for more, and it gets the whole picture after a restart. - if (s > heap_share()) { - static int said; - if (!said++) - rt_log("widescreen: the scene stays at %.0f%% of the picture until a restart: the game's render-target heap " - "was sized at the start, without an upscaler, and %.0f%% does not fit it", heap_share() * 100, s * 100); - s = heap_share(); - } - return s; -} - // Without DLSS or FSR 3.1 (FSR 1, bilinear) the game's dynamic resolution took the scene down to a third of the picture // under load, and a plain scale shows every step of that. The most the game's controller ever gives (wide_init) is held. #define PLAIN_SHARE 0.75 +static double heap_share(void); + +// Set by wide_init when the grown heap table (wide_heap_extra_kib) did not reach the game: romfs.c could not build +// it (a table in a form it does not rewrite, as a mod's may be) or could not write it to the profile folder. The game +// then has the shipped heap, made for its own picture with the scene at PLAIN_SHARE, and gets no more than that: +// a larger scene or picture in the shipped heap is the crash the growth is there to prevent. +static int heap_unserved; + +// The most the scene can be of the picture in the heap the game has. +static double fit_share(void) { + return heap_unserved ? PLAIN_SHARE : heap_share(); +} + +// The share of the picture the upscaler's mode renders (NX2NAVI_DLSS; NVIDIA's ratios). +static double mode_share(void) { + const char *m = cfg_value("NX2NAVI_DLSS"); + return !strcmp(m, "quality") ? 2.0 / 3 : !strcmp(m, "balanced") ? 0.58 : !strcmp(m, "performance") ? 0.5 + : !strcmp(m, "ultra_performance") ? 1.0 / 3 : 1.0; +} + +static double dlss_share(void) { + if (!vk_dlaa_wanted() || !cfg_bool("NX2NAVI_DLAA") || (!vk_dlaa_fsr3() && !vk_dlaa_available())) return 0; + double s = mode_share(); + // Never more than the texture heap the game has was made for (fit_share). A mode chosen in the menu that renders + // more than the start's did gets its size after a restart; a heap at its limit never holds the whole picture. + if (s > fit_share()) { + static int said; + if (!said++) + rt_log("widescreen: the scene is %.0f%% of the picture, not the mode's %.0f%%: the game's render-target heap " + "%s. A mode chosen in the menu gets its size after a restart, unless the start's log says the heap " + "is at its limit", fit_share() * 100, s * 100, + heap_unserved ? "is the shipped one (the start's log says why the grown one was not served)" + : "was sized at the start for that much"); + s = fit_share(); + } + return s; +} + static double scene_share(void) { static double plain = -1; if (plain < 0) { @@ -120,23 +139,45 @@ static double scene_share(void) { plain = e && *e ? atof(e) : PLAIN_SHARE; } double s = dlss_share(); - return s > 0 ? s : plain; + return s > 0 ? s : plain > fit_share() ? fit_share() : plain; // NX2NAVI_SCENE_SHARE above the heap the game has +} + +static int picture_size(unsigned *ow, unsigned *oh, int quiet); + +// The picture's pixels over the game's own 2560x1440. +static double picture_ratio(void) { + unsigned ow, oh; + if (!wide_available() || !picture_size(&ow, &oh, 1)) ow = GAME_W, oh = GAME_H; + return (double)ow * oh / ((double)GAME_W * GAME_H); } // The largest share of the picture the scene can have in this run, which the texture heap is made for // (wide_heap_extra_kib). Decided once, at the first call, which is the start (memory.c sizes the application pool -// from it before any device exists): the whole picture when the upscaler setting is on, whatever its mode, since -// the menu changes the mode in play and DLAA and FSR at native size render all of it; else the game's own most, -// PLAIN_SHARE, or NX2NAVI_SCENE_SHARE when that asks for more. From the settings, not from what runs: DLSS is known -// to run only once the device is up. +// from it before any device exists): the game's own most, PLAIN_SHARE, or more where the settings at the start ask +// for more: NX2NAVI_SCENE_SHARE, or the upscaler's mode when the upscaler is on (DLAA and FSR at native size render +// the whole picture; the modes below PLAIN_SHARE need nothing). From the settings, not from what runs: DLSS is known +// to run only once the device is up. A mode chosen in the menu later that renders more keeps the start's share until +// a restart (dlss_share). +// Never more than the application pool can take: the heap grows with the share's square (wide_heap_extra_kib), and +// past mem_heap_extra_max_kib the game's own SetHeapSize is refused and nothing starts. A 10240x2880 picture has room +// for 80% there, not for all of it, and the scene is then that much. static double heap_share(void) { static double share = -1; if (share < 0) { const char *e = cfg_get("NX2NAVI_SCENE_SHARE"); double s = e && *e ? atof(e) : PLAIN_SHARE; if (s < PLAIN_SHARE) s = PLAIN_SHARE; + if (rt_patch_set("res_fixed") >= 0 && cfg_bool("NX2NAVI_DLAA") && mode_share() > s) s = mode_share(); if (s > 1) s = 1; - if (rt_patch_set("res_fixed") >= 0 && cfg_bool("NX2NAVI_DLAA")) s = 1; + double room = (double)mem_heap_extra_max_kib() / GAME_DTA_KIB + 1; // the heap's most over the shipped one + double most = sqrt(room / picture_ratio()) * PLAIN_SHARE; + if (s > most) { + if (most < PLAIN_SHARE) most = PLAIN_SHARE; // wide_heap_extra_kib says what a picture this large lacks + rt_log("widescreen: the scene can be %.0f%% of the picture at most, not %.0f%%: the game's render-target " + "heap for more does not fit the application pool (%llu MiB of growth at most)", most * 100, s * 100, + (unsigned long long)(mem_heap_extra_max_kib() >> 10)); + s = most; + } share = s; } return share; @@ -191,15 +232,29 @@ static int picture_size(unsigned *ow, unsigned *oh, int quiet) { // is all of the picture, 2560x1440 where the game never goes past 1920x1080, and a player's game ended in that same // function, reading address 0x48, each time a dialogue closed in Castle Town (0.8.0, 1440 lines, dlss = dlaa). // The heap and its parent grow by the shipped size times the picture's pixels over 2560x1440 and times the scene's -// pixels over the game's most (heap_share: its targets are the picture's size or the scene's), in whole MiB; -// resolution.c rewrites the table (res_heapdef_build) and memory.c adds the amount to the application pool, so no -// other heap shrinks. 0 for the game's picture and scene, and for a larger picture without the widescreen sets. +// pixels over the game's most (heap_share: its targets are the picture's size or the scene's, so this is the most +// they can need), in whole MiB; resolution.c rewrites the table (res_heapdef_build) and memory.c adds the amount to +// the application pool, so no other heap shrinks. 0 for the game's picture and scene, and for a larger picture +// without the widescreen sets. In MiB, with the scene at the game's own most and with all of the picture (DLAA): +// 2560x1440 0 and 238, 3440x1440 105 and 424, 3840x2160 382 and 915, 5120x2160 610 and 1,322, 7680x2160 1,068 and +// 2,135. The heap is committed as the game first writes it (memory.c, NX2NAVI_HEAP_LAZY), so the room a run does not +// use costs address space and no memory. +// At most mem_heap_extra_max_kib: heap_share has taken the scene's share down to what fits, and a picture too large +// even with the game's own scene (above 10240x2880) gets the most there is, with a line that says so. uint64_t wide_heap_extra_kib(void) { - unsigned ow, oh; - if (!wide_available() || !picture_size(&ow, &oh, 1)) ow = GAME_W, oh = GAME_H; double k = heap_share() / PLAIN_SHARE; - double grow = (double)ow * oh / ((double)GAME_W * GAME_H) * k * k - 1; - return grow > 0 ? (uint64_t)ceil(GAME_DTA_KIB * grow / 1024) * 1024 : 0; + double grow = picture_ratio() * k * k - 1; + uint64_t kib = grow > 0 ? (uint64_t)ceil(GAME_DTA_KIB * grow / 1024) * 1024 : 0, most = mem_heap_extra_max_kib(); + if (kib > most) { + static int said; + if (!said++) + rt_log("widescreen: WARNING: a picture of %.1f times the game's own needs %llu MiB more render-target heap " + "and the application pool has room for %llu: the game may end when a screen needs more targets " + "than fit. A smaller game_res or a narrower game_aspect fits", picture_ratio(), + (unsigned long long)(kib >> 10), (unsigned long long)(most >> 10)); + kib = most; + } + return kib; } // The save's picture (recomp/patches-v111.txt, set wide_save): the game draws the whole picture into a target 543 @@ -221,7 +276,19 @@ void wide_init(void) { // (wide_present). double game = (double)GAME_W / GAME_H, f = 0.75; unsigned ow, oh; + // The grown heap table is served from here on, or is not (romfs.c heapdef_overlay says why in the log): before + // the guest runs and before the first present asks for the scene's share. + if (wide_heap_extra_kib()) { + fs_init(); + if (!res_heapdef_ok()) { + heap_unserved = 1; + rt_log("widescreen: the game's render-target heap stays the shipped one (the grown table was not served, see " + "the RomFS and resolution lines above): the scene is %.0f%% of the picture at most and the picture " + "the game's own 2560x1440", PLAIN_SHARE * 100); + } + } if (!picture_size(&ow, &oh, 0)) return; // the game's own sizes: every set stays off + if (heap_unserved) return; // a larger picture's targets do not fit the shipped heap if (so < 0 || ss < 0 || sa < 0 || sh < 0 || sm < 0) { rt_log("widescreen: the generated C has no widescreen patch sets (gen.py --patches recomp/patches-v111.txt); " "the game's own 2560x1440 picture"); @@ -274,7 +341,7 @@ void wide_init(void) { void wide_picture(unsigned *w, unsigned *h) { static unsigned pw, ph; if (!pw) { - if (!wide_available() || !picture_size(&pw, &ph, 1)) pw = GAME_W, ph = GAME_H; + if (!wide_available() || heap_unserved || !picture_size(&pw, &ph, 1)) pw = GAME_W, ph = GAME_H; } *w = pw, *h = ph; } diff --git a/tests/boot_warm_check.c b/tests/boot_warm_check.c index 480c4f03..52c6eb07 100644 --- a/tests/boot_warm_check.c +++ b/tests/boot_warm_check.c @@ -56,6 +56,26 @@ int main(void) { snprintf(text, sizeof text, "%s 2x\n", key); CHECK(boot_warm_decide(text, key, 0, &tries) == BOOT_WARM_NO, "a count with a tail owes a warm"); + // ---- a marker left by a start that ended while the boot built pipelines + int quiet = -1; + // A crash the handler saw: the prebuild is skipped at once, whatever the count. + snprintf(text, sizeof text, "%s 0\n", key); + CHECK(boot_end_decide(text, key, 1, &quiet) == BOOT_END_SKIP, "a reported crash does not skip the prebuild"); + // No note: the start was closed or stopped. It counts, and the builds are made again, until BOOT_QUIET_ENDS in a row. + for (int n = 0; n < BOOT_QUIET_ENDS; n++) { + snprintf(text, sizeof text, "%s %d\n", key, n); + int want = n + 1 >= BOOT_QUIET_ENDS ? BOOT_END_SKIP : BOOT_END_COUNTED; + CHECK(boot_end_decide(text, key, 0, &quiet) == want && quiet == n + 1, "a stopped start after %d before it", n); + } + // A marker of 0.9.2 holds the GPU and driver alone: the first of its row. + CHECK(boot_end_decide(key, key, 0, &quiet) == BOOT_END_COUNTED && quiet == 1, "an old marker skips the prebuild"); + CHECK(boot_end_decide(key, key, 1, &quiet) == BOOT_END_SKIP, "an old marker with a crash note does not skip"); + // Another driver's marker says nothing about this one, crash note or not. + snprintf(text, sizeof text, "%s 2\n", other); + CHECK(boot_end_decide(text, key, 0, &quiet) == BOOT_END_OTHER && quiet == 0, "another driver's marker counted"); + CHECK(boot_end_decide(text, key, 1, &quiet) == BOOT_END_OTHER, "another driver's crash blamed on this one"); + CHECK(boot_end_decide("", key, 0, &quiet) == BOOT_END_OTHER, "an empty marker counted"); + printf("boot_warm_check: %s\n", fails ? "FAILED" : "passed"); return fails ? 1 : 0; } diff --git a/tests/gpa_check.c b/tests/gpa_check.c index 7bda3d39..47a1c24d 100644 --- a/tests/gpa_check.c +++ b/tests/gpa_check.c @@ -49,6 +49,10 @@ VkResult vmaCreateBuffer(VmaAllocator a, const VkBufferCreateInfo *bi, const Vma return VK_SUCCESS; } void vmaDestroyBuffer(VmaAllocator a, VkBuffer buf, VmaAllocation alloc) { (void)a, (void)buf; free(alloc); } +VkResult vmaInvalidateAllocation(VmaAllocator a, VmaAllocation alloc, VkDeviceSize off, VkDeviceSize n) { + (void)a, (void)alloc, (void)off, (void)n; + return VK_SUCCESS; +} // A buffer's device address: its handle far above the GPU addresses the tests map. static uint64_t dev_of(VkBuffer b) { return (uint64_t)(uintptr_t)b << 28; } @@ -73,6 +77,22 @@ static void gpu_done(void) { // the submission being recorded is issued and fi nqueue = k; } +// A store the GPU runs in queue order (vk_gpa_flip_cmd): kept until gpu_runs_stores, as a command buffer keeps it +// until the GPU reaches it. +static struct { VkDeviceSize off, n; uint8_t data[32]; } stores[8]; +static int nstores; +void vkCmdUpdateBuffer(VkCommandBuffer cb, VkBuffer b, VkDeviceSize off, VkDeviceSize n, const void *data) { + (void)cb, (void)b; + if (nstores == 8 || n > sizeof stores[0].data || (off | n) & 3) { printf("FAIL: a store the stand-in cannot hold\n"); exit(1); } + stores[nstores].off = off, stores[nstores].n = n; + memcpy(stores[nstores].data, data, (size_t)n); + nstores++; +} +static void gpu_runs_stores(void) { + for (int i = 0; i < nstores; i++) memcpy((uint8_t *)(uintptr_t)table + stores[i].off, stores[i].data, (size_t)stores[i].n); + nstores = 0; +} + // ---- the table as a shader reads it (the layout at the top of vk_gpa.c) enum { L1_WORD = 16, L1_N = 65536, L2_WORD = L1_WORD + L1_N, L2_N = 4096, BLOCKS = 1024 }; static uint32_t block_of(uint64_t addr) { return table[L1_WORD + (addr >> 24)]; } @@ -125,27 +145,35 @@ int main(void) { CHECK(read_through(a0 + 4096, 1) == dev_of(b0) + 4096, "a plain page is its own twin"); gpu_done(); - // a new mirror takes the writes first and the reads once its first copy has run (vk_memory.c mirror_create, - // gpa_switch): until then the pages read the imported buffer and the mirror is their twin + // a new mirror takes the pool's record over by a store the GPU runs after the mirror's first copy (vk_memory.c + // mirror_create, vk_mem_mirror_prologue): what is queued before the store reads and writes the imported buffer + // alone, what comes after it reads the mirror and writes both VkBuffer m1 = new_buf(); - uint32_t w0 = vk_gpa_remap_writes(a0, a1, r0, m1, a0); - CHECK(w0 && w0 != r0, "a new mirror gets a record for the writes"); - expect(a0, a1, b0, a0, "a new mirror before its first copy: reads stay on the imported buffer"); - CHECK(read_through(a0 + 4096, 1) == dev_of(m1) + 4096 && read_through(a1 - 4096, 1) == dev_of(m1) + (a1 - 4096 - a0), - "a new mirror before its first copy is the twin, which takes the writes"); + CHECK(vk_gpa_flip_cmd(VK_NULL_HANDLE, r0, m1, a0, 1), "a new mirror's store is recorded"); + expect(a0, a1, b0, a0, "before the GPU reaches the store: reads stay on the imported buffer"); + CHECK(read_through(a0 + 4096, 1) == dev_of(b0) + 4096 && read_through(a1 - 4096, 1) == dev_of(b0) + (a1 - 4096 - a0), + "before the GPU reaches the store nothing is written into the mirror"); + gpu_runs_stores(); + expect(a0, a1, m1, a0, "after the store: reads are on the mirror"); + CHECK(read_through(a0 + 4096, 1) == dev_of(b0) + 4096 && read_through(a1 - 4096, 1) == dev_of(b0) + (a1 - 4096 - a0), + "after the store the imported buffer is the twin"); gpu_done(); - r0 = vk_gpa_remap(a0, a1, w0, m1, a0); - expect(a0, a1, m1, a0, "a new mirror after its first copy: reads move to it"); - CHECK(read_through(a0 + 4096, 1) == dev_of(b0) + 4096, "and the imported buffer is the twin again"); - // evicted while it still waits for its first copy: the pages go back to the imported buffer alone + // evicted: the pages go back to the imported buffer alone, on a record of their own + uint32_t was = r0; r0 = vk_gpa_remap_plain(a0, a1, r0, b0, a0); - gpu_done(); - VkBuffer m2 = new_buf(); - w0 = vk_gpa_remap_writes(a0, a1, r0, m2, a0); - r0 = vk_gpa_remap_plain(a0, a1, w0, b0, a0); - expect(a0, a1, b0, a0, "a mirror gone before its first copy"); + CHECK(r0 && r0 != was, "an evicted mirror's pages take a new record"); + expect(a0, a1, b0, a0, "an evicted mirror"); CHECK(read_through(a0 + 4096, 1) == dev_of(b0) + 4096, "leaves the imported buffer as its own twin"); - CHECK(!vk_gpa_remap_writes(a0, a1, 0, m2, a0), "a pool without a record gets none for a mirror's writes"); + gpu_done(); + // a mirror without a twin: the store names it twice + VkBuffer m2 = new_buf(); + CHECK(vk_gpa_flip_cmd(VK_NULL_HANDLE, r0, m2, a0, 0), "an untwinned mirror's store is recorded"); + gpu_runs_stores(); + expect(a0, a1, m2, a0, "an untwinned mirror"); + CHECK(read_through(a0 + 4096, 1) == dev_of(m2) + 4096, "is its own twin"); + r0 = vk_gpa_remap_plain(a0, a1, r0, b0, a0); + expect(a0, a1, b0, a0, "and goes back to the imported buffer"); + CHECK(!vk_gpa_flip_cmd(VK_NULL_HANDLE, 0, m2, a0, 1) && !nstores, "a pool without a record gets no store"); gpu_done(); // an unmapped range reads the zero buffer at once, and its blocks serve no other range while the GPU may still read