diff --git a/docs/architecture.md b/docs/architecture.md index 8243d2a0..8a8c61ae 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -209,7 +209,9 @@ keeps per-core data without locks and relies on that, so the host models it (`ru - The game's tensor-core upscaler is replaced by a decode-and-scale pass (`vk_tonemap.c`), which enlarges the scene with FSR 1's edge-adaptive filter (`NX2NAVI_STANDIN_FILTER`, `bilinear` for the plain one) unless DLSS or FSR 3.1 scales it; `NX2NAVI_NATIVE_TENSOR=1` runs the game's own network programs instead (`vk_exec_draw.c` `native_tensor`, - off by default). `vk_fsr.c` runs AMD FSR 1 (EASU, RCAS) on the present when the window is larger than the picture. + off by default). Without DLSS or FSR the scene is `scene_share` of the picture (`widescreen.c` `plain_share`, the + game's own 3/4 by default); at 1, the Upscaling row's Native, it is the picture's size and the pass only decodes it, + texel for texel. `vk_fsr.c` runs AMD FSR 1 (EASU, RCAS) on the present when the window is larger than the picture. - Anti-aliasing (`vk_aa.c`, `NX2NAVI_AA`): the game anti-aliases with its own SMAA, three compute passes after the bloom composite (edges, blending weights, composition into the scene target as square-rooted RGB9E5 words). `off`, `fxaa` and `smaa` leave out the first two and replace the composition with `shaders/aa_post.comp` or SMAA at its Ultra preset @@ -217,6 +219,10 @@ keeps per-core data without locks and relies on that, so the host models it (`ru `taa` is DLSS or FSR at the scene's size with the game's SMAA. While DLSS or FSR runs (`dlaa`, any mode), the other values still choose the pass: in the native modes it smooths the upscaler's picture after the bloom, in the scaling modes it runs on the scene before the upscaler scales it. + DLSS and FSR 3.1 read the bloom composite's output, which the game has already exposed and tonemapped; + `NX2NAVI_DLSS_PREBLOOM=1` (a test, off by default) runs them at the bloom chain's first pass on the scene's linear + light instead, and draws the lens flares and the bloom composite after them at the picture's size (`vk_dlaa.c`, + "before the bloom"). - HDR (`vk_hdr.c`): the game tonemaps its scene into a float target, upscales it, draws the HUD into a second float target and copies that into the 8-bit sRGB window texture it presents. With `NX2NAVI_HDR=1` the tonemap shader gets the function of `shaders/hdr_tonemap.frag` spliced in where its SDR curve clamps each channel, so the float picture @@ -260,6 +266,21 @@ keeps per-core data without locks and relies on that, so the host models it (`ru the holes of fuller ones, a few per frame and only in frames with time to spare; which blocks is decided from a table of what lies in them (`vk_tex_compact.h`), and the old image of a moved texture stays until no descriptor set and no submission can still name it (`vk_texture.c`, "the pass"). +- Memory report: beside each `mem: holes` line, `mem: by owner` splits the device-local memory the driver counts into + VMA's allocations by the file that made them (`vk_vma_budget.c`, "owners"; the textures further by the texture + cache's census), the free space inside VMA's blocks and what VMA never allocated (pipelines, descriptor memory, the + swapchain, DLSS and frame generation features); `mem: host` gives the host-visible allocations, the process's + commit and working set and the game's heap; `vk: released textures` how often released textures came back and how + old they were, and how many were made again after a drop (`vk_memory.c` `owner_log`). A figure that only climbs over + a session is a leak. +- Released textures by age: the game releases a texture when what uses it leaves the view and takes it back when that + returns; the cache keeps the released image so that the take-back finds it. In play the take-backs came within a + minute of the release or never, so a released texture goes once it has gone unused for a hold of three minutes, six + for content only the GPU made (`vk_texture.c` `drop_aged`, `vk_tex_release.h`, `tex_released_hold`). The size bound + (a quarter of the video memory budget) stays as the safety net for a burst of area changes, and young textures may + pass it, up to twice it, while the card and the system's commit have room. Times are wall seconds from a clock of + the presents, not present counts. `vram_use = high` (`cfg.c` `vram_use_value`) turns the hold off and keeps what a + switched-off upscaler or SMAA holds; the default, low, frees both. - Allocation: Vulkan Memory Allocator. Debugging: Vulkan validation layers, per-pass frame dumps (`NX2NAVI_DUMP`, `vk_dump.c`), a sampling profiler (`NX2NAVI_PROFILE`, `tools/prof_report.py`) and a flight recorder (`NX2NAVI_FLIGHT`). RenderDoc cannot capture, because it hides `VK_EXT_external_memory_host` from the application. diff --git a/docs/building.md b/docs/building.md index aabce40a..fc4a706a 100644 --- a/docs/building.md +++ b/docs/building.md @@ -662,6 +662,11 @@ bytes goes first, what moves fits the free bytes outside the chosen blocks, a pl and 400 random tables keep those rules. The part of `runtime/vk_texture.c` that builds the table and moves the images (`NX2NAVI_TEX_COMPACT`) is not built by it. No GPU, no game files. +A change to which entry of the upscaler stand-in's scratch images serves a new size (`runtime/vk_scratch_lru.h`, +`runtime/vk_exec_draw.c` `fallback_scratch`) passes `python tests\scratch_lru_check.py`, which builds the header alone +and checks that a match is a hit, a ninth size takes a free entry or the one used longest ago instead of being +refused, and two images of one frame never evict each other. No GPU, no game files. + A change to which frames FSR 3.1 frame generation is told are camera cuts (`runtime/vk_fg_reset.h`, `runtime/vk_fg.cpp` `generate` and `vk_fg_frame`) passes `python tests\fg_reset_check.py`, which builds the header alone and checks that each frame id gets its own frame's cut, that an id never noted or no longer held counts as a cut, and that a slot rewritten diff --git a/docs/releases/v0.9.3.md b/docs/releases/v0.9.3.md new file mode 100644 index 00000000..bbb568f3 --- /dev/null +++ b/docs/releases/v0.9.3.md @@ -0,0 +1,29 @@ +## Faster + +- **Experimental: the next frame no longer waits for the end of the last one's graphics work** (`tail_early = 1` in + nx2navi.ini, off by default; contributed by epicmeme546). The game waits each frame until the graphics card has + finished the frame before. The last of that work is the upscaler (FSR 4, FSR 3.1 or DLSS) and the interface drawn + after it, and nothing the game reads comes out of it. With the setting on, the copies of what the card wrote for the + game go before that last stretch and the game goes on once they are there, so the card finishes the picture while + the next frame is already being prepared. The card runs the same work as before. On an RX 7900 XT with a Ryzen 7 + 5800X3D at 3440x1440, FSR 4 Native AA, Hyrule Field: 50.7 instead of 42.5 fps standing and 56.0 instead of 45.9 along + a route; with `replay_workers = 3` as well, 63.5 instead of 49.4 standing. With FSR 3.1, whose pass is shorter, 51.8 + instead of 47.8 standing. Its pictures matched standing, along the route and through warps to five areas, but it is + new: leave it off to play, and report what you see if you try it. It does nothing without one of those upscalers, + and nothing yet on NVIDIA and Intel cards, where the lazy write-back is on. +- **Experimental: the replay workers take more of a frame off the game's render thread** (`replay_pipe` and + `replay_small` in nx2navi.ini, both 0 by default, both with `replay_workers`; contributed by epicmeme546). With + `replay_pipe = N`, 1 to 4, the thread goes on past up to N groups of draws its workers still record. With + `replay_small = N` as well, a group of N draws or more that is too small to share out goes whole to one worker. On + the same PC standing in Kokiri Forest, with `tail_early = 1`: 60.5 fps with 3 workers alone, 62.2 with 5 workers and + `replay_pipe = 4`, 63.2 with `replay_small = 16` added, against 45.4 with all four settings off. The pictures matched + there and through warps to five areas. + +## Changed + +- **The draw replay on several CPU cores is off by default again** (`replay_workers = 0`). With three workers on by + default in 0.9.1 and 0.9.2 some players' games ended with a lost graphics device; the cause is not found yet. The + settings menu has it under Game, Testing, as *CPU helper threads (experimental)*: 0 to 8, after a restart. +- **The other experimental performance settings have rows there too**: *Helpers work ahead* (`replay_pipe`, 0 to 4), + *Helpers take small jobs* (`replay_small`, 0 to 64) and *Start next frame early* (`tail_early`). A row whose setting + does nothing without another one is greyed out and says which it needs. diff --git a/nx2navi.ini.example b/nx2navi.ini.example index 71ac3ded..f0a77ea6 100644 --- a/nx2navi.ini.example +++ b/nx2navi.ini.example @@ -71,11 +71,30 @@ # of latency, and can show a dark or flickering frame now and then: the other thread reads vertex and buffer memory # the game has sometimes rewritten by then. # replay_async = 0 -# Record the large groups of draws of a submission on this many replay worker threads beside the submitting one, 0 to -# 16. Default: 3 (Hyrule Field standing on an RX 7900 XT: 51.7 instead of 43.6 fps). 0: the replay records everything -# on one thread, as before; try it if a picture looks wrong. With 1 a worker records each long run of draws while the -# submitting thread waits for it: no faster, it checks that the pictures stay the same. -# replay_workers = 3 +# EXPERIMENTAL. Record the large groups of draws of a submission on this many replay worker threads beside the +# submitting one: auto or 0 to 16. Default: auto, half the CPU's physical cores less one (2 on 6 cores, 3 on 8, 5 on +# 12, at most 6), none on four cores or fewer and none on AMD's driver for RX 6000 and older Radeon cards, where a +# player's game lost its graphics device with workers on; the log's "replay workers automatic" line says how many. +# 0: the replay records everything on one thread. 3 gave 51.7 instead of 43.6 fps standing in Hyrule Field on an RX +# 7900 XT, and more than 3 gave nothing more there. Also in the menu: Game, Testing, CPU helper threads. +# With 1 a worker records each long run of draws while the submitting thread waits for it: no faster, it checks that +# the pictures stay the same. +# replay_workers = auto +# EXPERIMENTAL. 1 lets the game start on its next frame while the graphics card still runs the end of the last one: the +# upscaler (FSR 4, FSR 3.1 or DLSS) and the interface drawn after it. The graphics card runs the same work, with the +# copies of what it wrote for the game moved before that end, and the game's waits end once those copies and the +# results it reads are there. A frame whose end hands the game anything else runs as before. Default: 0. Standing in +# Hyrule Field on an RX 7900 XT with FSR 4: 50.7 instead of 42.5 fps, and 63.5 instead of 49.4 with replay_workers = 3. +# Does nothing without one of those upscalers, nor with the lazy write-back (the default on NVIDIA and Intel cards). +# Also in the menu: Game, Testing, Start next frame early. +# tail_early = 0 +# EXPERIMENTAL, with replay_workers. replay_pipe = N (1 to 4, default 0): the submitting thread goes on past up to N +# groups its workers still record. replay_small = N (default 0, needs replay_pipe): a group of N draws or more that is +# too small to share out goes whole to one worker. Kokiri Forest standing on an RX 7900 XT with FSR 4, tail_early = 1, +# replay_workers = 5, replay_pipe = 4 and replay_small = 16: 63 instead of 45 fps with all of them off. Also in the +# menu: Game, Testing, Helpers work ahead (replay_pipe) and Helpers take small jobs (replay_small). +# replay_pipe = 0 +# replay_small = 0 # EXPERIMENTAL, THE GAME CRASHES WITH 1. 1 lifts the virtual core gate: guest threads run without the console's limit # of one thread per core, and the game keeps per-core data without locks. With 1 the game crashes or hangs, # usually at start. Keep 0 to play. @@ -333,8 +352,21 @@ # Percent of the video memory in use above which textures give way: first those the game released, then those no # shader sampled for a while (tex_disuse); each is made again when used. The default lies under the 87 percent past # which game memory gets no copy in video memory (memory = mirror), which costs far more frame time than a texture -# uploaded again. 0: textures never give way for video memory. +# uploaded again. 0: textures never give way for video memory. 70 frees more on a card short of memory, and more +# textures are made again (the low of the 0.10 previews set it). # tex_vram_pressure = 80 +# Seconds a texture the game released is kept once nothing uses it (0 to 900); then it goes, however much room is +# left. In play the game took released textures back within a minute or not at all; one taken back after its drop is +# made again and shows its stand-in for a frame. A texture whose content only the GPU made waits twice as long, and one +# the GPU drew into since its last copy back to game memory stays. 0: released textures stay up to tex_released_mb for +# the whole session (as before 0.10). Default: 180; vram_use = high sets 0. Read at start. +# tex_released_hold = 180 +# Megabytes the released textures may hold (0: no bound). Past it the oldest go even when young; while tex_released_hold +# is on, young ones may stay past it, up to twice it, as long as video memory and the system's commit have room. +# Default: a quarter of the video memory budget where the PC has an eighth of its memory to match, at least a +# thirty-second of the PC's memory (512 to 2048), at most 8192. 1024 (the low of the 0.10 previews) holds less, but +# textures the game takes back after their drop are made again and show their stand-in for a frame. Read at start. +# tex_released_mb = 4096 # Texture compaction (experimental). Video memory is taken in blocks (vma_block_mb), and a block goes back to the # driver only once everything in it is freed: after a long session the blocks hold textures scattered between holes, # and the game holds a gigabyte or two of video memory it does not use. 1: textures no shader sampled for ten seconds @@ -521,6 +553,14 @@ # is softer and costs a little less GPU time. DLSS and FSR 3.1 (dlaa = 1) scale the scene themselves, so this changes # nothing there, and it does not touch the movies. Changes while the game runs take effect at once. # standin_filter = easu +# The 3D scene's share of the game resolution without DLSS or FSR 3.1 (dlaa = 0), Ocarina of Time: 0.75, the default, +# is the game's own most (1920x1080 in its 2560x1440 picture), which the scaling above enlarges; 1 draws the scene at +# the picture's full size and nothing enlarges it (the menu's Upscaling row calls it Native (100%)); 0 leaves the size +# to the game's dynamic resolution, down to a third of the picture under load. A larger scene needs more of the +# memory the game keeps for its render targets, which is set aside at the start: 1 takes 238 MiB more than 0.75 at +# 2560x1440, 533 MiB more at 3840x2160 and 712 MiB more at 5120x2160, as dlss = dlaa does. Changes while the game runs +# take effect at once up to the share the game started with, a larger one at the next start. +# scene_share = 0.75 # Picture shape when the window's shape differs from the game resolution's (a 16:9 picture in a 21:9 fullscreen # window): fit, the default, keeps the shape, with black bars at the sides or above and below, so a 16:9 game in a wider # window is no longer stretched as it used to be; stretch fills the window. Changes while the game runs take effect at @@ -540,6 +580,15 @@ # cpu or astc_bc = 0) and the budget is under 20480 MiB, auto goes one step lower, as they take four times the memory. # The log says what was picked ("texture detail:"). Read at start. # texture_detail = auto +# Video memory use (the Screen tab's row): low or high. low, the default, lets go of released textures once nothing has +# used them for tex_released_hold (180 s), and frees what DLSS, frame generation's HUD-less picture, FSR 1 and SMAA +# held once they have been switched off for a while (about 0.9 GiB with DLSS off at 5120x2142). high keeps both for the +# whole session (tex_released_hold = 0: released textures up to tex_released_mb), so switching those features on again +# makes nothing. balanced, the default before 0.10, means low. The low of the 0.10 previews traded memory for work +# done again; its values are keys of their own: tex_released_mb = 1024, tex_vram_pressure = 70 and pipe_seed_boot = 0 +# (0.3-0.6 GiB of video memory and 2-4 GB of commit less, but a first visit to an area misses objects for a frame +# while their pipelines are made). Read at start. +# vram_use = low # Sun shadow map size: game (the game's own, which changes with the view and drops to about 400 near the ground, so # leaf shadows look blocky and jump) or a fixed 256, 512, 768, 1024, 1536 or 2048. Above 960 the texture heap grows with # it. 256 is coarser than the game's own and is for integrated GPUs short of memory bandwidth. Read at start. @@ -605,7 +654,9 @@ # characters, a little more flicker). Any other model by its letter, a to m: on slower RTX cards the older model e is # cheaper. Changes while the game runs take effect at once (DLSS starts a new history). # dlss_preset = default -# DLSS keeps one network per picture size (gameplay, the pause screen). 1: one unused for 3,600 frames is freed. +# DLSS keeps one network per picture size (gameplay, the pause screen). 1: one unused for 3,600 frames is freed, and +# ten seconds after the upscaler or frame generation is switched off in the menu, the networks and pictures they held +# are freed too (switched on again later, they are made again, which the first frame waits for). # 0: none is ever freed (about 0.5 GB more video memory after the first pause). 2: as 1, and the networks that stay # are not made again when another is made (shows striped pause screens; for tests). # dlss_idle_free = 1 @@ -616,6 +667,12 @@ # takes, and counts the changes of size that kept the picture against those that made a network. 0 is the default: # the shared network has been compiled and not yet run on a graphics card. Read at start. # dlss_range = 0 +# Where DLSS or FSR 3.1 works in the frame (a test, off by default). 0: on the finished picture, after the game has +# added its bloom, applied its exposure and tonemapped it. 1: on the scene's own light, before the bloom chain and the +# tonemap, as these upscalers are made for; the bloom, the lens flares and the tonemap then work on the upscaler's +# picture, drawn at the picture's size when the scene is smaller. Frames without the bloom (menus) stay as with 0. +# The log says where it ran (lines starting with "dlaa: prebloom"). Changes apply at the next frame. +# dlss_prebloom = 0 # Per-object motion vectors for DLSS: characters, animals and moving objects get their own motion instead of the # camera's, so DLSS keeps their history instead of smearing or rejecting it. Costs some CPU time each frame (about 1-2 ms # on a laptop CPU); 0 leaves camera-only motion. Read at start. diff --git a/runtime/cfg.c b/runtime/cfg.c index 981b2850..6c57c5e6 100644 --- a/runtime/cfg.c +++ b/runtime/cfg.c @@ -44,7 +44,7 @@ static const char *const known[] = { "BIND_SKIP", "BIND_STATS", "BIND_UP", "BIND_X", "BIND_Y", "BIND_ZL", "BIND_ZR", "BLOCK_VIEWS_3D", "CACHE_DIE", "CLOTH_REALTIME", "COLOR_BRIGHTNESS", "COMPUTE_KEEP", "COLOR_CONTRAST", "COLOR_GAMMA", "COLOR_PRESET", "COLOR_SATURATION", "CONTACT_DEBUG", "CONTACT_SHADOWS", "CORE_DIAG", "CRIT", "CRIT_OUT", - "DMABUF", "DMABUF_MB", "DRAW_CRUMBS", "GPA_LATE", "DESC_TEMPLATE", "DEVICE_PROFILE", "DISCORD_APP_ID", "DISCORD_IMAGE", "DISCORD_RPC", "DLAA", "DLAA_CAM_KEEP", "DLAA_CUT", "DLSS_IDLE_FREE", "DLAA_EDGE_FILL", "DLAA_AT_UPSCALER", "FFX_DEBUG", "FFX_KEYS", "FSR3_SHARPEN", "DLAA_JITTER", "DLAA_MV", "DLAA_MASKDEBUG", "DLAA_MVDEBUG", "DLAA_OBJMV", "DLAA_SURVEY", "DLAA_TARGET_KEEP", "DLAA_TIMES", "DLSS", "DLSS_CHAR_MASK", "DLSS_PRESET", "DLSS_RANGE", "DLSS_REACTIVE", "DUMP", "DUMP_BUF", "DUMP_DIR", + "DMABUF", "DMABUF_MB", "DRAW_CRUMBS", "GPA_LATE", "DESC_TEMPLATE", "DEVICE_PROFILE", "DISCORD_APP_ID", "DISCORD_IMAGE", "DISCORD_RPC", "DLAA", "DLAA_CAM_KEEP", "DLAA_CUT", "DLSS_IDLE_FREE", "DLAA_EDGE_FILL", "DLAA_AT_UPSCALER", "FFX_DEBUG", "FFX_KEYS", "FSR3_SHARPEN", "DLAA_JITTER", "DLAA_MV", "DLAA_MASKDEBUG", "DLAA_MVDEBUG", "DLAA_OBJMV", "DLAA_SURVEY", "DLAA_TARGET_KEEP", "DLAA_TIMES", "DLSS", "DLSS_CHAR_MASK", "DLSS_PRESET", "DLSS_PREBLOOM", "DLSS_RANGE", "DLSS_REACTIVE", "DUMP", "DUMP_BUF", "DUMP_DIR", "DUMP_LOG", "DUMP_RAW", "DUMP_SHADERS", "DUMP_FILE", "DS3", "LOG_CS_BINDS", "LOG_TEX_FLAGS", "LOG_MESH", "MESH", "POOL", "NATIVE_TENSOR", "LOG_RCOND", "LOG_RT_SIZES", "DUMP_WHITE", "DUMP_AFTER", "DUMP_MOVIE", "SUBMIT_FINISH", "ACQUIRE_WAIT", "CRASH_FILTER_GUARD", "CS_FALLBACK", "EXEFS", "FEEDBACK", "FIX_C0", "FIX_ZERO", "FLIGHT", "FLIGHT_AT", "FLIGHT_DIR", "FLIGHT_FRAMES", "GPU_TIMELINE", "FFX_PIPE_CACHE", "FLIGHT_SPIKE_MS", "FLIGHT_THUMBS", "FLIGHT_EVERY", "FLIGHT_FLASH", "FLIGHT_FLASH_DIFF", "FLIGHT_FLASH_CELLS", "FLUSH_SPAN", "FLUSH_UPLOAD", "FOG_CULL", "FOLIAGE_SHADOW_CULL", "FOLIAGE_SHADOW_VRS", "FOV", "FOV_PATCH", "FG_HUDLESS_SHOW", "FG_MULT", "FPS60", "FPS_LIMIT", "FRAMEGEN", "FRAMEGEN_MULTIPLIER", "FREEZE_IDLE", "FREEZE_ASYNC", "FREEZE_BUDGET_US", "FREE_CORES", "FULLSCREEN", "FXAA", "GAME_ASPECT", "GAME_RES", "GPA", "GPA_EMPTY", "GPA_TWIN", "GPA_TWIN_MAX_KB", "MIRROR_CAPTURE_KB", "WB_SKIP_MB", "GPL", "GPL_DRAW_BUILD", "GPL_FEEDBACK", "GPL_MESH", "PIPE_AHEAD", "PIPE_FAST_WORKERS", "PIPE_FRAME_WAIT_MS", "PIPE_LINK_DRAW_MS", "PIPE_SKIP_STEADY", "DRAW_WRITERS_WAIT", "PROG_LEAD", "GPU", "GPU_INDEX", "GPU_PASSES", "GPU_NAME", "HDR", @@ -65,7 +65,7 @@ static const char *const known[] = { "POOL_FAST", "POOL_GROUP", "POOL_GROUP_CHECK", "POOL_PRESNAP", "POOL_PRESNAP_RUNS", "SUBMIT_FINISH_GPU", "SVC_WAITS", "SNAP_WIDEN", "SUBMIT_FINISH_RESULTS", "SUBMIT_FINISH_KINDS", "FOLIAGE", "MIRROR_STAGE", "FLUSH_SKIP_CLEAN", "MIRROR_HOT", "MESH_EMULATION", "MESH_EMU_MB", "MESH_EMU_TASKS", "MESH_EMU_BATCH", "GUEST_OPT", "LOG_TEX_NEW", "DUMP_KEY", "VEC_INDEX_FIX", "PASS_TIMES", "PROFILE_MAIN", "PROFILE_RENDER", "PROMPTS", "PUSH_SKIP", "PUSH_STAGE", "RDOC", "RDOC_DIR", "RDOC_DLL", "REALTIME", "REC_END", "REEXEC", - "REFLEX", "LOW_LATENCY", "RENDER_SCALE", "REPLAY_ASYNC", "REPLAY_PIPE", "REPLAY_RUNS", "REPLAY_WORKERS", "RTSS", "RTSS_ENUM", "SCENE_SHARE", + "REFLEX", "LOW_LATENCY", "RENDER_SCALE", "REPLAY_ASYNC", "REPLAY_PIPE", "REPLAY_RUNS", "REPLAY_SMALL", "TAIL_EARLY", "REPLAY_WORKERS", "RTSS", "RTSS_ENUM", "SCENE_SHARE", "PRESNAP_PLACE", "REPLAY_CHECK", "RESOLVE_CHECK", "RESTART_WAIT", "ROMFS", "ROMFS_INDEX", "ROMFS_MMAP", "ROMFS_MODS", "ROMFS_PATCHES", "ROMFS_PROOF_DIR", "ROMFS_VERIFY", "RSTICK_DEADZONE", "RUMBLE", "RUMBLE_CURVE", "RUMBLE_HD", "RUMBLE_HD_MS", "RUMBLE_STRENGTH", "RUMBLE_TRIGGERS", "SHADER_CACHE", @@ -73,13 +73,13 @@ static const char *const known[] = { "SVC_SLOW_MS", "SHOTS_FULL", "SPEED_PROBE", "SPIN", "SPIN_HANDOFF_US", "SPLIT_DRAWS", "SPLIT_IDLE", "SPLIT_IN_PASS", "SPLIT_US", "SPV_MARK_TEST", "SPV_OVERRIDE", "STICK_DEADZONE", "STICK_SENS", "SWKBD_ANSWER", - "SET_AT", "SWKBD_TEST", "TAA", "DBG_TEST", "DEBUG_MENU", "MENU_TEST_DEBUG", "FSR4", "FSR4_FG", "FG4_VERSION", "FG4_SWAPCHAIN", "TEST_PAD", "TEST_PAD_AT", "TEST_PREPARE_SHADERS", "TEX_ALIAS", "DEPTH_INIT_INLINE", "DEPTH_ALIAS_LOG", "TEX_DISUSE", "TEX_DISUSE_EVICT", "TEX_GEN_UNDO", "TEX_MEMO", "TEX_VRAM_PRESSURE", "TEX_RELEASED_MB", "TEX_RELEASED_MAX", "TEX_REUSE", "TEX_SPLIT_FILL", "TEX_COMMIT_FREE_MB", "TEX_PROC_COMMIT_MB", "TEX_EVICT", "TEX_COMMIT_EVICT", "MIRROR_EVICT", "VRAM_BUDGET_MB", "TIMER_1MS", "TRACE", "TXD_TRANSPOSE", "UBO_SNAPSHOT", "EVO_TRACE", "EVENT_AT_SYNC", "UPSCALER", "VK_DYN_A2C", "VK_DYN_VERTEX", "VK_NAN_PRESERVE", "VK_NOOPT", "VK_NO_FLOAT64", + "SET_AT", "SWKBD_TEST", "TAA", "DBG_TEST", "DEBUG_MENU", "MENU_TEST_DEBUG", "FSR4", "FSR4_FG", "FG4_VERSION", "FG4_SWAPCHAIN", "TEST_PAD", "TEST_PAD_AT", "TEST_PREPARE_SHADERS", "TEX_ALIAS", "DEPTH_INIT_INLINE", "DEPTH_ALIAS_LOG", "TEX_DISUSE", "TEX_DISUSE_EVICT", "TEX_GEN_UNDO", "TEX_MEMO", "TEX_VRAM_PRESSURE", "TEX_RELEASED_MB", "TEX_RELEASED_MAX", "TEX_RELEASED_HOLD", "TEX_REUSE", "TEX_SPLIT_FILL", "TEX_COMMIT_FREE_MB", "TEX_PROC_COMMIT_MB", "TEX_EVICT", "TEX_COMMIT_EVICT", "MIRROR_EVICT", "VRAM_BUDGET_MB", "TIMER_1MS", "TRACE", "TXD_TRANSPOSE", "UBO_SNAPSHOT", "EVO_TRACE", "EVENT_AT_SYNC", "UPSCALER", "VK_DYN_A2C", "VK_DYN_VERTEX", "VK_NAN_PRESERVE", "VK_NOOPT", "VK_NO_FLOAT64", "VK_PRESENT", "VK_RECONVERGE", "VK_NAN_BITS", "VK_RECORDER", "VK_SSBO_ALIGN", "VK_SUBGROUP", "VK_SUBGROUP_PIN", "VK_VALIDATION", "VOLUME", "VRAM_HEADROOM", "VS_OUTS", "WATER_PROJECTIVE", "WAIT_WATCH_S", "WALK_KEY", "WALK_SCALE", "WB_EXACT", "WB_FENCE_LIMIT", "WB_MERGE", "WB_NO_REUP", "WB_MERGE_MAX", "WB_PROBE", "WB_SPLIT", "WB_LAZY", "WB_LAZY_CHUNK", "WB_LAZY_TOUCH", "WINDOW_SIZE", "XLATE", "XLATE_ID", "XLATE_KEEP", "XLATE_PROCS", "UPLOAD_BATCH_MB", "VRAM_HEAP_LIMIT", "VRAM_RESERVE_MB", "TEX_CLOBBER", "LOW_END_GPU", "ZSTD_CHECK", "ZSTD_NATIVE", "BIND_REUSE", "DYN_DELTA", "STG_ARENA", "SUBMIT_BATCH", "SUBMIT_THREAD", "ASTC_GPU_DESWIZZLE", "ASTC_SCRATCH_KEEP", "ASTC_ARENA", "ASTC_BC1", - "TEXUP_LOG_MS", "DYN_RES", "DYN_RES_TARGET", "DYN_RES_STEP", "TEXTURE_DETAIL", + "TEXUP_LOG_MS", "DYN_RES", "DYN_RES_TARGET", "DYN_RES_STEP", "TEXTURE_DETAIL", "VRAM_USE", "RECORDER_CORE", "RECORDER_PRIO", "REPLAY_TIME", "LOCK_TIME", "POOL_CREATE_BUDGET", "POOL_CREATE_MS", "POOL_FINI_IDLE", "POOL_LOAD_BACKLOG", "POOL_LOAD_MS", "VMA_BLOCK_MB", "SYNC_WAKE_ALL", "LOG_ASYNC", "DEVICE_LOST_AT", "STAGING_FAIL_AT", "VRAM_BALLAST_MB", "TEX_COMPACT", "TEX_COMPACT_MIN_MB", "TEX_COMPACT_FRAME_MB", // texture compaction (vk_texture.c); no menu row @@ -105,13 +105,13 @@ static const struct { const char *key, *section; } sections[] = { {"PHYSICS_STEPS", "frame rate"}, {"PICKUP_COUNT", "frame rate"}, {"MOVIE_PTS", "frame rate"}, {"STEP_STEADY", "frame rate"}, {"FULLSCREEN", "display"}, {"WINDOW_SIZE", "display"}, {"PERF_OVERLAY", "display"}, {"VK_PRESENT", "display"}, {"GPU_INDEX", "display"}, {"MONITOR", "display"}, {"OUTPUT_RES", "display"}, {"GAME_ASPECT", "display"}, {"GAME_RES", "display"}, {"RENDER_SCALE", "display"}, {"UPSCALER", "display"}, - {"SHARPNESS", "display"}, {"STANDIN_FILTER", "display"}, {"ASPECT", "display"}, {"FOV", "display"}, {"ANISO", "display"}, - {"TEXTURE_DETAIL", "display"}, + {"SHARPNESS", "display"}, {"STANDIN_FILTER", "display"}, {"SCENE_SHARE", "display"}, {"ASPECT", "display"}, {"FOV", "display"}, {"ANISO", "display"}, + {"TEXTURE_DETAIL", "display"}, {"VRAM_USE", "display"}, {"COLOR_PRESET", "display"}, {"COLOR_BRIGHTNESS", "display"}, {"COLOR_CONTRAST", "display"}, {"COLOR_SATURATION", "display"}, {"COLOR_GAMMA", "display"}, {"HDR", "display"}, {"HDR_LIVE", "display"}, {"HDR_PEAK_NITS", "display"}, {"HDR_GAME_NITS", "display"}, {"HDR_UI_NITS", "display"}, {"SHADOW_RES", "display"}, - {"DLAA", "display"}, {"FSR4", "display"}, {"FSR4_FG", "display"}, {"DLSS", "display"}, {"DLSS_PRESET", "display"}, {"DLSS_IDLE_FREE", "display"}, {"DLSS_RANGE", "display"}, {"DLSS_CHAR_MASK", "display"}, {"FXAA", "display"}, {"AA", "display"}, + {"DLAA", "display"}, {"FSR4", "display"}, {"FSR4_FG", "display"}, {"DLSS", "display"}, {"DLSS_PRESET", "display"}, {"DLSS_IDLE_FREE", "display"}, {"DLSS_RANGE", "display"}, {"DLSS_PREBLOOM", "display"}, {"DLSS_CHAR_MASK", "display"}, {"FXAA", "display"}, {"AA", "display"}, {"FRAMEGEN", "display"}, {"FRAMEGEN_MULTIPLIER", "display"}, {"FG_MULT", "display"}, {"SKIN_MOTION", "display"}, {"LOW_LATENCY", "display"}, {"DEVICE_PROFILE", "display"}, {"ASSUME_FOCUS", "display"}, @@ -135,7 +135,7 @@ static const struct { const char *key, *section; } sections[] = { {"BIND_RIGHT", "input"}, {"BIND_DOWN", "input"}, {"BIND_LSTICK_LEFT", "input"}, {"BIND_LSTICK_UP", "input"}, {"BIND_LSTICK_RIGHT", "input"}, {"BIND_LSTICK_DOWN", "input"}, {"BIND_RSTICK_LEFT", "input"}, {"BIND_RSTICK_UP", "input"}, {"BIND_RSTICK_RIGHT", "input"}, {"BIND_RSTICK_DOWN", "input"}, {"BIND_SHAKE", "input"}, - {"ASYNC_PREEMPT", "performance"}, {"REPLAY_ASYNC", "performance"}, {"REPLAY_WORKERS", "performance"}, {"REPLAY_PIPE", "performance"}, + {"ASYNC_PREEMPT", "performance"}, {"REPLAY_ASYNC", "performance"}, {"REPLAY_WORKERS", "performance"}, {"REPLAY_PIPE", "performance"}, {"TAIL_EARLY", "performance"}, {"REPLAY_SMALL", "performance"}, {"FREE_CORES", "performance"}, {"DYN_RES", "performance"}, {"DYN_RES_TARGET", "performance"}, {"DYN_RES_STEP", "performance"}, {"RECORDER_CORE", "performance"}, {"RECORDER_PRIO", "performance"}, {"BIND_REUSE", "performance"}, @@ -149,7 +149,7 @@ static const struct { const char *key, *section; } sections[] = { {"PIPE_SEED_BOOT", "performance"}, {"LIB_PREBUILD", "performance"}, {"LIB_PREBUILD_THREADS", "performance"}, {"LIB_WARM", "performance"}, {"PIPE_LINK_MS", "performance"}, {"PIPE_LINK_DRAW_MS", "performance"}, {"PIPE_AHEAD", "performance"}, {"LINE_WIDTH", "performance"}, {"POOL_SHOWN_FIRST", "performance"}, {"POOL_DATA_FIRST", "performance"}, {"TEX_REUSE_NAMED", "performance"}, {"TEX_OVERWRITTEN_USED", "performance"}, {"PIPE_FAST_WORKERS", "performance"}, {"GPL_FEEDBACK", "performance"}, {"GPL_MESH", "performance"}, {"FRAME_HOLD_MS", "performance"}, {"PACE_SMOOTH", "performance"}, {"OQ_BATCH", "performance"}, {"CLEAR_SMALL_CPU", "performance"}, {"PATCH_OCCLUSION_OFF", "performance"}, {"GATE_WAIT_MS", "performance"}, {"PIPE_PREBUILD", "performance"}, {"PIPE_SEED", "performance"}, {"COMPUTE_KEEP", "performance"}, {"PREWARM_XLATE", "performance"}, {"GPL", "performance"}, {"ASTC", "performance"}, {"MEMORY", "performance"}, - {"TEX_ALIAS", "performance"}, {"DEPTH_INIT_INLINE", "performance"}, {"DEPTH_ALIAS_LOG", "diagnostics"}, {"TEX_DISUSE", "performance"}, {"TEX_DISUSE_EVICT", "performance"}, {"TEX_GEN_UNDO", "performance"}, {"TEX_REUSE", "performance"}, {"TEX_SPLIT_FILL", "performance"}, {"TEX_VRAM_PRESSURE", "performance"}, {"TEX_PROC_COMMIT_MB", "performance"}, {"ASTC_BC", "performance"}, {"LOD_GUARD", "performance"}, {"FIX_C0", "performance"}, {"FIX_ZERO", "performance"}, {"WATER_PROJECTIVE", "performance"}, + {"TEX_ALIAS", "performance"}, {"DEPTH_INIT_INLINE", "performance"}, {"DEPTH_ALIAS_LOG", "diagnostics"}, {"TEX_DISUSE", "performance"}, {"TEX_DISUSE_EVICT", "performance"}, {"TEX_GEN_UNDO", "performance"}, {"TEX_REUSE", "performance"}, {"TEX_SPLIT_FILL", "performance"}, {"TEX_VRAM_PRESSURE", "performance"}, {"TEX_RELEASED_HOLD", "performance"}, {"TEX_PROC_COMMIT_MB", "performance"}, {"ASTC_BC", "performance"}, {"LOD_GUARD", "performance"}, {"FIX_C0", "performance"}, {"FIX_ZERO", "performance"}, {"WATER_PROJECTIVE", "performance"}, {"BIND_SKIP", "performance"}, {"SHADER_CACHE_FLUSH", "performance"}, {"SHADER_CARRY", "performance"}, {"PIPELINE_CACHE_PLAY", "performance"}, {"PIPELINE_CACHE_PLAY_MB", "performance"}, {"FFX_PIPE_CACHE", "performance"}, {"SHADER_MEMO", "performance"}, {"PIPE_MEMO", "performance"}, {"TEX_MEMO", "performance"}, {"DESC_TEMPLATE", "performance"}, {"TEX_CLOBBER", "performance"}, {"LOW_END_GPU", "performance"}, {"PUSH_SKIP", "performance"}, {"SPLIT_DRAWS", "performance"}, {"SPLIT_IDLE", "performance"}, {"SPLIT_IN_PASS", "performance"}, {"SPLIT_US", "performance"}, {"VK_VALIDATION", "performance"}, @@ -190,9 +190,13 @@ static const CfgMeta metas[] = { {"OUTPUT_RES", "1920x1080", NULL, RESTART}, {"RENDER_SCALE", "game", "game|50|67|75|100|125|150|200", RESTART}, {"UPSCALER", "fsr", "fsr|bilinear", LIVE}, {"SHARPNESS", "80", "0..100", LIVE}, {"STANDIN_FILTER", "easu", "easu|bilinear", LIVE}, // vk_tonemap.c vk_tonemap_record: how the scene is scaled to the picture + // widescreen.c plain_share: the scene's share of the picture without DLSS or FSR; 1 is the Upscaling row's Native, + // 0 the game's dynamic resolution. Live within the render-target heap the start made for it (heap_share). + {"SCENE_SHARE", "0.75", "0..1", LIVE}, {"DLAA", "0", "0|1", LIVE}, {"DLSS", "dlaa", "dlaa|quality|balanced|performance|ultra_performance", LIVE}, {"DLSS_PRESET", "default", "default|k|j", LIVE}, {"DLSS_IDLE_FREE", "1", "0|1|2", RESTART}, {"TAA", "dlss", "dlss|fsr3", LIVE}, // vk_dlaa.c fsr3 {"DLSS_RANGE", "0", "0|1", RESTART}, // vk_dlaa.c range_on: one DLSS feature for the render sizes of a mode + {"DLSS_PREBLOOM", "0", "0|1", LIVE}, // vk_dlaa.c prebloom_run: DLSS or FSR 3.1 before the bloom, not after the tonemap {"FSR4", "0", "0|1", LIVE}, // vk_ffx.cpp vk_ffx_fsr4 {"FSR4_FG", "0", "0|1", LIVE}, // vk_fg.cpp use4 {"FRAMEGEN", "off", "off|fsr3|dlss", LIVE}, // vk_fg.cpp, vk_dlssg.c @@ -208,6 +212,10 @@ static const CfgMeta metas[] = { {"ASPECT", "fit", "fit|stretch", LIVE}, {"FOV", "100", "70..140", LIVE}, {"ANISO", "16", "game|2|4|8|16", LIVE}, // vk_texture.c sampler_features, vk_sampler_aniso_live; 16: the game asks 1x or 2x {"TEXTURE_DETAIL", "auto", "auto|full|reduced|low", RESTART}, // vk_texture.c "texture detail", read at the first texture + // video memory use: low, the default, keeps released textures for the hold and frees what features switched off + // hold; high sets the VRAMHIGH defaults below and keeps those (cfg_vram_keep: vk_dlaa.c idle_release, vk_fsr.c + // idle_note, vk_aa.c work_idle). balanced, the default before 0.10, reads as low. + {"VRAM_USE", "low", "low|high", RESTART}, // colour adjustment (vk_color.c); the preset is the menu's, the four values are what the present uses {"COLOR_PRESET", "default", "default|vivid|clarity|custom", LIVE}, {"COLOR_BRIGHTNESS", "0", "-30..30", LIVE}, {"COLOR_CONTRAST", "100", "70..150", LIVE}, {"COLOR_SATURATION", "100", "50..200", LIVE}, @@ -279,7 +287,9 @@ static const CfgMeta metas[] = { {"COOP_ACCEPT", "ask", NULL, LIVE}, // Advanced: scheduling {"ASYNC_PREEMPT", "1", "0|1|2", RESTART}, {"REPLAY_ASYNC", "0", "0|1", RESTART}, - {"REPLAY_WORKERS", "0", "0..16", RESTART}, {"REPLAY_PIPE", "0", "0..4", RESTART}, + {"REPLAY_WORKERS", "auto", "auto|0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16", RESTART}, // vk_exec.c rw_init, vk_exec_replay_workers_auto {"REPLAY_PIPE", "0", "0..4", RESTART}, + {"TAIL_EARLY", "0", "0|1", RESTART}, // vk_exec.c "the early tail" + {"REPLAY_SMALL", "0", "0..1024", RESTART}, // vk_exec.c group_start {"FREE_CORES", "0", "0|1", RESTART}, {"TIMER_1MS", "1", "0|1", RESTART}, // dynamic resolution (dynres.c): the scene follows the GPU time toward DYN_RES_TARGET milliseconds {"DYN_RES", "0", "0|1", LIVE}, {"DYN_RES_TARGET", "16.7", "1..100", LIVE}, {"DYN_RES_STEP", "45", "8..600", LIVE}, @@ -354,7 +364,9 @@ static const CfgMeta metas[] = { {"PASS_TIMES", "0", "0|1", RESTART}, {"DUMP_KEY", "", NULL, RESTART}, // vk_crumbs.c, vk_dump.c // textures and memory {"ASTC", "auto", "auto|gpu|native|cpu", RESTART}, {"MEMORY", "auto", "auto|import|mirror|verify", RESTART}, - {"TEX_ALIAS", "1", "0|1", RESTART}, {"DEPTH_INIT_INLINE", "1", "0|1", RESTART}, {"DEPTH_ALIAS_LOG", "0", "0|1", RESTART}, {"TEX_DISUSE", "1", "0|1|2", RESTART}, {"TEX_DISUSE_EVICT", "0", "0|1", RESTART}, {"TEX_GEN_UNDO", "1", "0|1", RESTART}, {"TEX_REUSE", "1", "0|1", RESTART}, {"TEX_SPLIT_FILL", "1", "0|1|2", RESTART}, {"TEX_VRAM_PRESSURE", "80", "0..100", RESTART}, {"ASTC_BC", "", "0|1", RESTART}, {"ZSTD_NATIVE", "1", "0|1", RESTART}, {"ZSTD_CHECK", "0", "0|1", RESTART}, + {"TEX_ALIAS", "1", "0|1", RESTART}, {"DEPTH_INIT_INLINE", "1", "0|1", RESTART}, {"DEPTH_ALIAS_LOG", "0", "0|1", RESTART}, {"TEX_DISUSE", "1", "0|1|2", RESTART}, {"TEX_DISUSE_EVICT", "0", "0|1", RESTART}, {"TEX_GEN_UNDO", "1", "0|1", RESTART}, {"TEX_REUSE", "1", "0|1", RESTART}, {"TEX_SPLIT_FILL", "1", "0|1|2", RESTART}, {"TEX_VRAM_PRESSURE", "80", "0..100", RESTART}, + // vk_texture.c drop_aged: seconds a released texture is kept unused (vk_tex_release.h); vram_use = high sets 0 + {"TEX_RELEASED_HOLD", "180", "0..900", RESTART}, {"ASTC_BC", "", "0|1", RESTART}, {"ZSTD_NATIVE", "1", "0|1", RESTART}, {"ZSTD_CHECK", "0", "0|1", RESTART}, {"TEX_CLOBBER", "0", "0|1", RESTART}, {"LOW_END_GPU", "0", "0|1", RESTART}, // LOWEND below {"STG_ARENA", "1", "0|1", RESTART}, {"ASTC_GPU_DESWIZZLE", "1", "0|1", RESTART}, @@ -642,6 +654,40 @@ static const char *lowend_value(const char *key) { return NULL; } +// vram_use (NX2NAVI_VRAM_USE, the Screen tab's Video memory use row): low, the default, or high. low keeps a released +// texture until it has gone unused for the hold (vk_texture.c drop_aged, TEX_RELEASED_HOLD 180 s; vk_tex_release.h has +// the take-back figures behind it: in play the game took released textures back within a minute or not at all) and +// frees what a switched-off upscaler or SMAA holds. +// high sets the VRAMHIGH defaults for the keys the player does not set, after the device profile and the low-end mode: +// TEX_RELEASED_HOLD 0, released textures stay up to the size bound (a quarter of the video memory budget) for the whole +// session, as before 0.10; and it keeps what a switched-off feature holds (cfg_vram_keep). balanced, the default before +// 0.10, and anything else read as low. +// What the low of the 0.10 previews set and did cost something stays to be set on its own: +// - TEX_RELEASED_MB 1024: released textures keep at most 1 GiB; those the game takes back after a drop are made and +// uploaded or decoded again and show their stand-in for a frame (81 to 218 in each 3600 presents of a session on an +// RTX 5090, vk_texture.c released_cap_bytes). +// - TEX_VRAM_PRESSURE 70: textures that come back intact give way from 70% of the budget on, not 80% (sweep_pressure), +// and are made again when next used. +// - PIPE_SEED_BOOT 0: the shared list's pipelines this install never drew are made at their first draw, not at the +// start, which leaves out 0.3-0.6 GiB of video memory and 2-4 GB of commit; a session so set missed 1,091 pipelines +// at a draw, 115 of them for 8 to 34 ms, and their draws were skipped (vk_pipeline.c seed_boot). +// Unsampled live textures are not dropped in either (tex_disuse_evict stays 0): that eviction left the ground flat and +// see-through for good (vk_texture.c disuse_evict_on). +static const struct { const char *key, *value; } VRAMHIGH[] = { + {"TEX_RELEASED_HOLD", "0"}, +}; +static const char *vram_use_value(const char *key) { + const char *set = raw_get(PREFIX "VRAM_USE"); + if (!set || strcmp(set, "high")) return NULL; + for (size_t i = 0; i < sizeof VRAMHIGH / sizeof *VRAMHIGH; i++) + if (!strcasecmp(VRAMHIGH[i].key, key)) return VRAMHIGH[i].value; + return NULL; +} +int cfg_vram_keep(void) { + const char *set = cfg_get(PREFIX "VRAM_USE"); + return set && !strcmp(set, "high"); +} + static int profile_of(void) { int p = __atomic_load_n(&profile, __ATOMIC_ACQUIRE); if (p >= 0) return p; @@ -672,7 +718,8 @@ static const char *cfg_get_now(const char *name) { if (profile_of()) for (size_t i = 0; i < sizeof DECK / sizeof *DECK; i++) if (!strcasecmp(DECK[i].key, name + PREFIX_LEN)) return DECK[i].value; - return lowend_value(name + PREFIX_LEN); + const char *low = lowend_value(name + PREFIX_LEN); + return low ? low : vram_use_value(name + PREFIX_LEN); } // Whether name's value comes from the player (the environment or nx2navi.ini), not a profile or the table. @@ -774,6 +821,7 @@ const char *cfg_default(const char *key) { if (!strcasecmp(DECK[i].key, name)) return DECK[i].value; const char *low = lowend_value(name); if (low) return low; + if ((low = vram_use_value(name))) return low; const CfgMeta *m = cfg_meta(key); return m ? m->def : NULL; } diff --git a/runtime/cfg.h b/runtime/cfg.h index 75fa8db8..049aa3da 100644 --- a/runtime/cfg.h +++ b/runtime/cfg.h @@ -34,6 +34,8 @@ int cfg_ini_read(const char *path, const char *key, char *out, size_t cap); // // 1 when the player set name (the environment or nx2navi.ini), 0 when cfg_get gives a device profile's default or NULL. int cfg_is_set(const char *name); int cfg_low_end_active(void); // the low-end GPU mode was on when the game started (cfg.c LOWEND) +// vram_use = high: what a feature switched off holds stays for the run (cfg.c VRAMLOW, the comment above it) +int cfg_vram_keep(void); // The keys of the file's lines that are no settings of this version, lower-case with ", " between them ("" when there // are none; a list too long to keep ends early), and how many there are. Such a line does nothing: the log says so at // start and the settings menu shows it (ui/menu.cpp), so that nobody takes one for the cause of what they see. diff --git a/runtime/gfx.h b/runtime/gfx.h index 7a630acd..6aa6a065 100644 --- a/runtime/gfx.h +++ b/runtime/gfx.h @@ -164,6 +164,10 @@ typedef struct { // As fence_wait, but only until the GPU has executed the submission's own commands: its write-backs to guest // memory and its fence's side effects may follow (vk_sync.c exec_tl). The same as fence_wait where there is no split. int (*exec_wait)(uint64_t value, uint64_t timeout_ns); + // The game's own wait for a fence (nvnSyncWait): fence_wait, or where the back end knows the rest of the value's + // GPU work reads nothing the game can change and hands it nothing, until its results are there (vk_sync.c + // vk_sync_game_wait). + int (*game_wait)(uint64_t value, uint64_t timeout_ns); uint64_t (*fence_done)(void); // highest value whose side effects are written // Queues the present and returns at once; the host present goes out at at_ns (kern_now_ns clock), at once when that // has passed or at_ns is 0. diff --git a/runtime/gfx_null.c b/runtime/gfx_null.c index b73010a4..bd4d1b61 100644 --- a/runtime/gfx_null.c +++ b/runtime/gfx_null.c @@ -147,6 +147,7 @@ const GfxBackend gfx_null_backend = { .submit = null_submit, .fence_wait = null_fence_wait, .exec_wait = null_fence_wait, + .game_wait = null_fence_wait, .fence_done = null_fence_done, .present = null_present, .finish = null_finish, diff --git a/runtime/host.h b/runtime/host.h index 275dffc2..6b444980 100644 --- a/runtime/host.h +++ b/runtime/host.h @@ -196,6 +196,9 @@ int mem_start_note(int probe); // a game's start: a // below the game's need (mem_need.h); 1 when it is void mem_guest_census(uint64_t frame); // NX2NAVI_TRACE=perf: committed bytes per guest region void mem_lazy_heap_report(uint64_t frame); // NX2NAVI_TRACE=perf: the lazy heap's faults since the last line +// The game's own memory for the memory report (vk_memory.c owner_log): its heap's size, the part of it committed (the +// size itself unless the heap is lazy), and the physical memory it mapped. No lock: a log may read a figure late. +void mem_guest_usage(uint64_t *heap, uint64_t *heap_committed, uint64_t *phys_maps); uint32_t mem_set_heap_size(uint64_t size, uint64_t *addr); // 0x01 uint32_t mem_set_permission(uint64_t a, uint64_t n, uint32_t perm); // 0x02 uint32_t mem_set_attribute(uint64_t a, uint64_t n, uint32_t mask, uint32_t at); // 0x03 @@ -364,6 +367,10 @@ void wide_init(void); void wide_present(void); // each present: DLSS's render scale (res_fixed) // after rt_patches_init: widescreen.c, NX2NAVI_GAME_ASPECT int wide_available(void); // the generated C has the widescreen sets void wide_picture(unsigned *w, unsigned *h); // the picture the game draws (its scene is never larger) +int wide_scene_settable(void); // the generated C has res_fixed: the scene's share can be set +double wide_scene_share(void); // the scene's share of the picture now (0: the game's own) +double wide_scene_wanted(void); // ... what the settings ask for, before the heap's limit +int wide_scene_restart(void); // a restart gives the larger scene the settings ask for uint64_t wide_heap_extra_kib(void); // the texture heap's growth for the widescreen picture // The most res_heap_extra_kib may be (memory.c): with more the application pool passes the heap region and the // game's own SetHeapSize is refused. A whole number of 2 MiB. diff --git a/runtime/memory.c b/runtime/memory.c index 6940c49c..08bbb0c0 100644 --- a/runtime/memory.c +++ b/runtime/memory.c @@ -1806,6 +1806,12 @@ void mem_guest_census(uint64_t frame) { __atomic_load_n(&heap_size, __ATOMIC_ACQUIRE) / 1048576.0, (unsigned long long)heap_lazy_faults); } +void mem_guest_usage(uint64_t *heap, uint64_t *heap_committed, uint64_t *phys_maps) { + *heap = __atomic_load_n(&heap_size, __ATOMIC_RELAXED); + *heap_committed = heap_lazy ? __atomic_load_n(&heap_lazy_bytes, __ATOMIC_RELAXED) : *heap; + *phys_maps = __atomic_load_n(&alias_bytes, __ATOMIC_RELAXED); +} + // NX2NAVI_TRACE=perf "mem: sizes" (perf.c): the guest heap and physical maps, the kernel region's bump and commit, the // moved stacks, and the lazily written ranges with their array's room. void mem_sizes(PerfSizes *s) { diff --git a/runtime/nvn_api_core.c b/runtime/nvn_api_core.c index 5c6fa553..ac4a43b9 100644 --- a/runtime/nvn_api_core.c +++ b/runtime/nvn_api_core.c @@ -594,7 +594,7 @@ NVN_API(nvnSyncWait) { int ok; { NVN_BLOCKING(); - ok = b->fence_wait(s->fence, timeout); + ok = b->game_wait(s->fence, timeout); } if (ok) b->cpu_synced(s->fence); ret_x(c, ok ? NVN_SYNC_WAIT_RESULT_CONDITION_SATISFIED : NVN_SYNC_WAIT_RESULT_TIMEOUT_EXPIRED); diff --git a/runtime/ui/menu.cpp b/runtime/ui/menu.cpp index 00c5f6f9..bae7c379 100644 --- a/runtime/ui/menu.cpp +++ b/runtime/ui/menu.cpp @@ -67,6 +67,11 @@ void vk_queue_unlock(void); void res_sizes(unsigned out[2], unsigned scene[2]); int res_adjustable(void); int wide_available(void); +int wide_scene_settable(void); +void wide_picture(unsigned *w, unsigned *h); +double wide_scene_share(void); +double wide_scene_wanted(void); +int wide_scene_restart(void); const char *const *vk_color_preset(const char *name); int vk_hdr_wanted(void); int vk_hdr_active(void); @@ -87,6 +92,7 @@ const char *vk_fg4_name(void); // ... the version the library runs here int vk_texture_detail_auto(void); // vk_texture.c: the texture detail auto picks on this card, 0 full to 2 low int vk_mesh_shader_native(void); // vk_device.c: the GPU has mesh shaders (the mesh emulation does nothing) int vk_low_end_card(void); // vk_device.c: no mesh shaders, or under 8 GiB of video memory +int vk_exec_replay_workers_auto(const char **why); // vk_exec.c: replay_workers = auto's count on this PC double frame_step_limit_hz(void); int frame_limit_adjustable(void); int rtss_limit_state(void); @@ -766,6 +772,16 @@ static void row_texture_detail(void) { opt_choice_list("NX2NAVI_TEXTURE_DETAIL", UI_MENU_TEX_DETAIL, vals, names, 4, UI_MENU_TEX_DETAIL_DESC); } +// Video memory use (cfg.c vram_use_value, cfg_vram_keep): low, the default, lets released textures go after the hold +// (vk_texture.c drop_aged) and frees what switched-off features made; high keeps both. Any value but high is low, as +// cfg.c reads it (balanced, the default before 0.10, among them). Read at the start, so the row notes the restart. +static void row_vram_use(void) { + static const char *const vals[] = {"low", "high"}; + const char *names[] = {menu_text(UI_MENU_VRAM_USE_LOW), menu_text(UI_MENU_VRAM_USE_HIGH)}; + opt_choice_list("NX2NAVI_VRAM_USE", UI_MENU_VRAM_USE, vals, names, 2, UI_MENU_VRAM_USE_DESC, + cfg_vram_keep() ? 1 : 0); +} + static void row_shadow_scale(void) { static const char *const vals[] = {"1", "2", "3", "4"}; static const char *const names[] = {"960 (1x)", "1920 (2x)", "2880 (3x)", "3840 (4x)"}; @@ -928,6 +944,14 @@ static const char *fsr4_label(void) { return s; } +// Native (the Upscaling row): the scene drawn at the picture's own size, which the upscaler's stand-in then only +// decodes 1:1 (vk_tonemap.c), with no DLSS or FSR. It is scene_share = 1 (widescreen.c plain_share) with dlaa = 0; a +// scene_share of 1 or more set by hand in nx2navi.ini reads the same. +static bool share_native(void) { + const char *s = cfg_get("NX2NAVI_SCENE_SHARE"); + return s && *s && atof(s) >= 1; +} + // The Upscaling row's name of what is set now, for the performance overlay. static void upscaling_name(char *out, size_t size) { static const char *const modes[] = {"dlaa", "quality", "balanced", "performance", "ultra_performance"}; @@ -935,7 +959,8 @@ static void upscaling_name(char *out, size_t size) { static const UiTextId fsr3[] = {UI_MENU_FSR3_NATIVE, UI_MENU_DLSS_QUALITY, UI_MENU_DLSS_BALANCED, UI_MENU_DLSS_PERFORMANCE, UI_MENU_DLSS_ULTRA}; // FSR 3.1 set where it cannot load runs nothing (vk_dlaa.c vk_dlaa_fsr3): the game scales the picture as the rest says if (!cfg_bool("NX2NAVI_DLAA") || (!strcmp(cfg_value("NX2NAVI_TAA"), "fsr3") && !vk_ffx_available())) { - if (strcmp(cfg_value("NX2NAVI_UPSCALER"), "fsr")) snprintf(out, size, "%s", menu_text(UI_MENU_UPSCALER_BILINEAR)); + if (share_native() && wide_scene_settable()) snprintf(out, size, "%s", menu_text(UI_MENU_UPSCALING_NATIVE)); + else if (strcmp(cfg_value("NX2NAVI_UPSCALER"), "fsr")) snprintf(out, size, "%s", menu_text(UI_MENU_UPSCALER_BILINEAR)); else snprintf(out, size, menu_text(UI_MENU_OV_FMT_FSR), cfg_int("NX2NAVI_SHARPNESS")); return; } @@ -974,8 +999,9 @@ static void row_upscaling(void) { static const UiTextId fsr_names[] = {UI_MENU_FSR3_NATIVE, UI_MENU_DLSS_QUALITY, UI_MENU_DLSS_BALANCED, UI_MENU_DLSS_PERFORMANCE, UI_MENU_DLSS_ULTRA}; // FSR 4 (vk_ffx4.cpp) is the same setting as FSR 3.1 with NX2NAVI_FSR4 on, and takes the same modes: listed where - // it runs on this device, after FSR 3.1's - enum { BILINEAR, FSR1, DLSS, FSR3, FSR4 }; + // it runs on this device, after FSR 3.1's. Native (share_native) is listed where the scene's share can be set + // (widescreen.c wide_scene_settable), after the two entries that enlarge the game's smaller scene. + enum { BILINEAR, FSR1, NATIVE, DLSS, FSR3, FSR4 }; struct Entry { int kind, mode; }; static char text[15][96]; Entry list[18]; @@ -985,6 +1011,7 @@ static void row_upscaling(void) { list[n] = {kind, mode}; if (kind == BILINEAR) names[n] = menu_text(UI_MENU_UPSCALER_BILINEAR); else if (kind == FSR1) names[n] = menu_text(UI_MENU_UPSCALER_FSR); + else if (kind == NATIVE) names[n] = menu_text(UI_MENU_UPSCALING_NATIVE); else { char *t = text[(kind == FSR4 ? 10 : kind == FSR3 ? 5 : 0) + mode]; if (kind == DLSS) snprintf(t, sizeof text[0], mode ? "DLSS %s" : "NVIDIA %s", menu_text(mode_names[mode])); @@ -994,7 +1021,7 @@ static void row_upscaling(void) { n++; }; // what is set now - Entry now = {strcmp(cfg_value("NX2NAVI_UPSCALER"), "fsr") ? BILINEAR : FSR1, 0}; + Entry now = {share_native() ? NATIVE : strcmp(cfg_value("NX2NAVI_UPSCALER"), "fsr") ? BILINEAR : FSR1, 0}; if (cfg_bool("NX2NAVI_DLAA")) { now = {strcmp(cfg_value("NX2NAVI_TAA"), "fsr3") ? DLSS : vk_ffx_fsr4() ? FSR4 : FSR3, 0}; // set where it does for (int i = 0; i < 5; i++) // not run: FSR 3.1 does @@ -1003,6 +1030,7 @@ static void row_upscaling(void) { const bool dlss = dlss_offered(), fsr = vk_ffx_available(), fsr4 = fsr && vk_ffx4_available(); add(BILINEAR, 0); add(FSR1, 0); + if (wide_scene_settable() || now.kind == NATIVE) add(NATIVE, 0); for (int i = 0; dlss && i < 5; i++) add(DLSS, i); for (int i = 0; fsr && i < 5; i++) add(FSR3, i); for (int i = 0; fsr4 && i < 5; i++) add(FSR4, i); @@ -1011,10 +1039,21 @@ static void row_upscaling(void) { int cur = 0; for (int i = 0; i < n; i++) if (list[i].kind == now.kind && list[i].mode == now.mode) cur = i; - // FSR 3.1 still set where it cannot load: nothing upscales, so the row says so under its value + // FSR 3.1 still set where it cannot load: nothing upscales, so the row says so under its value. A scene larger than + // the render-target heap the start made (Native, or DLAA chosen in a run that started with a smaller scene) waits + // for a restart; at the heap's limit for this picture it stays smaller, and the note says how large it is + // (widescreen.c wide_scene_restart, heap_share) const char *note = nullptr; + static char in_use[64]; + const double share = wide_scene_settable() ? wide_scene_share() : 0; if (!fsr && now.kind == FSR3) note = menu_text(UI_MENU_FSR3_OFF); - else if (upscaling_restart()) note = menu_text(UI_MENU_AFTER_RESTART); + else if (upscaling_restart() || (share > 0 && wide_scene_restart())) note = menu_text(UI_MENU_AFTER_RESTART); + else if (share > 0 && wide_scene_wanted() > share + 0.001) { + char pct[16]; + snprintf(pct, sizeof pct, "%.0f %%", share * 100); + snprintf(in_use, sizeof in_use, menu_text(UI_MENU_PRESENT_IN_USE), pct); + note = in_use; + } int nv = opt_choice_index("NX2NAVI_UPSCALER", UI_MENU_UPSCALING, names, n, cur, 1, !fsr ? UI_MENU_UPSCALING_DESC_NO_FSR : dlss ? UI_MENU_UPSCALING_DESC : UI_MENU_UPSCALING_DESC_FSR, note); @@ -1023,6 +1062,10 @@ static void row_upscaling(void) { opt_apply("NX2NAVI_UPSCALER", e.kind == FSR1 ? "fsr" : "bilinear"); opt_apply("NX2NAVI_DLAA", e.kind >= DLSS ? "1" : "0"); if (e.kind < DLSS) opt_apply("NX2NAVI_FRAMEGEN", "off"); // it needs DLSS's or FSR 3.1's motion vectors (row_framegen) + // Native is scene_share = 1. Leaving it unsets the key (the game's own share, widescreen.c PLAIN_SHARE); a share of + // the whole picture set by hand beside DLSS or FSR stays while one of their modes is chosen + if (e.kind == NATIVE) opt_apply("NX2NAVI_SCENE_SHARE", "1"); + else if (share_native() && (e.kind < DLSS || now.kind == NATIVE)) opt_apply("NX2NAVI_SCENE_SHARE", ""); if (e.kind >= DLSS) opt_apply("NX2NAVI_DLSS", modes[e.mode]); if (e.kind >= DLSS) opt_apply("NX2NAVI_TAA", e.kind >= FSR3 ? "fsr3" : "dlss"); if (e.kind >= FSR3) opt_apply("NX2NAVI_FSR4", e.kind == FSR4 ? "1" : "0"); @@ -1320,6 +1363,7 @@ static void tab_screen(float scale) { const char *aniso_names[] = {menu_text(UI_MENU_ANISO_GAME), "2x", "4x", "8x", "16x"}; opt_choice_list("NX2NAVI_ANISO", UI_MENU_ANISO, aniso, aniso_names, 5, UI_MENU_ANISO_DESC); row_texture_detail(); + row_vram_use(); opt_end(); } @@ -1436,8 +1480,24 @@ static void tab_game(float scale) { rows_saves(); // Testing: experimental rows, then the debug menu where the game version has it opt_group_text("Testing"); - // vk_exec.c rw_init reads it at start: the row notes the restart and the bar's Restart does it - opt_slider("NX2NAVI_REPLAY_WORKERS", UI_MENU_REPLAY_WORKERS, 0, 8, UI_MENU_REPLAY_WORKERS_DESC, 1, "%.0f"); + // vk_exec.c rw_init and tail_enabled read these at start: each row notes the restart and the bar's Restart does + // it. A row whose setting does nothing without another one (rw_init) is greyed out until that one is set. + // Automatic first, the default (vk_exec.c vk_exec_replay_workers_auto), named with the count it gives on this PC + { + static const char *const vals[] = {"auto", "0", "1", "2", "3", "4", "5", "6", "7", "8"}; + static char auto_name[64]; + snprintf(auto_name, sizeof auto_name, "%s (%d)", menu_text(UI_MENU_AUTO), vk_exec_replay_workers_auto(nullptr)); + const char *names[] = {auto_name, "0", "1", "2", "3", "4", "5", "6", "7", "8"}; + opt_choice_list("NX2NAVI_REPLAY_WORKERS", UI_MENU_REPLAY_WORKERS, vals, names, 10, UI_MENU_REPLAY_WORKERS_DESC); + } + const char *wset = cfg_value("NX2NAVI_REPLAY_WORKERS"); + const bool workers = (!strcmp(wset, "auto") ? vk_exec_replay_workers_auto(nullptr) : atoi(wset)) > 0, + pipe = workers && cfg_int("NX2NAVI_REPLAY_PIPE") > 0; + if (!workers) opt_disable_next(UI_MENU_NEEDS_WORKERS); + opt_slider("NX2NAVI_REPLAY_PIPE", UI_MENU_REPLAY_PIPE, 0, 4, UI_MENU_REPLAY_PIPE_DESC, 1, "%.0f"); + if (!pipe) opt_disable_next(workers ? UI_MENU_NEEDS_PIPE : UI_MENU_NEEDS_WORKERS); + opt_slider("NX2NAVI_REPLAY_SMALL", UI_MENU_REPLAY_SMALL, 0, 64, UI_MENU_REPLAY_SMALL_DESC, 4, "%.0f"); + opt_toggle("NX2NAVI_TAIL_EARLY", UI_MENU_TAIL_EARLY, UI_MENU_TAIL_EARLY_DESC); // The debug menu, a tool for testing with English text of its own: off unless switched on here if (dbg_supported()) { const bool on = cfg_bool("NX2NAVI_DEBUG_MENU"); @@ -1557,6 +1617,7 @@ static int restart_pending(void) { bool prompts; prompts_note(&prompts); return cfg_pending_count() + upscaling_restart() + framegen_restart() + (vk_hdr_needs_restart() != 0) + + (wide_scene_settable() && wide_scene_share() > 0 && wide_scene_restart() && !upscaling_restart()) + (prompts && !cfg_pending("NX2NAVI_PROMPTS")); } @@ -2025,6 +2086,14 @@ static void perf_tick(uint64_t now, const MenuTarget *t) { static void perf_picture(const MenuTarget *t, PerfPanel *p) { unsigned out[2], scene[2]; res_sizes(out, scene); + // Ocarina's build has no output and scene patch sets, so res_sizes holds the other builds' placeholders there: the + // picture and the scene are widescreen.c's, the scene at the share it writes for the game (wide_present; in the + // first two seconds of a run the game's own dynamic resolution still sizes it) + const double share = !res_adjustable() && wide_scene_settable() ? wide_scene_share() : 0; + if (share > 0) { + wide_picture(&out[0], &out[1]); + scene[0] = (unsigned)lround(out[0] * share), scene[1] = (unsigned)lround(out[1] * share); + } char (*v)[64] = p->picture; snprintf(v[0], sizeof v[0], menu_text(UI_MENU_OV_FMT_SIZE), t->width, t->height); snprintf(v[1], sizeof v[1], menu_text(UI_MENU_OV_FMT_SIZE), (unsigned)t->image_w, (unsigned)t->image_h); diff --git a/runtime/ui_text.c b/runtime/ui_text.c index 1f261615..6ec1df44 100644 --- a/runtime/ui_text.c +++ b/runtime/ui_text.c @@ -82,6 +82,7 @@ static const char *const text_en[UI_TEXT_COUNT] = { "sharper than bilinear. Smaller window (Ocarina draws 2560 x 1440): the picture is shrunk " "and FSR sharpens it by the strength below.", [UI_MENU_UPSCALER_FSR] = "AMD FSR 1", [UI_MENU_UPSCALER_BILINEAR] = "Bilinear", + [UI_MENU_UPSCALING_NATIVE] = "Native (100%)", [UI_MENU_GAME_ASPECT] = "Widescreen", [UI_MENU_GAME_ASPECT_DESC] = "Widens the view. HUD and menus stay 16:9.", [UI_MENU_GAME_ASPECT_16_9] = "16:9 (the game's)", [UI_MENU_GAME_ASPECT_WINDOW] = "Window shape", @@ -98,6 +99,13 @@ static const char *const text_en[UI_TEXT_COUNT] = { [UI_MENU_TEX_DETAIL_DESC] = "Sharpness of large surface textures. Reduced and Low leave out their largest versions, so that an area fits the video memory of 6 and 8 GB cards; only surfaces seen up close look softer. Automatic picks by the card's video memory.", [UI_MENU_TEX_DETAIL_AUTO] = "Automatic: %s", [UI_MENU_TEX_DETAIL_FULL] = "Full", [UI_MENU_TEX_DETAIL_REDUCED] = "Reduced", [UI_MENU_TEX_DETAIL_LOW] = "Low", + [UI_MENU_VRAM_USE] = "Video memory use", + [UI_MENU_VRAM_USE_DESC] = "How much video memory the game keeps for later. Low, the default, lets go of the textures " + "of places you left once they have gone unused for three minutes, and of what an upscaler " + "or anti-aliasing switched off had made; in play the game asked for such textures again " + "within a minute or not at all. High keeps both for the whole session. Takes effect after " + "a restart.", + [UI_MENU_VRAM_USE_LOW] = "Low", [UI_MENU_VRAM_USE_HIGH] = "High", [UI_MENU_LOWEND_INTRO] = "The settings of the low-end GPU mode. Each can be changed here; most take effect after a restart.", [UI_MENU_LOWEND_RESTART] = "The mode turns on after a restart. Its settings are listed here then.", [UI_MENU_LOWEND_NOT_FOR_CARD] = "Not for this graphics card: it has mesh shaders and 8 GB of video memory or more. " @@ -499,11 +507,26 @@ static const char *const text_en[UI_TEXT_COUNT] = { "where the card has none, keeps video memory within what the driver reports free, waits less for the GPU and " "prepares frames on a thread of their own. Measured on a Radeon RX 580 only: it can show a dark or flickering " "frame now and then, so leave it off on a newer card.", - [UI_MENU_REPLAY_WORKERS] = "Replay workers (experimental)", - [UI_MENU_REPLAY_WORKERS_DESC] = "CPU threads that help translate each frame's draws for the GPU, beside the game's own " - "thread. 3 gave about 20% more frames a second in Hyrule Field on an 8-core CPU; more " - "than 3 gave nothing more. Some players' games crashed with them on: 0 (off) is the " - "safe choice. Takes effect after a restart.", + [UI_MENU_REPLAY_WORKERS] = "CPU helper threads (experimental)", + [UI_MENU_REPLAY_WORKERS_DESC] = "Uses more of your CPU's cores to prepare each frame for the graphics card. Automatic " + "picks the number for your CPU: half its cores less one, none on four cores or fewer " + "and none on RX 6000 and older Radeon cards. 0 turns it off. 3 gave about 20% more " + "frames a second on an 8-core CPU. If your game crashes with this on, set it to 0.", + [UI_MENU_REPLAY_PIPE] = "Helpers work ahead (experimental)", + [UI_MENU_REPLAY_PIPE_DESC] = "The game carries on with the rest of the frame while the helper threads are still " + "busy, instead of waiting for them. The number is how many batches of work the " + "helpers may have at once; 0 turns it off. A small gain: 4 gave 1 to 2% more frames a " + "second.", + [UI_MENU_REPLAY_SMALL] = "Helpers take small jobs (experimental)", + [UI_MENU_REPLAY_SMALL_DESC] = "Gives the helper threads the smaller batches of drawing work too, not only the large " + "ones. The number is the smallest batch a helper takes, counted in draw calls; 0 " + "turns it off. A small gain: 16 gave about 2% more frames a second.", + [UI_MENU_TAIL_EARLY] = "Start next frame early (experimental)", + [UI_MENU_TAIL_EARLY_DESC] = "The game starts on its next frame while the graphics card is still finishing the " + "current one. The biggest gain of these settings: 19 to 29% more frames a second with " + "FSR 4 on a Radeon RX 7900 XT, the only card it was measured on. It needs TAA, DLSS, " + "FSR 3.1 or FSR 4, and for now an AMD graphics card. Without them it changes nothing.", + [UI_MENU_NEEDS_WORKERS] = "needs CPU helper threads", [UI_MENU_NEEDS_PIPE] = "needs Helpers work ahead", [UI_MENU_DISCORD_RPC] = "Discord Rich Presence", [UI_MENU_DISCORD_RPC_DESC] = "Shows \"Playing NX2navi\" with the play time in your Discord status while the " "game runs. discord_app_id and discord_image in nx2navi.ini choose the application.", @@ -802,8 +825,10 @@ static const char *const text_en[UI_TEXT_COUNT] = { [UI_BOOT_STARTING] = "Starting", [UI_BOOT_SHADERS] = "Shaders: %lld of %lld", [UI_BOOT_PIPELINES] = "Pipelines: %lld of %lld", [UI_BOOT_COMPUTE] = "Compute pipelines: %lld of %lld", [UI_BOOT_DRIVER_CACHE] = "Driver cache: %lld of %lld", [UI_MENU_UPSCALING] = "Upscaling", - [UI_MENU_UPSCALING_DESC] = "How the picture is smoothed and scaled to your window.", - [UI_MENU_UPSCALING_DESC_FSR] = "How the picture is smoothed and scaled to your window.", + [UI_MENU_UPSCALING_DESC] = "How the picture is smoothed and scaled to your window. Native draws the 3D scene at " + "the picture's full size with no upscaler: the sharpest, and the most GPU work.", + [UI_MENU_UPSCALING_DESC_FSR] = "How the picture is smoothed and scaled to your window. Native draws the 3D scene " + "at the picture's full size with no upscaler: the sharpest, and the most GPU work.", [UI_MENU_FRAMEGEN] = "Frame generation", [UI_MENU_FRAMEGEN_DESC] = "Adds generated frames for smoother motion. Needs DLSS or FSR 3.1.", [UI_MENU_FRAMEGEN_OFF] = "Off", @@ -813,8 +838,10 @@ static const char *const text_en[UI_TEXT_COUNT] = { [UI_MENU_FRAMEGEN_NEEDS_FSR] = "choose an FSR 3.1 mode under Upscaling first", [UI_MENU_FRAMEGEN_NOTE_FSR] = "needs FSR 3.1", [UI_MENU_FRAMEGEN_DESC_FSR] = "Adds generated frames for smoother motion. Needs FSR 3.1.", - [UI_MENU_UPSCALING_DESC_NO_FSR] = "How the picture is smoothed and scaled to your window. AMD FSR 3.1 is not offered " - "here: it needs amd_fidelityfx_vk.dll, which cannot be loaded.", + [UI_MENU_UPSCALING_DESC_NO_FSR] = "How the picture is smoothed and scaled to your window. Native draws the 3D scene " + "at the picture's full size with no upscaler: the sharpest, and the most GPU " + "work. AMD FSR 3.1 is not offered here: it needs amd_fidelityfx_vk.dll, which " + "cannot be loaded.", [UI_MENU_FSR3_OFF] = "in use: off, FSR 3.1 is not available", [UI_MENU_FRAMEGEN_DESC_DLSS] = "Adds generated frames for smoother motion. Needs DLSS.", [UI_MENU_FRAMEGEN_NOTE_DLSS] = "needs DLSS", @@ -900,6 +927,17 @@ static const char *const text_ru[UI_TEXT_COUNT] = { "билинейного. Окно меньше (Ocarina рисует 2560 × 1440): картинка уменьшается, и FSR " "повышает её резкость на величину ниже.", [UI_MENU_UPSCALER_FSR] = "AMD FSR 1", [UI_MENU_UPSCALER_BILINEAR] = "Билинейное", + [UI_MENU_UPSCALING_NATIVE] = "Родное (100%)", + [UI_MENU_UPSCALING_DESC] = "Как картинка сглаживается и масштабируется под окно. «Родное» рисует 3D-сцену в " + "полном размере картинки без апскейлера: самая чёткая картинка и самая большая " + "нагрузка на видеокарту.", + [UI_MENU_UPSCALING_DESC_FSR] = "Как картинка сглаживается и масштабируется под окно. «Родное» рисует 3D-сцену в " + "полном размере картинки без апскейлера: самая чёткая картинка и самая большая " + "нагрузка на видеокарту.", + [UI_MENU_UPSCALING_DESC_NO_FSR] = "Как картинка сглаживается и масштабируется под окно. «Родное» рисует 3D-сцену " + "в полном размере картинки без апскейлера: самая чёткая картинка и самая " + "большая нагрузка на видеокарту. AMD FSR 3.1 здесь не предлагается: ему нужен " + "amd_fidelityfx_vk.dll, который не удаётся загрузить.", [UI_MENU_GAME_ASPECT] = "Широкий экран", [UI_MENU_GAME_ASPECT_DESC] = "Расширяет обзор. Интерфейс и меню остаются 16:9.", [UI_MENU_GAME_ASPECT_16_9] = "16:9 (как в игре)", [UI_MENU_GAME_ASPECT_WINDOW] = "По окну", @@ -916,6 +954,13 @@ static const char *const text_ru[UI_TEXT_COUNT] = { [UI_MENU_TEX_DETAIL_DESC] = "Чёткость больших текстур поверхностей. «Сниженная» и «Низкая» не загружают их самые крупные версии, чтобы область помещалась в видеопамять карт на 6 и 8 ГБ; мягче выглядят только поверхности вблизи. «Авто» выбирает по объёму видеопамяти.", [UI_MENU_TEX_DETAIL_AUTO] = "Авто: %s", [UI_MENU_TEX_DETAIL_FULL] = "Полная", [UI_MENU_TEX_DETAIL_REDUCED] = "Сниженная", [UI_MENU_TEX_DETAIL_LOW] = "Низкая", + [UI_MENU_VRAM_USE] = "Расход видеопамяти", + [UI_MENU_VRAM_USE_DESC] = "Сколько видеопамяти игра держит про запас. «Низкий», по умолчанию, освобождает текстуры " + "покинутых мест, не использованные три минуты, и то, что создали отключённые " + "масштабирование или сглаживание; в игре такие текстуры запрашивались снова в течение " + "минуты или не запрашивались вовсе. «Высокий» хранит и то и другое всю сессию. Действует " + "после перезапуска.", + [UI_MENU_VRAM_USE_LOW] = "Низкий", [UI_MENU_VRAM_USE_HIGH] = "Высокий", [UI_MENU_SHARPNESS] = "Резкость FSR", [UI_MENU_SHARPNESS_DESC] = "Насколько FSR повышает резкость. 0 отключает.", [UI_MENU_SHARPNESS_NEEDS_FSR] = "только с AMD FSR 1", @@ -1606,6 +1651,15 @@ static const char *const text_pt_br[UI_TEXT_COUNT] = { "mais nítidos que o bilinear. Janela menor (Ocarina desenha 2560 x 1440): a imagem é " "reduzida e o FSR aumenta a nitidez com a intensidade abaixo.", [UI_MENU_UPSCALER_FSR] = "AMD FSR 1", [UI_MENU_UPSCALER_BILINEAR] = "Bilinear", + [UI_MENU_UPSCALING_NATIVE] = "Nativo (100%)", + [UI_MENU_UPSCALING_DESC] = "Como a imagem é suavizada e ajustada à janela. Nativo desenha a cena 3D no tamanho " + "total da imagem, sem upscaler: a mais nítida, e a que mais exige da GPU.", + [UI_MENU_UPSCALING_DESC_FSR] = "Como a imagem é suavizada e ajustada à janela. Nativo desenha a cena 3D no " + "tamanho total da imagem, sem upscaler: a mais nítida, e a que mais exige da GPU.", + [UI_MENU_UPSCALING_DESC_NO_FSR] = "Como a imagem é suavizada e ajustada à janela. Nativo desenha a cena 3D no " + "tamanho total da imagem, sem upscaler: a mais nítida, e a que mais exige da " + "GPU. O AMD FSR 3.1 não é oferecido aqui: ele precisa do amd_fidelityfx_vk.dll, " + "que não pode ser carregado.", [UI_MENU_GAME_ASPECT] = "Tela larga", [UI_MENU_GAME_ASPECT_DESC] = "Amplia a visão. A interface e os menus continuam em 16:9.", [UI_MENU_GAME_ASPECT_16_9] = "16:9 (o do jogo)", [UI_MENU_GAME_ASPECT_WINDOW] = "Formato da janela", @@ -1887,6 +1941,13 @@ static const char *const text_pt_br[UI_TEXT_COUNT] = { "da placa.", [UI_MENU_TEX_DETAIL_AUTO] = "Automático: %s", [UI_MENU_TEX_DETAIL_FULL] = "Completo", [UI_MENU_TEX_DETAIL_REDUCED] = "Reduzido", [UI_MENU_TEX_DETAIL_LOW] = "Baixo", + [UI_MENU_VRAM_USE] = "Uso da memória de vídeo", + [UI_MENU_VRAM_USE_DESC] = "Quanta memória de vídeo o jogo guarda para depois. Baixo, o padrão, libera as texturas " + "dos lugares que você deixou após três minutos sem uso, e o que um escalonador ou " + "antisserrilhamento desligado criou; em jogo, essas texturas foram pedidas de novo em " + "menos de um minuto ou nunca. Alto guarda tudo isso durante a sessão inteira. Vale depois " + "de reiniciar.", + [UI_MENU_VRAM_USE_LOW] = "Baixo", [UI_MENU_VRAM_USE_HIGH] = "Alto", [UI_MENU_COLOR] = "Cor", [UI_MENU_COLOR_PRESET] = "Perfil de cor", [UI_MENU_COLOR_PRESET_DESC] = "Define os quatro valores abaixo. Vívido: mais contraste e saturação, sombras um " @@ -2317,6 +2378,14 @@ static const char *const text_zh_hans[UI_TEXT_COUNT] = { "窗口更小时(Ocarina 绘制 2560 × 1440):画面被缩小,FSR 按下方强度锐化。", [UI_MENU_UPSCALER_FSR] = "AMD FSR 1", [UI_MENU_UPSCALER_BILINEAR] = "双线性", + [UI_MENU_UPSCALING_NATIVE] = "原生(100%)", + [UI_MENU_UPSCALING_DESC] = "画面如何平滑并缩放到窗口。原生:以画面的完整尺寸绘制 3D 场景,不使用超分辨率:" + "最清晰,GPU 负担也最大。", + [UI_MENU_UPSCALING_DESC_FSR] = "画面如何平滑并缩放到窗口。原生:以画面的完整尺寸绘制 3D 场景,不使用超分辨率:" + "最清晰,GPU 负担也最大。", + [UI_MENU_UPSCALING_DESC_NO_FSR] = "画面如何平滑并缩放到窗口。原生:以画面的完整尺寸绘制 3D 场景,不使用超分辨率:" + "最清晰,GPU 负担也最大。此处不提供 AMD FSR 3.1:它需要 " + "amd_fidelityfx_vk.dll,但无法加载。", [UI_MENU_GAME_ASPECT] = "宽屏", [UI_MENU_GAME_ASPECT_DESC] = "扩大视野。界面和菜单保持 16:9。", [UI_MENU_GAME_ASPECT_16_9] = "16:9(游戏原生)", [UI_MENU_GAME_ASPECT_WINDOW] = "窗口比例", @@ -2719,6 +2788,10 @@ static const char *const text_zh_hans[UI_TEXT_COUNT] = { [UI_MENU_TEX_DETAIL_DESC] = "大型表面纹理的清晰度。“降低”和“低”不加载它们最大的版本,使一个区域能放进 6 GB 和 8 GB 显卡的显存;只有近处的表面看起来略微柔和。“自动”根据显卡的显存选择。", [UI_MENU_TEX_DETAIL_AUTO] = "自动:%s", [UI_MENU_TEX_DETAIL_FULL] = "完整", [UI_MENU_TEX_DETAIL_REDUCED] = "降低", [UI_MENU_TEX_DETAIL_LOW] = "低", + [UI_MENU_VRAM_USE] = "显存占用", + [UI_MENU_VRAM_USE_DESC] = "游戏为之后保留多少显存。“低”(默认)会释放已离开地点中三分钟未使用的纹理,以及已关闭的放大或抗锯齿" + "创建的内容;实际游玩中,这类纹理要么在一分钟内再次被使用,要么不再使用。“高”在整个游戏过程中保留两者。重启后生效。", + [UI_MENU_VRAM_USE_LOW] = "低", [UI_MENU_VRAM_USE_HIGH] = "高", [UI_MENU_HDR_GROUP] = "HDR", [UI_MENU_HDR] = "HDR 输出", [UI_MENU_HDR_DESC] = "在 HDR 显示器上呈现更亮的高光。请先在 Windows 中开启 HDR。", diff --git a/runtime/ui_text.h b/runtime/ui_text.h index 049492d9..95029dc7 100644 --- a/runtime/ui_text.h +++ b/runtime/ui_text.h @@ -50,6 +50,7 @@ typedef enum { UI_MENU_TEX_DETAIL, UI_MENU_TEX_DETAIL_DESC, UI_MENU_TEX_DETAIL_AUTO, // "Automatic: %s", %s the level it picks on this card UI_MENU_TEX_DETAIL_FULL, UI_MENU_TEX_DETAIL_REDUCED, UI_MENU_TEX_DETAIL_LOW, + UI_MENU_VRAM_USE, UI_MENU_VRAM_USE_DESC, UI_MENU_VRAM_USE_LOW, UI_MENU_VRAM_USE_HIGH, UI_MENU_SHARPNESS, UI_MENU_SHARPNESS_DESC, UI_MENU_SHARPNESS_NEEDS_FSR, UI_MENU_COLOR, // group heading over the colour rows UI_MENU_COLOR_PRESET, UI_MENU_COLOR_PRESET_DESC, UI_MENU_COLOR_DEFAULT, UI_MENU_COLOR_VIVID, UI_MENU_COLOR_CLARITY, @@ -174,7 +175,10 @@ typedef enum { // Menu tab UI_MENU_MENU_LANG, UI_MENU_MENU_LANG_DESC, UI_MENU_MENU_LANG_AUTO, UI_MENU_DISCORD_RPC, UI_MENU_DISCORD_RPC_DESC, UI_MENU_LOW_END_GPU, UI_MENU_LOW_END_GPU_DESC, - UI_MENU_REPLAY_WORKERS, UI_MENU_REPLAY_WORKERS_DESC, // the Game tab, English only + // the Game tab's Testing rows (ui/menu.cpp tab_game), English only + UI_MENU_REPLAY_WORKERS, UI_MENU_REPLAY_WORKERS_DESC, UI_MENU_REPLAY_PIPE, UI_MENU_REPLAY_PIPE_DESC, + UI_MENU_REPLAY_SMALL, UI_MENU_REPLAY_SMALL_DESC, UI_MENU_TAIL_EARLY, UI_MENU_TAIL_EARLY_DESC, + UI_MENU_NEEDS_WORKERS, UI_MENU_NEEDS_PIPE, // why a row is greyed out // Language names, each in its own language and the same in every row (game language and menu language lists) UI_MENU_LANGNAME_EN, UI_MENU_LANGNAME_JA, UI_MENU_LANGNAME_FR, UI_MENU_LANGNAME_DE, UI_MENU_LANGNAME_IT, @@ -334,6 +338,7 @@ typedef enum { // and on a build where FSR 3.1 does not load (vk_ffx_available): the menu leaves it out there, and these texts say so UI_MENU_UPSCALING_DESC_NO_FSR, UI_MENU_FSR3_OFF, // FSR 3.1 is still set but not loadable: "in use: off" and why + UI_MENU_UPSCALING_NATIVE, // the scene at the picture's own size, no upscaler (ui/menu.cpp share_native) UI_MENU_FRAMEGEN_DESC_DLSS, UI_MENU_FRAMEGEN_NOTE_DLSS, UI_MENU_FRAMEGEN_NEEDS_DLSS, // DLSS only UI_MENU_FRAMEGEN_DESC_NONE, UI_MENU_FRAMEGEN_NEEDS_NONE, // neither runs // DLSS frame generation (ui/menu.cpp row_framegen, row_framegen_mult): the multiplier row, its notes when this GPU diff --git a/runtime/vk.h b/runtime/vk.h index 7b6f0d50..81b980a2 100644 --- a/runtime/vk.h +++ b/runtime/vk.h @@ -8,17 +8,30 @@ #pragma clang diagnostic pop // Every VMA allocation goes by the device-local heaps' live budgets (vk_vma_budget.c): a heap within the reserve of its // budget takes nothing new while another heap can. +// Each allocation is also counted under its owner, by the source file that made it, until it is freed through the +// wrappers below (vk_vma_owner_bytes, for the memory report: vk_memory.c owner_log). VkResult vk_vma_create_image(VmaAllocator vma, const VkImageCreateInfo *ici, const VmaAllocationCreateInfo *ai, VkImage *img, - VmaAllocation *a, VmaAllocationInfo *info); + VmaAllocation *a, VmaAllocationInfo *info, const char *file); VkResult vk_vma_create_buffer(VmaAllocator vma, const VkBufferCreateInfo *bci, const VmaAllocationCreateInfo *ai, - VkBuffer *buf, VmaAllocation *a, VmaAllocationInfo *info); + VkBuffer *buf, VmaAllocation *a, VmaAllocationInfo *info, const char *file); VkResult vk_vma_allocate_memory(VmaAllocator vma, const VkMemoryRequirements *req, const VmaAllocationCreateInfo *ai, - VmaAllocation *a, VmaAllocationInfo *info); + VmaAllocation *a, VmaAllocationInfo *info, const char *file); +void vk_vma_destroy_image(VmaAllocator vma, VkImage img, VmaAllocation a); +void vk_vma_destroy_buffer(VmaAllocator vma, VkBuffer buf, VmaAllocation a); +void vk_vma_free_memory(VmaAllocator vma, VmaAllocation a); uint64_t vk_vma_moved(void); // allocations placed outside a full heap so far +// The owners of VMA allocations (vk_vma_budget.c owner_of says which files make each one's). +enum { VK_OWN_TEXTURES, VK_OWN_POOLS, VK_OWN_UPSCALER, VK_OWN_REPLAY, VK_OWN_USNAP, VK_OWN_GPA, VK_OWN_ASTC, + VK_OWN_PRESENT, VK_OWN_HEADROOM, VK_OWN_OTHER, VK_OWN_N }; +// The bytes of an owner's live allocations on the device-local heaps (local 1) or on the others (local 0). No lock. +uint64_t vk_vma_owner_bytes(int owner, int local); #ifndef VK_VMA_BUDGET_IMPL -#define vmaCreateImage vk_vma_create_image -#define vmaCreateBuffer vk_vma_create_buffer -#define vmaAllocateMemory vk_vma_allocate_memory +#define vmaCreateImage(v, ici, ai, img, a, info) vk_vma_create_image(v, ici, ai, img, a, info, __FILE__) +#define vmaCreateBuffer(v, bci, ai, buf, a, info) vk_vma_create_buffer(v, bci, ai, buf, a, info, __FILE__) +#define vmaAllocateMemory(v, req, ai, a, info) vk_vma_allocate_memory(v, req, ai, a, info, __FILE__) +#define vmaDestroyImage vk_vma_destroy_image +#define vmaDestroyBuffer vk_vma_destroy_buffer +#define vmaFreeMemory vk_vma_free_memory #endif #include "nvn_obj.h" // pool_t, tex_t, view_t, samp_par, NVNcopyRegion (memory and textures block) #include "gfx.h" // GfxShader, GfxShaderRes, GfxShaderAnswers (pipelines block) @@ -393,6 +406,23 @@ int vk_mem_take_live(void); // for the newest such submission up to the one it waits for. enum { VK_RES_EVENT = 1, VK_RES_COUNTER = 2, VK_RES_OQ = 4, VK_RES_WRITE = 8, VK_RES_WRITEBACK = 16, VK_RES_KINDS = 5 }; void vk_exec_note_results(int kind); // VK_RES_* +// The early tail (vk_exec.c "the early tail", NX2NAVI_TAIL_EARLY). vk_exec_tail: the replay is recording a submission's +// tail. vk_sync.c: the tail's command buffer set aside for the write-backs' one (hold), then the part with the +// write-backs handed over with the executed timeline's signal and the tail put behind it (release); whether a part can +// take that order now (ok); the game's wait for a fence (game_wait); for a value whose tail the GPU still runs, about +// how long that takes (tail_pending_ns, else 0). +struct VkExec; +void vk_exec_tail_begin(struct VkExec *ex); +int vk_exec_tail(void); +void vk_exec_tail_ub(uint64_t fence, uint64_t addr, const void *data, uint32_t size); // vk_exec_xfer.c +int vk_sync_tail_ok(void); +VkCommandBuffer vk_sync_tail_hold(void); +VkCommandBuffer vk_sync_tail_release(void); +int vk_sync_game_wait(uint64_t value, uint64_t timeout_ns); +uint64_t vk_sync_tail_pending_ns(uint64_t value); +int vk_mem_live_swap(int v); // vk_memory.c: the note of a late read of guest memory, exchanged +uint64_t vk_mem_gpu_wrote_count(void); // vk_memory.c: GPU writes of pool memory noted so far +int vk_mem_wb_lazy(void); // vk_memory.c: the lazy write-back runs int vk_exec_take_results(void); // the kinds noted since the last take void vk_mem_lazy_part_end(void); // lazy write-back: a part ends (before its prologue), and it is handed over void vk_mem_lazy_handed(void); @@ -688,10 +718,12 @@ VkDescriptorSetLayout vk_empty_set_layout(void); // set 1 of layouts without // image, which it returns for the caller to blit into the target; VK_NULL_HANDLE when it cannot. edge: 1 lets it scale // with the edge-adaptive filter (NX2NAVI_STANDIN_FILTER), 0 keeps the linear one (another upscaler's picture replaces // this one). The caller binds its own compute pipeline again afterwards. -// The tonemap pass's views of img go (vk_defer_view): img is destroyed or replaced. Any thread. -void vk_tonemap_image_gone(VkImage img); VkImage vk_tonemap_record(VkCommandBuffer cb, VkImage src, VkFormat sf, uint32_t sw, uint32_t sh, uint32_t dw, uint32_t dh, int edge); +// A source image of the stand-in goes (vk_exec_draw.c fallback_scratch, or a game image's destroy through +// vk_exec_image_gone): its views go through the deferred destruction, before another image can take its handle. Any +// thread. +void vk_tonemap_forget(VkImage src); // What vk_texture_by_id(id) returns can change only when this does: the slot is registered again, or a texture entry at // its storage address is created, dropped, released or taken back into use. Unlocked; read it before the lookup it // guards (a lookup may create an entry). 0 when the slot was never registered: nothing is kept under 0. @@ -765,10 +797,16 @@ void vk_texture_write_back(uint64_t addr, uint64_t size); // writes those imag void vk_texture_range_gone(uint64_t lo, uint64_t size, VmaAllocation alloc); // vk_pool_fini: drop the images there void vk_texture_stats(uint64_t *images, uint64_t *uploads, uint64_t *reuploads); // Entries now, live and released (TextureFinalize, kept until sweep_released), with the bytes of their own memory -// (images not bound into pool memory). -typedef struct { uint64_t live, released, own_live, own_released, bytes_live, bytes_released; } VkTexCensus; +// (images not bound into pool memory). Of the live bytes: those whose content only the GPU made (render targets and +// what the GPU drew or copied into), and the idle ones no lookup reached lately (vk_texture.c idle, what the VRAM +// pressure may evict); of the released bytes: those released too lately for any bound to drop them (stale_after). +typedef struct { + uint64_t live, released, own_live, own_released, bytes_live, bytes_released; + uint64_t bytes_live_gpu, bytes_live_idle, bytes_released_young; +} VkTexCensus; void vk_texture_census(VkTexCensus *s); void vk_texture_census_log(uint64_t frame); // vk_memory.c vram_log: the live entries' memory by kind and format +void vk_texture_reuse_log(uint64_t frame); // vk_memory.c owner_log: released textures taken back, made again // The holes in VMA's blocks that hold textures (vk_texture.c "compaction", vk_tex_compact.h), for vk_memory.c holes_log. // A VkBlockRow is an allocation of another kind that the caller knows to lie in one of VMA's blocks (a VRAM mirror): // the block's device memory handle and size, the allocation's size and memory type. diff --git a/runtime/vk_aa.c b/runtime/vk_aa.c index bcb95609..bdb13006 100644 --- a/runtime/vk_aa.c +++ b/runtime/vk_aa.c @@ -75,13 +75,13 @@ static struct { int none_told; } A = {.frame_read = ~0ull, .composed = 1}; -static void smaa_idle(int mode); +static void work_idle(uint64_t frame); int vk_aa_mode(void) { uint64_t f = __atomic_load_n(&gfx_counters.frames, __ATOMIC_RELAXED); if (f == A.frame_read) return A.mode; A.frame_read = f; - smaa_idle(A.mode); // the frame before's mode + work_idle(f); const char *v = cfg_value("NX2NAVI_AA"); int want = VK_AA_GAME; for (int i = 1; i < 4; i++) @@ -137,8 +137,24 @@ static struct { Img area, search; Img luma, edges, weights; // at the scene's size (w, h) uint32_t w, h; + uint64_t smaa_frame; // the last frame that recorded SMAA's passes } P; +static void img_free(Img *m); + +// SMAA's work images stayed for the rest of the run once aa left smaa: three pictures of the scene's size, 109 MiB at +// 5120x2142. They go WORK_IDLE presents after the last frame that used them, through the deferred destruction (no +// submission still names them then); smaa again makes them as a change of the scene's size does (work_images), in the +// command buffer of its first frame. A minute, not less: the passes run only where the game runs its own SMAA, so a +// long pause or loading screen with smaa set also counts as unused, and the images are made again after it. +#define WORK_IDLE 3600 +static void work_idle(uint64_t frame) { + if (!P.luma.img || frame - P.smaa_frame < WORK_IDLE || cfg_vram_keep()) return; // vram_use = high keeps them + rt_log("aa: SMAA's work images unused for %d frames: freed (%ux%u)", WORK_IDLE, P.w, P.h); + img_free(&P.luma), img_free(&P.edges), img_free(&P.weights); + P.w = P.h = 0; +} + static int pipes_init(void) { if (P.tried) return P.ok; P.tried = 1; @@ -265,22 +281,6 @@ static int tables_make(void) { return 1; } -// SMAA's work images (35 MiB at a 2560x1440 scene, 79 MiB at 2160 lines) were kept for the process once SMAA had run. -// After SMAA_IDLE_FRAMES frames of another mode they go, once the GPU is past the submission being recorded -// (img_free defers); SMAA chosen again makes them anew. The lookup textures, a few hundred KiB, stay. -#define SMAA_IDLE_FRAMES 600 -static void smaa_idle(int mode) { - static unsigned idle; - if (mode == VK_AA_SMAA || !P.luma.img) { - idle = 0; - return; - } - if (++idle < SMAA_IDLE_FRAMES) return; - img_free(&P.luma), img_free(&P.edges), img_free(&P.weights); - P.w = P.h = 0, idle = 0; - rt_log("aa: SMAA has not run for %d frames: its work images are freed", SMAA_IDLE_FRAMES); -} - // SMAA's work images at the scene's size, made again when it changes static int work_images(VkCommandBuffer cb, uint32_t w, uint32_t h) { if (P.luma.img && P.w == w && P.h == h) return 1; @@ -338,6 +338,7 @@ int vk_aa_record(VkCommandBuffer cb, int mode, VkImageView src, VkImageView dst, pass(cb, P_POST, v, s, &pc, sizeof pc, w, h); return 1; } + P.smaa_frame = __atomic_load_n(&gfx_counters.frames, __ATOMIC_RELAXED); const float metrics[4] = {1.0f / (float)w, 1.0f / (float)h, (float)w, (float)h}; const VkSampler lin[4] = {P.linear, P.linear, P.linear, P.linear}; pass(cb, P_LUMA, (const VkImageView[]){src, P.luma.view}, lin, metrics, sizeof metrics, w, h); diff --git a/runtime/vk_backend.c b/runtime/vk_backend.c index 6b40b6f0..6963c991 100644 --- a/runtime/vk_backend.c +++ b/runtime/vk_backend.c @@ -936,6 +936,7 @@ const GfxBackend gfx_vulkan_backend = { .submit = vk_be_submit, .fence_wait = vk_sync_fence_wait, .exec_wait = vk_be_exec_wait, + .game_wait = vk_sync_game_wait, .fence_done = vk_sync_fence_done, .present = vk_be_present, .frame_start = vk_be_frame_start, diff --git a/runtime/vk_dlaa.c b/runtime/vk_dlaa.c index 5b87907f..6cc0949c 100644 --- a/runtime/vk_dlaa.c +++ b/runtime/vk_dlaa.c @@ -31,7 +31,6 @@ #include "vk_dlaa.h" #include "vk_exec.h" #include "vk_ffx.h" -#include "vk_fg.h" #include "vk_rec.h" #include "dlaa_mv_spv.h" #include "dlaa_sharpen_spv.h" @@ -574,6 +573,15 @@ static struct { uint32_t hl_w, hl_h; VkFormat hl_fmt; int hl_fresh; + // before the bloom (NX2NAVI_DLSS_PREBLOOM, prebloom_run) + int pre_on; // the setting, read at each present + int pre; // this frame: the upscaler ran before the bloom chain; 1 back into the light target + // (no upscaling), 2 to the picture's size (PB), whose passes the retarget moves there + int pre_at; // where the last evaluation ran: 1 before the bloom, 0 at the game's upscaler + int comp; // this frame: the bloom composite was drawn into PB.comp (vk_dlaa_retarget) + VkImage comp_game; // the game's target that composite was meant for + uint32_t pic_w, pic_h; // the size of the game's upscaler's output, the last one seen (vk_dlaa_upscaler) + uint64_t pre_misses; // frames with the setting on that found no bloom chain to run before } D = {.mv_ok = -1, .cfg_seen = ~0u}; // What this frame drew into a D32F depth at least DLAA_SCENE_MIN_W wide before DLSS ran (vk_dlaa_draw), for the line a @@ -1485,6 +1493,7 @@ static struct Snap *mask_pass(VkCommandBuffer cb, VkImageView cv, uint32_t w, ui // ---------------------------------------------------------------- teardown // The HUD-less picture's images, a half-made pair too. The device must be idle, or the images not yet used. +static void pb_destroy(void); // the picture-size images of the upscaler before the bloom (prebloom_run) static void hl_destroy(void) { const VkCtx *c = vk_ctx(); if (D.hl_view) vkDestroyImageView(c->device, D.hl_view, NULL); @@ -1560,6 +1569,7 @@ void vk_dlaa_release(void) { for (int i = 0; i < 5; i++) destroy_size(&D.sizes[i]); for (int i = 0; i < SNAPS; i++) destroy_snap(&R.s[i]); hl_destroy(); + pb_destroy(); if (D.mv_pipe) vkDestroyPipeline(c->device, D.mv_pipe, NULL); if (D.mv_layout) vkDestroyPipelineLayout(c->device, D.mv_layout, NULL); if (D.mv_dsl) vkDestroyDescriptorSetLayout(c->device, D.mv_dsl, NULL); @@ -1615,6 +1625,10 @@ static int jitter_draw(const VkExec *ex) { } void vk_dlaa_viewport(VkExec *ex, VkViewport *v) { + float sx, sy; + // a pass moved to the picture's size (vk_dlaa_retarget); without a game viewport it covers the pass already + if (ex->has_vp && vk_dlaa_target_scale(ex, &sx, &sy)) + v->x *= sx, v->y *= sy, v->width *= sx, v->height *= sy; ex->vp_jittered = jitter_draw(ex); if (!ex->vp_jittered) return; ex->pass_jit = 1; @@ -1793,7 +1807,7 @@ static void window_line(void) { } // dst: the target DLSS's picture goes to, NULL for the light target itself (DLAA in place). -static void hudless(VkCommandBuffer cb, struct Size *e, uint32_t w, uint32_t h); +static void hudless(VkCommandBuffer cb, VkImage src, uint32_t w, uint32_t h); // Another light target than last frame's. The game draws its scene into one of a few pooled targets of the same size // and moves to another whenever its pass list changes, in the middle of play with the camera and the scene carrying @@ -2171,6 +2185,7 @@ static void evaluate(VkExec *ex, const VkExecTarget *dst) { const VkExecTarget *t = &D.color, *z = &D.depth; if (!dst) dst = t; uint32_t w = t->width, h = t->height, ow = dst->width, oh = dst->height; + vk_exec_tail_begin(ex); // NX2NAVI_TAIL_EARLY: from here on the frame's tail, in a command buffer of its own VkCommandBuffer cb = ex->cb; vk_exec_barrier(ex); // what drew the scene is done (no pass is open here) vk_gpu_gap_note(VK_GAP_UPSCALER, (uint64_t)ow * oh); @@ -2181,6 +2196,18 @@ static void evaluate(VkExec *ex, const VkExecTarget *dst) { } struct Size *e = size_for(cb, w, h, ow, oh); if (!e) return dl_fail("work image creation", 0); + // Before the bloom the input is the scene's light before the exposure and the tonemap, at the game's upscaler the + // tonemapped picture: a history built from one does not hold for the other (prebloom_run). + { + const int at = D.pre ? 1 : 0; + static int told; + if (D.evals && at != D.pre_at) { + D.reset = 1; + if (told < 8) + told++, rt_log("dlaa: the upscaler now runs %s: a new history", at ? "before the bloom" : "after the tonemap"); + } + D.pre_at = at; + } // Read once: vk_dlaa_frame runs on the present's thread and can clear fsr_frame while this replay is between the // checks below, which would send the rest of this function to DLSS's path without NGX after all. const int fsr = use_fsr(); @@ -2377,7 +2404,9 @@ evaluated: vkCmdBlitImage(cb, shown, VK_IMAGE_LAYOUT_GENERAL, dst->image, VK_IMAGE_LAYOUT_GENERAL, 1, &b, VK_FILTER_NEAREST); full_barrier(cb); if (dst != t) vk_dlaa_swap_img = t->img, vk_dlaa_swap_view = e->out_view; // the scaling draw reads DLSS's picture - if (vk_dlaa_hudless_wanted) hudless(cb, e, ow, oh); // frame generation keeps the HUD out of its generated frames + // frame generation keeps the HUD out of its generated frames; before the bloom, DLSS's output is not yet the picture + // and the HUD-less one is taken at the game's upscaler (vk_dlaa_upscaler) + if (vk_dlaa_hudless_wanted && !D.pre) hudless(cb, e->out, ow, oh); ex->recorded += 4; vk_exec_draw_new_cb(); // NGX bound pipelines and descriptor sets of its own: the next draw binds everything again D.jdepth = z->image, D.jw = z->width, D.jh = z->height; @@ -2412,7 +2441,9 @@ evaluated: // (vk_hdr_encode): DLSS's output is kept as RGBA16F for the same encode. The game's own HDR (native) encodes inside the // game, which this cannot follow: no HUD-less picture there. int vk_dlaa_hudless_wanted; -static void hudless(VkCommandBuffer cb, struct Size *e, uint32_t w, uint32_t h) { +// src: the picture without the HUD, linear, w x h, in GENERAL: DLSS's output where the upscaler ran at the game's +// upscaler, the game's upscaler's output where it ran before the bloom (vk_dlaa_upscaler). +static void hudless(VkCommandBuffer cb, VkImage src, uint32_t w, uint32_t h) { const VkCtx *c = vk_ctx(); int hdr = vk_hdr_wanted() && !vk_hdr_native(); if (vk_hdr_native()) return; @@ -2452,7 +2483,7 @@ static void hudless(VkCommandBuffer cb, struct Size *e, uint32_t w, uint32_t h) } VkImageBlit b = {.srcSubresource = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1}, .srcOffsets = {{0, 0, 0}, {(int32_t)w, (int32_t)h, 1}}, .dstSubresource = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1}, .dstOffsets = {{0, 0, 0}, {(int32_t)w, (int32_t)h, 1}}}; - vkCmdBlitImage(cb, e->out, VK_IMAGE_LAYOUT_GENERAL, hdr ? D.hl : D.hl_srgb, VK_IMAGE_LAYOUT_GENERAL, 1, &b, + vkCmdBlitImage(cb, src, VK_IMAGE_LAYOUT_GENERAL, hdr ? D.hl : D.hl_srgb, VK_IMAGE_LAYOUT_GENERAL, 1, &b, VK_FILTER_NEAREST); full_barrier(cb); if (hdr) { @@ -2493,6 +2524,227 @@ static int output_target(VkFormat f) { return srgb8(f) || vk_hdr_native_picture( double wide_dlss_share(void); // widescreen.c: the DLSS mode's share of the picture (1 DLAA), 0 off +// ---------------------------------------------------------------- before the bloom (NX2NAVI_DLSS_PREBLOOM) +// What the upscaler gets at the game's upscaler (vk_dlaa_upscaler) is the armed pair of the bloom composite: measured on +// frame dumps (NX2NAVI_DUMP; 2880x1620 scene in a 3840x2160 picture, and 1280x720 in 2560x1440 with DLSS), the light +// target holds the lit scene with its fog, transparents and lens flares as linear light up to about 160; the one +// full-screen draw that reads it and the bloom's smallest target writes the second light target with the scene depth +// attached, and that shader adds the bloom, applies the exposure, converts to the tonemap's colour space and ends in the +// fitted filmic curve and a 0..1 clamp (vk_hdr.c tonemap_splice finds the same curve). Its output, 0..1 and linear +// (the post pass after it stores its square root), is what DLSS anti-aliased and enlarged: a picture already exposed, +// tonemapped and bloomed, which the IsHDR and auto-exposure flags describe wrongly. +// With the setting on, the upscaler runs at the start of the bloom chain's first pass instead: two light-format targets +// smaller than the scene, no depth, right after the transparents (prebloom_start). Its input is then the scene's linear +// light before exposure and tonemap, at the render size, with the scene depth. +// - Without upscaling (DLAA) DLSS's picture goes back into the light target and the game's own passes (bloom, flares, +// composite and tonemap, post pass, its upscaler at 1:1, HUD) all run on it as they are. +// - With upscaling the game's passes after it are drawn at the render size, and the post pass and the game's upscaler +// are compute dispatches whose sizes the game sets, so they cannot be moved to the picture's size. What is moved +// (vk_dlaa_retarget) are the render passes from the light target on: DLSS's picture goes into PB.scene at the +// picture's size; a pass drawing into the light target (the lens flares, additive, no depth test) draws into +// PB.scene, and the bloom composite into PB.comp, both at the picture's size without the depth attachment, with +// their viewports and scissors scaled (vk_dlaa_viewport, vk_dlaa_target_scale). The composite samples the light +// target through its texture coordinates, so it reads PB.scene in its place (vk_dlaa_swap_img). The bloom chain +// reads the light target at its own small sizes; DLSS's picture is scaled back into it first, so the bloom starts +// from the resolved picture and does not shimmer with the jitter. The post pass and the game's upscaler then run on +// the game's composite target, which this frame never wrote, and at the game's upscaler PB.comp goes over their +// picture, as DLSS's did before (vk_dlaa_upscaler). +// A frame without that bloom chain (menus, a loading screen) runs the upscaler at the game's upscaler as before. +static struct { + VkImage scene, comp; // DLSS's picture with the flares, and the composite's picture: B10G11R11, picture size + VmaAllocation scene_mem, comp_mem; + VkImageView scene_view, comp_view; + uint32_t w, h; + uint64_t used; // D.presents when last used (idle_release frees them after IDLE_FREE_PRESENTS) +} PB; + +static void pb_destroy(void) { + const VkCtx *c = vk_ctx(); + if (PB.scene_view) vkDestroyImageView(c->device, PB.scene_view, NULL); + if (PB.comp_view) vkDestroyImageView(c->device, PB.comp_view, NULL); + if (PB.scene) vmaDestroyImage(c->vma, PB.scene, PB.scene_mem); + if (PB.comp) vmaDestroyImage(c->vma, PB.comp, PB.comp_mem); + memset(&PB, 0, sizeof PB); +} + +// The two images at w x h, made again for another picture size (the old ones after the device is idle, as hudless). +static int pb_images(VkCommandBuffer cb, uint32_t w, uint32_t h) { + PB.used = D.presents; + if (PB.scene && PB.w == w && PB.h == h) return 1; + if (PB.scene || PB.comp) { + vk_queue_lock(), vkDeviceWaitIdle(vk_ctx()->device), vk_queue_unlock(); + pb_destroy(); + } + VkImageUsageFlags u = VK_IMAGE_USAGE_COLOR_ATTACHMENT_BIT | VK_IMAGE_USAGE_SAMPLED_BIT | + VK_IMAGE_USAGE_TRANSFER_SRC_BIT | VK_IMAGE_USAGE_TRANSFER_DST_BIT; + const VkFormat f = VK_FORMAT_B10G11R11_UFLOAT_PACK32; + if (!image2d(w, h, f, u, &PB.scene, &PB.scene_mem, &PB.scene_view) || + !image2d(w, h, f, u, &PB.comp, &PB.comp_mem, &PB.comp_view)) { + pb_destroy(); + return 0; + } + VkImageMemoryBarrier2 ib[2]; + VkImage im[2] = {PB.scene, PB.comp}; + for (int i = 0; i < 2; i++) + ib[i] = (VkImageMemoryBarrier2){.sType = VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER_2, + .srcStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + .dstStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + .dstAccessMask = VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT, + .oldLayout = VK_IMAGE_LAYOUT_UNDEFINED, .newLayout = VK_IMAGE_LAYOUT_GENERAL, + .srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, + .dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, .image = im[i], + .subresourceRange = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 1, 0, 1}}; + VkDependencyInfo di = {.sType = VK_STRUCTURE_TYPE_DEPENDENCY_INFO, .imageMemoryBarrierCount = 2, + .pImageMemoryBarriers = ib}; + vkCmdPipelineBarrier2(cb, &di); + PB.w = w, PB.h = h; + rt_log("dlaa: prebloom: the passes after the upscaler draw at %ux%u (two B10G11R11 images, %.0f MiB)", w, h, + 2.0 * w * h * 4 / 1048576.0); + return 1; +} + +// The bloom chain's first pass, on Ocarina's frame: two light-format targets, each smaller than the armed light target +// both ways, no depth. In the dumps (2880x1620 and 1280x720 scenes) it is the only such pass and it comes right after +// the last pass of transparents into the light target; its targets are 480x270 at both scene sizes. The passes in +// between that read the light target at a third or half of its size have one target, or a light target with an R8 one. +static int prebloom_start(const VkExec *ex) { + if (ex->pass_ncolor != 2 || ex->depth.image) return 0; + for (int i = 0; i < 2; i++) { + const VkExecTarget *t = &ex->color[i]; + if (!t->image || t->format != VK_FORMAT_B10G11R11_UFLOAT_PACK32 || t->samples > 1 || t->is3d || + t->width >= D.color.width || t->height >= D.color.height) + return 0; + } + return 1; +} + +// Runs the upscaler at the bloom chain's first pass (prebloom_start), before that pass begins. 0: not here this frame +// (the picture's size not known yet, or not of the scene's shape): the game's upscaler runs it as before. +static int prebloom_run(VkExec *ex) { + const uint32_t w = D.color.width, h = D.color.height; + const double share = wide_dlss_share(); + const int native = share <= 0 || share >= 1; + uint32_t ow = D.pic_w, oh = D.pic_h; + if (!native) { + int known = ow > w && oh > h && (double)ow * h <= 1.03 * (double)oh * w && (double)oh * w <= 1.03 * (double)ow * h; + if (!known || !pb_images(ex->cb, ow, oh)) { + if (D.pre_misses++ < 4) + rt_log("dlaa: prebloom: the picture's size is not known yet (last %ux%u for a %ux%u scene): this frame " + "runs the upscaler after the tonemap", ow, oh, w, h); + return 0; + } + } + vk_exec_hook_turn(); // as in vk_dlaa_pass: the replay workers' draws read the jitter and its end + D.ended = D.done = 1; + D.pre = native ? 1 : 2; + const uint64_t evals = D.evals; + if (native) { + evaluate(ex, NULL); // into the light target, which the bloom and everything after it read + } else { + D.up = 1; + VkExecTarget dst = {.image = PB.scene, .format = VK_FORMAT_B10G11R11_UFLOAT_PACK32, + .aspect = VK_IMAGE_ASPECT_COLOR_BIT, .layers = 1, .width = ow, .height = oh, .samples = 1}; + evaluate(ex, &dst); + } + if (D.evals == evals) { // the evaluation failed (dl_fail) or did not run: nothing moves to the picture's size + D.pre = 0, D.up = 0; + return 1; + } + if (D.pre == 2) { + // the composite samples the light target: PB.scene, which the lens flares are then drawn into, in its place + vk_dlaa_swap_img = D.color.img, vk_dlaa_swap_view = PB.scene_view; + // and the bloom chain reads the light target itself, at its own sizes: the resolved picture, scaled back down + VkImageBlit b = {.srcSubresource = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1}, + .srcOffsets = {{0, 0, 0}, {(int32_t)ow, (int32_t)oh, 1}}, + .dstSubresource = {VK_IMAGE_ASPECT_COLOR_BIT, D.color.level, D.color.base_layer, 1}, + .dstOffsets = {{0, 0, 0}, {(int32_t)w, (int32_t)h, 1}}}; + vkCmdBlitImage(ex->cb, PB.scene, VK_IMAGE_LAYOUT_GENERAL, D.color.image, VK_IMAGE_LAYOUT_GENERAL, 1, &b, + VK_FILTER_LINEAR); + full_barrier(ex->cb); + ex->recorded += 2; + } + static int told; + if (!told++) + rt_log("dlaa: prebloom: %s runs before the bloom, at the start of its first pass (two %ux%u targets, format %d), " + "on the light target %ux%u format %d with the depth %ux%u format %d; %s", + use_fsr() ? "FSR 3.1" : "DLSS", ex->color[0].width, ex->color[0].height, (int)ex->color[0].format, w, h, + (int)D.color.format, D.depth.width, D.depth.height, (int)D.depth.format, + native ? "its picture goes back into the light target (no upscaling)" + : "its picture is the picture's size, where the lens flares and the bloom composite are drawn"); + return 1; +} + +// vk_exec.c vk_exec_pass_begin, in every replay thread, once the pass's targets are resolved and the pass hooks ran: +// after the upscaler ran before the bloom to the picture's size (prebloom_run), a pass drawing into the light target +// draws into PB.scene and the bloom composite (one light-format target of the scene's size, with the scene depth) into +// PB.comp, each at the picture's size and without its depth attachment (neither tests depth in the dumps). Its draws' +// viewports and scissors follow (vk_dlaa_viewport, vk_dlaa_target_scale). A pass of the same targets begun again (the +// pass broken by a copy) keeps what it got: its targets are not resolved again. +void vk_dlaa_retarget(VkExec *ex) { + if (D.pre != 2 || !PB.scene || ex->pass_ncolor != 1) return; + VkExecTarget *c = &ex->color[0]; + if (!c->image || c->image == PB.scene || c->image == PB.comp) return; + if (c->format != VK_FORMAT_B10G11R11_UFLOAT_PACK32 || c->samples > 1 || c->is3d || c->layers != 1) return; + VkImage to = VK_NULL_HANDLE; + int comp = 0; + if (c->image == D.color.image) to = PB.scene; + else if (c->width == D.color.width && c->height == D.color.height && ex->depth.image == D.depth.image && + (!D.comp_game || D.comp_game == c->image)) + to = PB.comp, comp = 1; + if (!to) return; + static int told[2]; + if (!told[comp]++) + rt_log("dlaa: prebloom: %s (%ux%u format %d, %s) drawn at %ux%u", comp ? "the bloom composite" : "a pass into the light target", + c->width, c->height, (int)c->format, ex->depth.image ? "with the scene depth" : "no depth", PB.w, PB.h); + if (comp) D.comp_game = c->image, D.comp = 1; + c->image = to, c->img = NULL; + c->width = PB.w, c->height = PB.h; + c->level = c->base_layer = 0, c->scale = 1, c->blocks = 0, c->view_type = VK_IMAGE_VIEW_TYPE_2D; + memset(&ex->depth, 0, sizeof ex->depth); + ex->pass_depth = ex->pass_stencil = VK_FORMAT_UNDEFINED; + ex->pipe.depth_fmt = 0; + ex->pass_w = PB.w, ex->pass_h = PB.h, ex->pass_scale = 1, ex->pass_layers = 1, ex->pass_noatt = 0; +} + +// The scale from the game's viewport and scissor to a pass vk_dlaa_retarget moved to the picture's size; 0 for any +// other pass. +int vk_dlaa_target_scale(const VkExec *ex, float *sx, float *sy) { + if (D.pre != 2 || !PB.scene || !D.color.width || !D.color.height) return 0; + VkImage t = ex->color[0].image; + if (!t || (t != PB.scene && t != PB.comp)) return 0; + *sx = (float)PB.w / (float)D.color.width, *sy = (float)PB.h / (float)D.color.height; + return 1; +} + +// At the game's upscaler, with the upscaler run before the bloom: notes the picture's size, and puts the composite +// drawn at the picture's size over the game's upscaler's output (2), or leaves that output, which is already made from +// DLSS's picture (1). The HUD-less picture for frame generation is the result either way. +static void prebloom_upscaler(VkExec *ex, VkImage out, uint32_t w, uint32_t h) { + VkCommandBuffer cb = ex->cb; + if (D.pre == 2) { + if (!D.comp) { + static int told; + if (told < 8) + told++, rt_log("dlaa: prebloom: frame %llu drew no bloom composite after the upscaler: the game's " + "picture stays", (unsigned long long)__atomic_load_n(&gfx_counters.frames, __ATOMIC_RELAXED)); + return; + } + // the same size but for a picture size changed since the upscaler ran: scaled then, for that frame + VkImageBlit b = {.srcSubresource = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1}, + .srcOffsets = {{0, 0, 0}, {(int32_t)PB.w, (int32_t)PB.h, 1}}, + .dstSubresource = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 1}, + .dstOffsets = {{0, 0, 0}, {(int32_t)w, (int32_t)h, 1}}}; + vkCmdBlitImage(cb, PB.comp, VK_IMAGE_LAYOUT_GENERAL, out, VK_IMAGE_LAYOUT_GENERAL, 1, &b, + PB.w == w && PB.h == h ? VK_FILTER_NEAREST : VK_FILTER_LINEAR); + full_barrier(cb); + ex->recorded += 2; + static int told; + if (!told++) rt_log("dlaa: prebloom: the composite drawn at %ux%u goes into the game's upscaler's %ux%u output", + PB.w, PB.h, w, h); + } + if (vk_dlaa_hudless_wanted) hudless(cb, out, w, h), ex->recorded += 2; +} + // Ocarina's own upscaler is the place for DLSS or FSR: its stand-in (vk_exec_draw.c untranslated_dispatch_fallback) has // just scaled the scene into the image the rest of the frame shows, and the upscaler's picture goes over it there. // Until this hook the upscaler ran at the pass after it (vk_dlaa_pass below), which was taken for the pass the HUD is @@ -2510,7 +2762,13 @@ double wide_dlss_share(void); // widescreen.c: the DLSS mode's share of the pi void vk_dlaa_upscaler(VkExec *ex, VkImage out, VkFormat fmt, uint32_t w, uint32_t h) { static int on = -1; if (on < 0) on = rt_knob("DLAA_AT_UPSCALER", "1"); - if (!on || !dl_active() || !D.armed || D.done || D.gbuf || !out || fmt != VK_FORMAT_B10G11R11_UFLOAT_PACK32) return; + if (!dl_active() || !out || fmt != VK_FORMAT_B10G11R11_UFLOAT_PACK32) return; + if (D.armed && !D.gbuf && w >= D.color.width && h >= D.color.height) D.pic_w = w, D.pic_h = h; // prebloom_run + if (D.pre) { + prebloom_upscaler(ex, out, w, h); + return; + } + if (!on || !D.armed || D.done || D.gbuf) return; uint32_t sw = D.color.width, sh = D.color.height; if (w < sw || h < sh || out == D.color.image) return; if ((double)w * sh > 1.03 * (double)h * sw || (double)h * sw > 1.03 * (double)w * sh) return; // not the scene's shape @@ -2524,56 +2782,47 @@ void vk_dlaa_upscaler(VkExec *ex, VkImage out, VkFormat fmt, uint32_t w, uint32_ evaluate(ex, &dst); } -// ---- what the upscaler holds after it is switched off -// Switched off in the menu, or failed (dl_fail), the upscaler kept everything it had made for the process: per scene -// size its output, motion vector, mask and sharpening images and a DLSS feature (at 2560x1440 with DLAA 81 MiB of -// images, at 2160 lines 182 MiB, and a DLSS feature is about 0.5 GB), the snapshots of the reactive mask, the HUD-less -// pair for frame generation (28 MiB, 63 MiB at 2160 lines) and the FSR 3.1 and FSR 4 contexts. RETIRE_PRESENTS presents -// after the upscaler goes off they are freed, here on the replay thread, which makes every evaluation and so cannot be -// inside one, after a wait for the device to be idle, as free_size does for a size it replaces: an evaluation may -// still run on the GPU, and the present thread only learns of the switch at its next frame. The HUD-less pair goes the -// same way while the upscaler runs and frame generation no longer asks for it (vk_dlaa_hudless_wanted, which the frame -// generator's release clears), once no frame generation swapchain is up that could still read it. Switched on again, -// the upscaler makes what it needs as at its first frame, with a new history, as any switch on does. -#define RETIRE_PRESENTS 8 -static uint64_t retire_off_at, retire_hl_at; // replay: D.presents when the upscaler, or the HUD-less picture, went unused -static uint64_t retired_n; // replay: times the upscaler's memory was given back +// ---- what idle_release (below) leaves after the upscaler is switched off +// idle_release frees the DLSS sizes and the HUD-less picture some presents after the setting goes off. What it does not +// reach stayed for the process: the reactive mask's snapshots, the FSR 3.1 and FSR 4 upscale contexts, and the DLSS +// sizes after a failure (dl_fail turns the upscaler off without the setting changing, so D.on stays set). They go +// here, RETIRE_PRESENTS presents after the upscaler stopped running, on the replay thread, which makes every +// evaluation, after a wait for the device to be idle, as free_size does for a size it replaces. Switched on again, the +// upscaler makes them anew, as at its first frame in a run. +#define RETIRE_PRESENTS 600 +static uint64_t retire_off_at; // replay: D.presents when the upscaler stopped running +static uint64_t retired_n; // replay: times this gave memory back static int up_holds(void) { - for (int i = 0; i < DLSS_SIZES; i++) - if (D.sizes[i].out || D.sizes[i].feat) return 1; for (int i = 0; i < SNAPS; i++) if (R.s[i].before) return 1; - return D.hl != VK_NULL_HANDLE || vk_ffx_holds(); + if (D.failed) + for (int i = 0; i < DLSS_SIZES; i++) + if (D.sizes[i].out || D.sizes[i].feat) return 1; + return vk_ffx_holds(); } static void retire_idle(void) { uint64_t now = __atomic_load_n(&D.presents, __ATOMIC_RELAXED); - int off = !dl_active(), hl_unused = D.hl && !vk_dlaa_hudless_wanted && !vk_fg_active(); - if (!off) retire_off_at = 0; - if (!hl_unused) retire_hl_at = 0; - if (off && !retire_off_at) retire_off_at = now ? now : 1; - if (hl_unused && !retire_hl_at) retire_hl_at = now ? now : 1; - int all = off && now >= retire_off_at + RETIRE_PRESENTS && up_holds(); - int hl = hl_unused && now >= retire_hl_at + RETIRE_PRESENTS; - if (!all && !hl) return; - vk_queue_lock(), vkDeviceWaitIdle(vk_ctx()->device), vk_queue_unlock(); - if (hl || !vk_fg_active()) hl_destroy(), D.hl_fresh = 0; - if (all) { - int64_t was = vk_dlaa_vram(); - for (int i = 0; i < DLSS_SIZES; i++) destroy_size(&D.sizes[i]); - for (int i = 0; i < SNAPS; i++) destroy_snap(&R.s[i]); - vk_ffx_release_idle(); - D.last = NULL, D.host = NULL; - retired_n++; - rt_log("dlaa: the upscaler is off (%s) since %d presents: its images, DLSS features and FSR contexts are freed " - "(%llu time(s); DLSS reported %.0f MiB of its own); made again when it is switched on", - D.failed ? "it failed" : "the setting", RETIRE_PRESENTS, (unsigned long long)retired_n, - was > 0 ? was / 1048576.0 : 0.0); - } else { - rt_log("dlaa: frame generation no longer asks for the HUD-less picture: its images are freed"); + if (!idle_free() || cfg_vram_keep()) return; // as idle_release: vram_use = high keeps them + if (dl_active()) { + retire_off_at = 0; + return; } - retire_off_at = retire_hl_at = 0; + if (!retire_off_at) retire_off_at = now ? now : 1; + if (now < retire_off_at + RETIRE_PRESENTS || !up_holds()) return; + vk_queue_lock(), vkDeviceWaitIdle(vk_ctx()->device), vk_queue_unlock(); + for (int i = 0; i < SNAPS; i++) destroy_snap(&R.s[i]); + vk_ffx_release_idle(); + if (D.failed) { // as idle_release does for a size: frame generation is handed nothing of one from here on + for (int i = 0; i < DLSS_SIZES; i++) destroy_size(&D.sizes[i]); + D.last = NULL, D.host = NULL, D.fg_fresh = 0, D.fg_mv_view = VK_NULL_HANDLE, D.fg_mv = (VkFfxImage){0}; + } + retired_n++; + rt_log("dlaa: the upscaler has not run for %d presents (%s): its mask snapshots and FSR contexts%s are freed (%llu " + "time(s)); made again when it runs", RETIRE_PRESENTS, D.failed ? "it failed" : "switched off", + D.failed ? " and DLSS sizes" : "", (unsigned long long)retired_n); + retire_off_at = 0; } void vk_dlaa_pass(VkExec *ex) { @@ -2581,7 +2830,8 @@ void vk_dlaa_pass(VkExec *ex) { retire_idle(); if (!dl_active()) return; if (D.done) { // diagnostics: another scene drawn into a light target in the same present interval - if (ex->pass_ncolor == 1 && c->image && scene_color(c->format) && z->image && z->format == VK_FORMAT_D32_SFLOAT && + // (not before the bloom: the flares and the composite after it are of that shape, and expected) + if (!D.pre && ex->pass_ncolor == 1 && c->image && scene_color(c->format) && z->image && z->format == VK_FORMAT_D32_SFLOAT && c->width == D.color.width && c->height == D.color.height && z->width == c->width) F.second++, F.second_img = c->image; return; @@ -2628,6 +2878,8 @@ void vk_dlaa_pass(VkExec *ex) { } if (!D.armed) return; if (!deferred) { // Ocarina's shape: the output pass ends the scene and runs DLSS + // NX2NAVI_DLSS_PREBLOOM: before the bloom chain's first pass instead, where there is one (prebloom_run) + if (D.pre_on && prebloom_start(ex) && prebloom_run(ex)) return; int one = ex->pass_ncolor == 1 && c->image && !z->image && c->image != D.color.image && c->samples <= 1 && !c->is3d; if (one) F.one_n++, F.one_w = c->width, F.one_h = c->height, F.one_fmt = c->format; @@ -2749,6 +3001,39 @@ int vk_dlaa_resized_take(void) { return 1; } +// What DLSS holds while nothing uses it (NX2NAVI_DLSS_IDLE_FREE 1 or 2, the default 1). With the upscaler switched off +// in the menu, the sizes kept their images and features for the rest of the run, since only an evaluation frees a size +// (stale_sizes): at a 5120x2142 output that is the output, sharpening, mask, class and distance images, about 0.4 GiB, +// and about 0.5 GiB per feature. With frame generation off and DLSS on, the HUD-less picture (hudless) stayed the same +// way. After IDLE_FREE_PRESENTS presents without their use they go here, on the replaying thread that makes every +// other NGX call, without waiting for the GPU: the last evaluation that used them is that many presents back, and the +// CPU is never more than a few ahead. A feature made later is made with none alive, the case release_kept leaves +// clean. What it costs: switching DLSS on again more than IDLE_FREE_PRESENTS presents after switching it off makes +// the size's images and feature again, which its first evaluation waits for, as at the first frame DLSS ran in a run. +#define IDLE_FREE_PRESENTS 600 +static uint64_t off_at; // D.presents when DLSS went off +static void idle_release(void) { + if (!idle_free() || cfg_vram_keep()) return; // vram_use = high keeps them + if (D.hl && D.presents - D.hl_at >= IDLE_FREE_PRESENTS && !D.hl_fresh) { + rt_log("dlaa: the HUD-less picture for frame generation went unused for %d presents: freed", IDLE_FREE_PRESENTS); + hl_destroy(); + } + if (PB.scene && D.presents - PB.used >= IDLE_FREE_PRESENTS) { // prebloom off, or no upscaling, for that long + rt_log("dlaa: prebloom: the picture-size images went unused for %d presents: freed", IDLE_FREE_PRESENTS); + pb_destroy(); + } + if (D.on || D.presents - off_at < IDLE_FREE_PRESENTS) return; + int n = 0; + for (int i = 0; i < DLSS_SIZES; i++) + if (D.sizes[i].out || D.sizes[i].feat) { + rt_log("dlaa: the DLSS size %ux%u -> %ux%u went unused for %d presents with the upscaler off: freed", + D.sizes[i].w, D.sizes[i].h, D.sizes[i].ow, D.sizes[i].oh, IDLE_FREE_PRESENTS); + destroy_size(&D.sizes[i]), n++; + } + if (n) // frame generation's motion vectors were a size's (evaluate); none is handed out this late (fg_current) + D.last = NULL, D.host = NULL, D.fg_fresh = 0, D.fg_mv_view = VK_NULL_HANDLE, D.fg_mv = (VkFfxImage){0}; +} + void vk_dlaa_frame(void) { static unsigned cfg_seen; static int cfg_known; @@ -2758,6 +3043,11 @@ void vk_dlaa_frame(void) { else if (setting_presents) setting_presents--; cfg_seen = v, cfg_known = 1; vk_dlaa_swap_img = NULL, vk_dlaa_swap_view = VK_NULL_HANDLE; + D.pre = D.comp = 0, D.comp_game = VK_NULL_HANDLE; // the frame before was recorded up to its present + int pre_on = cfg_bool("NX2NAVI_DLSS_PREBLOOM"); // live: the evaluation point changes at the next frame + if (pre_on != D.pre_on) rt_log("dlaa: prebloom %s", pre_on ? "on: the upscaler runs before the bloom where the frame has one" + : "off: the upscaler runs at the game's upscaler"); + D.pre_on = pre_on; dlss_mode(); preset_check(); if (!N.ngx_up) { // FSR 3.1 follows the setting @@ -2768,6 +3058,7 @@ void vk_dlaa_frame(void) { if (on != D.on) { rt_log("dlaa: %s", on ? "on" : "off"); D.on = on; + off_at = D.presents; // either way the next DLAA frame starts a new history from an unjittered frame D.reset = 1, D.cam_prev_ok = 0, D.jit_ok = D.jit_used = D.armed = D.done = D.gbuf = D.ended = D.up = 0; D.hold = 0; @@ -2775,6 +3066,7 @@ void vk_dlaa_frame(void) { memset(&F, 0, sizeof F); return; } + idle_release(); if (!dl_active()) return; if (D.share_off) D.share_off--; if (!D.done && D.armed) D.share_off = 120; // vk_dlaa_pass: the output pass without the mode's share for a while diff --git a/runtime/vk_dlaa.h b/runtime/vk_dlaa.h index cecbba18..5380b257 100644 --- a/runtime/vk_dlaa.h +++ b/runtime/vk_dlaa.h @@ -65,6 +65,13 @@ int64_t vk_dlaa_vram(void); // frame (vk_dlaa_frame). typedef struct VkExec VkExec; void vk_dlaa_pass(VkExec *ex); +// vk_exec.c vk_exec_pass_begin, in every replay thread, after the pass hooks: with the upscaler run before the bloom to +// the picture's size (NX2NAVI_DLSS_PREBLOOM), the passes after it that draw into the light target or the bloom +// composite's target are moved to images of the picture's size (vk_dlaa.c "before the bloom"). +void vk_dlaa_retarget(VkExec *ex); +// The scale from the game's viewport and scissor to such a moved pass; 0 for any other pass (vk_exec.c vk_exec_scissor, +// vk_dlaa_viewport). +int vk_dlaa_target_scale(const VkExec *ex, float *sx, float *sy); // vk_exec_draw.c untranslated_dispatch_fallback, after the stand-in of the game's own upscaler wrote its output image // (in GENERAL, of format fmt and w x h): DLSS or FSR runs there on the armed scene and its picture goes over it. void vk_dlaa_upscaler(VkExec *ex, VkImage out, VkFormat fmt, uint32_t w, uint32_t h); diff --git a/runtime/vk_exec.c b/runtime/vk_exec.c index a38050b0..5ac29395 100644 --- a/runtime/vk_exec.c +++ b/runtime/vk_exec.c @@ -817,6 +817,7 @@ void vk_exec_pass_begin(VkExec *ex) { vk_contact_pass_begin(ex); vk_dlaa_pass(ex); // DLAA runs before the bloom chain's first pass (vk_dlaa.c) } + vk_dlaa_retarget(ex); // after DLSS ran before the bloom: the passes it moves to the picture's size (vk_dlaa.c) // Learned targets (VkTexImage.fb_learned) are attached in the feedback layout VkImage want[9]; VkImageAspectFlags wasp[9]; @@ -967,6 +968,12 @@ VkRect2D vk_exec_scissor(const VkExec *ex) { int64_t k = ex->pass_scale > 1 ? ex->pass_scale : 1; int64_t x0 = ex->scissor.x * k, y0 = ex->scissor.y * k; int64_t x1 = x0 + (int64_t)ex->scissor.w * k, y1 = y0 + (int64_t)ex->scissor.h * k; + float sx, sy; + if (vk_dlaa_target_scale(ex, &sx, &sy)) { // a pass DLSS moved to the picture's size (vk_dlaa.c vk_dlaa_retarget) + x0 = (int64_t)floor(ex->scissor.x * (double)sx), y0 = (int64_t)floor(ex->scissor.y * (double)sy); + x1 = (int64_t)ceil(((double)ex->scissor.x + ex->scissor.w) * sx); + y1 = (int64_t)ceil(((double)ex->scissor.y + ex->scissor.h) * sy); + } if (x0 < 0) x0 = 0; if (y0 < 0) y0 = 0; if (x1 > w) x1 = w; @@ -1122,6 +1129,59 @@ static int split_n = -1, split_idle, split_in_pass, split_draws, split_at, part_ static uint64_t split_ns, part_t0; static uint64_t n_parts, n_subs, n_idle_parts, n_pass_splits, n_cuts; +// ---- the early tail (NX2NAVI_TAIL_EARLY, default 0) +// Standing in Hyrule Field with FSR 4 (RX 7900 XT, 2026-11-13) a frame's main submission ends with one part of 4.7 ms +// of GPU time, the temporal upscaler and the interface drawn after it, and one of 1.4 ms, the write-backs of what the +// shaders wrote into VRAM mirrors. The game waits for all of it before it writes the next frame's data (the wait after +// a frame's first submission, nvn_api_core.c, and its own fence waits), about 7 ms in which the replay thread has +// nothing to record. What the game reads of the submission is done before the upscaler runs: the occlusion and +// counter reports, the event values, and the write-backs' bytes, which only came last. After it the stream holds two +// timestamp reports, and inline uniform updates for the interface's draws (64 bytes into one mirrored block, some +// thirty times), and nothing else the CPU sees. +// So from the upscaler on (vk_dlaa.c evaluate) the replay records the tail into a command buffer of its own +// (vk_sync_cut), without splits or parallel groups. An inline uniform update of a mirrored block in it takes no +// write-back: the GPU copies the bytes into the mirror in stream order as ever, and guest memory gets them from the +// CPU with the submission's results (vk_exec_draw.c op_update_ub, vk_exec_xfer.c vk_exec_tail_ub). At the submission's +// end, when the tail turned out to hand the CPU nothing but timestamps (vk_exec_tail_results), to write no other pool +// memory (vk_mem_gpu_wrote_count) and to bind no memory the GPU reads from the game's own pages (vk_mem_live_swap: a +// pool without a mirror, a device address table), +// the write-backs are recorded into a command buffer that goes to the queue before the tail's, with the executed +// timeline's signal (vk_sync_tail_hold, vk_sync_tail_release): the GPU runs the same commands, the write-backs before +// the tail. The mirror copies the tail's first uses queued go with that part too, so they read the game's pages before +// the signal. The game's waits then end at that signal and the results the game reads (vk_sync.c vk_sync_exec_wait, +// vk_sync_game_wait), and the GPU runs the tail while the game and the replay are at the next frame. A timestamp +// reported in the tail is written then with the time the tail is expected to end at (vk_exec_xfer.c counter_cb), as +// the game measures the frame's GPU time with it. A tail that fails a test goes as before, the write-backs after it. +// Measured there, 150 s runs, each pair in one batch: 63.5 against 49.4 fps with three replay workers; without them +// 50.7 against 42.5 standing and 56.0 against 45.9 along the field route, and with FSR 3.1, whose tail the GPU runs +// for 1.2 ms where FSR 4's takes 3.5, 51.8 against 47.8 and 54.9 against 52.3. All but a few of the tails passed the +// tests there; in area loads and some places a tail hands the CPU results and goes as before. The pictures matched +// those without it standing, along the route and through warps to five areas. Not measured with the lazy write-back +// (NVIDIA's and Intel's default), where a tail's uniform updates are noted as writes and no tail goes early. +// Not with shadow storage, emulated mesh shaders, a frame dump or a pass capture, nor inside a chunk or a continuation. +static struct { + int on; // the submission's tail is being recorded + int live0; // vk_mem_live_swap's note from before it + uint64_t wrote0; // vk_mem_gpu_wrote_count when it began + // since the last perf line: tails; those that went early; those refused by results, writes, late reads, other + uint64_t n, early, why[4]; +} TL; +int vk_exec_tail_results(void); // vk_exec_xfer.c +void vk_sync_tail_stats(uint64_t *tails, uint64_t *waits, double *ms); +int vk_gpu_passes_capturing(void); +int vk_mesh_shader_native(void); // vk_device.c +int vk_exec_tail(void) { return TL.on; } + +static int tail_enabled(void) { + static int on = -1; + if (on < 0) { + on = cfg_bool("NX2NAVI_TAIL_EARLY"); + if (on) + rt_log("vk: the write-backs go before a frame's tail, and the game's waits end there (NX2NAVI_TAIL_EARLY)"); + } + return on; +} + int vk_exec_split_now(VkExec *ex) { int in_pass = ex->in_pass; if (in_pass) vk_exec_pass_end(ex); @@ -1153,7 +1213,7 @@ static void maybe_split(VkExec *ex) { split_idle ? "; a first part at once while the GPU is idle" : ""); split_at = split_n ? split_n : INT_MAX; } - if (!split_n) return; + if (!split_n || TL.on) return; // the tail stays one command buffer (the early tail) if (Q.drained < Q.out) { // groups the submitting thread went on past go into the part first, once done if (__atomic_load_n(&Q.done, __ATOMIC_ACQUIRE) < Q.out) return; // split_draws stays: a later op asks again group_drain(ex, PW_SPLIT); @@ -1783,8 +1843,9 @@ static void op_one(VkExec *ex, const op_hdr *h) { } } -// ---- replay workers (NX2NAVI_REPLAY_WORKERS, default 0: the replay records every op on the submitting thread; the -// Game tab's Replay workers row sets it) +// ---- replay workers (NX2NAVI_REPLAY_WORKERS, default auto: as many as the CPU's cores make worth it, +// vk_exec_replay_workers_auto; 0: the replay records every op on the submitting thread; the Game tab's CPU helper +// threads row sets it) // 3 workers: on an RX 7900 XT with a Ryzen 7 5800X3D, Hyrule Field standing went from 41.2 to 50.4 fps (2026-11-12, one // batch), with the same pictures there and in Kokiri Forest, Kakariko, Goron City, a graveyard house and Gerudo // Valley; 5 or 7 workers measured the same as 3. Default 0 again after players' games crashed with 3 on by default @@ -1877,6 +1938,8 @@ static const char *const pw_name[PW_N] = {"next group", "op", "pass hooks", "tur static struct { int n; // workers; 0 off, -1 not read yet int pipe; // NX2NAVI_REPLAY_PIPE: the groups out at once at most, 0 none + int small; // NX2NAVI_REPLAY_SMALL: the fewest draws of a group one worker takes whole + uint64_t small_groups, small_draws; // such groups and their draws since the last perf line void *help_dc; // the submitting thread's draw context for chunks it takes while it waits RwWorker *w[RW_MAX]; const op_hdr *skip_to[CALL_DEPTH + 1]; // per call depth: the walk read ahead to this op, a group too small @@ -1992,9 +2055,36 @@ static void rw_main(void *arg) { } } +// NX2NAVI_REPLAY_WORKERS = auto: the workers this PC gets. The game's own threads, the submitting thread, the present +// thread and the pipeline builders already keep about half of the cores busy, and on the 8-core CPU above 3 workers +// gave all there was (5 and 7 measured the same), so the count is half the physical cores less one: 2 on 6 cores, 3 on +// 8, 5 on 12, at most 6. Four cores or fewer get none: a worker would take a core from the game's threads. AMD's +// Windows driver for RX 6000 and older (vk_caps.amd_sc) gets none either, as that is where a player's device was lost +// with workers on and the cause is not known; a number set in nx2navi.ini or the menu still applies there. Physical +// cores of the process's affinity (plat_cpu_cores), or half the usable logical processors where that is unknown. +#define RW_AUTO_MAX 6 +int vk_exec_replay_workers_auto(const char **why) { + uint64_t all, first; + int cores = plat_cpu_cores(&all, &first); + if (cores <= 0) cores = (int)plat_cpu_usable() / 2; + int n = cores <= 4 ? 0 : cores / 2 - 1; + if (n > RW_AUTO_MAX) n = RW_AUTO_MAX; + const char *w = cores <= 4 ? "four cores or fewer" : "half the cores less one"; + if (vk_caps.amd_sc) n = 0, w = "AMD's driver for RX 6000 and older, where a game with workers lost its device"; + if (why) *why = w; + return n; +} + static void rw_init(void) { - int n = cfg_int("NX2NAVI_REPLAY_WORKERS"); + const char *set = cfg_value("NX2NAVI_REPLAY_WORKERS"), *why = NULL; + int autoset = !strcmp(set, "auto"); + int n = autoset ? vk_exec_replay_workers_auto(&why) : atoi(set); RW.n = n < 0 ? 0 : n > RW_MAX ? RW_MAX : n; + if (autoset) { + uint64_t all, first; + rt_log("vk: replay workers automatic: %d for %d physical cores (%s; NX2NAVI_REPLAY_WORKERS = a number sets " + "them)", RW.n, plat_cpu_cores(&all, &first), why); + } for (int k = 0; k < RW.n; k++) { RwWorker *w = calloc(1, sizeof *w); if (!w) rt_fatal("vk: out of memory (replay worker)"); @@ -2008,6 +2098,10 @@ static void rw_init(void) { int pipe = cfg_int("NX2NAVI_REPLAY_PIPE"); // the groups out at once RW.pipe = RW.n <= 0 || pipe <= 0 ? 0 : pipe > LIVE_GROUPS ? LIVE_GROUPS : pipe; if (RW.pipe) RW.help_dc = vk_exec_draw_ctx_new(); + RW.small = RW.pipe ? cfg_int("NX2NAVI_REPLAY_SMALL") : 0; + if (RW.small > 0) + rt_log("vk: a group of %d draws or more goes whole to one replay worker (NX2NAVI_REPLAY_SMALL)", RW.small); + else RW.small = 0; if (RW.pipe) rt_log("vk: groups of %d draws or more are recorded by %d replay worker%s while the submitting thread goes on " "past them, %d at once at most (NX2NAVI_REPLAY_WORKERS, NX2NAVI_REPLAY_PIPE)", 2 * CHUNK_DRAWS, RW.n, @@ -2319,7 +2413,14 @@ static const op_hdr *group_start(VkExec *ex, const op_hdr *h, int depth) { int n = (int)(draws / CHUNK_DRAWS); if (n > RW.n + 1) n = RW.n + 1; if (n > GROUP_CHUNKS) n = GROUP_CHUNKS; - if (n < 2 || vk_dump_armed || !vk_sync_append_room(n)) { + // NX2NAVI_REPLAY_SMALL=N (default 0, with NX2NAVI_REPLAY_PIPE): a group of N draws or more that is too small to cut + // goes whole to one worker, and the submitting thread goes on with what follows it without taking the chunk + // itself. Kokiri Forest standing, 5 workers, 4 groups out, the early tail on (2026-11-13): about 700 draws a frame + // were in such groups, 3.6 ms of the submitting thread; with 16 the replay took 10.2 instead of 11.0 ms a frame + // (63.2 against 62.2 fps: the game then waits longer for the GPU, which is busy 83% of the frame there). + int small = RW.pipe && RW.small && n < 2 && draws >= (uint32_t)RW.small; + if (small) n = 1; + if (n < 1 + !small || vk_dump_armed || TL.on || !vk_sync_append_room(n)) { RW.skip_to[depth] = end, RW.serial_draws += draws, RW.small_end[why] += draws; return h; } @@ -2393,7 +2494,8 @@ static const op_hdr *group_start(VkExec *ex, const op_hdr *h, int depth) { // the submitting thread takes chunks too, as the workers wake up, and goes on once none is left to take: the // GPU waits for the group's part until its last chunk is done rw_behind = 1; - group_take(RW.n); + if (small) RW.small_groups++, RW.small_draws += draws; // a worker takes it: this thread goes on at once + else group_take(RW.n); uint64_t to = kern_now_ns(); RW.front_ns += tf - t0, RW.own_ns += to - tf, RW.wait_ns += to - t0; return end; @@ -2511,6 +2613,25 @@ static void rec_claim(void) { } } +// The tail begins here (the early tail): what was recorded so far waits in the chain, the tail gets a command buffer of +// its own. Not in a chunk or a continuation, nor with groups still out: the cut needs every piece of the part. +void vk_exec_tail_begin(VkExec *ex) { + if (!tail_enabled() || TL.on || ex != x_cur || rw_chunk || evo_in) return; + if (Q.drained != Q.out) group_drain(ex, PW_OP); // the cut needs every piece of the part + if (!vk_sync_wb_split() || !vk_mesh_shader_native() || vk_dump_armed || vk_gpu_passes_capturing()) return; + if (!vk_sync_tail_ok()) return; + if (ex->in_pass) vk_exec_pass_end(ex); + vk_exec_barrier(ex); // the command buffer so far ends behind a full barrier, every image back in GENERAL + VkCommandBuffer cb = vk_sync_cut(); + if (!cb) return; + ex->cb = cb; + ex->dirty = VK_EXEC_DIRTY_ALL; + vk_exec_draw_new_cb(); + n_cuts++; + TL.on = 1, TL.live0 = vk_mem_live_swap(0), TL.wrote0 = vk_mem_gpu_wrote_count(); + vk_exec_tail_results(); +} + int vk_exec_record(VkCommandBuffer cb, const uint64_t *handles, int n, uint64_t fence) { if (vk_device_is_lost()) return 0; // the run is ending (vk_device.c vk_device_lost_check): nothing is recorded uint64_t in_ns = kern_now_ns(); @@ -2528,6 +2649,7 @@ int vk_exec_record(VkCommandBuffer cb, const uint64_t *handles, int n, uint64_t nvn_pins_sweep(nvn_submit_count(), keep, 9); ex->cb = cb; ex->fence = fence; + TL.on = 0; ex->recorded = 0; ex->in_pass = 0; ex->mark = NULL; @@ -2594,12 +2716,47 @@ int vk_exec_record(VkCommandBuffer cb, const uint64_t *handles, int n, uint64_t // NX2NAVI_WB_SPLIT: the write-backs in a part of their own, after one that signals the executed timeline // (vk_sync.c exec_tl): the wait after a frame's first submission then ends when the submission's passes are // through, not its write-backs. Off by default: vk_sync.c says what else that wait then leaves out. - if (synced && ex->recorded && vk_sync_wb_split() && vk_mem_mirror_wb_pending() && vk_exec_pool_split(ex->fence)) { + int tail = 0; // the early tail: the write-backs go before the tail, with the executed timeline's signal + if (TL.on) { + int live = vk_mem_live_swap(0), res = vk_exec_tail_results(); + vk_mem_live_swap(live | TL.live0); // the submission's own note, as it would have been + int wrote = vk_mem_gpu_wrote_count() != TL.wrote0; + int other = !synced || !ex->recorded || vk_dump_armed || vk_gpu_passes_capturing() || !vk_sync_tail_ok(); + TL.on = 0, TL.n++; + if (res) TL.why[0]++; + else if (wrote) TL.why[1]++; + else if (live) TL.why[2]++; + else if (other || !vk_exec_pool_split(ex->fence)) TL.why[3]++; + else tail = 1, TL.early++; + if (tail) { + ex->cb = vk_sync_tail_hold(); + ex->dirty = VK_EXEC_DIRTY_ALL; + vk_exec_draw_new_cb(); + } + } + if (!tail && synced && ex->recorded && vk_sync_wb_split() && vk_mem_mirror_wb_pending() && + vk_exec_pool_split(ex->fence)) { ex->cb = vk_sync_split_exec(); ex->dirty = VK_EXEC_DIRTY_ALL; vk_exec_draw_new_cb(); } if (synced) ex->recorded += vk_mem_mirror_end(ex->cb, fence), vk_usnap_end(fence); // VRAM mirror bytes the GPU wrote go back to guest memory + if (tail) { + ex->cb = vk_sync_tail_release(); + part_start = ex->recorded; // the last command buffer holds nothing: the tail is in the chain before it + } + if ((rt_trace & RT_TRACE_PERF) && TL.n >= 600) { + uint64_t tails, waits; + double ms; + vk_sync_tail_stats(&tails, &waits, &ms); + rt_log("perf: early tail: %llu tails, %llu with the write-backs before them (refused: %llu handed the CPU " + "results, %llu wrote pool memory, %llu read guest memory late, %llu other); %llu waits of the game " + "ended before their tail, which the GPU runs for about %.2f ms", + (unsigned long long)TL.n, (unsigned long long)TL.early, + (unsigned long long)TL.why[0], (unsigned long long)TL.why[1], (unsigned long long)TL.why[2], + (unsigned long long)TL.why[3], (unsigned long long)waits, ms); + TL.n = TL.early = 0, memset(TL.why, 0, sizeof TL.why); + } views_release(ex, fence); ex->cb = VK_NULL_HANDLE; if (n) n_subs++; @@ -2696,8 +2853,13 @@ int vk_exec_record(VkCommandBuffer cb, const uint64_t *handles, int n, uint64_t memset(RW.pw_n, 0, sizeof RW.pw_n), memset(RW.pw_ns, 0, sizeof RW.pw_ns); memset(RW.pd_op, 0, sizeof RW.pd_op); RW.behind_ns = 0; + if (RW.small) + rt_log("perf: replay workers: groups one worker took whole (NX2NAVI_REPLAY_SMALL): %.2f per " + "submission, %.1f draws each", (double)RW.small_groups / (double)n_subs, + RW.small_groups ? (double)RW.small_draws / (double)RW.small_groups : 0.0); } } + RW.small_groups = RW.small_draws = 0; RW.groups = RW.group_draws = RW.chunks = RW.serial_draws = RW.wait_ns = 0; evo_held = evo_resumed = evo_forced = evo_switches = 0; n_subs = n_parts = n_idle_parts = n_pass_splits = n_cuts = 0; diff --git a/runtime/vk_exec_draw.c b/runtime/vk_exec_draw.c index 1169af3a..2dc3db02 100644 --- a/runtime/vk_exec_draw.c +++ b/runtime/vk_exec_draw.c @@ -32,6 +32,7 @@ #include "vk_gpa.h" #include "mesh_fb.h" #include "vk_mesh_fb.h" +#include "vk_scratch_lru.h" #include "platform/platform.h" #include "vk_dlaa.h" #include "mesh_emu_spirv.h" @@ -920,8 +921,8 @@ VkImageView vk_exec_target_view(const VkTexImage *img, VkFormat format, VkImageV } int vk_exec_image_gone(VkImage image) { - vk_tonemap_image_gone(image); // its views in the tonemap pass, keyed by the handle a new image may get vk_exec_bc_gone(image); + vk_tonemap_forget(image); // the stand-in's views of a game image it read as it is vk_hdr_image_gone(image); if (!views_by_image.klen) return 0; int purged = 0; @@ -6828,37 +6829,23 @@ static int fmt_rgba8(VkFormat f) { f == VK_FORMAT_B8G8R8A8_SRGB || f == VK_FORMAT_A8B8G8R8_UNORM_PACK32 || f == VK_FORMAT_A8B8G8R8_SRGB_PACK32; } -// Scratch images for those blits, one per (size, format). Up to FB_SCRATCH are kept; a new size past them takes the -// place of the one used longest ago, which goes once the GPU is past the submission being recorded (vk_defer_image), -// with the tonemap pass's view of it. Before, the first eight sizes were kept for the process and a ninth got no -// scratch, so its blit was left out: the sizes come from the scene's resolution, which DLSS modes, the dynamic -// resolution and a window or picture size change move, at up to 32 MiB each at 2160 lines. One used in the -// submission being recorded is never the one replaced: the blit may have recorded it already. -#define FB_SCRATCH 8 +// Scratch images for those blits, one per (size, format), at most SCRATCH_N; a new size takes the place of the one used +// longest ago (vk_scratch_lru.h), whose image goes through the deferred destruction. +#define SCRATCH_N 8 static VkImage fallback_scratch(VkExec *ex, uint32_t w, uint32_t h, VkFormat f) { - static struct { uint32_t w, h; VkFormat f; VkImage img; VmaAllocation a; uint64_t used; } cache[FB_SCRATCH]; - static int n; - static PlatMutex lock = PLAT_MUTEX_INIT; // replay workers record blits side by side - plat_mutex_lock(&lock); - for (int i = 0; i < n; i++) - if (cache[i].w == w && cache[i].h == h && cache[i].f == f) { - cache[i].used = ex->fence; - VkImage img = cache[i].img; - plat_mutex_unlock(&lock); - return img; - } - int slot = n; - if (n == FB_SCRATCH) { // the one used longest ago, unless the submission being recorded uses it - slot = -1; - for (int i = 0; i < n; i++) - if (cache[i].used < ex->fence && (slot < 0 || cache[i].used < cache[slot].used)) slot = i; - if (slot < 0) { - plat_mutex_unlock(&lock); - return VK_NULL_HANDLE; - } - vk_tonemap_image_gone(cache[slot].img); - vk_defer_image(cache[slot].img, cache[slot].a); - cache[slot].img = VK_NULL_HANDLE; + static ScrEntry ent[SCRATCH_N]; + static struct { VkImage img; VmaAllocation a; } cache[SCRATCH_N]; + static int evictions; + int hit, evict, i = scr_pick(ent, SCRATCH_N, w, h, (uint32_t)f, + __atomic_load_n(&gfx_counters.frames, __ATOMIC_RELAXED), &hit, &evict); + if (hit) return cache[i].img; + if (evict) { + if (evictions++ < 8) + rt_log("vk: a stand-in scratch image of %ux%u (format %d) makes room for one of %ux%u (format %d)", + ent[i].w, ent[i].h, (int)ent[i].f, w, h, (int)f); + vk_tonemap_forget(cache[i].img); // the stand-in's views of it, before a new image can take its handle + vk_defer_image(cache[i].img, cache[i].a); + ent[i].live = 0, cache[i].img = VK_NULL_HANDLE, cache[i].a = NULL; } VkCtx *c = vk_ctx(); VkImageCreateInfo ii = {.sType = VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO, .imageType = VK_IMAGE_TYPE_2D, .format = f, @@ -6869,11 +6856,7 @@ static VkImage fallback_scratch(VkExec *ex, uint32_t w, uint32_t h, VkFormat f) .initialLayout = VK_IMAGE_LAYOUT_UNDEFINED}; VmaAllocationCreateInfo ia = {.usage = VMA_MEMORY_USAGE_AUTO_PREFER_DEVICE}; VkImage img; VmaAllocation a; - if (vmaCreateImage(c->vma, &ii, &ia, &img, &a, NULL) != VK_SUCCESS) { - if (slot < n) cache[slot] = cache[--n]; // the replaced entry's place stays empty - plat_mutex_unlock(&lock); - return VK_NULL_HANDLE; - } + if (vmaCreateImage(c->vma, &ii, &ia, &img, &a, NULL) != VK_SUCCESS) return VK_NULL_HANDLE; VkImageMemoryBarrier2 bar = {.sType = VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER_2, .dstStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, .dstAccessMask = VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT, @@ -6882,10 +6865,8 @@ static VkImage fallback_scratch(VkExec *ex, uint32_t w, uint32_t h, VkFormat f) .image = img, .subresourceRange = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 1, 0, 1}}; VkDependencyInfo di = {.sType = VK_STRUCTURE_TYPE_DEPENDENCY_INFO, .imageMemoryBarrierCount = 1, .pImageMemoryBarriers = &bar}; vkCmdPipelineBarrier2(ex->cb, &di); - cache[slot].w = w, cache[slot].h = h, cache[slot].f = f, cache[slot].img = img, cache[slot].a = a; - cache[slot].used = ex->fence; - if (slot == n) n++; - plat_mutex_unlock(&lock); + ent[i] = (ScrEntry){w, h, (uint32_t)f, __atomic_load_n(&gfx_counters.frames, __ATOMIC_RELAXED), 1}; + cache[i].img = img, cache[i].a = a; return img; } @@ -7391,7 +7372,14 @@ static void op_update_ub(VkExec *ex, const p_update_ub *u) { VK_MEM_PICK(sp); VkBufferCopy cp = {o, sp.offset, u->size}; vkCmdCopyBuffer(ex->cb, b, sp.buffer, 1, &cp); - vk_mem_gpu_wrote(&sp, u->size); + // In a submission's tail (vk_exec.c "the early tail") an update of a mirrored block reaches guest memory with the + // results the game reads, from the bytes here, and takes no write-back: one after the tail would land when the + // game is already writing the next frame's data. Not with the lazy write-back, where the block's pages may be + // armed and a store into them would wait for the GPU (memory.c lazy_fault): the update is noted as ever there, + // and the tail goes after the write-backs as before. + if (vk_exec_tail() && !vk_mem_wb_lazy() && vk_mem_mirrored(u->dst, u->size)) + vk_exec_tail_ub(ex->fence, u->dst, u->data, u->size); + else vk_mem_gpu_wrote(&sp, u->size); vk_usnap_drop(u->dst, u->size); ex->recorded++; vk_mem_count(sp.cls, VK_MUSE_GPU_WRITE, u->size); diff --git a/runtime/vk_exec_xfer.c b/runtime/vk_exec_xfer.c index 79026a89..6a15e19a 100644 --- a/runtime/vk_exec_xfer.c +++ b/runtime/vk_exec_xfer.c @@ -67,12 +67,32 @@ typedef struct { uint64_t addr, size; int back; } Range; // back: written into // for the newest such submission). Write-backs of the mirrors note it in vk_memory.c; the lazily written ones fetch on // the CPU's first access. static volatile int32_t ex_results; // VK_RES_* bits +static int tail_results; // those noted while the submission's tail records (vk_exec.c "the early tail") +int vk_exec_tail_results(void) { int r = tail_results; tail_results = 0; return r; } void vk_exec_note_results(int kind) { + if (vk_exec_tail()) tail_results |= kind; __atomic_fetch_or(&ex_results, kind, __ATOMIC_RELAXED); vk_gpu_passes_result(kind); // NX2NAVI_GPU_PASSES: where in the submission it is done } int vk_exec_take_results(void) { return __atomic_exchange_n(&ex_results, 0, __ATOMIC_RELAXED); } +// An inline uniform update in a submission's tail (vk_exec_draw.c op_update_ub, vk_exec.c "the early tail"): the GPU +// copies the bytes into the pool's mirror where the stream has the update, and guest memory gets them here, with the +// results the game reads, in place of a write-back. +typedef struct { uint64_t addr; uint32_t size; uint8_t data[]; } UbWrite; +static void ub_write_cb(void *arg) { + UbWrite *w = arg; + memcpy((void *)(uintptr_t)w->addr, w->data, w->size); + free(w); +} +void vk_exec_tail_ub(uint64_t fence, uint64_t addr, const void *data, uint32_t size) { + UbWrite *w = malloc(sizeof *w + size); + if (!w) rt_fatal("vk: out of memory"); + w->addr = addr, w->size = size; + memcpy(w->data, data, size); + vk_completion_add_result(fence, ub_write_cb, w); +} + static void guest_written_cb(void *arg) { Range *r = arg; if (r->back) vk_gpu_written_back(r->addr, r->size); @@ -114,8 +134,9 @@ static void event_cb(void *arg) { // NVNcounterData = {counter, timestamp}; timestamps are Tegra 19.2 MHz ticks (the base of GetSystemTick). // keep: the 4-byte quarters of the record the write leaves as they are (bit n: bytes 4n to 4n + 3), 0 none. -static void write_counter(uint64_t addr, uint64_t counter, unsigned keep) { - uint64_t rec[2] = {counter, (uint64_t)((unsigned __int128)kern_now_ns() * 12 / 625)}; +// ahead_ns: the timestamp is that much past now (a tail timestamp the GPU has not reached yet, counter_cb), else 0. +static void write_counter(uint64_t addr, uint64_t counter, unsigned keep, uint64_t ahead_ns) { + uint64_t rec[2] = {counter, (uint64_t)((unsigned __int128)(kern_now_ns() + ahead_ns) * 12 / 625)}; if (!keep) { memcpy((void *)(uintptr_t)addr, rec, sizeof rec); return; @@ -127,14 +148,16 @@ static void write_counter(uint64_t addr, uint64_t counter, unsigned keep) { // Reports other than samples passed (timestamps; the statistics counters are not modelled) write a counter of 0. The // counter word matters: Ocarina writes {1, 0} into a record from the CPU, reports TIMESTAMP_TOP into it and gates // clears on it differing from a zero record (mode 1), clears that must not run. -static void report_write(uint64_t addr, uint64_t seq, uint64_t counter); // below, with the pending table +static void report_write(uint64_t addr, uint64_t seq, uint64_t counter, uint64_t ahead_ns); // below, with the pending table -typedef struct { uint64_t addr, seq; } CounterWrite; +// tail: the submission's value for a timestamp reported in its tail (vk_exec.c "the early tail"), else 0 +typedef struct { uint64_t addr, seq, tail; } CounterWrite; static void counter_cb(void *arg) { CounterWrite *w = arg; if (vk_sync_effect_late()) __atomic_fetch_add(&late_counters, 1, __ATOMIC_RELAXED); - report_write(w->addr, w->seq, 0); + // A timestamp of the tail, written while the GPU still runs the tail: the time the tail is expected to end at + report_write(w->addr, w->seq, 0, w->tail ? vk_sync_tail_pending_ns(w->tail) : 0); free(w); } @@ -308,7 +331,7 @@ static unsigned clear_quarters(const Pending *e) { return e->clr_seq >= e->seq ? // Completion thread, or the thread whose wait runs the results: report seq to addr goes into guest memory, but for // the quarters a small clear replayed after it has filled. -static void report_write(uint64_t addr, uint64_t seq, uint64_t counter) { +static void report_write(uint64_t addr, uint64_t seq, uint64_t counter, uint64_t ahead_ns) { int lk = __atomic_load_n(&clr_any, __ATOMIC_ACQUIRE); unsigned keep = 0; if (lk) { @@ -317,7 +340,7 @@ static void report_write(uint64_t addr, uint64_t seq, uint64_t counter) { if (e && seq <= e->clr_seq) keep = e->clr_mask; if (keep) __atomic_fetch_add(&CS.kept, 1, __ATOMIC_RELAXED); } - write_counter(addr, counter, keep); + write_counter(addr, counter, keep, ahead_ns); if (lk) plat_mutex_unlock(&clr_lock); pending_written(addr, seq); } @@ -644,7 +667,7 @@ static void oq_cb(void *arg) { if (s->lost && !oq_count) oq_count = 1; if (s->addr) { if (vk_sync_effect_late()) __atomic_fetch_add(&late_counts, 1, __ATOMIC_RELAXED); - report_write(s->addr, s->seq, oq_count); + report_write(s->addr, s->seq, oq_count, 0); } } free(s->q), free(s); @@ -893,10 +916,13 @@ static void report_counter(VkExec *ex, const p_report_counter *p) { clear_note_report(p->addr); if (samples) oq_report(ex, p->addr); else { + // a timestamp in the tail is the one result a tail may hand the CPU: its time is written ahead (counter_cb) + int ts_tail = vk_exec_tail() && p->type == NVN_COUNTER_TYPE_TIMESTAMP, keep = tail_results; vk_exec_note_results(VK_RES_COUNTER); + if (ts_tail) tail_results = keep; CounterWrite *a = malloc(sizeof *a); if (!a) rt_fatal("vk: out of memory"); - *a = (CounterWrite){p->addr, report_seq}; + *a = (CounterWrite){p->addr, report_seq, ts_tail ? ex->fence : 0}; vk_completion_add_result(ex->fence, counter_cb, a); } Pending *e = pending_slot(p->addr, 1); diff --git a/runtime/vk_fsr.c b/runtime/vk_fsr.c index cd1864aa..09b9b18e 100644 --- a/runtime/vk_fsr.c +++ b/runtime/vk_fsr.c @@ -39,8 +39,26 @@ static struct { uint32_t w, h; // their size VkImageView src_view[SLOTS]; // the source view each slot's last frame read int logged; + uint32_t used_slots, since; // slots whose frames used the images / that recorded without them since the last use + uint32_t idle; // presents recorded without them since their last use } F; +// The two images while the upscaler is not fsr (or the picture needs no scaling): they stayed for the rest of the run, +// two RGBA16F pictures of the window's size, 168 MiB at 5120x2142. They go once IDLE_FREE presents recorded without +// them and every slot that ever used them has recorded since: the caller waited for that slot's fence before it +// recorded (see the top of the file), so no submission still reads them. Taking fsr up again makes them as a window +// size change does (ensure_images: a wait for the idle device and two images). +#define IDLE_FREE 600 +static void free_images(VkCtx *c); +static void idle_note(unsigned slot) { + if (!F.img[0] || cfg_vram_keep()) return; // vram_use = high keeps them + F.since |= 1u << (slot % SLOTS); + if (++F.idle < IDLE_FREE || (F.since & F.used_slots) != F.used_slots) return; + rt_log("vk: FSR's images unused for %u presents: freed (%ux%u)", F.idle, F.w, F.h); + free_images(vk_ctx()); + F.used_slots = F.since = F.idle = 0; +} + static VkPipeline make_pipeline(VkCtx *c, const uint32_t *code, size_t size) { VkShaderModuleCreateInfo mi = {.sType = VK_STRUCTURE_TYPE_SHADER_MODULE_CREATE_INFO, .codeSize = size, .pCode = code}; VkShaderModule m = VK_NULL_HANDLE; @@ -198,40 +216,23 @@ static int shrink_sharpen(VkCommandBuffer cb, VkImage src, const int crop[4], Vk return 2; } -// Presents in a row that FSR 1 did not scale (another upscaler chosen, or a picture the window shows at its own size) -// after which its two images at the window's size go (56 MiB at 2560x1440, 127 MiB at 2160 lines; kept for the process -// before). A present that scales again makes them anew. -#define FSR_IDLE_PRESENTS 300 -static unsigned fsr_idle; - -static void fsr_unused(void) { - if (!F.img[0] || ++fsr_idle < FSR_IDLE_PRESENTS) return; - VkCtx *c = vk_ctx(); - vk_queue_lock(); - vkDeviceWaitIdle(c->device); // earlier presents may still read them - vk_queue_unlock(); - free_images(c); - fsr_idle = 0; - rt_log("vk: FSR 1 has not scaled a picture for %d presents: its images are freed", FSR_IDLE_PRESENTS); -} - int vk_fsr_record(VkCommandBuffer cb, unsigned slot, VkImage src, VkFormat fmt, int sw, int sh, const int crop[4], VkImage dst, const int out[4]) { int cw = crop[2], ch = crop[3]; uint32_t dw = (uint32_t)out[2], dh = (uint32_t)out[3]; if (strcmp(cfg_value("NX2NAVI_UPSCALER"), "fsr") || !fmt || (dw == (uint32_t)cw && dh == (uint32_t)ch)) - return fsr_unused(), 0; + return idle_note(slot), 0; int sharp = cfg_int("NX2NAVI_SHARPNESS"); if (sharp < 0) sharp = 0; if (sharp > 100) sharp = 100; // EASU only enlarges: a picture smaller than the crop in either direction takes the shrink path, which without // sharpening is the plain blit of the caller int shrink = dw < (uint32_t)cw || dh < (uint32_t)ch; - if (shrink && !sharp) return fsr_unused(), 0; - fsr_idle = 0; + if (shrink && !sharp) return idle_note(slot), 0; VkCtx *c = vk_ctx(); if (!F.tried) init(c); if (!F.ok || !ensure_images(c, dw, dh)) return 0; + F.used_slots |= 1u << (slot % SLOTS), F.since = 0, F.idle = 0; if (shrink) return shrink_sharpen(cb, src, crop, dst, out, sharp); slot %= SLOTS; if (F.src_view[slot]) vkDestroyImageView(c->device, F.src_view[slot], NULL); diff --git a/runtime/vk_memory.c b/runtime/vk_memory.c index 15b705a9..a50a3c94 100644 --- a/runtime/vk_memory.c +++ b/runtime/vk_memory.c @@ -966,6 +966,7 @@ static int dead_pending(void); static void vram_log(uint64_t frame); #define HOLES_PERIOD 3600 // frames between two "mem: holes" lines (holes_log) static void holes_log(uint64_t frame); +static void owner_log(uint64_t frame); // the memory report, after the staging pool (vk_mem_give_back) static void compact_look(void); static void outside_watch(uint64_t frame); static void up_ts_log(uint64_t frame); @@ -983,7 +984,7 @@ void vk_mem_census(uint64_t frame) { if (dmabuf_on && frame % 60 == 0) span_gc(frame, 0); static uint64_t window_start; if ((rt_trace & RT_TRACE_PERF) && frame % 600 == 0) vram_log(frame), mirror_pool_log(frame); - if (frame && frame % ((rt_trace & RT_TRACE_PERF) ? 600 : HOLES_PERIOD) == 0) holes_log(frame); + if (frame && frame % ((rt_trace & RT_TRACE_PERF) ? 600 : HOLES_PERIOD) == 0) holes_log(frame), owner_log(frame); if (vk_texture_compact_due()) compact_look(); // NX2NAVI_TEX_COMPACT: 0 at once with the setting off if ((rt_trace & RT_TRACE_PERF) && frame % 60 == 0) outside_watch(frame), up_ts_log(frame), vk_gtl_log(frame, 60); if ((rt_trace & RT_TRACE_PERF) && !vk_mirror_on) dirty_frame(frame); @@ -3725,7 +3726,11 @@ static uint64_t wb_skip(void) { } static int64_t m_wb_skipped; // bytes wb_skip kept in mirrors since the last summary +static volatile uint64_t n_gpu_wrote; // vk_mem_gpu_wrote_count +uint64_t vk_mem_gpu_wrote_count(void) { return __atomic_load_n(&n_gpu_wrote, __ATOMIC_RELAXED); } + static void gpu_wrote(const VkMemSpan *sp, uint64_t size, int shader) { + __atomic_fetch_add(&n_gpu_wrote, 1, __ATOMIC_RELAXED); PoolRec *r = sp->pool; if (!r || r->mstate != MS_ON || sp->buffer != r->mbuf || !size || !wb_lazy()) { uint64_t skip = r && r->mstate == MS_ON && sp->buffer == r->mbuf ? wb_skip() : 0; @@ -4100,6 +4105,9 @@ static uint64_t capture_cap(void) { } void vk_mem_note_live(void) { __atomic_store_n(&mirror_live, 1, __ATOMIC_RELAXED); } +int vk_mem_live_swap(int v) { return __atomic_exchange_n(&mirror_live, v, __ATOMIC_RELAXED); } +static int wb_lazy(void); +int vk_mem_wb_lazy(void) { return wb_lazy(); } // the lazy write-back runs (vk_exec_draw.c op_update_ub) int vk_mem_take_live(void) { int live = __atomic_exchange_n(&mirror_live, 0, __ATOMIC_RELAXED); @@ -5360,6 +5368,88 @@ uint64_t vk_mem_give_back(void) { return freed + vk_headroom_release(); } +// ---- The memory report: every HOLES_PERIOD presents (600 with NX2NAVI_TRACE=perf), after the holes line, in every run. +// The device-local memory the driver counts for the game split by what holds it: VMA's allocations by owner +// (vk_vma_budget.c, "owners"), with the textures split further by the texture cache's census (vk_texture_census) and +// the pools into VRAM mirrors and physical pools; the free space inside VMA's blocks; and what VMA never allocated +// (compiled pipelines, descriptor memory, the swapchain, the DLSS and frame generation features, the driver's own), +// with DLSS's share as its statistics report it. Then the host-visible allocations (staging above all) and the +// process's system memory with the game's own heap. Then the released textures' reuse (vk_texture_reuse_log). +// A category that only climbs from line to line over a long session is a leak. The figures are sums the owners keep +// already, one driver query, VMA's budget query and one walk of the texture entries (vk_texture_census): well under a +// millisecond once a minute, at the present that logs. +static void owner_log(uint64_t frame) { + VkCtx *c = vk_ctx(); + if (!c || !c->vma) return; + VkPhysicalDeviceMemoryBudgetPropertiesEXT bp = {.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_MEMORY_BUDGET_PROPERTIES_EXT}; + if (c->has_memory_budget) { + VkPhysicalDeviceMemoryProperties2 p2 = {.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_MEMORY_PROPERTIES_2, .pNext = &bp}; + vkGetPhysicalDeviceMemoryProperties2(c->phys, &p2); + } + VmaBudget vb[VK_MAX_MEMORY_HEAPS]; + vmaGetHeapBudgets(c->vma, vb); + uint64_t usage = 0, blocks = 0, allocs = 0, hblocks = 0, hallocs = 0; + for (uint32_t h = 0; h < c->mem_props.memoryHeapCount && h < VK_MAX_MEMORY_HEAPS; h++) { + if (c->mem_props.memoryHeaps[h].flags & VK_MEMORY_HEAP_DEVICE_LOCAL_BIT) + usage += c->has_memory_budget ? bp.heapUsage[h] : vb[h].statistics.blockBytes, + blocks += vb[h].statistics.blockBytes, allocs += vb[h].statistics.allocationBytes; + else hblocks += vb[h].statistics.blockBytes, hallocs += vb[h].statistics.allocationBytes; + } + uint64_t phys = 0; + plat_mutex_lock_shared(&pool_lock); + for (int i = 0; i < nphys; i++) + if (phys_list[i]->alloc) phys += phys_list[i]->size; + plat_mutex_unlock_shared(&pool_lock); + VkTexCensus t; + vk_texture_census(&t); + uint64_t own[VK_OWN_N], hown[VK_OWN_N]; + for (int o = 0; o < VK_OWN_N; o++) own[o] = vk_vma_owner_bytes(o, 1), hown[o] = vk_vma_owner_bytes(o, 0); + uint64_t stg_blocks = 0, kept; + if (stg_pool) { + VmaStatistics st; + vmaGetPoolStatistics(c->vma, stg_pool, &st); + stg_blocks = st.blockBytes; + } + // Read without their locks: an upload holds cmd_lock while it waits for a batch (stg_create), tens of milliseconds + // the present would wait too; a figure a log reads a moment late does no harm. + kept = __atomic_load_n(&stg_cache_bytes, __ATOMIC_RELAXED) + + (uint64_t)__atomic_load_n(&narena_free, __ATOMIC_RELAXED) * ARENA_CHUNK; + uint64_t snaps = __atomic_load_n(&snap_pool_bytes, __ATOMIC_RELAXED); + int64_t dlss = vk_dlaa_vram(); + char ds[48] = "nothing"; + if (dlss >= 0) snprintf(ds, sizeof ds, "%.0f MiB", dlss / 1048576.0); + const double M = 1048576.0; + uint64_t mir = (uint64_t)__atomic_load_n(&m_live_bytes, __ATOMIC_RELAXED); + rt_log("mem: by owner frame %llu: device-local in use %.0f MiB (the driver's figure) = VMA's allocations %.0f + free " + "inside VMA's blocks %.0f + outside VMA %.0f (compiled pipelines, descriptor memory, the swapchain, DLSS and " + "frame generation features, the driver; DLSS reports %s). VMA's allocations: textures %.0f (live %.0f, of them " + "render targets and other GPU-drawn %.0f and idle %.0f; released %.0f, of them too lately to drop %.0f), pools " + "%.0f (VRAM mirrors %.0f, physical pools %.0f), upscalers %.0f, replay buffers %.0f, uniform snapshots %.0f, " + "device address table %.0f, ASTC decoder %.0f, present and post-processing %.0f, headroom spares %.0f, other " + "%.0f", (unsigned long long)frame, usage / M, allocs / M, blocks > allocs ? (blocks - allocs) / M : 0.0, + usage > blocks ? (usage - blocks) / M : 0.0, ds, own[VK_OWN_TEXTURES] / M, t.bytes_live / M, + t.bytes_live_gpu / M, t.bytes_live_idle / M, t.bytes_released / M, t.bytes_released_young / M, + own[VK_OWN_POOLS] / M, mir / M, phys / M, own[VK_OWN_UPSCALER] / M, own[VK_OWN_REPLAY] / M, + own[VK_OWN_USNAP] / M, own[VK_OWN_GPA] / M, own[VK_OWN_ASTC] / M, own[VK_OWN_PRESENT] / M, + own[VK_OWN_HEADROOM] / M, own[VK_OWN_OTHER] / M); + double proc, sys_used, sys_limit; + float ram_used, ram_total, ws; + plat_commit_info(&proc, &sys_used, &sys_limit); + plat_mem_info(&ram_used, &ram_total, &ws); + uint64_t heap, heap_c, maps; + mem_guest_usage(&heap, &heap_c, &maps); + rt_log("mem: host frame %llu: VMA's host-visible blocks %.0f MiB hold %.0f MiB: staging pool blocks %.0f (kept staging " + "buffers and arena chunks %.0f), write-back snapshots %.0f; by owner: pools and staging %.0f, textures %.0f, " + "upscalers %.0f, replay buffers %.0f, uniform snapshots %.0f, present and post-processing (the flight " + "recorder's thumbnails among them) %.0f, other %.0f. Process commit %.0f MB, working set %.0f MB; system commit " + "%.0f of %.0f MB; the game's heap %.0f MiB (%.0f committed), its physical maps %.0f MiB", + (unsigned long long)frame, hblocks / M, hallocs / M, stg_blocks / M, kept / M, snaps / M, hown[VK_OWN_POOLS] / M, + hown[VK_OWN_TEXTURES] / M, hown[VK_OWN_UPSCALER] / M, hown[VK_OWN_REPLAY] / M, hown[VK_OWN_USNAP] / M, + hown[VK_OWN_PRESENT] / M, (hown[VK_OWN_OTHER] + hown[VK_OWN_GPA] + hown[VK_OWN_ASTC] + hown[VK_OWN_HEADROOM]) / M, + proc, (double)ws, sys_used, sys_limit, heap / M, heap_c / M, maps / M); + vk_texture_reuse_log(frame); +} + // NX2NAVI_STAGING_FAIL_AT=n (test): the n-th staging buffer the driver makes is taken as refused for memory once, so // the give-back runs (vk_mem_give_back). static int staging_fail_now(void) { diff --git a/runtime/vk_scratch_lru.h b/runtime/vk_scratch_lru.h new file mode 100644 index 00000000..d71d815b --- /dev/null +++ b/runtime/vk_scratch_lru.h @@ -0,0 +1,38 @@ +// Which entry of a small cache of scratch images serves a request for an image of one size and format +// (vk_exec_draw.c fallback_scratch). Header only, so that tests/scratch_lru_check.c checks it without a device. +// +// The cache held its first SCRATCH_N images for the whole run and refused every other size: each scene size the +// stand-in of the game's upscaler blits through takes one, and after a run of window, resolution or DLSS mode +// changes the blit of a new size found no image and was left out. Now a request that matches no live entry takes a +// free one, else the one used longest ago, which the caller frees (deferred: a submission may still name it) before +// it makes the new image there. The images in use stay; one used in no frame for a while is what goes. +#pragma once +#include + +typedef struct { + uint32_t w, h, f; // the image's size and format (VkFormat) + uint64_t used; // the frame of its last request + int live; // holds an image +} ScrEntry; + +// The entry for (w, h, f) among e[0..n-1]: a live match (*hit 1), else a free entry, else the live one used longest ago +// (*evict 1: its image is to be freed before the slot is reused). A hit's used becomes now; the caller fills a new +// entry once its image is made. +static inline int scr_pick(ScrEntry *e, int n, uint32_t w, uint32_t h, uint32_t f, uint64_t now, int *hit, int *evict) { + int free_at = -1, lru = -1; + *hit = *evict = 0; + for (int i = 0; i < n; i++) { + if (!e[i].live) { + if (free_at < 0) free_at = i; + continue; + } + if (e[i].w == w && e[i].h == h && e[i].f == f) { + e[i].used = now, *hit = 1; + return i; + } + if (lru < 0 || e[i].used < e[lru].used) lru = i; + } + if (free_at >= 0) return free_at; + *evict = 1; + return lru; +} diff --git a/runtime/vk_sync.c b/runtime/vk_sync.c index 9cdc2868..d864e659 100644 --- a/runtime/vk_sync.c +++ b/runtime/vk_sync.c @@ -63,6 +63,16 @@ static uint64_t live_last_of[LIVE_RING], live_last; // per value (index & LIVE_RING - 1): the newest value up to it whose completion hands results to the CPU (vk_exec_take_results) static uint64_t res_of[LIVE_RING], res_last; static uint8_t res_kinds[LIVE_RING]; // per value: its VK_RES_* kinds +// The early tail (vk_exec.c "the early tail"), per value under st_lock: early_of holds the value when its part with +// the write-backs signalled the executed timeline before its tail; empty_of when the submission handed the GPU nothing +// and the CPU no results (a submission held at its first event wait: Ocarina's first of a frame in the field). +static uint64_t early_of[LIVE_RING], empty_of[LIVE_RING]; +static VkCommandBuffer tail_cb; // sub_lock: the tail's command buffer between hold and release +// sub_lock: this submission's tail went behind its write-backs; the parts of it handed over before the last +static int tail_early, sub_parts; +static volatile uint64_t tail_ns; // about how long the GPU takes for a tail (a moving average) +static volatile uint64_t tail_t0, tail_val; // the first early return for tail_val, for tail_ns +static volatile uint64_t tail_n, tail_waits; // since the last vk_sync_tail_stats: early tails; waits they ended static int res_mask = 0x1f; // NX2NAVI_SUBMIT_FINISH_KINDS: the kinds the wait counts static int stop; static Pending *queue; // side effects in the order they were added @@ -920,6 +930,12 @@ static void completion_main(void *arg) { // finished submissions held (vk_mem_collect: upload batches, destroyed images, views and buffers, each // checked against the timeline itself) is nothing they read, and it takes cmd_lock, which the upload // path holds, so done waited behind uploads. + uint64_t tv = __atomic_load_n(&tail_val, __ATOMIC_ACQUIRE); + if (tv && reached >= tv) { // the tail a wait went ahead of is through: how long it took from there + uint64_t d = kern_now_ns() - __atomic_load_n(&tail_t0, __ATOMIC_RELAXED), a = tail_ns; + if (d < 50000000ull) __atomic_store_n(&tail_ns, a ? (a * 7 + d) / 8 : d, __ATOMIC_RELAXED); + __atomic_compare_exchange_n(&tail_val, &tv, 0, 0, __ATOMIC_RELEASE, __ATOMIC_RELAXED); + } uint64_t was = done; __atomic_store_n(&done, reached, __ATOMIC_RELEASE); plat_cond_broadcast(&cv_done); @@ -1197,6 +1213,63 @@ VkCommandBuffer vk_sync_worker_begin(int w) { void vk_sync_worker_end(VkCommandBuffer cb) { vkEndCommandBuffer(cb); } static VkCommandBuffer split_part(int exec); + +// ---- the early tail (vk_exec.c "the early tail"). The replay cut the part where the tail began (vk_sync_cut), so the +// chain holds what came before and the command buffer being recorded is the tail's. vk_sync_tail_hold sets that one +// aside, ended and timed as a part of its own, and begins the one the write-backs are recorded into; +// vk_sync_tail_release hands the chain and the write-backs to the queue with the executed timeline's signal +// (split_part), and puts the tail into the chain for the submission's last part, which signals the timeline. The queue +// runs them in that order, so the GPU's work is what it was with the write-backs moved before the tail. +int vk_sync_tail_ok(void) { return !pre_cb && !tail_cb && exec_tl && exec_sig < cur_fence && nchain + 2 <= CHAIN_MAX; } + +VkCommandBuffer vk_sync_tail_hold(void) { + VkCtx *c = vk_ctx(); + VkCommandBuffer cb = slots[cur_slot].cb; + vk_gpu_passes_part(cb, 1); + ts_end(cb, cur_fence); + if (!rec_end_cb(cb, 0, 0)) vkEndCommandBuffer(cb); + tail_cb = cb; + CmdSlot *s = take_slot(c, cur_fence); // the tail keeps its slot until the value is reached + VkCommandBufferBeginInfo bi = {.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO, + .flags = VK_COMMAND_BUFFER_USAGE_ONE_TIME_SUBMIT_BIT}; + vkBeginCommandBuffer(s->cb, &bi); + ts_begin(s->cb); + rec_begin(s->cb, cur_fence); + vk_gpu_passes_part(s->cb, 0); + return s->cb; +} + +VkCommandBuffer vk_sync_tail_release(void) { + VkCommandBuffer cb = split_part(1); + chain[nchain++] = tail_cb; + tail_cb = VK_NULL_HANDLE; + tail_early = 1; + return cb; +} + +// The newest value up to `value` whose GPU work a wait of the game is for: the empty submissions after it are passed +// over. *early: that one's write-backs went before its tail, so the executed timeline reaches it there. +static uint64_t wait_target(uint64_t value, int *early) { + plat_mutex_lock(&st_lock); + uint64_t t = value; + while (t > done && empty_of[t & (LIVE_RING - 1)] == t) t--; + *early = t > done && early_of[t & (LIVE_RING - 1)] == t; + plat_mutex_unlock(&st_lock); + return t; +} + +uint64_t vk_sync_tail_pending_ns(uint64_t value) { + plat_mutex_lock(&st_lock); + int pending = value > done && early_of[value & (LIVE_RING - 1)] == value; + plat_mutex_unlock(&st_lock); + return pending ? __atomic_load_n(&tail_ns, __ATOMIC_RELAXED) : 0; +} + +void vk_sync_tail_stats(uint64_t *tails, uint64_t *waits, double *ms) { + *tails = __atomic_exchange_n(&tail_n, 0, __ATOMIC_RELAXED); + *waits = __atomic_exchange_n(&tail_waits, 0, __ATOMIC_RELAXED); + *ms = (double)__atomic_load_n(&tail_ns, __ATOMIC_RELAXED) / 1e6; +} VkCommandBuffer vk_sync_split(void) { return split_part(0); } // The part that ends here is the last with the submission's own commands: it signals the executed timeline. VkCommandBuffer vk_sync_split_exec(void) { return split_part(1); } @@ -1218,6 +1291,7 @@ static VkCommandBuffer split_part(int exec) { si.signalSemaphoreInfoCount = 1, si.pSignalSemaphoreInfos = &esig, exec_sig = cur_fence; ts_submitted(cbi, ncb); queue_submit(c, &si, 0); // held or handed to the queue (submit batching) + sub_parts++; vk_mem_lazy_handed(); vk_mem_note_submit(cur_fence); // objects destroyed from here on may be in use by the part just submitted CmdSlot *s = take_slot(c, cur_fence); // the submitted part keeps its slot until the value is reached @@ -1259,6 +1333,7 @@ static uint64_t submit(const uint64_t *handles, int n, const OwnCommands *own) { vk_mem_dirty_submit(); // NX2NAVI_TRACE=perf: what a mirror would upload before this submit uint64_t fence = __atomic_load_n(&issued, __ATOMIC_ACQUIRE) + 1; cur_fence = fence; + tail_early = sub_parts = 0; CmdSlot *s = take_slot(c, fence); VkCommandBufferBeginInfo bi = {.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO, .flags = VK_COMMAND_BUFFER_USAGE_ONE_TIME_SUBMIT_BIT}; @@ -1318,6 +1393,9 @@ static uint64_t submit(const uint64_t *handles, int n, const OwnCommands *own) { if (live) live_last = fence; live_last_of[fence & (LIVE_RING - 1)] = live_last; res_kinds[fence & (LIVE_RING - 1)] = (uint8_t)res; + early_of[fence & (LIVE_RING - 1)] = tail_early ? fence : 0; + empty_of[fence & (LIVE_RING - 1)] = !own && !sent && !sub_parts && !res ? fence : 0; + if (tail_early) __atomic_fetch_add(&tail_n, 1, __ATOMIC_RELAXED); if (res & res_mask) res_last = fence; res_of[fence & (LIVE_RING - 1)] = res_last; __atomic_store_n(&issued, fence, __ATOMIC_RELEASE); @@ -1530,8 +1608,13 @@ int vk_sync_exec_wait(uint64_t value, uint64_t timeout_ns) { if (vk_device_is_lost()) return __atomic_load_n(&done, __ATOMIC_ACQUIRE) >= value; if (!vk_sync_wb_split()) return vk_sync_fence_wait(value, timeout_ns); if (__atomic_load_n(&done, __ATOMIC_ACQUIRE) >= value) return 1; + // The early tail: a value whose GPU work is that of an earlier submission with its write-backs before its tail (the + // submissions since handed the GPU nothing) is reached where that one's executed signal is, its tail still to run. + int early = 0; + uint64_t asked = value, target = wb_results ? wait_target(value, &early) : value; + if (early) value = target; uint64_t p0 = perf_begin(); - hold_release(value, 0, 0); // a held submission the waiter needs goes to the queue now + hold_release(asked, 0, 0); // a held submission the waiter needs goes to the queue now VkCtx *c = vk_ctx(); VkSemaphoreWaitInfo wi = {.sType = VK_STRUCTURE_TYPE_SEMAPHORE_WAIT_INFO, .semaphoreCount = 1, .pSemaphores = &exec_tl, .pValues = &value}; @@ -1545,6 +1628,13 @@ int vk_sync_exec_wait(uint64_t value, uint64_t timeout_ns) { if (r == VK_SUCCESS && wb_results) results_run(value); perf_end(PERF_FENCE, p0); perf_gpu_wait(p0); + if (r == VK_SUCCESS && early && __atomic_load_n(&done, __ATOMIC_ACQUIRE) < value) { + __atomic_fetch_add(&tail_waits, 1, __ATOMIC_RELAXED); + uint64_t none = 0; // the first wait back for this tail starts its clock (the completion thread ends it) + if (__atomic_load_n(&tail_val, __ATOMIC_ACQUIRE) != value && + __atomic_compare_exchange_n(&tail_val, &none, value, 0, __ATOMIC_RELEASE, __ATOMIC_RELAXED)) + __atomic_store_n(&tail_t0, kern_now_ns(), __ATOMIC_RELAXED); + } if (r == VK_SUCCESS) { __atomic_fetch_add(&exw_n, 1, __ATOMIC_RELAXED); if (__atomic_load_n(&done, __ATOMIC_ACQUIRE) < value) { @@ -1556,6 +1646,19 @@ int vk_sync_exec_wait(uint64_t value, uint64_t timeout_ns) { return r == VK_SUCCESS; } +// The game's own wait for a fence (nvn_api_core.c nvnSyncWait): the fence's wait, or with the early tail the wait for +// the results of the submission the value's GPU work belongs to (vk_sync_exec_wait), when that one's tail is all the +// GPU still has of it: the tail reads nothing the game can change and hands it nothing. +int vk_sync_game_wait(uint64_t value, uint64_t timeout_ns) { + vk_sync_start(); + if (vk_sync_wb_split() && wb_results && __atomic_load_n(&done, __ATOMIC_ACQUIRE) < value) { + int early = 0; + wait_target(value, &early); + if (early) return vk_sync_exec_wait(value, timeout_ns); + } + return vk_sync_fence_wait(value, timeout_ns); +} + uint64_t vk_sync_fence_done(void) { return __atomic_load_n(&done, __ATOMIC_ACQUIRE); } uint64_t vk_sync_issued(void) { return __atomic_load_n(&issued, __ATOMIC_ACQUIRE); } diff --git a/runtime/vk_tex_release.h b/runtime/vk_tex_release.h new file mode 100644 index 00000000..0e3dec36 --- /dev/null +++ b/runtime/vk_tex_release.h @@ -0,0 +1,95 @@ +// Released textures by age, the part without a GPU: how long a released texture is kept, how many bytes the released +// ones may hold before the size bound drops young ones, and the wall clock that turns a present count into seconds. +// vk_texture.c ("released textures by age", drop_aged and sweep_pressure) does what these say; +// tests/tex_release_check.py runs this file alone. +// +// The game releases a texture when what uses it leaves the view and takes it back when that returns. The cache keeps a +// released texture's image so that the take-back finds it; a texture taken back after its image was dropped is made +// again (created and uploaded, or decoded on the GPU), and the frame that asks for it samples its stand-in. How long +// after the release the take-backs come is what decides how long an image is worth keeping. Two sessions on an RTX +// 5090 (2026-11-12, one at the old Balanced, one at the old Low; vk_texture.c vk_texture_reuse_log) took back, per +// line of 3600 presents: under 2 s 22,000 to 121,000; 2 to 10 s 1 to 126; 10 to 60 s 0 to 156; 1 to 5 min and longer +// none. Balanced kept every release up to its bound of 4 GiB, so its zero past a minute is not a dropped texture's +// absence: nothing released for longer was asked for again. A released texture not taken back within a minute was +// never taken back. +// +// So a released texture goes once it has gone unused for the hold (TRL_HOLD_S by default, NX2NAVI_TEX_RELEASED_HOLD), +// three times the longest gap seen, whether or not the bound is reached: the cache then holds what the last few +// minutes released, not what the bound allows, and it empties within minutes of the last area change. Unused counts +// from the release or from the last time a shader sampled the released image (a finalized slot may show it), whichever +// is later. The hold is the same for a texture uploaded from the game's memory and for an ASTC one the GPU decodes: +// either made again shows its stand-in for a frame, the decode only costs more of the render thread. A texture whose +// content only the GPU made (a render target) cannot be made again at all, it comes back empty: it waits twice the +// hold. +// +// The size bound (released_cap_bytes, a quarter of the video memory budget by default) stays as the safety net it was: +// past it the oldest released textures go even when young. With the hold on, young ones past it are kept while the card +// and the system have room (trl_room), up to TRL_ROOM_TIMES the bound: the old Balanced dropped 31 young ones in a +// burst of area changes at its bound of 4 GiB on a card with 20 GiB free, and each was made again within a minute. The +// hold empties that excess within minutes, so the room it takes is temporary; without the hold (vram_use = high) the +// bound acts as before. +#pragma once +#include + +enum { TRL_STORAGE, TRL_GPU }; // content made again from the game's memory; content only the GPU made + +#define TRL_HOLD_S 180 // the default hold, in seconds +#define TRL_HOLD_MAX_S 900 // the longest hold the setting takes: twice it fits the clock below +#define TRL_ROOM_TIMES 2 // with room, the released textures may hold this many times the bound +#define TRL_ROOM_MARGIN 10 // room: the card at least this many points of its budget under the VRAM pressure + +// The hold of a class, in seconds; 0: no hold. +static inline uint32_t trl_hold(uint32_t hold_s, int cls) { return cls == TRL_GPU ? hold_s * 2 : hold_s; } + +// Whether a released texture unused for unused_s seconds has passed its hold. +static inline int trl_aged(uint32_t hold_s, int cls, uint32_t unused_s) { + return hold_s && unused_s >= trl_hold(hold_s, cls); +} + +// Whether the card and the system have room for young released textures past the bound: the device-local memory in +// use (the driver's figure, holes counted) is TRL_ROOM_MARGIN points or more under the VRAM pressure's share of the +// budget, and the system's commit has at least twice the free space under which it counts as low (commit_free_mb +// below 0: the commit is not tracked here, which counts as no room). +static inline int trl_room(uint64_t used, uint64_t budget, int pressure_pct, double commit_free_mb, + double commit_low_mb) { + if (!budget || pressure_pct <= TRL_ROOM_MARGIN || commit_free_mb < 0) return 0; + return used <= budget / 100 * (uint64_t)(pressure_pct - TRL_ROOM_MARGIN) && commit_free_mb >= 2 * commit_low_mb; +} + +// The bytes the released textures may hold before the bound drops young ones (drop_stale's keep): the bound, or +// TRL_ROOM_TIMES it while the hold is on and there is room. cap UINT64_MAX: no bound. +static inline uint64_t trl_keep(uint32_t hold_s, uint64_t cap, int room) { + if (cap == UINT64_MAX || !hold_s || !room) return cap; + return cap > UINT64_MAX / TRL_ROOM_TIMES ? UINT64_MAX : cap * TRL_ROOM_TIMES; +} + +// The wall clock of the presents. The cache stamps its textures with present counts (gfx_counters.frames), and the +// presents per second change with the scene and the menus: a hold counted in presents would be shorter in wall time +// after a fast stretch. The clock notes the present count once a second or so (tick) and answers how many seconds ago +// a present was, rounded down to the tick after it: a texture is taken as at most that old, never older. +#define TRL_CLOCK_N 2048 // ticks kept: at least 2048 s, more than twice TRL_HOLD_MAX_S +typedef struct { + uint64_t frame[TRL_CLOCK_N]; // the present count at each tick + uint32_t sec[TRL_CLOCK_N]; // the tick's second of the run + uint32_t n; // ticks so far +} TrlClock; + +static inline void trl_clock_tick(TrlClock *c, uint64_t frame, uint32_t sec) { + c->frame[c->n % TRL_CLOCK_N] = frame, c->sec[c->n % TRL_CLOCK_N] = sec; + c->n++; +} + +// Seconds from present `frame` to now_sec: from the first tick at or after it. A present after the last tick is 0 s +// old; one before the oldest tick kept is as old as that tick, at least TRL_CLOCK_N - 1 seconds. +static inline uint32_t trl_clock_age(const TrlClock *c, uint64_t frame, uint32_t now_sec) { + uint32_t kept = c->n < TRL_CLOCK_N ? c->n : TRL_CLOCK_N, first = c->n - kept; + if (!kept || c->frame[(c->n - 1) % TRL_CLOCK_N] < frame) return 0; + uint32_t lo = first, hi = c->n - 1; // the first tick with frame >= frame lies in [lo, hi] + while (lo < hi) { + uint32_t mid = lo + (hi - lo) / 2; + if (c->frame[mid % TRL_CLOCK_N] >= frame) hi = mid; + else lo = mid + 1; + } + uint32_t s = c->sec[lo % TRL_CLOCK_N]; + return now_sec > s ? now_sec - s : 0; +} diff --git a/runtime/vk_texture.c b/runtime/vk_texture.c index edf46f74..2b215a4b 100644 --- a/runtime/vk_texture.c +++ b/runtime/vk_texture.c @@ -28,6 +28,7 @@ #include "vk_dlaa.h" #include "vk_exec.h" #include "vk_tex_compact.h" +#include "vk_tex_release.h" #include "prof.h" #include "platform/platform.h" @@ -182,12 +183,36 @@ static uint32_t submit_epoch = 1; // bumped per queue submit: draw bindings ch static uint32_t tex_rate = 256; #define PRESENTS(n) ((uint64_t)(n) * tex_rate >> 8) +// The wall clock of the presents (vk_tex_release.h TrlClock): rate_tick notes the present count about once a second, +// so that the released textures' hold (drop_aged) and the take-back ages (vk_texture_reuse_log) count seconds, not +// presents at whatever rate the game runs. tex_lock. +// The seconds are seconds of play, not of the wall: a gap between two submissions counts at most PLAY_GAP_MAX_NS. A +// window minimized for ten minutes (freeze_idle), a game stopped at a breakpoint or a long load then does not age the +// released textures past their hold while nothing could take them back, and the first frames after it find the +// textures the frames just before it released. +#define PLAY_GAP_MAX_NS 1000000000ull +static TrlClock pclock; +static uint64_t pclock_t0; // kern_now_ns at the first tick +static uint64_t play_ns; // seconds of play so far, in ns (rate_tick) +static uint64_t play_last; // kern_now_ns at the last submission rate_tick saw + +static uint32_t now_sec(void) { + if (!pclock_t0) return 0; + uint64_t gap = kern_now_ns() - play_last; + return (uint32_t)((play_ns + (gap < PLAY_GAP_MAX_NS ? gap : PLAY_GAP_MAX_NS)) / 1000000000ull); +} + +static uint32_t secs_since(uint64_t frame) { return trl_clock_age(&pclock, frame, now_sec()); } + static void rate_tick(void) { // each submission, tex_lock held static uint64_t t0, f0; static uint32_t ring[RATE_N], at; uint64_t t = kern_now_ns(), f = __atomic_load_n(&gfx_counters.frames, __ATOMIC_RELAXED); - if (!t0) t0 = t, f0 = f; + if (play_last) play_ns += t - play_last < PLAY_GAP_MAX_NS ? t - play_last : PLAY_GAP_MAX_NS; + play_last = t; + if (!t0) t0 = t, f0 = f, pclock_t0 = t, trl_clock_tick(&pclock, f, 0); if (t - t0 < 1000000000ull) return; + trl_clock_tick(&pclock, f, (uint32_t)(play_ns / 1000000000ull)); uint64_t r = (f - f0) * 256ull * 1000000000ull / ((t - t0) * 60); ring[at++ % RATE_N] = r < 256 ? 256 : r > 2048 ? 2048 : (uint32_t)r; t0 = t, f0 = f; @@ -723,8 +748,12 @@ static int slot_names(const Tex *e) { // Why an entry was destroyed (drop_tex), in the order of drop_names. enum { DROP_NONE, DROP_MAX, DROP_CAP, DROP_COMMIT, DROP_VRAM, DROP_IDLE_VRAM, DROP_IDLE_COMMIT, DROP_DISUSE, - DROP_OVERWRITTEN, DROP_REPLACED, DROP_REUSED, DROP_RANGE, DROP_EXACT, DROP_SHUTDOWN, DROP_COMPACT, DROP_N }; + DROP_OVERWRITTEN, DROP_REPLACED, DROP_REUSED, DROP_RANGE, DROP_EXACT, DROP_SHUTDOWN, DROP_COMPACT, + DROP_AGE, DROP_N }; static void drop_tex(Tex *e, int why); +static void reuse_dropped(const Tex *e, int why); // "reuse of released textures", after vk_texture_census +static void reuse_made(const Tex *e); +static void reuse_taken_back(const Tex *e); static int compact_on(void); // NX2NAVI_TEX_COMPACT ("compaction", near the end of the file) static void compact_step(void); static void compact_forget(Tex *e); @@ -1569,8 +1598,10 @@ static void rel_append(Tex *e) { } static Tex *sweep_at; // where sweep_released goes on +static Tex *aged_at; // where drop_aged goes on static void rel_unlink(Tex *e) { if (sweep_at == e) sweep_at = e->rel_next; + if (aged_at == e) aged_at = e->rel_next; ru_unlink(e); rel_bytes -= e->bytes; if (e->rel_prev) e->rel_prev->rel_next = e->rel_next; @@ -1885,7 +1916,8 @@ static const char *const drop_names[DROP_N] = { "", "past RELEASED_MAX released entries", "past the released entries' size cap", "under commit pressure", "under VRAM pressure", "idle, under VRAM pressure", "idle, under commit pressure", "unsampled (disuse)", "overwritten at another address", "replaced by another descriptor at its address", "its image went to another texture", - "its storage went", "made whole again (texture detail)", "shutdown", "released, in a block a compaction empties"}; + "its storage went", "made whole again (texture detail)", "shutdown", "released, in a block a compaction empties", + "released and unused for longer than the hold"}; static const char *const start_names[START_N] = { "started on its own", "filled from storage over a live entry", "content copied from a live entry", "content copied from a released entry", "started empty over a live entry", "started empty over a released entry", @@ -2015,6 +2047,7 @@ static void drop_tex(Tex *e, int why) { else if (wb_wanted(e)) write_back_n(&e, 1), n_drop_wb++; } n_drop_by[why]++; + reuse_dropped(e, why); if (rt_trace & RT_TRACE_PERF) drops[ndrops++ % DROPS_N] = (DropRec){key_hash(&e->key), e->key.addr, frames_now(), (uint32_t)e->t.d.format, (uint32_t)e->t.d.flags, e->t.d.width, e->t.d.height, (uint8_t)why, @@ -2361,12 +2394,15 @@ void vk_texture_pool_ctx(int on) { pool_ctx = on; } static int idle(const Tex *e, uint64_t now); // NX2NAVI_TEX_RELEASED_MB (default a thirty-second of the installed memory, 512 MiB to 2 GiB, or more by the card's -// budget, released_cap_bytes; 0 = no limit) bounds the -// own memory of the released entries: RELEASED_MAX alone kept 5-7 GiB of released render targets and decoded textures -// after an hour of play at a 2160p game resolution, device memory the driver also charges to the system's commit (the -// process's private bytes reached 31 GiB). A sixteenth (1 to 4 GiB) still let them hold 1.3 GiB on a 32 GB PC whose -// commit then ran out at an area load (device lost); only entries released STALE_FRAMES presents ago go, so the -// transient ones the game takes back within frames stay. NX2NAVI_TEX_COMMIT_FREE_MB (default the larger of 4 GiB and a tenth of the commit +// budget, released_cap_bytes; 0 = no limit) bounds the own memory of the released entries. With the hold on (drop_aged, +// the default) what they hold is what the last minutes released, and the bound is the safety net for a burst of area +// changes within the hold; sweep_pressure lets young entries past it while the card and the system have room +// (vk_tex_release.h trl_keep). Before the hold the bound was all that limited them: RELEASED_MAX alone kept 5-7 GiB of +// released render targets and decoded textures after an hour of play at a 2160p game resolution, device memory the +// driver also charges to the system's commit (the process's private bytes reached 31 GiB). A sixteenth (1 to 4 GiB) +// still let them hold 1.3 GiB on a 32 GB PC whose commit then ran out at an area load (device lost); only entries +// released STALE_FRAMES presents ago go, so the transient ones the game takes back within frames stay. +// NX2NAVI_TEX_COMMIT_FREE_MB (default the larger of 4 GiB and a tenth of the commit // limit; 0 = off) lowers that bound to COMMIT_LOW_KEEP while less of the system's commit is free: an allocation that // fails there ends the game. On Linux nothing fails at the limit: the OOM killer ends the game when its cgroup or the // machine runs out of memory, and plat_commit_info reports the tighter of the two. Both drop only stale entries @@ -2400,6 +2436,9 @@ static long long proc_commit_mb; // this process's bound in MB; 0 = none static int commit_low, commit_logs; // commit_low: either of the two below at the last check static int sys_low, proc_high; // the system's commit is low; this process's is above its bound static uint64_t n_cap_drops, n_commit_drops, n_commit_past; +// The system's commit at the last check, for the room young released entries past the bound may take (trl_room): MB +// free and the free space under which it counts as low; -1 free: not tracked. +static double commit_free_now = -1, commit_low_mark; static uint64_t released_cap_bytes(void) { if (!released_cap) { @@ -2456,6 +2495,7 @@ static int commit_is_low(void) { "textures give way only when the system's commit is low", plat_commit_counts_vram() ? ", less the video memory in use on a discrete card," : "", proc_commit_mb); } + commit_free_now = -1; if (!commit_free_mb && !proc_commit_mb) return 0; double proc, used, limit; plat_commit_info(&proc, &used, &limit); @@ -2464,6 +2504,7 @@ static int commit_is_low(void) { if (commit_free_mb && used >= 0 && limit > 0) { double want = commit_free_mb > 0 ? (double)commit_free_mb : limit / 10 > 4096 ? limit / 10 : 4096; low = limit - used < want; + commit_free_now = limit - used, commit_low_mark = want; commit_critical = limit - used < want / 2; // the pause for drops that come back ends here } double dev = 0; // device-local memory in use, MB: in the commit (plat_commit_counts_vram), not in RAM (see above) @@ -2616,6 +2657,62 @@ static void drop_overwritten(void) { } } +// ---- released textures by age (vk_tex_release.h has the take-back figures behind the hold) +// NX2NAVI_TEX_RELEASED_HOLD (seconds, default TRL_HOLD_S, at most TRL_HOLD_MAX_S; 0 = no hold; vram_use = high sets 0): +// a released entry that owns memory goes once it has gone unused that long, twice that for content only the GPU made, +// whatever the size bound says. Unused counts from the release or the last sample of the released image, whichever is +// later, in seconds of the wall clock (secs_since). The rules drop_stale keeps hold here too: not before stale_after +// (an entry the game once took back after a long gap waits GAP_TIMES that gap) and not while it is not idle (a lookup +// other than the pool's reached it, or a shader sampled it, lately). Nor an entry the GPU drew into since its last +// write-back: its drop would first copy it back to the game's memory and wait for the GPU (drop_tex, write_back_n), +// a stall the sweep is not to make in play; such an entry stays to the size bound, as before the hold. Each check +// looks at up to AGED_LOOK entries from where the last one stopped (aged_at), in release order, and starts from the +// oldest again at the first one released less than the hold ago: the rest are younger. +#define AGED_LOOK 256 +static int rel_hold = -1; // seconds; -1: not read yet +static uint64_t n_age_drops, n_age_bytes; // since the last vk_texture_reuse_log line + +static uint32_t released_hold(void) { + if (rel_hold < 0) { + const char *s = cfg_get("NX2NAVI_TEX_RELEASED_HOLD"); + long long v = s && *s ? atoll(s) : TRL_HOLD_S; + rel_hold = v < 0 ? 0 : v > TRL_HOLD_MAX_S ? TRL_HOLD_MAX_S : (int)v; + if (rel_hold) + rt_log("vk: released textures go once unused for %d s, %d s for content only the GPU made " + "(NX2NAVI_TEX_RELEASED_HOLD); young ones past the size bound stay while the card and the system " + "have room, up to %d times the bound", rel_hold, (int)trl_hold((uint32_t)rel_hold, TRL_GPU), + TRL_ROOM_TIMES); + else + rt_log("vk: released textures stay until the size bound or a pressure takes them " + "(NX2NAVI_TEX_RELEASED_HOLD=0)"); + } + return (uint32_t)rel_hold; +} + +static void drop_aged(void) { // tex_lock held + uint32_t hold = released_hold(); + if (!hold) return; + uint64_t now = frames_now(); + Tex *e = aged_at ? aged_at : rel_head; + int dropped = 0; + for (int n = 0; n < AGED_LOOK && dropped < PRESSURE_STEP; n++) { + if (!e || secs_since(e->rel_frame) < hold) { // the rest is younger: from the oldest next time + e = NULL; + break; + } + Tex *next = e->rel_next; + int cls = comes_back_intact(e) || e->overwritten ? TRL_STORAGE : TRL_GPU; + uint64_t last = e->samp_frame > e->rel_frame ? e->samp_frame : e->rel_frame; + if (e->bytes && !e->gpu_dirty && trl_aged(hold, cls, secs_since(last)) && + now - e->rel_frame >= stale_after(e) && idle(e, now)) { + n_age_drops++, n_age_bytes += e->bytes, dropped++; + drop_tex(e, DROP_AGE); + } + e = next; + } + aged_at = e; +} + // NX2NAVI_TEX_EVICT (default on; 0 = only released entries go under VRAM pressure, as before): above the VRAM pressure // threshold, once the released entries that come back intact are gone, live entries go too, least recently looked up // first: those that come back intact and that no lookup reached for IDLE_ROUNDS pool walks and STALE_FRAMES presents @@ -3175,9 +3272,17 @@ static void sweep_pressure(void) { // tex_lock held int held = low && commit_drops_held(); // the commit drops came back (commit_back_note) if (rel_head) { drop_overwritten(); - uint64_t cap = released_cap_bytes(); - if (rel_bytes > cap) drop_stale(cap, &n_cap_drops, DROP_CAP); - if (cap != UINT64_MAX && rel_bytes > cap + cap / 4) rel_census(cap); // well over it still: what holds the rest + drop_aged(); + // the bound, or more of it while the hold empties the released entries within minutes and the card and the + // system have room (vk_tex_release.h trl_keep): a young entry dropped there is one the game may still take back + uint64_t cap = released_cap_bytes(), keep = cap, vused, vbudget; + int vexact; + if (rel_bytes > cap && released_hold() && vk_mem_vram(&vused, &vbudget, &vexact)) + keep = trl_keep(released_hold(), cap, + trl_room(vused, vbudget, vram_pressure ? vram_pressure : VRAM_PRESSURE_DEFAULT, + commit_free_now, commit_low_mark)); + if (rel_bytes > keep) drop_stale(keep, &n_cap_drops, DROP_CAP); + if (keep != UINT64_MAX && rel_bytes > keep + keep / 4) rel_census(keep); // well over it: what holds the rest if (rel_bytes > COMMIT_LOW_KEEP && low && !held) more = drop_stale(COMMIT_LOW_KEEP, &n_commit_drops, DROP_COMMIT) == PRESSURE_STEP; } @@ -4246,6 +4351,7 @@ static Tex *entry_of(const tex_t *t) { } if (!(e = create_tex(t, &k, tex_alias))) return create_failed(&k), NULL; origin_made(e); + reuse_made(e); perf_count(PERF_CNT_TEX_NEW); start_at(e, &k, 0); flight_ev(e, FLIGHT_TEX_NEW, e->og_start, 0); @@ -4263,6 +4369,7 @@ static Tex *entry_of(const tex_t *t) { e->samp_frame = __atomic_load_n(&gfx_counters.frames, __ATOMIC_RELAXED); // in use again, as a new one uint64_t gap = e->samp_frame - e->rel_frame; if (gap > e->back_gap) e->back_gap = gap > UINT32_MAX ? UINT32_MAX : (uint32_t)gap; + reuse_taken_back(e); e->released = 0, nreleased--, rel_unlink(e), n_site[2]++, tex_gen_flip(e, e->superseded || e->overwritten); if (e->pidx) pent[e->pidx - 1].rel = 0; // the pool's check looks at it again e->overwritten = 0; @@ -6324,25 +6431,102 @@ void vk_texture_stats(uint64_t *images, uint64_t *uploads, uint64_t *reuploads) unlock(); } +// A walk over the entries under the texture lock, with the sizes the entries keep (Tex.bytes): about 0.1 ms with ten +// thousand of them, at most once a HOLES_PERIOD (vk_memory.c owner_log) without the perf trace. void vk_texture_census(VkTexCensus *s) { - VkCtx *c = vk_ctx(); memset(s, 0, sizeof *s); lock(); + uint64_t now = frames_now(); for (uint32_t i = 0; i < by_key.cap; i++) { const Tex *e = by_key.vals[i]; if (!e) continue; - uint64_t bytes = 0; - if (e->alloc && c) { - VmaAllocationInfo ai; - vmaGetAllocationInfo(c->vma, e->alloc, &ai); - bytes = ai.size; + uint64_t bytes = e->alloc ? e->bytes : 0; + if (e->released) { + s->released++, s->own_released += e->alloc != NULL, s->bytes_released += bytes; + if (now - e->rel_frame < stale_after(e)) s->bytes_released_young += bytes; + } else { + s->live++, s->own_live += e->alloc != NULL, s->bytes_live += bytes; + if (!comes_back_intact(e)) s->bytes_live_gpu += bytes; + if (idle(e, now)) s->bytes_live_idle += bytes; } - if (e->released) s->released++, s->own_released += e->alloc != NULL, s->bytes_released += bytes; - else s->live++, s->own_live += e->alloc != NULL, s->bytes_live += bytes; } unlock(); } +// ---- reuse of released textures (vk_texture_reuse_log) +// How often a released entry is taken back and how long it had been released then, and how often an entry is made +// again after a drop that a bound or a pressure made (the image made and its content uploaded or decoded once more). +// The two tell what the bounds on the released entries cost: an entry taken back after minutes is one a smaller bound +// may have dropped, and one made again soon after its drop is one the bound did drop. Counted since the last line; the +// dropped keys are remembered in a table of DROPPED_N slots indexed by the key's hash, a newer drop taking the slot. +// Ages are wall seconds (secs_since); 1-3 min and 3-5 min lie either side of the default hold (TRL_HOLD_S). +enum { AGE_YOUNG, AGE_10S, AGE_1M, AGE_3M, AGE_5M, AGE_OLD, AGE_N }; +static const char *const age_names[AGE_N] = {"under 2 s", "2-10 s", "10-60 s", "1-3 min", "3-5 min", "longer"}; +static uint64_t rb_n[AGE_N], rb_bytes[AGE_N]; // take-backs by the time released (tex_lock) +static uint64_t ra_n[AGE_N], ra_bytes, ra_why[DROP_N]; // made again by the time since the drop, and by its reason +#define DROPPED_N 8192 +static struct { uint64_t kh, frame; uint8_t why; } dropped[DROPPED_N]; // tex_lock + +static int age_of(uint32_t s) { // seconds + return s < 2 ? AGE_YOUNG : s < 10 ? AGE_10S : s < 60 ? AGE_1M : s < 180 ? AGE_3M : s < 300 ? AGE_5M : AGE_OLD; +} + +static void reuse_taken_back(const Tex *e) { // tex_lock held; e is being taken back, rel_frame still its release + int a = age_of(secs_since(e->rel_frame)); + rb_n[a]++, rb_bytes[a] += e->alloc ? e->bytes : 0; +} + +// Drops of an entry the game may still ask for: past a bound, or under a pressure, or unsampled. +static int drop_costs(int why) { + return why == DROP_MAX || why == DROP_CAP || why == DROP_COMMIT || why == DROP_VRAM || why == DROP_IDLE_VRAM || + why == DROP_IDLE_COMMIT || why == DROP_DISUSE || why == DROP_AGE; +} + +static void reuse_dropped(const Tex *e, int why) { // tex_lock held + if (!drop_costs(why)) return; + uint64_t kh = key_hash(&e->key); + dropped[kh % DROPPED_N].kh = kh, dropped[kh % DROPPED_N].frame = frames_now(), dropped[kh % DROPPED_N].why = (uint8_t)why; +} + +static void reuse_made(const Tex *e) { // tex_lock held; e was just made + uint64_t kh = key_hash(&e->key); + if (dropped[kh % DROPPED_N].kh != kh || !dropped[kh % DROPPED_N].why) return; + int a = age_of(secs_since(dropped[kh % DROPPED_N].frame)); + ra_n[a]++, ra_bytes += e->alloc ? e->bytes : 0, ra_why[dropped[kh % DROPPED_N].why]++; + dropped[kh % DROPPED_N].why = 0; +} + +void vk_texture_reuse_log(uint64_t frame) { + char line[1536]; + const double M = 1048576.0; + lock(); + int o = snprintf(line, sizeof line, "vk: released textures frame %llu, since the last line: taken back by the time " + "released:", (unsigned long long)frame); + for (int a = 0; a < AGE_N; a++) + o += snprintf(line + o, sizeof line - (size_t)o, " %s %llu (%.0f MiB);", age_names[a], (unsigned long long)rb_n[a], + rb_bytes[a] / M); + uint64_t again = 0; + for (int a = 0; a < AGE_N; a++) again += ra_n[a]; + o += snprintf(line + o, sizeof line - (size_t)o, " made again after a drop %llu (%.0f MiB; by the time since the drop:", + (unsigned long long)again, ra_bytes / M); + for (int a = 0; a < AGE_N; a++) + o += snprintf(line + o, sizeof line - (size_t)o, "%s %s %llu", a ? "," : "", age_names[a], (unsigned long long)ra_n[a]); + snprintf(line + o, sizeof line - (size_t)o, "; by the drop: released past the hold %llu, past the size bound " + "%llu, past the count %llu, VRAM pressure %llu released and %llu idle, commit %llu released and %llu " + "idle, unsampled %llu); dropped past the hold %llu (%.0f MiB); now %d released, %.0f MiB of a bound of " + "%.0f MiB, hold %u s", + (unsigned long long)ra_why[DROP_AGE], (unsigned long long)ra_why[DROP_CAP], + (unsigned long long)ra_why[DROP_MAX], (unsigned long long)ra_why[DROP_VRAM], + (unsigned long long)ra_why[DROP_IDLE_VRAM], (unsigned long long)ra_why[DROP_COMMIT], + (unsigned long long)ra_why[DROP_IDLE_COMMIT], (unsigned long long)ra_why[DROP_DISUSE], + (unsigned long long)n_age_drops, n_age_bytes / M, nreleased, rel_bytes / M, + released_cap_bytes() == UINT64_MAX ? 0.0 : released_cap_bytes() / M, released_hold()); + memset(rb_n, 0, sizeof rb_n), memset(rb_bytes, 0, sizeof rb_bytes), memset(ra_n, 0, sizeof ra_n); + memset(ra_why, 0, sizeof ra_why), ra_bytes = 0, n_age_drops = n_age_bytes = 0; + unlock(); + rt_log("%s", line); +} + // With NX2NAVI_TRACE=perf every 600 frames, after vk_memory.c's vram line: what the live entries' own memory holds. By // kind: decoded ASTC (as BC3 or RGBA8) and other textures whose content the CPU provides, all of which a re-upload // brings back (comes_back_intact), and the GPU-written ones (render targets, textures the GPU drew or copied into), diff --git a/runtime/vk_tonemap.c b/runtime/vk_tonemap.c index 452447c8..49864f13 100644 --- a/runtime/vk_tonemap.c +++ b/runtime/vk_tonemap.c @@ -25,8 +25,10 @@ // (vk_fsr.c), built here with its SQUARE constant set: it filters the same stored values, which are the perceptual // encoding that filter is made for, and squares the result as the linear pass does, so only how the scaled sample is // taken differs. bilinear keeps the linear pass. The linear pass also stays where nothing is enlarged (the filter only -// enlarges, and at the same size the linear pass copies each texel exactly) and where the caller says another -// upscaler's picture replaces this one (vk_exec_draw.c untranslated_dispatch_fallback: DLSS or FSR 3.1, vk_dlaa.c). +// enlarges) and where the caller says another upscaler's picture replaces this one (vk_exec_draw.c +// untranslated_dispatch_fallback: DLSS or FSR 3.1, vk_dlaa.c). A scene as large as the picture (scene_share = 1, the +// menu's Native: widescreen.c plain_share) is not scaled at all: the linear pass samples it with a nearest-texel +// sampler, so each output texel is its source texel squared, whatever the filtering hardware's sub-texel precision. // The movie's frames do not come through here: the caller blits those (RGBA8) itself. static struct { int tried, ok; @@ -37,8 +39,9 @@ static struct { VkPipeline easu; // scale with the edge-adaptive filter, then square (shaders/fsr_easu.comp) int easu_tried; int filter_logs; // lines tm_filter_log has written - int filter_last; // what its last line named: 1 the linear filter, 2 the edge-adaptive one + int filter_last; // what its last line named: 1 the linear filter, 2 the edge-adaptive one, 3 a copy 1:1 VkSampler sampler; + VkSampler nearest; // the same with nearest-texel filtering: a source of the output's size (copy 1:1) struct { VkImage img; VkFormat f; VkImageView view; } views[16]; int nviews; VkImage out; // the pass's own output (RGBA16F, the target's size); the caller blits it in @@ -102,8 +105,11 @@ static int tm_init(void) { int ok = TM.push && vkCreateDescriptorSetLayout(c->device, &dci, NULL, &TM.dsl) == VK_SUCCESS; VkPipelineLayoutCreateInfo li = {.sType = VK_STRUCTURE_TYPE_PIPELINE_LAYOUT_CREATE_INFO, .setLayoutCount = 1, .pSetLayouts = &TM.dsl, .pushConstantRangeCount = 1, .pPushConstantRanges = &pr}; + VkSamplerCreateInfo ni = si; + ni.magFilter = ni.minFilter = VK_FILTER_NEAREST; ok = ok && vkCreatePipelineLayout(c->device, &li, NULL, &TM.layout) == VK_SUCCESS && vkCreateSampler(c->device, &si, NULL, &TM.sampler) == VK_SUCCESS && + vkCreateSampler(c->device, &ni, NULL, &TM.nearest) == VK_SUCCESS && vkCreateShaderModule(c->device, &mi, NULL, &m) == VK_SUCCESS; if (ok) { VkComputePipelineCreateInfo ci = {.sType = VK_STRUCTURE_TYPE_COMPUTE_PIPELINE_CREATE_INFO, @@ -148,55 +154,50 @@ static VkPipeline tm_easu(void) { // One log line when the filter the pass scales with changes (the setting, DLSS or FSR switched on or off, a scene as // large as the picture), 8 at most: a scene whose size changes from frame to frame can cross the picture's size often. static void tm_filter_log(int easu, int edge, uint32_t sw, uint32_t sh, uint32_t dw, uint32_t dh) { - int now = easu ? 2 : 1; + int same = sw == dw && sh == dh, now = easu ? 2 : same && edge ? 3 : 1; if (TM.filter_last == now || TM.filter_logs >= 8) return; TM.filter_last = now, TM.filter_logs++; if (easu) rt_log("vk_exec: upscaler stand-in scales %ux%u to %ux%u with the edge-adaptive filter " "(NX2NAVI_STANDIN_FILTER=bilinear: the linear one)", sw, sh, dw, dh); + else if (now == 3) + rt_log("vk_exec: upscaler stand-in copies %ux%u 1:1, decoded and not scaled: the scene is the picture's size", sw, + sh); else rt_log("vk_exec: upscaler stand-in scales %ux%u to %ux%u with the linear filter: %s", sw, sh, dw, dh, !edge ? "DLSS or FSR scales the scene and its picture goes over this one" - : dw < sw || dh < sh || (dw == sw && dh == sh) ? "nothing to enlarge" + : dw < sw || dh < sh ? "nothing to enlarge" : strcmp(cfg_value("NX2NAVI_STANDIN_FILTER"), "easu") ? "NX2NAVI_STANDIN_FILTER asks for it" : "the edge-adaptive pass is not there"); } -// Views of the images the pass samples: the fallback blit's scratch images, and a game image the pass reads as it is. -// A view goes with its image (vk_tonemap_image_gone: the scratch image's replacement, vk_exec_image_gone for a game -// image), so a new image that gets the old handle never finds the old image's view. Kept before for the process, with -// a game image's view left naming a destroyed image, and no view at all past sixteen. -static PlatMutex tm_view_lock = PLAT_MUTEX_INIT; // the replay records, a texture's destroy (any thread) takes views away +// Views of the images the pass samples, kept as long as the image: vk_tonemap_forget drops those of an image +// fallback_scratch frees, and those of a game image the pass read as it is when the game's texture goes +// (vk_exec_image_gone), which also gives their places back. Without the second, a game image's view stayed naming a +// destroyed image whose handle a new image could get. The texture's destroy runs on any thread, hence the lock. +static PlatMutex tm_view_lock = PLAT_MUTEX_INIT; + +void vk_tonemap_forget(VkImage src) { + plat_mutex_lock(&tm_view_lock); + for (int i = 0; i < TM.nviews;) + if (TM.views[i].img == src) vk_defer_view(TM.views[i].view), TM.views[i] = TM.views[--TM.nviews]; + else i++; + plat_mutex_unlock(&tm_view_lock); +} static VkImageView tm_view(VkImage img, VkFormat f) { plat_mutex_lock(&tm_view_lock); - for (int i = 0; i < TM.nviews; i++) - if (TM.views[i].img == img && TM.views[i].f == f) { - VkImageView v = TM.views[i].view; - plat_mutex_unlock(&tm_view_lock); - return v; - } VkImageView v = VK_NULL_HANDLE; + for (int i = 0; i < TM.nviews && !v; i++) + if (TM.views[i].img == img && TM.views[i].f == f) v = TM.views[i].view; VkImageViewCreateInfo vi = {.sType = VK_STRUCTURE_TYPE_IMAGE_VIEW_CREATE_INFO, .image = img, .viewType = VK_IMAGE_VIEW_TYPE_2D, .format = f, .subresourceRange = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 1, 0, 1}}; - if (TM.nviews < 16 && vkCreateImageView(vk_ctx()->device, &vi, NULL, &v) == VK_SUCCESS) + if (!v && TM.nviews < 16 && vkCreateImageView(vk_ctx()->device, &vi, NULL, &v) == VK_SUCCESS) TM.views[TM.nviews].img = img, TM.views[TM.nviews].f = f, TM.views[TM.nviews].view = v, TM.nviews++; plat_mutex_unlock(&tm_view_lock); return v; } -void vk_tonemap_image_gone(VkImage img) { - plat_mutex_lock(&tm_view_lock); - for (int i = 0; i < TM.nviews;) - if (TM.views[i].img == img) { - vk_defer_view(TM.views[i].view); // a submission being recorded may sample it - TM.views[i] = TM.views[--TM.nviews]; - } else { - i++; - } - plat_mutex_unlock(&tm_view_lock); -} - // Records the pass from src (sampled, sw x sh) into the pass's own RGBA16F image of dw x dh and returns it (the caller // blits it into the target); VK_NULL_HANDLE when it cannot (the caller blits the source instead). edge: the picture is // what the screen shows, so the edge-adaptive filter may scale it (0: another upscaler's picture replaces it). @@ -218,7 +219,9 @@ VkImage vk_tonemap_record(VkCommandBuffer cb, VkImage src, VkFormat sf, uint32_t VkDependencyInfo dep = {.sType = VK_STRUCTURE_TYPE_DEPENDENCY_INFO, .memoryBarrierCount = 1, .pMemoryBarriers = &mb}; vkCmdPipelineBarrier2(cb, &dep); vkCmdBindPipeline(cb, VK_PIPELINE_BIND_POINT_COMPUTE, easu ? easu : TM.pipe); - VkDescriptorImageInfo ii[2] = {{TM.sampler, sv, VK_IMAGE_LAYOUT_GENERAL}, {VK_NULL_HANDLE, dv, VK_IMAGE_LAYOUT_GENERAL}}; + // the same size both ways: each output texel takes its own source texel (see the top of the file) + VkSampler smp = !easu && dw == sw && dh == sh ? TM.nearest : TM.sampler; + VkDescriptorImageInfo ii[2] = {{smp, sv, VK_IMAGE_LAYOUT_GENERAL}, {VK_NULL_HANDLE, dv, VK_IMAGE_LAYOUT_GENERAL}}; VkWriteDescriptorSet w[2] = { {.sType = VK_STRUCTURE_TYPE_WRITE_DESCRIPTOR_SET, .dstBinding = 0, .descriptorCount = 1, .descriptorType = VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER, .pImageInfo = &ii[0]}, diff --git a/runtime/vk_vma_budget.c b/runtime/vk_vma_budget.c index b70a849d..3a7a887d 100644 --- a/runtime/vk_vma_budget.c +++ b/runtime/vk_vma_budget.c @@ -1,5 +1,5 @@ // Video memory by the heaps' live budgets, for every allocation through VMA (vk.h maps vmaCreateImage, vmaCreateBuffer -// and vmaAllocateMemory here). +// and vmaAllocateMemory here, and the frees that go with them), and the allocations' sums by owner ("owners", below). // // An RX 580 (8 GiB, AMD 23.10) has a 6 GiB device-local heap and a 2 GiB device-local, host-visible one. The heap limits // of vk_device.c (vram_heap_limits) are fixed when the device is made: each heap's budget then, less a tenth for the @@ -19,6 +19,7 @@ // headroom's anchors, vk_headroom.c) are made as asked. #define VK_VMA_BUDGET_IMPL #include +#include #include "vk.h" #include "cfg.h" #include "host.h" @@ -83,42 +84,121 @@ static uint32_t all_bits(VmaAllocator vma, uint32_t bits) { return mp->memoryTypeCount >= 32 ? ~0u : (1u << mp->memoryTypeCount) - 1; } +// ---- owners +// Every allocation made through the wrappers below is counted under the owner of the file that made it, the device-local +// heaps apart from the others, and leaves the count when vk.h's vmaDestroyImage, vmaDestroyBuffer or +// vmaFreeMemory frees it. The allocation's user data (unused otherwise) holds 1 + owner * 2 + local, so the free knows +// where it was counted; one made before its owner was known, or by a file that does not include vk.h, has none and is +// never counted. The memory report (vk_memory.c owner_log) reads the sums: a category that only climbs over a session +// is where a leak shows. A count costs one VMA info query and two atomic adds per allocation and per free. +static volatile int64_t own_bytes[VK_OWN_N][2]; + +static int owner_of(const char *file) { + static const struct { const char *name; int owner; } map[] = { + {"vk_texture.c", VK_OWN_TEXTURES}, {"vk_memory.c", VK_OWN_POOLS}, // mirrors, physical pools, staging + {"vk_dlaa.c", VK_OWN_UPSCALER}, {"vk_fsr.c", VK_OWN_UPSCALER}, {"vk_dlssg.c", VK_OWN_UPSCALER}, + {"vk_exec_draw.c", VK_OWN_REPLAY}, {"vk_exec.c", VK_OWN_REPLAY}, {"vk_exec_xfer.c", VK_OWN_REPLAY}, + {"vk_mesh_fb.c", VK_OWN_REPLAY}, {"vk_usnap.c", VK_OWN_USNAP}, {"vk_gpa.c", VK_OWN_GPA}, + {"vk_astc.c", VK_OWN_ASTC}, {"gfx_present.c", VK_OWN_PRESENT}, {"vk_hdr.c", VK_OWN_PRESENT}, + {"vk_tonemap.c", VK_OWN_PRESENT}, {"vk_aa.c", VK_OWN_PRESENT}, {"vk_color.c", VK_OWN_PRESENT}, + {"vk_headroom.c", VK_OWN_HEADROOM}, + }; + if (!file) return VK_OWN_OTHER; + const char *b = file; + for (const char *p = file; *p; p++) + if (*p == '/' || *p == '\\') b = p + 1; + for (unsigned i = 0; i < sizeof map / sizeof *map; i++) + if (!strcmp(b, map[i].name)) return map[i].owner; + return VK_OWN_OTHER; +} + +static void own_add(VmaAllocator vma, VmaAllocation a, const char *file) { + if (!a) return; + VmaAllocationInfo i; + vmaGetAllocationInfo(vma, a, &i); + const VkPhysicalDeviceMemoryProperties *mp; + vmaGetMemoryProperties(vma, &mp); + int local = i.memoryType < mp->memoryTypeCount && + (mp->memoryHeaps[mp->memoryTypes[i.memoryType].heapIndex].flags & VK_MEMORY_HEAP_DEVICE_LOCAL_BIT); + int owner = owner_of(file); + vmaSetAllocationUserData(vma, a, (void *)(uintptr_t)(1 + owner * 2 + local)); + __atomic_add_fetch(&own_bytes[owner][local], (int64_t)i.size, __ATOMIC_RELAXED); +} + +static void own_sub(VmaAllocator vma, VmaAllocation a) { + if (!a) return; + VmaAllocationInfo i; + vmaGetAllocationInfo(vma, a, &i); + uintptr_t tag = (uintptr_t)i.pUserData; + if (!tag || tag > VK_OWN_N * 2) return; + __atomic_sub_fetch(&own_bytes[(tag - 1) / 2][(tag - 1) & 1], (int64_t)i.size, __ATOMIC_RELAXED); +} + +uint64_t vk_vma_owner_bytes(int owner, int local) { + if ((unsigned)owner >= VK_OWN_N) return 0; + int64_t v = __atomic_load_n(&own_bytes[owner][local != 0], __ATOMIC_RELAXED); + return v > 0 ? (uint64_t)v : 0; +} + +void vk_vma_destroy_image(VmaAllocator vma, VkImage img, VmaAllocation a) { + own_sub(vma, a); + vmaDestroyImage(vma, img, a); +} + +void vk_vma_destroy_buffer(VmaAllocator vma, VkBuffer buf, VmaAllocation a) { + own_sub(vma, a); + vmaDestroyBuffer(vma, buf, a); +} + +void vk_vma_free_memory(VmaAllocator vma, VmaAllocation a) { + own_sub(vma, a); + vmaFreeMemory(vma, a); +} + +static VkResult made(VkResult r, VmaAllocator vma, VmaAllocation *a, const char *file) { + if (r == VK_SUCCESS) own_add(vma, *a, file); + return r; +} + VkResult vk_vma_create_image(VmaAllocator vma, const VkImageCreateInfo *ici, const VmaAllocationCreateInfo *ai, VkImage *img, - VmaAllocation *a, VmaAllocationInfo *info) { + VmaAllocation *a, VmaAllocationInfo *info, const char *file) { if (!ai->pool) { uint32_t bits = all_bits(vma, ai->memoryTypeBits), nb = budget_bits(vma, bits); if (nb != bits) { VmaAllocationCreateInfo b = *ai; b.memoryTypeBits = nb; - if (vmaCreateImage(vma, ici, &b, img, a, info) == VK_SUCCESS) return note_moved(), VK_SUCCESS; + if (vmaCreateImage(vma, ici, &b, img, a, info) == VK_SUCCESS) + return note_moved(), made(VK_SUCCESS, vma, a, file); } } - return vmaCreateImage(vma, ici, ai, img, a, info); + return made(vmaCreateImage(vma, ici, ai, img, a, info), vma, a, file); } VkResult vk_vma_create_buffer(VmaAllocator vma, const VkBufferCreateInfo *bci, const VmaAllocationCreateInfo *ai, - VkBuffer *buf, VmaAllocation *a, VmaAllocationInfo *info) { + VkBuffer *buf, VmaAllocation *a, VmaAllocationInfo *info, const char *file) { if (!ai->pool) { uint32_t bits = all_bits(vma, ai->memoryTypeBits), nb = budget_bits(vma, bits); if (nb != bits) { VmaAllocationCreateInfo b = *ai; b.memoryTypeBits = nb; - if (vmaCreateBuffer(vma, bci, &b, buf, a, info) == VK_SUCCESS) return note_moved(), VK_SUCCESS; + if (vmaCreateBuffer(vma, bci, &b, buf, a, info) == VK_SUCCESS) + return note_moved(), made(VK_SUCCESS, vma, a, file); } } - return vmaCreateBuffer(vma, bci, ai, buf, a, info); + return made(vmaCreateBuffer(vma, bci, ai, buf, a, info), vma, a, file); } VkResult vk_vma_allocate_memory(VmaAllocator vma, const VkMemoryRequirements *req, const VmaAllocationCreateInfo *ai, - VmaAllocation *a, VmaAllocationInfo *info) { + VmaAllocation *a, VmaAllocationInfo *info, const char *file) { uint32_t bits = req->memoryTypeBits & all_bits(vma, ai->memoryTypeBits); if (!ai->pool && (bits & (bits - 1))) { // more than one type: one alone is asked for on purpose uint32_t nb = budget_bits(vma, bits); if (nb != bits) { VkMemoryRequirements r = *req; r.memoryTypeBits = nb; - if (vmaAllocateMemory(vma, &r, ai, a, info) == VK_SUCCESS) return note_moved(), VK_SUCCESS; + if (vmaAllocateMemory(vma, &r, ai, a, info) == VK_SUCCESS) + return note_moved(), made(VK_SUCCESS, vma, a, file); } } - return vmaAllocateMemory(vma, req, ai, a, info); + return made(vmaAllocateMemory(vma, req, ai, a, info), vma, a, file); } diff --git a/runtime/widescreen.c b/runtime/widescreen.c index 14b95c7c..65aab1e7 100644 --- a/runtime/widescreen.c +++ b/runtime/widescreen.c @@ -79,7 +79,9 @@ static unsigned picture_height(void) { // While DLSS is on, the scene is the DLSS mode's share of the picture (NVIDIA's ratios: DLAA all of it, Quality 2/3, // Balanced 0.58, Performance 1/2, Ultra Performance 1/3) instead of what the game's dynamic-resolution controller // picks. The share is written to +240 at every change of the setting (the menu switches the mode live); the set goes on -// RES_FIXED_AFTER presents after the first, once the controller has filled the fields the fixed path keeps. +// RES_FIXED_AFTER presents after the first, once the controller has filled the fields the fixed path keeps. Without +// DLSS or FSR the share is NX2NAVI_SCENE_SHARE's (plain_share): the game's own most unless set, all of the picture +// for the menu's Native. #define RES_FIXED_AFTER 120 // The share follows the upscaler that runs, not the setting: DLSS starts only after a restart and only where NGX came up // (vk_dlaa_available), and FSR 3.1 only where its library loads (vk_dlaa_fsr3), so a mode picked in the menu, or one @@ -114,8 +116,13 @@ static double mode_share(void) { : !strcmp(m, "ultra_performance") ? 1.0 / 3 : 1.0; } +// DLSS or FSR 3.1 scales the scene in this run: the setting is on and the upscaler came up. +static int upscaler_runs(void) { + return vk_dlaa_wanted() && cfg_bool("NX2NAVI_DLAA") && (vk_dlaa_fsr3() || vk_dlaa_available()); +} + static double dlss_share(void) { - if (!vk_dlaa_wanted() || !cfg_bool("NX2NAVI_DLAA") || (!vk_dlaa_fsr3() && !vk_dlaa_available())) return 0; + if (!upscaler_runs()) return 0; double s = mode_share(); // Never more than the texture heap the game has was made for (fit_share). A mode chosen in the menu that renders // more than the start's did gets its size after a restart; a heap at its limit never holds the whole picture. @@ -132,16 +139,26 @@ static double dlss_share(void) { return s; } +// The share without DLSS or FSR 3.1 (NX2NAVI_SCENE_SHARE, 1 at most): PLAIN_SHARE unset, 1 for the menu's Native +// (ui/menu.cpp row_upscaling: the scene at the picture's size, which the stand-in then only decodes, vk_tonemap.c), +// 0 for the game's dynamic resolution, as with no share at all. Read at every call, so the menu's choice applies at +// once within the heap the start made (fit_share). +static double plain_share(void) { + const char *e = cfg_get("NX2NAVI_SCENE_SHARE"); + double s = e && *e ? atof(e) : PLAIN_SHARE; + return s > 1 ? 1 : s; +} + static double scene_share(void) { - static double plain = -1; - if (plain < 0) { - const char *e = cfg_get("NX2NAVI_SCENE_SHARE"); // 0: the game's dynamic resolution, as with no share at all - plain = e && *e ? atof(e) : PLAIN_SHARE; - } - double s = dlss_share(); + double s = dlss_share(), plain = plain_share(); return s > 0 ? s : plain > fit_share() ? fit_share() : plain; // NX2NAVI_SCENE_SHARE above the heap the game has } +// The share the settings ask for, before the heap's limit: the upscaler's mode while one runs, else plain_share. +static double wanted_share(void) { + return upscaler_runs() ? mode_share() : plain_share(); +} + static int picture_size(unsigned *ow, unsigned *oh, int quiet); // The picture's pixels over the game's own 2560x1440. @@ -151,6 +168,14 @@ static double picture_ratio(void) { return (double)ow * oh / ((double)GAME_W * GAME_H); } +// The most share of the picture the application pool has room for at the picture the settings ask for (the heap +// grows with the share's square, see heap_share). Computed anew at each call, so the menu can tell whether a restart +// would give a larger scene (wide_scene_restart). +static double share_most(void) { + double room = (double)mem_heap_extra_max_kib() / GAME_DTA_KIB + 1; // the heap's most over the shipped one + return sqrt(room / picture_ratio()) * PLAIN_SHARE; +} + // The largest share of the picture the scene can have in this run, which the texture heap is made for // (wide_heap_extra_kib). Decided once, at the first call, which is the start (memory.c sizes the application pool // from it before any device exists): the game's own most, PLAIN_SHARE, or more where the settings at the start ask @@ -164,13 +189,10 @@ static double picture_ratio(void) { static double heap_share(void) { static double share = -1; if (share < 0) { - const char *e = cfg_get("NX2NAVI_SCENE_SHARE"); - double s = e && *e ? atof(e) : PLAIN_SHARE; + double s = plain_share(); if (s < PLAIN_SHARE) s = PLAIN_SHARE; if (rt_patch_set("res_fixed") >= 0 && cfg_bool("NX2NAVI_DLAA") && mode_share() > s) s = mode_share(); - if (s > 1) s = 1; - double room = (double)mem_heap_extra_max_kib() / GAME_DTA_KIB + 1; // the heap's most over the shipped one - double most = sqrt(room / picture_ratio()) * PLAIN_SHARE; + double most = share_most(); if (s > most) { if (most < PLAIN_SHARE) most = PLAIN_SHARE; // wide_heap_extra_kib says what a picture this large lacks rt_log("widescreen: the scene can be %.0f%% of the picture at most, not %.0f%%: the game's render-target " @@ -189,6 +211,36 @@ static double heap_share(void) { // 810-line scene, three quarters of the picture), so there the share goes in as it is: wide_init sets the factor. static double scale_k = 1.0; +// One line for the scene in effect: its size, the picture's and how the scene reaches the picture's size. Written when +// the fixed share first takes over from the game's dynamic resolution and again when the share or the way it is scaled +// changes (a setting changed in the menu), not at every settings change. The scene's size is the share of the +// picture's as the game sizes it (a 50% scene of 5120x2160 is 2560x1080). +static void scene_log(double s, int first) { + static double last_s = -1; + static int last_how = -1; + // 1 DLSS, 2 FSR, 3 not scaled (the stand-in decodes 1:1), 4 the stand-in's edge-adaptive filter, 5 its linear one + int how = dlss_share() > 0 ? (vk_dlaa_fsr3() ? 2 : 1) : s >= 1 ? 3 + : !strcmp(cfg_value("NX2NAVI_STANDIN_FILTER"), "easu") ? 4 : 5; + if (!first && how == last_how && fabs(s - last_s) < 0.001) return; + last_s = s, last_how = how; + static const char *const hows[] = { + "", "DLSS scales it to the picture", "AMD FSR scales it to the picture", + "native: no upscaling, the upscaler's stand-in only decodes it 1:1", + "the upscaler's stand-in enlarges it with the edge-adaptive filter (no DLSS or FSR)", + "the upscaler's stand-in enlarges it with the linear filter (no DLSS or FSR)"}; + unsigned pw, ph; + wide_picture(&pw, &ph); + rt_log("widescreen: scene %ux%u, picture %ux%u (%.0f%% of it): %s%s", (unsigned)lround(pw * s), + (unsigned)lround(ph * s), pw, ph, s * 100, hows[how], first ? " (the game's dynamic resolution off)" : ""); + // The menu's Native, or another share, larger than the heap the start made: say why the scene is smaller + double want = wanted_share(); + if (how >= 3 && want > s + 0.001) + rt_log("widescreen: scene_share asks for %.0f%% of the picture; the game's render-target heap %s", want * 100, + heap_unserved ? "is the shipped one (the start's log says why the grown one was not served)" + : wide_scene_restart() ? "was sized at the start for less: the next start gives it" + : "is at its limit for this picture (the start's log says so)"); +} + void wide_present(void) { static int set = -2; static uint64_t presents; @@ -196,17 +248,37 @@ void wide_present(void) { if (set == -2) set = rt_patch_set("res_fixed"); if (set < 0) return; unsigned v = cfg_version(); - if (v != seen) { - seen = v; - float s = (float)scene_share(), page_s = (float)(s * scale_k); - if (s > 0) memcpy((uint8_t *)rt_patch_page() + 240, &page_s, 4); - if (s <= 0 && rt_patch_enabled(set)) rt_patch_switch(set, 0); - else if (rt_patch_enabled(set)) rt_log("widescreen: render scale %.0f%% of the picture", s * 100); - } - if (++presents == RES_FIXED_AFTER && scene_share() > 0) { - rt_patch_switch(set, 1); - rt_log("widescreen: render scale %.0f%% of the picture (the game's dynamic resolution off)", scene_share() * 100); + int at = ++presents == RES_FIXED_AFTER, due = presents >= RES_FIXED_AFTER; + if (v == seen && !at) return; + seen = v; + double s = scene_share(); + float page_s = (float)(s * scale_k); + if (s > 0) memcpy((uint8_t *)rt_patch_page() + 240, &page_s, 4); + int on = rt_patch_enabled(set); + if (s <= 0) { + if (on) rt_patch_switch(set, 0); + return; } + // On at RES_FIXED_AFTER, once the controller has filled its fields; later too, after a share of 0 (the game's + // dynamic resolution) gave way to a fixed one + if (!due) return; + if (!on) rt_patch_switch(set, 1); + scene_log(s, !on); +} + +int wide_scene_settable(void) { return rt_patch_set("res_fixed") >= 0; } + +double wide_scene_share(void) { return scene_share(); } + +double wide_scene_wanted(void) { return wanted_share(); } + +// The settings ask for a larger scene than the heap the start made holds, and the next start's heap holds more: the +// share it asks for within what the application pool has room for at the picture the settings ask for (share_most). +// A heap the game was not served (heap_unserved) stays the shipped one at the next start too, as far as is known here. +int wide_scene_restart(void) { + double want = wanted_share(), now = fit_share(), next = share_most(); + if (want > 1) want = 1; + return !heap_unserved && want > now + 0.001 && next > now + 0.001; } // The picture the settings ask for, *ow x *oh; 0 when it is the game's own 2560x1440 (every set stays off). Read at the diff --git a/tests/gpa_check.c b/tests/gpa_check.c index 47a1c24d..7c58c2dd 100644 --- a/tests/gpa_check.c +++ b/tests/gpa_check.c @@ -38,9 +38,10 @@ void rt_log(const char *fmt, ...) { static const uint32_t *table; // the first buffer vk_gpa.c creates static uintptr_t next_handle = 0x1000; -VkResult vmaCreateBuffer(VmaAllocator a, const VkBufferCreateInfo *bi, const VmaAllocationCreateInfo *ai, VkBuffer *buf, - VmaAllocation *alloc, VmaAllocationInfo *info) { - (void)a, (void)ai; +// vk.h sends vmaCreateBuffer and vmaDestroyBuffer to these wrappers (vk_vma_budget.c) +VkResult vk_vma_create_buffer(VmaAllocator a, const VkBufferCreateInfo *bi, const VmaAllocationCreateInfo *ai, VkBuffer *buf, + VmaAllocation *alloc, VmaAllocationInfo *info, const char *file) { + (void)a, (void)ai, (void)file; void *mem = calloc(1, (size_t)bi->size); if (!table) table = mem; *buf = (VkBuffer)(next_handle += 0x1000); diff --git a/tests/scratch_lru_check.c b/tests/scratch_lru_check.c new file mode 100644 index 00000000..f341c191 --- /dev/null +++ b/tests/scratch_lru_check.c @@ -0,0 +1,59 @@ +// runtime/vk_scratch_lru.h: which entry of the stand-in's scratch image cache serves a request (vk_exec_draw.c +// fallback_scratch). Built and run by tests/scratch_lru_check.py; prints each failed check and exits 1 on any. +// The old cache kept its first eight sizes and refused the ninth: the sizes below are those a run walks through when +// the window and the DLSS mode change a few times. +#include +#include + +#include "vk_scratch_lru.h" + +static int fails; +#define CHECK(c, ...) do { if (!(c)) { fails++; printf("FAIL line %d: ", __LINE__); printf(__VA_ARGS__); printf("\n"); } } while (0) + +#define N 8 +static ScrEntry e[N]; + +// A request as fallback_scratch makes it: a hit returns its entry, a miss fills the entry it was given (after freeing +// the evicted image). Returns the entry; *was_hit, *was_evict as scr_pick said. +static int request(uint32_t w, uint32_t h, uint32_t f, uint64_t frame, int *was_hit, int *was_evict) { + int hit, evict, i = scr_pick(e, N, w, h, f, frame, &hit, &evict); + if (!hit) e[i] = (ScrEntry){w, h, f, frame, 1}; + if (was_hit) *was_hit = hit; + if (was_evict) *was_evict = evict; + return i; +} + +int main(void) { + int hit, evict; + // ---- empty: free entries in order, nothing evicted + for (int k = 0; k < N; k++) { + int i = request(1920 + (uint32_t)k, 1080, 37, (uint64_t)k + 1, &hit, &evict); + CHECK(i == k && !hit && !evict, "size %d took entry %d (hit %d, evict %d)", k, i, hit, evict); + } + // ---- a match is a hit and refreshes its frame + int i = request(1923, 1080, 37, 100, &hit, &evict); + CHECK(i == 3 && hit && !evict && e[3].used == 100, "the match was entry %d (hit %d, evict %d, used %llu)", i, hit, + evict, (unsigned long long)e[3].used); + // same size, another format: not a match + i = request(1923, 1080, 43, 101, &hit, &evict); + CHECK(!hit && evict, "another format matched (hit %d, evict %d)", hit, evict); + // ---- the ninth size: the one used longest ago goes (entry 0, frame 1), not the full cache refusing it + CHECK(i == 0, "the other format evicted entry %d, not the oldest (0)", i); + i = request(5120, 2142, 37, 102, &hit, &evict); + CHECK(i == 1 && evict, "the next new size evicted entry %d (evict %d), not entry 1", i, evict); + // the refreshed entry 3 (frame 100) outlives the untouched ones + for (int k = 0; k < 4; k++) request(800 + (uint32_t)k, 600, 37, 103 + (uint64_t)k, &hit, &evict); + CHECK(e[3].live && e[3].w == 1923 && e[3].f == 37, "entry 3, used at frame 100, was evicted before older ones"); + // ---- two images in one frame never evict each other while older entries remain (g8 then g8s of one dispatch) + int a = request(2560, 1440, 37, 200, &hit, &evict); + int b = request(2560, 1440, 43, 200, &hit, &evict); + CHECK(a != b && e[a].live && e[a].w == 2560 && e[a].f == 37, "the second image of a frame took the first's entry"); + CHECK(request(2560, 1440, 37, 201, &hit, NULL) == a && hit, "the first image of the pair was not kept"); + // ---- a freed entry (an image that failed to be made stays not live) is taken before any eviction + e[5].live = 0; + i = request(640, 360, 37, 300, &hit, &evict); + CHECK(i == 5 && !hit && !evict, "a free entry was passed over (entry %d, evict %d)", i, evict); + if (fails) return printf("scratch_lru_check: %d checks failed\n", fails), 1; + printf("scratch_lru_check: all checks passed\n"); + return 0; +} diff --git a/tests/scratch_lru_check.py b/tests/scratch_lru_check.py new file mode 100644 index 00000000..a364bc74 --- /dev/null +++ b/tests/scratch_lru_check.py @@ -0,0 +1,40 @@ +"""Build and run tests/scratch_lru_check.c: runtime/vk_scratch_lru.h, which entry of the upscaler stand-in's scratch +image cache serves a new size (vk_exec_draw.c fallback_scratch): a free one, else the one used longest ago. + +No GPU, no game files, any Python. + +Usage: scratch_lru_check.py +""" +import os +import shutil +import subprocess +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +MSYS_BIN = Path(r'C:\msys64\ucrt64\bin') + + +def compiler(): + """$CC, else MSYS2 UCRT64's clang, else clang on PATH (as tests/diff/common.py picks it, without its Unicorn).""" + if os.environ.get('CC'): + return os.environ['CC'] + if os.name == 'nt' and (MSYS_BIN / 'clang.exe').exists(): + return str(MSYS_BIN / 'clang.exe') + return shutil.which('clang') or 'clang' + + +def main(): + work = ROOT / 'build' / 'test' / 'scratch_lru' + work.mkdir(parents=True, exist_ok=True) + exe = work / ('scratch_lru_check.exe' if os.name == 'nt' else 'scratch_lru_check') + env = dict(os.environ) + if os.name == 'nt' and MSYS_BIN.exists(): + env['PATH'] = str(MSYS_BIN) + os.pathsep + env.get('PATH', '') + subprocess.check_call([compiler(), '-std=gnu11', '-O1', '-Wall', '-Werror', '-I', str(ROOT / 'runtime'), + str(ROOT / 'tests' / 'scratch_lru_check.c'), '-o', str(exe)], env=env) + return subprocess.run([str(exe)], env=env).returncode + + +if __name__ == '__main__': + sys.exit(main()) diff --git a/tests/tex_release_check.c b/tests/tex_release_check.c new file mode 100644 index 00000000..90cdee7b --- /dev/null +++ b/tests/tex_release_check.c @@ -0,0 +1,267 @@ +// runtime/vk_tex_release.h: how long a released texture is kept, the bytes the released ones may hold, and the clock +// that turns presents into seconds. Built and run by tests/tex_release_check.py; prints each failed check and a table +// of the simulated sessions, and exits 1 on any failure. +// +// The simulation follows vk_texture.c sweep_pressure: once a second (the cache checks every 8 submissions) take-backs +// that fall due find their entry or make it again, drop_aged takes the entries past their hold, and drop_stale takes +// the oldest entries released 2 s ago or earlier while the released bytes are above trl_keep. The streams are shaped +// by the figures of two sessions' logs on an RTX 5090 (vk_tex_release.h). +#include +#include +#include + +#include "vk_tex_release.h" + +static int fails; +#define CHECK(c, ...) do { if (!(c)) { fails++; printf("FAIL line %d: ", __LINE__); printf(__VA_ARGS__); printf("\n"); } } while (0) + +#define MB(n) ((uint64_t)(n) << 20) + +static uint64_t rng_state = 0x243f6a8885a308d3ull; +static uint32_t rnd(uint32_t n) { + rng_state ^= rng_state << 13, rng_state ^= rng_state >> 7, rng_state ^= rng_state << 17; + return (uint32_t)(rng_state >> 33) % n; +} + +// ---- the clock +static TrlClock clk; + +static void clock_checks(void) { + // presents at a rate that changes every few seconds (60 to 240 a second); a tick each second, as rate_tick makes + memset(&clk, 0, sizeof clk); + uint64_t frame = 0, at[4000]; // at[s]: the present count at second s + for (uint32_t s = 0; s < 4000; s++) { + at[s] = frame; + trl_clock_tick(&clk, frame, s); + frame += 60 + (s / 7 % 4) * 60; + } + uint32_t now = 3999; + for (uint32_t s = 0; s < 4000; s += 13) { + uint32_t age = trl_clock_age(&clk, at[s] + 1, now), real = now - s; // a present just after tick s + if (now - s < TRL_CLOCK_N - 1) { + CHECK(age <= real && age + 1 >= real, "present after second %u: %u s old, %u by the clock", s, real, age); + } else { + CHECK(age >= TRL_CLOCK_N - 2, "present after second %u, before the ticks kept: %u by the clock", s, age); + } + } + CHECK(trl_clock_age(&clk, frame + 5, now) == 0, "a present after the last tick is 0 s old"); + CHECK(trl_clock_age(&clk, at[3999], now) == 0, "the last tick's present is 0 s old"); + // the hold in seconds does not depend on the rate: 180 s at 60 or at 240 presents a second + uint32_t s60 = 2500, s240 = 0; // within the ticks kept (the last TRL_CLOCK_N seconds) + for (uint32_t s = 2500; s < 4000; s++) + if ((s / 7 % 4) == 3) { s240 = s; break; } + CHECK(trl_clock_age(&clk, at[s60], s60 + 180) == 180 && trl_clock_age(&clk, at[s240], s240 + 180) == 180, + "180 s are 180 s at any rate"); + // a run shorter than the ring, and an empty clock + TrlClock c2 = {0}; + CHECK(trl_clock_age(&c2, 100, 50) == 0, "no tick yet: 0 s"); + trl_clock_tick(&c2, 0, 0), trl_clock_tick(&c2, 100, 1), trl_clock_tick(&c2, 250, 2); + CHECK(trl_clock_age(&c2, 50, 10) == 9 && trl_clock_age(&c2, 0, 10) == 10 && trl_clock_age(&c2, 251, 10) == 0, + "short run: %u %u %u", trl_clock_age(&c2, 50, 10), trl_clock_age(&c2, 0, 10), trl_clock_age(&c2, 251, 10)); +} + +// ---- the rules +static void rule_checks(void) { + CHECK(trl_hold(TRL_HOLD_S, TRL_STORAGE) == 180 && trl_hold(TRL_HOLD_S, TRL_GPU) == 360, "holds 180 and 360 s"); + CHECK(!trl_aged(0, TRL_STORAGE, 100000), "no hold: nothing ages out"); + CHECK(!trl_aged(180, TRL_STORAGE, 179) && trl_aged(180, TRL_STORAGE, 180), "the hold's edge"); + CHECK(!trl_aged(180, TRL_GPU, 359) && trl_aged(180, TRL_GPU, 360), "GPU content waits twice the hold"); + CHECK(2 * TRL_HOLD_MAX_S < TRL_CLOCK_N - 1, "the clock covers the longest hold of GPU content"); + uint64_t budget = MB(31 * 1024); + // the Balanced session's line at frame 7200: 11786 MiB in use of about 31 GiB, 27.3 GB of 71 GB commit free + CHECK(trl_room(MB(11786), budget, 80, 70981 - 43683, 7098), "RTX 5090 session: room"); + CHECK(!trl_room(MB(23000), budget, 80, 27000, 7098), "card above 70%% of its budget: no room"); + CHECK(!trl_room(MB(5000), budget, 80, 10000, 7098), "commit under twice its low mark: no room"); + CHECK(!trl_room(MB(5000), budget, 80, -1, 7098), "commit not tracked: no room"); + CHECK(!trl_room(MB(1000), 0, 80, 30000, 4096), "no budget known: no room"); + CHECK(!trl_room(MB(1), budget, 5, 30000, 4096), "pressure under the margin: no room"); + // an 8 GiB card in play: 6.3 of 7.5 GiB in use + CHECK(!trl_room(MB(6450), MB(7680), 80, 20000, 4096), "8 GiB card in play: no room"); + CHECK(trl_keep(180, MB(4008), 1) == MB(8016) && trl_keep(180, MB(4008), 0) == MB(4008), "keep: twice with room"); + CHECK(trl_keep(0, MB(4008), 1) == MB(4008), "keep: the bound without the hold, as before"); + CHECK(trl_keep(180, UINT64_MAX, 1) == UINT64_MAX, "keep: no bound stays none"); +} + +// ---- the take-backs of the logs against the hold +// "vk: released textures frame N" lines of the two sessions (2026-11-12): take-backs by the time released, the buckets +// under 2 s, 2-10 s, 10-60 s, 1-5 min, longer, with each bucket's upper edge in seconds (the last open). +static const struct { const char *session; uint64_t n[5]; } TAKEN[] = { + {"low, frame 3600", {46626, 18, 0, 0, 0}}, {"low, frame 7200", {115111, 126, 69, 0, 0}}, + {"low, frame 10800", {88575, 79, 156, 0, 0}}, {"balanced, frame 3600", {22034, 1, 0, 0, 0}}, + {"balanced, frame 7200", {120779, 91, 1, 0, 0}}, +}; +static const uint32_t EDGE[5] = {2, 10, 60, 300, UINT32_MAX}; + +static void taken_back_checks(void) { + uint32_t longest = 0; + for (size_t i = 0; i < sizeof TAKEN / sizeof *TAKEN; i++) + for (int b = 0; b < 5; b++) + if (TAKEN[i].n[b] && EDGE[b] > longest) longest = EDGE[b]; + CHECK(longest == 60, "the longest take-back of the logs lies under %u s", longest); + // nothing the logs took back would have gone past the hold: every take-back came before 60 s, the hold is 180 s + for (int cls = TRL_STORAGE; cls <= TRL_GPU; cls++) + CHECK(!trl_aged(TRL_HOLD_S, cls, longest) && trl_hold(TRL_HOLD_S, cls) >= 3 * longest, + "class %d: hold %u s against take-backs up to %u s", cls, trl_hold(TRL_HOLD_S, cls), longest); +} + +// ---- simulated sessions +typedef struct { + uint32_t rel; // second of the release + uint32_t back; // second the game takes it back; 0: never + uint32_t bytes; + uint8_t cls, gone, taken; +} Ent; + +typedef struct { + const char *name; + uint64_t cap; // bytes + uint32_t hold; // s +} Policy; + +typedef struct { + uint64_t remade, remade_bytes, dropped_age, dropped_bound, peak, end, at_burst_end; +} Result; + +#define MAXE 400000 +static Ent ents[MAXE]; +static uint32_t nents; +static uint32_t backs[MAXE], nbacks; // the entries the game takes back, by the second it does + +static int back_cmp(const void *a, const void *b) { + uint32_t x = ents[*(const uint32_t *)a].back, y = ents[*(const uint32_t *)b].back; + return x < y ? -1 : x > y; +} + +// A stream: `burst_s` seconds of area changes at burst_mbs MiB/s of releases the game keeps released, then calm play to +// end_s at calm_mbs. Sizes 0.1 to 1.9 MiB (the Balanced session held 3946 MiB in 7754 entries, half a MiB each); a +// tenth hold content only the GPU made. A share `back_pct` per thousand comes back, after a time drawn from the +// logs' buckets past 2 s (2-10 s and 10-60 s about as often); `late` adds take-backs late_s after the release. +static void make_stream(uint32_t burst_s, double burst_mbs, uint32_t end_s, double calm_mbs, uint32_t back_pml, + uint32_t late_s) { + nents = 0; + double carry = 0; + for (uint32_t s = 1; s < end_s; s++) { + carry += (s <= burst_s ? burst_mbs : calm_mbs) * 1048576.0; + while (carry > 0 && nents < MAXE) { + Ent *e = &ents[nents++]; + memset(e, 0, sizeof *e); + e->rel = s, e->bytes = (uint32_t)(MB(1) / 10 + rnd((uint32_t)MB(18) / 10)); + e->cls = rnd(10) == 0 ? TRL_GPU : TRL_STORAGE; + if (rnd(1000) < back_pml) e->back = s + (rnd(2) ? 2 + rnd(8) : 10 + rnd(50)); + if (late_s && rnd(1000) < back_pml) e->back = s + late_s; + carry -= e->bytes; + } + } + nbacks = 0; + for (uint32_t i = 0; i < nents; i++) + if (ents[i].back) backs[nbacks++] = i; + qsort(backs, nbacks, sizeof *backs, back_cmp); +} + +static Result run(const Policy *p, int room, uint32_t burst_s, uint32_t end_s) { + Result r = {0}; + for (uint32_t i = 0; i < nents; i++) ents[i].gone = ents[i].taken = 0; + uint64_t held = 0; + uint32_t next = 0, head = 0; // next: the first entry not released yet; head: the oldest that may still be held + uint32_t nb = 0; // the next take-back in backs[] + for (uint32_t s = 1; s < end_s; s++) { + while (next < nents && ents[next].rel <= s) held += ents[next++].bytes; + for (; nb < nbacks && ents[backs[nb]].back <= s; nb++) { // the take-backs due now + Ent *e = &ents[backs[nb]]; + if (e->gone) r.remade++, r.remade_bytes += e->bytes; + else held -= e->bytes; + e->taken = 1; + } + for (uint32_t i = head; p->hold && i < next; i++) { // drop_aged + Ent *e = &ents[i]; + if (e->gone || e->taken) continue; + if (s - e->rel < p->hold) break; // in release order: the rest is younger + if (trl_aged(p->hold, e->cls, s - e->rel)) e->gone = 1, held -= e->bytes, r.dropped_age++; + } + uint64_t keep = trl_keep(p->hold, p->cap, room); + for (uint32_t i = head; i < next && held > keep; i++) { // drop_stale, oldest first, released 2 s ago or more + Ent *e = &ents[i]; + if (e->gone || e->taken || s - e->rel < 2) continue; + e->gone = 1, held -= e->bytes, r.dropped_bound++; + } + while (head < next && (ents[head].gone || ents[head].taken)) head++; + if (held > r.peak) r.peak = held; + if (s == burst_s) r.at_burst_end = held; + } + r.end = held; + return r; +} + +static void print_result(const char *scenario, const Policy *p, int room, const Result *r) { + printf(" %-34s %-12s room %d: made again %5llu (%6.0f MiB), dropped by age %6llu, by the bound %6llu; held at " + "the burst's end %5.0f MiB, peak %5.0f MiB, at the end %5.0f MiB\n", + scenario, p->name, room, (unsigned long long)r->remade, r->remade_bytes / 1048576.0, + (unsigned long long)r->dropped_age, (unsigned long long)r->dropped_bound, r->at_burst_end / 1048576.0, + r->peak / 1048576.0, r->end / 1048576.0); +} + +static void session_checks(void) { + // RTX 5090, 31 GiB budget: the size bound a quarter of it (4008 MiB, the logs' figure) + const Policy old_low = {"old low", MB(1024), 0}, balanced = {"old balanced", MB(4008), 0}, + low = {"new low", MB(4008), TRL_HOLD_S}; + // an 8 GiB card: bound 1.9 GiB, no room in play + const Policy b8 = {"old balanced", MB(1920), 0}, l8 = {"new low", MB(1920), TRL_HOLD_S}, + o8 = {"old low", MB(1024), 0}; + struct { const char *name; uint32_t burst_s; double burst_mbs; uint32_t end_s; double calm_mbs; } sc[] = { + // the Balanced session's second line: 3.2 GiB released and kept within 35 s; then 10 minutes in one area + {"area changes 70 s, calm 10 min", 70, 93.0, 670, 1.0}, + // a long session: an area change every few minutes, shown as a slower stream throughout + {"steady 20 MiB/s for 20 min", 1200, 20.0, 1500, 0.5}, + // a teleport tour: 4 minutes at the burst rate, past twice the bound + {"teleports 4 min, calm 10 min", 240, 93.0, 840, 1.0}, + }; + printf("simulated sessions (made again: taken back after its drop; MiB: own memory of released textures)\n"); + for (size_t i = 0; i < sizeof sc / sizeof *sc; i++) { + rng_state = 0x9e3779b97f4a7c15ull + i; + make_stream(sc[i].burst_s, sc[i].burst_mbs, sc[i].end_s, sc[i].calm_mbs, 20, 0); + Result rl = run(&low, 1, sc[i].burst_s, sc[i].end_s), rb = run(&balanced, 1, sc[i].burst_s, sc[i].end_s); + Result ro = run(&old_low, 1, sc[i].burst_s, sc[i].end_s); + Result rl8 = run(&l8, 0, sc[i].burst_s, sc[i].end_s), rb8 = run(&b8, 0, sc[i].burst_s, sc[i].end_s); + Result ro8 = run(&o8, 0, sc[i].burst_s, sc[i].end_s); + print_result(sc[i].name, &old_low, 1, &ro), print_result("", &balanced, 1, &rb), print_result("", &low, 1, &rl); + print_result(" same on an 8 GiB card", &o8, 0, &ro8), print_result("", &b8, 0, &rb8); + print_result("", &l8, 0, &rl8); + // with room the new default makes nothing again, unless a burst passes twice the bound within the hold + uint64_t total = 0; + for (uint32_t k = 0; k < nents && ents[k].rel <= sc[i].burst_s; k++) total += ents[k].bytes; + if (total < 2 * low.cap - MB(256)) + CHECK(rl.remade == 0, "%s: new low made %llu again", sc[i].name, (unsigned long long)rl.remade); + // never more made again than the old Balanced, on either card + CHECK(rl.remade <= rb.remade && rl8.remade <= rb8.remade, "%s: new low %llu/%llu against balanced %llu/%llu", + sc[i].name, (unsigned long long)rl.remade, (unsigned long long)rl8.remade, (unsigned long long)rb.remade, + (unsigned long long)rb8.remade); + // what it holds at the end: what the last hold released, far under the old Balanced's + CHECK(rl.end <= rb.end / 2 || rb.end < MB(256), "%s: new low holds %.0f MiB at the end, balanced %.0f", + sc[i].name, rl.end / 1048576.0, rb.end / 1048576.0); + // the old low's 1 GiB made textures again in every burst, as the log showed + CHECK(ro.remade > 0, "%s: old low made nothing again", sc[i].name); + CHECK(!rl.dropped_bound || total >= 2 * low.cap - MB(256), "%s: new low dropped %llu by the bound with room", + sc[i].name, (unsigned long long)rl.dropped_bound); + } + // a game that takes textures back late: at 150 s nothing goes before (hold 180), at 200 s the hold costs + rng_state = 7; + make_stream(70, 93.0, 670, 1.0, 20, 150); + Result r150 = run(&low, 1, 70, 670); + CHECK(r150.remade == 0, "take-backs at 150 s: new low made %llu again", (unsigned long long)r150.remade); + rng_state = 7; + make_stream(70, 93.0, 670, 1.0, 20, 200); + Result r200 = run(&low, 1, 70, 670); + CHECK(r200.remade > 0, "take-backs at 200 s: the hold should have dropped some first"); + printf(" late take-backs: at 150 s made again %llu, at 200 s %llu (the hold is %u s)\n", + (unsigned long long)r150.remade, (unsigned long long)r200.remade, TRL_HOLD_S); +} + +int main(void) { + clock_checks(); + rule_checks(); + taken_back_checks(); + session_checks(); + if (fails) printf("%d checks failed\n", fails); + else printf("tex_release_check: all checks passed\n"); + return fails ? 1 : 0; +} diff --git a/tests/tex_release_check.py b/tests/tex_release_check.py new file mode 100644 index 00000000..e92eabe3 --- /dev/null +++ b/tests/tex_release_check.py @@ -0,0 +1,41 @@ +"""Build and run tests/tex_release_check.c: runtime/vk_tex_release.h, which says how long a released texture is kept, +how much the released ones may hold past the size bound, and how many seconds ago a present was. The check also runs +simulated sessions shaped by the take-back figures of two players' logs against the old and new defaults. + +No GPU, no game files, any Python. + +Usage: tex_release_check.py +""" +import os +import shutil +import subprocess +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +MSYS_BIN = Path(r'C:\msys64\ucrt64\bin') + + +def compiler(): + """$CC, else MSYS2 UCRT64's clang, else clang on PATH (as tests/diff/common.py picks it, without its Unicorn).""" + if os.environ.get('CC'): + return os.environ['CC'] + if os.name == 'nt' and (MSYS_BIN / 'clang.exe').exists(): + return str(MSYS_BIN / 'clang.exe') + return shutil.which('clang') or 'clang' + + +def main(): + work = ROOT / 'build' / 'test' / 'tex_release' + work.mkdir(parents=True, exist_ok=True) + exe = work / ('tex_release_check.exe' if os.name == 'nt' else 'tex_release_check') + env = dict(os.environ) + if os.name == 'nt' and MSYS_BIN.exists(): + env['PATH'] = str(MSYS_BIN) + os.pathsep + env.get('PATH', '') + subprocess.check_call([compiler(), '-std=gnu11', '-O1', '-Wall', '-Werror', '-I', str(ROOT / 'runtime'), + str(ROOT / 'tests' / 'tex_release_check.c'), '-o', str(exe)], env=env) + return subprocess.run([str(exe)], env=env).returncode + + +if __name__ == '__main__': + sys.exit(main())