Keep the driver's cache of stage libraries warm, link a missing pipeline instead of skipping its draw, and count the texture cache's waits in time.

The preparation builds every stage library once so the driver's disk cache holds it and a library a draw needs later takes under a millisecond. Turning dynamic vertex input off for AMD's RDNA 3 and newer driver changed the dynamic states every library lists, which gave each one another entry in the driver's cache, and nothing built them again: an update carried the prepared shaders over without the driver step. On an RX 7900 XT (26.10.1) all 42,142 prepared libraries were cold, 93 ms a vertex and 242 ms a fragment library on average. With no pipeline recorded, a walk through a village skipped 11,180 draws while their pipelines built and made 94 draws wait 5.5 s in all, four of them 1.0 to 2.0 s; a player's session log shows up to 114,834 draws skipped in 600 frames and 181 waits of 251 ms on average. Skipped ground and walls appear and disappear, and a skipped lighting pass is a black frame with the interface intact.

- The game checks the driver's cache at every start (vk_pipeline.c lw_start): a sample of the prepared fragment libraries is built, and when their median time says the cache is cold the whole prepared list is built into it again on idle-priority threads while the game runs. shader-warm.txt records a warm in progress, so the next start goes on with one a run ended, and stops the tries on a driver that does not keep the libraries. This also covers a driver update and a cache the driver emptied.
- An update that carries the prepared shaders over runs the game's --warm-shaders (tools/make_game.py warm_driver_cache): 11 s with the libraries there, about 8 minutes with all of them cold on that card.
- A draw whose pipeline is missing waits for its fast link from a budget of 6 ms a presented frame, 3 ms at most each (NX2NAVI_PIPE_LINK_MS), before it is skipped: with warm libraries the link is done within that.
- Full-screen draws wait from their 50 ms budget again instead of being skipped (NX2NAVI_PIPE_SKIP = any is the default; fullscreen stays selectable). With warm libraries the wait is a millisecond.

The same walk afterwards: libraries 0.3 and 0.7 ms on average, 1,605 draws skipped, 1,473 of them in the scene's first 600 frames and 8 to 80 in each 600 after, and 19 waits of 60 ms in all. First visits to four other saves (a mountain pass, the Lost Woods, Kokiri Forest, a river) waited 12 to 84 ms in all except the Lost Woods, where mesh shader pipelines, which have no stage libraries, and vertex libraries that need added outputs still compiled once: 23 waits, 2.1 s in all, the longest 776 ms.

The texture cache counted its waits in presents chosen at 60 a second: at 165 presents a second the 1800 that stood for thirty seconds were 11 s, so a released render target went sooner than meant and a texture out of view for 11 s was retired and showed its stand-in for some frames when it came back. The counts are now taken times the present rate over 60 (presents_for), and a texture is retired after two minutes without a sample instead of thirty seconds.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
epicmeme546
2026-11-10 01:36:29 -05:00
parent 8d91667c28
commit e10ef60fd5
9 changed files with 381 additions and 44 deletions

View File

@@ -1372,30 +1372,12 @@ def shader_carry(out):
return (out / 'shader-prepared.bin').is_file() and bool(cache_identities(out))
def step_shaders(out, update=False):
"""Every shader of the game files into the game's shader cache and the driver's own cache, before the first start.
Its output is read as it comes: the counts nx2navi.exe prints become progress events, the other lines go on to
the log, and while nothing moves an event every few seconds keeps saying the step is running and for how long.
update: after a kit update the shaders prepared before stay in use and the game translates them again in the
background (shader_carry), so the update does not wait the minutes a full preparation takes."""
# The game translates with the copy beside it. The compile step refreshes that copy, but a kit that changes only
# the translator runs this step again alone (step_key), which would then prepare with the old one. Only a kit that
# has one: the compile step, which always runs first, stops a build without it.
for d in (KIT / 'bin', KIT / 'build' / 'xlate'):
if (d / XLATE_NAME).is_file():
note_cache_translator(out)
for name in (XLATE_NAME, 'nx2navi_xlate.id'):
copy_if_changed(d / name, out / name)
break
if update and shader_carry(out):
say(' the shaders prepared before this update stay in use; the game translates them again in the background')
if loose_shaders(out) >= PACK_LOOSE_MIN:
pack_shader_cache(out)
return
cmd = [sys.executable, '-u', str(KIT / 'tools' / 'prepare_shaders.py'), str(out), '--romfs', str(out / 'romfs'),
'--exefs', str(out / 'exefs'), '--no-list', '--exe', EXE_NAME]
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, encoding='utf-8',
errors='replace', bufsize=1)
def shader_run(cmd, cwd=None):
"""Runs a shader preparation command and returns its exit code. Its output is read as it comes: the counts
nx2navi.exe prints become progress events, the other lines go on to the log, and while nothing moves an event
every few seconds keeps saying the step is running and for how long."""
proc = subprocess.Popen(cmd, cwd=cwd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True,
encoding='utf-8', errors='replace', bufsize=1)
t0 = time.monotonic()
state = {'phase': None, 'done': 0, 'total': 0, 'moved': t0}
lock = threading.Lock()
@@ -1435,6 +1417,56 @@ def step_shaders(out, update=False):
proc.stdout.close()
stop.set()
beat.join()
return rc
def warm_driver_cache(out):
"""After an update that carries the shaders over: every prepared stage library is built into the graphics driver's
cache again (`nx2navi.exe --warm-shaders`). It takes seconds while the driver still has them. It takes minutes when
the update changed what a library is built with, which gives each one another entry in the driver's cache: without
this run every pipeline a draw first needed then compiled in play, which skipped that draw for some frames or made
it wait (on an RX 7900 XT 100 to 340 ms a library against 3 ms from the cache, after dynamic vertex input was
turned off for that driver). A run that fails is not an error of the build: the game checks the cache at every
start and warms it in the background itself (runtime/vk_pipeline.c lw_start)."""
exe = out / EXE_NAME
if not exe.is_file() or not (out / 'shader-prepared.bin').is_file():
return
say(" the prepared shaders are built into the graphics driver's cache again")
t0 = time.monotonic()
try:
rc = shader_run([str(exe), '--warm-shaders'], cwd=str(out))
except OSError as e:
rc = str(e)
if rc:
say(f' that did not finish ({rc}); the game does it in the background while it runs')
else:
say(f" driver's cache warm in {time.monotonic() - t0:.0f} s")
def step_shaders(out, update=False):
"""Every shader of the game files into the game's shader cache and the driver's own cache, before the first start
(shader_run reports its progress).
update: after a kit update the shaders prepared before stay in use and the game translates them again in the
background (shader_carry), so the update does not wait the minutes a full preparation takes; the driver's cache
gets their stage libraries again (warm_driver_cache)."""
# The game translates with the copy beside it. The compile step refreshes that copy, but a kit that changes only
# the translator runs this step again alone (step_key), which would then prepare with the old one. Only a kit that
# has one: the compile step, which always runs first, stops a build without it.
for d in (KIT / 'bin', KIT / 'build' / 'xlate'):
if (d / XLATE_NAME).is_file():
note_cache_translator(out)
for name in (XLATE_NAME, 'nx2navi_xlate.id'):
copy_if_changed(d / name, out / name)
break
if update and shader_carry(out):
say(' the shaders prepared before this update stay in use; the game translates them again in the background')
if loose_shaders(out) >= PACK_LOOSE_MIN:
pack_shader_cache(out)
warm_driver_cache(out)
return
cmd = [sys.executable, '-u', str(KIT / 'tools' / 'prepare_shaders.py'), str(out), '--romfs', str(out / 'romfs'),
'--exefs', str(out / 'exefs'), '--no-list', '--exe', EXE_NAME]
rc = shader_run(cmd)
if rc:
fail('preparing the shaders failed (the lines above say why); the game still runs, compiling shaders as it '
'meets them. Run the build again to retry this step')