Add the shared pipeline list: mesh pipelines players met are built at the shader preparation, before the first start.

The preparation translates every shader and builds each one's stage library, so a pipeline a draw meets for the first time is linked in 2 to 6 ms. A mesh pipeline has no stage libraries: on an install that never drew it, it compiles whole in play, 0.2 to 1.1 s with its draws left out of the picture (18 of them on a first visit of Kokiri Forest). Whole pipelines were built ahead only from the folder's own pipeline-keys.bin.

tools/pipeline_list.py merge folds the pipeline-keys.bin and shader-answers.bin of game folders that were played into one list file: every key record once, with the answers record of each shader it names. A record names its shaders by input id, which depends on the game's code and the answers alone, and holds the game's pipeline state; what depends on the GPU is derived when the pipeline is built, so a record made on one machine is good on another with the same game version and runtime. Four folders on this PC give 5,432 records, 317 of them mesh pipelines (147 KB of keys).

tools/prepare_shaders.py reads the list (tools/data/pipeline_list.bin, data/pipeline_list.bin in a release folder, or --pipeline-list), adds its shader variants to the seed and writes its records as seed-pipeline-keys.bin. nx2navi.exe --prepare-shaders builds the mesh pipelines among them after the seed is translated and saves the pipeline cache before the stage libraries are built (NX2NAVI_PREPARE_PIPELINES=all builds the vertex pipelines too); --warm-shaders does it again after an update or a new driver, and --prepare-pipelines is that step alone. The pipelines go into the caches and are not kept or added to the folder's own key file, so they cost no start-up time or video memory until a draw meets them. A marker file keeps a driver that crashes on one of them from failing every preparation.

The list is derived from game data and stays out of the repository: check_content.py refuses files recorded from a game folder by their first bytes, and make_kit.py, make_launcher.py and make_release.py carry a list given with --pipeline-list. All three now also carry pipeline_list.py, which prepare_shaders.py imports.

Checked without the game: tests/pipeline_list_test.py, and prepare_shaders.py --no-run on the game files with a list of 317 mesh pipelines (all 266 shader variants found in this game version, 177 new to the seed). The build step in nx2navi.exe compiles and links and has not been run.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
chrg6
2026-11-10 14:52:46 -05:00
parent 682d9c5c50
commit 918d399d55
15 changed files with 627 additions and 14 deletions

View File

@@ -21,6 +21,13 @@ shader-prepared.bin (one record per variant, in priority order) and builds every
pipeline once, so the GPU driver's own disk cache holds them (vk_pipeline.c "stage library prebuild"). The seed files
can be deleted afterwards.
The shared pipeline list (tools/pipeline_list.py: graphics pipelines players met, with the shader variants they name)
is read beside the shader list when there is one: tools/data/pipeline_list.bin, data/pipeline_list.bin in a release
folder, or --pipeline-list. Its variants of shaders this game version has join the seed, and its key records are
written as seed-pipeline-keys.bin, which --prepare-shaders builds into the pipeline caches once the seed is translated
(vk_pipeline.c "the shared pipeline list"; mesh pipelines by default). That file stays in the build directory:
`nx2navi.exe --warm-shaders` and `--prepare-pipelines` build from it again after an update or a new driver.
A driver whose shader compiler crashes on a variant crashes on it in every run (NVIDIA 615.78.08 on an RTX 5080, issue
#30). The translations and shader-prepared.bin are written by then, and the journal the run leaves names the variants
the driver was building (vk_pipeline.c skip_init), so only the driver step runs again (`nx2navi.exe --warm-shaders`), up to
@@ -47,6 +54,7 @@ ROOT = Path(__file__).resolve().parents[1]
while not (ROOT / 'tools' / 'shader_census.py').is_file() and ROOT != ROOT.parent:
ROOT = ROOT.parent
sys.path.insert(0, str(ROOT / 'tools'))
import pipeline_list # noqa: E402
import shader_list # noqa: E402
from shader_census import inventory # noqa: E402
@@ -83,6 +91,44 @@ if not BUNDLED_LIST.is_file():
BUNDLED_LIST = ROOT / 'data' / 'shader_list.tsv' # release folder layout (tools/make_release.py)
BUNDLED_PIPELINES = ROOT / 'tools' / 'data' / 'pipeline_list.bin' # as BUNDLED_LIST: out of the repository
if not BUNDLED_PIPELINES.is_file():
BUNDLED_PIPELINES = ROOT / 'data' / 'pipeline_list.bin'
SEED_KEYS = 'seed-pipeline-keys.bin'
LIST_PRIORITY = 1 # a pipeline list variant's seed priority: with the warm-up rows
def read_pipeline_list(path):
"""(answers records, pipeline-keys.bin image) of the pipeline list at path, or of the bundled one; None without
a list. A file that is not a list stops the run: a release that names one must not go on without it."""
p = Path(path) if path else BUNDLED_PIPELINES
if not p.is_file():
if path:
sys.exit(f'prepare_shaders: no pipeline list at {p}')
return None
got = pipeline_list.unpack(p.read_bytes())
if got is None:
sys.exit(f'prepare_shaders: {p} is not a pipeline list (tools/pipeline_list.py merge writes one)')
return got
def list_variants(answers_recs, have, seen):
"""Seed records for the pipeline list's shader variants: those whose shader (runtime code hash) is in `have` and
whose (hash, answers) the seed does not hold yet. The input id is left zero, as for every seed record: the runtime
finds it, and finds the id the list's key records name, since it depends on the code and the answers alone."""
out, absent = [], 0
for a in answers_recs:
h, packed = a[4:36], a[68:68 + 147]
if h not in have:
absent += 1
continue
if (h, packed) in seen:
continue
seen.add((h, packed))
out.append(a[:36] + bytes(32) + packed + bytes([LIST_PRIORITY]))
return out, absent
def read_list(path):
if path:
return Path(path).read_text(encoding='utf-8')
@@ -190,6 +236,8 @@ def main():
ap.add_argument('--list', help='a shader_list.tsv instead of tools/data/shader_list.tsv or refs/MacNativeGames')
ap.add_argument('--no-run', action='store_true', help='write the seed files only')
ap.add_argument('--no-list', action='store_true', help='default variants only, without a shared shader list')
ap.add_argument('--pipeline-list', help='a pipeline list instead of tools/data/pipeline_list.bin (tools/pipeline_list.py)')
ap.add_argument('--no-pipeline-list', action='store_true', help='no shared pipeline list, also when one is bundled')
ap.add_argument('--all-shaders', action='store_true',
help='also the default variants SLOW_UNUSED leaves out (an hour more of GPU driver builds)')
ap.add_argument('--exe', default='nx2navi.exe' if os.name == 'nt' else 'nx2navi',
@@ -235,8 +283,17 @@ def main():
rec = struct.pack('<I', stage) + h + bytes(32) + packed + bytes([prio.get(key, 2) | (DEFAULT_FLAG if default else 0)])
assert len(rec) == ANS_REC
recs.append((prio.get(key, 2), rec))
recs.sort(key=lambda r: r[0]) # stable: list order within a priority
out = a.build.resolve()
plist = None if a.no_pipeline_list else read_pipeline_list(a.pipeline_list)
(out / SEED_KEYS).unlink(missing_ok=True) # a list this build no longer has
if plist:
extra, absent = list_variants(plist[0], {h for h, _ in progs.values()}, seen)
recs += [(LIST_PRIORITY, r) for r in extra]
(out / SEED_KEYS).write_bytes(plist[1])
n_keys = len(pipeline_list.read_keys(plist[1])[3])
print(f'pipeline list: {n_keys} pipelines; of the {len(plist[0])} shader variants they name, {len(extra)} join the '
f'seed, {absent} are of shaders {a.version} does not have, the others are in it already', flush=True)
recs.sort(key=lambda r: r[0]) # stable: list order within a priority
with open(out / 'seed-programs.bin', 'wb') as f:
f.write(b'NX2NSPRG' + struct.pack('<II', 1, 0))
for _, body in progs.values():