From 10ea0a2eb2492ae51601c4164efa8fddbc30fe7c Mon Sep 17 00:00:00 2001 From: chrg6 Date: Mon, 9 Nov 2026 10:38:09 -0500 Subject: [PATCH] Keep serving the previous translator's shader translations after a translator update, and translate them again in the background. Every shader cache key names the translator, so a release with a new translator had no cached translation at all: the launcher re-translated about 42,000 programs before play, and play waited for any program not prepared. The cache now lists the translator identities it holds translations of in shader-cache/identities.txt, newest first. A shader the current translator has no translation of uses a previous one's, looked up with that identity mixed with this machine's host options and read with the usual checks; it keeps that translation for the rest of the session, so no pipeline is built twice. After the boot prewarm a background pass translates every stored input that has a previous translator's entry again under the current one, on its own translator child at idle priority that waits while the game translates. A program the new translator refuses keeps its old translation and is logged. Once the pass has gone through every such input, the old entries of what it covered are dropped: loose files are deleted, and a pruned copy of shader-cache.pack replaces the pack at the next start when the pack is still the one it was copied from. --prepare-shaders still translates everything under the current translator. NX2NAVI_SHADER_CARRY=0 keeps the earlier behaviour. Co-Authored-By: Claude Opus 5.5 --- docs/architecture.md | 11 +- nx2navi.ini.example | 6 + runtime/boot.c | 3 + runtime/cfg.c | 5 +- runtime/gfx.h | 7 +- runtime/gfx_shader.c | 801 ++++++++++++++++++++++++++++++++++-- runtime/platform/linux.c | 4 + runtime/platform/platform.h | 3 + runtime/platform/win32.c | 4 + 9 files changed, 800 insertions(+), 44 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 2f5461ba..29fbf0c7 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -86,12 +86,21 @@ keeps per-core data without locks and relies on that, so the host models it (`ru pinned shader translator (`tools/vendor/ryujinx`, 1.3.2) with our patches (`tools/xlate/patches/`), built with .NET 10 NativeAOT as `nx2navi_xlate.exe` and run as separate processes. Translations are cached on disk; shaders and pipelines seen in play are prepared at the next boot. The game waits for it (`runtime/boot.c` `boot_wait`): about half a second - when warm; after a translator, GPU or driver change a "Preparing shaders" screen shows the progress (`ui/menu.cpp` + when warm; after a GPU or driver change, or a translator change with nothing to carry (below), a "Preparing + shaders" screen shows the progress (`ui/menu.cpp` `menu_boot_frame`). With the graphics pipeline library the boot builds recorded pipelines in record order for at most 20 s after the shaders are ready and leaves the rest to the pipeline workers below normal priority; a draw that meets one before it is built gets a fast-linked pipeline (`vk_pipeline.c` `PREWARM_PIPE_MS`). `NX2NAVI_PREWARM_FULL=1`, and drivers without the library, wait for every pipeline. `NX2NAVI_PREWARM_WAIT_MS` caps the whole wait, after which the rest finishes in the background. +- Translator updates: every cache key names the translator, so a new one has no translations. The cache carries the + previous translator's (`NX2NAVI_SHADER_CARRY`, `runtime/gfx_shader.c` "Carry"): `shader-cache/identities.txt` + lists the translators it holds translations of, and a shader the new translator has not translated yet uses the + previous one's for the whole session, so neither the start nor play waits for a translation the update caused. + After the boot prewarm a background pass on its own translator process at idle priority, which waits while the game + translates, translates every stored program again for the next start, then deletes the old translations (a pruned + copy of `shader-cache.pack` replaces the pack at the next start). A program the new translator refuses keeps its + old translation. `--prepare-shaders` still translates everything with the current translator. - Mesh and task shaders run as `VK_EXT_mesh_shader`; indirect mesh draws read NVN's `{count, first}` record through a ring of Vulkan `{x, y, z}` records filled on the GPU. - Pipelines: `VK_EXT_graphics_pipeline_library` where the driver has it: stage libraries compiled ahead, a fast link diff --git a/nx2navi.ini.example b/nx2navi.ini.example index c67309b5..fc249ed7 100644 --- a/nx2navi.ini.example +++ b/nx2navi.ini.example @@ -103,6 +103,12 @@ # prewarm_full = 0 # Threads that translate known shaders at start (1 to 64; default: half the CPU threads, at most 16). # prewarm_xlate = 8 +# After an update that brings a new shader translator, 1 keeps using the shader cache's translations from the earlier +# translator, so the update adds no shader translation at start or in play: while you play, a background pass at the +# lowest priority translates them again with the new translator for the next start, then deletes the old ones. A shader +# the new translator cannot translate keeps its old translation. 0: shaders the new translator has not translated yet +# are translated before they are used (at start or when the game first draws them), as earlier versions did. +# shader_carry = 1 # Vulkan graphics pipeline library, 0 or 1. Default: on. # gpl = 1 diff --git a/runtime/boot.c b/runtime/boot.c index 939f6726..97d4736a 100644 --- a/runtime/boot.c +++ b/runtime/boot.c @@ -347,6 +347,9 @@ static void prewarm_main(void *arg) { int cpus = (int)plat_cpu_usable(); gfx_shader_prepare_recorded(cpus, cpus / 2 > 16 ? 16 : cpus / 2 < 1 ? 1 : cpus / 2); } + // Translations of a previous translator are made again in the background from here on, after the boot's own + // translations. Not for a --shot image: which translation a shader gets would then depend on the pass's timing. + if (!rt_config()->shot) gfx_shader_carry_start(); return; } diff --git a/runtime/cfg.c b/runtime/cfg.c index 15fe1439..928300ab 100644 --- a/runtime/cfg.c +++ b/runtime/cfg.c @@ -56,7 +56,7 @@ static const char *const known[] = { "REFLEX", "RENDER_SCALE", "REPLAY_ASYNC", "RTSS", "REPLAY_CHECK", "RESOLVE_CHECK", "RESTART_WAIT", "ROMFS", "ROMFS_INDEX", "ROMFS_MMAP", "ROMFS_MODS", "ROMFS_PATCHES", "ROMFS_PROOF_DIR", "ROMFS_VERIFY", "RSTICK_DEADZONE", "RUMBLE", "RUMBLE_CURVE", "RUMBLE_STRENGTH", "RUMBLE_TRIGGERS", "SHADER_CACHE", - "SHADER_CACHE_FLUSH", "SHADER_MEMO", "SHADOW_COMPARE", "SHADOW_RANGE", "SHADOW_RES", "SHADOW_SCALE", "SHARED_FONTS", "SHARPNESS", "SHOTS", "SHOTS_DIR", + "SHADER_CACHE_FLUSH", "SHADER_CARRY", "SHADER_MEMO", "SHADOW_COMPARE", "SHADOW_RANGE", "SHADOW_RES", "SHADOW_SCALE", "SHARED_FONTS", "SHARPNESS", "SHOTS", "SHOTS_DIR", "SVC_SLOW_MS", "SHOTS_FULL", "SPEED_PROBE", "SPIN", "SPIN_HANDOFF_US", "SPLIT_DRAWS", "SPLIT_IDLE", "SPLIT_IN_PASS", "SPLIT_US", "SPV_OVERRIDE", "STICK_DEADZONE", "STICK_SENS", "SWKBD_ANSWER", @@ -110,7 +110,7 @@ static const struct { const char *key, *section; } sections[] = { {"LIB_PREBUILD", "performance"}, {"LIB_PREBUILD_THREADS", "performance"}, {"PIPE_PREBUILD", "performance"}, {"PREWARM_XLATE", "performance"}, {"GPL", "performance"}, {"ASTC", "performance"}, {"MEMORY", "performance"}, {"TEX_ALIAS", "performance"}, {"TEX_REUSE", "performance"}, {"TEX_VRAM_PRESSURE", "performance"}, {"ASTC_BC", "performance"}, {"LOD_GUARD", "performance"}, {"FIX_C0", "performance"}, {"FIX_ZERO", "performance"}, - {"BIND_SKIP", "performance"}, {"SHADER_CACHE_FLUSH", "performance"}, {"SHADER_MEMO", "performance"}, {"PIPE_MEMO", "performance"}, {"TEX_MEMO", "performance"}, {"DESC_TEMPLATE", "performance"}, + {"BIND_SKIP", "performance"}, {"SHADER_CACHE_FLUSH", "performance"}, {"SHADER_CARRY", "performance"}, {"SHADER_MEMO", "performance"}, {"PIPE_MEMO", "performance"}, {"TEX_MEMO", "performance"}, {"DESC_TEMPLATE", "performance"}, {"PUSH_SKIP", "performance"}, {"SPLIT_DRAWS", "performance"}, {"SPLIT_IDLE", "performance"}, {"SPLIT_IN_PASS", "performance"}, {"SPLIT_US", "performance"}, {"VK_VALIDATION", "performance"}, {"VK_DYN_VERTEX", "performance"}, {"VK_DYN_A2C", "performance"}, {"VK_SUBGROUP_PIN", "performance"}, {"VK_RECONVERGE", "performance"}, {"TRACE", "diagnostics"}, {"FLIGHT", "diagnostics"}, {"PASS_CRUMBS", "diagnostics"}, {"FLIGHT_SPIKE_MS", "diagnostics"}, @@ -204,6 +204,7 @@ static const CfgMeta metas[] = { {"LIB_PREBUILD_BIG_BG", "1", "0|1", RESTART}, {"PIPE_FEEDBACK", "1", "0|1|2", RESTART}, {"PIPELINE_CACHE_MAX_MB", "512", "0..65536", RESTART}, {"PIPE_BINARIES", "0", "0|1", RESTART}, {"ONESHOT_WAIT_MS", "30000", "0..60000", RESTART}, {"ONESHOT_BUDGET", "0", "0|1", RESTART}, {"PIPE_BINARIES_MAX_MB", "1024", "0..65536", RESTART}, {"BIG_BUILD_FREE_MB", "8192", "0..262144", RESTART}, {"BIG_BUILD_MAX_MB", "4096", "0..262144", RESTART}, + {"SHADER_CARRY", "1", "0|1", RESTART}, // gfx_shader.c "Carry": a previous translator's translations until translated again // textures and memory {"ASTC", "auto", "auto|gpu|native|cpu", RESTART}, {"MEMORY", "auto", "auto|import|mirror|verify", RESTART}, {"TEX_ALIAS", "1", "0|1", RESTART}, {"TEX_REUSE", "1", "0|1", RESTART}, {"TEX_VRAM_PRESSURE", "90", "0..100", RESTART}, {"ASTC_BC", "", "0|1", RESTART}, {"ZSTD_NATIVE", "1", "0|1", RESTART}, {"ZSTD_CHECK", "0", "0|1", RESTART}, diff --git a/runtime/gfx.h b/runtime/gfx.h index 372e86ad..50836592 100644 --- a/runtime/gfx.h +++ b/runtime/gfx.h @@ -62,8 +62,13 @@ const uint8_t *gfx_shader_input_id(const GfxShader *s); // the 32-byte input // By input id from memory or disk; xlate 1 also translates it from its stored inputs. NULL if absent. const GfxShader *gfx_shader_load_input(const uint8_t iid[32], int xlate); // Boot prewarm: out[i] = gfx_shader_load_input(iids[i], 1) on `threads` threads with up to `procs` translators, then -// every other stored input not cached under this translator is translated. Logs progress and the total. +// every other stored input not cached under this translator is translated (one a previous translator's translation +// serves is left to gfx_shader_carry_start's pass). Logs progress and the total. void gfx_shader_prewarm(const uint8_t (*iids)[32], size_t n, const GfxShader **out, int threads, int procs); +// After the boot prewarm of a game run (boot.c): when the shader cache holds translations of a previous translator +// (NX2NAVI_SHADER_CARRY), starts the background pass that translates them again under the current one for the next +// start and then drops the old ones (gfx_shader.c, "Carry"). +void gfx_shader_carry_start(void); // How far the running (or last) gfx_shader_prewarm is: shaders done of total, and how many of them it translated (the // rest were cached). Any thread; the numbers lag the workers by at most 100 ms. void gfx_shader_prewarm_progress(int64_t *done, int64_t *total, int64_t *translated); diff --git a/runtime/gfx_shader.c b/runtime/gfx_shader.c index d54fe8a1..0d6c3b8f 100644 --- a/runtime/gfx_shader.c +++ b/runtime/gfx_shader.c @@ -18,6 +18,34 @@ // (64 hex digits) overrides it. A translator change therefore changes every cache key without a hand-made KEY_VERSION // bump; KEY_VERSION is for changes on this side (key layout, how the output is used). // +// Carry (NX2NAVI_SHADER_CARRY, default 1; section "carry"). Since every cache key names the translator, a release with a +// new translator would otherwise translate every shader again, before play or while the game waits. /identities.txt +// lists the translator identities the cache holds translations of, newest first: base identities, before +// gfx_shader_set_host mixes the host options in. The current one goes first once a translation under it is stored or +// read (ids_note). A shader the current translator has no translation of takes a previous translator's: its use mask, +// input id and cache key are computed with that identity mixed with this machine's host options, so a translation +// made with other host options never matches, and the .spv and .res.json come from one place (the pack or the loose +// files) and pass the same checks as any entry. Lookups go: the current key in memory, the previous keys in memory, the +// current key on disk, the previous keys on disk, then a translation (carry_find). A shader served from a previous +// translator thus keeps that translation for the rest of the session, deliberately: no pipeline is built a second +// time for it, and the pass below needs no way to swap shaders under the renderer. A shader first met after the pass +// translated it gets the new translation. +// The background pass (gfx_shader_carry_start, after the boot prewarm) translates every stored input that has a +// previous translator's entry again under the current identity, on a translator child of its own that runs at idle +// priority, outside the pool, and waits while on-demand translations are queued or running. It writes cache files only; +// nothing enters the memory cache. A program the new translator refuses keeps its previous translation, and the log +// names it. When the pass has gone through every such input it drops the previous translators' entries of the inputs it +// covered: loose files are deleted, packed ones are left out of a pruned copy of the pack, which replaces the pack at +// the next start if the pack is still the file it was copied from. The stored inputs (in/) are never dropped. An +// identity leaves identities.txt when nothing of it is left. Only the process holding /carry.lock runs the pass +// or replaces the pack. The preparation (prepare_files, --prepare-shaders) does not carry: it translates everything +// under the current identity. +// One limit: vk_exec_draw.c learns a program's texture slot mask from the first shader it gets for it +// (slot_mask_learn), so a carried shader teaches it the previous translator's mask. Should the current translator read +// the type of a slot the previous one did not, a variant of that program translated anew in the same session is asked +// without that slot's type (the translator's default). The pass translates from the stored full answers, so the next +// session is not affected. +// // Pack: tools/pack_shader_cache.py folds the .spv, .res.json, .use and in/*.in files into /shader-cache.pack // (one file instead of four per shader: ~117,000 files after an install) and deletes them; the installer runs it after // preparing the shaders. The pack is read-only here. Every read looks in the pack first and then for the loose file, @@ -99,6 +127,8 @@ struct GfxShader { uint32_t mvprev_banks; // .res.json "mvprev_banks" (patch 0015), 0 none uint32_t omap; // FS: .res.json "frag_out_map", the colour components it writes // (4 bits per target, the SPH omap); 0 unknown (older translations) + uint8_t carried; // 1 + the previous identity (C.prev) that translated it; 0 the + // current translator }; static PlatMutex cache_lock = PLAT_MUTEX_INIT; // memory cache @@ -131,10 +161,32 @@ static struct { // translations in fl static char cache_dir[PATH_CAP]; static char xlate_exe[PATH_CAP]; static uint8_t xlate_id[32]; // translator identity (see the keys at the top) +static uint8_t xlate_base[32]; // the same before gfx_shader_set_host mixes in host options static int host_f64 = 1, host_zero_locals, host_bda, host_set; // gfx_shader_set_host static uint32_t host_sbuf, host_sg[3] = {32, 32, 32}, host_gather; static PlatOnce paths_once = PLAT_ONCE_INIT; +// Carry state (see "Carry" at the top). prev and prev_eff are written before ready is set and never change after it. +#define IDS_MAX 8 // identities.txt holds at most this many identities +static struct { + int on; // NX2NAVI_SHADER_CARRY, read in init_paths + int n_prev; // previous identities in identities.txt, newest first + uint8_t prev[IDS_MAX][32]; // their base identities + uint8_t prev_eff[IDS_MAX][32]; // mixed with this machine's host options: their cache keys + volatile int32_t ready; // prev_eff is set (gfx_shader_set_host) + volatile int32_t suspended; // a preparation runs: everything is translated anew + volatile int32_t noted; // identities.txt names the current identity first, or will + uint8_t dropped[IDS_MAX][32]; // identities the pass found nothing of (under ids_lock) + int n_dropped; + volatile int64_t served; // shaders served from a previous translator this session + int64_t served_logged; +} C; +static PlatMutex ids_lock = PLAT_MUTEX_INIT; // identities.txt rewrites, C.dropped + +static int carry_on(void) { + return C.n_prev && __atomic_load_n(&C.ready, __ATOMIC_ACQUIRE) && !__atomic_load_n(&C.suspended, __ATOMIC_ACQUIRE); +} + // ---- small helpers #define file_exists plat_file_exists @@ -284,10 +336,14 @@ static void xlate_identity(void) { } free(f); } + memcpy(xlate_base, xlate_id, 32); rt_log("gfx_shader: translator identity %02x%02x%02x%02x%02x%02x%02x%02x (%s)", xlate_id[0], xlate_id[1], xlate_id[2], xlate_id[3], xlate_id[4], xlate_id[5], xlate_id[6], xlate_id[7], how); } +static void ids_load(void); +static void pack_swap_pruned(void); + // The repository root is the nearest parent of nx2navi.exe that holds build/xlate/nx2navi_xlate.exe. The shader cache lives // next to nx2navi.exe, so every build dir has its own (as the pipeline cache, vk_pipeline.c cache_path_init). static void use_preload_start(void); @@ -317,6 +373,8 @@ static void init_paths(void *param) { if (cache_cfg && *cache_cfg && strlen(cache_cfg) < sizeof cache_dir) strcpy(cache_dir, cache_cfg); else if (exe_dir[0]) snprintf(cache_dir, sizeof cache_dir, "%s/shader-cache", exe_dir); xlate_identity(); + C.on = cfg_bool("NX2NAVI_SHADER_CARRY"); + if (cache_dir[0] && C.on) ids_load(), pack_swap_pruned(); // the swap before the pack is opened if (cache_dir[0]) pack_open(); use_preload_start(); return; @@ -324,26 +382,136 @@ static void init_paths(void *param) { static void paths(void) { plat_once(&paths_once, init_paths, NULL); } +// The host options as mixed into an identity (gfx_shader_set_host) into t (96 bytes): their length, 0 for the +// defaults, which are not mixed in. +static int host_text(char *t) { + if (host_f64 && !host_sbuf && host_sg[0] == 32 && host_sg[1] == 32 && host_sg[2] == 32 && !host_gather && + !host_zero_locals && !host_bda) + return 0; + int n = snprintf(t, 96, "TKHO f64=%d sbuf=%u sg=%u,%u,%u", host_f64, host_sbuf, host_sg[0], host_sg[1], host_sg[2]); + if (host_gather) n += snprintf(t + n, (size_t)(96 - n), " gather=%u", host_gather); + if (host_zero_locals) n += snprintf(t + n, (size_t)(96 - n), " zero_locals"); + if (host_bda) n += snprintf(t + n, (size_t)(96 - n), " bda"); + return n; +} + +// A base identity with this machine's host options mixed in: the identity its cache keys use here. +static void host_mix(const uint8_t base[32], uint8_t out[32]) { + uint8_t buf[32 + 96]; + int n = host_text((char *)buf + 32); + if (!n) { + memmove(out, base, 32); + return; + } + memcpy(buf, base, 32); + rt_sha256(buf, 32 + (size_t)n, out); +} + // Host answers (vk_caps.c, at device creation, before any translation). Answers other than the defaults (Float64, // exact storage offsets, 32 lanes, no gather bias, undefined locals) are mixed into the translator identity, so they -// get their own cache keys. +// get their own cache keys. The previous identities of identities.txt get the same options mixed in (carry). void gfx_shader_set_host(int float64, uint32_t sbuf_align, const uint32_t subgroup[3], uint32_t gather_bits, int zero_locals, int bda) { paths(); if (host_set++) return; host_f64 = float64, host_sbuf = sbuf_align, host_gather = gather_bits, host_zero_locals = zero_locals, host_bda = bda; memcpy(host_sg, subgroup, sizeof host_sg); - if (float64 && !sbuf_align && subgroup[0] == 32 && subgroup[1] == 32 && subgroup[2] == 32 && !gather_bits && !zero_locals && !bda) - return; - uint8_t buf[32 + 96]; - int n = snprintf((char *)buf + 32, 96, "TKHO f64=%d sbuf=%u sg=%u,%u,%u", float64, sbuf_align, subgroup[0], subgroup[1], - subgroup[2]); - if (gather_bits) n += snprintf((char *)buf + 32 + n, (size_t)(96 - n), " gather=%u", gather_bits); - if (zero_locals) n += snprintf((char *)buf + 32 + n, (size_t)(96 - n), " zero_locals"); - if (bda) n += snprintf((char *)buf + 32 + n, (size_t)(96 - n), " bda"); - memcpy(buf, xlate_id, 32); - rt_sha256(buf, 32 + (size_t)n, xlate_id); - rt_log("gfx_shader: host options %s: own shader cache keys", (char *)buf + 37); + char t[96]; + int n = host_text(t); + host_mix(xlate_base, xlate_id); + for (int i = 0; i < C.n_prev; i++) host_mix(C.prev[i], C.prev_eff[i]); + __atomic_store_n(&C.ready, 1, __ATOMIC_RELEASE); + if (n) rt_log("gfx_shader: host options %s: own shader cache keys", t + 5); +} + +// ---- identities.txt (see "Carry" at the top): "#" comment lines, then one base identity per line in 64 hex digits, +// newest first, at most IDS_MAX. + +static void key_hex(const uint8_t key[32], char out[65]); + +static void ids_path(char *p) { snprintf(p, PATH_CAP + 32, "%s/identities.txt", cache_dir); } + +// The identities of the text f (may be NULL) into out; their count. Lines that are no identity are skipped. +static int ids_parse(const char *f, uint8_t out[IDS_MAX][32]) { + int n = 0; + while (f && *f && n < IDS_MAX) { + const char *e = strchr(f, '\n'); + size_t len = e ? (size_t)(e - f) : strlen(f); + char line[128]; + uint8_t id[32]; + if (len < sizeof line && *f != '#') { + memcpy(line, f, len), line[len] = 0; + int ok = hex32(line, id); + for (int i = 0; ok && i < n; i++) ok = memcmp(out[i], id, 32) != 0; // a repeated line counts once + if (ok) memcpy(out[n++], id, 32); + } + f += len + (e != NULL); + } + return n; +} + +static int id_zero(const uint8_t id[32]) { + static const uint8_t zero[32]; + return !memcmp(id, zero, 32); +} + +// init_paths: the previous identities, and whether the current one is first already. +static void ids_load(void) { + char p[PATH_CAP + 32]; + uint8_t ids[IDS_MAX][32]; + uint64_t size = 0; + ids_path(p); + char *f = (char *)read_file(p, &size); + int n = ids_parse(f, ids); + free(f); + for (int i = 0; i < n; i++) + if (memcmp(ids[i], xlate_base, 32)) memcpy(C.prev[C.n_prev++], ids[i], 32); + C.noted = id_zero(xlate_base) || (n && !memcmp(ids[0], xlate_base, 32)); // nothing to write + if (C.n_prev) + rt_log("gfx_shader: carry: the cache holds translations of %d previous translators (newest %02x%02x%02x%02x); " + "shaders the current one has not translated yet use theirs", C.n_prev, C.prev[0][0], C.prev[0][1], + C.prev[0][2], C.prev[0][3]); +} + +// Rewrites identities.txt from what it holds now: the current identity first when a translation under it was stored or +// read (C.noted), then the others in their order, without the ones the pass dropped. Read again first, so an identity +// another process wrote meanwhile stays. +static void ids_write(void) { + char p[PATH_CAP + 32], text[128 + IDS_MAX * 66]; + uint8_t ids[IDS_MAX][32], out[IDS_MAX][32]; + uint64_t size = 0; + plat_mutex_lock(&ids_lock); + ids_path(p); + char *f = (char *)read_file(p, &size); + int n = ids_parse(f, ids), k = 0; + free(f); + if (__atomic_load_n(&C.noted, __ATOMIC_ACQUIRE) && !id_zero(xlate_base)) memcpy(out[k++], xlate_base, 32); + for (int i = 0; i < n && k < IDS_MAX; i++) { + int skip = !memcmp(ids[i], xlate_base, 32); + for (int j = 0; j < C.n_dropped && !skip; j++) skip = !memcmp(ids[i], C.dropped[j], 32); + if (!skip) memcpy(out[k++], ids[i], 32); + } + int len = snprintf(text, sizeof text, "# Translator identities this shader cache holds translations of, newest first\n"); + for (int i = 0; i < k; i++) { + key_hex(out[i], text + len); + len += 64; + text[len++] = '\n'; + } + mkdir_p(cache_dir); + if (!plat_file_write_atomic(p, text, (size_t)len)) rt_log("gfx_shader: carry: cannot write %s", p); + plat_mutex_unlock(&ids_lock); +} + +static void ids_write_main(void *arg) { + (void)arg; + ids_write(); +} + +// A translation under the current identity was stored or read: once per process, identities.txt gets the current +// identity first (on a thread of its own: the caller may be the render thread, and the write waits for the disk). +static void ids_note(void) { + if (!C.on || plat_cas32(&C.noted, 1, 0)) return; + if (!plat_thread_spawn(ids_write_main, NULL, "shader ids", 0)) ids_write(); } // ---- the translator children @@ -524,8 +692,18 @@ static void shader_free(GfxShader *s) { static void reflect(GfxShader *s); +// A shader of a previous translator entered the memory cache: counted, and the first one logged. +static void carry_served(const GfxShader *s) { + if (plat_inc64(&C.served) != 1) return; + const uint8_t *id = C.prev[s->carried - 1]; + rt_log("gfx_shader: carry: serving translations of a previous translator (%02x%02x%02x%02x) where the current one has " + "none yet; each keeps its translation for this session, and a background pass translates them again for the " + "next start", id[0], id[1], id[2], id[3]); +} + // Inserts s, or frees it and returns the entry another thread inserted first. static GfxShader *mem_insert(GfxShader *s) { + int added = 0; plat_mutex_lock(&cache_lock); GfxShader **head = &buckets[(s->key[0] | s->key[1] << 8) % BUCKETS], *r = *head; while (r && memcmp(r->key, s->key, 32)) r = r->next; @@ -536,8 +714,11 @@ static GfxShader *mem_insert(GfxShader *s) { s->next = *head; *head = s; n_keys++; + added = 1; } plat_mutex_unlock(&cache_lock); + if (added && s->carried) carry_served(s); + else if (added) ids_note(); return s; } @@ -575,9 +756,13 @@ static int spirv_complete(const uint32_t *w, uint64_t n) { return i == n; } +static void reflect_spirv(GfxShader *s); +static void reflect_stage(GfxShader *s); + // Reads .spv and .res.json from the pack (packed 1) or the loose files; NULL if either is missing or -// malformed. iid: the input id, NULL when the caller sets it. -static GfxShader *disk_load_from(const uint8_t key[32], const uint8_t *iid, int packed) { +// malformed. iid: the input id, NULL when the caller sets it. full 0: only the stage and the interface Locations are +// reflected and the SPIR-V is left as the translator wrote it (the carry pass checks a translation it does not keep). +static GfxShader *disk_load_from(const uint8_t key[32], const uint8_t *iid, int packed, int full) { static volatile int32_t n_bad; char hex[65], p[PATH_CAP + 80]; key_hex(key, hex); @@ -611,6 +796,10 @@ static GfxShader *disk_load_from(const uint8_t key[32], const uint8_t *iid, int s->words = spv_size / 4; s->reflection = (char *)res; s->reflection_size = res_size; + if (!full) { + reflect_stage(s), reflect_spirv(s); + return s; + } reflect(s); if (s->stage == GFX_STAGE_FS) vk_hdr_shader_swap(&s->spirv, &s->words); // the in-memory copy; the cache keeps its own return s; @@ -638,8 +827,20 @@ const GfxShader *gfx_shader_own(const uint32_t *spirv, uint64_t words, const cha // The pack, then the loose files (a newer translation of a key the pack holds a bad copy of); NULL when neither has a // valid pair (the caller translates again and writes loose files). static GfxShader *disk_load(const uint8_t key[32], const uint8_t *iid) { - GfxShader *s = pack_n ? disk_load_from(key, iid, 1) : NULL; - return s ? s : disk_load_from(key, iid, 0); + GfxShader *s = pack_n ? disk_load_from(key, iid, 1, 1) : NULL; + return s ? s : disk_load_from(key, iid, 0, 1); +} + +// Whether the cache holds .spv and .res.json together in the pack or together as loose files (as disk_load +// reads them), without reading them. +static int entry_present(const uint8_t key[32]) { + if (pack_find(PACK_SPV, key) && pack_find(PACK_RES, key)) return 1; + char hex[65], p[PATH_CAP + 80]; + key_hex(key, hex); + snprintf(p, sizeof p, "%s/%s.spv", cache_dir, hex); + if (!file_exists(p)) return 0; + snprintf(p, sizeof p, "%s/%s.res.json", cache_dir, hex); + return file_exists(p); } // ---- request building @@ -867,13 +1068,18 @@ out: free(cc); } -static void reflect(GfxShader *s) { - static const char *const kinds[GFX_RES_KINDS] = {"cbufs", "sbufs", "textures", "images"}; +static void reflect_stage(GfxShader *s) { const char *js = s->reflection; char stage[8]; json_str(js, js + strlen(js), "stage", stage, sizeof stage); s->stage = !strcmp(stage, "cs") ? GFX_STAGE_CS : !strcmp(stage, "ps") ? GFX_STAGE_FS : !strcmp(stage, "ts") ? GFX_STAGE_TS : !strcmp(stage, "ms") ? GFX_STAGE_MS : GFX_STAGE_VS; +} + +static void reflect(GfxShader *s) { + static const char *const kinds[GFX_RES_KINDS] = {"cbufs", "sbufs", "textures", "images"}; + const char *js = s->reflection; + reflect_stage(s); s->mvprev_banks = (uint32_t)json_int(js, js + strlen(js), "mvprev_banks", 0); s->omap = s->stage == GFX_STAGE_FS ? (uint32_t)json_int(js, js + strlen(js), "frag_out_map", 0) : 0; for (int k = 0; k < GFX_RES_KINDS; k++) { @@ -913,12 +1119,14 @@ static UseMask use_of(const GfxShader *s) { } // .use: masks are per translator. -static void use_name(const uint8_t h[32], uint8_t name[32]) { +static void use_name_id(const uint8_t id[32], const uint8_t h[32], uint8_t name[32]) { uint8_t buf[4 + 32 + 32]; - memcpy(buf, "TKSU", 4), memcpy(buf + 4, xlate_id, 32), memcpy(buf + 36, h, 32); + memcpy(buf, "TKSU", 4), memcpy(buf + 4, id, 32), memcpy(buf + 36, h, 32); rt_sha256(buf, sizeof buf, name); } +static void use_name(const uint8_t h[32], uint8_t name[32]) { use_name_id(xlate_id, h, name); } + static void use_path_of(const uint8_t name[32], char *p) { char hex[65]; key_hex(name, hex); @@ -1131,28 +1339,37 @@ static void mask_answers(const GfxShaderAnswers *a, const UseMask *m, GfxShaderA o->mvprev = a->mvprev; // another translation, not an answer the use mask covers } -// The cache key of an input id under the current translator. -static void key_of_input(const uint8_t iid[32], uint8_t key[32]) { +// The cache key of an input id under the translator identity id (with the host options mixed in). +static void key_of_input_id(const uint8_t id[32], const uint8_t iid[32], uint8_t key[32]) { uint8_t buf[8 + 32 + 32], *p = buf; memcpy(p, "TKSK", 4), p += 4; put_u32(&p, KEY_VERSION); - memcpy(p, xlate_id, 32), p += 32; + memcpy(p, id, 32), p += 32; memcpy(p, iid, 32), p += 32; rt_sha256(buf, (size_t)(p - buf), key); } +// The cache key of an input id under the current translator. +static void key_of_input(const uint8_t iid[32], uint8_t key[32]) { key_of_input_id(xlate_id, iid, key); } + +// The input id of a code hash and answers under use mask m. +static void iid_of(const uint8_t code_hash[32], const GfxShaderAnswers *a, const UseMask *m, uint8_t iid[32]) { + GfxShaderAnswers masked; + uint8_t buf[8 + 32 + 256], *p = buf; + memcpy(p, "TKSI", 4), p += 4; + put_u32(&p, IN_VERSION); + memcpy(p, code_hash, 32), p += 32; + mask_answers(a, m, &masked); + p += pack_answers(p, &masked); + rt_sha256(buf, (size_t)(p - buf), iid); +} + // The input id and cache key (m given), or the provisional key (m NULL: full answers; iid untouched). static void build_key(const uint8_t code_hash[32], const GfxShaderAnswers *a, const UseMask *m, uint8_t key[32], uint8_t iid[32]) { - GfxShaderAnswers masked; uint8_t buf[8 + 32 + 32 + 256], *p = buf; if (m) { - memcpy(p, "TKSI", 4), p += 4; - put_u32(&p, IN_VERSION); - memcpy(p, code_hash, 32), p += 32; - mask_answers(a, m, &masked); - p += pack_answers(p, &masked); - rt_sha256(buf, (size_t)(p - buf), iid); + iid_of(code_hash, a, m, iid); key_of_input(iid, key); return; } @@ -1251,6 +1468,92 @@ static size_t build_request(char *o0, const char *hex, const char *stage, const return (size_t)(o - o0); } +// ---- carry: lookups under the previous identities (see "Carry" at the top) + +// The use mask of code hash h under identity id: the preloaded table once it is complete, else the pack and the loose +// file. Not remembered in use_buckets, which holds the current identity's masks. +static int use_lookup(const uint8_t id[32], const uint8_t h[32], UseMask *m) { + uint8_t name[32]; + use_name_id(id, h, name); + if (plat_cas32(&use_files_ready, 0, 0)) { + UseFile *u = *use_file_slot(name); + if (u) *m = u->m; + return u != NULL; + } + char p[PATH_CAP + 80]; + uint64_t n = 0; + uint32_t *f = (uint32_t *)pack_read(PACK_USE, name, &n); + if (!f) use_path_of(name, p), f = (uint32_t *)read_file(p, &n); + int known = f && n == 20 && f[0] == USE_MAGIC && f[1] == USE_VERSION; + if (known) m->slots = f[2], m->attrs = f[3], m->fouts = f[4]; + free(f); + return known; +} + +// Previous identity j's entry `key` from the disk cache into the memory cache; NULL when the cache has no valid one. +// stage (GFX_STAGE_*, -1 any): an entry of another stage than the request's is not used. Every entry passes +// disk_load's checks (whole SPIR-V, a .res.json naming its stage, both from the pack or both loose), which matters most +// for compute shaders: the replay thread dispatches one the moment it has it (vk_exec_draw.c shader_for). +static GfxShader *carry_load(const uint8_t key[32], const uint8_t iid[32], int stage, int j) { + GfxShader *s = disk_load(key, iid); + if (!s) return NULL; + if (stage >= 0 && s->stage != stage) { + static volatile int32_t n_logged; + if (plat_inc32(&n_logged) <= 4) + rt_log("gfx_shader: carry: a previous translation of stage %d is not used for a stage %d request", s->stage, stage); + shader_free(s); + return NULL; + } + s->carried = (uint8_t)(j + 1); + return mem_insert(s); +} + +// A previous translator's translation of (code hash, answers) for a stage, newest identity first: from the memory cache +// only (disk 0), or from the memory cache and then the disk cache (disk 1). NULL when none has one or carry is off. +static GfxShader *carry_find(const uint8_t h[32], const GfxShaderAnswers *a, int stage, int disk) { + if (!carry_on()) return NULL; + for (int j = 0; j < C.n_prev; j++) { + UseMask m; + uint8_t iid[32], key[32]; + if (!use_lookup(C.prev_eff[j], h, &m)) continue; + iid_of(h, a, &m, iid); + key_of_input_id(C.prev_eff[j], iid, key); + GfxShader *s = mem_find(key); + if (s || (disk && (s = carry_load(key, iid, stage, j)))) return s; + } + return NULL; +} + +// The same by input id. owned given (gfx_shader_load_private): an entry read from the disk is handed to the caller +// with *owned 1 instead of entering the memory cache. +static GfxShader *carry_find_iid(const uint8_t iid[32], int disk, int *owned) { + if (!carry_on()) return NULL; + for (int j = 0; j < C.n_prev; j++) { + uint8_t key[32]; + key_of_input_id(C.prev_eff[j], iid, key); + GfxShader *s = mem_find(key); + if (s || !disk) { + if (s) return s; + continue; + } + if (!owned) s = carry_load(key, iid, -1, j); + else if ((s = disk_load(key, iid))) s->carried = (uint8_t)(j + 1), *owned = 1; + if (s) return s; + } + return NULL; +} + +// Whether the cache holds a previous translator's entry of input iid. +static int carry_has_iid(const uint8_t iid[32]) { + if (!carry_on()) return 0; + for (int j = 0; j < C.n_prev; j++) { + uint8_t key[32]; + key_of_input_id(C.prev_eff[j], iid, key); + if (entry_present(key)) return 1; + } + return 0; +} + // ---- public // Raw line exchange for the probe: one line to the service, its reply in `reply`. Starts the service if needed. @@ -1397,13 +1700,16 @@ static int failed_slot(const uint8_t key[32], int insert) { return found; } +// The lookup order with carry (see "Carry" at the top): the current key in memory, a previous translator's entry in +// memory, the current key on disk, a previous translator's entry on disk. static const GfxShader *get_one(const Req *r, int stage, const uint8_t *control, const uint8_t *code, const GfxShaderAnswers *a) { GfxShader *s = mem_find(r->key); - if (s) return plat_inc64(&n_hits), s; + if (s || (s = carry_find(r->code_hash, a, stage, 0))) return plat_inc64(&n_hits), s; paths(); if (!cache_dir[0]) return fail("no cache directory", NULL); if (r->final && (s = disk_load(r->key, r->iid))) return plat_inc64(&n_hits), mem_insert(s); + if ((s = carry_find(r->code_hash, a, stage, 1))) return plat_inc64(&n_hits), s; if (failed_slot(r->key, 0)) return NULL; const GfxShader *t = translate(r, stage, control, code, a); if (!t) failed_slot(r->key, 1); @@ -1647,7 +1953,7 @@ const GfxShader *gfx_shader_get_async(int stage, const uint8_t *control, const u static const GfxShader *get_async_one(Req r, int stage, const uint8_t *control, const uint8_t *code, const GfxShaderAnswers *a, int *pending) { GfxShader *s = mem_find(r.key); - if (s) return plat_inc64(&n_hits), s; + if (s || (s = carry_find(r.code_hash, a, stage, 0))) return plat_inc64(&n_hits), s; // get_one's order plat_mutex_lock_shared(&X.lock); XWait *w = *wait_slot(r.key); int failed = w && w->failed; @@ -1656,6 +1962,7 @@ static const GfxShader *get_async_one(Req r, int stage, const uint8_t *control, paths(); if (!cache_dir[0]) return fail("no cache directory", NULL); if (r.final && (s = disk_load(r.key, r.iid))) return plat_inc64(&n_hits), mem_insert(s); + if ((s = carry_find(r.code_hash, a, stage, 1))) return plat_inc64(&n_hits), s; XJob *j = calloc(1, sizeof *j); XWait *e = calloc(1, sizeof *e); @@ -1728,15 +2035,17 @@ const uint8_t *gfx_shader_key(const GfxShader *s) { return s->key; } const uint8_t *gfx_shader_input_id(const GfxShader *s) { return s->iid; } // By input id: memory, then the disk cache under the current translator, then (xlate 1) a translation from the stored -// inputs. NULL when none has it; a failed translation is counted as a failure. +// inputs; a previous translator's entry in between as get_one takes them. NULL when none has it; a failed translation +// is counted as a failure. const GfxShader *gfx_shader_load_input(const uint8_t iid[32], int xlate) { paths(); if (!cache_dir[0]) return NULL; uint8_t key[32]; key_of_input(iid, key); GfxShader *s = mem_find(key); - if (s) return s; + if (s || (s = carry_find_iid(iid, 0, NULL))) return s; if ((s = disk_load(key, iid))) return mem_insert(s); + if ((s = carry_find_iid(iid, 1, NULL))) return s; Inputs in; if (!xlate || !in_load(iid, &in)) return NULL; const GfxShader *t = gfx_shader_get(in.stage, in.control, in.code, in.length, in.c1, &in.a); @@ -1754,8 +2063,9 @@ const GfxShader *gfx_shader_load_private(const uint8_t iid[32], int *owned) { uint8_t key[32]; key_of_input(iid, key); GfxShader *s = mem_find(key); - if (s) return s; - if ((s = disk_load(key, iid))) *owned = 1; + if (s || (s = carry_find_iid(iid, 0, NULL))) return s; + if ((s = disk_load(key, iid))) *owned = 1, ids_note(); + if (!s) s = carry_find_iid(iid, 1, owned); return s; } @@ -1772,7 +2082,7 @@ typedef struct { uint8_t (*other)[32]; // other stored inputs: translated when not cached, not loaded size_t n_other; volatile int64_t next, done; - volatile int32_t missing, other_cached; + volatile int32_t missing, other_cached, other_carried; // carried: a previous translator's entry, left to the pass } Warm; // The running prewarm's numbers for gfx_shader_prewarm_progress (boot.c's wait screen), copied from its Warm while it @@ -1800,6 +2110,7 @@ static void warm_worker(void *arg) { key_hex(key, hex); snprintf(p, PATH_CAP + 80, "%s/%s.spv", cache_dir, hex); if (pack_find(PACK_SPV, key) || file_exists(p)) plat_inc32(&w->other_cached); + else if (carry_has_iid(iid)) plat_inc32(&w->other_carried); // the background pass translates it else gfx_shader_load_input(iid, 1); } plat_inc64(&w->done); @@ -1885,9 +2196,10 @@ void gfx_shader_prewarm(const uint8_t (*iids)[32], size_t n, const GfxShader **o __atomic_store_n(&pw_xlated, (int64_t)((uint64_t)n_xlated - xl0), __ATOMIC_RELAXED); svc_set_limit(svc_runtime_limit()); rt_log("gfx_shader: prewarm: %zu shaders named by pipeline keys (%ld not available), %zu other stored inputs (%ld " - "cached); %llu translated, %llu failed, with up to %d translators and %d threads, in %.0f ms", w.n_named, - (long)w.missing, w.n_other, (long)w.other_cached, (unsigned long long)((uint64_t)n_xlated - xl0), - (unsigned long long)((uint64_t)n_failures - fail0), procs, k, (double)plat_now_ms() - t0); + "cached, %ld by a previous translator only); %llu translated, %llu failed, with up to %d translators and %d " + "threads, in %.0f ms", w.n_named, (long)w.missing, w.n_other, (long)w.other_cached, (long)w.other_carried, + (unsigned long long)((uint64_t)n_xlated - xl0), (unsigned long long)((uint64_t)n_failures - fail0), procs, k, + (double)plat_now_ms() - t0); free(w.other); } const uint32_t *gfx_shader_spirv(const GfxShader *s, uint64_t *words) { @@ -2157,6 +2469,9 @@ static PlatMutex rec_io = PLAT_MUTEX_INIT; // the record files: appends in ord void gfx_shader_record_flush(int force) { static volatile int64_t last; int64_t now = (int64_t)plat_now_ms(); + int64_t served = __atomic_load_n(&C.served, __ATOMIC_RELAXED); + if (force && served != __atomic_exchange_n(&C.served_logged, served, __ATOMIC_RELAXED)) // the saves and the exit + rt_log("gfx_shader: carry: %lld shaders served from a previous translator this session", (long long)served); if (!RC.on || (!force && now - last < 2000)) return; last = now; // The files are written outside RC.lock, which the render thread takes to record a new answer. @@ -2246,6 +2561,7 @@ static long long prepare_files(const char *ans_path, const char *prg_path, int t if (!p.progs.idx || (iid_out && !p.iid)) return free(p.progs.idx), free(p.iid), free(ab), free(pb), -1; prg_walk(pb, pn, prg_collect, &p.progs); qsort(p.progs.idx, p.progs.n, sizeof *p.progs.idx, prg_cmp); + plat_inc32(&C.suspended); // a preparation translates everything under the current translator, carrying nothing uint64_t xl0 = (uint64_t)n_xlated; threads = threads < 1 ? 1 : threads > 64 ? 64 : threads; svc_set_limit(procs); @@ -2266,6 +2582,7 @@ static long long prepare_files(const char *ans_path, const char *prg_path, int t } printf("prepare-shaders: progress translate %zu %zu\n", p.n_ans, p.n_ans); fflush(stdout); + plat_dec32(&C.suspended); svc_set_limit(svc_runtime_limit()); rt_log("gfx_shader: prepare: %zu %s of %zu programs: %llu translated, %lld failed, %lld without a program, the rest " "cached; %lld default variants with refined answers; %.0f ms", p.n_ans, what, p.progs.n, @@ -2300,6 +2617,9 @@ int gfx_shader_prepare_seed(const char *dir, int threads, int procs) { double t0 = (double)plat_now_ms(); uint64_t fail0 = (uint64_t)n_failures; uint8_t *ab = NULL, (*iid)[32] = NULL; + // For the whole run: the variants built into the driver's cache after it (vk_pipeline.c lb_warm_prepared) are the + // current translator's, as before carrying existed. + plat_inc32(&C.suspended); long long n = prepare_files(ans, prg, threads, procs, "seed variants", &ab, &iid); if (n < 0) return 1; uint8_t *recs = calloc((size_t)n + 1, PRP_REC); @@ -2367,3 +2687,404 @@ size_t gfx_shader_warm_list(GfxWarmEnt **out) { *out = e; return n; } + +// ---- carry: the background pass, pruning and the pruned pack (see "Carry" at the top) + +// /carry.lock, held for the rest of the process: 1 held, 0 another process holds it, -1 it cannot be made. +static int carry_lock(void) { + char p[PATH_CAP + 32]; + snprintf(p, sizeof p, "%s/carry.lock", cache_dir); + return plat_lock_file(p); +} + +#define PACK_SRC_MAGIC "TKSPSRC1" // the pruned pack's trailer: this, then the source pack's +#define PACK_TRAILER 48 // PlatFileId as five u64 + +static void file_id_put(uint8_t *o, const PlatFileId *id) { + uint64_t v[5] = {id->volume, id->file_id, id->size, id->write_time, id->create_time}; + memcpy(o, v, sizeof v); +} + +// Whether /shader-cache.pack.pruned exists and was copied from the pack file src; *size its size. +static int pruned_of(const PlatFileId *src, uint64_t *size) { + char p[PATH_CAP + 48]; + snprintf(p, sizeof p, "%s/shader-cache.pack.pruned", cache_dir); + PlatFile *f = plat_file_open(p); + PlatFileId id; + uint8_t tr[PACK_TRAILER], want[40]; + int same = f && plat_file_id(f, &id) && id.size >= 16 + PACK_TRAILER && + plat_file_pread(f, tr, PACK_TRAILER, id.size - PACK_TRAILER) == PACK_TRAILER && !memcmp(tr, PACK_SRC_MAGIC, 8); + if (f) plat_file_close(f); + if (same) file_id_put(want, src), same = !memcmp(tr + 8, want, sizeof want), *size = id.size; + return same; +} + +// init_paths, before the pack is opened: the pruned copy a pass wrote (carry_repack) replaces shader-cache.pack when the +// pack is still the file it was copied from. Otherwise something wrote a new pack since (tools/pack_shader_cache.py, an +// install), the copy would lose what that one added, and it is deleted; a later pass prunes the new pack. +static void pack_swap_pruned(void) { + char pend[PATH_CAP + 48], pack[PATH_CAP + 32]; + snprintf(pend, sizeof pend, "%s/shader-cache.pack.pruned", cache_dir); + if (!file_exists(pend) || carry_lock() != 1) return; // another process uses the cache: a later start does it + snprintf(pack, sizeof pack, "%s/shader-cache.pack", cache_dir); + PlatFile *b = plat_file_open(pack); + PlatFileId ib; + uint64_t size = 0; + int same = b && plat_file_id(b, &ib); + if (b) plat_file_close(b); + same = same && pruned_of(&ib, &size); + if (!same) { + plat_file_delete(pend); + rt_log("gfx_shader: carry: the cache pack changed after its pruned copy was made; the copy is deleted"); + } else if (plat_file_replace(pend, pack, 1)) { + rt_log("gfx_shader: carry: the cache pack is now its copy without previous translators' files (%.0f MiB)", + (double)size / 1048576.0); + } else { + rt_log("gfx_shader: carry: cannot put the pruned copy in place of %s now; a later start tries again", pack); + } +} + +typedef struct { + uint8_t iid[32], h[32]; // input id; code hash once known (h_known) + uint8_t has; // bit j: an entry under C.prev_eff[j] + uint8_t state, h_known; // CI_* +} CarryItem; +enum { CI_TODO, CI_COVERED, CI_FAILED, CI_SKIPPED }; // skipped: unreadable stored input, or no translator + +static Svc carry_svc; // the pass's own translator child, outside the pool (S) + +// Whether on-demand translation is running or queued (F, X): the pass waits until none is. +static int carry_busy(void) { + return __atomic_load_n(&F.n, __ATOMIC_RELAXED) > 0 || __atomic_load_n(&X.head, __ATOMIC_RELAXED) != NULL; +} + +static void carry_wait(void) { + while (carry_busy()) plat_sleep_ms(50); +} + +// One request on the pass's child, started below every normal process when needed; a dead child is restarted once. +static int carry_exchange(const char *line, char *reply, size_t n) { + for (int attempt = 0; attempt < 2; attempt++) { + if (!carry_svc.child) { + if (!child_start(&carry_svc)) return 0; + plat_child_lower(carry_svc.child); + } + if (child_exchange_raw(&carry_svc, line, reply, n)) return 1; + child_stop(&carry_svc); + } + return 0; +} + +// Translates a stored input again under the current identity and stores the files as translate_one does, leaving the +// memory cache alone. The job's files have names of their own (sha256("TKSC", key)), so an on-demand translation of the +// same key at the same time never writes the same files; each renames its own onto the key's, with the same content. +// 1 stored, 0 the translator refused the program or wrote no usable translation (why says which), -1 no translator or +// a file error. +static int carry_translate(const Req *r, const Inputs *in, char *why, size_t nwhy) { + uint8_t t[32], tb[36]; + memcpy(tb, "TKSC", 4), memcpy(tb + 4, r->key, 32); + rt_sha256(tb, sizeof tb, t); + char hex[65], code_path[PATH_CAP + 80], spv_path[PATH_CAP + 80], res_path[PATH_CAP + 80]; + key_hex(t, hex); + mkdir_p(cache_dir); + snprintf(code_path, sizeof code_path, "%s/%s.code", cache_dir, hex); + snprintf(spv_path, sizeof spv_path, "%s/%s.spv", cache_dir, hex); + snprintf(res_path, sizeof res_path, "%s/%s.res.json", cache_dir, hex); + if (!write_file(code_path, in->code + r->skip, (size_t)(r->length - r->skip))) + return snprintf(why, nwhy, "cannot write the code file"), -1; + size_t cap = 2 * PATH_CAP * 6 + 2 * (size_t)r->c1_len + 8192; + char *req = malloc(cap), reply[LINE_MAX_LEN]; + if (!req) return plat_file_delete(code_path), snprintf(why, nwhy, "out of memory"), -1; + build_request(req, hex, r->stage_name, code_path, r->c1, r->c1_len, &in->a, in->control); + int ok = carry_exchange(req, reply, sizeof reply), refused = ok && strncmp(reply, "ok ", 3); + if (refused) { // as translate_one: the job and its code stay, for a translator fix to be tested on + char job_path[PATH_CAP + 80]; + snprintf(job_path, sizeof job_path, "%s/%s.job", cache_dir, hex); + write_file(job_path, req, strlen(req)); + } else { + plat_file_delete(code_path); + } + free(req); + if (!ok) return snprintf(why, nwhy, "translator unavailable"), -1; + GfxShader *s = refused ? NULL : disk_load_from(t, NULL, 0, 0); + if (!s || s->stage != in->stage) { + snprintf(why, nwhy, "%s", refused ? reply : s ? "translated to another stage" : "translator output missing or malformed"); + plat_file_delete(spv_path), plat_file_delete(res_path); + if (s) shader_free(s); + return 0; + } + UseMask m = {0}; + uint8_t key[32], iid[32]; + if (r->final) memcpy(key, r->key, 32), memcpy(iid, r->iid, 32); + else m = use_of(s), build_key(r->code_hash, &in->a, &m, key, iid); + shader_free(s); + char khex[65], to[PATH_CAP + 80]; + key_hex(key, khex); + snprintf(to, sizeof to, "%s/%s.spv", cache_dir, khex); + int moved = plat_file_replace(spv_path, to, cache_flush()); + snprintf(to, sizeof to, "%s/%s.res.json", cache_dir, khex); + moved = moved && plat_file_replace(res_path, to, cache_flush()); + if (!moved) { // a reader has the key's files open: an on-demand translation stored them meanwhile + plat_file_delete(spv_path), plat_file_delete(res_path); + if (!entry_present(key)) return snprintf(why, nwhy, "cannot rename %s", hex), -1; + } + if (!r->final) { + plat_thread_priority(PLAT_PRIO_NORMAL); // use_store takes the cache lock the render thread reads under + use_store(r->code_hash, &m); + plat_thread_priority(PLAT_PRIO_LOWEST); + } + in_store(iid, in->stage, in->control, in->code, r, &in->a); + ids_note(); + return 1; +} + +static int set_has(const Set64 *s, uint64_t h) { + h = h ? h : 1; + for (uint32_t j = s->cap ? (uint32_t)h & (s->cap - 1) : 0; s->cap && s->v[j]; j = (j + 1) & (s->cap - 1)) + if (s->v[j] == h) return 1; + return 0; +} + +typedef struct { uint32_t kind; uint8_t name[32]; } PackDrop; + +static int drop_cmp(const void *x, const void *y) { + const PackDrop *a = x, *b = y; + if (a->kind != b->kind) return a->kind < b->kind ? -1 : 1; + return memcmp(a->name, b->name, 32); +} + +typedef struct { + PackDrop *v; // packed files to leave out of the pruned pack + size_t n, cap; + size_t deleted; // loose files deleted + size_t left; // files still there: a delete failed, or packed +} Drops; + +// One file of a previous translator: the loose file is deleted, a packed one is listed for the pruned pack. +static void drop_file(Drops *d, uint32_t kind, const uint8_t name[32]) { + char hex[65], p[PATH_CAP + 80]; + key_hex(name, hex); + snprintf(p, sizeof p, "%s/%s%s", cache_dir, hex, kind == PACK_USE ? ".use" : kind == PACK_SPV ? ".spv" : ".res.json"); + if (file_exists(p)) { + plat_file_delete(p); + if (file_exists(p)) d->left++; // open in a reader: the next pass deletes it + else d->deleted++; + } + if (!pack_find(kind, name)) return; + d->left++; + if (d->n == d->cap) { + size_t c = d->cap ? d->cap * 2 : 4096; + PackDrop *t = realloc(d->v, c * sizeof *t); + if (!t) return; + d->v = t, d->cap = c; + } + d->v[d->n].kind = kind, memcpy(d->v[d->n].name, name, 32), d->n++; +} + +// Previous identity j's files of the inputs the pass covered: their .spv and .res.json, and the .use of each program +// none of whose other inputs still needs it. The inputs it did not cover (refused, skipped, not reached) keep theirs, +// so carry_find still serves them. Returns how many of j's entries remain (packed ones until the pruned pack is in +// place); 0 drops j from identities.txt. +static size_t carry_prune_one(const CarryItem *it, size_t m, int j, Drops *d) { + Set64 keep = {0}, done = {0}; + size_t left0 = d->left, kept = 0; + for (size_t i = 0; i < m; i++) + if ((it[i].has >> j & 1) && it[i].state != CI_COVERED) { + kept++; + if (it[i].h_known) set_add(&keep, hash8(it[i].h)); + } + for (size_t i = 0; i < m; i++) { + if (!(it[i].has >> j & 1) || it[i].state != CI_COVERED) continue; + uint8_t key[32], name[32]; + key_of_input_id(C.prev_eff[j], it[i].iid, key); + drop_file(d, PACK_SPV, key), drop_file(d, PACK_RES, key); + if (!set_add(&done, hash8(it[i].h)) || set_has(&keep, hash8(it[i].h))) continue; + use_name_id(C.prev_eff[j], it[i].h, name); + drop_file(d, PACK_USE, name); + } + free(keep.v), free(done.v); + return kept + (d->left - left0); +} + +// Writes shader-cache.pack.pruned: the open pack without the dropped files, with a trailer naming the pack it was copied +// from (pack_swap_pruned puts it in place at the next start; the open pack cannot be replaced while this process reads +// it). Written under another name, flushed, then renamed, so a copy that exists is whole. 1 when it was written. +static int carry_repack(Drops *d) { + char tmp[PATH_CAP + 48], out[PATH_CAP + 48]; + qsort(d->v, d->n, sizeof *d->v, drop_cmp); + PackEnt *nix = malloc(((size_t)pack_n + 1) * sizeof *nix); + uint64_t *src = malloc(((size_t)pack_n + 1) * sizeof *src), at; + uint32_t k = 0; + PlatFileId id; + uint64_t have = 0; + if (!nix || !src || !plat_file_id(pack_file, &id) || pruned_of(&id, &have)) { + // A copy of this pack already waits for the next start (its swap failed, or this pass ran before): the + // whole pack is not copied a second time; the pass after the swap prunes what it left in. + if (have) rt_log("gfx_shader: carry: a pruned copy of the cache pack waits for the next start already"); + free(nix), free(src); + return 0; + } + for (uint32_t i = 0; i < pack_n; i++) { + PackDrop key = {pack_ix[i].kind, {0}}; + memcpy(key.name, pack_ix[i].name, 32); + if (bsearch(&key, d->v, d->n, sizeof *d->v, drop_cmp)) continue; + src[k] = pack_ix[i].offset, nix[k] = pack_ix[i], k++; + } + at = 16 + (uint64_t)k * sizeof *nix; + for (uint32_t i = 0; i < k; i++) nix[i].offset = at, at += nix[i].size; + snprintf(tmp, sizeof tmp, "%s/shader-cache.pack.pruned.tmp", cache_dir); + snprintf(out, sizeof out, "%s/shader-cache.pack.pruned", cache_dir); + plat_file_delete(tmp); + int err = 0; + PlatFile *f = plat_fs_create(tmp, &err); + uint8_t head[16] = "TKSPACK1", trailer[PACK_TRAILER] = PACK_SRC_MAGIC; + memcpy(head + 8, &k, 4); + file_id_put(trailer + 8, &id); + size_t cap = 4u << 20, fill = 0; + uint8_t *buf = malloc(cap); + at = 16 + (uint64_t)k * sizeof *nix; // where buf's first byte goes + int ok = f && buf && plat_file_pwrite(f, head, 16, 0) == 16 && + plat_file_pwrite(f, nix, k * (uint32_t)sizeof *nix, 16) == (int64_t)k * (int64_t)sizeof *nix; + for (uint32_t i = 0; ok && i <= k; i++) { + uint32_t size = i < k ? nix[i].size : 0; + if (i == k || fill + size > cap) { // the buffer out; between writes on-demand translation goes first + ok = !fill || plat_file_pwrite(f, buf, (uint32_t)fill, at) == (int64_t)fill; + at += fill, fill = 0; + carry_wait(); + } + if (i == k || !ok) continue; + if (size > cap) { + uint8_t *big = malloc(size); + ok = big && pack_read_at(src[i], big, size) && plat_file_pwrite(f, big, size, at) == (int64_t)size; + at += size; + free(big); + } else { + ok = pack_read_at(src[i], buf + fill, size); + fill += size; + } + } + ok = ok && plat_file_pwrite(f, trailer, PACK_TRAILER, at) == PACK_TRAILER && plat_file_fsync(f); + if (f) plat_file_close(f); + ok = ok && plat_file_replace(tmp, out, 1); + if (!ok) plat_file_delete(tmp); + if (ok) + rt_log("gfx_shader: carry: wrote %s: %u of the pack's %u files (%.0f MiB), without %zu of previous translators; " + "it takes the pack's place at the next start", out, k, pack_n, (double)(at + PACK_TRAILER) / 1048576.0, d->n); + else rt_log("gfx_shader: carry: cannot write %s; the pack keeps previous translators' files for now", out); + free(buf), free(nix), free(src); + return ok; +} + +static void carry_main(void *arg) { + (void)arg; + plat_thread_priority(PLAT_PRIO_LOWEST); + double t0 = (double)plat_now_ms(), logged = t0; + char tmp[PATH_CAP + 48]; + snprintf(tmp, sizeof tmp, "%s/shader-cache.pack.pruned.tmp", cache_dir); + plat_file_delete(tmp); // a copy an earlier process did not finish + uint8_t (*ids)[32] = NULL; + size_t n = other_inputs(NULL, 0, &ids), m = 0, per[IDS_MAX] = {0}; + CarryItem *it = calloc(n + 1, sizeof *it); + for (size_t i = 0; it && i < n; i++) { // the stored inputs a previous translator has an entry of + uint8_t has = 0; + for (int j = 0; j < C.n_prev; j++) { + uint8_t key[32]; + key_of_input_id(C.prev_eff[j], ids[i], key); + if (entry_present(key)) has |= (uint8_t)(1u << j), per[j]++; + } + if (has) memcpy(it[m].iid, ids[i], 32), it[m++].has = has; + } + free(ids); + if (!it) { + rt_log("gfx_shader: carry: out of memory; no background pass"); + return; + } + char list[IDS_MAX * 32] = ""; + for (int j = 0, k = 0; j < C.n_prev && k < (int)sizeof list; j++) + k += snprintf(list + k, sizeof list - (size_t)k, "%s%02x%02x%02x%02x: %zu", j ? ", " : "", C.prev[j][0], + C.prev[j][1], C.prev[j][2], C.prev[j][3], per[j]); + rt_log("gfx_shader: carry: background pass started: %zu of %zu stored inputs have a previous translator's " + "translation (%s); they are translated again at idle priority for the next start", m, n, list); + size_t had = 0, xlated = 0, failed = 0, skipped = 0; + int unavailable = 0; + for (size_t i = 0; i < m && unavailable < 3; i++) { + carry_wait(); + CarryItem *c = &it[i]; + Inputs in; + Req r; + plat_thread_priority(PLAT_PRIO_NORMAL); // req_init takes the cache lock the render thread reads under + int ok = in_load(c->iid, &in); + if (ok && !req_init(&r, in.stage, in.control, in.code, in.length, in.c1, &in.a)) free(in.buf), ok = 0; + plat_thread_priority(PLAT_PRIO_LOWEST); + if (!ok) { + c->state = CI_SKIPPED, skipped++; + continue; + } + memcpy(c->h, r.code_hash, 32), c->h_known = 1; + if (r.final && entry_present(r.key)) { // the current translator has it already + c->state = CI_COVERED, had++; + } else { + char why[512], hex[65]; + int k = carry_translate(&r, &in, why, sizeof why); + if (k > 0) { + c->state = CI_COVERED, xlated++, unavailable = 0; + } else if (!k) { + c->state = CI_FAILED, failed++; + key_hex(c->iid, hex); + if (failed <= 64) + rt_log("gfx_shader: carry: stored input %s does not translate with the current translator (%s); " + "its previous translation stays in use", hex, why); + } else { + c->state = CI_SKIPPED, skipped++, unavailable++; + if (unavailable == 3) rt_log("gfx_shader: carry: %s; the background pass stops", why); + } + } + free(in.buf); + if ((double)plat_now_ms() - logged >= 10000) { + logged = (double)plat_now_ms(); + rt_log("gfx_shader: carry: %zu of %zu done: %zu translated, %zu had the current translator's already, %zu " + "failed, %zu skipped; %lld shaders served from a previous translator so far (%.0f s)", i + 1, m, xlated, + had, failed, skipped, (long long)C.served, (logged - t0) / 1000); + } + } + child_stop(&carry_svc); + rt_log("gfx_shader: carry: background pass finished in %.0f s: %zu stored inputs, %zu translated again, %zu had the " + "current translator's already, %zu failed (their previous translations stay in use), %zu skipped or not " + "reached; %lld shaders served from a previous translator so far", ((double)plat_now_ms() - t0) / 1000, m, + xlated, had, failed, m - xlated - had - failed, (long long)C.served); + // The entries the pass no longer needs. In this process each input it covered now has a current key on the disk + // whose use mask is known, and lookups try that before a previous translator's disk entry; the shaders already + // served stay in memory as they are. + Drops d = {0}; + uint8_t gone[IDS_MAX][32]; + int n_gone = 0; + for (int j = 0; j < C.n_prev; j++) + if (!carry_prune_one(it, m, j, &d)) memcpy(gone[n_gone++], C.prev[j], 32); + free(it); + int repacked = d.n && carry_repack(&d); + plat_mutex_lock(&ids_lock); + for (int j = 0; j < n_gone && C.n_dropped < IDS_MAX; j++) memcpy(C.dropped[C.n_dropped++], gone[j], 32); + plat_mutex_unlock(&ids_lock); + // The cache holds the current translator's translations of what the pass covered: the current identity stays + // listed, first, even when this process stored or read nothing else under it. + if (had || xlated) plat_xchg32(&C.noted, 1); + ids_write(); + rt_log("gfx_shader: carry: %zu files of previous translators deleted, %zu packed ones %s; %d of %d previous " + "translators stay in identities.txt", d.deleted, d.n, + repacked ? "left out of the pruned pack" : "stay in the pack for now", C.n_prev - n_gone, C.n_prev); + free(d.v); +} + +void gfx_shader_carry_start(void) { + static volatile int32_t started; + paths(); + if (!carry_on() || !cache_dir[0] || plat_xchg32(&started, 1)) return; + mkdir_p(cache_dir); + int lk = carry_lock(); + if (lk != 1) { + rt_log("gfx_shader: carry: %s; no background pass in this process", + lk ? "cannot make carry.lock in the shader cache" : "another process uses this shader cache"); + return; + } + if (!plat_thread_spawn(carry_main, NULL, "shader carry", 0)) rt_log("gfx_shader: carry: cannot start the background pass"); +} diff --git a/runtime/platform/linux.c b/runtime/platform/linux.c index 9b44cdf4..9cdae4b5 100644 --- a/runtime/platform/linux.c +++ b/runtime/platform/linux.c @@ -1835,6 +1835,10 @@ void plat_child_stop(PlatChild *c) { free(c); } +void plat_child_lower(PlatChild *c) { // a higher nice value needs no privilege + if (c) setpriority(PRIO_PROCESS, (id_t)c->pid, 19); +} + // A child that died makes the write fail with EPIPE; SIGPIPE is ignored for the process (main.c), as Windows has none. int plat_child_write(PlatChild *c, const void *data, size_t n) { return write_all(c->in, data, n); } diff --git a/runtime/platform/platform.h b/runtime/platform/platform.h index b54bbfa6..d261896e 100644 --- a/runtime/platform/platform.h +++ b/runtime/platform/platform.h @@ -459,6 +459,9 @@ typedef struct PlatChild PlatChild; // Children start one at a time so none inherits another pipe ends. PlatChild *plat_child_start(const char *exe, const char *arg, uint32_t *err); void plat_child_stop(PlatChild *c); // kills the child and frees c +// Runs the child below every normal-priority process (Windows: the idle priority class; elsewhere nice 19), for work +// that must not slow the game down (gfx_shader.c, the carry pass). +void plat_child_lower(PlatChild *c); int plat_child_write(PlatChild *c, const void *data, size_t n); // 1 when all n bytes were written #define PLAT_CHILD_READ_MS 120000 // plat_child_read gives up on a child that sends nothing for this long long plat_child_read(PlatChild *c, void *buf, size_t cap); // bytes read (at least 1); 0 or negative: pipe broke or timed out diff --git a/runtime/platform/win32.c b/runtime/platform/win32.c index 33aaa645..bc58aa1f 100644 --- a/runtime/platform/win32.c +++ b/runtime/platform/win32.c @@ -1913,6 +1913,10 @@ void plat_child_stop(PlatChild *c) { free(c); } +void plat_child_lower(PlatChild *c) { + if (c) SetPriorityClass(c->proc, IDLE_PRIORITY_CLASS); +} + int plat_child_write(PlatChild *c, const void *data, size_t n) { DWORD put = 0; return WriteFile(c->in, data, (DWORD)n, &put, NULL) && put == n;