Read a pipeline cache larger than 2 GB on Windows, and log every refused file with its size.

cache_load measured pipeline-cache.bin with ftell into a long, which is 32 bits on Windows: a file past 2 GB measured as -1, so the run started with an empty cache and the log said nothing, not even that the file was over pipeline_cache_max_mb. The read now lives in runtime/vk_cache_file.h: the length comes from the 64-bit tell, the read runs in 64 MB pieces, and a refusal returns its reason and the file's size, which cache_load logs. The Windows atomic write no longer cuts a block of 4 GB or more to a 32-bit count.

tests/pipeline_cache_file_check.py reads a sparse file just over 2 GB through the header. With --probe-prewarm and such a file carrying this GPU's header, the log reads "holds 2048 MB, over NX2NAVI_PIPELINE_CACHE_MAX_MB 512" under the default limit and "2048.0 MB read" with the limit off; the exe before this change logged nothing for the same file.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
chrg6
2026-11-11 04:29:31 -05:00
parent 84b0672ec2
commit 871b06e2e7
6 changed files with 247 additions and 17 deletions

View File

@@ -48,6 +48,7 @@
#include "nvn_hmap.h"
#include "platform/platform.h"
#include "tex.h"
#include "vk_cache_file.h"
#include "vk_gpa.h"
#include "vk_warm_hint.h"
@@ -601,33 +602,32 @@ static void cache_load(VkCtx *c) {
}
void *data = NULL;
size_t size = 0;
FILE *f = P.cache_path[0] ? fopen(P.cache_path, "rb") : NULL;
if (f) {
fseek(f, 0, SEEK_END);
long len = ftell(f);
fseek(f, 0, SEEK_SET);
if (P.cache_path[0]) {
// NX2NAVI_PIPELINE_CACHE_MAX_MB (default 512; 0 no limit): a larger file starts the run empty, and the run's
// saves replace it. The file only grows: entries of earlier translators and builds stay, and on NVIDIA
// (621.01) an entry another executable wrote is neither used nor written again, so a renamed or updated exe
// compiles while the file keeps its size (2026-11-09: 650 MB, 15-18 % of the pipelines read from it). The
// whole file sits in process memory twice at the boot (this read and the driver's copy) and is copied out
// at every save, which a 16 GB machine near its commit limit cannot spare. The size is checked before the
// read, so a file that is dropped never takes its size in memory.
uint64_t cap = (uint64_t)strtoull(cfg_value("NX2NAVI_PIPELINE_CACHE_MAX_MB"), NULL, 10) << 20;
if (len > 0 && cap && (uint64_t)len > cap) {
// read, so a file that is dropped never takes its size in memory. Sizes are 64-bit all the way
// (vk_cache_file.h cache_file_read): a file past 2 GB is read with the limit raised or off, and refused by
// its real size otherwise.
uint64_t cap = (uint64_t)strtoull(cfg_value("NX2NAVI_PIPELINE_CACHE_MAX_MB"), NULL, 10) << 20, len = 0;
int why = CACHE_FILE_NONE;
data = cache_file_read(P.cache_path, sizeof(VkPipelineCacheHeaderVersionOne), cap, &len, &why);
if (data) size = (size_t)len;
else if (why == CACHE_FILE_OVER_CAP)
rt_log("vk_pipeline: %s holds %.0f MB, over NX2NAVI_PIPELINE_CACHE_MAX_MB %llu: starting empty; this run "
"rewrites it", P.cache_path, len / 1048576.0, (unsigned long long)(cap >> 20));
len = 0;
}
if (len > (long)sizeof(VkPipelineCacheHeaderVersionOne) && (data = malloc((size_t)len)) &&
fread(data, 1, (size_t)len, f) == (size_t)len)
size = (size_t)len;
fclose(f);
else if (why != CACHE_FILE_NONE)
rt_log("vk_pipeline: %s (%llu bytes, %.1f MB) is not read: %s; starting empty", P.cache_path,
(unsigned long long)len, len / 1048576.0, cache_file_why(why));
VkPipelineCacheHeaderVersionOne h;
if (size) memcpy(&h, data, sizeof h);
if (size && (h.headerVersion != VK_PIPELINE_CACHE_HEADER_VERSION_ONE || h.vendorID != c->props.vendorID ||
h.deviceID != c->props.deviceID || memcmp(h.pipelineCacheUUID, c->props.pipelineCacheUUID, VK_UUID_SIZE))) {
rt_log("vk_pipeline: %s was written by another device or driver; starting empty", P.cache_path);
rt_log("vk_pipeline: %s (%.1f MB) was written by another device or driver; starting empty", P.cache_path,
size / 1048576.0);
size = 0;
}
}