Make the page write generations 64-bit, so a long session no longer wraps them and every texture, mirror and shadow check after the wrap answers wrong.

vk_memory.c handed out page write generations from a 32-bit counter (gen_now) and every consumer compared them as
32-bit values, with 0 meaning "never stamped". At 124 to 336 generations a frame (a 0.10.0 log with trace = perf) the
counter wrapped after about 66 h of play at 60 fps, 27.6 h at 144 and 15 to 17 h at 240. Past the wrap a stamp made
before it compared newer than every later check and one made after it older: textures and VRAM mirrors missed the
CPU's writes (stale textures, and the lazy fetch copying older mirror bytes over a CPU write), pages nothing wrote read
as written at every check (a hash or upload every time, and every such page of a mirrored pool copied up at each
sync), and the generation handed out at the wrap was 0. At 64 bits nothing wraps: 2^64 lasts millions of years at
336 a frame and 240 fps. Below 2^32 every value and every compare is the same as before.

What changed: gen_now, both tables (gen_chunk, cgen_chunk), reg_max and creg_max, PoolRec's synced_gen, mgen and sgen,
dirty_gen and DIRTY_UNSYNCED, VSample.mgen, every local and parameter that holds a generation in vk_memory.c,
vk_texture.c (Tex.wgen and gpu_gen, PEnt.since, storage_rewritten and its callers, sweep_hard's write-back check
included) and tex_probe.c, and vk.h's vk_mem_changed, vk_mem_changed_snap and vk_mem_unchanged(_why)_locked. The verify
mode's mismatch line prints the generations with %llu. The readers that take no gen_lock (lazy_clean, cpu_wrote_since,
verify_cb, vk_mem_sizes) stay lock-free at 64 bits: the slots, mgen, gen_now and a new chunk pointer are now stored
atomically (relaxed, or release where a reader acquires), the same instructions on x86-64, and a static assertion
holds that 64-bit atomics are lock-free. No lock, wait or thread changes.

gen_next, which hands out every generation, no longer warns at 0xC0000000 and up or at the wrap, which cannot come;
it logs one line when the count reaches 2^32, where earlier versions ran out, with the frame, the hours and the rate.

Cost: the tables take 8 bytes a page instead of 4 (512 KiB more per 256 MiB of guest address space stamped, both
tables together, allocated as before when a chunk is first stamped), reg_max and creg_max 2 MiB each instead of 1,
Tex and PoolRec 8 bytes more, PEnt 8 more. The page walks read 8 bytes a slot; a micro-benchmark of the walk measured no
difference.

Not reproduced in the game before the change: the wrong answers were reproduced by compiling vk_memory.c's generation
code with the counter seeded below 2^32. tests/gen_wrap_check now runs that code, with the readers outside its
section, across 2^32, 2^40 and 2^63 and checks that the compares answer as below the boundary; on the code before the
change 19 of its checks fail at 2^32 and the other two seeds do not fit the counter. The new tests/gen_width_check
compiles vk_memory.c, vk_texture.c and tex_probe.c with -Wshorten-64-to-32, -Wformat and -Wincompatible-pointer-types
as errors, so a 32-bit local left behind is found (shown by reverting single locals, fields and formats). Checks that
passed: gen_wrap_check, gen_width_check, pass_cc_check of every runtime file, no new -Wall -Wextra warnings in the
three files, tex_release_check, tex_compact_check, wb_lazy_check, gpa_check, mem_growth_check, mem_budget_check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
chrg6
2026-11-13 18:57:10 -05:00
parent 879c76f91a
commit 6b95c867c8
8 changed files with 426 additions and 181 deletions

View File

@@ -755,9 +755,17 @@ every copy returns, a heap page nothing touched yet reads zero and stays reserve
left out. Windows only, no game files. left out. Windows only, no game files.
A change to the page write generations (`runtime/vk_memory.c`, from its "page write generations" banner to its "dirty A change to the page write generations (`runtime/vk_memory.c`, from its "page write generations" banner to its "dirty
census" banner: `gen_next`, `gen_mark`) passes `python tests\gen_wrap_check.py`, which builds that part of the file census" banner, and the readers of its tables `cpu_wrote_since`, `shadow_pool_clean`, `lazy_clean`, `page_dirty`)
alone: the lines logged as the 32-bit generations near their wrap, and the four comparisons known to answer wrong across passes `python tests\gen_wrap_check.py`, which builds those parts of the file alone and runs the generations across
it. No GPU, no game files. 2^32 (where 32-bit generations wrapped), 2^40 and 2^63: texture checks, the snapshot path and the region check, the
CPU-side tables, and mirror and shadow cleanliness answer as below the boundary, and one line is logged when the count
reaches 2^32. No GPU, no game files.
A change to `runtime/vk_memory.c`, `runtime/vk_texture.c` or `runtime/tex_probe.c` passes
`python tests\gen_width_check.py`, which compiles the three files with the build's flags and `-Wshorten-64-to-32`,
`-Wformat` and `-Wincompatible-pointer-types` as errors: the page write generations are 64 bits, and a local, field or
format left at 32 bits would bring their wrap back without a word. Any narrowing in these files takes an explicit cast.
It needs the shader headers (`tools/build_shaders.py`); no GPU, no game files.
A change to which frames FSR 3.1 frame generation is told are camera cuts (`runtime/vk_fg_reset.h`, `runtime/vk_fg.cpp` A change to which frames FSR 3.1 frame generation is told are camera cuts (`runtime/vk_fg_reset.h`, `runtime/vk_fg.cpp`
`generate` and `vk_fg_frame`) passes `python tests\fg_reset_check.py`, which builds the header alone and checks that each `generate` and `vk_fg_frame`) passes `python tests\fg_reset_check.py`, which builds the header alone and checks that each

View File

@@ -398,7 +398,7 @@ static int test_import(pool_t *pool) {
vmaDestroyBuffer(c->vma, sb, sa); vmaDestroyBuffer(c->vma, sb, sa);
// generations: written after `since` -> 1, then 0, explicit host write -> 1 // generations: written after `since` -> 1, then 0, explicit host write -> 1
uint32_t g0, g1, g2, g3; uint64_t g0, g1, g2, g3;
vk_mem_changed(a, pool->size, 0, &g0); vk_mem_changed(a, pool->size, 0, &g0);
m[0x8000] = 1; m[0x8000] = 1;
GCHECK(vk_mem_changed(a + 0x8000, 0x1000, g0, &g1) == 1, "a CPU write after a generation is not seen"); GCHECK(vk_mem_changed(a + 0x8000, 0x1000, g0, &g1) == 1, "a CPU write after a generation is not seen");
@@ -421,7 +421,7 @@ static int test_snap_runs(uint8_t *h) {
uint64_t base = (uintptr_t)h; uint64_t base = (uintptr_t)h;
GCHECK(!(base & 0x1fffff), "the heap at %#llx does not start on a 2 MiB region", (unsigned long long)base); GCHECK(!(base & 0x1fffff), "the heap at %#llx does not start on a 2 MiB region", (unsigned long long)base);
int left = -1; int left = -1;
uint32_t g0, now; uint64_t g0, now;
vk_texture_new_submit(); vk_texture_new_submit();
uint32_t ep = vk_texture_epoch(); uint32_t ep = vk_texture_epoch();
vk_mem_gen_lock(); vk_mem_gen_lock();

View File

@@ -527,11 +527,11 @@ int vk_mem_lookup(uint64_t addr, uint64_t size, VkMemSpan *out);
uint64_t vk_mem_gen(void); uint64_t vk_mem_gen(void);
// Page write generations over guest memory (mem_written_pages plus vk_guest_written). Returns 1 when a page of the // Page write generations over guest memory (mem_written_pages plus vk_guest_written). Returns 1 when a page of the
// range was written after generation `since`, 0 when not, -1 when the range is not write-watched (treat as written); // range was written after generation `since`, 0 when not, -1 when the range is not write-watched (treat as written);
// *now is the generation to pass next time. // *now is the generation to pass next time. Generations are 64 bits and do not wrap; 0 is never handed out.
int vk_mem_changed(uint64_t addr, uint64_t size, uint32_t since, uint32_t *now); int vk_mem_changed(uint64_t addr, uint64_t size, uint64_t since, uint64_t *now);
// The same answer from per-epoch write-watch snapshots of 2 MiB regions: no system call once the regions were queried in // The same answer from per-epoch write-watch snapshots of 2 MiB regions: no system call once the regions were queried in
// this epoch (a guest submission). For checks that may see writes made during the epoch one epoch late. // this epoch (a guest submission). For checks that may see writes made during the epoch one epoch late.
int vk_mem_changed_snap(uint64_t addr, uint64_t size, uint32_t since, uint32_t epoch, uint32_t *now); int vk_mem_changed_snap(uint64_t addr, uint64_t size, uint64_t since, uint32_t epoch, uint64_t *now);
void vk_mem_snap_stats(uint64_t *checks, uint64_t *calls, uint64_t *untracked); // since the last call void vk_mem_snap_stats(uint64_t *checks, uint64_t *calls, uint64_t *untracked); // since the last call
// Upload batch: one command buffer on gfx_queue that texture uploads and texel copies record into. begin/end bracket // Upload batch: one command buffer on gfx_queue that texture uploads and texel copies record into. begin/end bracket
@@ -801,10 +801,10 @@ int vk_texture_slot_retry_locked(int id);
// have changed), 0 when a closer look is due (vk_mem_changed_snap). // have changed), 0 when a closer look is due (vk_mem_changed_snap).
void vk_mem_gen_lock_shared(void); void vk_mem_gen_lock_shared(void);
void vk_mem_gen_unlock_shared(void); void vk_mem_gen_unlock_shared(void);
int vk_mem_unchanged_locked(uint64_t addr, uint64_t size, uint32_t since, uint32_t epoch); int vk_mem_unchanged_locked(uint64_t addr, uint64_t size, uint64_t since, uint32_t epoch);
// The same check's reason when it fails: 1 a region without this epoch's snapshot yet, 2 one untracked, 3 one written // The same check's reason when it fails: 1 a region without this epoch's snapshot yet, 2 one untracked, 3 one written
// since, 4 nothing to check (no size, beyond the regions); 0 unchanged. // since, 4 nothing to check (no size, beyond the regions); 0 unchanged.
int vk_mem_unchanged_why_locked(uint64_t addr, uint64_t size, uint32_t since, uint32_t epoch); int vk_mem_unchanged_why_locked(uint64_t addr, uint64_t size, uint64_t since, uint32_t epoch);
// The lock exclusively, for the snapshot of the regions of [addr, addr+size) that have none in epoch yet (the number of // The lock exclusively, for the snapshot of the regions of [addr, addr+size) that have none in epoch yet (the number of
// write-watch queries it took). // write-watch queries it took).
void vk_mem_gen_lock(void); void vk_mem_gen_lock(void);

View File

@@ -60,7 +60,7 @@ typedef struct PoolRec {
volatile int32_t used; // resolved at least once: the GPU may still read it at finalize volatile int32_t used; // resolved at least once: the GPU may still read it at finalize
// NX2NAVI_TRACE=perf dirty census (vk_mem_dirty_submit), replay thread under the submit lock only: // NX2NAVI_TRACE=perf dirty census (vk_mem_dirty_submit), replay thread under the submit lock only:
uint32_t bound_sub; // last submit that bound it as a buffer (dirty_sub), 0 never uint32_t bound_sub; // last submit that bound it as a buffer (dirty_sub), 0 never
uint32_t synced_gen; // page generation its lazy-mirror count covers; DIRTY_UNSYNCED until scanned uint64_t synced_gen; // page generation its lazy-mirror count covers; DIRTY_UNSYNCED until scanned
uint8_t bufbound; // ever bound as a vertex, index, uniform, storage or indirect buffer uint8_t bufbound; // ever bound as a vertex, index, uniform, storage or indirect buffer
uint8_t untracked; // the last dirty scan found no write watch over it uint8_t untracked; // the last dirty scan found no write watch over it
// VRAM mirror (POOL_IMPORT, "VRAM mirror" below). mstate and mbuf change on the replay thread only. // VRAM mirror (POOL_IMPORT, "VRAM mirror" below). mstate and mbuf change on the replay thread only.
@@ -86,7 +86,7 @@ typedef struct PoolRec {
uint8_t twin; // MS_ON: the table's writes reach guest memory too (twin_wanted); 0: the mirror only uint8_t twin; // MS_ON: the table's writes reach guest memory too (twin_wanted); 0: the mirror only
uint32_t vd_wrote_sub, vd_use_sub; // verify mode: the last submission that bound it written / used its mirror uint32_t vd_wrote_sub, vd_use_sub; // verify mode: the last submission that bound it written / used its mirror
uint64_t vd_wrote_n; // verify mode: written bindings of it so far uint64_t vd_wrote_n; // verify mode: written bindings of it so far
uint32_t mgen; // gen_lock: every page write stamped up to this generation is in the mirror uint64_t mgen; // gen_lock: every page write stamped up to this generation is in the mirror
// (or in a copy queued before its next use) // (or in a copy queued before its next use)
FlushRange fl[FLUSH_RANGES + 1]; // flush_lock: nvnMemoryPoolFlushMappedRange pages not copied yet (flush_add) FlushRange fl[FLUSH_RANGES + 1]; // flush_lock: nvnMemoryPoolFlushMappedRange pages not copied yet (flush_add)
int fl_n; // flush_lock: ranges in fl; read without the lock to skip an empty list int fl_n; // flush_lock: ranges in fl; read without the lock to skip an empty list
@@ -96,7 +96,7 @@ typedef struct PoolRec {
// Shadow storage (POOL_IMPORT in shadow mode, "shadow storage" below): buf is a host-visible copy of the guest pages. // Shadow storage (POOL_IMPORT in shadow mode, "shadow storage" below): buf is a host-visible copy of the guest pages.
VmaAllocation salloc; VmaAllocation salloc;
uint8_t *smap; // its mapping; offsets as in buf (from import_lo) uint8_t *smap; // its mapping; offsets as in buf (from import_lo)
uint32_t sgen; // gen_lock: every page write stamped up to this generation is in the shadow uint64_t sgen; // gen_lock: every page write stamped up to this generation is in the shadow
uint64_t sborn; // the value of the first submission that can have used the shadow (shadow_pool) uint64_t sborn; // the value of the first submission that can have used the shadow (shadow_pool)
volatile uint8_t stouched; // on the touched list: the GPU may use it in the submission being recorded volatile uint8_t stouched; // on the touched list: the GPU may use it in the submission being recorded
uint32_t stab_i; // shadow_lock: 1 + its index in stab (the table's shadows), 0 when not there uint32_t stab_i; // shadow_lock: 1 + its index in stab (the table's shadows), 0 when not there
@@ -1160,49 +1160,52 @@ int vk_mem_vram_live(uint64_t *used, uint64_t *budget) {
// ---------------------------------------------------------------- page write generations // ---------------------------------------------------------------- page write generations
// Two-level table over the guest address space (below 2^39): 256 MiB chunks of 65,536 page generations, allocated // Two-level table over the guest address space (below 2^39): 256 MiB chunks of 65,536 page generations, allocated
// when a page of the chunk is first stamped. 0 = never stamped. // when a page of the chunk is first stamped. 0 = never stamped.
// Generations are 64 bits and never run out (at 336 a frame and 240 fps, 2^64 lasts millions of years). At 32 bits,
// 336 a frame (the most measured, 0.10.0) and 240 fps ran them out after about 15 h of play, and past the wrap a stamp
// from before it compared newer than every later check and one made after it older: textures and pool mirrors missed
// the CPU's writes, and pages nothing wrote read as written at every check. Below 2^32 every value and every compare
// is what it was at 32 bits.
#define GEN_CHUNK_SHIFT 28 #define GEN_CHUNK_SHIFT 28
#define GEN_PAGE_SHIFT 12 #define GEN_PAGE_SHIFT 12
#define GEN_CHUNKS (RT_ADDRESS_LIMIT >> GEN_CHUNK_SHIFT) #define GEN_CHUNKS (RT_ADDRESS_LIMIT >> GEN_CHUNK_SHIFT)
#define GEN_PER_CHUNK (1u << (GEN_CHUNK_SHIFT - GEN_PAGE_SHIFT)) #define GEN_PER_CHUNK (1u << (GEN_CHUNK_SHIFT - GEN_PAGE_SHIFT))
static uint32_t *gen_chunk[GEN_CHUNKS]; static uint64_t *gen_chunk[GEN_CHUNKS];
static uint32_t gen_now; // last generation handed out static uint64_t gen_now; // last generation handed out (gen_lock; read without it atomically)
static PlatMutex gen_lock = PLAT_MUTEX_INIT; static PlatMutex gen_lock = PLAT_MUTEX_INIT;
static uint64_t gen_pages[1024]; static uint64_t gen_pages[1024];
// lazy_clean, cpu_wrote_since, verify_cb and vk_mem_sizes read generations without gen_lock: 64-bit loads and stores
// must be single instructions on every target.
_Static_assert(__atomic_always_lock_free(sizeof(uint64_t), 0), "64-bit page generations need lock-free atomics");
// Generations are 32 bits, and nothing compares them across a wrap: past it a stamp from before reads as newer than // When the count reaches 2^32, where the 32-bit generations of earlier versions ran out, the log says so once, with
// every later check, and a stamp of 0 as never stamped, so textures and pool mirrors the CPU writes can miss a write // the frame, the hours of play and the rate, so a long session's log shows how far it got. gen_lock held exclusively;
// and every check reads as a write. At about 181 a frame that comes after about 110 h of play at 60 fps, 27 h at 240. // the log takes no lock of this file.
// Only trace=perf's "mem: sizes" counted them, so gen_next, which hands each one out, logs where the session stands at static void gen_mark(uint64_t g) {
// 0xC0000000, 0xD0000000, 0xE0000000 and 0xF0000000, and the wrap itself (gen_mark): five lines at most.
static void gen_mark(uint32_t g) { // gen_lock held exclusively; the log takes no lock of this file
uint64_t frames = __atomic_load_n(&gfx_counters.frames, __ATOMIC_RELAXED); uint64_t frames = __atomic_load_n(&gfx_counters.frames, __ATOMIC_RELAXED);
double secs = rt_guest_start_ns ? (double)(kern_now_ns() - rt_guest_start_ns) / 1e9 : 0, all = 4294967296.0; double secs = rt_guest_start_ns ? (double)(kern_now_ns() - rt_guest_start_ns) / 1e9 : 0;
if (!g) { rt_log("mem: page generations reached %llu at frame %llu after %.1f h of play (%.0f a frame); before 0.10.2 they "
rt_log("mem: page generations ran out at frame %llu after %.1f h: from here on textures and pool mirrors the " "were 32 bits and ran out here, after which textures and pool mirrors could miss the CPU's writes",
"CPU writes can show old content and every check uploads again; restart the game", (unsigned long long)g, (unsigned long long)frames, secs / 3600, frames ? (double)g / (double)frames : 0.0);
(unsigned long long)frames, secs / 3600);
return;
}
rt_log("mem: page generations: %u of 4294967295 handed out at frame %llu (%.0f a frame); at this rate they run out "
"in about %.1f h of play, after which textures and pool mirrors the CPU writes can miss a write and every "
"check reads as a write: restart the game before then", g, (unsigned long long)frames,
frames ? g / (double)frames : 0.0, secs > 0 ? (all - g) * secs / g / 3600 : 0.0);
} }
static inline uint32_t gen_next(void) { // gen_lock held exclusively
uint32_t g = ++gen_now; // gen_lock held exclusively: the only place a generation is handed out. The store is atomic: vk_mem_sizes and
if (__builtin_expect(!(g & 0x0fffffffu) && (g >= 0xc0000000u || !g), 0)) gen_mark(g); // verify_cb read gen_now without gen_lock.
static inline uint64_t gen_next(void) {
uint64_t g = gen_now + 1;
if (__builtin_expect(g == 1ull << 32, 0)) gen_mark(g);
__atomic_store_n(&gen_now, g, __ATOMIC_RELAXED);
return g; return g;
} }
// The newest generation stamped on any page of each 2 MiB region (the write-watch snapshot regions below): the mirror // The newest generation stamped on any page of each 2 MiB region (the write-watch snapshot regions below): the mirror
// sync skips regions with nothing newer than its pool's mirror. // sync skips regions with nothing newer than its pool's mirror.
#define REG_SHIFT 21 #define REG_SHIFT 21
static uint32_t reg_max[RT_ADDRESS_LIMIT >> REG_SHIFT]; static uint64_t reg_max[RT_ADDRESS_LIMIT >> REG_SHIFT];
// The same two tables without the stamps of a mirror's own write-back (vk_gpu_written_back): what the CPU and the host // The same two tables without the stamps of a mirror's own write-back (vk_gpu_written_back): what the CPU and the host
// paths wrote. The mirror sync reads these, so the bytes a mirror wrote back into guest memory are not copied up into it // paths wrote. The mirror sync reads these, so the bytes a mirror wrote back into guest memory are not copied up into it
// again at its next sync: the storage buffers the game's compute shaders write went down and came back up every frame, // again at its next sync: the storage buffers the game's compute shaders write went down and came back up every frame,
// 20 MiB a frame over PCIe in Kokiri Forest (2026-11-09). // 20 MiB a frame over PCIe in Kokiri Forest (2026-11-09).
static uint32_t *cgen_chunk[GEN_CHUNKS]; static uint64_t *cgen_chunk[GEN_CHUNKS];
static uint32_t creg_max[RT_ADDRESS_LIMIT >> REG_SHIFT]; static uint64_t creg_max[RT_ADDRESS_LIMIT >> REG_SHIFT];
// Every write-watch query of the generation table (gen_lock held exclusively): calls, bytes and time since the last // Every write-watch query of the generation table (gen_lock held exclusively): calls, bytes and time since the last
// mirror summary, all consumers together; the queries that found written pages (they also flush TLBs) apart. // mirror summary, all consumers together; the queries that found written pages (they also flush TLBs) apart.
@@ -1269,35 +1272,38 @@ static uint32_t *fcand; // gen_lock: regions to try at the
static int nfcand, cfcand; static int nfcand, cfcand;
static uint64_t n_frz_refused; // gen_lock: candidates mem_freeze refused static uint64_t n_frz_refused; // gen_lock: candidates mem_freeze refused
static uint32_t *slot_in(uint32_t **tab, uint64_t page_addr, int create) { static uint64_t *slot_in(uint64_t **tab, uint64_t page_addr, int create) {
uint64_t ci = page_addr >> GEN_CHUNK_SHIFT; uint64_t ci = page_addr >> GEN_CHUNK_SHIFT;
if (ci >= GEN_CHUNKS) return NULL; if (ci >= GEN_CHUNKS) return NULL;
if (!tab[ci]) { if (!tab[ci]) {
if (!create) return NULL; if (!create) return NULL;
tab[ci] = calloc(GEN_PER_CHUNK, sizeof(uint32_t)); uint64_t *p = calloc(GEN_PER_CHUNK, sizeof *p);
if (!tab[ci]) rt_fatal("vk: out of memory for page generations"); if (!p) rt_fatal("vk: out of memory for page generations");
// published with release: lazy_clean and cpu_wrote_since load the chunk pointers without gen_lock
__atomic_store_n(&tab[ci], p, __ATOMIC_RELEASE);
} }
return &tab[ci][(page_addr >> GEN_PAGE_SHIFT) & (GEN_PER_CHUNK - 1)]; return &tab[ci][(page_addr >> GEN_PAGE_SHIFT) & (GEN_PER_CHUNK - 1)];
} }
static uint32_t *gen_slot(uint64_t page_addr, int create) { return slot_in(gen_chunk, page_addr, create); } static uint64_t *gen_slot(uint64_t page_addr, int create) { return slot_in(gen_chunk, page_addr, create); }
// gen_lock held exclusively; g is the newest generation handed out. back: a mirror's write-back, which leaves the // gen_lock held exclusively; g is the newest generation handed out. back: a mirror's write-back, which leaves the
// CPU-side tables alone. // CPU-side tables alone. The slots are stored atomically: lazy_clean, cpu_wrote_since and verify_cb read them
static void stamp_page_as(uint64_t a, uint32_t g, int back) { // without gen_lock.
uint32_t *s = gen_slot(a, 1); static void stamp_page_as(uint64_t a, uint64_t g, int back) {
uint64_t *s = gen_slot(a, 1);
if (!s) return; if (!s) return;
*s = g; __atomic_store_n(s, g, __ATOMIC_RELAXED);
reg_max[a >> REG_SHIFT] = g; reg_max[a >> REG_SHIFT] = g;
reg_last[a >> REG_SHIFT] = ww_ep; reg_last[a >> REG_SHIFT] = ww_ep;
if (back) return; if (back) return;
uint32_t *c = slot_in(cgen_chunk, a, 1); uint64_t *c = slot_in(cgen_chunk, a, 1);
if (c) *c = g, creg_max[a >> REG_SHIFT] = g; if (c) __atomic_store_n(c, g, __ATOMIC_RELAXED), creg_max[a >> REG_SHIFT] = g;
} }
static void stamp_page(uint64_t a, uint32_t g) { stamp_page_as(a, g, 0); } static void stamp_page(uint64_t a, uint64_t g) { stamp_page_as(a, g, 0); }
static void stamp_range(uint64_t lo, uint64_t hi, uint32_t g) { static void stamp_range(uint64_t lo, uint64_t hi, uint64_t g) {
for (uint64_t a = lo; a < hi; a += 1ull << GEN_PAGE_SHIFT) stamp_page(a, g); for (uint64_t a = lo; a < hi; a += 1ull << GEN_PAGE_SHIFT) stamp_page(a, g);
} }
@@ -1309,11 +1315,11 @@ void vk_guest_written(uint64_t addr, uint64_t size) {
plat_mutex_unlock(&gen_lock); plat_mutex_unlock(&gen_lock);
} }
int vk_mem_changed(uint64_t addr, uint64_t size, uint32_t since, uint32_t *now) { int vk_mem_changed(uint64_t addr, uint64_t size, uint64_t since, uint64_t *now) {
if (!size) { *now = since; return 0; } if (!size) { *now = since; return 0; }
uint64_t lo = addr & ~0xfffull, hi = (addr + size + 0xfff) & ~0xfffull; uint64_t lo = addr & ~0xfffull, hi = (addr + size + 0xfff) & ~0xfffull;
plat_mutex_lock(&gen_lock); plat_mutex_lock(&gen_lock);
uint32_t g = gen_next(); uint64_t g = gen_next();
int tracked = 1, frozen = 1; int tracked = 1, frozen = 1;
for (uint64_t a = lo & ~((1ull << REG_SHIFT) - 1); a < hi && frozen; a += 1ull << REG_SHIFT) frozen = mem_frozen(a); for (uint64_t a = lo & ~((1ull << REG_SHIFT) - 1); a < hi && frozen; a += 1ull << REG_SHIFT) frozen = mem_frozen(a);
while (!frozen) { while (!frozen) {
@@ -1324,7 +1330,7 @@ int vk_mem_changed(uint64_t addr, uint64_t size, uint32_t since, uint32_t *now)
} }
int changed = 0; int changed = 0;
for (uint64_t a = lo; tracked && a < hi && !changed; a += 1ull << GEN_PAGE_SHIFT) { for (uint64_t a = lo; tracked && a < hi && !changed; a += 1ull << GEN_PAGE_SHIFT) {
uint32_t *s = gen_slot(a, 0); uint64_t *s = gen_slot(a, 0);
if (s && *s > since) changed = 1; if (s && *s > since) changed = 1;
} }
plat_mutex_unlock(&gen_lock); plat_mutex_unlock(&gen_lock);
@@ -1346,7 +1352,7 @@ static uint64_t n_snap_checks, n_snap_calls, n_snap_untracked;
// gen_lock held exclusively. Queries the regions r0..r1 that were not snapped in epoch, one query per run of them; // gen_lock held exclusively. Queries the regions r0..r1 that were not snapped in epoch, one query per run of them;
// returns the number of queries. // returns the number of queries.
static uint64_t snap_regions(uint64_t r0, uint64_t r1, uint32_t epoch) { static uint64_t snap_regions(uint64_t r0, uint64_t r1, uint32_t epoch) {
uint32_t g = 0; uint64_t g = 0;
uint64_t calls = 0; uint64_t calls = 0;
for (uint64_t r = r0; r <= r1;) { for (uint64_t r = r0; r <= r1;) {
if (snap_at[r] == epoch) { r++; continue; } if (snap_at[r] == epoch) { r++; continue; }
@@ -1419,7 +1425,7 @@ static void snap_widen(uint64_t lo, uint64_t hi, uint64_t *r0, uint64_t *r1) {
plat_mutex_unlock_shared(&pool_lock); plat_mutex_unlock_shared(&pool_lock);
} }
int vk_mem_changed_snap(uint64_t addr, uint64_t size, uint32_t since, uint32_t epoch, uint32_t *now) { int vk_mem_changed_snap(uint64_t addr, uint64_t size, uint64_t since, uint32_t epoch, uint64_t *now) {
if (!size) { *now = since; return 0; } if (!size) { *now = since; return 0; }
uint64_t lo = addr & ~0xfffull, hi = (addr + size + 0xfff) & ~0xfffull; uint64_t lo = addr & ~0xfffull, hi = (addr + size + 0xfff) & ~0xfffull;
uint64_t r0 = lo >> SNAP_SHIFT, r1 = (hi - 1) >> SNAP_SHIFT; uint64_t r0 = lo >> SNAP_SHIFT, r1 = (hi - 1) >> SNAP_SHIFT;
@@ -1443,7 +1449,7 @@ int vk_mem_changed_snap(uint64_t addr, uint64_t size, uint32_t since, uint32_t e
if (reg_max[r] <= since) continue; if (reg_max[r] <= since) continue;
uint64_t a = r << REG_SHIFT > lo ? r << REG_SHIFT : lo, b = (r + 1) << REG_SHIFT < hi ? (r + 1) << REG_SHIFT : hi; uint64_t a = r << REG_SHIFT > lo ? r << REG_SHIFT : lo, b = (r + 1) << REG_SHIFT < hi ? (r + 1) << REG_SHIFT : hi;
for (; a < b && !changed; a += 1ull << GEN_PAGE_SHIFT) { for (; a < b && !changed; a += 1ull << GEN_PAGE_SHIFT) {
uint32_t *s = gen_slot(a, 0); uint64_t *s = gen_slot(a, 0);
if (s && *s > since) changed = 1; if (s && *s > since) changed = 1;
} }
} }
@@ -1461,7 +1467,7 @@ void vk_mem_gen_unlock_shared(void) { plat_mutex_unlock_shared(&gen_lock); }
void vk_mem_gen_lock(void) { plat_mutex_lock(&gen_lock); } void vk_mem_gen_lock(void) { plat_mutex_lock(&gen_lock); }
void vk_mem_gen_unlock(void) { plat_mutex_unlock(&gen_lock); } void vk_mem_gen_unlock(void) { plat_mutex_unlock(&gen_lock); }
int vk_mem_unchanged_why_locked(uint64_t addr, uint64_t size, uint32_t since, uint32_t epoch) { int vk_mem_unchanged_why_locked(uint64_t addr, uint64_t size, uint64_t since, uint32_t epoch) {
if (!size || !epoch) return 4; if (!size || !epoch) return 4;
uint64_t lo = addr & ~0xfffull, hi = (addr + size + 0xfff) & ~0xfffull; uint64_t lo = addr & ~0xfffull, hi = (addr + size + 0xfff) & ~0xfffull;
uint64_t r0 = lo >> SNAP_SHIFT, r1 = (hi - 1) >> SNAP_SHIFT; uint64_t r0 = lo >> SNAP_SHIFT, r1 = (hi - 1) >> SNAP_SHIFT;
@@ -1474,7 +1480,7 @@ int vk_mem_unchanged_why_locked(uint64_t addr, uint64_t size, uint32_t since, ui
return 0; return 0;
} }
int vk_mem_unchanged_locked(uint64_t addr, uint64_t size, uint32_t since, uint32_t epoch) { int vk_mem_unchanged_locked(uint64_t addr, uint64_t size, uint64_t since, uint32_t epoch) {
return !vk_mem_unchanged_why_locked(addr, size, since, epoch); return !vk_mem_unchanged_why_locked(addr, size, since, epoch);
} }
@@ -1561,7 +1567,7 @@ void vk_mem_snap_stats(uint64_t *checks, uint64_t *calls, uint64_t *untracked) {
// - lazy: pages stamped after the pool's previous count, in pools the previous submit bound (upload when bound); // - lazy: pages stamped after the pool's previous count, in pools the previous submit bound (upload when bound);
// - first use: the size of a pool the first time it is bound (a mirror copies the whole pool then). // - first use: the size of a pool the first time it is bound (a mirror copies the whole pool then).
// Classes: CPU_CACHED, CPU_UNCACHED, other (CPU_NO_ACCESS). Logged as `mem: dirty` every 60 frames. // Classes: CPU_CACHED, CPU_UNCACHED, other (CPU_NO_ACCESS). Logged as `mem: dirty` every 60 frames.
#define DIRTY_UNSYNCED UINT32_MAX #define DIRTY_UNSYNCED UINT64_MAX
#define DIRTY_GAP 0x100000ull // pools closer than this share one query (the gap's pages are stamped too) #define DIRTY_GAP 0x100000ull // pools closer than this share one query (the gap's pages are stamped too)
#define DIRTY_WIN 60 #define DIRTY_WIN 60
enum { DC_CACHED, DC_UNC, DC_OTHER, DC_N }; enum { DC_CACHED, DC_UNC, DC_OTHER, DC_N };
@@ -1569,7 +1575,8 @@ typedef struct {
uint64_t eager[DC_N], lazy[DC_N], first[DC_N]; uint64_t eager[DC_N], lazy[DC_N], first[DC_N];
uint64_t submits, queries, scan_ns, sub_max, frames; uint64_t submits, queries, scan_ns, sub_max, frames;
} DirtyAcc; } DirtyAcc;
static uint32_t dirty_sub = 1, dirty_gen; // submit lock: the current submit number, generation of the last scan static uint32_t dirty_sub = 1; // submit lock: the current submit number
static uint64_t dirty_gen; // submit lock: the generation of the last scan
static uint64_t first_pending[DC_N], ever_n[DC_N], ever_bytes[DC_N]; // submit lock static uint64_t first_pending[DC_N], ever_n[DC_N], ever_bytes[DC_N]; // submit lock
static PoolRec **dlist; // submit lock: the buffer-bound pools of a scan static PoolRec **dlist; // submit lock: the buffer-bound pools of a scan
static int dcap; static int dcap;
@@ -1597,7 +1604,7 @@ static uint64_t page_lo(const PoolRec *r) { return r->lo & ~0xfffull; }
static uint64_t page_hi(const PoolRec *r) { return (r->lo + r->size + 0xfff) & ~0xfffull; } static uint64_t page_hi(const PoolRec *r) { return (r->lo + r->size + 0xfff) & ~0xfffull; }
// gen_lock held exclusively. 0: the range is not write-watched. // gen_lock held exclusively. 0: the range is not write-watched.
static int query_stamp(uint64_t lo, uint64_t hi, uint32_t g) { static int query_stamp(uint64_t lo, uint64_t hi, uint64_t g) {
for (;;) { for (;;) {
uint64_t n = ww_query(lo, hi - lo); uint64_t n = ww_query(lo, hi - lo);
if (n == UINT64_MAX) return 0; if (n == UINT64_MAX) return 0;
@@ -1998,7 +2005,7 @@ static void shadow_sync_pool(PoolRec *r, uint32_t epoch, int cpu) {
s_queries += snap_regions(r0, r1, epoch); s_queries += snap_regions(r0, r1, epoch);
for (uint64_t k = r0; k <= r1 && tracked; k++) tracked = !snap_untracked[k]; for (uint64_t k = r0; k <= r1 && tracked; k++) tracked = !snap_untracked[k];
} else { // outside the snapshot regions, or before the first guest submission: one query per unfrozen run } else { // outside the snapshot regions, or before the first guest submission: one query per unfrozen run
uint32_t g = gen_next(); uint64_t g = gen_next();
for (uint64_t a = lo; a < hi;) { for (uint64_t a = lo; a < hi;) {
uint64_t b = (a | ((1ull << REG_SHIFT) - 1)) + 1; uint64_t b = (a | ((1ull << REG_SHIFT) - 1)) + 1;
if (b > hi) b = hi; if (b > hi) b = hi;
@@ -2010,7 +2017,7 @@ static void shadow_sync_pool(PoolRec *r, uint32_t epoch, int cpu) {
} }
} }
s_q_ticks += qpc() - t0; s_q_ticks += qpc() - t0;
uint32_t since = r->sgen; uint64_t since = r->sgen;
r->sgen = gen_now; r->sgen = gen_now;
if (!tracked) { // nothing tells which pages changed: all of them if (!tracked) { // nothing tells which pages changed: all of them
s_untracked++; s_untracked++;
@@ -2022,7 +2029,7 @@ static void shadow_sync_pool(PoolRec *r, uint32_t epoch, int cpu) {
uint64_t a = k << REG_SHIFT, b = a + (1ull << REG_SHIFT); uint64_t a = k << REG_SHIFT, b = a + (1ull << REG_SHIFT);
if (a < lo) a = lo; if (a < lo) a = lo;
if (b > hi) b = hi; if (b > hi) b = hi;
const uint32_t *ch = (cpu ? cgen_chunk : gen_chunk)[a >> GEN_CHUNK_SHIFT]; // a 2 MiB region lies in one chunk const uint64_t *ch = (cpu ? cgen_chunk : gen_chunk)[a >> GEN_CHUNK_SHIFT]; // a 2 MiB region lies in one chunk
if ((cpu ? creg_max : reg_max)[k] <= since || !ch) { if ((cpu ? creg_max : reg_max)[k] <= since || !ch) {
if (run) shadow_copy(r, run, a), run = 0; if (run) shadow_copy(r, run, a), run = 0;
continue; continue;
@@ -2086,8 +2093,8 @@ static int shadow_bulk_write(void) {
// such a page may have come after a CPU write it should have come before ("The game must not get ahead of the // such a page may have come after a CPU write it should have come before ("The game must not get ahead of the
// write-backs" above); a page of which the CPU wrote other bytes counts too. // write-backs" above); a page of which the CPU wrote other bytes counts too.
// No gen_lock, as lazy_clean: a stamp that lands meanwhile only blurs a count. // No gen_lock, as lazy_clean: a stamp that lands meanwhile only blurs a count.
static int cpu_wrote_since(uint64_t pg, uint32_t sgen) { static int cpu_wrote_since(uint64_t pg, uint64_t sgen) {
const uint32_t *ch = __atomic_load_n(&cgen_chunk[pg >> GEN_CHUNK_SHIFT], __ATOMIC_ACQUIRE); const uint64_t *ch = __atomic_load_n(&cgen_chunk[pg >> GEN_CHUNK_SHIFT], __ATOMIC_ACQUIRE);
if (ch && __atomic_load_n(&ch[(pg >> GEN_PAGE_SHIFT) & (GEN_PER_CHUNK - 1)], __ATOMIC_RELAXED) > sgen) return 1; if (ch && __atomic_load_n(&ch[(pg >> GEN_PAGE_SHIFT) & (GEN_PER_CHUNK - 1)], __ATOMIC_RELAXED) > sgen) return 1;
uint64_t p[1], n = mem_written_peek(pg, SB_PAGE, p, 1); uint64_t p[1], n = mem_written_peek(pg, SB_PAGE, p, 1);
return n && n != UINT64_MAX; return n && n != UINT64_MAX;
@@ -2168,7 +2175,7 @@ static void table_wb_cb(void *arg) {
s_tab_wb += moved, s_tab_wb_seen += seen, s_tab_wb_jobs++; s_tab_wb += moved, s_tab_wb_seen += seen, s_tab_wb_jobs++;
if (nwrote) { if (nwrote) {
plat_mutex_lock(&gen_lock); plat_mutex_lock(&gen_lock);
uint32_t g = gen_next(); uint64_t g = gen_next();
for (int i = 0; i < nwrote; i++) stamp_page_as(wrote[i], g, back); for (int i = 0; i < nwrote; i++) stamp_page_as(wrote[i], g, back);
plat_mutex_unlock(&gen_lock); plat_mutex_unlock(&gen_lock);
} }
@@ -2184,7 +2191,7 @@ static void gpu_written(uint64_t addr, uint64_t size, int back) {
uint64_t lo = addr & ~0xfffull, hi = (addr + size + 0xfff) & ~0xfffull; uint64_t lo = addr & ~0xfffull, hi = (addr + size + 0xfff) & ~0xfffull;
back = back && wb_no_reup(); // read before gen_lock: its first call logs back = back && wb_no_reup(); // read before gen_lock: its first call logs
plat_mutex_lock(&gen_lock); plat_mutex_lock(&gen_lock);
uint32_t g = gen_next(); uint64_t g = gen_next();
for (uint64_t a = lo; a < hi; a += 1ull << GEN_PAGE_SHIFT) stamp_page_as(a, g, back); for (uint64_t a = lo; a < hi; a += 1ull << GEN_PAGE_SHIFT) stamp_page_as(a, g, back);
plat_mutex_unlock(&gen_lock); plat_mutex_unlock(&gen_lock);
} }
@@ -2340,7 +2347,7 @@ void vk_mem_dirty_submit(void) {
if (r->kind == POOL_IMPORT && r->bufbound) dlist[n++] = r; if (r->kind == POOL_IMPORT && r->bufbound) dlist[n++] = r;
} }
plat_mutex_lock(&gen_lock); plat_mutex_lock(&gen_lock);
uint32_t g = gen_next(), since = dirty_gen; uint64_t g = gen_next(), since = dirty_gen;
dirty_gen = g; dirty_gen = g;
for (int i = 0; i < n;) { // one query per run of pools (sorted by address) for (int i = 0; i < n;) { // one query per run of pools (sorted by address)
uint64_t lo = page_lo(dlist[i]), hi = page_hi(dlist[i]); uint64_t lo = page_lo(dlist[i]), hi = page_hi(dlist[i]);
@@ -2361,7 +2368,7 @@ void vk_mem_dirty_submit(void) {
int k = dc_class(r); int k = dc_class(r);
ln[k]++, lb[k] += r->size; ln[k]++, lb[k] += r->size;
int fresh = r->synced_gen == DIRTY_UNSYNCED, lz = !fresh && r->bound_sub == prev; int fresh = r->synced_gen == DIRTY_UNSYNCED, lz = !fresh && r->bound_sub == prev;
uint32_t ls = r->synced_gen; uint64_t ls = r->synced_gen;
if (fresh || r->bound_sub == prev) r->synced_gen = g; if (fresh || r->bound_sub == prev) r->synced_gen = g;
if (r->untracked) { untracked += r->size; continue; } if (r->untracked) { untracked += r->size; continue; }
if (fresh) continue; // counted as first use if (fresh) continue; // counted as first use
@@ -2372,10 +2379,10 @@ void vk_mem_dirty_submit(void) {
for (uint64_t a = lo; a < hi;) { for (uint64_t a = lo; a < hi;) {
uint64_t ci = a >> GEN_CHUNK_SHIFT, ce = (ci + 1) << GEN_CHUNK_SHIFT; uint64_t ci = a >> GEN_CHUNK_SHIFT, ce = (ci + 1) << GEN_CHUNK_SHIFT;
if (ce > hi) ce = hi; if (ce > hi) ce = hi;
const uint32_t *ch = ci < GEN_CHUNKS ? gen_chunk[ci] : NULL; const uint64_t *ch = ci < GEN_CHUNKS ? gen_chunk[ci] : NULL;
if (ch) if (ch)
for (uint64_t x = a; x < ce; x += 1ull << GEN_PAGE_SHIFT) { for (uint64_t x = a; x < ce; x += 1ull << GEN_PAGE_SHIFT) {
uint32_t v = ch[(x >> GEN_PAGE_SHIFT) & (GEN_PER_CHUNK - 1)]; uint64_t v = ch[(x >> GEN_PAGE_SHIFT) & (GEN_PER_CHUNK - 1)];
ne += v > since; ne += v > since;
nl += lz && v > ls; nl += lz && v > ls;
} }
@@ -2955,7 +2962,7 @@ static void mirror_create(PoolRec *r) {
mrep.sync_queries += snap_regions(r0, r1, mepoch); mrep.sync_queries += snap_regions(r0, r1, mepoch);
int untracked = 0; int untracked = 0;
for (uint64_t k = r0; k <= r1; k++) untracked |= snap_untracked[k]; for (uint64_t k = r0; k <= r1; k++) untracked |= snap_untracked[k];
r->mgen = gen_now; // the whole-range copy below holds every write observed so far __atomic_store_n(&r->mgen, gen_now, __ATOMIC_RELEASE); // the whole-range copy below holds every write so far
plat_mutex_unlock(&gen_lock); plat_mutex_unlock(&gen_lock);
if (untracked) return refuse(r, MR_UNTRACKED, "its range is not write-watched"); if (untracked) return refuse(r, MR_UNTRACKED, "its range is not write-watched");
VkBufferCreateInfo bi = {.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, .size = n, .usage = pool_usage(c), VkBufferCreateInfo bi = {.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, .size = n, .usage = pool_usage(c),
@@ -3106,7 +3113,7 @@ static void mirror_sync(PoolRec *r) {
plat_mutex_lock(&gen_lock); plat_mutex_lock(&gen_lock);
mrep.sync_queries += snap_regions(r0, r1, mepoch); mrep.sync_queries += snap_regions(r0, r1, mepoch);
} }
uint32_t since = r->mgen; uint64_t since = r->mgen;
int untracked = 0; int untracked = 0;
for (uint64_t k = r0; k <= r1; k++) { for (uint64_t k = r0; k <= r1; k++) {
untracked |= snap_untracked[k]; untracked |= snap_untracked[k];
@@ -3114,7 +3121,7 @@ static void mirror_sync(PoolRec *r) {
uint64_t a = k << SNAP_SHIFT, b = a + (1ull << SNAP_SHIFT); uint64_t a = k << SNAP_SHIFT, b = a + (1ull << SNAP_SHIFT);
if (a < lo) a = lo; if (a < lo) a = lo;
if (b > hi) b = hi; if (b > hi) b = hi;
const uint32_t *ch = cgen_chunk[a >> GEN_CHUNK_SHIFT]; // a 2 MiB region lies in one chunk const uint64_t *ch = cgen_chunk[a >> GEN_CHUNK_SHIFT]; // a 2 MiB region lies in one chunk
if (!ch) continue; if (!ch) continue;
for (uint64_t x = a; x < b; x += 1ull << GEN_PAGE_SHIFT) for (uint64_t x = a; x < b; x += 1ull << GEN_PAGE_SHIFT)
if (ch[(x >> GEN_PAGE_SHIFT) & (GEN_PER_CHUNK - 1)] > since) { if (ch[(x >> GEN_PAGE_SHIFT) & (GEN_PER_CHUNK - 1)] > since) {
@@ -3122,7 +3129,7 @@ static void mirror_sync(PoolRec *r) {
r->st_pages += 1ull << GEN_PAGE_SHIFT, r->hot_bytes += 1ull << GEN_PAGE_SHIFT, r->hot_up += 1ull << GEN_PAGE_SHIFT; r->st_pages += 1ull << GEN_PAGE_SHIFT, r->hot_bytes += 1ull << GEN_PAGE_SHIFT, r->hot_up += 1ull << GEN_PAGE_SHIFT;
} }
} }
r->mgen = gen_now; __atomic_store_n(&r->mgen, gen_now, __ATOMIC_RELEASE); // lazy_clean reads it without gen_lock
if (excl) plat_mutex_unlock(&gen_lock); if (excl) plat_mutex_unlock(&gen_lock);
else plat_mutex_unlock_shared(&gen_lock); else plat_mutex_unlock_shared(&gen_lock);
if (__atomic_load_n(&r->fl_n, __ATOMIC_ACQUIRE)) { if (__atomic_load_n(&r->fl_n, __ATOMIC_ACQUIRE)) {
@@ -3181,7 +3188,7 @@ void vk_mem_shadow_table_read(void) {
static int shadow_pool_clean(const PoolRec *r, uint32_t epoch, int cpu) { static int shadow_pool_clean(const PoolRec *r, uint32_t epoch, int cpu) {
uint64_t r0 = r->import_lo >> SNAP_SHIFT, r1 = (r->import_hi - 1) >> SNAP_SHIFT; uint64_t r0 = r->import_lo >> SNAP_SHIFT, r1 = (r->import_hi - 1) >> SNAP_SHIFT;
if (!epoch || r1 >= SNAP_REGIONS) return 0; if (!epoch || r1 >= SNAP_REGIONS) return 0;
const uint32_t *mx = cpu ? creg_max : reg_max; const uint64_t *mx = cpu ? creg_max : reg_max;
for (uint64_t k = r0; k <= r1; k++) for (uint64_t k = r0; k <= r1; k++)
if (snap_at[k] != epoch || snap_untracked[k] || mx[k] > r->sgen) return 0; if (snap_at[k] != epoch || snap_untracked[k] || mx[k] > r->sgen) return 0;
return 1; return 1;
@@ -3543,7 +3550,7 @@ static void swb_run(uint64_t a, uint64_t b) {
if (b <= a || !shadow_write_back(a, b - a) || nvn_gpu_only(a)) return; if (b <= a || !shadow_write_back(a, b - a) || nvn_gpu_only(a)) return;
int back = wb_no_reup(); int back = wb_no_reup();
plat_mutex_lock(&gen_lock); plat_mutex_lock(&gen_lock);
uint32_t g = gen_next(); uint64_t g = gen_next();
for (uint64_t x = a & ~0xfffull; x < b; x += 1ull << GEN_PAGE_SHIFT) stamp_page_as(x, g, back); for (uint64_t x = a & ~0xfffull; x < b; x += 1ull << GEN_PAGE_SHIFT) stamp_page_as(x, g, back);
plat_mutex_unlock(&gen_lock); plat_mutex_unlock(&gen_lock);
} }
@@ -3755,9 +3762,9 @@ static void lazy_fetch(uint64_t h0, uint64_t h1, uint64_t off, uint64_t size) {
// No lock: the pages are inaccessible, so nothing stamps them meanwhile; the mirror's mgen only grows. // No lock: the pages are inaccessible, so nothing stamps them meanwhile; the mirror's mgen only grows.
static int lazy_clean(uint64_t lo, uint64_t hi, void *ctx) { static int lazy_clean(uint64_t lo, uint64_t hi, void *ctx) {
const PoolRec *r = ctx; const PoolRec *r = ctx;
uint32_t mgen = __atomic_load_n(&r->mgen, __ATOMIC_ACQUIRE); uint64_t mgen = __atomic_load_n(&r->mgen, __ATOMIC_ACQUIRE);
for (uint64_t a = lo; a < hi; a += 1ull << GEN_PAGE_SHIFT) { for (uint64_t a = lo; a < hi; a += 1ull << GEN_PAGE_SHIFT) {
const uint32_t *ch = __atomic_load_n(&cgen_chunk[a >> GEN_CHUNK_SHIFT], __ATOMIC_ACQUIRE); const uint64_t *ch = __atomic_load_n(&cgen_chunk[a >> GEN_CHUNK_SHIFT], __ATOMIC_ACQUIRE);
if (ch && __atomic_load_n(&ch[(a >> GEN_PAGE_SHIFT) & (GEN_PER_CHUNK - 1)], __ATOMIC_RELAXED) > mgen) return 0; if (ch && __atomic_load_n(&ch[(a >> GEN_PAGE_SHIFT) & (GEN_PER_CHUNK - 1)], __ATOMIC_RELAXED) > mgen) return 0;
} }
uint64_t p[1]; uint64_t p[1];
@@ -4336,14 +4343,14 @@ void vk_mem_mirror_prologue(VkCommandBuffer cb) {
} }
// ---- verify mode // ---- verify mode
typedef struct { PoolRec *r; uint64_t off, len, dst; uint32_t mgen; } VSample; typedef struct { PoolRec *r; uint64_t off, len, dst, mgen; } VSample;
typedef struct { uint32_t sub; int n; VkBuffer buf; VmaAllocation alloc; const uint8_t *map; VSample s[VERIFY_SAMPLES]; } VJob; typedef struct { uint32_t sub; int n; VkBuffer buf; VmaAllocation alloc; const uint8_t *map; VSample s[VERIFY_SAMPLES]; } VJob;
static int ver_i; static int ver_i;
static uint64_t ver_off; static uint64_t ver_off;
// The CPU wrote the page since the mirror's sync that mgen records: stamped later, or its write-watch bit is set. // The CPU wrote the page since the mirror's sync that mgen records: stamped later, or its write-watch bit is set.
static int page_dirty(uint64_t a, uint32_t mgen) { static int page_dirty(uint64_t a, uint64_t mgen) {
uint32_t *s = gen_slot(a, 0); uint64_t *s = gen_slot(a, 0);
if (s && __atomic_load_n(s, __ATOMIC_RELAXED) > mgen) return 1; if (s && __atomic_load_n(s, __ATOMIC_RELAXED) > mgen) return 1;
uint64_t p[1]; uint64_t p[1];
return mem_written_peek(a, 1ull << GEN_PAGE_SHIFT, p, 1) != 0; // UINT64_MAX (not watched) counts as written return mem_written_peek(a, 1ull << GEN_PAGE_SHIFT, p, 1) != 0; // UINT64_MAX (not watched) counts as written
@@ -4382,13 +4389,15 @@ static void verify_cb(void *arg) { // completion thread
rt_log("mem: verify mismatch: pool %#llx+%#llx (%s) offset %#llx: bytes %#x..%#x of the page differ, " rt_log("mem: verify mismatch: pool %#llx+%#llx (%s) offset %#llx: bytes %#x..%#x of the page differ, "
"mirror %02x guest %02x at the first (submission %u)", (unsigned long long)r->lo, "mirror %02x guest %02x at the first (submission %u)", (unsigned long long)r->lo,
(unsigned long long)r->size, f, (unsigned long long)(a - r->lo), k, e, m[k], g[k], j->sub); (unsigned long long)r->size, f, (unsigned long long)(a - r->lo), k, e, m[k], g[k], j->sub);
const uint32_t *gs = gen_slot(a & ~((1ull << GEN_PAGE_SHIFT) - 1), 0); const uint64_t *gs = gen_slot(a & ~((1ull << GEN_PAGE_SHIFT) - 1), 0);
char mh[100], gh[100]; char mh[100], gh[100];
for (int b = 0; b < 16; b++) snprintf(mh + 3 * b, 4, "%02x ", m[b]), snprintf(gh + 3 * b, 4, "%02x ", g[b]); for (int b = 0; b < 16; b++) snprintf(mh + 3 * b, 4, "%02x ", m[b]), snprintf(gh + 3 * b, 4, "%02x ", g[b]);
rt_log("mem: verify mismatch detail: written binds %llu, last %u; used %u; written back %u; gpa record %u; " rt_log("mem: verify mismatch detail: written binds %llu, last %u; used %u; written back %u; gpa record %u; "
"page cpu stamp %u (synced to %u, now %u); mirror %s| guest %s", "page cpu stamp %llu (synced to %llu, now %llu); mirror %s| guest %s",
(unsigned long long)r->vd_wrote_n, r->vd_wrote_sub, r->vd_use_sub, wb, r->gpa_rec, gs ? *gs : 0, (unsigned long long)r->vd_wrote_n, r->vd_wrote_sub, r->vd_use_sub, wb, r->gpa_rec,
s->mgen, gen_now, mh, gh); (unsigned long long)(gs ? __atomic_load_n(gs, __ATOMIC_RELAXED) : 0),
(unsigned long long)s->mgen, (unsigned long long)__atomic_load_n(&gen_now, __ATOMIC_RELAXED),
mh, gh);
} }
} }
} }
@@ -6266,6 +6275,6 @@ void vk_mem_sizes(PerfSizes *s) {
perf_size(s, "free staging arena chunks", arena_n, ARENA_KEEP, arena_n * ARENA_CHUNK); perf_size(s, "free staging arena chunks", arena_n, ARENA_KEEP, arena_n * ARENA_CHUNK);
perf_size(s, "write-back snapshot pool", snap_n, SNAP_POOL_N, snap_b); perf_size(s, "write-back snapshot pool", snap_n, SNAP_POOL_N, snap_b);
perf_size(s, "shadow base copies", sb, 0, sb * SB_PAGE); perf_size(s, "shadow base copies", sb, 0, sb * SB_PAGE);
// the page generations handed out so far (gen_now, 32 bits, no wrap handling): how fast a session uses them up // the page generations handed out so far (gen_now, 64 bits: they do not run out)
perf_size(s, "page generations handed out", (uint64_t)__atomic_load_n(&gen_now, __ATOMIC_RELAXED), 0, PERF_SIZE_NONE); perf_size(s, "page generations handed out", (uint64_t)__atomic_load_n(&gen_now, __ATOMIC_RELAXED), 0, PERF_SIZE_NONE);
} }

View File

@@ -53,7 +53,7 @@ typedef struct Tex {
uint64_t blk_mem, blk_size; // where alloc lies (own_memory): the device memory handle and size of its block uint64_t blk_mem, blk_size; // where alloc lies (own_memory): the device memory handle and size of its block
uint8_t blk_type, blk_own; // that memory's type; blk_own: memory of its own, not a part of one of VMA's blocks uint8_t blk_type, blk_own; // that memory's type; blk_own: memory of its own, not a part of one of VMA's blocks
VmaAllocation bound_to; // the pool memory it is bound into VmaAllocation bound_to; // the pool memory it is bound into
uint32_t wgen; // page generation of the last upload check uint64_t wgen; // page generation of the last upload check
uint32_t checked; // submit_epoch of the last check from a draw binding uint32_t checked; // submit_epoch of the last check from a draw binding
uint32_t use_round; // look_round when a lookup last reached it (touch) uint32_t use_round; // look_round when a lookup last reached it (touch)
uint64_t use_frame; // gfx_counters.frames then uint64_t use_frame; // gfx_counters.frames then
@@ -72,7 +72,7 @@ typedef struct Tex {
int last_ch; // last vk_mem_changed result of make_fresh: 1 written, 0 clean, -1 not tracked int last_ch; // last vk_mem_changed result of make_fresh: 1 written, 0 clean, -1 not tracked
struct Tex *next_at; // other entries at the same storage address struct Tex *next_at; // other entries at the same storage address
uint8_t gpu_dirty; // mirrored, and the GPU wrote the image after the last write-back to the storage uint8_t gpu_dirty; // mirrored, and the GPU wrote the image after the last write-back to the storage
uint32_t gpu_gen; // page generation when the GPU write that set gpu_dirty was recorded uint64_t gpu_gen; // page generation when the GPU write that set gpu_dirty was recorded
uint64_t gpu_seq; // order of that GPU write among all of them (gpu_writes) uint64_t gpu_seq; // order of that GPU write among all of them (gpu_writes)
uint64_t gpu_fence; // fence of the submission that made that write (0: not recorded in one) uint64_t gpu_fence; // fence of the submission that made that write (0: not recorded in one)
uint64_t run_frame; // mirrored: the frame of the newest GPU write uint64_t run_frame; // mirrored: the frame of the newest GPU write
@@ -1661,7 +1661,7 @@ static void rel_unlink(Tex *e) {
// to 8,192 released entries, they stay listed, and since stays as it was at their release, so each one whose regions // to 8,192 released entries, they stay listed, and since stays as it was at their release, so each one whose regions
// took a write later missed the region check at every submission from then on: 1,000 to 6,300 entries a check in // took a write later missed the region check at every submission from then on: 1,000 to 6,300 entries a check in
// Kakariko and Castle Town, a third to a half of the list. // Kakariko and Castle Town, a third to a half of the list.
typedef struct { Tex *e; uint64_t addr, size; uint32_t since; uint8_t ok, rel; } PEnt; // ok: uploaded as of since typedef struct { Tex *e; uint64_t addr, size, since; uint8_t ok, rel; } PEnt; // ok: uploaded as of since
static PEnt *pent; static PEnt *pent;
static int npent, cpent, *pq, cpq; static int npent, cpent, *pq, cpq;
static uint64_t n_pool_checks, n_pool_skips, n_pool_rel; static uint64_t n_pool_checks, n_pool_skips, n_pool_rel;
@@ -2069,7 +2069,7 @@ static int commit_drops_held(void) {
// shutdown. // shutdown.
static int wb_wanted(const Tex *e); static int wb_wanted(const Tex *e);
static void wb_cfg_init(void); static void wb_cfg_init(void);
static int storage_rewritten(const Tex *e, uint32_t since, uint32_t *now); static int storage_rewritten(const Tex *e, uint64_t since, uint64_t *now);
static void write_back_n(Tex **v, int n); static void write_back_n(Tex **v, int n);
static uint64_t n_drop_wb; // entries written back before their drop since the last perf line static uint64_t n_drop_wb; // entries written back before their drop since the last perf line
@@ -2077,7 +2077,7 @@ static void drop_tex(Tex *e, int why) {
flight_ev(e, FLIGHT_TEX_DROP, (uint32_t)why, 0); flight_ev(e, FLIGHT_TEX_DROP, (uint32_t)why, 0);
if (why == DROP_COMMIT || why == DROP_IDLE_COMMIT) commit_drop_note(e); if (why == DROP_COMMIT || why == DROP_IDLE_COMMIT) commit_drop_note(e);
if (e->gpu_dirty && why != DROP_OVERWRITTEN && why != DROP_SHUTDOWN) { if (e->gpu_dirty && why != DROP_OVERWRITTEN && why != DROP_SHUTDOWN) {
uint32_t now; uint64_t now;
wb_cfg_init(); wb_cfg_init();
if (storage_rewritten(e, e->gpu_gen, &now) == 1) e->gpu_dirty = 0; if (storage_rewritten(e, e->gpu_gen, &now) == 1) e->gpu_dirty = 0;
else if (wb_wanted(e)) write_back_n(&e, 1), n_drop_wb++; else if (wb_wanted(e)) write_back_n(&e, 1), n_drop_wb++;
@@ -2459,7 +2459,7 @@ static void sweep_hard(void) { // tex_lock held
wb_cfg_init(); wb_cfg_init();
for (int i = 0; i < k; i++) { for (int i = 0; i < k; i++) {
Tex *e = walk[out[i]]; Tex *e = walk[out[i]];
uint32_t t; uint64_t t;
if (!e->gpu_dirty) continue; if (!e->gpu_dirty) continue;
if (storage_rewritten(e, e->gpu_gen, &t) == 1) e->gpu_dirty = 0; if (storage_rewritten(e, e->gpu_gen, &t) == 1) e->gpu_dirty = 0;
else if (wb_wanted(e)) wbv[m++] = e; else if (wb_wanted(e)) wbv[m++] = e;
@@ -4089,7 +4089,7 @@ static volatile int32_t n_kept;
// slots used in turn, and the CPU reads them a frame later. One slot's rect target shares its page with data the CPU // slots used in turn, and the CPU reads them a frame later. One slot's rect target shares its page with data the CPU
// writes every frame: without the copy its renders were never written back, the game read a zero rect every other // writes every frame: without the copy its renders were never written back, the game read a zero rect every other
// frame and the target's name label blinked. NX2NAVI_WB_EXACT=0 restores the page-only decision. // frame and the target's name label blinked. NX2NAVI_WB_EXACT=0 restores the page-only decision.
static int storage_rewritten(const Tex *e, uint32_t since, uint32_t *now) { static int storage_rewritten(const Tex *e, uint64_t since, uint64_t *now) {
uint64_t lo = (e->pub.addr + 0xfff) & ~0xfffull, hi = (e->pub.addr + e->t.L.size) & ~0xfffull; uint64_t lo = (e->pub.addr + 0xfff) & ~0xfffull, hi = (e->pub.addr + e->t.L.size) & ~0xfffull;
if (hi <= lo) lo = e->pub.addr, hi = e->pub.addr + e->t.L.size; if (hi <= lo) lo = e->pub.addr, hi = e->pub.addr + e->t.L.size;
int ch = vk_mem_changed(lo, hi - lo, since, now); int ch = vk_mem_changed(lo, hi - lo, since, now);
@@ -4131,7 +4131,7 @@ static void make_fresh(Tex *e) {
upload(e); upload(e);
return; return;
} }
uint32_t now; uint64_t now;
int ch = vk_mem_changed(e->pub.addr, size, e->wgen, &now); int ch = vk_mem_changed(e->pub.addr, size, e->wgen, &now);
e->wgen = now; e->wgen = now;
e->last_ch = ch; e->last_ch = ch;
@@ -4144,7 +4144,7 @@ static void make_fresh(Tex *e) {
static void make_fresh_snap(Tex *e) { static void make_fresh_snap(Tex *e) {
if (!from_storage(e) || e->pub.gpu_owned || (e->sfill && e->released)) return; if (!from_storage(e) || e->pub.gpu_owned || (e->sfill && e->released)) return;
if (!e->uploaded && up_wait(e)) return; if (!e->uploaded && up_wait(e)) return;
uint32_t now; uint64_t now;
n_checks++; n_checks++;
int ch = vk_mem_changed_snap(e->pub.addr, e->t.L.size, e->uploaded ? e->wgen : 0, submit_epoch, &now); int ch = vk_mem_changed_snap(e->pub.addr, e->t.L.size, e->uploaded ? e->wgen : 0, submit_epoch, &now);
e->wgen = now; e->wgen = now;
@@ -4190,7 +4190,7 @@ static int alias_compatible(const Tex *e, const Tex *src, const char **why) {
static int alias_has_gpu_content(const Tex *src) { static int alias_has_gpu_content(const Tex *src) {
if (!src->pub.gpu_owned) return 0; if (!src->pub.gpu_owned) return 0;
if (!src->pub.mirrored) return 1; if (!src->pub.mirrored) return 1;
uint32_t now; uint64_t now;
return src->gpu_dirty && !storage_rewritten(src, src->gpu_gen, &now); return src->gpu_dirty && !storage_rewritten(src, src->gpu_gen, &now);
} }
@@ -4360,7 +4360,7 @@ static void start_at(Tex *e, const TexKey *k, int taken_back) {
// split fill: e is to start with its storage's bytes; msrc: the released CPU-written entry whose image holds them // split fill: e is to start with its storage's bytes; msrc: the released CPU-written entry whose image holds them
// (destroyed after the copy), msrc_gpu: it holds what the GPU wrote instead, msrc_gen: the storage's generation // (destroyed after the copy), msrc_gpu: it holds what the GPU wrote instead, msrc_gen: the storage's generation
int fill = !taken_back && tex_alias && split_wants(&e->t, &e->f), msrc_gpu = 0; int fill = !taken_back && tex_alias && split_wants(&e->t, &e->f), msrc_gpu = 0;
uint32_t msrc_gen = 0; uint64_t msrc_gen = 0;
Tex *msrc = NULL; Tex *msrc = NULL;
const char *why = "no live texture with GPU content"; // for the log of an alias left empty const char *why = "no live texture with GPU content"; // for the log of an alias left empty
const Tex *other = NULL; const Tex *other = NULL;
@@ -4381,7 +4381,7 @@ static void start_at(Tex *e, const TexKey *k, int taken_back) {
} }
gone++; gone++;
const char *no; const char *no;
uint32_t gen = 0; uint64_t gen = 0;
int gpu = alias_has_gpu_content(p); int gpu = alias_has_gpu_content(p);
if (fill && !msrc && (gpu || (p->uploaded && !p->pub.gpu_owned && if (fill && !msrc && (gpu || (p->uploaded && !p->pub.gpu_owned &&
!vk_mem_changed(p->pub.addr, p->t.L.size, p->wgen, &gen))) && !vk_mem_changed(p->pub.addr, p->t.L.size, p->wgen, &gen))) &&
@@ -4588,7 +4588,7 @@ static Tex *entry_of(const tex_t *t) {
void vk_texture_debug(const VkTexImage *img, uint32_t *uploads, int *last_change, int *pages_written) { void vk_texture_debug(const VkTexImage *img, uint32_t *uploads, int *last_change, int *pages_written) {
lock(); lock();
const Tex *e = (const Tex *)img; // pub is the first member const Tex *e = (const Tex *)img; // pub is the first member
uint32_t now; uint64_t now;
*uploads = e->n_up; *uploads = e->n_up;
*last_change = e->last_ch; *last_change = e->last_ch;
*pages_written = e->pub.mirrored && e->uploaded ? vk_mem_changed(e->pub.addr, e->t.L.size, e->wgen, &now) : 0; *pages_written = e->pub.mirrored && e->uploaded ? vk_mem_changed(e->pub.addr, e->t.L.size, e->wgen, &now) : 0;
@@ -5705,7 +5705,7 @@ void vk_texture_write_back(uint64_t addr, uint64_t size) {
if (!hit) rt_fatal("vk: out of memory for a texture write-back"); if (!hit) rt_fatal("vk: out of memory for a texture write-back");
int n = owned_dirty(addr, size, hit), k = 0; // owned[] cannot change while tex_lock is held int n = owned_dirty(addr, size, hit), k = 0; // owned[] cannot change while tex_lock is held
for (int i = 0; i < n; i++) { for (int i = 0; i < n; i++) {
uint32_t now; uint64_t now;
if (storage_rewritten(hit[i], hit[i]->gpu_gen, &now) == 1) hit[i]->gpu_dirty = 0; if (storage_rewritten(hit[i], hit[i]->gpu_gen, &now) == 1) hit[i]->gpu_dirty = 0;
else hit[k++] = hit[i]; else hit[k++] = hit[i];
} }
@@ -5778,7 +5778,7 @@ void vk_texture_write_back_small(uint64_t fence) {
if (later) __atomic_store_n(&small_pending, 1, __ATOMIC_RELEASE); if (later) __atomic_store_n(&small_pending, 1, __ATOMIC_RELEASE);
int k = 0; int k = 0;
for (int i = 0; i < n; i++) { for (int i = 0; i < n; i++) {
uint32_t now; uint64_t now;
if (storage_rewritten(hit[i], hit[i]->gpu_gen, &now) == 1) hit[i]->gpu_dirty = 0; if (storage_rewritten(hit[i], hit[i]->gpu_gen, &now) == 1) hit[i]->gpu_dirty = 0;
else hit[k++] = hit[i]; else hit[k++] = hit[i];
} }

71
tests/gen_width_check.py Normal file
View File

@@ -0,0 +1,71 @@
"""No 64-to-32-bit narrowing in the files that hold page write generations: runtime/vk_memory.c, runtime/vk_texture.c
and runtime/tex_probe.c, compiled with the build's flags plus -Wshorten-64-to-32, -Wformat and
-Wincompatible-pointer-types as errors (clang -fsyntax-only). The generations are 64 bits (vk_memory.c "page write
generations"); one local, field or format left at 32 bits would bring the wrap of 2^32 back without a word, and the
normal build sees a narrowing only as a silent conversion. The baseline is zero such diagnostics in the whole of the
three files: a narrowing elsewhere in them takes an explicit cast, which also says it is meant.
Needs clang, the Vulkan headers, VMA and the shader headers of tools/build_shaders.py (runtime/*_spv.h). No GPU, no
game files, any Python.
Usage: gen_width_check.py [--cc <clang>]
"""
import argparse
import os
import re
import subprocess
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / 'tools'))
import build_game # noqa: E402 (RT_FLAGS, the compiler and the include folders of the build)
FILES = ['vk_memory.c', 'vk_texture.c', 'tex_probe.c']
STRICT = ['-Werror=shorten-64-to-32', '-Werror=format', '-Werror=incompatible-pointer-types']
# the diagnostics this check is about, by the flag clang names at the end of the line
WATCHED = re.compile(r'^(?P<where>[^\n]*?:\d+:\d+): (?:error|warning): (?P<what>[^\n]*?)'
r'\[(?:-Werror,)?(?P<flag>-Wshorten-64-to-32|-Wformat[\w-]*|-Wincompatible[\w-]*pointer-types)\]',
re.M)
def main():
ap = argparse.ArgumentParser()
ap.add_argument('--cc', help='the C compiler (default: the build\'s clang)')
args = ap.parse_args()
if not (ROOT / 'runtime' / 'wb_merge_spv.h').is_file():
print('gen_width_check: FAIL: runtime/wb_merge_spv.h is missing: run tools/build_shaders.py first')
return 1
cc = args.cc or os.environ.get('CC') or str(build_game.CLANG)
build_game.set_cpu('v3') # adds -DRT_CPU_LEVEL=3, as the build does
flags = [f for f in build_game.RT_FLAGS if f != '-g']
inc = ['-I', str(ROOT / 'runtime'), '-I', str(build_game.vma_include())]
vk = build_game.vulkan_include(cc)
if vk:
inc = ['-I', str(vk)] + inc
env = dict(os.environ)
if os.name == 'nt':
env['PATH'] = str(Path(cc).parent) + os.pathsep + env.get('PATH', '')
bad = 0
for name in FILES:
src = ROOT / 'runtime' / name
p = subprocess.run([cc, '-fsyntax-only', '-std=gnu11'] + flags + STRICT + inc + [str(src)], capture_output=True,
text=True, errors='replace', env=env)
hits = [m for m in WATCHED.finditer(p.stderr)]
for m in hits:
print(f'gen_width_check: FAIL: {m.group("where")}: {m.group("what").strip()} [{m.group("flag")}]')
if p.returncode and not hits:
print(f'gen_width_check: FAIL: runtime/{name} does not compile:\n{p.stderr[-3000:]}')
bad += 1
bad += len(hits)
if not hits and not p.returncode:
print(f'gen_width_check: runtime/{name}: no narrowing, format or pointer-type diagnostic')
if bad:
print(f'gen_width_check: {bad} problems')
return 1
print('gen_width_check: all checks passed')
return 0
if __name__ == '__main__':
sys.exit(main())

View File

@@ -1,11 +1,14 @@
// runtime/vk_memory.c's page write generations (the section from its "page write generations" banner to its "dirty // runtime/vk_memory.c's page write generations across a boundary: the section from its "page write generations" banner
// census" banner, extracted by tests/gen_wrap_check.py into gen_section.inc): the lines gen_next logs as the 32-bit // to its "dirty census" banner and the readers of its tables outside it (cpu_wrote_since, shadow_pool_clean,
// generations near their wrap, and what the comparisons answer across it. Prints each failed check and exits 1 on any. // lazy_clean, page_dirty), extracted by tests/gen_wrap_check.py into gen_section.inc, start just below a seed's
// Stubs: no write-watch (mem_written_pages reports no page), nothing frozen, no pools. // boundary and run across it. Every compare the texture checks, the snapshot path, the VRAM mirrors' lazy fetch, the
// // shadow storage and the verify mode make must answer as it does below the boundary. Seeds: 2^32 - 8 (where 32-bit
// The comparisons across the wrap are wrong today (AUDIT F-8-01: the generations have no wrap handling; its fix waits // generations wrapped), 2^40 - 8 (the low 32 bits pass through 0 again, so a truncation to 32 bits anywhere fails
// for approval). The check expects the four wrong answers and says so: the change that handles the wrap turns them // there too) and 2^63 - 8. gen_next must log one line when the count reaches 2^32, and no other.
// into failures here, and flips the expectations with it. // Stubs: no-op locks; a write watch that reports the pages a check marks as written by the CPU; nothing frozen; no
// pools in the index.
// Usage: gen_wrap_check <seed>. Prints each failed check and "gen_wrap_check: seed ...: N of M checks failed"; exit 1
// on any.
#include <stdarg.h> #include <stdarg.h>
#include <stdint.h> #include <stdint.h>
#include <stdio.h> #include <stdio.h>
@@ -30,11 +33,36 @@ static uint64_t kern_now_ns(void) { return fake_now; }
static void grow(void **arr, int *cap, size_t elem) { static void grow(void **arr, int *cap, size_t elem) {
int n = *cap ? *cap * 2 : 64; int n = *cap ? *cap * 2 : 64;
*arr = realloc(*arr, (size_t)n * elem); *arr = realloc(*arr, (size_t)n * elem);
if (!*arr) exit(70);
*cap = n; *cap = n;
} }
#define SB_PAGE 4096ull
// The write watch: pages the CPU wrote since the last query over them (mem_written_pages resets them,
// mem_written_peek does not).
static uint64_t ww_pend[64];
static int nww_pend;
static void cpu_write(uint64_t page) { ww_pend[nww_pend++] = page & ~0xfffull; }
static uint64_t ww_take(uint64_t addr, uint64_t size, uint64_t *pages, uint64_t max, int reset) {
uint64_t n = 0;
for (int i = 0; i < nww_pend && n < max;) {
if (ww_pend[i] >= addr && ww_pend[i] < addr + size) {
if (pages) pages[n] = ww_pend[i];
n++;
if (reset) {
ww_pend[i] = ww_pend[--nww_pend];
continue;
}
}
i++;
}
return n;
}
uint64_t mem_written_pages(uint64_t addr, uint64_t size, uint64_t *pages, uint64_t max) { uint64_t mem_written_pages(uint64_t addr, uint64_t size, uint64_t *pages, uint64_t max) {
(void)addr, (void)size, (void)pages, (void)max; return ww_take(addr, size, pages, max, 1);
return 0; }
uint64_t mem_written_peek(uint64_t addr, uint64_t size, uint64_t *pages, uint64_t max) {
return ww_take(addr, size, pages, max, 0);
} }
int mem_frozen(uint64_t a) { (void)a; return 0; } int mem_frozen(uint64_t a) { (void)a; return 0; }
int mem_watched(uint64_t addr, uint64_t size) { (void)addr, (void)size; return 1; } int mem_watched(uint64_t addr, uint64_t size) { (void)addr, (void)size; return 1; }
@@ -46,9 +74,8 @@ static pidx pool_index;
static PlatMutex pool_lock; static PlatMutex pool_lock;
static const pidx_ent *pidx_find(const pidx *x, uintptr_t a, uint64_t l) { (void)x, (void)a, (void)l; return NULL; } static const pidx_ent *pidx_find(const pidx *x, uintptr_t a, uint64_t l) { (void)x, (void)a, (void)l; return NULL; }
enum { POOL_IMPORT = 1 }; enum { POOL_IMPORT = 1 };
typedef struct { int kind; void *buf; uint64_t import_lo, import_hi; } PoolRec;
// The log: the generation lines are kept, the others dropped. // The log: the page generation lines are kept, the others dropped.
static char gen_lines[16][512]; static char gen_lines[16][512];
static int n_gen_lines; static int n_gen_lines;
static void rt_log(const char *fmt, ...) { static void rt_log(const char *fmt, ...) {
@@ -73,73 +100,149 @@ static void rt_fatal(const char *fmt, ...) {
#include "gen_section.inc" #include "gen_section.inc"
static int fails; typedef __typeof__(gen_now) gen_t; // the width the source gives the generations
#define CHECK(c, ...) do { if (!(c)) { fails++; printf("FAIL line %d: ", __LINE__); printf(__VA_ARGS__); printf("\n"); } } while (0)
static int checks, failed;
#define CHECK(cond, ...) \
do { \
checks++; \
if (!(cond)) failed++, printf("FAIL line %d: ", __LINE__), printf(__VA_ARGS__), printf("\n"); \
} while (0)
#define TEX 0x100000000ull // a texture of 4 pages in one 2 MiB region #define TEX 0x100000000ull // a texture of 4 pages in one 2 MiB region
#define OTHER 0x100200000ull // a second one in the next region #define OTHER 0x100200000ull // a second one in the next region
#define SPIN 0x200000000ull // an unrelated range other checks hand out generations for
#define LONE 0x100400000ull // a page written exactly at the boundary
// Hands out generations with vk_guest_written from just below `to` until gen_now reaches `to` + 3. static uint64_t last_now; // every *now handed back, in order: must grow
static void walk_to(uint32_t to) { static void seen(gen_t now, const char *what) {
gen_now = to - 4; CHECK((uint64_t)now > last_now, "%s: generation %llu after %llu: not newer", what, (unsigned long long)now,
for (int i = 0; i < 7; i++) vk_guest_written(0x300000000ull, 0x1000); (unsigned long long)last_now);
last_now = now;
} }
int main(void) { int main(int argc, char **argv) {
// ---- the lines: one at each of 0xC0000000, 0xD0000000, 0xE0000000, 0xF0000000 and at the wrap, no other uint64_t seed = argc > 1 ? strtoull(argv[1], NULL, 0) : (1ull << 32) - 8;
uint64_t edge = seed + 8; // the boundary under test: 2^32 for the first seed
gen_t w_tex, w_oth, now, d;
rt_guest_start_ns = 1000000000ull; rt_guest_start_ns = 1000000000ull;
fake_now = rt_guest_start_ns + 3600ull * 1000000000ull; // an hour in fake_now = rt_guest_start_ns + 3600ull * 1000000000ull; // an hour in
gfx_counters.frames = 216000; // at 60 fps gfx_counters.frames = 216000; // at 60 fps
walk_to(0x10000000u); // below the first mark: nothing gen_now = (gen_t)seed;
walk_to(0xb0000000u); CHECK((uint64_t)gen_now == seed, "the seed %#llx does not fit the generations (%u bits)", (unsigned long long)seed,
CHECK(n_gen_lines == 0, "%d lines before 0xC0000000", n_gen_lines); (unsigned)(8 * sizeof(gen_t)));
static const uint32_t marks[] = {0xc0000000u, 0xd0000000u, 0xe0000000u, 0xf0000000u}; if (failed) { // the generations cannot reach the boundary: nothing below can run
for (int k = 0; k < 4; k++) { printf("gen_wrap_check: seed %#llx: 1 of 1 checks failed\n", (unsigned long long)seed);
int was = n_gen_lines; return 1;
walk_to(marks[k]);
char want[32];
snprintf(want, sizeof want, ": %u of 4294967295", marks[k]);
CHECK(n_gen_lines == was + 1 && strstr(gen_lines[was], want), "the mark at %#x gave %d lines (%s)", marks[k],
n_gen_lines - was, n_gen_lines > was ? gen_lines[was] : "none");
} }
// 0xC0000000 in an hour of 216,000 frames: about 14,913 a frame, and a quarter of the range left: 20 min last_now = (uint64_t)gen_now;
CHECK(n_gen_lines >= 1 && strstr(gen_lines[0], "(14913 a frame)") && strstr(gen_lines[0], "in about 0.3 h"),
"the first line's rate or hours: %s", n_gen_lines ? gen_lines[0] : "none");
int was = n_gen_lines;
walk_to(0); // across the wrap
CHECK(n_gen_lines == was + 1 && strstr(gen_lines[was], "ran out at frame 216000 after 1.0 h"),
"the wrap gave %d lines (%s)", n_gen_lines - was, n_gen_lines > was ? gen_lines[was] : "none");
walk_to(0x20000000u);
CHECK(n_gen_lines == 5, "%d lines in all, not 5", n_gen_lines);
for (int i = 0; i < n_gen_lines && i < 16; i++) printf(" %s\n", gen_lines[i]);
// ---- the comparisons across the wrap: the four answers known to be wrong (expected, see the top) // Before the boundary: both textures loaded (written) and checked.
int known = 0;
gen_now = 999;
vk_guest_written(TEX, 0x4000); vk_guest_written(TEX, 0x4000);
vk_guest_written(OTHER, 0x4000); vk_guest_written(OTHER, 0x4000);
uint32_t w1, w2, now; CHECK(vk_mem_changed(TEX, 0x4000, 0, &w_tex) == 1, "TEX: its load is not seen");
vk_mem_changed(TEX, 0x4000, 0, &w1); seen(w_tex, "TEX first check");
vk_mem_changed(OTHER, 0x4000, 0, &w2); CHECK(vk_mem_changed(OTHER, 0x4000, 0, &w_oth) == 1, "OTHER: its load is not seen");
gen_now = UINT32_MAX - 6; seen(w_oth, "OTHER first check");
CHECK(vk_mem_changed(TEX, 0x4000, w1, &w1) == 0, "an unwritten texture read as written before the wrap"); uint64_t pre = (uint64_t)gen_now; // a generation before the boundary: what a mirror or shadow last synced to
vk_mem_changed(OTHER, 0x4000, w2, &w2);
for (int i = 0; i < 10; i++) vk_mem_changed(0x200000000ull, 0x1000, 0, &now); // other checks use generations
vk_guest_written(TEX, 0x1000); // the CPU writes the texture after the wrap
known += vk_mem_changed(TEX, 0x4000, w1, &now) != 1; // 1. the write is missed
vk_mem_changed(OTHER, 0x4000, w2, &w2);
known += vk_mem_changed(OTHER, 0x4000, w2, &now) == 1; // 2. an unwritten texture reads as written at every check
w2 = now;
known += vk_mem_changed_snap(OTHER, 0x4000, w2, 7, &now) == 1; // 3. the same through the snapshot path
gen_now = UINT32_MAX;
vk_guest_written(TEX + 0x2000, 0x1000); // 4. a write stamped at the wrap gets 0, which reads as never stamped
known += vk_mem_changed(TEX + 0x2000, 0x1000, 0, &now) != 1;
CHECK(known == 4, "%d of the 4 known wrong answers across the wrap remain: if the wrap is handled now, flip these "
"expectations", known);
printf("gen_wrap_check: across the wrap %d of 4 comparisons still answer wrong (known, expected until the wrap "
"is handled)\n", known);
if (fails) return printf("gen_wrap_check: %d checks failed\n", fails), 1; // Other checks hand out generations up to edge - 1; the page written next is stamped exactly at edge.
printf("gen_wrap_check: all checks passed\n"); while ((uint64_t)gen_now + 1 < edge) {
return 0; vk_mem_changed(SPIN, 0x1000, 0, &d);
seen(d, "spin");
}
vk_guest_written(LONE, 0x1000);
gen_t *ls = gen_slot(LONE, 0);
CHECK(ls && (uint64_t)*ls == edge, "the page written at the boundary is stamped %llu, not %llu",
ls ? (unsigned long long)*ls : 0ull, (unsigned long long)edge);
CHECK(vk_mem_changed(LONE, 0x1000, 0, &now) == 1, "the page written at the boundary reads as never written");
seen(now, "LONE");
CHECK(vk_mem_changed(LONE, 0x1000, (gen_t)(edge - 1), &now) == 1, "the boundary's write is older than edge - 1");
seen(now, "LONE again");
for (int i = 0; i < 6; i++) {
vk_mem_changed(SPIN, 0x1000, 0, &d);
seen(d, "spin past");
}
// After the boundary: a CPU write found by the write watch and one by a host path are both seen.
cpu_write(TEX);
CHECK(vk_mem_changed(TEX, 0x4000, w_tex, &now) == 1, "TEX: a CPU write after the boundary is missed");
seen(now, "TEX after a CPU write");
w_tex = now;
vk_guest_written(TEX + 0x1000, 4);
CHECK(vk_mem_changed(TEX, 0x4000, w_tex, &now) == 1, "TEX: a host write after the boundary is missed");
seen(now, "TEX after a host write");
w_tex = now;
CHECK(vk_mem_changed(TEX, 0x4000, w_tex, &now) == 0, "TEX: no write since its check, but it reads as written");
seen(now, "TEX unwritten");
w_tex = now;
// A texture nobody wrote since before the boundary stays unchanged, check after check.
CHECK(vk_mem_changed(OTHER, 0x4000, w_oth, &now) == 0,
"OTHER: unwritten since before the boundary, reads as written");
seen(now, "OTHER 1");
w_oth = now;
CHECK(vk_mem_changed(OTHER, 0x4000, w_oth, &now) == 0, "OTHER: unwritten, reads as written at the next check too");
seen(now, "OTHER 2");
w_oth = now;
// The snapshot path (draw bindings and the texture pool's check) and the region check (reg_max).
uint32_t ep = 7;
CHECK(vk_mem_changed_snap(OTHER, 0x4000, w_oth, ep, &now) == 0, "OTHER via the snapshot: reads as written");
seen(now, "OTHER snap");
CHECK(vk_mem_unchanged_why_locked(OTHER, 0x4000, w_oth, ep) == 0, "OTHER's region check answers %d, not 0",
vk_mem_unchanged_why_locked(OTHER, 0x4000, w_oth, ep));
cpu_write(OTHER + 0x2000);
ep = 8;
CHECK(vk_mem_changed_snap(OTHER, 0x4000, w_oth, ep, &now) == 1, "OTHER via the snapshot: a CPU write is missed");
seen(now, "OTHER snap written");
CHECK(vk_mem_unchanged_why_locked(OTHER, 0x4000, w_oth, ep) == 3,
"OTHER's region check answers %d, not 3 (written)",
vk_mem_unchanged_why_locked(OTHER, 0x4000, w_oth, ep));
CHECK(vk_mem_unchanged_why_locked(OTHER, 0x4000, now, ep) == 0, "OTHER's region check after its check answers %d",
vk_mem_unchanged_why_locked(OTHER, 0x4000, now, ep));
// The CPU-side tables (what mirror_sync, lazy_clean and cpu_wrote_since read) hold the same stamps.
gen_t *gs = gen_slot(TEX + 0x1000, 0), *cs = slot_in(cgen_chunk, TEX + 0x1000, 0);
CHECK(gs && cs && *gs == *cs && (uint64_t)*cs > edge, "TEX page 1: texture stamp %llu, CPU-side stamp %llu",
gs ? (unsigned long long)*gs : 0ull, cs ? (unsigned long long)*cs : 0ull);
CHECK((uint64_t)creg_max[TEX >> REG_SHIFT] > edge && reg_max[TEX >> REG_SHIFT] == creg_max[TEX >> REG_SHIFT],
"TEX's region: newest stamps %llu / %llu", (unsigned long long)reg_max[TEX >> REG_SHIFT],
(unsigned long long)creg_max[TEX >> REG_SHIFT]);
// A VRAM mirror and a shadow copy synced before the boundary, TEX written after it.
PoolRec p = {.kind = POOL_IMPORT, .import_lo = TEX, .import_hi = TEX + 0x4000};
p.mgen = (__typeof__(p.mgen))pre, p.sgen = (__typeof__(p.sgen))pre;
CHECK((uint64_t)p.mgen == pre && (uint64_t)p.sgen == pre, "PoolRec's mgen and sgen do not hold %llu",
(unsigned long long)pre);
CHECK(!lazy_clean(TEX, TEX + 0x4000, &p), "lazy_clean: the mirror lacks a CPU write after the boundary, yet clean");
CHECK(cpu_wrote_since(TEX + 0x1000, p.sgen), "cpu_wrote_since: a CPU write after the boundary is missed");
CHECK(page_dirty(TEX + 0x1000, p.mgen), "page_dirty: a CPU write after the boundary is missed");
vk_mem_changed_snap(TEX, 0x4000, w_tex, ep, &now); // the region gets this epoch's snapshot
seen(now, "TEX snap");
CHECK(!shadow_pool_clean(&p, ep, 1), "shadow_pool_clean (CPU-side tables): a write after the boundary is missed");
CHECK(!shadow_pool_clean(&p, ep, 0), "shadow_pool_clean: a write after the boundary is missed");
// Synced again now: nothing is newer.
p.mgen = (__typeof__(p.mgen))gen_now, p.sgen = (__typeof__(p.sgen))gen_now;
CHECK(lazy_clean(TEX, TEX + 0x4000, &p), "lazy_clean: synced after every write, yet not clean");
CHECK(!cpu_wrote_since(TEX + 0x1000, p.sgen), "cpu_wrote_since: nothing written since the sync, yet written");
CHECK(!page_dirty(TEX + 0x1000, p.mgen), "page_dirty: nothing written since the sync, yet dirty");
CHECK(shadow_pool_clean(&p, ep, 1) && shadow_pool_clean(&p, ep, 0), "shadow_pool_clean: synced, yet not clean");
// The log: one line where the count reached 2^32, none at any other boundary.
int reached = seed < (1ull << 32) && (uint64_t)gen_now >= (1ull << 32);
CHECK(n_gen_lines == reached, "%d page generation lines, not %d (first: %s)", n_gen_lines, reached,
n_gen_lines ? gen_lines[0] : "none");
if (reached && n_gen_lines)
CHECK(strstr(gen_lines[0], "reached 4294967296 at frame 216000 after 1.0 h of play (19884 a frame)"),
"the line at 2^32: %s", gen_lines[0]);
for (int i = 0; i < n_gen_lines && i < 16; i++) {
CHECK(!strstr(gen_lines[i], "run out") && !strstr(gen_lines[i], "ran out at"), "a line of running out: %s",
gen_lines[i]);
printf(" %s\n", gen_lines[i]);
}
printf("gen_wrap_check: seed %#llx, %u-bit generations, gen_now %#llx: %d of %d checks failed\n",
(unsigned long long)seed, (unsigned)(8 * sizeof(gen_t)), (unsigned long long)gen_now, failed, checks);
return failed ? 1 : 0;
} }

View File

@@ -1,7 +1,13 @@
"""Build and run tests/gen_wrap_check.c: runtime/vk_memory.c's page write generations, extracted verbatim from its """Build and run tests/gen_wrap_check.c: runtime/vk_memory.c's page write generations across 2^32, 2^40 and 2^63.
"page write generations" banner to its "dirty census" banner, with the lines gen_next logs as the 32-bit generations
near their wrap (one at each of 0xC0000000, 0xD0000000, 0xE0000000, 0xF0000000 and one at the wrap), and the four From runtime/vk_memory.c it takes, verbatim, into build/test/gen_wrap/gen_section.inc:
comparisons known to answer wrong across the wrap. - the declarations of PoolRec's mgen and sgen (the generations a VRAM mirror and a shadow copy hold), so the stand-in
PoolRec has their real width;
- the section from the "page write generations" banner to the "dirty census" banner;
- the readers of the tables outside that section: cpu_wrote_since, shadow_pool_clean, lazy_clean and page_dirty.
The program starts the generations just below each seed's boundary and checks that every compare answers across it as
below it (texture checks, the snapshot path, the region check, the CPU-side tables, mirror and shadow cleanliness),
and that gen_next logs one line when the count reaches 2^32 and no other.
No GPU, no game files, any Python. No GPU, no game files, any Python.
@@ -9,6 +15,7 @@ Usage: gen_wrap_check.py [--cc <clang>]
""" """
import argparse import argparse
import os import os
import re
import shutil import shutil
import subprocess import subprocess
import sys import sys
@@ -18,6 +25,9 @@ ROOT = Path(__file__).resolve().parents[1]
MSYS_BIN = Path(r'C:\msys64\ucrt64\bin') MSYS_BIN = Path(r'C:\msys64\ucrt64\bin')
BEGIN = '// ---------------------------------------------------------------- page write generations' BEGIN = '// ---------------------------------------------------------------- page write generations'
END = '// ---------------------------------------------------------------- dirty census' END = '// ---------------------------------------------------------------- dirty census'
READERS = ('static int cpu_wrote_since(', 'static int shadow_pool_clean(', 'static int lazy_clean(',
'static int page_dirty(')
SEEDS = ((1 << 32) - 8, (1 << 40) - 8, (1 << 63) - 8) # the old wrap; low 32 bits through 0 again; near 2^63
def compiler(cc): def compiler(cc):
@@ -30,19 +40,56 @@ def compiler(cc):
return shutil.which('clang') or 'clang' return shutil.which('clang') or 'clang'
def definition(src, start):
"""The definition that begins with start (not a prototype), up to its closing brace at column 0."""
i = 0
while True:
i = src.find(start, i)
if i < 0:
return None
eol = src.index('\n', i)
if src[eol - 1] == '{':
return src[i:src.index('\n}\n', i) + 3]
i = eol
def extract(src):
"""gen_section.inc's text, or None and the reason."""
a, b = src.find(BEGIN), src.find(END)
if a < 0 or b < a:
return None, 'the section banners are not in runtime/vk_memory.c'
s0 = src.find('typedef struct PoolRec {')
s1 = src.find('} PoolRec;', s0)
if s0 < 0 or s1 < 0:
return None, 'PoolRec is not in runtime/vk_memory.c'
fields = []
for name in ('mgen', 'sgen'):
m = re.search(r'^\s*(u?int\d+_t)\s+' + name + r';', src[s0:s1], re.M)
if not m:
return None, f'PoolRec.{name} is not declared as an integer of fixed width'
fields.append(f'{m.group(1)} {name};')
out = ['// Extracted from runtime/vk_memory.c by tests/gen_wrap_check.py.\n',
'typedef struct PoolRec { int kind; void *buf; uint64_t import_lo, import_hi; ' + ' '.join(fields) +
' } PoolRec;\n', src[a:b]]
for start in READERS:
d = definition(src, start)
if not d:
return None, f'{start.split()[-1]}...) is not in runtime/vk_memory.c'
out.append(d)
return '\n'.join(out), None
def main(): def main():
ap = argparse.ArgumentParser() ap = argparse.ArgumentParser()
ap.add_argument('--cc', help='the C compiler (default: $CC, MSYS2 UCRT64 clang, clang on PATH)') ap.add_argument('--cc', help='the C compiler (default: $CC, MSYS2 UCRT64 clang, clang on PATH)')
args = ap.parse_args() args = ap.parse_args()
src = (ROOT / 'runtime' / 'vk_memory.c').read_text(encoding='utf-8-sig') inc, why = extract((ROOT / 'runtime' / 'vk_memory.c').read_text(encoding='utf-8-sig'))
a, b = src.find(BEGIN), src.find(END) if not inc:
if a < 0 or b < a: print(f'gen_wrap_check: FAIL: {why}')
print('gen_wrap_check: FAIL: the section banners are not in runtime/vk_memory.c')
return 1 return 1
work = ROOT / 'build' / 'test' / 'gen_wrap' work = ROOT / 'build' / 'test' / 'gen_wrap'
work.mkdir(parents=True, exist_ok=True) work.mkdir(parents=True, exist_ok=True)
(work / 'gen_section.inc').write_text('// Extracted from runtime/vk_memory.c by tests/gen_wrap_check.py.\n' + (work / 'gen_section.inc').write_text(inc, encoding='utf-8')
src[a:b], encoding='utf-8')
exe = work / ('gen_wrap_check.exe' if os.name == 'nt' else 'gen_wrap_check') exe = work / ('gen_wrap_check.exe' if os.name == 'nt' else 'gen_wrap_check')
env = dict(os.environ) env = dict(os.environ)
if os.name == 'nt' and MSYS_BIN.exists(): if os.name == 'nt' and MSYS_BIN.exists():
@@ -50,7 +97,14 @@ def main():
subprocess.check_call([compiler(args.cc), '-std=gnu11', '-O1', '-Wall', '-Wno-unused-function', subprocess.check_call([compiler(args.cc), '-std=gnu11', '-O1', '-Wall', '-Wno-unused-function',
'-Wno-unused-variable', '-I', str(work), str(ROOT / 'tests' / 'gen_wrap_check.c'), '-o', '-Wno-unused-variable', '-I', str(work), str(ROOT / 'tests' / 'gen_wrap_check.c'), '-o',
str(exe)], env=env) str(exe)], env=env)
return subprocess.run([str(exe)], env=env, timeout=120).returncode bad = 0
for seed in SEEDS:
bad += subprocess.run([str(exe), str(seed)], env=env, timeout=120).returncode != 0
if bad:
print(f'gen_wrap_check: {bad} of {len(SEEDS)} runs failed')
return 1
print('gen_wrap_check: all checks passed')
return 0
if __name__ == '__main__': if __name__ == '__main__':