mirror of
https://github.com/tangerineturtle1/NX2navi.git
synced 2026-11-19 12:59:58 -04:00
Make the page write generations 64-bit, so a long session no longer wraps them and every texture, mirror and shadow check after the wrap answers wrong.
vk_memory.c handed out page write generations from a 32-bit counter (gen_now) and every consumer compared them as 32-bit values, with 0 meaning "never stamped". At 124 to 336 generations a frame (a 0.10.0 log with trace = perf) the counter wrapped after about 66 h of play at 60 fps, 27.6 h at 144 and 15 to 17 h at 240. Past the wrap a stamp made before it compared newer than every later check and one made after it older: textures and VRAM mirrors missed the CPU's writes (stale textures, and the lazy fetch copying older mirror bytes over a CPU write), pages nothing wrote read as written at every check (a hash or upload every time, and every such page of a mirrored pool copied up at each sync), and the generation handed out at the wrap was 0. At 64 bits nothing wraps: 2^64 lasts millions of years at 336 a frame and 240 fps. Below 2^32 every value and every compare is the same as before. What changed: gen_now, both tables (gen_chunk, cgen_chunk), reg_max and creg_max, PoolRec's synced_gen, mgen and sgen, dirty_gen and DIRTY_UNSYNCED, VSample.mgen, every local and parameter that holds a generation in vk_memory.c, vk_texture.c (Tex.wgen and gpu_gen, PEnt.since, storage_rewritten and its callers, sweep_hard's write-back check included) and tex_probe.c, and vk.h's vk_mem_changed, vk_mem_changed_snap and vk_mem_unchanged(_why)_locked. The verify mode's mismatch line prints the generations with %llu. The readers that take no gen_lock (lazy_clean, cpu_wrote_since, verify_cb, vk_mem_sizes) stay lock-free at 64 bits: the slots, mgen, gen_now and a new chunk pointer are now stored atomically (relaxed, or release where a reader acquires), the same instructions on x86-64, and a static assertion holds that 64-bit atomics are lock-free. No lock, wait or thread changes. gen_next, which hands out every generation, no longer warns at 0xC0000000 and up or at the wrap, which cannot come; it logs one line when the count reaches 2^32, where earlier versions ran out, with the frame, the hours and the rate. Cost: the tables take 8 bytes a page instead of 4 (512 KiB more per 256 MiB of guest address space stamped, both tables together, allocated as before when a chunk is first stamped), reg_max and creg_max 2 MiB each instead of 1, Tex and PoolRec 8 bytes more, PEnt 8 more. The page walks read 8 bytes a slot; a micro-benchmark of the walk measured no difference. Not reproduced in the game before the change: the wrong answers were reproduced by compiling vk_memory.c's generation code with the counter seeded below 2^32. tests/gen_wrap_check now runs that code, with the readers outside its section, across 2^32, 2^40 and 2^63 and checks that the compares answer as below the boundary; on the code before the change 19 of its checks fail at 2^32 and the other two seeds do not fit the counter. The new tests/gen_width_check compiles vk_memory.c, vk_texture.c and tex_probe.c with -Wshorten-64-to-32, -Wformat and -Wincompatible-pointer-types as errors, so a 32-bit local left behind is found (shown by reverting single locals, fields and formats). Checks that passed: gen_wrap_check, gen_width_check, pass_cc_check of every runtime file, no new -Wall -Wextra warnings in the three files, tex_release_check, tex_compact_check, wb_lazy_check, gpa_check, mem_growth_check, mem_budget_check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -755,9 +755,17 @@ every copy returns, a heap page nothing touched yet reads zero and stays reserve
|
|||||||
left out. Windows only, no game files.
|
left out. Windows only, no game files.
|
||||||
|
|
||||||
A change to the page write generations (`runtime/vk_memory.c`, from its "page write generations" banner to its "dirty
|
A change to the page write generations (`runtime/vk_memory.c`, from its "page write generations" banner to its "dirty
|
||||||
census" banner: `gen_next`, `gen_mark`) passes `python tests\gen_wrap_check.py`, which builds that part of the file
|
census" banner, and the readers of its tables `cpu_wrote_since`, `shadow_pool_clean`, `lazy_clean`, `page_dirty`)
|
||||||
alone: the lines logged as the 32-bit generations near their wrap, and the four comparisons known to answer wrong across
|
passes `python tests\gen_wrap_check.py`, which builds those parts of the file alone and runs the generations across
|
||||||
it. No GPU, no game files.
|
2^32 (where 32-bit generations wrapped), 2^40 and 2^63: texture checks, the snapshot path and the region check, the
|
||||||
|
CPU-side tables, and mirror and shadow cleanliness answer as below the boundary, and one line is logged when the count
|
||||||
|
reaches 2^32. No GPU, no game files.
|
||||||
|
|
||||||
|
A change to `runtime/vk_memory.c`, `runtime/vk_texture.c` or `runtime/tex_probe.c` passes
|
||||||
|
`python tests\gen_width_check.py`, which compiles the three files with the build's flags and `-Wshorten-64-to-32`,
|
||||||
|
`-Wformat` and `-Wincompatible-pointer-types` as errors: the page write generations are 64 bits, and a local, field or
|
||||||
|
format left at 32 bits would bring their wrap back without a word. Any narrowing in these files takes an explicit cast.
|
||||||
|
It needs the shader headers (`tools/build_shaders.py`); no GPU, no game files.
|
||||||
|
|
||||||
A change to which frames FSR 3.1 frame generation is told are camera cuts (`runtime/vk_fg_reset.h`, `runtime/vk_fg.cpp`
|
A change to which frames FSR 3.1 frame generation is told are camera cuts (`runtime/vk_fg_reset.h`, `runtime/vk_fg.cpp`
|
||||||
`generate` and `vk_fg_frame`) passes `python tests\fg_reset_check.py`, which builds the header alone and checks that each
|
`generate` and `vk_fg_frame`) passes `python tests\fg_reset_check.py`, which builds the header alone and checks that each
|
||||||
|
|||||||
@@ -398,7 +398,7 @@ static int test_import(pool_t *pool) {
|
|||||||
vmaDestroyBuffer(c->vma, sb, sa);
|
vmaDestroyBuffer(c->vma, sb, sa);
|
||||||
|
|
||||||
// generations: written after `since` -> 1, then 0, explicit host write -> 1
|
// generations: written after `since` -> 1, then 0, explicit host write -> 1
|
||||||
uint32_t g0, g1, g2, g3;
|
uint64_t g0, g1, g2, g3;
|
||||||
vk_mem_changed(a, pool->size, 0, &g0);
|
vk_mem_changed(a, pool->size, 0, &g0);
|
||||||
m[0x8000] = 1;
|
m[0x8000] = 1;
|
||||||
GCHECK(vk_mem_changed(a + 0x8000, 0x1000, g0, &g1) == 1, "a CPU write after a generation is not seen");
|
GCHECK(vk_mem_changed(a + 0x8000, 0x1000, g0, &g1) == 1, "a CPU write after a generation is not seen");
|
||||||
@@ -421,7 +421,7 @@ static int test_snap_runs(uint8_t *h) {
|
|||||||
uint64_t base = (uintptr_t)h;
|
uint64_t base = (uintptr_t)h;
|
||||||
GCHECK(!(base & 0x1fffff), "the heap at %#llx does not start on a 2 MiB region", (unsigned long long)base);
|
GCHECK(!(base & 0x1fffff), "the heap at %#llx does not start on a 2 MiB region", (unsigned long long)base);
|
||||||
int left = -1;
|
int left = -1;
|
||||||
uint32_t g0, now;
|
uint64_t g0, now;
|
||||||
vk_texture_new_submit();
|
vk_texture_new_submit();
|
||||||
uint32_t ep = vk_texture_epoch();
|
uint32_t ep = vk_texture_epoch();
|
||||||
vk_mem_gen_lock();
|
vk_mem_gen_lock();
|
||||||
|
|||||||
10
runtime/vk.h
10
runtime/vk.h
@@ -527,11 +527,11 @@ int vk_mem_lookup(uint64_t addr, uint64_t size, VkMemSpan *out);
|
|||||||
uint64_t vk_mem_gen(void);
|
uint64_t vk_mem_gen(void);
|
||||||
// Page write generations over guest memory (mem_written_pages plus vk_guest_written). Returns 1 when a page of the
|
// Page write generations over guest memory (mem_written_pages plus vk_guest_written). Returns 1 when a page of the
|
||||||
// range was written after generation `since`, 0 when not, -1 when the range is not write-watched (treat as written);
|
// range was written after generation `since`, 0 when not, -1 when the range is not write-watched (treat as written);
|
||||||
// *now is the generation to pass next time.
|
// *now is the generation to pass next time. Generations are 64 bits and do not wrap; 0 is never handed out.
|
||||||
int vk_mem_changed(uint64_t addr, uint64_t size, uint32_t since, uint32_t *now);
|
int vk_mem_changed(uint64_t addr, uint64_t size, uint64_t since, uint64_t *now);
|
||||||
// The same answer from per-epoch write-watch snapshots of 2 MiB regions: no system call once the regions were queried in
|
// The same answer from per-epoch write-watch snapshots of 2 MiB regions: no system call once the regions were queried in
|
||||||
// this epoch (a guest submission). For checks that may see writes made during the epoch one epoch late.
|
// this epoch (a guest submission). For checks that may see writes made during the epoch one epoch late.
|
||||||
int vk_mem_changed_snap(uint64_t addr, uint64_t size, uint32_t since, uint32_t epoch, uint32_t *now);
|
int vk_mem_changed_snap(uint64_t addr, uint64_t size, uint64_t since, uint32_t epoch, uint64_t *now);
|
||||||
void vk_mem_snap_stats(uint64_t *checks, uint64_t *calls, uint64_t *untracked); // since the last call
|
void vk_mem_snap_stats(uint64_t *checks, uint64_t *calls, uint64_t *untracked); // since the last call
|
||||||
|
|
||||||
// Upload batch: one command buffer on gfx_queue that texture uploads and texel copies record into. begin/end bracket
|
// Upload batch: one command buffer on gfx_queue that texture uploads and texel copies record into. begin/end bracket
|
||||||
@@ -801,10 +801,10 @@ int vk_texture_slot_retry_locked(int id);
|
|||||||
// have changed), 0 when a closer look is due (vk_mem_changed_snap).
|
// have changed), 0 when a closer look is due (vk_mem_changed_snap).
|
||||||
void vk_mem_gen_lock_shared(void);
|
void vk_mem_gen_lock_shared(void);
|
||||||
void vk_mem_gen_unlock_shared(void);
|
void vk_mem_gen_unlock_shared(void);
|
||||||
int vk_mem_unchanged_locked(uint64_t addr, uint64_t size, uint32_t since, uint32_t epoch);
|
int vk_mem_unchanged_locked(uint64_t addr, uint64_t size, uint64_t since, uint32_t epoch);
|
||||||
// The same check's reason when it fails: 1 a region without this epoch's snapshot yet, 2 one untracked, 3 one written
|
// The same check's reason when it fails: 1 a region without this epoch's snapshot yet, 2 one untracked, 3 one written
|
||||||
// since, 4 nothing to check (no size, beyond the regions); 0 unchanged.
|
// since, 4 nothing to check (no size, beyond the regions); 0 unchanged.
|
||||||
int vk_mem_unchanged_why_locked(uint64_t addr, uint64_t size, uint32_t since, uint32_t epoch);
|
int vk_mem_unchanged_why_locked(uint64_t addr, uint64_t size, uint64_t since, uint32_t epoch);
|
||||||
// The lock exclusively, for the snapshot of the regions of [addr, addr+size) that have none in epoch yet (the number of
|
// The lock exclusively, for the snapshot of the regions of [addr, addr+size) that have none in epoch yet (the number of
|
||||||
// write-watch queries it took).
|
// write-watch queries it took).
|
||||||
void vk_mem_gen_lock(void);
|
void vk_mem_gen_lock(void);
|
||||||
|
|||||||
@@ -60,7 +60,7 @@ typedef struct PoolRec {
|
|||||||
volatile int32_t used; // resolved at least once: the GPU may still read it at finalize
|
volatile int32_t used; // resolved at least once: the GPU may still read it at finalize
|
||||||
// NX2NAVI_TRACE=perf dirty census (vk_mem_dirty_submit), replay thread under the submit lock only:
|
// NX2NAVI_TRACE=perf dirty census (vk_mem_dirty_submit), replay thread under the submit lock only:
|
||||||
uint32_t bound_sub; // last submit that bound it as a buffer (dirty_sub), 0 never
|
uint32_t bound_sub; // last submit that bound it as a buffer (dirty_sub), 0 never
|
||||||
uint32_t synced_gen; // page generation its lazy-mirror count covers; DIRTY_UNSYNCED until scanned
|
uint64_t synced_gen; // page generation its lazy-mirror count covers; DIRTY_UNSYNCED until scanned
|
||||||
uint8_t bufbound; // ever bound as a vertex, index, uniform, storage or indirect buffer
|
uint8_t bufbound; // ever bound as a vertex, index, uniform, storage or indirect buffer
|
||||||
uint8_t untracked; // the last dirty scan found no write watch over it
|
uint8_t untracked; // the last dirty scan found no write watch over it
|
||||||
// VRAM mirror (POOL_IMPORT, "VRAM mirror" below). mstate and mbuf change on the replay thread only.
|
// VRAM mirror (POOL_IMPORT, "VRAM mirror" below). mstate and mbuf change on the replay thread only.
|
||||||
@@ -86,7 +86,7 @@ typedef struct PoolRec {
|
|||||||
uint8_t twin; // MS_ON: the table's writes reach guest memory too (twin_wanted); 0: the mirror only
|
uint8_t twin; // MS_ON: the table's writes reach guest memory too (twin_wanted); 0: the mirror only
|
||||||
uint32_t vd_wrote_sub, vd_use_sub; // verify mode: the last submission that bound it written / used its mirror
|
uint32_t vd_wrote_sub, vd_use_sub; // verify mode: the last submission that bound it written / used its mirror
|
||||||
uint64_t vd_wrote_n; // verify mode: written bindings of it so far
|
uint64_t vd_wrote_n; // verify mode: written bindings of it so far
|
||||||
uint32_t mgen; // gen_lock: every page write stamped up to this generation is in the mirror
|
uint64_t mgen; // gen_lock: every page write stamped up to this generation is in the mirror
|
||||||
// (or in a copy queued before its next use)
|
// (or in a copy queued before its next use)
|
||||||
FlushRange fl[FLUSH_RANGES + 1]; // flush_lock: nvnMemoryPoolFlushMappedRange pages not copied yet (flush_add)
|
FlushRange fl[FLUSH_RANGES + 1]; // flush_lock: nvnMemoryPoolFlushMappedRange pages not copied yet (flush_add)
|
||||||
int fl_n; // flush_lock: ranges in fl; read without the lock to skip an empty list
|
int fl_n; // flush_lock: ranges in fl; read without the lock to skip an empty list
|
||||||
@@ -96,7 +96,7 @@ typedef struct PoolRec {
|
|||||||
// Shadow storage (POOL_IMPORT in shadow mode, "shadow storage" below): buf is a host-visible copy of the guest pages.
|
// Shadow storage (POOL_IMPORT in shadow mode, "shadow storage" below): buf is a host-visible copy of the guest pages.
|
||||||
VmaAllocation salloc;
|
VmaAllocation salloc;
|
||||||
uint8_t *smap; // its mapping; offsets as in buf (from import_lo)
|
uint8_t *smap; // its mapping; offsets as in buf (from import_lo)
|
||||||
uint32_t sgen; // gen_lock: every page write stamped up to this generation is in the shadow
|
uint64_t sgen; // gen_lock: every page write stamped up to this generation is in the shadow
|
||||||
uint64_t sborn; // the value of the first submission that can have used the shadow (shadow_pool)
|
uint64_t sborn; // the value of the first submission that can have used the shadow (shadow_pool)
|
||||||
volatile uint8_t stouched; // on the touched list: the GPU may use it in the submission being recorded
|
volatile uint8_t stouched; // on the touched list: the GPU may use it in the submission being recorded
|
||||||
uint32_t stab_i; // shadow_lock: 1 + its index in stab (the table's shadows), 0 when not there
|
uint32_t stab_i; // shadow_lock: 1 + its index in stab (the table's shadows), 0 when not there
|
||||||
@@ -1160,49 +1160,52 @@ int vk_mem_vram_live(uint64_t *used, uint64_t *budget) {
|
|||||||
// ---------------------------------------------------------------- page write generations
|
// ---------------------------------------------------------------- page write generations
|
||||||
// Two-level table over the guest address space (below 2^39): 256 MiB chunks of 65,536 page generations, allocated
|
// Two-level table over the guest address space (below 2^39): 256 MiB chunks of 65,536 page generations, allocated
|
||||||
// when a page of the chunk is first stamped. 0 = never stamped.
|
// when a page of the chunk is first stamped. 0 = never stamped.
|
||||||
|
// Generations are 64 bits and never run out (at 336 a frame and 240 fps, 2^64 lasts millions of years). At 32 bits,
|
||||||
|
// 336 a frame (the most measured, 0.10.0) and 240 fps ran them out after about 15 h of play, and past the wrap a stamp
|
||||||
|
// from before it compared newer than every later check and one made after it older: textures and pool mirrors missed
|
||||||
|
// the CPU's writes, and pages nothing wrote read as written at every check. Below 2^32 every value and every compare
|
||||||
|
// is what it was at 32 bits.
|
||||||
#define GEN_CHUNK_SHIFT 28
|
#define GEN_CHUNK_SHIFT 28
|
||||||
#define GEN_PAGE_SHIFT 12
|
#define GEN_PAGE_SHIFT 12
|
||||||
#define GEN_CHUNKS (RT_ADDRESS_LIMIT >> GEN_CHUNK_SHIFT)
|
#define GEN_CHUNKS (RT_ADDRESS_LIMIT >> GEN_CHUNK_SHIFT)
|
||||||
#define GEN_PER_CHUNK (1u << (GEN_CHUNK_SHIFT - GEN_PAGE_SHIFT))
|
#define GEN_PER_CHUNK (1u << (GEN_CHUNK_SHIFT - GEN_PAGE_SHIFT))
|
||||||
static uint32_t *gen_chunk[GEN_CHUNKS];
|
static uint64_t *gen_chunk[GEN_CHUNKS];
|
||||||
static uint32_t gen_now; // last generation handed out
|
static uint64_t gen_now; // last generation handed out (gen_lock; read without it atomically)
|
||||||
static PlatMutex gen_lock = PLAT_MUTEX_INIT;
|
static PlatMutex gen_lock = PLAT_MUTEX_INIT;
|
||||||
static uint64_t gen_pages[1024];
|
static uint64_t gen_pages[1024];
|
||||||
|
// lazy_clean, cpu_wrote_since, verify_cb and vk_mem_sizes read generations without gen_lock: 64-bit loads and stores
|
||||||
|
// must be single instructions on every target.
|
||||||
|
_Static_assert(__atomic_always_lock_free(sizeof(uint64_t), 0), "64-bit page generations need lock-free atomics");
|
||||||
|
|
||||||
// Generations are 32 bits, and nothing compares them across a wrap: past it a stamp from before reads as newer than
|
// When the count reaches 2^32, where the 32-bit generations of earlier versions ran out, the log says so once, with
|
||||||
// every later check, and a stamp of 0 as never stamped, so textures and pool mirrors the CPU writes can miss a write
|
// the frame, the hours of play and the rate, so a long session's log shows how far it got. gen_lock held exclusively;
|
||||||
// and every check reads as a write. At about 181 a frame that comes after about 110 h of play at 60 fps, 27 h at 240.
|
// the log takes no lock of this file.
|
||||||
// Only trace=perf's "mem: sizes" counted them, so gen_next, which hands each one out, logs where the session stands at
|
static void gen_mark(uint64_t g) {
|
||||||
// 0xC0000000, 0xD0000000, 0xE0000000 and 0xF0000000, and the wrap itself (gen_mark): five lines at most.
|
|
||||||
static void gen_mark(uint32_t g) { // gen_lock held exclusively; the log takes no lock of this file
|
|
||||||
uint64_t frames = __atomic_load_n(&gfx_counters.frames, __ATOMIC_RELAXED);
|
uint64_t frames = __atomic_load_n(&gfx_counters.frames, __ATOMIC_RELAXED);
|
||||||
double secs = rt_guest_start_ns ? (double)(kern_now_ns() - rt_guest_start_ns) / 1e9 : 0, all = 4294967296.0;
|
double secs = rt_guest_start_ns ? (double)(kern_now_ns() - rt_guest_start_ns) / 1e9 : 0;
|
||||||
if (!g) {
|
rt_log("mem: page generations reached %llu at frame %llu after %.1f h of play (%.0f a frame); before 0.10.2 they "
|
||||||
rt_log("mem: page generations ran out at frame %llu after %.1f h: from here on textures and pool mirrors the "
|
"were 32 bits and ran out here, after which textures and pool mirrors could miss the CPU's writes",
|
||||||
"CPU writes can show old content and every check uploads again; restart the game",
|
(unsigned long long)g, (unsigned long long)frames, secs / 3600, frames ? (double)g / (double)frames : 0.0);
|
||||||
(unsigned long long)frames, secs / 3600);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
rt_log("mem: page generations: %u of 4294967295 handed out at frame %llu (%.0f a frame); at this rate they run out "
|
|
||||||
"in about %.1f h of play, after which textures and pool mirrors the CPU writes can miss a write and every "
|
|
||||||
"check reads as a write: restart the game before then", g, (unsigned long long)frames,
|
|
||||||
frames ? g / (double)frames : 0.0, secs > 0 ? (all - g) * secs / g / 3600 : 0.0);
|
|
||||||
}
|
}
|
||||||
static inline uint32_t gen_next(void) { // gen_lock held exclusively
|
|
||||||
uint32_t g = ++gen_now;
|
// gen_lock held exclusively: the only place a generation is handed out. The store is atomic: vk_mem_sizes and
|
||||||
if (__builtin_expect(!(g & 0x0fffffffu) && (g >= 0xc0000000u || !g), 0)) gen_mark(g);
|
// verify_cb read gen_now without gen_lock.
|
||||||
|
static inline uint64_t gen_next(void) {
|
||||||
|
uint64_t g = gen_now + 1;
|
||||||
|
if (__builtin_expect(g == 1ull << 32, 0)) gen_mark(g);
|
||||||
|
__atomic_store_n(&gen_now, g, __ATOMIC_RELAXED);
|
||||||
return g;
|
return g;
|
||||||
}
|
}
|
||||||
// The newest generation stamped on any page of each 2 MiB region (the write-watch snapshot regions below): the mirror
|
// The newest generation stamped on any page of each 2 MiB region (the write-watch snapshot regions below): the mirror
|
||||||
// sync skips regions with nothing newer than its pool's mirror.
|
// sync skips regions with nothing newer than its pool's mirror.
|
||||||
#define REG_SHIFT 21
|
#define REG_SHIFT 21
|
||||||
static uint32_t reg_max[RT_ADDRESS_LIMIT >> REG_SHIFT];
|
static uint64_t reg_max[RT_ADDRESS_LIMIT >> REG_SHIFT];
|
||||||
// The same two tables without the stamps of a mirror's own write-back (vk_gpu_written_back): what the CPU and the host
|
// The same two tables without the stamps of a mirror's own write-back (vk_gpu_written_back): what the CPU and the host
|
||||||
// paths wrote. The mirror sync reads these, so the bytes a mirror wrote back into guest memory are not copied up into it
|
// paths wrote. The mirror sync reads these, so the bytes a mirror wrote back into guest memory are not copied up into it
|
||||||
// again at its next sync: the storage buffers the game's compute shaders write went down and came back up every frame,
|
// again at its next sync: the storage buffers the game's compute shaders write went down and came back up every frame,
|
||||||
// 20 MiB a frame over PCIe in Kokiri Forest (2026-11-09).
|
// 20 MiB a frame over PCIe in Kokiri Forest (2026-11-09).
|
||||||
static uint32_t *cgen_chunk[GEN_CHUNKS];
|
static uint64_t *cgen_chunk[GEN_CHUNKS];
|
||||||
static uint32_t creg_max[RT_ADDRESS_LIMIT >> REG_SHIFT];
|
static uint64_t creg_max[RT_ADDRESS_LIMIT >> REG_SHIFT];
|
||||||
|
|
||||||
// Every write-watch query of the generation table (gen_lock held exclusively): calls, bytes and time since the last
|
// Every write-watch query of the generation table (gen_lock held exclusively): calls, bytes and time since the last
|
||||||
// mirror summary, all consumers together; the queries that found written pages (they also flush TLBs) apart.
|
// mirror summary, all consumers together; the queries that found written pages (they also flush TLBs) apart.
|
||||||
@@ -1269,35 +1272,38 @@ static uint32_t *fcand; // gen_lock: regions to try at the
|
|||||||
static int nfcand, cfcand;
|
static int nfcand, cfcand;
|
||||||
static uint64_t n_frz_refused; // gen_lock: candidates mem_freeze refused
|
static uint64_t n_frz_refused; // gen_lock: candidates mem_freeze refused
|
||||||
|
|
||||||
static uint32_t *slot_in(uint32_t **tab, uint64_t page_addr, int create) {
|
static uint64_t *slot_in(uint64_t **tab, uint64_t page_addr, int create) {
|
||||||
uint64_t ci = page_addr >> GEN_CHUNK_SHIFT;
|
uint64_t ci = page_addr >> GEN_CHUNK_SHIFT;
|
||||||
if (ci >= GEN_CHUNKS) return NULL;
|
if (ci >= GEN_CHUNKS) return NULL;
|
||||||
if (!tab[ci]) {
|
if (!tab[ci]) {
|
||||||
if (!create) return NULL;
|
if (!create) return NULL;
|
||||||
tab[ci] = calloc(GEN_PER_CHUNK, sizeof(uint32_t));
|
uint64_t *p = calloc(GEN_PER_CHUNK, sizeof *p);
|
||||||
if (!tab[ci]) rt_fatal("vk: out of memory for page generations");
|
if (!p) rt_fatal("vk: out of memory for page generations");
|
||||||
|
// published with release: lazy_clean and cpu_wrote_since load the chunk pointers without gen_lock
|
||||||
|
__atomic_store_n(&tab[ci], p, __ATOMIC_RELEASE);
|
||||||
}
|
}
|
||||||
return &tab[ci][(page_addr >> GEN_PAGE_SHIFT) & (GEN_PER_CHUNK - 1)];
|
return &tab[ci][(page_addr >> GEN_PAGE_SHIFT) & (GEN_PER_CHUNK - 1)];
|
||||||
}
|
}
|
||||||
|
|
||||||
static uint32_t *gen_slot(uint64_t page_addr, int create) { return slot_in(gen_chunk, page_addr, create); }
|
static uint64_t *gen_slot(uint64_t page_addr, int create) { return slot_in(gen_chunk, page_addr, create); }
|
||||||
|
|
||||||
// gen_lock held exclusively; g is the newest generation handed out. back: a mirror's write-back, which leaves the
|
// gen_lock held exclusively; g is the newest generation handed out. back: a mirror's write-back, which leaves the
|
||||||
// CPU-side tables alone.
|
// CPU-side tables alone. The slots are stored atomically: lazy_clean, cpu_wrote_since and verify_cb read them
|
||||||
static void stamp_page_as(uint64_t a, uint32_t g, int back) {
|
// without gen_lock.
|
||||||
uint32_t *s = gen_slot(a, 1);
|
static void stamp_page_as(uint64_t a, uint64_t g, int back) {
|
||||||
|
uint64_t *s = gen_slot(a, 1);
|
||||||
if (!s) return;
|
if (!s) return;
|
||||||
*s = g;
|
__atomic_store_n(s, g, __ATOMIC_RELAXED);
|
||||||
reg_max[a >> REG_SHIFT] = g;
|
reg_max[a >> REG_SHIFT] = g;
|
||||||
reg_last[a >> REG_SHIFT] = ww_ep;
|
reg_last[a >> REG_SHIFT] = ww_ep;
|
||||||
if (back) return;
|
if (back) return;
|
||||||
uint32_t *c = slot_in(cgen_chunk, a, 1);
|
uint64_t *c = slot_in(cgen_chunk, a, 1);
|
||||||
if (c) *c = g, creg_max[a >> REG_SHIFT] = g;
|
if (c) __atomic_store_n(c, g, __ATOMIC_RELAXED), creg_max[a >> REG_SHIFT] = g;
|
||||||
}
|
}
|
||||||
|
|
||||||
static void stamp_page(uint64_t a, uint32_t g) { stamp_page_as(a, g, 0); }
|
static void stamp_page(uint64_t a, uint64_t g) { stamp_page_as(a, g, 0); }
|
||||||
|
|
||||||
static void stamp_range(uint64_t lo, uint64_t hi, uint32_t g) {
|
static void stamp_range(uint64_t lo, uint64_t hi, uint64_t g) {
|
||||||
for (uint64_t a = lo; a < hi; a += 1ull << GEN_PAGE_SHIFT) stamp_page(a, g);
|
for (uint64_t a = lo; a < hi; a += 1ull << GEN_PAGE_SHIFT) stamp_page(a, g);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1309,11 +1315,11 @@ void vk_guest_written(uint64_t addr, uint64_t size) {
|
|||||||
plat_mutex_unlock(&gen_lock);
|
plat_mutex_unlock(&gen_lock);
|
||||||
}
|
}
|
||||||
|
|
||||||
int vk_mem_changed(uint64_t addr, uint64_t size, uint32_t since, uint32_t *now) {
|
int vk_mem_changed(uint64_t addr, uint64_t size, uint64_t since, uint64_t *now) {
|
||||||
if (!size) { *now = since; return 0; }
|
if (!size) { *now = since; return 0; }
|
||||||
uint64_t lo = addr & ~0xfffull, hi = (addr + size + 0xfff) & ~0xfffull;
|
uint64_t lo = addr & ~0xfffull, hi = (addr + size + 0xfff) & ~0xfffull;
|
||||||
plat_mutex_lock(&gen_lock);
|
plat_mutex_lock(&gen_lock);
|
||||||
uint32_t g = gen_next();
|
uint64_t g = gen_next();
|
||||||
int tracked = 1, frozen = 1;
|
int tracked = 1, frozen = 1;
|
||||||
for (uint64_t a = lo & ~((1ull << REG_SHIFT) - 1); a < hi && frozen; a += 1ull << REG_SHIFT) frozen = mem_frozen(a);
|
for (uint64_t a = lo & ~((1ull << REG_SHIFT) - 1); a < hi && frozen; a += 1ull << REG_SHIFT) frozen = mem_frozen(a);
|
||||||
while (!frozen) {
|
while (!frozen) {
|
||||||
@@ -1324,7 +1330,7 @@ int vk_mem_changed(uint64_t addr, uint64_t size, uint32_t since, uint32_t *now)
|
|||||||
}
|
}
|
||||||
int changed = 0;
|
int changed = 0;
|
||||||
for (uint64_t a = lo; tracked && a < hi && !changed; a += 1ull << GEN_PAGE_SHIFT) {
|
for (uint64_t a = lo; tracked && a < hi && !changed; a += 1ull << GEN_PAGE_SHIFT) {
|
||||||
uint32_t *s = gen_slot(a, 0);
|
uint64_t *s = gen_slot(a, 0);
|
||||||
if (s && *s > since) changed = 1;
|
if (s && *s > since) changed = 1;
|
||||||
}
|
}
|
||||||
plat_mutex_unlock(&gen_lock);
|
plat_mutex_unlock(&gen_lock);
|
||||||
@@ -1346,7 +1352,7 @@ static uint64_t n_snap_checks, n_snap_calls, n_snap_untracked;
|
|||||||
// gen_lock held exclusively. Queries the regions r0..r1 that were not snapped in epoch, one query per run of them;
|
// gen_lock held exclusively. Queries the regions r0..r1 that were not snapped in epoch, one query per run of them;
|
||||||
// returns the number of queries.
|
// returns the number of queries.
|
||||||
static uint64_t snap_regions(uint64_t r0, uint64_t r1, uint32_t epoch) {
|
static uint64_t snap_regions(uint64_t r0, uint64_t r1, uint32_t epoch) {
|
||||||
uint32_t g = 0;
|
uint64_t g = 0;
|
||||||
uint64_t calls = 0;
|
uint64_t calls = 0;
|
||||||
for (uint64_t r = r0; r <= r1;) {
|
for (uint64_t r = r0; r <= r1;) {
|
||||||
if (snap_at[r] == epoch) { r++; continue; }
|
if (snap_at[r] == epoch) { r++; continue; }
|
||||||
@@ -1419,7 +1425,7 @@ static void snap_widen(uint64_t lo, uint64_t hi, uint64_t *r0, uint64_t *r1) {
|
|||||||
plat_mutex_unlock_shared(&pool_lock);
|
plat_mutex_unlock_shared(&pool_lock);
|
||||||
}
|
}
|
||||||
|
|
||||||
int vk_mem_changed_snap(uint64_t addr, uint64_t size, uint32_t since, uint32_t epoch, uint32_t *now) {
|
int vk_mem_changed_snap(uint64_t addr, uint64_t size, uint64_t since, uint32_t epoch, uint64_t *now) {
|
||||||
if (!size) { *now = since; return 0; }
|
if (!size) { *now = since; return 0; }
|
||||||
uint64_t lo = addr & ~0xfffull, hi = (addr + size + 0xfff) & ~0xfffull;
|
uint64_t lo = addr & ~0xfffull, hi = (addr + size + 0xfff) & ~0xfffull;
|
||||||
uint64_t r0 = lo >> SNAP_SHIFT, r1 = (hi - 1) >> SNAP_SHIFT;
|
uint64_t r0 = lo >> SNAP_SHIFT, r1 = (hi - 1) >> SNAP_SHIFT;
|
||||||
@@ -1443,7 +1449,7 @@ int vk_mem_changed_snap(uint64_t addr, uint64_t size, uint32_t since, uint32_t e
|
|||||||
if (reg_max[r] <= since) continue;
|
if (reg_max[r] <= since) continue;
|
||||||
uint64_t a = r << REG_SHIFT > lo ? r << REG_SHIFT : lo, b = (r + 1) << REG_SHIFT < hi ? (r + 1) << REG_SHIFT : hi;
|
uint64_t a = r << REG_SHIFT > lo ? r << REG_SHIFT : lo, b = (r + 1) << REG_SHIFT < hi ? (r + 1) << REG_SHIFT : hi;
|
||||||
for (; a < b && !changed; a += 1ull << GEN_PAGE_SHIFT) {
|
for (; a < b && !changed; a += 1ull << GEN_PAGE_SHIFT) {
|
||||||
uint32_t *s = gen_slot(a, 0);
|
uint64_t *s = gen_slot(a, 0);
|
||||||
if (s && *s > since) changed = 1;
|
if (s && *s > since) changed = 1;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -1461,7 +1467,7 @@ void vk_mem_gen_unlock_shared(void) { plat_mutex_unlock_shared(&gen_lock); }
|
|||||||
void vk_mem_gen_lock(void) { plat_mutex_lock(&gen_lock); }
|
void vk_mem_gen_lock(void) { plat_mutex_lock(&gen_lock); }
|
||||||
void vk_mem_gen_unlock(void) { plat_mutex_unlock(&gen_lock); }
|
void vk_mem_gen_unlock(void) { plat_mutex_unlock(&gen_lock); }
|
||||||
|
|
||||||
int vk_mem_unchanged_why_locked(uint64_t addr, uint64_t size, uint32_t since, uint32_t epoch) {
|
int vk_mem_unchanged_why_locked(uint64_t addr, uint64_t size, uint64_t since, uint32_t epoch) {
|
||||||
if (!size || !epoch) return 4;
|
if (!size || !epoch) return 4;
|
||||||
uint64_t lo = addr & ~0xfffull, hi = (addr + size + 0xfff) & ~0xfffull;
|
uint64_t lo = addr & ~0xfffull, hi = (addr + size + 0xfff) & ~0xfffull;
|
||||||
uint64_t r0 = lo >> SNAP_SHIFT, r1 = (hi - 1) >> SNAP_SHIFT;
|
uint64_t r0 = lo >> SNAP_SHIFT, r1 = (hi - 1) >> SNAP_SHIFT;
|
||||||
@@ -1474,7 +1480,7 @@ int vk_mem_unchanged_why_locked(uint64_t addr, uint64_t size, uint32_t since, ui
|
|||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
int vk_mem_unchanged_locked(uint64_t addr, uint64_t size, uint32_t since, uint32_t epoch) {
|
int vk_mem_unchanged_locked(uint64_t addr, uint64_t size, uint64_t since, uint32_t epoch) {
|
||||||
return !vk_mem_unchanged_why_locked(addr, size, since, epoch);
|
return !vk_mem_unchanged_why_locked(addr, size, since, epoch);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1561,7 +1567,7 @@ void vk_mem_snap_stats(uint64_t *checks, uint64_t *calls, uint64_t *untracked) {
|
|||||||
// - lazy: pages stamped after the pool's previous count, in pools the previous submit bound (upload when bound);
|
// - lazy: pages stamped after the pool's previous count, in pools the previous submit bound (upload when bound);
|
||||||
// - first use: the size of a pool the first time it is bound (a mirror copies the whole pool then).
|
// - first use: the size of a pool the first time it is bound (a mirror copies the whole pool then).
|
||||||
// Classes: CPU_CACHED, CPU_UNCACHED, other (CPU_NO_ACCESS). Logged as `mem: dirty` every 60 frames.
|
// Classes: CPU_CACHED, CPU_UNCACHED, other (CPU_NO_ACCESS). Logged as `mem: dirty` every 60 frames.
|
||||||
#define DIRTY_UNSYNCED UINT32_MAX
|
#define DIRTY_UNSYNCED UINT64_MAX
|
||||||
#define DIRTY_GAP 0x100000ull // pools closer than this share one query (the gap's pages are stamped too)
|
#define DIRTY_GAP 0x100000ull // pools closer than this share one query (the gap's pages are stamped too)
|
||||||
#define DIRTY_WIN 60
|
#define DIRTY_WIN 60
|
||||||
enum { DC_CACHED, DC_UNC, DC_OTHER, DC_N };
|
enum { DC_CACHED, DC_UNC, DC_OTHER, DC_N };
|
||||||
@@ -1569,7 +1575,8 @@ typedef struct {
|
|||||||
uint64_t eager[DC_N], lazy[DC_N], first[DC_N];
|
uint64_t eager[DC_N], lazy[DC_N], first[DC_N];
|
||||||
uint64_t submits, queries, scan_ns, sub_max, frames;
|
uint64_t submits, queries, scan_ns, sub_max, frames;
|
||||||
} DirtyAcc;
|
} DirtyAcc;
|
||||||
static uint32_t dirty_sub = 1, dirty_gen; // submit lock: the current submit number, generation of the last scan
|
static uint32_t dirty_sub = 1; // submit lock: the current submit number
|
||||||
|
static uint64_t dirty_gen; // submit lock: the generation of the last scan
|
||||||
static uint64_t first_pending[DC_N], ever_n[DC_N], ever_bytes[DC_N]; // submit lock
|
static uint64_t first_pending[DC_N], ever_n[DC_N], ever_bytes[DC_N]; // submit lock
|
||||||
static PoolRec **dlist; // submit lock: the buffer-bound pools of a scan
|
static PoolRec **dlist; // submit lock: the buffer-bound pools of a scan
|
||||||
static int dcap;
|
static int dcap;
|
||||||
@@ -1597,7 +1604,7 @@ static uint64_t page_lo(const PoolRec *r) { return r->lo & ~0xfffull; }
|
|||||||
static uint64_t page_hi(const PoolRec *r) { return (r->lo + r->size + 0xfff) & ~0xfffull; }
|
static uint64_t page_hi(const PoolRec *r) { return (r->lo + r->size + 0xfff) & ~0xfffull; }
|
||||||
|
|
||||||
// gen_lock held exclusively. 0: the range is not write-watched.
|
// gen_lock held exclusively. 0: the range is not write-watched.
|
||||||
static int query_stamp(uint64_t lo, uint64_t hi, uint32_t g) {
|
static int query_stamp(uint64_t lo, uint64_t hi, uint64_t g) {
|
||||||
for (;;) {
|
for (;;) {
|
||||||
uint64_t n = ww_query(lo, hi - lo);
|
uint64_t n = ww_query(lo, hi - lo);
|
||||||
if (n == UINT64_MAX) return 0;
|
if (n == UINT64_MAX) return 0;
|
||||||
@@ -1998,7 +2005,7 @@ static void shadow_sync_pool(PoolRec *r, uint32_t epoch, int cpu) {
|
|||||||
s_queries += snap_regions(r0, r1, epoch);
|
s_queries += snap_regions(r0, r1, epoch);
|
||||||
for (uint64_t k = r0; k <= r1 && tracked; k++) tracked = !snap_untracked[k];
|
for (uint64_t k = r0; k <= r1 && tracked; k++) tracked = !snap_untracked[k];
|
||||||
} else { // outside the snapshot regions, or before the first guest submission: one query per unfrozen run
|
} else { // outside the snapshot regions, or before the first guest submission: one query per unfrozen run
|
||||||
uint32_t g = gen_next();
|
uint64_t g = gen_next();
|
||||||
for (uint64_t a = lo; a < hi;) {
|
for (uint64_t a = lo; a < hi;) {
|
||||||
uint64_t b = (a | ((1ull << REG_SHIFT) - 1)) + 1;
|
uint64_t b = (a | ((1ull << REG_SHIFT) - 1)) + 1;
|
||||||
if (b > hi) b = hi;
|
if (b > hi) b = hi;
|
||||||
@@ -2010,7 +2017,7 @@ static void shadow_sync_pool(PoolRec *r, uint32_t epoch, int cpu) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
s_q_ticks += qpc() - t0;
|
s_q_ticks += qpc() - t0;
|
||||||
uint32_t since = r->sgen;
|
uint64_t since = r->sgen;
|
||||||
r->sgen = gen_now;
|
r->sgen = gen_now;
|
||||||
if (!tracked) { // nothing tells which pages changed: all of them
|
if (!tracked) { // nothing tells which pages changed: all of them
|
||||||
s_untracked++;
|
s_untracked++;
|
||||||
@@ -2022,7 +2029,7 @@ static void shadow_sync_pool(PoolRec *r, uint32_t epoch, int cpu) {
|
|||||||
uint64_t a = k << REG_SHIFT, b = a + (1ull << REG_SHIFT);
|
uint64_t a = k << REG_SHIFT, b = a + (1ull << REG_SHIFT);
|
||||||
if (a < lo) a = lo;
|
if (a < lo) a = lo;
|
||||||
if (b > hi) b = hi;
|
if (b > hi) b = hi;
|
||||||
const uint32_t *ch = (cpu ? cgen_chunk : gen_chunk)[a >> GEN_CHUNK_SHIFT]; // a 2 MiB region lies in one chunk
|
const uint64_t *ch = (cpu ? cgen_chunk : gen_chunk)[a >> GEN_CHUNK_SHIFT]; // a 2 MiB region lies in one chunk
|
||||||
if ((cpu ? creg_max : reg_max)[k] <= since || !ch) {
|
if ((cpu ? creg_max : reg_max)[k] <= since || !ch) {
|
||||||
if (run) shadow_copy(r, run, a), run = 0;
|
if (run) shadow_copy(r, run, a), run = 0;
|
||||||
continue;
|
continue;
|
||||||
@@ -2086,8 +2093,8 @@ static int shadow_bulk_write(void) {
|
|||||||
// such a page may have come after a CPU write it should have come before ("The game must not get ahead of the
|
// such a page may have come after a CPU write it should have come before ("The game must not get ahead of the
|
||||||
// write-backs" above); a page of which the CPU wrote other bytes counts too.
|
// write-backs" above); a page of which the CPU wrote other bytes counts too.
|
||||||
// No gen_lock, as lazy_clean: a stamp that lands meanwhile only blurs a count.
|
// No gen_lock, as lazy_clean: a stamp that lands meanwhile only blurs a count.
|
||||||
static int cpu_wrote_since(uint64_t pg, uint32_t sgen) {
|
static int cpu_wrote_since(uint64_t pg, uint64_t sgen) {
|
||||||
const uint32_t *ch = __atomic_load_n(&cgen_chunk[pg >> GEN_CHUNK_SHIFT], __ATOMIC_ACQUIRE);
|
const uint64_t *ch = __atomic_load_n(&cgen_chunk[pg >> GEN_CHUNK_SHIFT], __ATOMIC_ACQUIRE);
|
||||||
if (ch && __atomic_load_n(&ch[(pg >> GEN_PAGE_SHIFT) & (GEN_PER_CHUNK - 1)], __ATOMIC_RELAXED) > sgen) return 1;
|
if (ch && __atomic_load_n(&ch[(pg >> GEN_PAGE_SHIFT) & (GEN_PER_CHUNK - 1)], __ATOMIC_RELAXED) > sgen) return 1;
|
||||||
uint64_t p[1], n = mem_written_peek(pg, SB_PAGE, p, 1);
|
uint64_t p[1], n = mem_written_peek(pg, SB_PAGE, p, 1);
|
||||||
return n && n != UINT64_MAX;
|
return n && n != UINT64_MAX;
|
||||||
@@ -2168,7 +2175,7 @@ static void table_wb_cb(void *arg) {
|
|||||||
s_tab_wb += moved, s_tab_wb_seen += seen, s_tab_wb_jobs++;
|
s_tab_wb += moved, s_tab_wb_seen += seen, s_tab_wb_jobs++;
|
||||||
if (nwrote) {
|
if (nwrote) {
|
||||||
plat_mutex_lock(&gen_lock);
|
plat_mutex_lock(&gen_lock);
|
||||||
uint32_t g = gen_next();
|
uint64_t g = gen_next();
|
||||||
for (int i = 0; i < nwrote; i++) stamp_page_as(wrote[i], g, back);
|
for (int i = 0; i < nwrote; i++) stamp_page_as(wrote[i], g, back);
|
||||||
plat_mutex_unlock(&gen_lock);
|
plat_mutex_unlock(&gen_lock);
|
||||||
}
|
}
|
||||||
@@ -2184,7 +2191,7 @@ static void gpu_written(uint64_t addr, uint64_t size, int back) {
|
|||||||
uint64_t lo = addr & ~0xfffull, hi = (addr + size + 0xfff) & ~0xfffull;
|
uint64_t lo = addr & ~0xfffull, hi = (addr + size + 0xfff) & ~0xfffull;
|
||||||
back = back && wb_no_reup(); // read before gen_lock: its first call logs
|
back = back && wb_no_reup(); // read before gen_lock: its first call logs
|
||||||
plat_mutex_lock(&gen_lock);
|
plat_mutex_lock(&gen_lock);
|
||||||
uint32_t g = gen_next();
|
uint64_t g = gen_next();
|
||||||
for (uint64_t a = lo; a < hi; a += 1ull << GEN_PAGE_SHIFT) stamp_page_as(a, g, back);
|
for (uint64_t a = lo; a < hi; a += 1ull << GEN_PAGE_SHIFT) stamp_page_as(a, g, back);
|
||||||
plat_mutex_unlock(&gen_lock);
|
plat_mutex_unlock(&gen_lock);
|
||||||
}
|
}
|
||||||
@@ -2340,7 +2347,7 @@ void vk_mem_dirty_submit(void) {
|
|||||||
if (r->kind == POOL_IMPORT && r->bufbound) dlist[n++] = r;
|
if (r->kind == POOL_IMPORT && r->bufbound) dlist[n++] = r;
|
||||||
}
|
}
|
||||||
plat_mutex_lock(&gen_lock);
|
plat_mutex_lock(&gen_lock);
|
||||||
uint32_t g = gen_next(), since = dirty_gen;
|
uint64_t g = gen_next(), since = dirty_gen;
|
||||||
dirty_gen = g;
|
dirty_gen = g;
|
||||||
for (int i = 0; i < n;) { // one query per run of pools (sorted by address)
|
for (int i = 0; i < n;) { // one query per run of pools (sorted by address)
|
||||||
uint64_t lo = page_lo(dlist[i]), hi = page_hi(dlist[i]);
|
uint64_t lo = page_lo(dlist[i]), hi = page_hi(dlist[i]);
|
||||||
@@ -2361,7 +2368,7 @@ void vk_mem_dirty_submit(void) {
|
|||||||
int k = dc_class(r);
|
int k = dc_class(r);
|
||||||
ln[k]++, lb[k] += r->size;
|
ln[k]++, lb[k] += r->size;
|
||||||
int fresh = r->synced_gen == DIRTY_UNSYNCED, lz = !fresh && r->bound_sub == prev;
|
int fresh = r->synced_gen == DIRTY_UNSYNCED, lz = !fresh && r->bound_sub == prev;
|
||||||
uint32_t ls = r->synced_gen;
|
uint64_t ls = r->synced_gen;
|
||||||
if (fresh || r->bound_sub == prev) r->synced_gen = g;
|
if (fresh || r->bound_sub == prev) r->synced_gen = g;
|
||||||
if (r->untracked) { untracked += r->size; continue; }
|
if (r->untracked) { untracked += r->size; continue; }
|
||||||
if (fresh) continue; // counted as first use
|
if (fresh) continue; // counted as first use
|
||||||
@@ -2372,10 +2379,10 @@ void vk_mem_dirty_submit(void) {
|
|||||||
for (uint64_t a = lo; a < hi;) {
|
for (uint64_t a = lo; a < hi;) {
|
||||||
uint64_t ci = a >> GEN_CHUNK_SHIFT, ce = (ci + 1) << GEN_CHUNK_SHIFT;
|
uint64_t ci = a >> GEN_CHUNK_SHIFT, ce = (ci + 1) << GEN_CHUNK_SHIFT;
|
||||||
if (ce > hi) ce = hi;
|
if (ce > hi) ce = hi;
|
||||||
const uint32_t *ch = ci < GEN_CHUNKS ? gen_chunk[ci] : NULL;
|
const uint64_t *ch = ci < GEN_CHUNKS ? gen_chunk[ci] : NULL;
|
||||||
if (ch)
|
if (ch)
|
||||||
for (uint64_t x = a; x < ce; x += 1ull << GEN_PAGE_SHIFT) {
|
for (uint64_t x = a; x < ce; x += 1ull << GEN_PAGE_SHIFT) {
|
||||||
uint32_t v = ch[(x >> GEN_PAGE_SHIFT) & (GEN_PER_CHUNK - 1)];
|
uint64_t v = ch[(x >> GEN_PAGE_SHIFT) & (GEN_PER_CHUNK - 1)];
|
||||||
ne += v > since;
|
ne += v > since;
|
||||||
nl += lz && v > ls;
|
nl += lz && v > ls;
|
||||||
}
|
}
|
||||||
@@ -2955,7 +2962,7 @@ static void mirror_create(PoolRec *r) {
|
|||||||
mrep.sync_queries += snap_regions(r0, r1, mepoch);
|
mrep.sync_queries += snap_regions(r0, r1, mepoch);
|
||||||
int untracked = 0;
|
int untracked = 0;
|
||||||
for (uint64_t k = r0; k <= r1; k++) untracked |= snap_untracked[k];
|
for (uint64_t k = r0; k <= r1; k++) untracked |= snap_untracked[k];
|
||||||
r->mgen = gen_now; // the whole-range copy below holds every write observed so far
|
__atomic_store_n(&r->mgen, gen_now, __ATOMIC_RELEASE); // the whole-range copy below holds every write so far
|
||||||
plat_mutex_unlock(&gen_lock);
|
plat_mutex_unlock(&gen_lock);
|
||||||
if (untracked) return refuse(r, MR_UNTRACKED, "its range is not write-watched");
|
if (untracked) return refuse(r, MR_UNTRACKED, "its range is not write-watched");
|
||||||
VkBufferCreateInfo bi = {.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, .size = n, .usage = pool_usage(c),
|
VkBufferCreateInfo bi = {.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, .size = n, .usage = pool_usage(c),
|
||||||
@@ -3106,7 +3113,7 @@ static void mirror_sync(PoolRec *r) {
|
|||||||
plat_mutex_lock(&gen_lock);
|
plat_mutex_lock(&gen_lock);
|
||||||
mrep.sync_queries += snap_regions(r0, r1, mepoch);
|
mrep.sync_queries += snap_regions(r0, r1, mepoch);
|
||||||
}
|
}
|
||||||
uint32_t since = r->mgen;
|
uint64_t since = r->mgen;
|
||||||
int untracked = 0;
|
int untracked = 0;
|
||||||
for (uint64_t k = r0; k <= r1; k++) {
|
for (uint64_t k = r0; k <= r1; k++) {
|
||||||
untracked |= snap_untracked[k];
|
untracked |= snap_untracked[k];
|
||||||
@@ -3114,7 +3121,7 @@ static void mirror_sync(PoolRec *r) {
|
|||||||
uint64_t a = k << SNAP_SHIFT, b = a + (1ull << SNAP_SHIFT);
|
uint64_t a = k << SNAP_SHIFT, b = a + (1ull << SNAP_SHIFT);
|
||||||
if (a < lo) a = lo;
|
if (a < lo) a = lo;
|
||||||
if (b > hi) b = hi;
|
if (b > hi) b = hi;
|
||||||
const uint32_t *ch = cgen_chunk[a >> GEN_CHUNK_SHIFT]; // a 2 MiB region lies in one chunk
|
const uint64_t *ch = cgen_chunk[a >> GEN_CHUNK_SHIFT]; // a 2 MiB region lies in one chunk
|
||||||
if (!ch) continue;
|
if (!ch) continue;
|
||||||
for (uint64_t x = a; x < b; x += 1ull << GEN_PAGE_SHIFT)
|
for (uint64_t x = a; x < b; x += 1ull << GEN_PAGE_SHIFT)
|
||||||
if (ch[(x >> GEN_PAGE_SHIFT) & (GEN_PER_CHUNK - 1)] > since) {
|
if (ch[(x >> GEN_PAGE_SHIFT) & (GEN_PER_CHUNK - 1)] > since) {
|
||||||
@@ -3122,7 +3129,7 @@ static void mirror_sync(PoolRec *r) {
|
|||||||
r->st_pages += 1ull << GEN_PAGE_SHIFT, r->hot_bytes += 1ull << GEN_PAGE_SHIFT, r->hot_up += 1ull << GEN_PAGE_SHIFT;
|
r->st_pages += 1ull << GEN_PAGE_SHIFT, r->hot_bytes += 1ull << GEN_PAGE_SHIFT, r->hot_up += 1ull << GEN_PAGE_SHIFT;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
r->mgen = gen_now;
|
__atomic_store_n(&r->mgen, gen_now, __ATOMIC_RELEASE); // lazy_clean reads it without gen_lock
|
||||||
if (excl) plat_mutex_unlock(&gen_lock);
|
if (excl) plat_mutex_unlock(&gen_lock);
|
||||||
else plat_mutex_unlock_shared(&gen_lock);
|
else plat_mutex_unlock_shared(&gen_lock);
|
||||||
if (__atomic_load_n(&r->fl_n, __ATOMIC_ACQUIRE)) {
|
if (__atomic_load_n(&r->fl_n, __ATOMIC_ACQUIRE)) {
|
||||||
@@ -3181,7 +3188,7 @@ void vk_mem_shadow_table_read(void) {
|
|||||||
static int shadow_pool_clean(const PoolRec *r, uint32_t epoch, int cpu) {
|
static int shadow_pool_clean(const PoolRec *r, uint32_t epoch, int cpu) {
|
||||||
uint64_t r0 = r->import_lo >> SNAP_SHIFT, r1 = (r->import_hi - 1) >> SNAP_SHIFT;
|
uint64_t r0 = r->import_lo >> SNAP_SHIFT, r1 = (r->import_hi - 1) >> SNAP_SHIFT;
|
||||||
if (!epoch || r1 >= SNAP_REGIONS) return 0;
|
if (!epoch || r1 >= SNAP_REGIONS) return 0;
|
||||||
const uint32_t *mx = cpu ? creg_max : reg_max;
|
const uint64_t *mx = cpu ? creg_max : reg_max;
|
||||||
for (uint64_t k = r0; k <= r1; k++)
|
for (uint64_t k = r0; k <= r1; k++)
|
||||||
if (snap_at[k] != epoch || snap_untracked[k] || mx[k] > r->sgen) return 0;
|
if (snap_at[k] != epoch || snap_untracked[k] || mx[k] > r->sgen) return 0;
|
||||||
return 1;
|
return 1;
|
||||||
@@ -3543,7 +3550,7 @@ static void swb_run(uint64_t a, uint64_t b) {
|
|||||||
if (b <= a || !shadow_write_back(a, b - a) || nvn_gpu_only(a)) return;
|
if (b <= a || !shadow_write_back(a, b - a) || nvn_gpu_only(a)) return;
|
||||||
int back = wb_no_reup();
|
int back = wb_no_reup();
|
||||||
plat_mutex_lock(&gen_lock);
|
plat_mutex_lock(&gen_lock);
|
||||||
uint32_t g = gen_next();
|
uint64_t g = gen_next();
|
||||||
for (uint64_t x = a & ~0xfffull; x < b; x += 1ull << GEN_PAGE_SHIFT) stamp_page_as(x, g, back);
|
for (uint64_t x = a & ~0xfffull; x < b; x += 1ull << GEN_PAGE_SHIFT) stamp_page_as(x, g, back);
|
||||||
plat_mutex_unlock(&gen_lock);
|
plat_mutex_unlock(&gen_lock);
|
||||||
}
|
}
|
||||||
@@ -3755,9 +3762,9 @@ static void lazy_fetch(uint64_t h0, uint64_t h1, uint64_t off, uint64_t size) {
|
|||||||
// No lock: the pages are inaccessible, so nothing stamps them meanwhile; the mirror's mgen only grows.
|
// No lock: the pages are inaccessible, so nothing stamps them meanwhile; the mirror's mgen only grows.
|
||||||
static int lazy_clean(uint64_t lo, uint64_t hi, void *ctx) {
|
static int lazy_clean(uint64_t lo, uint64_t hi, void *ctx) {
|
||||||
const PoolRec *r = ctx;
|
const PoolRec *r = ctx;
|
||||||
uint32_t mgen = __atomic_load_n(&r->mgen, __ATOMIC_ACQUIRE);
|
uint64_t mgen = __atomic_load_n(&r->mgen, __ATOMIC_ACQUIRE);
|
||||||
for (uint64_t a = lo; a < hi; a += 1ull << GEN_PAGE_SHIFT) {
|
for (uint64_t a = lo; a < hi; a += 1ull << GEN_PAGE_SHIFT) {
|
||||||
const uint32_t *ch = __atomic_load_n(&cgen_chunk[a >> GEN_CHUNK_SHIFT], __ATOMIC_ACQUIRE);
|
const uint64_t *ch = __atomic_load_n(&cgen_chunk[a >> GEN_CHUNK_SHIFT], __ATOMIC_ACQUIRE);
|
||||||
if (ch && __atomic_load_n(&ch[(a >> GEN_PAGE_SHIFT) & (GEN_PER_CHUNK - 1)], __ATOMIC_RELAXED) > mgen) return 0;
|
if (ch && __atomic_load_n(&ch[(a >> GEN_PAGE_SHIFT) & (GEN_PER_CHUNK - 1)], __ATOMIC_RELAXED) > mgen) return 0;
|
||||||
}
|
}
|
||||||
uint64_t p[1];
|
uint64_t p[1];
|
||||||
@@ -4336,14 +4343,14 @@ void vk_mem_mirror_prologue(VkCommandBuffer cb) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// ---- verify mode
|
// ---- verify mode
|
||||||
typedef struct { PoolRec *r; uint64_t off, len, dst; uint32_t mgen; } VSample;
|
typedef struct { PoolRec *r; uint64_t off, len, dst, mgen; } VSample;
|
||||||
typedef struct { uint32_t sub; int n; VkBuffer buf; VmaAllocation alloc; const uint8_t *map; VSample s[VERIFY_SAMPLES]; } VJob;
|
typedef struct { uint32_t sub; int n; VkBuffer buf; VmaAllocation alloc; const uint8_t *map; VSample s[VERIFY_SAMPLES]; } VJob;
|
||||||
static int ver_i;
|
static int ver_i;
|
||||||
static uint64_t ver_off;
|
static uint64_t ver_off;
|
||||||
|
|
||||||
// The CPU wrote the page since the mirror's sync that mgen records: stamped later, or its write-watch bit is set.
|
// The CPU wrote the page since the mirror's sync that mgen records: stamped later, or its write-watch bit is set.
|
||||||
static int page_dirty(uint64_t a, uint32_t mgen) {
|
static int page_dirty(uint64_t a, uint64_t mgen) {
|
||||||
uint32_t *s = gen_slot(a, 0);
|
uint64_t *s = gen_slot(a, 0);
|
||||||
if (s && __atomic_load_n(s, __ATOMIC_RELAXED) > mgen) return 1;
|
if (s && __atomic_load_n(s, __ATOMIC_RELAXED) > mgen) return 1;
|
||||||
uint64_t p[1];
|
uint64_t p[1];
|
||||||
return mem_written_peek(a, 1ull << GEN_PAGE_SHIFT, p, 1) != 0; // UINT64_MAX (not watched) counts as written
|
return mem_written_peek(a, 1ull << GEN_PAGE_SHIFT, p, 1) != 0; // UINT64_MAX (not watched) counts as written
|
||||||
@@ -4382,13 +4389,15 @@ static void verify_cb(void *arg) { // completion thread
|
|||||||
rt_log("mem: verify mismatch: pool %#llx+%#llx (%s) offset %#llx: bytes %#x..%#x of the page differ, "
|
rt_log("mem: verify mismatch: pool %#llx+%#llx (%s) offset %#llx: bytes %#x..%#x of the page differ, "
|
||||||
"mirror %02x guest %02x at the first (submission %u)", (unsigned long long)r->lo,
|
"mirror %02x guest %02x at the first (submission %u)", (unsigned long long)r->lo,
|
||||||
(unsigned long long)r->size, f, (unsigned long long)(a - r->lo), k, e, m[k], g[k], j->sub);
|
(unsigned long long)r->size, f, (unsigned long long)(a - r->lo), k, e, m[k], g[k], j->sub);
|
||||||
const uint32_t *gs = gen_slot(a & ~((1ull << GEN_PAGE_SHIFT) - 1), 0);
|
const uint64_t *gs = gen_slot(a & ~((1ull << GEN_PAGE_SHIFT) - 1), 0);
|
||||||
char mh[100], gh[100];
|
char mh[100], gh[100];
|
||||||
for (int b = 0; b < 16; b++) snprintf(mh + 3 * b, 4, "%02x ", m[b]), snprintf(gh + 3 * b, 4, "%02x ", g[b]);
|
for (int b = 0; b < 16; b++) snprintf(mh + 3 * b, 4, "%02x ", m[b]), snprintf(gh + 3 * b, 4, "%02x ", g[b]);
|
||||||
rt_log("mem: verify mismatch detail: written binds %llu, last %u; used %u; written back %u; gpa record %u; "
|
rt_log("mem: verify mismatch detail: written binds %llu, last %u; used %u; written back %u; gpa record %u; "
|
||||||
"page cpu stamp %u (synced to %u, now %u); mirror %s| guest %s",
|
"page cpu stamp %llu (synced to %llu, now %llu); mirror %s| guest %s",
|
||||||
(unsigned long long)r->vd_wrote_n, r->vd_wrote_sub, r->vd_use_sub, wb, r->gpa_rec, gs ? *gs : 0,
|
(unsigned long long)r->vd_wrote_n, r->vd_wrote_sub, r->vd_use_sub, wb, r->gpa_rec,
|
||||||
s->mgen, gen_now, mh, gh);
|
(unsigned long long)(gs ? __atomic_load_n(gs, __ATOMIC_RELAXED) : 0),
|
||||||
|
(unsigned long long)s->mgen, (unsigned long long)__atomic_load_n(&gen_now, __ATOMIC_RELAXED),
|
||||||
|
mh, gh);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -6266,6 +6275,6 @@ void vk_mem_sizes(PerfSizes *s) {
|
|||||||
perf_size(s, "free staging arena chunks", arena_n, ARENA_KEEP, arena_n * ARENA_CHUNK);
|
perf_size(s, "free staging arena chunks", arena_n, ARENA_KEEP, arena_n * ARENA_CHUNK);
|
||||||
perf_size(s, "write-back snapshot pool", snap_n, SNAP_POOL_N, snap_b);
|
perf_size(s, "write-back snapshot pool", snap_n, SNAP_POOL_N, snap_b);
|
||||||
perf_size(s, "shadow base copies", sb, 0, sb * SB_PAGE);
|
perf_size(s, "shadow base copies", sb, 0, sb * SB_PAGE);
|
||||||
// the page generations handed out so far (gen_now, 32 bits, no wrap handling): how fast a session uses them up
|
// the page generations handed out so far (gen_now, 64 bits: they do not run out)
|
||||||
perf_size(s, "page generations handed out", (uint64_t)__atomic_load_n(&gen_now, __ATOMIC_RELAXED), 0, PERF_SIZE_NONE);
|
perf_size(s, "page generations handed out", (uint64_t)__atomic_load_n(&gen_now, __ATOMIC_RELAXED), 0, PERF_SIZE_NONE);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -53,7 +53,7 @@ typedef struct Tex {
|
|||||||
uint64_t blk_mem, blk_size; // where alloc lies (own_memory): the device memory handle and size of its block
|
uint64_t blk_mem, blk_size; // where alloc lies (own_memory): the device memory handle and size of its block
|
||||||
uint8_t blk_type, blk_own; // that memory's type; blk_own: memory of its own, not a part of one of VMA's blocks
|
uint8_t blk_type, blk_own; // that memory's type; blk_own: memory of its own, not a part of one of VMA's blocks
|
||||||
VmaAllocation bound_to; // the pool memory it is bound into
|
VmaAllocation bound_to; // the pool memory it is bound into
|
||||||
uint32_t wgen; // page generation of the last upload check
|
uint64_t wgen; // page generation of the last upload check
|
||||||
uint32_t checked; // submit_epoch of the last check from a draw binding
|
uint32_t checked; // submit_epoch of the last check from a draw binding
|
||||||
uint32_t use_round; // look_round when a lookup last reached it (touch)
|
uint32_t use_round; // look_round when a lookup last reached it (touch)
|
||||||
uint64_t use_frame; // gfx_counters.frames then
|
uint64_t use_frame; // gfx_counters.frames then
|
||||||
@@ -72,7 +72,7 @@ typedef struct Tex {
|
|||||||
int last_ch; // last vk_mem_changed result of make_fresh: 1 written, 0 clean, -1 not tracked
|
int last_ch; // last vk_mem_changed result of make_fresh: 1 written, 0 clean, -1 not tracked
|
||||||
struct Tex *next_at; // other entries at the same storage address
|
struct Tex *next_at; // other entries at the same storage address
|
||||||
uint8_t gpu_dirty; // mirrored, and the GPU wrote the image after the last write-back to the storage
|
uint8_t gpu_dirty; // mirrored, and the GPU wrote the image after the last write-back to the storage
|
||||||
uint32_t gpu_gen; // page generation when the GPU write that set gpu_dirty was recorded
|
uint64_t gpu_gen; // page generation when the GPU write that set gpu_dirty was recorded
|
||||||
uint64_t gpu_seq; // order of that GPU write among all of them (gpu_writes)
|
uint64_t gpu_seq; // order of that GPU write among all of them (gpu_writes)
|
||||||
uint64_t gpu_fence; // fence of the submission that made that write (0: not recorded in one)
|
uint64_t gpu_fence; // fence of the submission that made that write (0: not recorded in one)
|
||||||
uint64_t run_frame; // mirrored: the frame of the newest GPU write
|
uint64_t run_frame; // mirrored: the frame of the newest GPU write
|
||||||
@@ -1661,7 +1661,7 @@ static void rel_unlink(Tex *e) {
|
|||||||
// to 8,192 released entries, they stay listed, and since stays as it was at their release, so each one whose regions
|
// to 8,192 released entries, they stay listed, and since stays as it was at their release, so each one whose regions
|
||||||
// took a write later missed the region check at every submission from then on: 1,000 to 6,300 entries a check in
|
// took a write later missed the region check at every submission from then on: 1,000 to 6,300 entries a check in
|
||||||
// Kakariko and Castle Town, a third to a half of the list.
|
// Kakariko and Castle Town, a third to a half of the list.
|
||||||
typedef struct { Tex *e; uint64_t addr, size; uint32_t since; uint8_t ok, rel; } PEnt; // ok: uploaded as of since
|
typedef struct { Tex *e; uint64_t addr, size, since; uint8_t ok, rel; } PEnt; // ok: uploaded as of since
|
||||||
static PEnt *pent;
|
static PEnt *pent;
|
||||||
static int npent, cpent, *pq, cpq;
|
static int npent, cpent, *pq, cpq;
|
||||||
static uint64_t n_pool_checks, n_pool_skips, n_pool_rel;
|
static uint64_t n_pool_checks, n_pool_skips, n_pool_rel;
|
||||||
@@ -2069,7 +2069,7 @@ static int commit_drops_held(void) {
|
|||||||
// shutdown.
|
// shutdown.
|
||||||
static int wb_wanted(const Tex *e);
|
static int wb_wanted(const Tex *e);
|
||||||
static void wb_cfg_init(void);
|
static void wb_cfg_init(void);
|
||||||
static int storage_rewritten(const Tex *e, uint32_t since, uint32_t *now);
|
static int storage_rewritten(const Tex *e, uint64_t since, uint64_t *now);
|
||||||
static void write_back_n(Tex **v, int n);
|
static void write_back_n(Tex **v, int n);
|
||||||
static uint64_t n_drop_wb; // entries written back before their drop since the last perf line
|
static uint64_t n_drop_wb; // entries written back before their drop since the last perf line
|
||||||
|
|
||||||
@@ -2077,7 +2077,7 @@ static void drop_tex(Tex *e, int why) {
|
|||||||
flight_ev(e, FLIGHT_TEX_DROP, (uint32_t)why, 0);
|
flight_ev(e, FLIGHT_TEX_DROP, (uint32_t)why, 0);
|
||||||
if (why == DROP_COMMIT || why == DROP_IDLE_COMMIT) commit_drop_note(e);
|
if (why == DROP_COMMIT || why == DROP_IDLE_COMMIT) commit_drop_note(e);
|
||||||
if (e->gpu_dirty && why != DROP_OVERWRITTEN && why != DROP_SHUTDOWN) {
|
if (e->gpu_dirty && why != DROP_OVERWRITTEN && why != DROP_SHUTDOWN) {
|
||||||
uint32_t now;
|
uint64_t now;
|
||||||
wb_cfg_init();
|
wb_cfg_init();
|
||||||
if (storage_rewritten(e, e->gpu_gen, &now) == 1) e->gpu_dirty = 0;
|
if (storage_rewritten(e, e->gpu_gen, &now) == 1) e->gpu_dirty = 0;
|
||||||
else if (wb_wanted(e)) write_back_n(&e, 1), n_drop_wb++;
|
else if (wb_wanted(e)) write_back_n(&e, 1), n_drop_wb++;
|
||||||
@@ -2459,7 +2459,7 @@ static void sweep_hard(void) { // tex_lock held
|
|||||||
wb_cfg_init();
|
wb_cfg_init();
|
||||||
for (int i = 0; i < k; i++) {
|
for (int i = 0; i < k; i++) {
|
||||||
Tex *e = walk[out[i]];
|
Tex *e = walk[out[i]];
|
||||||
uint32_t t;
|
uint64_t t;
|
||||||
if (!e->gpu_dirty) continue;
|
if (!e->gpu_dirty) continue;
|
||||||
if (storage_rewritten(e, e->gpu_gen, &t) == 1) e->gpu_dirty = 0;
|
if (storage_rewritten(e, e->gpu_gen, &t) == 1) e->gpu_dirty = 0;
|
||||||
else if (wb_wanted(e)) wbv[m++] = e;
|
else if (wb_wanted(e)) wbv[m++] = e;
|
||||||
@@ -4089,7 +4089,7 @@ static volatile int32_t n_kept;
|
|||||||
// slots used in turn, and the CPU reads them a frame later. One slot's rect target shares its page with data the CPU
|
// slots used in turn, and the CPU reads them a frame later. One slot's rect target shares its page with data the CPU
|
||||||
// writes every frame: without the copy its renders were never written back, the game read a zero rect every other
|
// writes every frame: without the copy its renders were never written back, the game read a zero rect every other
|
||||||
// frame and the target's name label blinked. NX2NAVI_WB_EXACT=0 restores the page-only decision.
|
// frame and the target's name label blinked. NX2NAVI_WB_EXACT=0 restores the page-only decision.
|
||||||
static int storage_rewritten(const Tex *e, uint32_t since, uint32_t *now) {
|
static int storage_rewritten(const Tex *e, uint64_t since, uint64_t *now) {
|
||||||
uint64_t lo = (e->pub.addr + 0xfff) & ~0xfffull, hi = (e->pub.addr + e->t.L.size) & ~0xfffull;
|
uint64_t lo = (e->pub.addr + 0xfff) & ~0xfffull, hi = (e->pub.addr + e->t.L.size) & ~0xfffull;
|
||||||
if (hi <= lo) lo = e->pub.addr, hi = e->pub.addr + e->t.L.size;
|
if (hi <= lo) lo = e->pub.addr, hi = e->pub.addr + e->t.L.size;
|
||||||
int ch = vk_mem_changed(lo, hi - lo, since, now);
|
int ch = vk_mem_changed(lo, hi - lo, since, now);
|
||||||
@@ -4131,7 +4131,7 @@ static void make_fresh(Tex *e) {
|
|||||||
upload(e);
|
upload(e);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
uint32_t now;
|
uint64_t now;
|
||||||
int ch = vk_mem_changed(e->pub.addr, size, e->wgen, &now);
|
int ch = vk_mem_changed(e->pub.addr, size, e->wgen, &now);
|
||||||
e->wgen = now;
|
e->wgen = now;
|
||||||
e->last_ch = ch;
|
e->last_ch = ch;
|
||||||
@@ -4144,7 +4144,7 @@ static void make_fresh(Tex *e) {
|
|||||||
static void make_fresh_snap(Tex *e) {
|
static void make_fresh_snap(Tex *e) {
|
||||||
if (!from_storage(e) || e->pub.gpu_owned || (e->sfill && e->released)) return;
|
if (!from_storage(e) || e->pub.gpu_owned || (e->sfill && e->released)) return;
|
||||||
if (!e->uploaded && up_wait(e)) return;
|
if (!e->uploaded && up_wait(e)) return;
|
||||||
uint32_t now;
|
uint64_t now;
|
||||||
n_checks++;
|
n_checks++;
|
||||||
int ch = vk_mem_changed_snap(e->pub.addr, e->t.L.size, e->uploaded ? e->wgen : 0, submit_epoch, &now);
|
int ch = vk_mem_changed_snap(e->pub.addr, e->t.L.size, e->uploaded ? e->wgen : 0, submit_epoch, &now);
|
||||||
e->wgen = now;
|
e->wgen = now;
|
||||||
@@ -4190,7 +4190,7 @@ static int alias_compatible(const Tex *e, const Tex *src, const char **why) {
|
|||||||
static int alias_has_gpu_content(const Tex *src) {
|
static int alias_has_gpu_content(const Tex *src) {
|
||||||
if (!src->pub.gpu_owned) return 0;
|
if (!src->pub.gpu_owned) return 0;
|
||||||
if (!src->pub.mirrored) return 1;
|
if (!src->pub.mirrored) return 1;
|
||||||
uint32_t now;
|
uint64_t now;
|
||||||
return src->gpu_dirty && !storage_rewritten(src, src->gpu_gen, &now);
|
return src->gpu_dirty && !storage_rewritten(src, src->gpu_gen, &now);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -4360,7 +4360,7 @@ static void start_at(Tex *e, const TexKey *k, int taken_back) {
|
|||||||
// split fill: e is to start with its storage's bytes; msrc: the released CPU-written entry whose image holds them
|
// split fill: e is to start with its storage's bytes; msrc: the released CPU-written entry whose image holds them
|
||||||
// (destroyed after the copy), msrc_gpu: it holds what the GPU wrote instead, msrc_gen: the storage's generation
|
// (destroyed after the copy), msrc_gpu: it holds what the GPU wrote instead, msrc_gen: the storage's generation
|
||||||
int fill = !taken_back && tex_alias && split_wants(&e->t, &e->f), msrc_gpu = 0;
|
int fill = !taken_back && tex_alias && split_wants(&e->t, &e->f), msrc_gpu = 0;
|
||||||
uint32_t msrc_gen = 0;
|
uint64_t msrc_gen = 0;
|
||||||
Tex *msrc = NULL;
|
Tex *msrc = NULL;
|
||||||
const char *why = "no live texture with GPU content"; // for the log of an alias left empty
|
const char *why = "no live texture with GPU content"; // for the log of an alias left empty
|
||||||
const Tex *other = NULL;
|
const Tex *other = NULL;
|
||||||
@@ -4381,7 +4381,7 @@ static void start_at(Tex *e, const TexKey *k, int taken_back) {
|
|||||||
}
|
}
|
||||||
gone++;
|
gone++;
|
||||||
const char *no;
|
const char *no;
|
||||||
uint32_t gen = 0;
|
uint64_t gen = 0;
|
||||||
int gpu = alias_has_gpu_content(p);
|
int gpu = alias_has_gpu_content(p);
|
||||||
if (fill && !msrc && (gpu || (p->uploaded && !p->pub.gpu_owned &&
|
if (fill && !msrc && (gpu || (p->uploaded && !p->pub.gpu_owned &&
|
||||||
!vk_mem_changed(p->pub.addr, p->t.L.size, p->wgen, &gen))) &&
|
!vk_mem_changed(p->pub.addr, p->t.L.size, p->wgen, &gen))) &&
|
||||||
@@ -4588,7 +4588,7 @@ static Tex *entry_of(const tex_t *t) {
|
|||||||
void vk_texture_debug(const VkTexImage *img, uint32_t *uploads, int *last_change, int *pages_written) {
|
void vk_texture_debug(const VkTexImage *img, uint32_t *uploads, int *last_change, int *pages_written) {
|
||||||
lock();
|
lock();
|
||||||
const Tex *e = (const Tex *)img; // pub is the first member
|
const Tex *e = (const Tex *)img; // pub is the first member
|
||||||
uint32_t now;
|
uint64_t now;
|
||||||
*uploads = e->n_up;
|
*uploads = e->n_up;
|
||||||
*last_change = e->last_ch;
|
*last_change = e->last_ch;
|
||||||
*pages_written = e->pub.mirrored && e->uploaded ? vk_mem_changed(e->pub.addr, e->t.L.size, e->wgen, &now) : 0;
|
*pages_written = e->pub.mirrored && e->uploaded ? vk_mem_changed(e->pub.addr, e->t.L.size, e->wgen, &now) : 0;
|
||||||
@@ -5705,7 +5705,7 @@ void vk_texture_write_back(uint64_t addr, uint64_t size) {
|
|||||||
if (!hit) rt_fatal("vk: out of memory for a texture write-back");
|
if (!hit) rt_fatal("vk: out of memory for a texture write-back");
|
||||||
int n = owned_dirty(addr, size, hit), k = 0; // owned[] cannot change while tex_lock is held
|
int n = owned_dirty(addr, size, hit), k = 0; // owned[] cannot change while tex_lock is held
|
||||||
for (int i = 0; i < n; i++) {
|
for (int i = 0; i < n; i++) {
|
||||||
uint32_t now;
|
uint64_t now;
|
||||||
if (storage_rewritten(hit[i], hit[i]->gpu_gen, &now) == 1) hit[i]->gpu_dirty = 0;
|
if (storage_rewritten(hit[i], hit[i]->gpu_gen, &now) == 1) hit[i]->gpu_dirty = 0;
|
||||||
else hit[k++] = hit[i];
|
else hit[k++] = hit[i];
|
||||||
}
|
}
|
||||||
@@ -5778,7 +5778,7 @@ void vk_texture_write_back_small(uint64_t fence) {
|
|||||||
if (later) __atomic_store_n(&small_pending, 1, __ATOMIC_RELEASE);
|
if (later) __atomic_store_n(&small_pending, 1, __ATOMIC_RELEASE);
|
||||||
int k = 0;
|
int k = 0;
|
||||||
for (int i = 0; i < n; i++) {
|
for (int i = 0; i < n; i++) {
|
||||||
uint32_t now;
|
uint64_t now;
|
||||||
if (storage_rewritten(hit[i], hit[i]->gpu_gen, &now) == 1) hit[i]->gpu_dirty = 0;
|
if (storage_rewritten(hit[i], hit[i]->gpu_gen, &now) == 1) hit[i]->gpu_dirty = 0;
|
||||||
else hit[k++] = hit[i];
|
else hit[k++] = hit[i];
|
||||||
}
|
}
|
||||||
|
|||||||
71
tests/gen_width_check.py
Normal file
71
tests/gen_width_check.py
Normal file
@@ -0,0 +1,71 @@
|
|||||||
|
"""No 64-to-32-bit narrowing in the files that hold page write generations: runtime/vk_memory.c, runtime/vk_texture.c
|
||||||
|
and runtime/tex_probe.c, compiled with the build's flags plus -Wshorten-64-to-32, -Wformat and
|
||||||
|
-Wincompatible-pointer-types as errors (clang -fsyntax-only). The generations are 64 bits (vk_memory.c "page write
|
||||||
|
generations"); one local, field or format left at 32 bits would bring the wrap of 2^32 back without a word, and the
|
||||||
|
normal build sees a narrowing only as a silent conversion. The baseline is zero such diagnostics in the whole of the
|
||||||
|
three files: a narrowing elsewhere in them takes an explicit cast, which also says it is meant.
|
||||||
|
|
||||||
|
Needs clang, the Vulkan headers, VMA and the shader headers of tools/build_shaders.py (runtime/*_spv.h). No GPU, no
|
||||||
|
game files, any Python.
|
||||||
|
|
||||||
|
Usage: gen_width_check.py [--cc <clang>]
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
sys.path.insert(0, str(ROOT / 'tools'))
|
||||||
|
import build_game # noqa: E402 (RT_FLAGS, the compiler and the include folders of the build)
|
||||||
|
|
||||||
|
FILES = ['vk_memory.c', 'vk_texture.c', 'tex_probe.c']
|
||||||
|
STRICT = ['-Werror=shorten-64-to-32', '-Werror=format', '-Werror=incompatible-pointer-types']
|
||||||
|
# the diagnostics this check is about, by the flag clang names at the end of the line
|
||||||
|
WATCHED = re.compile(r'^(?P<where>[^\n]*?:\d+:\d+): (?:error|warning): (?P<what>[^\n]*?)'
|
||||||
|
r'\[(?:-Werror,)?(?P<flag>-Wshorten-64-to-32|-Wformat[\w-]*|-Wincompatible[\w-]*pointer-types)\]',
|
||||||
|
re.M)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument('--cc', help='the C compiler (default: the build\'s clang)')
|
||||||
|
args = ap.parse_args()
|
||||||
|
if not (ROOT / 'runtime' / 'wb_merge_spv.h').is_file():
|
||||||
|
print('gen_width_check: FAIL: runtime/wb_merge_spv.h is missing: run tools/build_shaders.py first')
|
||||||
|
return 1
|
||||||
|
cc = args.cc or os.environ.get('CC') or str(build_game.CLANG)
|
||||||
|
build_game.set_cpu('v3') # adds -DRT_CPU_LEVEL=3, as the build does
|
||||||
|
flags = [f for f in build_game.RT_FLAGS if f != '-g']
|
||||||
|
inc = ['-I', str(ROOT / 'runtime'), '-I', str(build_game.vma_include())]
|
||||||
|
vk = build_game.vulkan_include(cc)
|
||||||
|
if vk:
|
||||||
|
inc = ['-I', str(vk)] + inc
|
||||||
|
env = dict(os.environ)
|
||||||
|
if os.name == 'nt':
|
||||||
|
env['PATH'] = str(Path(cc).parent) + os.pathsep + env.get('PATH', '')
|
||||||
|
bad = 0
|
||||||
|
for name in FILES:
|
||||||
|
src = ROOT / 'runtime' / name
|
||||||
|
p = subprocess.run([cc, '-fsyntax-only', '-std=gnu11'] + flags + STRICT + inc + [str(src)], capture_output=True,
|
||||||
|
text=True, errors='replace', env=env)
|
||||||
|
hits = [m for m in WATCHED.finditer(p.stderr)]
|
||||||
|
for m in hits:
|
||||||
|
print(f'gen_width_check: FAIL: {m.group("where")}: {m.group("what").strip()} [{m.group("flag")}]')
|
||||||
|
if p.returncode and not hits:
|
||||||
|
print(f'gen_width_check: FAIL: runtime/{name} does not compile:\n{p.stderr[-3000:]}')
|
||||||
|
bad += 1
|
||||||
|
bad += len(hits)
|
||||||
|
if not hits and not p.returncode:
|
||||||
|
print(f'gen_width_check: runtime/{name}: no narrowing, format or pointer-type diagnostic')
|
||||||
|
if bad:
|
||||||
|
print(f'gen_width_check: {bad} problems')
|
||||||
|
return 1
|
||||||
|
print('gen_width_check: all checks passed')
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
sys.exit(main())
|
||||||
@@ -1,11 +1,14 @@
|
|||||||
// runtime/vk_memory.c's page write generations (the section from its "page write generations" banner to its "dirty
|
// runtime/vk_memory.c's page write generations across a boundary: the section from its "page write generations" banner
|
||||||
// census" banner, extracted by tests/gen_wrap_check.py into gen_section.inc): the lines gen_next logs as the 32-bit
|
// to its "dirty census" banner and the readers of its tables outside it (cpu_wrote_since, shadow_pool_clean,
|
||||||
// generations near their wrap, and what the comparisons answer across it. Prints each failed check and exits 1 on any.
|
// lazy_clean, page_dirty), extracted by tests/gen_wrap_check.py into gen_section.inc, start just below a seed's
|
||||||
// Stubs: no write-watch (mem_written_pages reports no page), nothing frozen, no pools.
|
// boundary and run across it. Every compare the texture checks, the snapshot path, the VRAM mirrors' lazy fetch, the
|
||||||
//
|
// shadow storage and the verify mode make must answer as it does below the boundary. Seeds: 2^32 - 8 (where 32-bit
|
||||||
// The comparisons across the wrap are wrong today (AUDIT F-8-01: the generations have no wrap handling; its fix waits
|
// generations wrapped), 2^40 - 8 (the low 32 bits pass through 0 again, so a truncation to 32 bits anywhere fails
|
||||||
// for approval). The check expects the four wrong answers and says so: the change that handles the wrap turns them
|
// there too) and 2^63 - 8. gen_next must log one line when the count reaches 2^32, and no other.
|
||||||
// into failures here, and flips the expectations with it.
|
// Stubs: no-op locks; a write watch that reports the pages a check marks as written by the CPU; nothing frozen; no
|
||||||
|
// pools in the index.
|
||||||
|
// Usage: gen_wrap_check <seed>. Prints each failed check and "gen_wrap_check: seed ...: N of M checks failed"; exit 1
|
||||||
|
// on any.
|
||||||
#include <stdarg.h>
|
#include <stdarg.h>
|
||||||
#include <stdint.h>
|
#include <stdint.h>
|
||||||
#include <stdio.h>
|
#include <stdio.h>
|
||||||
@@ -30,11 +33,36 @@ static uint64_t kern_now_ns(void) { return fake_now; }
|
|||||||
static void grow(void **arr, int *cap, size_t elem) {
|
static void grow(void **arr, int *cap, size_t elem) {
|
||||||
int n = *cap ? *cap * 2 : 64;
|
int n = *cap ? *cap * 2 : 64;
|
||||||
*arr = realloc(*arr, (size_t)n * elem);
|
*arr = realloc(*arr, (size_t)n * elem);
|
||||||
|
if (!*arr) exit(70);
|
||||||
*cap = n;
|
*cap = n;
|
||||||
}
|
}
|
||||||
|
#define SB_PAGE 4096ull
|
||||||
|
|
||||||
|
// The write watch: pages the CPU wrote since the last query over them (mem_written_pages resets them,
|
||||||
|
// mem_written_peek does not).
|
||||||
|
static uint64_t ww_pend[64];
|
||||||
|
static int nww_pend;
|
||||||
|
static void cpu_write(uint64_t page) { ww_pend[nww_pend++] = page & ~0xfffull; }
|
||||||
|
static uint64_t ww_take(uint64_t addr, uint64_t size, uint64_t *pages, uint64_t max, int reset) {
|
||||||
|
uint64_t n = 0;
|
||||||
|
for (int i = 0; i < nww_pend && n < max;) {
|
||||||
|
if (ww_pend[i] >= addr && ww_pend[i] < addr + size) {
|
||||||
|
if (pages) pages[n] = ww_pend[i];
|
||||||
|
n++;
|
||||||
|
if (reset) {
|
||||||
|
ww_pend[i] = ww_pend[--nww_pend];
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
i++;
|
||||||
|
}
|
||||||
|
return n;
|
||||||
|
}
|
||||||
uint64_t mem_written_pages(uint64_t addr, uint64_t size, uint64_t *pages, uint64_t max) {
|
uint64_t mem_written_pages(uint64_t addr, uint64_t size, uint64_t *pages, uint64_t max) {
|
||||||
(void)addr, (void)size, (void)pages, (void)max;
|
return ww_take(addr, size, pages, max, 1);
|
||||||
return 0;
|
}
|
||||||
|
uint64_t mem_written_peek(uint64_t addr, uint64_t size, uint64_t *pages, uint64_t max) {
|
||||||
|
return ww_take(addr, size, pages, max, 0);
|
||||||
}
|
}
|
||||||
int mem_frozen(uint64_t a) { (void)a; return 0; }
|
int mem_frozen(uint64_t a) { (void)a; return 0; }
|
||||||
int mem_watched(uint64_t addr, uint64_t size) { (void)addr, (void)size; return 1; }
|
int mem_watched(uint64_t addr, uint64_t size) { (void)addr, (void)size; return 1; }
|
||||||
@@ -46,9 +74,8 @@ static pidx pool_index;
|
|||||||
static PlatMutex pool_lock;
|
static PlatMutex pool_lock;
|
||||||
static const pidx_ent *pidx_find(const pidx *x, uintptr_t a, uint64_t l) { (void)x, (void)a, (void)l; return NULL; }
|
static const pidx_ent *pidx_find(const pidx *x, uintptr_t a, uint64_t l) { (void)x, (void)a, (void)l; return NULL; }
|
||||||
enum { POOL_IMPORT = 1 };
|
enum { POOL_IMPORT = 1 };
|
||||||
typedef struct { int kind; void *buf; uint64_t import_lo, import_hi; } PoolRec;
|
|
||||||
|
|
||||||
// The log: the generation lines are kept, the others dropped.
|
// The log: the page generation lines are kept, the others dropped.
|
||||||
static char gen_lines[16][512];
|
static char gen_lines[16][512];
|
||||||
static int n_gen_lines;
|
static int n_gen_lines;
|
||||||
static void rt_log(const char *fmt, ...) {
|
static void rt_log(const char *fmt, ...) {
|
||||||
@@ -73,73 +100,149 @@ static void rt_fatal(const char *fmt, ...) {
|
|||||||
|
|
||||||
#include "gen_section.inc"
|
#include "gen_section.inc"
|
||||||
|
|
||||||
static int fails;
|
typedef __typeof__(gen_now) gen_t; // the width the source gives the generations
|
||||||
#define CHECK(c, ...) do { if (!(c)) { fails++; printf("FAIL line %d: ", __LINE__); printf(__VA_ARGS__); printf("\n"); } } while (0)
|
|
||||||
|
static int checks, failed;
|
||||||
|
#define CHECK(cond, ...) \
|
||||||
|
do { \
|
||||||
|
checks++; \
|
||||||
|
if (!(cond)) failed++, printf("FAIL line %d: ", __LINE__), printf(__VA_ARGS__), printf("\n"); \
|
||||||
|
} while (0)
|
||||||
|
|
||||||
#define TEX 0x100000000ull // a texture of 4 pages in one 2 MiB region
|
#define TEX 0x100000000ull // a texture of 4 pages in one 2 MiB region
|
||||||
#define OTHER 0x100200000ull // a second one in the next region
|
#define OTHER 0x100200000ull // a second one in the next region
|
||||||
|
#define SPIN 0x200000000ull // an unrelated range other checks hand out generations for
|
||||||
|
#define LONE 0x100400000ull // a page written exactly at the boundary
|
||||||
|
|
||||||
// Hands out generations with vk_guest_written from just below `to` until gen_now reaches `to` + 3.
|
static uint64_t last_now; // every *now handed back, in order: must grow
|
||||||
static void walk_to(uint32_t to) {
|
static void seen(gen_t now, const char *what) {
|
||||||
gen_now = to - 4;
|
CHECK((uint64_t)now > last_now, "%s: generation %llu after %llu: not newer", what, (unsigned long long)now,
|
||||||
for (int i = 0; i < 7; i++) vk_guest_written(0x300000000ull, 0x1000);
|
(unsigned long long)last_now);
|
||||||
|
last_now = now;
|
||||||
}
|
}
|
||||||
|
|
||||||
int main(void) {
|
int main(int argc, char **argv) {
|
||||||
// ---- the lines: one at each of 0xC0000000, 0xD0000000, 0xE0000000, 0xF0000000 and at the wrap, no other
|
uint64_t seed = argc > 1 ? strtoull(argv[1], NULL, 0) : (1ull << 32) - 8;
|
||||||
|
uint64_t edge = seed + 8; // the boundary under test: 2^32 for the first seed
|
||||||
|
gen_t w_tex, w_oth, now, d;
|
||||||
rt_guest_start_ns = 1000000000ull;
|
rt_guest_start_ns = 1000000000ull;
|
||||||
fake_now = rt_guest_start_ns + 3600ull * 1000000000ull; // an hour in
|
fake_now = rt_guest_start_ns + 3600ull * 1000000000ull; // an hour in
|
||||||
gfx_counters.frames = 216000; // at 60 fps
|
gfx_counters.frames = 216000; // at 60 fps
|
||||||
walk_to(0x10000000u); // below the first mark: nothing
|
gen_now = (gen_t)seed;
|
||||||
walk_to(0xb0000000u);
|
CHECK((uint64_t)gen_now == seed, "the seed %#llx does not fit the generations (%u bits)", (unsigned long long)seed,
|
||||||
CHECK(n_gen_lines == 0, "%d lines before 0xC0000000", n_gen_lines);
|
(unsigned)(8 * sizeof(gen_t)));
|
||||||
static const uint32_t marks[] = {0xc0000000u, 0xd0000000u, 0xe0000000u, 0xf0000000u};
|
if (failed) { // the generations cannot reach the boundary: nothing below can run
|
||||||
for (int k = 0; k < 4; k++) {
|
printf("gen_wrap_check: seed %#llx: 1 of 1 checks failed\n", (unsigned long long)seed);
|
||||||
int was = n_gen_lines;
|
return 1;
|
||||||
walk_to(marks[k]);
|
|
||||||
char want[32];
|
|
||||||
snprintf(want, sizeof want, ": %u of 4294967295", marks[k]);
|
|
||||||
CHECK(n_gen_lines == was + 1 && strstr(gen_lines[was], want), "the mark at %#x gave %d lines (%s)", marks[k],
|
|
||||||
n_gen_lines - was, n_gen_lines > was ? gen_lines[was] : "none");
|
|
||||||
}
|
}
|
||||||
// 0xC0000000 in an hour of 216,000 frames: about 14,913 a frame, and a quarter of the range left: 20 min
|
last_now = (uint64_t)gen_now;
|
||||||
CHECK(n_gen_lines >= 1 && strstr(gen_lines[0], "(14913 a frame)") && strstr(gen_lines[0], "in about 0.3 h"),
|
|
||||||
"the first line's rate or hours: %s", n_gen_lines ? gen_lines[0] : "none");
|
|
||||||
int was = n_gen_lines;
|
|
||||||
walk_to(0); // across the wrap
|
|
||||||
CHECK(n_gen_lines == was + 1 && strstr(gen_lines[was], "ran out at frame 216000 after 1.0 h"),
|
|
||||||
"the wrap gave %d lines (%s)", n_gen_lines - was, n_gen_lines > was ? gen_lines[was] : "none");
|
|
||||||
walk_to(0x20000000u);
|
|
||||||
CHECK(n_gen_lines == 5, "%d lines in all, not 5", n_gen_lines);
|
|
||||||
for (int i = 0; i < n_gen_lines && i < 16; i++) printf(" %s\n", gen_lines[i]);
|
|
||||||
|
|
||||||
// ---- the comparisons across the wrap: the four answers known to be wrong (expected, see the top)
|
// Before the boundary: both textures loaded (written) and checked.
|
||||||
int known = 0;
|
|
||||||
gen_now = 999;
|
|
||||||
vk_guest_written(TEX, 0x4000);
|
vk_guest_written(TEX, 0x4000);
|
||||||
vk_guest_written(OTHER, 0x4000);
|
vk_guest_written(OTHER, 0x4000);
|
||||||
uint32_t w1, w2, now;
|
CHECK(vk_mem_changed(TEX, 0x4000, 0, &w_tex) == 1, "TEX: its load is not seen");
|
||||||
vk_mem_changed(TEX, 0x4000, 0, &w1);
|
seen(w_tex, "TEX first check");
|
||||||
vk_mem_changed(OTHER, 0x4000, 0, &w2);
|
CHECK(vk_mem_changed(OTHER, 0x4000, 0, &w_oth) == 1, "OTHER: its load is not seen");
|
||||||
gen_now = UINT32_MAX - 6;
|
seen(w_oth, "OTHER first check");
|
||||||
CHECK(vk_mem_changed(TEX, 0x4000, w1, &w1) == 0, "an unwritten texture read as written before the wrap");
|
uint64_t pre = (uint64_t)gen_now; // a generation before the boundary: what a mirror or shadow last synced to
|
||||||
vk_mem_changed(OTHER, 0x4000, w2, &w2);
|
|
||||||
for (int i = 0; i < 10; i++) vk_mem_changed(0x200000000ull, 0x1000, 0, &now); // other checks use generations
|
|
||||||
vk_guest_written(TEX, 0x1000); // the CPU writes the texture after the wrap
|
|
||||||
known += vk_mem_changed(TEX, 0x4000, w1, &now) != 1; // 1. the write is missed
|
|
||||||
vk_mem_changed(OTHER, 0x4000, w2, &w2);
|
|
||||||
known += vk_mem_changed(OTHER, 0x4000, w2, &now) == 1; // 2. an unwritten texture reads as written at every check
|
|
||||||
w2 = now;
|
|
||||||
known += vk_mem_changed_snap(OTHER, 0x4000, w2, 7, &now) == 1; // 3. the same through the snapshot path
|
|
||||||
gen_now = UINT32_MAX;
|
|
||||||
vk_guest_written(TEX + 0x2000, 0x1000); // 4. a write stamped at the wrap gets 0, which reads as never stamped
|
|
||||||
known += vk_mem_changed(TEX + 0x2000, 0x1000, 0, &now) != 1;
|
|
||||||
CHECK(known == 4, "%d of the 4 known wrong answers across the wrap remain: if the wrap is handled now, flip these "
|
|
||||||
"expectations", known);
|
|
||||||
printf("gen_wrap_check: across the wrap %d of 4 comparisons still answer wrong (known, expected until the wrap "
|
|
||||||
"is handled)\n", known);
|
|
||||||
|
|
||||||
if (fails) return printf("gen_wrap_check: %d checks failed\n", fails), 1;
|
// Other checks hand out generations up to edge - 1; the page written next is stamped exactly at edge.
|
||||||
printf("gen_wrap_check: all checks passed\n");
|
while ((uint64_t)gen_now + 1 < edge) {
|
||||||
return 0;
|
vk_mem_changed(SPIN, 0x1000, 0, &d);
|
||||||
|
seen(d, "spin");
|
||||||
|
}
|
||||||
|
vk_guest_written(LONE, 0x1000);
|
||||||
|
gen_t *ls = gen_slot(LONE, 0);
|
||||||
|
CHECK(ls && (uint64_t)*ls == edge, "the page written at the boundary is stamped %llu, not %llu",
|
||||||
|
ls ? (unsigned long long)*ls : 0ull, (unsigned long long)edge);
|
||||||
|
CHECK(vk_mem_changed(LONE, 0x1000, 0, &now) == 1, "the page written at the boundary reads as never written");
|
||||||
|
seen(now, "LONE");
|
||||||
|
CHECK(vk_mem_changed(LONE, 0x1000, (gen_t)(edge - 1), &now) == 1, "the boundary's write is older than edge - 1");
|
||||||
|
seen(now, "LONE again");
|
||||||
|
for (int i = 0; i < 6; i++) {
|
||||||
|
vk_mem_changed(SPIN, 0x1000, 0, &d);
|
||||||
|
seen(d, "spin past");
|
||||||
|
}
|
||||||
|
|
||||||
|
// After the boundary: a CPU write found by the write watch and one by a host path are both seen.
|
||||||
|
cpu_write(TEX);
|
||||||
|
CHECK(vk_mem_changed(TEX, 0x4000, w_tex, &now) == 1, "TEX: a CPU write after the boundary is missed");
|
||||||
|
seen(now, "TEX after a CPU write");
|
||||||
|
w_tex = now;
|
||||||
|
vk_guest_written(TEX + 0x1000, 4);
|
||||||
|
CHECK(vk_mem_changed(TEX, 0x4000, w_tex, &now) == 1, "TEX: a host write after the boundary is missed");
|
||||||
|
seen(now, "TEX after a host write");
|
||||||
|
w_tex = now;
|
||||||
|
CHECK(vk_mem_changed(TEX, 0x4000, w_tex, &now) == 0, "TEX: no write since its check, but it reads as written");
|
||||||
|
seen(now, "TEX unwritten");
|
||||||
|
w_tex = now;
|
||||||
|
|
||||||
|
// A texture nobody wrote since before the boundary stays unchanged, check after check.
|
||||||
|
CHECK(vk_mem_changed(OTHER, 0x4000, w_oth, &now) == 0,
|
||||||
|
"OTHER: unwritten since before the boundary, reads as written");
|
||||||
|
seen(now, "OTHER 1");
|
||||||
|
w_oth = now;
|
||||||
|
CHECK(vk_mem_changed(OTHER, 0x4000, w_oth, &now) == 0, "OTHER: unwritten, reads as written at the next check too");
|
||||||
|
seen(now, "OTHER 2");
|
||||||
|
w_oth = now;
|
||||||
|
|
||||||
|
// The snapshot path (draw bindings and the texture pool's check) and the region check (reg_max).
|
||||||
|
uint32_t ep = 7;
|
||||||
|
CHECK(vk_mem_changed_snap(OTHER, 0x4000, w_oth, ep, &now) == 0, "OTHER via the snapshot: reads as written");
|
||||||
|
seen(now, "OTHER snap");
|
||||||
|
CHECK(vk_mem_unchanged_why_locked(OTHER, 0x4000, w_oth, ep) == 0, "OTHER's region check answers %d, not 0",
|
||||||
|
vk_mem_unchanged_why_locked(OTHER, 0x4000, w_oth, ep));
|
||||||
|
cpu_write(OTHER + 0x2000);
|
||||||
|
ep = 8;
|
||||||
|
CHECK(vk_mem_changed_snap(OTHER, 0x4000, w_oth, ep, &now) == 1, "OTHER via the snapshot: a CPU write is missed");
|
||||||
|
seen(now, "OTHER snap written");
|
||||||
|
CHECK(vk_mem_unchanged_why_locked(OTHER, 0x4000, w_oth, ep) == 3,
|
||||||
|
"OTHER's region check answers %d, not 3 (written)",
|
||||||
|
vk_mem_unchanged_why_locked(OTHER, 0x4000, w_oth, ep));
|
||||||
|
CHECK(vk_mem_unchanged_why_locked(OTHER, 0x4000, now, ep) == 0, "OTHER's region check after its check answers %d",
|
||||||
|
vk_mem_unchanged_why_locked(OTHER, 0x4000, now, ep));
|
||||||
|
|
||||||
|
// The CPU-side tables (what mirror_sync, lazy_clean and cpu_wrote_since read) hold the same stamps.
|
||||||
|
gen_t *gs = gen_slot(TEX + 0x1000, 0), *cs = slot_in(cgen_chunk, TEX + 0x1000, 0);
|
||||||
|
CHECK(gs && cs && *gs == *cs && (uint64_t)*cs > edge, "TEX page 1: texture stamp %llu, CPU-side stamp %llu",
|
||||||
|
gs ? (unsigned long long)*gs : 0ull, cs ? (unsigned long long)*cs : 0ull);
|
||||||
|
CHECK((uint64_t)creg_max[TEX >> REG_SHIFT] > edge && reg_max[TEX >> REG_SHIFT] == creg_max[TEX >> REG_SHIFT],
|
||||||
|
"TEX's region: newest stamps %llu / %llu", (unsigned long long)reg_max[TEX >> REG_SHIFT],
|
||||||
|
(unsigned long long)creg_max[TEX >> REG_SHIFT]);
|
||||||
|
|
||||||
|
// A VRAM mirror and a shadow copy synced before the boundary, TEX written after it.
|
||||||
|
PoolRec p = {.kind = POOL_IMPORT, .import_lo = TEX, .import_hi = TEX + 0x4000};
|
||||||
|
p.mgen = (__typeof__(p.mgen))pre, p.sgen = (__typeof__(p.sgen))pre;
|
||||||
|
CHECK((uint64_t)p.mgen == pre && (uint64_t)p.sgen == pre, "PoolRec's mgen and sgen do not hold %llu",
|
||||||
|
(unsigned long long)pre);
|
||||||
|
CHECK(!lazy_clean(TEX, TEX + 0x4000, &p), "lazy_clean: the mirror lacks a CPU write after the boundary, yet clean");
|
||||||
|
CHECK(cpu_wrote_since(TEX + 0x1000, p.sgen), "cpu_wrote_since: a CPU write after the boundary is missed");
|
||||||
|
CHECK(page_dirty(TEX + 0x1000, p.mgen), "page_dirty: a CPU write after the boundary is missed");
|
||||||
|
vk_mem_changed_snap(TEX, 0x4000, w_tex, ep, &now); // the region gets this epoch's snapshot
|
||||||
|
seen(now, "TEX snap");
|
||||||
|
CHECK(!shadow_pool_clean(&p, ep, 1), "shadow_pool_clean (CPU-side tables): a write after the boundary is missed");
|
||||||
|
CHECK(!shadow_pool_clean(&p, ep, 0), "shadow_pool_clean: a write after the boundary is missed");
|
||||||
|
// Synced again now: nothing is newer.
|
||||||
|
p.mgen = (__typeof__(p.mgen))gen_now, p.sgen = (__typeof__(p.sgen))gen_now;
|
||||||
|
CHECK(lazy_clean(TEX, TEX + 0x4000, &p), "lazy_clean: synced after every write, yet not clean");
|
||||||
|
CHECK(!cpu_wrote_since(TEX + 0x1000, p.sgen), "cpu_wrote_since: nothing written since the sync, yet written");
|
||||||
|
CHECK(!page_dirty(TEX + 0x1000, p.mgen), "page_dirty: nothing written since the sync, yet dirty");
|
||||||
|
CHECK(shadow_pool_clean(&p, ep, 1) && shadow_pool_clean(&p, ep, 0), "shadow_pool_clean: synced, yet not clean");
|
||||||
|
|
||||||
|
// The log: one line where the count reached 2^32, none at any other boundary.
|
||||||
|
int reached = seed < (1ull << 32) && (uint64_t)gen_now >= (1ull << 32);
|
||||||
|
CHECK(n_gen_lines == reached, "%d page generation lines, not %d (first: %s)", n_gen_lines, reached,
|
||||||
|
n_gen_lines ? gen_lines[0] : "none");
|
||||||
|
if (reached && n_gen_lines)
|
||||||
|
CHECK(strstr(gen_lines[0], "reached 4294967296 at frame 216000 after 1.0 h of play (19884 a frame)"),
|
||||||
|
"the line at 2^32: %s", gen_lines[0]);
|
||||||
|
for (int i = 0; i < n_gen_lines && i < 16; i++) {
|
||||||
|
CHECK(!strstr(gen_lines[i], "run out") && !strstr(gen_lines[i], "ran out at"), "a line of running out: %s",
|
||||||
|
gen_lines[i]);
|
||||||
|
printf(" %s\n", gen_lines[i]);
|
||||||
|
}
|
||||||
|
|
||||||
|
printf("gen_wrap_check: seed %#llx, %u-bit generations, gen_now %#llx: %d of %d checks failed\n",
|
||||||
|
(unsigned long long)seed, (unsigned)(8 * sizeof(gen_t)), (unsigned long long)gen_now, failed, checks);
|
||||||
|
return failed ? 1 : 0;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,7 +1,13 @@
|
|||||||
"""Build and run tests/gen_wrap_check.c: runtime/vk_memory.c's page write generations, extracted verbatim from its
|
"""Build and run tests/gen_wrap_check.c: runtime/vk_memory.c's page write generations across 2^32, 2^40 and 2^63.
|
||||||
"page write generations" banner to its "dirty census" banner, with the lines gen_next logs as the 32-bit generations
|
|
||||||
near their wrap (one at each of 0xC0000000, 0xD0000000, 0xE0000000, 0xF0000000 and one at the wrap), and the four
|
From runtime/vk_memory.c it takes, verbatim, into build/test/gen_wrap/gen_section.inc:
|
||||||
comparisons known to answer wrong across the wrap.
|
- the declarations of PoolRec's mgen and sgen (the generations a VRAM mirror and a shadow copy hold), so the stand-in
|
||||||
|
PoolRec has their real width;
|
||||||
|
- the section from the "page write generations" banner to the "dirty census" banner;
|
||||||
|
- the readers of the tables outside that section: cpu_wrote_since, shadow_pool_clean, lazy_clean and page_dirty.
|
||||||
|
The program starts the generations just below each seed's boundary and checks that every compare answers across it as
|
||||||
|
below it (texture checks, the snapshot path, the region check, the CPU-side tables, mirror and shadow cleanliness),
|
||||||
|
and that gen_next logs one line when the count reaches 2^32 and no other.
|
||||||
|
|
||||||
No GPU, no game files, any Python.
|
No GPU, no game files, any Python.
|
||||||
|
|
||||||
@@ -9,6 +15,7 @@ Usage: gen_wrap_check.py [--cc <clang>]
|
|||||||
"""
|
"""
|
||||||
import argparse
|
import argparse
|
||||||
import os
|
import os
|
||||||
|
import re
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
@@ -18,6 +25,9 @@ ROOT = Path(__file__).resolve().parents[1]
|
|||||||
MSYS_BIN = Path(r'C:\msys64\ucrt64\bin')
|
MSYS_BIN = Path(r'C:\msys64\ucrt64\bin')
|
||||||
BEGIN = '// ---------------------------------------------------------------- page write generations'
|
BEGIN = '// ---------------------------------------------------------------- page write generations'
|
||||||
END = '// ---------------------------------------------------------------- dirty census'
|
END = '// ---------------------------------------------------------------- dirty census'
|
||||||
|
READERS = ('static int cpu_wrote_since(', 'static int shadow_pool_clean(', 'static int lazy_clean(',
|
||||||
|
'static int page_dirty(')
|
||||||
|
SEEDS = ((1 << 32) - 8, (1 << 40) - 8, (1 << 63) - 8) # the old wrap; low 32 bits through 0 again; near 2^63
|
||||||
|
|
||||||
|
|
||||||
def compiler(cc):
|
def compiler(cc):
|
||||||
@@ -30,19 +40,56 @@ def compiler(cc):
|
|||||||
return shutil.which('clang') or 'clang'
|
return shutil.which('clang') or 'clang'
|
||||||
|
|
||||||
|
|
||||||
|
def definition(src, start):
|
||||||
|
"""The definition that begins with start (not a prototype), up to its closing brace at column 0."""
|
||||||
|
i = 0
|
||||||
|
while True:
|
||||||
|
i = src.find(start, i)
|
||||||
|
if i < 0:
|
||||||
|
return None
|
||||||
|
eol = src.index('\n', i)
|
||||||
|
if src[eol - 1] == '{':
|
||||||
|
return src[i:src.index('\n}\n', i) + 3]
|
||||||
|
i = eol
|
||||||
|
|
||||||
|
|
||||||
|
def extract(src):
|
||||||
|
"""gen_section.inc's text, or None and the reason."""
|
||||||
|
a, b = src.find(BEGIN), src.find(END)
|
||||||
|
if a < 0 or b < a:
|
||||||
|
return None, 'the section banners are not in runtime/vk_memory.c'
|
||||||
|
s0 = src.find('typedef struct PoolRec {')
|
||||||
|
s1 = src.find('} PoolRec;', s0)
|
||||||
|
if s0 < 0 or s1 < 0:
|
||||||
|
return None, 'PoolRec is not in runtime/vk_memory.c'
|
||||||
|
fields = []
|
||||||
|
for name in ('mgen', 'sgen'):
|
||||||
|
m = re.search(r'^\s*(u?int\d+_t)\s+' + name + r';', src[s0:s1], re.M)
|
||||||
|
if not m:
|
||||||
|
return None, f'PoolRec.{name} is not declared as an integer of fixed width'
|
||||||
|
fields.append(f'{m.group(1)} {name};')
|
||||||
|
out = ['// Extracted from runtime/vk_memory.c by tests/gen_wrap_check.py.\n',
|
||||||
|
'typedef struct PoolRec { int kind; void *buf; uint64_t import_lo, import_hi; ' + ' '.join(fields) +
|
||||||
|
' } PoolRec;\n', src[a:b]]
|
||||||
|
for start in READERS:
|
||||||
|
d = definition(src, start)
|
||||||
|
if not d:
|
||||||
|
return None, f'{start.split()[-1]}...) is not in runtime/vk_memory.c'
|
||||||
|
out.append(d)
|
||||||
|
return '\n'.join(out), None
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
ap = argparse.ArgumentParser()
|
ap = argparse.ArgumentParser()
|
||||||
ap.add_argument('--cc', help='the C compiler (default: $CC, MSYS2 UCRT64 clang, clang on PATH)')
|
ap.add_argument('--cc', help='the C compiler (default: $CC, MSYS2 UCRT64 clang, clang on PATH)')
|
||||||
args = ap.parse_args()
|
args = ap.parse_args()
|
||||||
src = (ROOT / 'runtime' / 'vk_memory.c').read_text(encoding='utf-8-sig')
|
inc, why = extract((ROOT / 'runtime' / 'vk_memory.c').read_text(encoding='utf-8-sig'))
|
||||||
a, b = src.find(BEGIN), src.find(END)
|
if not inc:
|
||||||
if a < 0 or b < a:
|
print(f'gen_wrap_check: FAIL: {why}')
|
||||||
print('gen_wrap_check: FAIL: the section banners are not in runtime/vk_memory.c')
|
|
||||||
return 1
|
return 1
|
||||||
work = ROOT / 'build' / 'test' / 'gen_wrap'
|
work = ROOT / 'build' / 'test' / 'gen_wrap'
|
||||||
work.mkdir(parents=True, exist_ok=True)
|
work.mkdir(parents=True, exist_ok=True)
|
||||||
(work / 'gen_section.inc').write_text('// Extracted from runtime/vk_memory.c by tests/gen_wrap_check.py.\n' +
|
(work / 'gen_section.inc').write_text(inc, encoding='utf-8')
|
||||||
src[a:b], encoding='utf-8')
|
|
||||||
exe = work / ('gen_wrap_check.exe' if os.name == 'nt' else 'gen_wrap_check')
|
exe = work / ('gen_wrap_check.exe' if os.name == 'nt' else 'gen_wrap_check')
|
||||||
env = dict(os.environ)
|
env = dict(os.environ)
|
||||||
if os.name == 'nt' and MSYS_BIN.exists():
|
if os.name == 'nt' and MSYS_BIN.exists():
|
||||||
@@ -50,7 +97,14 @@ def main():
|
|||||||
subprocess.check_call([compiler(args.cc), '-std=gnu11', '-O1', '-Wall', '-Wno-unused-function',
|
subprocess.check_call([compiler(args.cc), '-std=gnu11', '-O1', '-Wall', '-Wno-unused-function',
|
||||||
'-Wno-unused-variable', '-I', str(work), str(ROOT / 'tests' / 'gen_wrap_check.c'), '-o',
|
'-Wno-unused-variable', '-I', str(work), str(ROOT / 'tests' / 'gen_wrap_check.c'), '-o',
|
||||||
str(exe)], env=env)
|
str(exe)], env=env)
|
||||||
return subprocess.run([str(exe)], env=env, timeout=120).returncode
|
bad = 0
|
||||||
|
for seed in SEEDS:
|
||||||
|
bad += subprocess.run([str(exe), str(seed)], env=env, timeout=120).returncode != 0
|
||||||
|
if bad:
|
||||||
|
print(f'gen_wrap_check: {bad} of {len(SEEDS)} runs failed')
|
||||||
|
return 1
|
||||||
|
print('gen_wrap_check: all checks passed')
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
if __name__ == '__main__':
|
||||||
|
|||||||
Reference in New Issue
Block a user