mirror of
https://github.com/tangerineturtle1/NX2navi.git
synced 2026-11-17 00:39:57 -04:00
Fix Ocarina's white dialogue frames, the fountain floor speckle and the button prompt text
The texture pool descriptors are now a ring of sets, one per submission, brought up to date from a change log, and the first submission of each frame waits for its GPU work before the game goes on: the GPU ran late and part of a frame read data of the next one, and distant characters drew NaN that bloom spread over the frame. The window acquire also waits for the image's previous frame. Function locals start at zero unless DLAA is on, which removes the per-warp light tiles on the Great Fairy Fountain floor and Link's dark, speckled look. Texture slots follow only live entries of their own target, so the volumetric fog gets its 3D volume. The pool refresh walks only the slots that changed. The translator implements PRMT (patch 0021), which it only logged before; the button prompts render again. HFMA2.RELU no longer depends on it, HMMA.F16 rounds once and HMMA.F32 is lowered, checked against GPU vectors with a new Maxwell warp emulator. The frame dump gains a re-arming white-frame trigger, a trigger on a depth target's first pass, NaN pixels in magenta, image handles and view targets, and depth copies of sampled depth textures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -257,6 +257,11 @@ Shaders: `.venv-py314\Scripts\python tools\shader_census.py --version <ver> --tr
|
||||
to `build\<ver>\shaders\` and runs it through the translator and `spirv-val`; failure groups go to `failures.txt`.
|
||||
`tools\xlate_smoke.py` checks the translator alone, without game files.
|
||||
|
||||
A change to the SM86 tensor lowering (`tools/xlate/AmpereLowering.cs` Hmma, Ldsm, Movm, Half) passes
|
||||
`python tests\tensor_lowering_check.py`: it lowers GPU-captured vectors with `nx2navi_xlate --lower` and runs them on
|
||||
`tests\maxwell_warp_emu.py`, a model of what the Ryujinx translator makes of Maxwell code (the docstring names the
|
||||
vector file and what is checked). `tests\maxwell_warp_emu_selftest.py` checks the emulator itself. No game files.
|
||||
|
||||
## Reverse engineering (optional)
|
||||
|
||||
- Ghidra 12.1.2 with Adubbz's Ghidra-Switch-Loader in `toolchain\ghidra\` (JDK 21) opens the game modules (NSO).
|
||||
|
||||
@@ -88,7 +88,7 @@ Source shorthands:
|
||||
| 3.9 | **Integer vertex inputs read as raw bits** (bone indices `R8UI`). | Float-declared inputs convert integers. | NXN Mario: stretched characters; re-translated with `uint` inputs. Ours: attribute types passed to `QueryGraphicsState` (`Program.cs`). | NXN |
|
||||
| 3.10 | **BFE / BFI fields past bit 31** (as PTX `bfe` / `bfi`): a field that runs past bit 31 is cut there; one that starts past it extracts 0 (signed: bit 31 repeated) and inserts nothing. NX2navi 1.4.3's Depths terrain vertex shaders (42 variants) take the position from a register, `((2y + x) * 8) & 0xff` with 5-bit x and y, so up to 248. | `OpBitField*Extract` / `OpBitFieldInsert` undefined when offset or offset + count exceeds 32; AMD's bitfield instructions take offset and width modulo 32. | Ryujinx passes the fields through. UltraTears saw the Depths walls and ground missing on the Steam Deck and a Linux RX 570 (RADV). Ours: clamped where the field is not a constant inside the word (translator patch 0012). RX 9070 on Windows drew the Depths the same before and after. | PTX ISA `bfe`, `bfi` |
|
||||
| 3.11 | **SHF by 0** gives the unshifted word (C for `SHF.L`, A for `SHF.R`). | Ryujinx's funnel shifts the other word by 32: undefined in SPIR-V; AMD shifts take the low 5 bits and give A \| C. | the hardware fixed Depths terrain gaps in its funnel shift (PR 10459). Ours: a select on a zero amount (translator patch 0013). In 1.4.3 every register SHF pairs its word with 0, so the result was right already. | — |
|
||||
| 3.12 | **Registers read before any write** on a path (paths the structurizer adds included) have no defined value in the game's code; UltraTears reports 0 from the NVIDIA driver. | An `OpVariable` without an initializer loads an undefined value; Mesa folds undefined values freely. | Ours: on Mesa drivers (RADV, ANV, NVK; `vk_caps_driver`) every function-local variable starts at 0 (`OpConstantNull`, job flag `zero_locals`, translator patch 0016; `NX2NAVI_ZERO_LOCALS` overrides). Elsewhere the load stays undefined: a zero start on every driver (the old patch 0014) made the whole picture shake by about 0.9 output pixels a frame with DLSS or DLAA on an RTX 4080 SUPER (#612). It also moved the SPIR-V hash of the game's FXAA shader out of `FXAA_FS` (`vk_dlaa.c` `vk_dlaa_skip_draw`), the likely cause of the shake; that path is not traced. | — |
|
||||
| 3.12 | **Registers read before any write** on a path (paths the structurizer adds included) have no defined value in the game's code; UltraTears reports 0 from the NVIDIA driver. | An `OpVariable` without an initializer loads an undefined value; Mesa folds undefined values freely. | Ours: every function-local variable starts at 0 (`OpConstantNull`, job flag `zero_locals`, translator patch 0016; `NX2NAVI_ZERO_LOCALS` overrides) on Mesa drivers (RADV, ANV, NVK; `vk_caps_driver`) and, with DLAA off, on every other driver too: on NVIDIA an undefined read takes another warp lane's value, and Ocarina's clustered-light loop lit its floors in 4x8-pixel tiles (one warp each) and darkened Link (runtime/vk_caps.c). With DLAA on, off Mesa, the load stays undefined: a zero start on every driver (the old patch 0014) made the whole picture shake by about 0.9 output pixels a frame with DLSS or DLAA on an RTX 4080 SUPER (#612). It also moved the SPIR-V hash of the game's FXAA shader out of `FXAA_FS` (`vk_dlaa.c` `vk_dlaa_skip_draw`), the likely cause of the shake; that path is not traced. | — |
|
||||
|
||||
## 4. Memory model
|
||||
|
||||
|
||||
@@ -40,7 +40,7 @@ static const char *const known[] = {
|
||||
"BIND_ZL", "BIND_ZR", "BLOCK_VIEWS_3D", "CLOTH_REALTIME", "COLOR_BRIGHTNESS", "COLOR_CONTRAST", "COLOR_GAMMA",
|
||||
"COLOR_PRESET", "COLOR_SATURATION", "CONTACT_DEBUG", "CONTACT_SHADOWS", "CORE_DIAG", "CRIT", "CRIT_OUT",
|
||||
"DESC_TEMPLATE", "DEVICE_PROFILE", "DISCORD_APP_ID", "DISCORD_IMAGE", "DISCORD_RPC", "DLAA", "DLAA_JITTER", "DLAA_MV", "DLAA_MASKDEBUG", "DLAA_MVDEBUG", "DLAA_OBJMV", "DLAA_SURVEY", "DLAA_TIMES", "DLSS", "DLSS_CHAR_MASK", "DLSS_PRESET", "DLSS_REACTIVE", "DUMP", "DUMP_BUF", "DUMP_DIR",
|
||||
"DUMP_LOG", "LOG_CS_BINDS", "LOG_TEX_FLAGS", "LOG_MESH", "MESH", "POOL", "NATIVE_TENSOR", "LOG_RCOND", "CS_FALLBACK", "EXEFS", "FEEDBACK", "FIX_C0", "FIX_ZERO", "FLIGHT", "FLIGHT_AT", "FLIGHT_DIR", "FLIGHT_FRAMES",
|
||||
"DUMP_LOG", "LOG_CS_BINDS", "LOG_TEX_FLAGS", "LOG_MESH", "MESH", "POOL", "NATIVE_TENSOR", "LOG_RCOND", "DUMP_WHITE", "DUMP_AFTER", "SUBMIT_FINISH", "ACQUIRE_WAIT", "CS_FALLBACK", "EXEFS", "FEEDBACK", "FIX_C0", "FIX_ZERO", "FLIGHT", "FLIGHT_AT", "FLIGHT_DIR", "FLIGHT_FRAMES",
|
||||
"FLIGHT_SPIKE_MS", "FLIGHT_THUMBS", "FLUSH_SPAN", "FOV", "FOV_PATCH", "FPS60", "FPS_LIMIT", "FREEZE_IDLE", "FREE_CORES",
|
||||
"FULLSCREEN", "FXAA", "GPL", "GPL_DRAW_BUILD", "GPU", "GPU_INDEX", "GPU_NAME", "HDR", "HDR_GAME_NITS", "HDR_HUE",
|
||||
"HDR_PEAK_NITS", "HDR_UI_NITS", "HIGHFPS_FIX", "INPUT", "INPUT_RECORD", "INPUT_SCRIPT", "LANG", "LIB_PREBUILD",
|
||||
|
||||
@@ -6,7 +6,8 @@
|
||||
#define FLIGHT_THUMB_W 320
|
||||
#define FLIGHT_THUMB_H 180
|
||||
|
||||
extern int flight_on; // NX2NAVI_FLIGHT=1, set once by flight_init before the first present
|
||||
extern int flight_on;
|
||||
extern int vk_dump_white; // vk_dump.c, NX2NAVI_DUMP_WHITE // NX2NAVI_FLIGHT=1, set once by flight_init before the first present
|
||||
|
||||
void flight_init(void); // prof_start calls it
|
||||
// perf_present, once per present on the render thread: the interval since the previous present, the render thread's
|
||||
|
||||
@@ -45,6 +45,7 @@ const pool_t *nvn_pool_find(uint64_t addr, uint64_t len); // GPU-visible pool h
|
||||
uint64_t nvn_new_serial(void); // next tex_t / prog_t serial, from 1
|
||||
uint64_t nvn_fence_last(void); // fence value of the newest submit (0 before the first)
|
||||
int nvn_depth_minus_w(void); // 1 after DeviceSetDepthMode(NEAR_IS_MINUS_W), else 0
|
||||
extern int nvn_submits_in_frame; // nvnQueueSubmitCommands calls since the last present (NX2NAVI_SUBMIT_FINISH)
|
||||
void nvn_guest_written(uint64_t addr, uint64_t size); // the host or the game wrote pool storage: back end re-reads
|
||||
// NX2NAVI_TRACE=perf: MemoryPoolFlushMappedRange [0] and BufferFlushMappedRange [1] calls [0] and bytes [1] by the pool's
|
||||
// CPU class (0 CPU_CACHED, 1 CPU_UNCACHED, 2 other) since the previous call.
|
||||
|
||||
@@ -622,6 +622,7 @@ NVN_API(nvnQueueFinish) {
|
||||
// lod_entered / lod_issued count submits that entered QueueSubmitCommands / reached the back end; lod_fence holds
|
||||
// the fence of the last LOD_RING. NX2NAVI_LOD_GUARD=0 turns the wait off.
|
||||
#define LOD_RING 64
|
||||
int nvn_submits_in_frame; // nvnQueueSubmitCommands calls since the last nvnQueuePresentTexture (NX2NAVI_SUBMIT_FINISH)
|
||||
static uint64_t lod_entered, lod_issued, lod_fence[LOD_RING];
|
||||
|
||||
// void nvnQueueSubmitCommands(NVNqueue *q, int n, const NVNcommandHandle *handles): replayed on this thread with its
|
||||
@@ -655,6 +656,23 @@ NVN_API(nvnQueueSubmitCommands) {
|
||||
while (f > prev && !__atomic_compare_exchange_n(&last_fence, &prev, f, 1, __ATOMIC_RELEASE, __ATOMIC_RELAXED)) {}
|
||||
__atomic_store_n(&lod_fence[no % LOD_RING], f, __ATOMIC_RELAXED);
|
||||
__atomic_store_n(&lod_issued, no, __ATOMIC_SEQ_CST);
|
||||
// The first submission of a frame (since the last present) returns once the GPU has run it. NVN starts the GPU when
|
||||
// the submit returns and it is through a frame's first submission (Ocarina's frame-start compute: light grid, probes)
|
||||
// within a few milliseconds; here the replay records first and the GPU runs later. Ocarina rewrites data those
|
||||
// passes and the frame's lit materials read once it has submitted its later work, with no fence: with the GPU late,
|
||||
// part of a frame drew with the next frame's data, distant characters wrote NaN and bloom spread it (white dialogue
|
||||
// frames, 3% of them in 100 s of dialogue; none with this wait, at the same 58-59 fps: the GPU is idle at that
|
||||
// point). Copies of the data at submit made it worse, a mix of two frames' versions.
|
||||
// NX2NAVI_SUBMIT_FINISH=<mask>: bit i waits after the i-th submission of a frame (default 0x1); a plain 1 waits
|
||||
// after every submission, 0 never.
|
||||
static long long finish_mask = -2;
|
||||
if (finish_mask == -2) {
|
||||
const char *e = cfg_get("NX2NAVI_SUBMIT_FINISH");
|
||||
finish_mask = e && *e ? strtoll(e, NULL, 0) : 1;
|
||||
if (e && *e && finish_mask == 1 && !(e[0] == '0' && (e[1] == 'x' || e[1] == 'X'))) finish_mask = -1; // "1": all
|
||||
}
|
||||
int idx = nvn_submits_in_frame++;
|
||||
if (finish_mask && f && (finish_mask == -1 || (idx < 63 && (finish_mask >> idx & 1)))) gfx_backend()->fence_wait(f, UINT64_MAX);
|
||||
}
|
||||
|
||||
// The producer (1.2.1 main f_70008c6460) releases the read lock on obj+0x538 at lr 0x70008c70c8; when
|
||||
|
||||
@@ -756,6 +756,19 @@ static void sleep_until(uint64_t deadline) {
|
||||
}
|
||||
}
|
||||
|
||||
// The submit fence of each window image's newest present (win_t has no room: a side table for the first windows).
|
||||
static struct { const win_t *w; uint64_t fence[4]; } img_fence[4];
|
||||
static uint64_t last_present_fence; // the submit fence at the newest present of any window
|
||||
|
||||
static uint64_t *img_fence_of(const win_t *o, int idx) {
|
||||
if (idx < 0 || idx >= 4) return NULL;
|
||||
for (int i = 0; i < 4; i++) {
|
||||
if (img_fence[i].w == o) return &img_fence[i].fence[idx];
|
||||
if (!img_fence[i].w) return img_fence[i].w = o, &img_fence[i].fence[idx];
|
||||
}
|
||||
return NULL;
|
||||
}
|
||||
|
||||
// NVNwindowAcquireTextureResult nvnWindowAcquireTexture(NVNwindow *w, NVNsync *s, int *index): blocks here, not in the
|
||||
// present, until the image's previous present plus one period has passed (the game often presents with a lock held).
|
||||
// That is the time the next frame goes to the display, so the image is free as soon as that time is reached.
|
||||
@@ -772,6 +785,24 @@ NVN_API(nvnWindowAcquireTexture) {
|
||||
sleep_until(at);
|
||||
perf_end(PERF_PACE, p0);
|
||||
}
|
||||
// On the Switch the image comes back only after the display has shown it, which needs the GPU to have finished the
|
||||
// frame drawn into it, so the CPU never runs more than the swapchain's depth ahead of the GPU. Here the pacer frees
|
||||
// images by the clock alone, so the acquire also waits for the GPU to finish the image's newest present. With the
|
||||
// GPU slowed (frame dump logging), this alone cut Ocarina's NaN frames from 56 to 23 of 61; the wait after a frame's
|
||||
// first submission (nvn_api_core.c nvnQueueSubmitCommands) removes the rest. NX2NAVI_ACQUIRE_WAIT=0 leaves it to the
|
||||
// clock; =2 waits for the newest present of any image (one frame in flight at most).
|
||||
static int acq_wait = -1;
|
||||
if (acq_wait < 0) acq_wait = rt_knob("ACQUIRE_WAIT", "1");
|
||||
uint64_t *fe = img_fence_of(o, idx), want = fe ? *fe : 0;
|
||||
if (acq_wait == 2) want = last_present_fence;
|
||||
if (acq_wait && want) {
|
||||
const GfxBackend *b = gfx_backend();
|
||||
if (b->fence_done() < want) {
|
||||
uint64_t p0 = perf_begin();
|
||||
b->fence_wait(want, UINT64_MAX);
|
||||
perf_end(PERF_PACE, p0);
|
||||
}
|
||||
}
|
||||
// The sync means "the display no longer reads this image". Presents are copies queued in submit order, so it is
|
||||
// already satisfied: fenced = 3 with fence 0.
|
||||
if (s) {
|
||||
@@ -794,12 +825,16 @@ NVN_API(nvnQueuePresentTexture) {
|
||||
if (o->magic != NVN_WIN_MAGIC || idx < 0 || idx >= o->ntex || idx >= 4 || !o->tex[idx])
|
||||
rt_fatal("nvn: QueuePresentTexture of image %d on window %p", idx, (void *)o);
|
||||
o->presents++;
|
||||
nvn_submits_in_frame = 0;
|
||||
const tex_t *t = (const tex_t *)o->tex[idx];
|
||||
uint64_t period = present_period(o), now = kern_now_ns();
|
||||
uint64_t slot = now + pace(&o->pace, now, period);
|
||||
gfx_backend()->present(t, o->crop, slot);
|
||||
o->ticket[idx] = o->presents;
|
||||
o->avail[idx] = slot + period;
|
||||
uint64_t *fe = img_fence_of(o, idx);
|
||||
if (fe) *fe = nvn_fence_last(); // the frame drawn into it (nvnWindowAcquireTexture waits for it)
|
||||
last_present_fence = nvn_fence_last();
|
||||
frame_step_present(slot, o->interval); // frame_rate.c: the real-time step follows the display times
|
||||
rt_frame_presented(t, o->presents);
|
||||
}
|
||||
|
||||
@@ -425,6 +425,11 @@ VkImage vk_tonemap_record(VkCommandBuffer cb, VkImage src, VkFormat sf, uint32_t
|
||||
// its storage address is created, dropped, released or taken back into use. Unlocked; read it before the lookup it
|
||||
// guards (a lookup may create an entry). 0 when the slot was never registered: nothing is kept under 0.
|
||||
uint64_t vk_texture_slot_token(int id);
|
||||
// Slots whose token may have changed since *pos (0 at first; then the value these calls leave): fn is called for each
|
||||
// (some more than once) and *pos advanced. 0 when too many changes came since: the caller looks at every slot. *pos
|
||||
// from vk_texture_dirty_pos_locked after such a walk. Texture lock held.
|
||||
int vk_texture_dirty_slots_locked(uint64_t *pos, void (*fn)(int id, void *arg), void *arg);
|
||||
uint64_t vk_texture_dirty_pos_locked(void);
|
||||
void vk_sampler_dlss_bias(float bias); // texture LOD bias added for DLSS upscaling (material samplers)
|
||||
uint32_t vk_sampler_gen(void); // changes whenever a sampler slot is registered
|
||||
void vk_texture_gpu_written(const tex_t *t); // rendered or copied into by the GPU: no re-upload from storage
|
||||
|
||||
@@ -58,7 +58,7 @@ static void present_main(void *arg) {
|
||||
busy = 1;
|
||||
plat_cond_broadcast(&pcv);
|
||||
plat_mutex_unlock(&pl);
|
||||
if (flight_on) vk_thumbs_frame(j.frame);
|
||||
if (flight_on || vk_dump_white) vk_thumbs_frame(j.frame); // NX2NAVI_DUMP_WHITE reads them too
|
||||
vk_present_blit(j.image, VK_IMAGE_LAYOUT_GENERAL, j.format, j.w, j.h, j.has_crop ? j.crop : NULL);
|
||||
uint64_t f = vk_sync_submit(NULL, 0); // a timeline value after the blit: deferred destruction waits for the blit too
|
||||
if (j.value && f != j.value)
|
||||
|
||||
@@ -116,9 +116,14 @@ void vk_caps_init(VkCtx *c) {
|
||||
// BotW and NX2navi on AMD). The translator biases gather coordinates by the host's sub-texel precision; NVIDIA rounds
|
||||
// as Maxwell does. NX2NAVI_GATHER_BIAS=<bits> forces it (0: off).
|
||||
uint32_t gather = env_u32("NX2NAVI_GATHER_BIAS", c->props.vendorID == 0x10de ? 0 : l->subTexelPrecisionBits);
|
||||
// A register read before any write is an undefined load in SPIR-V, and Mesa folds undefined values freely, so on
|
||||
// Mesa function-local variables start at 0 (docs/maxwell.md 3.12). NX2NAVI_ZERO_LOCALS=1 forces it, 0 turns it off.
|
||||
uint32_t zero_locals = env_u32("NX2NAVI_ZERO_LOCALS", (uint32_t)vk_caps.mesa);
|
||||
// A register read before any write is an undefined load in SPIR-V, so function-local variables start at 0
|
||||
// (docs/maxwell.md 3.12). Mesa folds undefined values freely; NVIDIA's compiler gives such a read a value of another
|
||||
// lane of the warp: Ocarina's clustered-light loop reads registers that only some paths write, and on an RTX 5090
|
||||
// the Great Fairy Fountain floor took an extra light term in 4x8-pixel tiles (one 32-lane warp each) and Link went
|
||||
// dark with green speckles. With DLAA on (vk_dlaa.c, which finds the game's FXAA by its SPIR-V hash) they stay
|
||||
// undefined off Mesa, as before: zeroing moved that hash and the picture shook under DLAA (maxwell.md 3.12).
|
||||
// NX2NAVI_ZERO_LOCALS=1 / 0 forces either.
|
||||
uint32_t zero_locals = env_u32("NX2NAVI_ZERO_LOCALS", vk_caps.mesa || !cfg_bool("NX2NAVI_DLAA"));
|
||||
gfx_shader_set_host(vk_caps.float64, vk_caps.sbuf_align, told, gather, zero_locals != 0);
|
||||
if (gather) rt_log("vk caps: gather coordinates biased for %u sub-texel bits (NX2NAVI_GATHER_BIAS)", gather);
|
||||
|
||||
|
||||
@@ -5,7 +5,8 @@
|
||||
// written in the frame are copied once each, slices side by side. Files in NX2NAVI_DUMP_DIR (default "dump"):
|
||||
// pass_<seq>_t<i>_<w>x<h>_<vkformat>_<name>.png colour target i (td: depth; img<slot>: storage image; lut<id>: 3D;
|
||||
// a BC target drawn through a block view: its blocks as that view)
|
||||
// ..._tm.png float formats: x/(1+x) of the same image (the plain one clamps)
|
||||
// ..._tm.png float formats: x/(1+x) of the same image (the plain one clamps);
|
||||
// in both, a pixel with a NaN channel is magenta
|
||||
// passes.txt per pass or dispatch: targets, draws, each distinct pipeline + texture set with its shaders' SPIR-V
|
||||
// hashes (sh_<hash>.spv written once) and the bound textures with their samplers
|
||||
// images.txt per image: channel min/max, distinct texel values over the image and over its top quarter
|
||||
@@ -35,6 +36,7 @@
|
||||
#include "vk.h"
|
||||
|
||||
#include "vk_exec.h"
|
||||
#include "flight.h"
|
||||
uint64_t vk_exec_cs_storage_addr(const VkExec *ex, const GfxShaderRes *r); // vk_exec_draw.c
|
||||
int vk_exec_raw_9e5(VkImage img, VkFormat f); // vk_exec_draw.c
|
||||
#include "platform/platform.h"
|
||||
@@ -49,6 +51,10 @@ int vk_exec_raw_9e5(VkImage img, VkFormat f); // v
|
||||
int vk_dump_armed;
|
||||
static uint64_t dump_frame, log_from; // log_from: NX2NAVI_DUMP_LOG, first frame logged to passes.txt without images
|
||||
static char dir[400];
|
||||
static char base_dir[400]; // NX2NAVI_DUMP_DIR; with NX2NAVI_DUMP_WHITE each attempt goes to base_dir/f<frame>
|
||||
static uint64_t white_last; // NX2NAVI_DUMP_WHITE: the frame dumped last (0: none), checked 3 presents on
|
||||
static int white_tries, white_caught;
|
||||
static uint64_t dump_after; // NX2NAVI_DUMP_AFTER
|
||||
static FILE *passes_f, *images_f;
|
||||
static int seq, finished, copied; // copied: passes and dispatches of the dump frame whose images were copied
|
||||
static int skip_runs, dump_ubo; // NX2NAVI_DUMP_SKIP_RUNS, NX2NAVI_DUMP_UBO (vec4 per block, 0 for none)
|
||||
@@ -221,8 +227,8 @@ static void write_images(const Job *j) {
|
||||
uint32_t pw = j->w * j->d, ph = j->h;
|
||||
uint64_t n = (uint64_t)pw * ph;
|
||||
float *px = malloc(n * 16);
|
||||
uint8_t *out = malloc(n * 4);
|
||||
if (!px || !out) rt_fatal("vk_dump: out of memory for %s", j->name);
|
||||
uint8_t *out = malloc(n * 4), *isnan_ = calloc(n, 1); // isnan_: a pixel with a NaN channel, drawn magenta
|
||||
if (!px || !out || !isnan_) rt_fatal("vk_dump: out of memory for %s", j->name);
|
||||
float mn[4] = {INFINITY, INFINITY, INFINITY, INFINITY}, mx[4] = {-INFINITY, -INFINITY, -INFINITY, -INFINITY};
|
||||
uint64_t nans = 0;
|
||||
for (uint32_t z = 0; z < j->d; z++)
|
||||
@@ -231,7 +237,7 @@ static void write_images(const Job *j) {
|
||||
float *o = px + 4 * ((uint64_t)y * pw + (uint64_t)z * j->w + x);
|
||||
decode(j->f, j->map + (((uint64_t)z * j->h + y) * j->w + x) * j->f->bytes, o);
|
||||
for (int c = 0; c < 4; c++) {
|
||||
if (isnan(o[c])) { nans++, o[c] = 0; continue; }
|
||||
if (isnan(o[c])) { nans++, o[c] = 0, isnan_[(uint64_t)y * pw + (uint64_t)z * j->w + x] = 1; continue; }
|
||||
if (o[c] < mn[c]) mn[c] = o[c];
|
||||
if (o[c] > mx[c]) mx[c] = o[c];
|
||||
}
|
||||
@@ -255,6 +261,7 @@ static void write_images(const Job *j) {
|
||||
out[4 * i + c] = u8(x);
|
||||
}
|
||||
out[4 * i + 3] = 255;
|
||||
if (isnan_[i]) out[4 * i] = 255, out[4 * i + 1] = 0, out[4 * i + 2] = 255;
|
||||
}
|
||||
char path[600];
|
||||
snprintf(path, sizeof path, "%s/%s%s.png", dir, j->name, pass ? "_tm" : "");
|
||||
@@ -270,6 +277,7 @@ static void write_images(const Job *j) {
|
||||
}
|
||||
plat_mutex_unlock(&img_lock);
|
||||
free(out);
|
||||
free(isnan_);
|
||||
free(px);
|
||||
}
|
||||
|
||||
@@ -527,9 +535,12 @@ static void flush_luts(VkExec *ex, int n) {
|
||||
VkFormat bf = block_format(l->format);
|
||||
snprintf(name, sizeof name, "pass_%04d_%s%d_%ux%ux%u_%d_%s%s", n, l->is2d ? "tex" : "lut", l->id, l->w, l->h, l->d,
|
||||
(int)l->format, fmt_name(l->format, fn), bf ? "_blocks" : "");
|
||||
// a sampled depth texture copies its depth aspect (the colour aspect of a depth image read as zeros)
|
||||
VkImageAspectFlags asp = l->format >= VK_FORMAT_D16_UNORM && l->format <= VK_FORMAT_D32_SFLOAT_S8_UINT
|
||||
? VK_IMAGE_ASPECT_DEPTH_BIT : VK_IMAGE_ASPECT_COLOR_BIT;
|
||||
const char *why = bf ? copy_region(ex, l->image, bf, VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, 0,
|
||||
(VkExtent3D){l->w, l->h, l->d}, (l->w + 3) / 4, (l->h + 3) / 4, l->d, name)
|
||||
: copy_out(ex, l->image, l->format, VK_IMAGE_ASPECT_COLOR_BIT, 0, 0, l->w, l->h, l->d, name);
|
||||
: copy_out(ex, l->image, l->format, asp, 0, 0, l->w, l->h, l->d, name);
|
||||
fprintf(passes_f, " %s tex %d -> %s.png%s%s\n", l->is2d ? "2D" : "3D", l->id, name, why ? " skipped: " : "",
|
||||
why ? why : "");
|
||||
}
|
||||
@@ -572,8 +583,12 @@ static void describe_bindings(Str *b, const VkExec *ex, const VkPipeInfo *info)
|
||||
}
|
||||
if (st == 5) sp(b, " -> %#llx", (unsigned long long)vk_exec_cs_storage_addr(ex, r));
|
||||
for (int k = 0; k < 16; k++)
|
||||
if (ex->ssbo[st][k].addr)
|
||||
sp(b, " %d=%#llx+%#llx", k, (unsigned long long)ex->ssbo[st][k].addr, (unsigned long long)ex->ssbo[st][k].size);
|
||||
if (ex->ssbo[st][k].addr) {
|
||||
VkDescriptorBufferInfo cb; // "(copy)": a binding that only reads it uses the submit-time copy (vk_usnap.c)
|
||||
int copy = vk_usnap_find(ex->ssbo[st][k].addr, ex->ssbo[st][k].size, 1, &cb);
|
||||
sp(b, " %d=%#llx+%#llx%s", k, (unsigned long long)ex->ssbo[st][k].addr, (unsigned long long)ex->ssbo[st][k].size,
|
||||
copy ? "(copy)" : "");
|
||||
}
|
||||
sp(b, "\n");
|
||||
// NX2NAVI_DUMP_UBO: also the small storage buffers' guest bytes (GPU results as of the last write-back); per-view
|
||||
// constants (exposure and such) live in such buffers.
|
||||
@@ -624,10 +639,11 @@ static void describe_bindings(Str *b, const VkExec *ex, const VkPipeInfo *info)
|
||||
const VkTexImage *t = vk_texture_by_id(NVN_HANDLE_TEX(h), &v, &hv);
|
||||
if (!t) { sp(b, "tex %d unresolved (dummy)\n", NVN_HANDLE_TEX(h)); continue; }
|
||||
static const char *vt[] = {"1D", "2D", "3D", "cube", "1DArray", "2DArray", "cubeArray"};
|
||||
sp(b, "tex %d %s(%d) %ux%ux%u levels %u layers %u image %s", NVN_HANDLE_TEX(h), fmt_name(t->format, fn),
|
||||
sp(b, "tex %d %s(%d) %ux%ux%u levels %u layers %u image %s vkimage %p", NVN_HANDLE_TEX(h), fmt_name(t->format, fn),
|
||||
(int)t->format, t->width, t->height, t->depth, t->levels, t->layers,
|
||||
t->view_type <= 6 ? vt[t->view_type] : "?");
|
||||
t->view_type <= 6 ? vt[t->view_type] : "?", (void *)t->image);
|
||||
if (hv && v.set_format) sp(b, " view-nvn-fmt %#x", v.format);
|
||||
if (hv && v.set_target) sp(b, " view-target %u", v.target);
|
||||
if (hv && v.set_levels) sp(b, " view-levels %u+%u", v.base_level, v.nlevels);
|
||||
if (hv && v.set_layers) sp(b, " view-layers %u+%u", v.min_layer, v.nlayers);
|
||||
samp_par s;
|
||||
@@ -730,9 +746,9 @@ static void target_line(VkExec *ex, const VkExecTarget *t, const char *tag, int
|
||||
t->height, 1, name) :
|
||||
t->is3d ? "3D slice (the volume is dumped as lut<id> where it is sampled)" :
|
||||
copy_out(ex, t->image, t->format, t->aspect, t->level, t->base_layer, t->width, t->height, 1, name);
|
||||
fprintf(passes_f, " %s: nvn %#x %s(%d) %ux%u level %u layers %u+%u samples %u addr %#llx -> %s.png%s%s\n", tag,
|
||||
t->nvn_format, fmt_name(t->format, fn), (int)t->format, t->width, t->height, t->level, t->base_layer,
|
||||
t->layers, t->samples, (unsigned long long)(uintptr_t)t->tex->mem, name, why ? " skipped: " : "",
|
||||
fprintf(passes_f, " %s: nvn %#x %s(%d) %ux%u level %u layers %u+%u samples %u addr %#llx vkimage %p -> %s.png%s%s\n",
|
||||
tag, t->nvn_format, fmt_name(t->format, fn), (int)t->format, t->width, t->height, t->level, t->base_layer,
|
||||
t->layers, t->samples, (unsigned long long)(uintptr_t)t->tex->mem, (void *)t->image, name, why ? " skipped: " : "",
|
||||
why ? why : "");
|
||||
}
|
||||
|
||||
@@ -755,6 +771,11 @@ static void pass_decide(int same_targets) {
|
||||
|
||||
void vk_dump_pass_end(VkExec *ex) {
|
||||
plat_mutex_lock(&rec_lock);
|
||||
if (dump_after && dump_frame == UINT64_MAX && ex->depth.tex && (uint64_t)(uintptr_t)ex->depth.tex->mem == dump_after) {
|
||||
dump_frame = frame_now(); // this pass and the rest of its frame
|
||||
rt_log("vk_dump: a pass drew into %#llx in frame %llu: it and the rest of the frame go to %s",
|
||||
(unsigned long long)dump_after, (unsigned long long)dump_frame, dir);
|
||||
}
|
||||
if (!active()) goto out;
|
||||
int n = seq++;
|
||||
uint64_t sig = targets_sig(ex);
|
||||
@@ -893,9 +914,34 @@ out:
|
||||
|
||||
static void make_dirs(char *path) { plat_file_mkdir_p(path); }
|
||||
|
||||
// NX2NAVI_DUMP_WHITE=1 (diagnosis): no frame is named; from frame 1800 each present looks at the thumbnail of the frame
|
||||
// three presents back (gfx_present.c vk_thumbs_*), and the first one whose picture left of the overlay and above the
|
||||
// dialogue box is blank white makes the dump take the next frame (white flashes come in runs of several frames).
|
||||
int vk_dump_white; // vk_backend.c asks for a thumbnail of every present while it is set
|
||||
#define dump_white vk_dump_white
|
||||
|
||||
// passes.txt and images.txt in dir (a new folder per NX2NAVI_DUMP_WHITE attempt)
|
||||
static void open_files(void) {
|
||||
make_dirs(dir);
|
||||
char path[600];
|
||||
snprintf(path, sizeof path, "%s/passes.txt", dir);
|
||||
passes_f = fopen(path, "w");
|
||||
snprintf(path, sizeof path, "%s/images.txt", dir);
|
||||
images_f = fopen(path, "w");
|
||||
if (!passes_f || !images_f) rt_fatal("vk_dump: cannot write to %s", dir);
|
||||
}
|
||||
|
||||
void vk_dump_init(void) {
|
||||
const char *e = cfg_get("NX2NAVI_DUMP");
|
||||
if (!e || !*e || !(dump_frame = strtoull(e, NULL, 10))) return;
|
||||
dump_white = rt_knob("DUMP_WHITE", "0");
|
||||
// NX2NAVI_DUMP_AFTER=<address>: the first render pass whose depth target lives at that address and the rest of its frame
|
||||
// (a target drawn once, at an area load)
|
||||
const char *da = cfg_get("NX2NAVI_DUMP_AFTER");
|
||||
dump_after = da && *da ? strtoull(da, NULL, 0) : 0;
|
||||
if (dump_white || dump_after) {
|
||||
dump_frame = UINT64_MAX;
|
||||
vk_thumbs_enable(16);
|
||||
} else if (!e || !*e || !(dump_frame = strtoull(e, NULL, 10))) return;
|
||||
const char *lf = cfg_get("NX2NAVI_DUMP_LOG");
|
||||
if (lf && *lf) log_from = strtoull(lf, NULL, 10);
|
||||
const char *sr = cfg_get("NX2NAVI_DUMP_SKIP_RUNS");
|
||||
@@ -912,14 +958,10 @@ void vk_dump_init(void) {
|
||||
w[1] = *q == 'x' ? (uint32_t)strtoul(q + 1, &q, 10) : 0;
|
||||
}
|
||||
const char *d = cfg_get("NX2NAVI_DUMP_DIR");
|
||||
snprintf(dir, sizeof dir, "%s", d && *d ? d : "dump");
|
||||
make_dirs(dir);
|
||||
snprintf(base_dir, sizeof base_dir, "%s", d && *d ? d : "dump");
|
||||
snprintf(dir, sizeof dir, "%s", base_dir);
|
||||
char path[600];
|
||||
snprintf(path, sizeof path, "%s/passes.txt", dir);
|
||||
passes_f = fopen(path, "w");
|
||||
snprintf(path, sizeof path, "%s/images.txt", dir);
|
||||
images_f = fopen(path, "w");
|
||||
if (!passes_f || !images_f) rt_fatal("vk_dump: cannot write to %s", dir);
|
||||
if (!dump_white) open_files();
|
||||
const char *bf = cfg_get("NX2NAVI_DUMP_BUF");
|
||||
if (bf && *bf) {
|
||||
char *q;
|
||||
@@ -937,7 +979,43 @@ void vk_dump_init(void) {
|
||||
rt_log("vk_dump: frame %llu goes to %s", (unsigned long long)dump_frame, dir);
|
||||
}
|
||||
|
||||
// The thumbnail of frame f is blank white left of the overlay and above the dialogue box: -1 not available, 0 no, 1 yes.
|
||||
static int thumb_white(uint64_t f) {
|
||||
static uint8_t px[FLIGHT_THUMB_W * FLIGHT_THUMB_H * 4];
|
||||
if (!vk_thumbs_read(f, px)) return -1;
|
||||
int n = 0, bright = 0;
|
||||
for (int y = 0; y < 105; y++)
|
||||
for (int x = 0; x < 250; x++, n++) {
|
||||
const uint8_t *q = px + 4 * (y * FLIGHT_THUMB_W + x);
|
||||
bright += (q[0] * 30 + q[1] * 59 + q[2] * 11) / 100 > 245;
|
||||
}
|
||||
return bright * 100 > n * 97;
|
||||
}
|
||||
|
||||
static void white_check(uint64_t frame) {
|
||||
static int run; // consecutive white thumbnails: white runs are mostly 1-2 frames, too short for a dump 4 frames on
|
||||
if (frame < 1800) return;
|
||||
int w = thumb_white(frame - 3);
|
||||
if (w < 0) return;
|
||||
run = w ? run + 1 : 0;
|
||||
if (run >= 3) { // frames frame-5..frame-3 white: the run is likely still on 4 frames later
|
||||
run = 0;
|
||||
dump_frame = frame + 1;
|
||||
snprintf(dir, sizeof dir, "%s/f%llu", base_dir, (unsigned long long)dump_frame);
|
||||
open_files();
|
||||
rt_log("vk_dump: frames %llu-%llu were white; frame %llu goes to %s", (unsigned long long)(frame - 5),
|
||||
(unsigned long long)(frame - 3), (unsigned long long)dump_frame, dir);
|
||||
}
|
||||
}
|
||||
|
||||
void vk_dump_present(uint64_t frame) {
|
||||
if (dump_white && white_last && frame == white_last + 3) { // was the dumped frame itself white?
|
||||
int w = thumb_white(white_last);
|
||||
rt_log("vk_dump: dumped frame %llu was %s", (unsigned long long)white_last, w > 0 ? "white: done" : "not white: trying again");
|
||||
if (w > 0 || ++white_tries >= 8) white_caught = 1;
|
||||
white_last = 0;
|
||||
}
|
||||
if (dump_white && !white_caught && !white_last && dump_frame == UINT64_MAX) white_check(frame);
|
||||
if (frame != dump_frame) return;
|
||||
plat_mutex_lock(&rec_lock);
|
||||
finished = 1;
|
||||
@@ -952,4 +1030,13 @@ void vk_dump_present(uint64_t frame) {
|
||||
passes_f = NULL;
|
||||
rt_log("vk_dump: frame %llu: %d passes and dispatches, about %llu MB of PNGs in %s", (unsigned long long)frame, seq,
|
||||
(unsigned long long)(png_bytes >> 20), dir);
|
||||
if (dump_white) { // ready for another attempt (vk_dump_present checks this one's thumbnail 3 presents on)
|
||||
white_last = frame;
|
||||
plat_mutex_lock(&rec_lock);
|
||||
seq = copied = 0, last_targets = 0, pass_skip = NULL, png_bytes = 0;
|
||||
nluts = nluts_done = nents = ents_over = pass_draws = 0;
|
||||
dump_frame = UINT64_MAX;
|
||||
finished = 0;
|
||||
plat_mutex_unlock(&rec_lock);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -63,6 +63,7 @@ void vk_exec_frame_written(const tex_t *t) {
|
||||
}
|
||||
|
||||
void vk_exec_frame_log(const tex_t *p, uint64_t frame) {
|
||||
vk_exec_cond_frame_end(frame); // a frame boundary in replay order (present_image runs on the replay thread)
|
||||
if (!(rt_trace & RT_TRACE_GFX)) return;
|
||||
VkExecFrame *f = &vk_exec_frame;
|
||||
int hit = 0;
|
||||
@@ -743,6 +744,13 @@ static void clear_attachment(VkExec *ex, const VkClearAttachment *a) {
|
||||
static void clear_cond_log(const VkExec *ex, const char *what, int index) {
|
||||
static int on = -1, n;
|
||||
if (on < 0) on = rt_knob("LOG_RCOND", "0");
|
||||
if (on == 2 && index < 0 && ex->has_render_cond && ex->depth.width >= 1600) { // full-size depth clears, all of them
|
||||
char why[256];
|
||||
vk_exec_cond_why(&ex->render_cond, why, sizeof why);
|
||||
rt_log("SEQ depth clear %#llx %s; %s", ex->depth.img ? (unsigned long long)ex->depth.img->addr : 0ull,
|
||||
ex->render_disabled ? "skipped" : "RUN", why);
|
||||
return;
|
||||
}
|
||||
if (!on || !ex->has_render_cond || n >= 400) return;
|
||||
n++;
|
||||
const VkExecTarget *t = index < 0 ? &ex->depth : index < (int)ex->pass_ncolor ? &ex->color[index] : NULL;
|
||||
@@ -754,6 +762,7 @@ static void clear_cond_log(const VkExec *ex, const char *what, int index) {
|
||||
static void clear_color(VkExec *ex, const p_clear_color *c) {
|
||||
vk_exec_frame.clears++;
|
||||
clear_cond_log(ex, "colour", c->index);
|
||||
if (ex->has_render_cond) ex->render_disabled ? vk_cond_clears_skip++ : vk_cond_clears_run++;
|
||||
if (ex->render_disabled) return;
|
||||
resolve_rt(ex);
|
||||
int i = c->index;
|
||||
@@ -785,6 +794,7 @@ static void clear_color(VkExec *ex, const p_clear_color *c) {
|
||||
|
||||
static void clear_ds(VkExec *ex, const p_clear_ds *c) {
|
||||
clear_cond_log(ex, "depth", -1);
|
||||
if (ex->has_render_cond) ex->render_disabled ? vk_cond_clears_skip++ : (vk_cond_clears_run++, vk_cond_depth_run++);
|
||||
if (ex->render_disabled) return;
|
||||
resolve_rt(ex);
|
||||
const VkExecTarget *t = &ex->depth;
|
||||
|
||||
@@ -38,6 +38,10 @@ typedef struct {
|
||||
uint8_t scale; // VkTexImage.scale
|
||||
} VkExecTarget;
|
||||
|
||||
void vk_exec_cond_frame_end(uint64_t frame); // vk_exec_xfer.c: a present, in replay order (NX2NAVI_LOG_RCOND)
|
||||
extern unsigned vk_cond_clears_run, vk_cond_clears_skip, vk_cond_draws_skip, vk_cond_depth_run;
|
||||
void vk_exec_cond_why(const p_render_cond *c, char *buf, size_t cap); // NX2NAVI_LOG_RCOND=ab
|
||||
|
||||
// The replay state. One instance lives for the run: NVN state is queue state and carries over from one submission to
|
||||
// the next, as in MacNativeGames' and LAS Recomp's replays. The "per command buffer" fields are reset by vk_exec_record.
|
||||
typedef struct VkExec {
|
||||
|
||||
@@ -1526,13 +1526,33 @@ static void tex_info_full(VkExec *ex, const GfxShaderRes *r, uint64_t h, int sto
|
||||
// rewrites the entries whose slot changed (vk_texture_slot_token) and fills the kinds a new pipeline asked for; with
|
||||
// update-after-bind (descriptorIndexing) a write never invalidates a command buffer being recorded. Each submit's first
|
||||
// sync also looks every registered slot up once, the freshness check a bound texture gets (vk_texture_by_id_locked).
|
||||
// The entries are not double-buffered: a slot registered again while a frame using it is in flight is read new by that
|
||||
// frame, as on the Switch, where the pool is memory the GPU reads as it goes.
|
||||
// The set is one of a ring of POOL_SETS, one per submission: on the Switch the GPU reads the pool as it goes, but it
|
||||
// runs a submission right after nvnQueueSubmitCommands, so what it reads is the pool as of about the submit. Here the
|
||||
// GPU runs a submission several milliseconds later, and Ocarina registers slots for its next submission right after
|
||||
// submitting: with one set, an in-flight submission sampled a texture registered for the next one, the lit materials
|
||||
// of distant characters read the wrong texture as their light or probe data, wrote NaN and bloom spread it (white
|
||||
// dialogue frames, a fifth of them; none once each submission waited for its GPU work, NX2NAVI_SUBMIT_FINISH). A
|
||||
// submission's first pool_sync takes the next set (pool_take_set), waits for the GPU only if that set's last
|
||||
// submission is still running, and brings it up to date from the change log (every entry written since the set was
|
||||
// last current; the whole set when the log has wrapped). Writes within a submission go to its own set at once, as
|
||||
// before.
|
||||
volatile uint32_t vk_pool_kinds_used;
|
||||
#define POOL_SETS 4
|
||||
#define POOL_LOG (1u << 20) // entries of the change log: binding * VK_POOL_TEX + id, one per write
|
||||
static struct {
|
||||
VkDescriptorSetLayout layout, empty;
|
||||
VkDescriptorPool dpool;
|
||||
VkDescriptorSet set;
|
||||
VkDescriptorSet set; // the current submission's (sets[cur])
|
||||
VkDescriptorSet sets[POOL_SETS];
|
||||
uint64_t set_fence[POOL_SETS]; // the timeline value of the last submission that bound it
|
||||
uint64_t set_synced[POOL_SETS]; // the change-log position up to which it holds every write
|
||||
int nsets, cur, all; // all: the first fill, written to every set
|
||||
uint64_t cur_fence; // the submission sets[cur] serves
|
||||
uint32_t *log; // POOL_LOG entries
|
||||
uint64_t logn; // writes so far
|
||||
uint64_t *last; // per entry: the log position of its newest write (catch-up writes it once, there)
|
||||
uint64_t switches, waits, catchup, written, rewrites, catch_ns, sync_ns, n_slots, n_refill, n_fresh;
|
||||
uint64_t dirty_pos; // vk_texture_dirty_slots_locked position (0: the next sync walks every slot)
|
||||
int ready, uab;
|
||||
uint32_t kinds, gen, epoch, sgen;
|
||||
int retry; // the last sync left slots to the creation budget
|
||||
@@ -1567,19 +1587,33 @@ static VkDescriptorSetLayout pool_layout_make(void) {
|
||||
if (vkCreateDescriptorSetLayout(c->device, &li, NULL, &l) != VK_SUCCESS) rt_fatal("vk: the texture pool layout failed");
|
||||
VkDescriptorPoolSize ps[2] = {{VK_DESCRIPTOR_TYPE_SAMPLED_IMAGE, VK_POOL_KINDS * VK_POOL_TEX},
|
||||
{VK_DESCRIPTOR_TYPE_SAMPLER, VK_POOL_SAMP}};
|
||||
VkDescriptorPoolCreateInfo pi = {.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_POOL_CREATE_INFO,
|
||||
.flags = PL.uab ? VK_DESCRIPTOR_POOL_CREATE_UPDATE_AFTER_BIND_BIT : 0,
|
||||
.maxSets = 1, .poolSizeCount = 2, .pPoolSizes = ps};
|
||||
if (vkCreateDescriptorPool(c->device, &pi, NULL, &PL.dpool) != VK_SUCCESS) rt_fatal("vk: the texture pool descriptor pool failed");
|
||||
VkDescriptorSetAllocateInfo ai = {.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_SET_ALLOCATE_INFO, .descriptorPool = PL.dpool,
|
||||
.descriptorSetCount = 1, .pSetLayouts = &l};
|
||||
if (vkAllocateDescriptorSets(c->device, &ai, &PL.set) != VK_SUCCESS) rt_fatal("vk: the texture pool set failed");
|
||||
// POOL_SETS sets, fewer where the device has no room for them (one at the least, as before)
|
||||
for (int n = POOL_SETS; n >= 1 && !PL.nsets; n--) {
|
||||
VkDescriptorPoolSize psn[2] = {{ps[0].type, ps[0].descriptorCount * (uint32_t)n},
|
||||
{ps[1].type, ps[1].descriptorCount * (uint32_t)n}};
|
||||
VkDescriptorPoolCreateInfo pi = {.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_POOL_CREATE_INFO,
|
||||
.flags = PL.uab ? VK_DESCRIPTOR_POOL_CREATE_UPDATE_AFTER_BIND_BIT : 0,
|
||||
.maxSets = (uint32_t)n, .poolSizeCount = 2, .pPoolSizes = psn};
|
||||
if (vkCreateDescriptorPool(c->device, &pi, NULL, &PL.dpool) != VK_SUCCESS) continue;
|
||||
VkDescriptorSetLayout ls[POOL_SETS];
|
||||
for (int i = 0; i < n; i++) ls[i] = l;
|
||||
VkDescriptorSetAllocateInfo ai = {.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_SET_ALLOCATE_INFO, .descriptorPool = PL.dpool,
|
||||
.descriptorSetCount = (uint32_t)n, .pSetLayouts = ls};
|
||||
if (vkAllocateDescriptorSets(c->device, &ai, PL.sets) == VK_SUCCESS) PL.nsets = n;
|
||||
else vkDestroyDescriptorPool(c->device, PL.dpool, NULL), PL.dpool = VK_NULL_HANDLE;
|
||||
}
|
||||
if (!PL.nsets) rt_fatal("vk: the texture pool set failed");
|
||||
PL.set = PL.sets[0];
|
||||
PL.log = malloc(sizeof *PL.log * POOL_LOG);
|
||||
PL.last = calloc((size_t)VK_POOL_KINDS * VK_POOL_TEX + VK_POOL_SAMP, sizeof *PL.last);
|
||||
if (!PL.log || !PL.last) rt_fatal("vk: out of memory");
|
||||
PL.tok = calloc(VK_POOL_TEX, sizeof *PL.tok);
|
||||
PL.view = calloc((size_t)VK_POOL_KINDS * VK_POOL_TEX, sizeof *PL.view);
|
||||
PL.samp = calloc(VK_POOL_SAMP, sizeof *PL.samp);
|
||||
if (!PL.tok || !PL.view || !PL.samp) rt_fatal("vk: out of memory");
|
||||
rt_log("vk: bindless texture pools: %d kinds x %d textures, %d samplers, %s", VK_POOL_KINDS, VK_POOL_TEX, VK_POOL_SAMP,
|
||||
PL.uab ? "update after bind" : "no update-after-bind on this GPU (entries written while bound are undefined)");
|
||||
rt_log("vk: bindless texture pools: %d kinds x %d textures, %d samplers, %s, %d sets (one per submission in turn)",
|
||||
VK_POOL_KINDS, VK_POOL_TEX, VK_POOL_SAMP,
|
||||
PL.uab ? "update after bind" : "no update-after-bind on this GPU (entries written while bound are undefined)", PL.nsets);
|
||||
return l;
|
||||
}
|
||||
|
||||
@@ -1620,16 +1654,83 @@ static void pool_flush(void) {
|
||||
pool_nw = 0;
|
||||
}
|
||||
|
||||
static void pool_write(uint32_t binding, uint32_t id, VkImageView view, VkSampler samp) {
|
||||
static void pool_write_to(VkDescriptorSet set, uint32_t binding, uint32_t id, VkImageView view, VkSampler samp) {
|
||||
if (pool_nw == POOL_WBATCH) pool_flush();
|
||||
pool_ii[pool_nw] = (VkDescriptorImageInfo){samp, view, VK_IMAGE_LAYOUT_GENERAL};
|
||||
pool_w[pool_nw] = (VkWriteDescriptorSet){.sType = VK_STRUCTURE_TYPE_WRITE_DESCRIPTOR_SET, .dstSet = PL.set,
|
||||
pool_w[pool_nw] = (VkWriteDescriptorSet){.sType = VK_STRUCTURE_TYPE_WRITE_DESCRIPTOR_SET, .dstSet = set,
|
||||
.dstBinding = binding, .dstArrayElement = id, .descriptorCount = 1,
|
||||
.descriptorType = view ? VK_DESCRIPTOR_TYPE_SAMPLED_IMAGE : VK_DESCRIPTOR_TYPE_SAMPLER,
|
||||
.pImageInfo = &pool_ii[pool_nw]};
|
||||
pool_nw++;
|
||||
}
|
||||
|
||||
// An entry whose wanted value PL.view / PL.samp already hold: into the current set, and into the log for the others
|
||||
// (the first fill goes into every set).
|
||||
static void pool_write(uint32_t binding, uint32_t id, VkImageView view, VkSampler samp) {
|
||||
if (PL.all) {
|
||||
for (int i = 0; i < PL.nsets; i++) pool_write_to(PL.sets[i], binding, id, view, samp);
|
||||
return;
|
||||
}
|
||||
pool_write_to(PL.set, binding, id, view, samp);
|
||||
uint32_t e = binding * VK_POOL_TEX + id;
|
||||
PL.last[e] = PL.logn;
|
||||
PL.log[PL.logn++ % POOL_LOG] = e;
|
||||
}
|
||||
|
||||
// Writes entry e (binding * VK_POOL_TEX + id) of set as PL.view / PL.samp hold it now.
|
||||
static void pool_entry_to(VkDescriptorSet set, uint32_t e) {
|
||||
uint32_t binding = e / VK_POOL_TEX, id = e % VK_POOL_TEX;
|
||||
if (binding < VK_POOL_KINDS) {
|
||||
VkImageView v = PL.view[(size_t)binding * VK_POOL_TEX + id];
|
||||
if (v) pool_write_to(set, binding, id, v, VK_NULL_HANDLE);
|
||||
} else if (id < VK_POOL_SAMP && PL.samp[id]) {
|
||||
pool_write_to(set, binding, id, VK_NULL_HANDLE, PL.samp[id]);
|
||||
}
|
||||
}
|
||||
|
||||
// A submission's first pool use: the next set of the ring, once the GPU is done with it. 1 when it switched; *from is
|
||||
// the log position the set holds writes up to (pool_catch_up brings it level once pool_sync has refreshed the entries:
|
||||
// a view written into a set must be alive, and the refresh is what replaces the views of destroyed textures).
|
||||
static int pool_take_set(VkExec *ex, uint64_t *from) {
|
||||
if (PL.cur_fence == ex->fence) return 0;
|
||||
int i = (PL.cur + 1) % PL.nsets;
|
||||
if (PL.nsets > 1) {
|
||||
const GfxBackend *b = gfx_backend();
|
||||
if (b->fence_done() < PL.set_fence[i]) PL.waits++, b->fence_wait(PL.set_fence[i], UINT64_MAX);
|
||||
}
|
||||
*from = PL.set_synced[i];
|
||||
PL.cur = i, PL.set = PL.sets[i], PL.cur_fence = ex->fence, PL.set_fence[i] = ex->fence;
|
||||
return PL.nsets > 1;
|
||||
}
|
||||
|
||||
// The current set gets the entries written into other sets in [from, to) of the log, as PL.view / PL.samp hold them now.
|
||||
static void pool_catch_up(uint64_t from, uint64_t to) {
|
||||
VkDescriptorSet set = PL.set;
|
||||
if (to - from > POOL_LOG) { // the log has wrapped since: every entry
|
||||
PL.rewrites++;
|
||||
for (uint32_t e = 0; e < VK_POOL_KINDS * VK_POOL_TEX + VK_POOL_SAMP; e++) pool_entry_to(set, e);
|
||||
} else {
|
||||
// each entry once, at its newest write: an entry written again later (also by this submission's own sync,
|
||||
// which wrote it into this set already) is skipped at the older positions
|
||||
PL.catchup += to - from;
|
||||
for (uint64_t q = from; q < to; q++) {
|
||||
uint32_t e = PL.log[q % POOL_LOG];
|
||||
if (PL.last[e] == q) pool_entry_to(set, e), PL.written++;
|
||||
}
|
||||
}
|
||||
pool_flush();
|
||||
PL.switches++;
|
||||
if (PL.switches % 3600 == 0)
|
||||
rt_log("vk: texture pool sets: %llu switches, %llu waited for the GPU, %.1f log entries and %.1f writes per "
|
||||
"catch-up, %llu whole rewrites; %.3f ms catch-up and %.3f ms entry refresh per switch (%.0f slots, %.0f "
|
||||
"refilled, %.0f checked)",
|
||||
(unsigned long long)PL.switches, (unsigned long long)PL.waits, (double)PL.catchup / (double)PL.switches,
|
||||
(double)PL.written / (double)PL.switches, (unsigned long long)PL.rewrites,
|
||||
PL.catch_ns / 1e6 / (double)PL.switches, PL.sync_ns / 1e6 / (double)PL.switches,
|
||||
(double)PL.n_slots / (double)PL.switches, (double)PL.n_refill / (double)PL.switches,
|
||||
(double)PL.n_fresh / (double)PL.switches);
|
||||
}
|
||||
|
||||
// Entry id of every kind in mask: the slot's view of that kind or the kind's stand-in. Texture lock held. 0: the
|
||||
// slot's texture was left to a later sync by the creation budget (pool_sync).
|
||||
// NX2NAVI_SKIP_SHADERS=<hash>,...: draws and dispatches whose shader's SPIR-V has one of these FNV-1a hashes (the "vs"/"fs"/
|
||||
@@ -1684,8 +1785,38 @@ static int pool_fill(VkExec *ex, uint32_t id, uint32_t mask, uint64_t tok) {
|
||||
return !vk_texture_create_deferred_locked();
|
||||
}
|
||||
|
||||
static void pool_sync_entries(VkExec *ex);
|
||||
|
||||
static void pool_sync(VkExec *ex) {
|
||||
if (!PL.layout) vk_pool_set_layout();
|
||||
uint64_t from = 0, pre = PL.logn;
|
||||
int switched = pool_take_set(ex, &from); // every submission its own set, before the shortcut
|
||||
uint64_t t0 = kern_now_ns();
|
||||
pool_sync_entries(ex);
|
||||
uint64_t t1 = kern_now_ns();
|
||||
PL.sync_ns += t1 - t0;
|
||||
if (switched) {
|
||||
vk_texture_lock();
|
||||
pool_catch_up(from, pre); // what the other sets got since; this sync's own writes went into it already
|
||||
vk_texture_unlock();
|
||||
PL.catch_ns += kern_now_ns() - t1;
|
||||
}
|
||||
PL.set_synced[PL.cur] = PL.logn;
|
||||
}
|
||||
|
||||
typedef struct { VkExec *ex; uint32_t used; int n, retry; } PoolDirty;
|
||||
|
||||
static void pool_dirty_slot(int id, void *arg) { // vk_texture_dirty_slots_locked: a slot whose token may have changed
|
||||
PoolDirty *d = arg;
|
||||
if (id >= d->n) return;
|
||||
uint64_t t = vk_texture_slot_token(id);
|
||||
if (t == PL.tok[id]) return;
|
||||
PL.n_refill++;
|
||||
if (pool_fill(d->ex, (uint32_t)id, d->used, t)) PL.tok[id] = t;
|
||||
else d->retry = 1;
|
||||
}
|
||||
|
||||
static void pool_sync_entries(VkExec *ex) {
|
||||
uint32_t used = __atomic_load_n(&vk_pool_kinds_used, __ATOMIC_RELAXED);
|
||||
uint32_t gen = vk_texture_slots_gen(), ep = vk_texture_epoch(), sg = vk_sampler_gen();
|
||||
if (PL.ready && !PL.retry && used == PL.kinds && gen == PL.gen && ep == PL.epoch && sg == PL.sgen) return;
|
||||
@@ -1694,24 +1825,37 @@ static void pool_sync(VkExec *ex) {
|
||||
int n = vk_texture_slot_count_locked();
|
||||
if (n > VK_POOL_TEX) n = VK_POOL_TEX;
|
||||
if (!PL.ready) { // every entry a stand-in first: partially bound arrays are still read out of range by guests
|
||||
PL.all = 1;
|
||||
for (uint32_t id = 0; id < VK_POOL_TEX; id++) pool_fill(ex, id, 0xffffu, 0);
|
||||
for (uint32_t id = 0; id < VK_POOL_SAMP; id++) PL.samp[id] = Z.samp, pool_write(VK_POOL_KINDS, id, VK_NULL_HANDLE, Z.samp);
|
||||
pool_flush();
|
||||
PL.all = 0;
|
||||
PL.ready = 1, PL.sgen = 0;
|
||||
}
|
||||
int retry = 0; // slots whose texture the creation budget left: filled again by the next sync
|
||||
if (fresh || gen != PL.gen || ep != PL.epoch || PL.retry) {
|
||||
vk_texture_create_budget_locked(POOL_CREATE_BUDGET);
|
||||
for (int id = 0; id < n; id++) { // slots past n were never registered: stand-ins since the first sync
|
||||
// Only a slot change since the last sync (no new kinds, no new submission, nothing left over): the slots that
|
||||
// changed (vk_texture_dirty_slots_locked), not the whole table
|
||||
PoolDirty pd = {ex, used, n, 0};
|
||||
int walked = !fresh && ep == PL.epoch && !PL.retry && PL.dirty_pos &&
|
||||
vk_texture_dirty_slots_locked(&PL.dirty_pos, pool_dirty_slot, &pd);
|
||||
retry = pd.retry;
|
||||
if (!walked) PL.dirty_pos = vk_texture_dirty_pos_locked();
|
||||
for (int id = 0; id < n && !walked; id++) { // slots past n were never registered: stand-ins since the first sync
|
||||
uint64_t t = vk_texture_slot_token(id);
|
||||
if (t != PL.tok[id]) {
|
||||
PL.n_refill++;
|
||||
if (pool_fill(ex, (uint32_t)id, used, t)) PL.tok[id] = t;
|
||||
else retry = 1;
|
||||
} else if (t && fresh) {
|
||||
pool_fill(ex, (uint32_t)id, fresh, t);
|
||||
} else if (t && ep != PL.epoch) {
|
||||
PL.n_fresh++;
|
||||
vk_texture_by_id_locked(id, NULL, NULL); // the submit's freshness check
|
||||
}
|
||||
}
|
||||
if (!walked) PL.n_slots += (uint64_t)n;
|
||||
vk_texture_create_budget_locked(-1);
|
||||
}
|
||||
PL.retry = retry;
|
||||
@@ -2620,7 +2764,7 @@ static int draw_skipped(const VkExec *ex, int mesh, int why) {
|
||||
// mesh: a mesh draw (op_draw_mesh): the program's mesh and task stages stand in for the vertex stage, and nothing
|
||||
// vertex-related (vertex buffers, motion vector bookkeeping of vertex shaders) is bound or noted.
|
||||
static int prepare_draw_ex(VkExec *ex, int prim, int32_t count, int restart, int mesh) {
|
||||
if (ex->render_disabled) return draw_skipped(ex, mesh, SK_DISABLED);
|
||||
if (ex->render_disabled) return vk_cond_draws_skip += ex->has_render_cond, draw_skipped(ex, mesh, SK_DISABLED);
|
||||
cb_res(ex);
|
||||
dummies_init();
|
||||
if (ex->pass_noatt) { // no attachment: the pass covers this draw's viewport or scissor, reopened when it needs more
|
||||
|
||||
@@ -124,9 +124,29 @@ static void counter_cb(void *arg) {
|
||||
// report counts as pending until then, not until its fence passes: the callback runs after the fence, and in between
|
||||
// the record still held what the CPU wrote (Ocarina re-arms its condition records with 1 for the next frame), so a
|
||||
// condition read 1 where the GPU had written 0, a depth clear ran after the depth copy and frames showed only sky.
|
||||
typedef struct { uint64_t addr, seq; volatile uint64_t written; int unknown, type; } Pending;
|
||||
// frame: the present count (vk_exec_cond_frame_end, in replay order) when the report was replayed. The game arms its
|
||||
// condition records from the CPU between frames, so within a frame a timestamp report decides the record's value even
|
||||
// after it reached memory: Ocarina's CPU, running a frame ahead, had re-armed a record with 1 by the time a later depth
|
||||
// clear of the frame read it, the clear ran after the depth copy and the frame showed only sky (white flashes).
|
||||
typedef struct { uint64_t addr, seq, frame; volatile uint64_t written; int unknown, type; } Pending;
|
||||
static Pending pending[PENDING];
|
||||
static uint64_t report_seq;
|
||||
static uint64_t cond_frame;
|
||||
|
||||
// Per-frame counts of conditional work (NX2NAVI_LOG_RCOND=1, vk_exec_cond_frame_end logs them): clears and
|
||||
// draws issued under a render condition, run or skipped.
|
||||
unsigned vk_cond_clears_run, vk_cond_clears_skip, vk_cond_draws_skip, vk_cond_depth_run;
|
||||
|
||||
void vk_exec_cond_frame_end(uint64_t frame) {
|
||||
static int on = -1;
|
||||
if (on < 0) on = rt_knob("LOG_RCOND", "0");
|
||||
if (on && (vk_cond_clears_run || vk_cond_clears_skip || vk_cond_draws_skip))
|
||||
rt_log("vk: frame %llu conditions: clears run %u skipped %u, draws skipped %u, depth clears run %u",
|
||||
(unsigned long long)frame, vk_cond_clears_run, vk_cond_clears_skip, vk_cond_draws_skip, vk_cond_depth_run);
|
||||
vk_cond_clears_run = vk_cond_clears_skip = vk_cond_draws_skip = vk_cond_depth_run = 0;
|
||||
if (on == 2) rt_log("SEQ present %llu", (unsigned long long)frame);
|
||||
cond_frame++;
|
||||
}
|
||||
|
||||
static Pending *pending_slot(uint64_t addr, int insert);
|
||||
|
||||
@@ -149,11 +169,12 @@ static Pending *pending_slot(uint64_t addr, int insert) {
|
||||
if (a == addr) return e;
|
||||
if (!insert) continue;
|
||||
if (!a) { if (!free_) free_ = e; continue; }
|
||||
if (__atomic_load_n(&e->written, __ATOMIC_ACQUIRE) >= e->seq && (!victim || e->seq < victim->seq)) victim = e;
|
||||
if (__atomic_load_n(&e->written, __ATOMIC_ACQUIRE) >= e->seq && e->frame != cond_frame && (!victim || e->seq < victim->seq))
|
||||
victim = e; // done, and not from this frame (whose reports still decide their records)
|
||||
}
|
||||
Pending *e = free_ ? free_ : victim;
|
||||
if (!e) return NULL; // every probed entry still has a report in flight
|
||||
e->seq = 0, e->unknown = 0, e->type = -1;
|
||||
e->seq = 0, e->frame = 0, e->unknown = 0, e->type = -1;
|
||||
__atomic_store_n(&e->written, 0, __ATOMIC_RELEASE);
|
||||
__atomic_store_n(&e->addr, addr, __ATOMIC_RELEASE);
|
||||
return e;
|
||||
@@ -342,7 +363,7 @@ static void report_counter(VkExec *ex, const p_report_counter *p) {
|
||||
vk_completion_add(ex->fence, counter_cb, a);
|
||||
}
|
||||
Pending *e = pending_slot(p->addr, 1);
|
||||
if (e) e->seq = report_seq, e->unknown = samples, e->type = (int)p->type;
|
||||
if (e) e->seq = report_seq, e->frame = cond_frame, e->unknown = samples, e->type = (int)p->type;
|
||||
else LOG_ONCE("vk: a counter report's table neighbourhood is all in flight: conditions on %#llx read memory",
|
||||
(unsigned long long)p->addr);
|
||||
}
|
||||
@@ -352,6 +373,7 @@ static void report_counter(VkExec *ex, const p_report_counter *p) {
|
||||
// 1: *v read, 0: not readable, 2: a samples-passed report not in memory yet (unknown)
|
||||
static int counter_word(uint64_t addr, uint64_t *v) {
|
||||
const Pending *e = pending_slot(addr, 0); // the last report to addr decides
|
||||
if (e && !e->unknown && e->frame == cond_frame) return *v = 0, 1; // a timestamp/statistics report of this frame
|
||||
if (e && __atomic_load_n(&e->written, __ATOMIC_ACQUIRE) < e->seq) return *v = 0, e->unknown ? 2 : 1;
|
||||
VkMemSpan m;
|
||||
if (nvn_gpu_only(addr) || !vk_mem_resolve(addr, sizeof *v, &m)) return 0;
|
||||
@@ -366,6 +388,21 @@ static int last_type(uint64_t addr) {
|
||||
return e ? e->type : -1;
|
||||
}
|
||||
|
||||
// NX2NAVI_LOG_RCOND=ab (diagnosis): the words a condition compares and where each came from: "f" a report replayed
|
||||
// this frame (0), "p" a pending report (0 or unknown), "m" memory, with the newest report's type and frame.
|
||||
void vk_exec_cond_why(const p_render_cond *c, char *buf, size_t cap) {
|
||||
int o = snprintf(buf, cap, "%#llx mode %d:", (unsigned long long)c->addr, c->mode);
|
||||
for (int k = 0; k < 2 && o < (int)cap; k++) {
|
||||
uint64_t a = c->addr + 16 * (uint64_t)k;
|
||||
const Pending *e = pending_slot(a, 0);
|
||||
const char *src = !e ? "m" : !e->unknown && e->frame == cond_frame ? "f"
|
||||
: __atomic_load_n(&e->written, __ATOMIC_ACQUIRE) < e->seq ? "p" : "m";
|
||||
o += snprintf(buf + o, cap - (size_t)o, " %llu %s(type %d, frame %lld of %llu)",
|
||||
(unsigned long long)*(const uint64_t *)(uintptr_t)a, src, e ? e->type : -1,
|
||||
e ? (long long)e->frame : -1LL, (unsigned long long)cond_frame);
|
||||
}
|
||||
}
|
||||
|
||||
static int cond_enabled(const p_render_cond *c) {
|
||||
uint64_t x, y;
|
||||
int rx = counter_word(c->addr, &x), ry = counter_word(c->addr + 16, &y);
|
||||
@@ -546,6 +583,10 @@ static void copy_t2t(VkExec *ex, const p_copy_t2t *p) {
|
||||
}
|
||||
ex->recorded++;
|
||||
vk_texture_gpu_written(p->dst);
|
||||
static int lg = -1;
|
||||
if (lg < 0) lg = rt_knob("LOG_RCOND", "0");
|
||||
if (lg == 2 && (d.img->aspect & VK_IMAGE_ASPECT_DEPTH_BIT) && d.w >= 1600)
|
||||
rt_log("SEQ depth copy %#llx -> %#llx", (unsigned long long)s.img->addr, (unsigned long long)d.img->addr);
|
||||
}
|
||||
|
||||
// ---- buffer to buffer
|
||||
|
||||
@@ -126,11 +126,59 @@ static uint32_t reg_serial;
|
||||
|
||||
static uint32_t agen_idx(uint64_t addr) { return (uint32_t)(addr * 0x9E3779B97F4A7C15ull >> 52); }
|
||||
|
||||
// Slots that may resolve differently since a position, for the texture pool descriptors (vk_exec_draw.c
|
||||
// pool_sync_entries), which otherwise looked at every registered slot after each change: Ocarina registers slots all
|
||||
// through a submission, and the refresh walked the whole table about twenty times per submission (3.5 ms). A ring of
|
||||
// changes, each a slot id (registered) or an address bucket (an entry at that agen index changed: every slot registered
|
||||
// over it, kept in per-bucket lists). Lock held for all of it.
|
||||
#define DIRTY_N (1u << 16)
|
||||
#define DIRTY_BUCKET 0x80000000u
|
||||
static uint32_t dirty[DIRTY_N];
|
||||
static uint64_t ndirty;
|
||||
static int32_t bucket_head[AGEN_N];
|
||||
static int32_t *slot_next, *slot_prev; // per slot id: its bucket list links (slot_prev -2: in no list)
|
||||
static uint16_t *slot_bucket;
|
||||
|
||||
static void dirty_push(uint32_t x) { dirty[ndirty++ % DIRTY_N] = x; }
|
||||
|
||||
static void tex_gen_bump(uint64_t addr) {
|
||||
if (!++tex_gen) tex_gen = 1;
|
||||
slots_gen++;
|
||||
uint32_t *g = &agen[agen_idx(addr)];
|
||||
uint32_t b = agen_idx(addr);
|
||||
uint32_t *g = &agen[b];
|
||||
__atomic_store_n(g, *g + 1, __ATOMIC_RELEASE);
|
||||
dirty_push(DIRTY_BUCKET | b);
|
||||
}
|
||||
|
||||
// Moves slot id into bucket b's list (the lists exist once a slot was registered: vk_texture_register).
|
||||
static void slot_rebucket(int id, uint32_t b) {
|
||||
if (slot_prev[id] != -2) { // out of its old list
|
||||
if (slot_prev[id] >= 0) slot_next[slot_prev[id]] = slot_next[id];
|
||||
else bucket_head[slot_bucket[id]] = slot_next[id];
|
||||
if (slot_next[id] >= 0) slot_prev[slot_next[id]] = slot_prev[id];
|
||||
}
|
||||
slot_bucket[id] = (uint16_t)b;
|
||||
slot_prev[id] = -1;
|
||||
slot_next[id] = bucket_head[b];
|
||||
if (bucket_head[b] >= 0) slot_prev[bucket_head[b]] = id;
|
||||
bucket_head[b] = id;
|
||||
}
|
||||
|
||||
uint64_t vk_texture_dirty_pos_locked(void) { return ndirty; }
|
||||
|
||||
int vk_texture_dirty_slots_locked(uint64_t *pos, void (*fn)(int id, void *arg), void *arg) {
|
||||
if (!slot_next || ndirty - *pos > DIRTY_N) {
|
||||
*pos = ndirty;
|
||||
return 0;
|
||||
}
|
||||
for (uint64_t q = *pos; q < ndirty; q++) {
|
||||
uint32_t x = dirty[q % DIRTY_N];
|
||||
if (x & DIRTY_BUCKET)
|
||||
for (int32_t id = bucket_head[x & ~DIRTY_BUCKET]; id >= 0; id = slot_next[id]) fn(id, arg);
|
||||
else fn((int)x, arg);
|
||||
}
|
||||
*pos = ndirty;
|
||||
return 1;
|
||||
}
|
||||
|
||||
static volatile int32_t tex_ready;
|
||||
@@ -1109,7 +1157,11 @@ void vk_texture_register(int id, const tex_t *t, const view_t *v) {
|
||||
}
|
||||
if (!sinfo) {
|
||||
uint64_t *p = calloc(1 << 20, sizeof *p);
|
||||
if (!p) rt_fatal("vk: out of memory for texture slot tokens");
|
||||
slot_next = malloc(sizeof *slot_next << 20), slot_prev = malloc(sizeof *slot_prev << 20);
|
||||
slot_bucket = calloc(1 << 20, sizeof *slot_bucket);
|
||||
if (!p || !slot_next || !slot_prev || !slot_bucket) rt_fatal("vk: out of memory for texture slot tokens");
|
||||
for (int i = 0; i < (1 << 20); i++) slot_prev[i] = -2, slot_next[i] = -1;
|
||||
for (int i = 0; i < AGEN_N; i++) bucket_head[i] = -1;
|
||||
__atomic_store_n(&sinfo, p, __ATOMIC_RELEASE);
|
||||
}
|
||||
TexSlot *s = &tslots[id];
|
||||
@@ -1119,6 +1171,8 @@ void vk_texture_register(int id, const tex_t *t, const view_t *v) {
|
||||
slots_gen++;
|
||||
if (!++reg_serial) reg_serial = 1;
|
||||
__atomic_store_n(&sinfo[id], (uint64_t)reg_serial << 16 | agen_idx((uintptr_t)t->mem), __ATOMIC_RELEASE);
|
||||
slot_rebucket(id, agen_idx((uintptr_t)t->mem));
|
||||
dirty_push((uint32_t)id);
|
||||
s->has_view = v != NULL;
|
||||
if (v) s->v = *v;
|
||||
else memset(&s->v, 0, sizeof s->v);
|
||||
@@ -1149,6 +1203,11 @@ static Tex *slot_resolve(TexSlot *s, view_t *view, int *has_view) {
|
||||
e = NULL;
|
||||
for (Tex *p = nvn_hm_get(&by_addr, &k.addr); p; p = p->next_at) {
|
||||
if (p->released) continue;
|
||||
// Only an entry of the slot's target: Ocarina's volumetric fog registers a 3D two-channel volume (32x24x64)
|
||||
// over the storage of the bloom pass's 480x270 2D target; followed, the fog computes and the fog pass got
|
||||
// that 2D image, no 3D view can be made of it (vk_exec_draw.c get_view) and they read and wrote stand-ins.
|
||||
// With the target kept they get the volume itself.
|
||||
if (p->key.d.target != k.d.target) continue;
|
||||
if (!any) any = p;
|
||||
if (same_but_swizzle(&p->key, &k)) { e = p; break; }
|
||||
}
|
||||
|
||||
1306
tests/maxwell_warp_emu.py
Normal file
1306
tests/maxwell_warp_emu.py
Normal file
File diff suppressed because it is too large
Load Diff
351
tests/maxwell_warp_emu_selftest.py
Normal file
351
tests/maxwell_warp_emu_selftest.py
Normal file
@@ -0,0 +1,351 @@
|
||||
"""Self-test of tests/maxwell_warp_emu.py: hand-encoded Maxwell sequences against hand-computed results.
|
||||
|
||||
Part 1 runs Maxwell words encoded here from the field layouts of Ryujinx Decoders/InstDecoders.cs (integer and logic
|
||||
ops, SHFL.IDX with clamp and segment masks, LDS/STS sizes, F2F, HFMA2, FFMA rounding, PRMT, predicates, EXIT, BRA).
|
||||
Part 2, when build/xlate/nx2navi_xlate.exe exists, lowers SM86 HMMA.1688 / HMMA.16816, LDSM and MOVM with
|
||||
nx2navi_xlate --lower and checks the emulated result against exactly representable reference values (small
|
||||
integers, so every F16 rounding is exact and only the fragment layout can be wrong), and HFMA2.RELU as the translator
|
||||
runs it (Ryujinx 1.3.2 drops PRMT, so AmpereLowering.cs Half must not use it). tests/tensor_lowering_check.py checks
|
||||
the same lowerings against hardware results.
|
||||
|
||||
python tests/maxwell_warp_emu_selftest.py
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import maxwell_warp_emu as emu # noqa: E402
|
||||
|
||||
PT = 7 << 16 # guard field: predicate PT
|
||||
LANES = 32
|
||||
lane = np.arange(LANES, dtype=np.int64)
|
||||
failures: list[str] = []
|
||||
checks = 0
|
||||
|
||||
|
||||
def check(name: str, got, want):
|
||||
global checks
|
||||
checks += 1
|
||||
got = np.asarray(got).astype(np.int64)
|
||||
want = np.broadcast_to(np.asarray(want).astype(np.int64), got.shape)
|
||||
if not np.array_equal(got, want):
|
||||
bad = np.nonzero(got != want)[0][:6]
|
||||
failures.append(f"{name}: " + ", ".join(f"lane {b} got {int(got[b]):#x} want {int(want[b]):#x}" for b in bad))
|
||||
|
||||
|
||||
# ---- Maxwell encoders (InstDecoders.cs field positions) -----------------------------------------------------------
|
||||
|
||||
def guard(p: int, inv: bool = False) -> int:
|
||||
return (p | (8 if inv else 0)) << 16
|
||||
|
||||
|
||||
def iadd32i(d, a, imm, g=PT): return 0x1C00000000000000 | (imm & 0xFFFFFFFF) << 20 | a << 8 | d | g
|
||||
def mov32i(d, imm, g=PT): return 0x0100000000000000 | 0xF << 12 | (imm & 0xFFFFFFFF) << 20 | d | g
|
||||
def mov(d, s, g=PT): return 0x5C98000000000000 | 0xF << 39 | s << 20 | d | g
|
||||
def s2r(d, sr=0): return 0xF0C8000000000000 | sr << 20 | d | PT
|
||||
def lop32i(d, a, imm, op): return 0x0400000000000000 | op << 53 | (imm & 0xFFFFFFFF) << 20 | a << 8 | d | PT
|
||||
def lop3(d, a, b, c, lut): return 0x5BE0000000000000 | 7 << 48 | c << 39 | lut << 28 | b << 20 | a << 8 | d | PT
|
||||
def iscadd(d, a, b, sh): return 0x5C18000000000000 | sh << 39 | b << 20 | a << 8 | d | PT
|
||||
def iadd3(d, a, b, c, neg_b=False): return 0x5CC0000000000000 | c << 39 | b << 20 | a << 8 | d | (1 << 50 if neg_b else 0) | PT
|
||||
def shl_i(d, a, n): return 0x3848000000000000 | n << 20 | a << 8 | d | PT
|
||||
def shr_i(d, a, n, signed=False): return 0x3828000000000000 | (1 << 48 if signed else 0) | n << 20 | a << 8 | d | PT
|
||||
def shr_r(d, a, b): return 0x5C28000000000000 | b << 20 | a << 8 | d | PT
|
||||
def prmt(d, a, sel, c): return 0x5BC0000000000000 | c << 39 | sel << 20 | a << 8 | d | PT
|
||||
def isetp_i(pd, a, imm, cmp, signed=False): # ISETP.cmp Pd, PT, Ra, imm, PT (AND)
|
||||
return 0x3660000000000000 | cmp << 49 | (1 << 48 if signed else 0) | 7 << 39 | (imm & 0x7FFFF) << 20 | a << 8 | pd << 3 | 7 | PT
|
||||
def shfl(d, a, b, c, mode=0, b_imm=False, c_imm=False, pd=7):
|
||||
return (0xEF10000000000000 | pd << 48 | (c << 34 if c_imm else c << 39) | (1 << 29 if c_imm else 0)
|
||||
| (1 << 28 if b_imm else 0) | mode << 30 | b << 20 | a << 8 | d | PT)
|
||||
def lds(d, a, off, size): return 0xEF48000000000000 | size << 48 | (off & 0xFFFFFF) << 20 | a << 8 | d | PT
|
||||
def sts(v, a, off, size): return 0xEF58000000000000 | size << 48 | (off & 0xFFFFFF) << 20 | a << 8 | v | PT
|
||||
def f2f(d, b, dst, src, high=False): return 0x5CA8000000000000 | dst << 8 | src << 10 | b << 20 | (1 << 41 if high else 0) | d | PT
|
||||
def hfma2(d, a, b, c, neg_c=False, sat=False):
|
||||
return 0x5D00000000000000 | c << 39 | (1 << 30 if neg_c else 0) | (1 << 32 if sat else 0) | b << 20 | a << 8 | d | PT
|
||||
def ffma(d, a, b, c): return 0x5980000000000000 | c << 39 | b << 20 | a << 8 | d | PT
|
||||
def bra(offset): return 0xE24000000000000F | (offset & 0xFFFFFF) << 20 | PT
|
||||
EXIT = 0xE30000000007000F
|
||||
|
||||
|
||||
def f32bits(x): return int(np.array([x], dtype=np.float32).view(np.uint32)[0])
|
||||
def h16bits(x): return int(np.array([x], dtype=np.float16).view(np.uint16)[0])
|
||||
def hpair(lo, hi): return h16bits(lo) | h16bits(hi) << 16
|
||||
|
||||
|
||||
def regs_all(**regs):
|
||||
return {"*": {k: v for k, v in regs.items()}}
|
||||
|
||||
|
||||
def per_lane(reg, values):
|
||||
return {l: {reg: int(v) & 0xFFFFFFFF} for l, v in enumerate(values)}
|
||||
|
||||
|
||||
def merge(*specs):
|
||||
out: dict = {}
|
||||
for s in specs:
|
||||
for k, v in s.items():
|
||||
out.setdefault(k, {}).update(v)
|
||||
return out
|
||||
|
||||
|
||||
# ---- part 1: hand-encoded sequences -------------------------------------------------------------------------------
|
||||
|
||||
def part1():
|
||||
# IADD32I, MOV32I, IADD3 with a negated operand, ISCADD.
|
||||
r = emu.run([mov32i(1, 0x7FFFFFFF), iadd32i(2, 0, 5), iadd32i(3, 1, 1), iadd32i(4, 0, -3),
|
||||
iadd3(5, 0, 2, 1, neg_b=True), iscadd(6, 0, 2, 4), EXIT], per_lane("R0", lane * 3))
|
||||
check("IADD32I", r[:, 2], lane * 3 + 5)
|
||||
check("IADD32I wraps", r[:, 3], 0x80000000)
|
||||
check("IADD32I negative", r[:, 4], (lane.astype(np.int64) * 3 - 3) & 0xFFFFFFFF)
|
||||
check("IADD3 -B", r[:, 5], (lane.astype(np.int64) * 3 - (lane * 3 + 5) + 0x7FFFFFFF) & 0xFFFFFFFF)
|
||||
check("ISCADD", r[:, 6], ((lane.astype(np.int64) * 3) << 4) + lane * 3 + 5)
|
||||
check("RZ stays 0", r[:, 255], 0)
|
||||
|
||||
# S2R lane id, LOP32I, SHL/SHR, LOP3, SHR by register.
|
||||
r = emu.run([s2r(1), lop32i(2, 1, 3, 0), shl_i(3, 1, 3), shr_i(4, 1, 2), lop32i(5, 1, 0xF0, 1),
|
||||
lop32i(6, 1, 0x5, 2), lop3(7, 1, 2, 3, 0x96), mov32i(8, 0x80000000), shr_i(9, 8, 4, signed=True),
|
||||
shr_r(10, 3, 2), EXIT], {})
|
||||
check("S2R SR_LANEID", r[:, 1], lane)
|
||||
check("LOP32I.AND", r[:, 2], lane & 3)
|
||||
check("SHL imm", r[:, 3], lane << 3)
|
||||
check("SHR imm", r[:, 4], lane >> 2)
|
||||
check("LOP32I.OR", r[:, 5], lane | 0xF0)
|
||||
check("LOP32I.XOR", r[:, 6], lane ^ 5)
|
||||
check("LOP3 0x96 (xor3)", r[:, 7], lane ^ (lane & 3) ^ (lane << 3))
|
||||
check("SHR.S32", r[:, 9], 0xF8000000)
|
||||
check("SHR by register", r[:, 10], (lane << 3) >> (lane & 3))
|
||||
|
||||
# SHFL.IDX: the whole warp by a lane register; a segment of 4 lanes; out of range lanes keep their own value.
|
||||
vals = lane * 0x101 + 7
|
||||
seq = [s2r(1), iadd32i(2, 1, 1), lop32i(2, 2, 31, 0),
|
||||
shfl(3, 0, 2, 31, c_imm=True), # (lane + 1) % 32
|
||||
shfl(4, 0, 1, 0x1C03, b_imm=True, c_imm=True), # segments of 4: lane (t & ~3) | 1
|
||||
shfl(5, 0, 5, 0x1803, b_imm=True, c_imm=True, pd=0), # segments of 8, clamp 3: index 5 invalid
|
||||
shfl(6, 0, 2, 0x1803, b_imm=True, c_imm=True, pd=1), # index 2 valid
|
||||
shfl(7, 0, 1, 0x1F, mode=3, b_imm=True, c_imm=True), # BFLY 1
|
||||
shfl(8, 0, 2, 0x00, mode=2, b_imm=True, c_imm=True), # DOWN 2, clamp 0: lanes > 0... invalid
|
||||
shfl(9, 0, 2, 0x1F, mode=2, b_imm=True, c_imm=True), # DOWN 2
|
||||
shfl(10, 0, 1, 0, mode=1, b_imm=True, c_imm=True), # UP 1
|
||||
EXIT]
|
||||
w = emu.run_warp(seq, per_lane("R0", vals))
|
||||
r = w.r
|
||||
check("SHFL.IDX lane register", r[:, 3], vals[(lane + 1) % 32])
|
||||
check("SHFL.IDX segment 4", r[:, 4], vals[(lane & ~3) | 1])
|
||||
check("SHFL.IDX clamp invalid keeps own value", r[:, 5], vals)
|
||||
check("SHFL.IDX clamp invalid predicate", w.p[:, 0], False)
|
||||
check("SHFL.IDX in clamp", r[:, 6], vals[(lane & ~7) | 2])
|
||||
check("SHFL.IDX valid predicate", w.p[:, 1], True)
|
||||
check("SHFL.BFLY", r[:, 7], vals[lane ^ 1])
|
||||
check("SHFL.DOWN clamp 0", r[:, 8], vals)
|
||||
check("SHFL.DOWN", r[:, 9], np.where(lane + 2 <= 31, vals[np.minimum(lane + 2, 31)], vals))
|
||||
check("SHFL.UP", r[:, 10], np.where(lane >= 1, vals[np.maximum(lane.astype(np.int64) - 1, 0)], vals))
|
||||
|
||||
# STS / LDS: 32, 16 and 8 bits, sign extension, 64 and 128 bits, the immediate offset.
|
||||
seq = [s2r(1), shl_i(2, 1, 2), # R2 = 4 * lane
|
||||
sts(1, 2, 0x100, 4), # word [0x100 + 4t] = t
|
||||
lds(3, 2, 0x100, 4),
|
||||
lds(4, 2, 0x100, 2), lds(5, 2, 0x102, 2), lds(6, 2, 0x101, 0),
|
||||
mov32i(7, 0x8001F0F0), sts(7, 255, 0x40, 4), lds(8, 255, 0x40, 3), lds(9, 255, 0x42, 3),
|
||||
lds(10, 255, 0x40, 1), lds(11, 255, 0x41, 0),
|
||||
lds(12, 255, 0x100, 6), # R12-R15 = words 0..3 of the table
|
||||
mov32i(16, 0xABCD), sts(16, 255, 0x82, 2), lds(17, 255, 0x80, 4),
|
||||
EXIT]
|
||||
r = emu.run(seq, {})
|
||||
check("LDS.32", r[:, 3], lane)
|
||||
check("LDS.U16", r[:, 4], lane)
|
||||
check("LDS.U16 high half", r[:, 5], 0)
|
||||
check("LDS.U8 byte 1", r[:, 6], 0)
|
||||
check("LDS.S16", r[:, 8], 0xFFFFF0F0)
|
||||
check("LDS.S16 high", r[:, 9], 0xFFFF8001)
|
||||
check("LDS.S8", r[:, 10], 0xFFFFFFF0)
|
||||
check("LDS.U8", r[:, 11], 0xF0)
|
||||
for k in range(4):
|
||||
check(f"LDS.128 word {k}", r[:, 12 + k], k)
|
||||
check("STS.U16 at +2", r[:, 17], 0xABCD0000)
|
||||
|
||||
# F2F: F32 from either F16 half (exact), F16 from F32 (round to nearest even, high half zero).
|
||||
src = hpair(1.5, -0.333251953125)
|
||||
seq = [mov32i(1, src), f2f(2, 1, 2, 1), f2f(3, 1, 2, 1, high=True),
|
||||
mov32i(4, f32bits(1 / 3)), f2f(5, 4, 1, 2),
|
||||
mov32i(6, f32bits(1 + 2 ** -11)), f2f(7, 6, 1, 2), # tie: to even (1.0)
|
||||
mov32i(8, f32bits(1 + 3 * 2 ** -11)), f2f(9, 8, 1, 2), # tie: to even (1 + 2^-9)
|
||||
mov32i(10, f32bits(70000.0)), f2f(11, 10, 1, 2), # overflow: inf
|
||||
EXIT]
|
||||
r = emu.run(seq, {})
|
||||
check("F2F.F32.F16 H0", r[:, 2], f32bits(1.5))
|
||||
check("F2F.F32.F16 H1", r[:, 3], f32bits(-0.333251953125))
|
||||
check("F2F.F16.F32 1/3", r[:, 5], 0x3555)
|
||||
check("F2F.F16.F32 tie down", r[:, 7], 0x3C00)
|
||||
check("F2F.F16.F32 tie up", r[:, 9], 0x3C02)
|
||||
check("F2F.F16.F32 overflow", r[:, 11], 0x7C00)
|
||||
|
||||
# HFMA2: per half F32 FMA then F16; neg C; saturate.
|
||||
seq = [mov32i(1, hpair(1.5, 2.0)), mov32i(2, hpair(2.0, -0.5)), mov32i(3, hpair(0.25, 1.0)),
|
||||
hfma2(4, 1, 2, 3), hfma2(5, 1, 2, 3, neg_c=True), hfma2(6, 1, 2, 3, sat=True),
|
||||
mov32i(7, hpair(1 + 2 ** -10, 1.0)), mov32i(8, hpair(1 + 2 ** -10, 2 ** -11)), mov32i(9, hpair(0.0, 1.0)),
|
||||
hfma2(10, 7, 8, 9), EXIT]
|
||||
r = emu.run(seq, {})
|
||||
check("HFMA2", r[:, 4], hpair(3.25, 0.0))
|
||||
check("HFMA2 -C", r[:, 5], hpair(2.75, -2.0))
|
||||
check("HFMA2.SAT", r[:, 6], hpair(1.0, 0.0))
|
||||
# (1 + 2^-10)^2 = 1 + 2^-9 + 2^-20 -> F16 1 + 2^-9; 1 + 2^-11 is an F16 tie -> 1.0.
|
||||
check("HFMA2 rounding", r[:, 10], hpair(1 + 2 ** -9, 1.0))
|
||||
|
||||
# Ryujinx Optimizer.PropagatePack: an F16 result unpacked again without an integer operation in between is the
|
||||
# unrounded F32 value (directly or through a MOV); LOP in between, or a guarded pack, keeps the F16 rounding.
|
||||
third, third16 = f32bits(1 / 3), f32bits(float(np.float16(1 / 3)))
|
||||
seq = [mov32i(1, third), f2f(2, 1, 1, 2), f2f(3, 2, 2, 1), # pack, unpack: folded
|
||||
mov(4, 2), f2f(5, 4, 2, 1), # through a MOV: folded
|
||||
lop32i(6, 2, 0xFFFFFFFF, 0), f2f(7, 6, 2, 1), # through LOP: rounded
|
||||
s2r(8), isetp_i(0, 8, 32, 1), # P0 = lane < 32, true everywhere
|
||||
f2f(9, 1, 1, 2) & ~(0xF << 16) | guard(0), f2f(10, 9, 2, 1), # guarded pack: rounded
|
||||
mov32i(11, hpair(40000.0, 1.0)), mov32i(12, hpair(2.0, 2.0)), mov32i(13, 0), mov32i(16, hpair(0.5, 0.5)),
|
||||
hfma2(14, 11, 12, 13), hfma2(15, 14, 16, 13), # 40000 * 2 = inf as F16, * 0.5 again
|
||||
EXIT]
|
||||
r = emu.run(seq, {})
|
||||
check("F2F.F16 then F2F.F32: unrounded (PropagatePack)", r[:, 3], third)
|
||||
check("... through MOV: unrounded", r[:, 5], third)
|
||||
check("... through LOP: rounded", r[:, 7], third16)
|
||||
check("... guarded pack: rounded", r[:, 10], third16)
|
||||
check("HFMA2 overflow stored as F16 inf", r[:, 14], hpair(np.inf, 2.0))
|
||||
check("HFMA2 into HFMA2: the unrounded 80000 halved", r[:, 15], hpair(40000.0, 1.0))
|
||||
|
||||
# FFMA: rounded once. Exact a*b + c lies just below an F32 halfway point; F64 alone would round to the tie and
|
||||
# then to even (0x3f800002), the fused result is 0x3f800001.
|
||||
seq = [mov32i(1, 0x39800001), mov32i(2, 0x397FFFFE), mov32i(3, 0x3F800001), ffma(4, 1, 2, 3),
|
||||
mov32i(5, f32bits(3.0)), mov32i(6, f32bits(0.5)), ffma(7, 5, 6, 5), EXIT]
|
||||
r = emu.run(seq, {})
|
||||
check("FFMA single rounding at a tie", r[:, 4], 0x3F800001)
|
||||
check("FFMA", r[:, 7], f32bits(4.5))
|
||||
|
||||
# PRMT: a no-op as Ryujinx translates it, the Maxwell byte permute otherwise (0xbb99: sign of bytes 1 and 3).
|
||||
seq = [mov32i(1, 0x7F80C000), mov32i(2, 0xBB99), mov32i(3, 0x11111111), prmt(3, 1, 2, 255),
|
||||
mov32i(4, 0x3210), mov32i(5, 0x22222222), prmt(5, 1, 4, 255), EXIT]
|
||||
r = emu.run(seq, {}, prmt="ryujinx")
|
||||
check("PRMT (Ryujinx: no-op)", r[:, 3], 0x11111111)
|
||||
r = emu.run(seq, {}, prmt="maxwell")
|
||||
check("PRMT sign replicate (Maxwell)", r[:, 3], 0x0000FFFF)
|
||||
check("PRMT identity (Maxwell)", r[:, 5], 0x7F80C000)
|
||||
|
||||
# Predicates: ISETP, a guarded write, an inverted guard, EXIT of some lanes, a uniform BRA over one instruction.
|
||||
seq = [s2r(1), isetp_i(0, 1, 16, 1), # P0 = lane < 16
|
||||
mov32i(2, 0xAAAA, g=guard(0)), mov32i(3, 0xBBBB, g=guard(0, inv=True)),
|
||||
bra(emu.slot_address(6) - emu.slot_address(4) - 8), mov32i(4, 0xDEAD), # skipped
|
||||
mov32i(5, 0x5555),
|
||||
(EXIT & ~(0xF << 16)) | guard(0, inv=True), # lanes >= 16 exit
|
||||
mov32i(6, 0x6666), EXIT]
|
||||
w = emu.run_warp(seq, regs_all(R2=1, R3=1, R6=1))
|
||||
check("ISETP.LT", w.p[:, 0], lane < 16)
|
||||
check("guarded write", w.r[:, 2], np.where(lane < 16, 0xAAAA, 1))
|
||||
check("inverted guard", w.r[:, 3], np.where(lane < 16, 1, 0xBBBB))
|
||||
check("BRA skips", w.r[:, 4], 0)
|
||||
check("after BRA", w.r[:, 5], 0x5555)
|
||||
check("guarded EXIT", w.r[:, 6], np.where(lane < 16, 0x6666, 1))
|
||||
|
||||
# Unknown opcodes raise with the word.
|
||||
global checks
|
||||
checks += 1
|
||||
try:
|
||||
emu.run([0xFFFFFFFFFFFFFFFF], {})
|
||||
failures.append("unknown opcode did not raise")
|
||||
except emu.UnknownOpcode as e:
|
||||
if "0xffffffffffffffff" not in str(e):
|
||||
failures.append(f"unknown opcode message: {e}")
|
||||
|
||||
|
||||
# ---- part 2: lowered SM86 tensor instructions ---------------------------------------------------------------------
|
||||
|
||||
def hmma_case(k16: bool, rng):
|
||||
"""HMMA.{1688,16816}.F16 R4, R8, R12, R16 with small integers; returns (sm86, regs_in, expected D regs)."""
|
||||
K = 16 if k16 else 8
|
||||
A = rng.integers(-3, 4, size=(16, K)).astype(np.float64)
|
||||
B = rng.integers(-3, 4, size=(K, 8)).astype(np.float64)
|
||||
C = rng.integers(-8, 9, size=(16, 8)).astype(np.float64)
|
||||
D = A @ B + C
|
||||
regs_in, want = {}, {}
|
||||
for t in range(LANES):
|
||||
g, q = t // 4, t % 4
|
||||
regs = {}
|
||||
for j in range(K // 8):
|
||||
for r in range(2):
|
||||
regs[f"R{8 + 2 * j + r}"] = hpair(A[g + 8 * r, 8 * j + 2 * q], A[g + 8 * r, 8 * j + 2 * q + 1])
|
||||
regs[f"R{12 + j}"] = hpair(B[8 * j + 2 * q, g], B[8 * j + 2 * q + 1, g])
|
||||
for r in range(2):
|
||||
regs[f"R{16 + r}"] = hpair(C[g + 8 * r, 2 * q], C[g + 8 * r, 2 * q + 1])
|
||||
regs_in[t] = regs
|
||||
want[t] = {4 + r: hpair(D[g + 8 * r, 2 * q], D[g + 8 * r, 2 * q + 1]) for r in range(2)}
|
||||
w0 = 0x23C | 7 << 12 | 4 << 16 | 8 << 24
|
||||
sm86 = [w0, 12, 16 | (1 << 11 if k16 else 0), 0]
|
||||
return sm86, regs_in, want
|
||||
|
||||
|
||||
def part2(exe: Path):
|
||||
rng = np.random.default_rng(86)
|
||||
for k16 in (False, True):
|
||||
sm86, regs_in, want = hmma_case(k16, rng)
|
||||
words, urbase, _ = emu.lower(sm86, exe)
|
||||
r = emu.run(words, regs_in)
|
||||
for reg in (4, 5):
|
||||
check(f"HMMA.{'16816' if k16 else '1688'} D R{reg} (lowered, {len(words)} words)",
|
||||
r[:, reg], [want[t][reg] for t in range(LANES)])
|
||||
|
||||
# LDSM.16.M88.4 R4, [R1] and LDSM.16.MT88.1 R8, [R1]: matrix m row r at 0x200 * m + 0x20 * r (16 bytes used),
|
||||
# element (m, row, col) = 0x1000 * m + 0x100 * row + col.
|
||||
smem = bytearray(0x1000)
|
||||
for m in range(4):
|
||||
for row in range(8):
|
||||
for col in range(8):
|
||||
at = 0x200 * m + 0x20 * row + 2 * col
|
||||
smem[at:at + 2] = (0x1000 * m + 0x100 * row + col).to_bytes(2, "little")
|
||||
addr = {t: {"R1": 0x200 * (t // 8) + 0x20 * (t % 8)} for t in range(LANES)}
|
||||
el = lambda m, row, col: 0x1000 * m + 0x100 * row + col
|
||||
for transpose, size, dst in ((False, 2, 4), (True, 0, 8), (True, 2, 12), (False, 0, 16)):
|
||||
sm86 = [0x83B | 7 << 12 | dst << 16 | 1 << 24, 0, size << 8 | (0x4000 if transpose else 0), 0]
|
||||
words, _, _ = emu.lower(sm86, exe)
|
||||
r = emu.run(words, addr, bytes(smem))
|
||||
for m in range(1 << size):
|
||||
if transpose: # column t/4 of rows 2(t%4) (low) and 2(t%4)+1 (high)
|
||||
want = [el(m, 2 * (t % 4), t // 4) | el(m, 2 * (t % 4) + 1, t // 4) << 16 for t in range(LANES)]
|
||||
else: # row t/4, columns 2(t%4), 2(t%4)+1
|
||||
want = [el(m, t // 4, 2 * (t % 4)) | el(m, t // 4, 2 * (t % 4) + 1) << 16 for t in range(LANES)]
|
||||
check(f"LDSM.16.{'MT88' if transpose else 'M88'}.{1 << size} matrix {m}", r[:, dst + m], want)
|
||||
|
||||
# MOVM.16.MT88 R2, R1: element (row, col) = 8 * row + col at lane 4 row + col / 2, half col % 2.
|
||||
regs_in = {t: {"R1": (8 * (t // 4) + 2 * (t % 4)) | (8 * (t // 4) + 2 * (t % 4) + 1) << 16} for t in range(LANES)}
|
||||
words, _, _ = emu.lower([0x23A | 7 << 12 | 2 << 16 | 1 << 24, 0, 0, 0], exe)
|
||||
r = emu.run(words, regs_in)
|
||||
want = [(8 * (2 * (t % 4)) + t // 4) | (8 * (2 * (t % 4) + 1) + t // 4) << 16 for t in range(LANES)]
|
||||
check("MOVM.16.MT88", r[:, 2], want)
|
||||
|
||||
# HFMA2.RELU R0, R1, R2, R3 (PT) as the translator runs it (no PRMT: AmpereLowering.cs Half builds the mask with
|
||||
# integer operations), whatever the scratch registers held before: negative sums become +0, positive ones stay.
|
||||
sm86 = [0x231 | 7 << 12 | 0 << 16 | 1 << 24, 2, 3 | 0x8000 | 0x03800000, 0]
|
||||
words, _, _ = emu.lower(sm86, exe)
|
||||
for a, b, c, want in (((1.0, -2.0), (-3.0, 0.5), (1.0, 0.25), (0.0, 0.0)), # sums -2.0, -0.75
|
||||
((1.0, 2.0), (3.0, -0.5), (1.0, 0.25), (4.0, 0.0))): # sums 4.0, -0.75
|
||||
regs_in = regs_all(R1=hpair(*a), R2=hpair(*b), R3=hpair(*c))
|
||||
for fill in (0, 0x7FC0DEAD):
|
||||
w = emu.run_warp(words, regs_in, prmt="ryujinx", fill=fill)
|
||||
check(f"HFMA2.RELU -> {want} as translated (fill {fill:#x})", w.r[:, 0], hpair(*want))
|
||||
check(f"HFMA2.RELU uses no PRMT (fill {fill:#x})", [len(w.warnings)], 0)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
part1()
|
||||
exe = emu.DEFAULT_XLATE
|
||||
if exe.exists():
|
||||
part2(exe)
|
||||
else:
|
||||
print(f" skip: part 2 needs {exe}")
|
||||
if failures:
|
||||
print("\n".join("FAIL " + f for f in failures))
|
||||
print(f"{checks - len(failures)}/{checks} checks passed")
|
||||
return 1 if failures else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
441
tests/tensor_lowering_check.py
Normal file
441
tests/tensor_lowering_check.py
Normal file
@@ -0,0 +1,441 @@
|
||||
"""Checks the lowered SM86 tensor instructions against results captured on an NVIDIA GPU, in one command.
|
||||
|
||||
tools/xlate/AmpereLowering.cs rewrites the SM86 instructions of the game's neural-network passes as Maxwell code:
|
||||
Lowerer.Hmma (HMMA.1688 / HMMA.16816, .F16 and .F32), Lowerer.Ldsm (LDSM.16.M88 / MT88 .1/.2/.4), Lowerer.Movm
|
||||
(MOVM.16.MT88) and Lowerer.Half (HFMA2, HFMA2.RELU). This script lowers every case of the vector file with
|
||||
`nx2navi_xlate --lower`, runs the Maxwell words on tests/maxwell_warp_emu.py (which executes them as the Ryujinx
|
||||
translator does) and compares with the registers the GPU wrote. Exit status 1 on any failure. The vector file names
|
||||
the GPU that produced it ("device"). So far that is an RTX 5090 running the compute_86 PTX through the driver's JIT,
|
||||
as no SM86 part was at hand: the results follow the PTX semantics but need not show SM86's internal precision.
|
||||
|
||||
C:/Users/PC/AppData/Local/Programs/Python/Python312/python tests/tensor_lowering_check.py [vectors.json]
|
||||
[--xlate EXE] [--case NAME] [-v]
|
||||
|
||||
Defaults: vectors.json is <workspace>/work/tensor-gt/vectors.json (<workspace>/work/tensor-gt/gen_vectors.py writes
|
||||
it: each case is one SM86 instruction in inline PTX, run on one warp), the translator
|
||||
<workspace>/build/xlate/nx2navi_xlate.exe (python tools/build_xlate.py builds it); <workspace> is the directory two
|
||||
levels above this repository. numpy is needed.
|
||||
|
||||
Every case runs several times and must pass each run:
|
||||
* with the registers the vectors do not set first 0, then 0x7fc0dead, so a read of a register the lowering never
|
||||
wrote shows;
|
||||
* then under @P0 (the guard field of the SM86 word set to P0) with P0 false in every third lane: lanes with P0 get
|
||||
the GPU's result, the others keep their old destination values. The other lanes still supply their fragments,
|
||||
rows and addresses, which works only if the shuffles run in every lane and just the destination writes take the
|
||||
guard, the rule AmpereLowering.cs follows.
|
||||
Each run checks:
|
||||
* destination registers bit for bit, except that a NaN matches any NaN in a float result (maxwell_warp_emu
|
||||
same_value: SPIR-V leaves NaN payloads to the host, the GPU writes 0x7fff). HMMA gets no tolerance: the GPU rounds
|
||||
every element once (f16 or f32 of the exact A*B+C), and so does the lowering for these values, whose sums are
|
||||
exact in F32; the largest difference in ulp is reported;
|
||||
* every other register R0-R236 (guest and uniform registers; R237 up is the lowering's own) and P0-P6 keep their
|
||||
values, shared memory is untouched, and the emulator has nothing to report (a shuffle reading a lane its guard
|
||||
left out, a PRMT the translator drops);
|
||||
* urbase, the register the lowering mapped UR0 to, is the highest register the instruction names plus two, rounded
|
||||
down to even: AmpereLowering HighestGpr counted every register of the instruction's fragments and quads.
|
||||
Read off the lowered words (both lowerings): every write to a guest register carries the instruction's guard and no
|
||||
SHFL is guarded.
|
||||
Derived cases, beyond the hardware vectors, for forms the game uses: reference models of HFMA2(.RELU) and LDSM, each
|
||||
first checked against every hardware case of its instruction, give the expected values of HFMA2.RELU with an
|
||||
immediate B (opcode 0x831, as in the game's HFMA2.RELU Rd, Ra, 1, 1, Rc) and LDSM .M88.4 / .MT88.4 with an offset.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from fractions import Fraction
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import maxwell_warp_emu as emu # noqa: E402
|
||||
|
||||
LANES = 32
|
||||
RZ = 255
|
||||
GUEST_REGS = 237 # R0-R236: guest and uniform registers; AmpereLowering.cs uses R237 up itself
|
||||
FILLS = (0x00000000, 0x7FC0DEAD)
|
||||
EXIT = 0xE30000000007000F # Maxwell EXIT: runs no code, gives the emulator's initial state
|
||||
DEFAULT_VECTORS = emu.ROOT.parent.parent / "work" / "tensor-gt" / "vectors.json"
|
||||
|
||||
|
||||
def guard_lanes() -> np.ndarray:
|
||||
"""P0 for the guarded run: false in lanes 1, 4, 7, ... so every quad and row group mixes both."""
|
||||
return np.arange(LANES) % 3 != 1
|
||||
|
||||
|
||||
# ---- the instructions of the vector file ----------------------------------------------------------------------
|
||||
|
||||
def words_of(sm86) -> list[int]:
|
||||
return [emu.parse_int(x) & 0xFFFFFFFF for x in sm86]
|
||||
|
||||
|
||||
def operands(sm86) -> tuple[list[int], list[int]]:
|
||||
"""(destination registers, source registers) one SM86 instruction names, from its fields (work/tensor-gt/README.md
|
||||
"Encoding bits"; the same fields AmpereLowering.cs decodes)."""
|
||||
w0, w1, w2, _ = words_of(sm86)
|
||||
op = w0 & 0xFFF
|
||||
rd, ra, rb, rc = (w0 >> 16) & 0xFF, (w0 >> 24) & 0xFF, w1 & 0xFF, w2 & 0xFF
|
||||
|
||||
def regs(base, n):
|
||||
return [] if base == RZ else list(range(base, base + n))
|
||||
if op == 0x23C: # HMMA: shape bit 75, .F32 bit 76
|
||||
k16, f32 = (w2 >> 11) & 1, (w2 >> 12) & 1
|
||||
cd = 4 if f32 else 2
|
||||
return regs(rd, cd), regs(ra, 4 if k16 else 2) + regs(rb, 2 if k16 else 1) + regs(rc, cd)
|
||||
if op == 0x83B: # LDSM: count 1 << bits 72-73
|
||||
return regs(rd, 1 << ((w2 >> 8) & 3)), regs(ra, 1)
|
||||
if op == 0x23A: # MOVM
|
||||
return regs(rd, 1), regs(ra, 1)
|
||||
if op in (0x231, 0x831): # HFMA2: B a register (0x231) or an immediate (0x831)
|
||||
return regs(rd, 1), regs(ra, 1) + (regs(rb, 1) if op == 0x231 else []) + regs(rc, 1)
|
||||
raise ValueError(f"no operand table for SM86 opcode 0x{op:03x}")
|
||||
|
||||
|
||||
def expected_urbase(sm86) -> int:
|
||||
dests, srcs = operands(sm86)
|
||||
return (max(dests + srcs + [0]) + 2) & ~1
|
||||
|
||||
|
||||
def shared_memory(case: dict) -> bytes | None:
|
||||
"""The case's shared memory as maxwell_warp_emu.run_case places it: smem at smem_base, 64 KiB at least."""
|
||||
if case.get("smem") is None:
|
||||
return None
|
||||
base = emu.parse_int(case.get("smem_base", 0))
|
||||
data = emu.parse_bytes(case["smem"])
|
||||
buf = bytearray(max(0x10000, base + len(data)))
|
||||
buf[base:base + len(data)] = data
|
||||
return bytes(buf)
|
||||
|
||||
|
||||
def consistency(case: dict) -> list[str]:
|
||||
"""Problems of the case itself: regs_out not exactly the destinations, LDSM addresses that disagree."""
|
||||
problems = []
|
||||
dests, _ = operands(case["sm86"])
|
||||
for lane, regs in emu.load_regs(case["regs_out"]).items():
|
||||
if sorted(n for kind, n in regs if kind == "R") != sorted(dests):
|
||||
problems.append(f"lane {lane}: regs_out has {sorted(n for _, n in regs)}, the instruction writes {dests}")
|
||||
break
|
||||
if case.get("lane_addresses") is not None:
|
||||
ra = (words_of(case["sm86"])[0] >> 24) & 0xFF
|
||||
base = emu.parse_int(case.get("smem_base", 0))
|
||||
regs_in = emu.load_regs(case["regs_in"])
|
||||
for lane, a in enumerate(case["lane_addresses"]):
|
||||
given = regs_in[lane].get(("R", ra))
|
||||
if given is not None and given != (emu.parse_int(a) + base) & 0xFFFFFFFF:
|
||||
problems.append(f"lane {lane}: R{ra} = {given:#x} but lane_addresses says {emu.parse_int(a) + base:#x}")
|
||||
break
|
||||
return problems
|
||||
|
||||
|
||||
# ---- running and comparing --------------------------------------------------------------------------------------
|
||||
|
||||
class Outcome:
|
||||
def __init__(self):
|
||||
self.problems: list[str] = []
|
||||
self.worst_ulp = 0
|
||||
self.nan_matches = 0
|
||||
self.runs = 0
|
||||
|
||||
|
||||
def check_run(out: Outcome, label: str, words, case: dict, fill: int, guard: np.ndarray | None, verbose: bool):
|
||||
"""One run of the lowered words; appends what is wrong to out.problems."""
|
||||
sm86 = case["sm86"]
|
||||
dests, _ = operands(sm86)
|
||||
kind = emu.result_kind(sm86)
|
||||
regs_in = emu.load_regs(case["regs_in"])
|
||||
if case.get("lane_addresses") is not None:
|
||||
ra = (words_of(sm86)[0] >> 24) & 0xFF
|
||||
base = emu.parse_int(case.get("smem_base", 0))
|
||||
for lane, a in enumerate(case["lane_addresses"]):
|
||||
regs_in[lane].setdefault(("R", ra), (emu.parse_int(a) + base) & 0xFFFFFFFF)
|
||||
if guard is not None:
|
||||
for lane in range(LANES):
|
||||
regs_in[lane][("P", 0)] = bool(guard[lane])
|
||||
expected = emu.load_regs(case["regs_out"])
|
||||
smem = shared_memory(case)
|
||||
start = emu.run_warp([EXIT], regs_in, smem, fill=fill)
|
||||
warp = emu.run_warp(words, regs_in, smem, fill=fill)
|
||||
out.runs += 1
|
||||
where = f"{label}, fill {fill:#010x}"
|
||||
bad = []
|
||||
for lane in range(LANES):
|
||||
on = guard is None or guard[lane]
|
||||
for n in dests:
|
||||
got = int(warp.r[lane, n])
|
||||
want = expected[lane][("R", n)] if on else int(start.r[lane, n])
|
||||
k = kind if on else "bits"
|
||||
if k != "bits":
|
||||
out.worst_ulp = max(out.worst_ulp, emu.ulps(got, want, k))
|
||||
if not emu.same_value(got, want, k):
|
||||
bad.append(f"lane {lane:2d} R{n}: got {emu._fmt(got)} want {emu._fmt(want & 0xFFFFFFFF)}"
|
||||
+ ("" if on else " (guard false: old value)"))
|
||||
elif got != want:
|
||||
out.nan_matches += 1
|
||||
if bad:
|
||||
out.problems.append(f"{where}: {len(bad)} wrong destination values")
|
||||
out.problems += [" " + b for b in bad[:12 if verbose else 4]]
|
||||
keep = [n for n in range(GUEST_REGS) if n not in dests]
|
||||
lanes, cols = np.nonzero(warp.r[:, keep] != start.r[:, keep])
|
||||
if len(lanes):
|
||||
regs = sorted({keep[c] for c in cols})
|
||||
out.problems.append(f"{where}: registers outside the destination changed: R{regs[:8]} in lanes {sorted(set(lanes.tolist()))[:8]}")
|
||||
if (warp.p[:, :7] != start.p[:, :7]).any():
|
||||
out.problems.append(f"{where}: guest predicates P0-P6 changed")
|
||||
if bytes(warp.smem) != bytes(start.smem):
|
||||
out.problems.append(f"{where}: shared memory changed")
|
||||
for m in dict.fromkeys(warp.warnings):
|
||||
out.problems.append(f"{where}: emulator: {m}")
|
||||
|
||||
|
||||
# Maxwell instructions of maxwell_warp_emu.TABLE without a destination register in bits 0-7.
|
||||
NO_REGISTER_RESULT = {"EXIT", "BRA", "NOP", "STS", "STL", "ISETP", "ISETP.I", "R2P", "R2P.I"}
|
||||
|
||||
|
||||
def guard_rule_problems(words, guard: int, label: str) -> list[str]:
|
||||
"""AmpereLowering.cs's guard rule read off the lowered words: a write to a guest register (below R237) carries the
|
||||
SM86 instruction's guard (Maxwell guard field `guard`), a shuffle runs unguarded, in every lane."""
|
||||
problems = []
|
||||
for k, w in enumerate(words):
|
||||
name, _ = emu.decode(w)
|
||||
g = (w >> 16) & 0xF
|
||||
if name == "SHFL" and g != 7:
|
||||
problems.append(f"{label}: word {k} {w:016x}: SHFL under a guard")
|
||||
if name not in NO_REGISTER_RESULT and (w & 0xFF) < GUEST_REGS and g != guard:
|
||||
problems.append(f"{label}: word {k} {w:016x}: {name} writes guest register R{w & 0xFF} without the instruction's guard")
|
||||
return problems
|
||||
|
||||
|
||||
def check_case(case: dict, exe: Path, verbose: bool = False) -> Outcome:
|
||||
out = Outcome()
|
||||
out.problems += [f"vector file: {p}" for p in consistency(case)]
|
||||
try:
|
||||
words, urbase, _ = emu.lower(case["sm86"], exe)
|
||||
except RuntimeError as e:
|
||||
out.problems.append(str(e))
|
||||
return out
|
||||
if urbase != expected_urbase(case["sm86"]):
|
||||
out.problems.append(f"urbase {urbase}, expected {expected_urbase(case['sm86'])}: HighestGpr misses a register range")
|
||||
out.problems += guard_rule_problems(words, (words_of(case["sm86"])[0] >> 12) & 0xF, "lowered words")
|
||||
for fill in FILLS:
|
||||
check_run(out, "all lanes", words, case, fill, None, verbose)
|
||||
# The same instruction under @P0: SM86 guard bits 12-15 = 0.
|
||||
w = words_of(case["sm86"])
|
||||
guarded = dict(case, sm86=[w[0] & ~0xF000] + w[1:])
|
||||
gwords, _, _ = emu.lower(guarded["sm86"], exe)
|
||||
out.problems += guard_rule_problems(gwords, 0, "lowered words under @P0")
|
||||
check_run(out, "@P0", gwords, guarded, FILLS[1], guard_lanes(), verbose)
|
||||
return out
|
||||
|
||||
|
||||
# ---- reference models for the derived cases ------------------------------------------------------------------------
|
||||
|
||||
def half_value(h: int):
|
||||
"""An F16 bit pattern as ("nan"), ("inf", sign) or ("num", Fraction, sign)."""
|
||||
sign = h >> 15
|
||||
e, m = (h >> 10) & 0x1F, h & 0x3FF
|
||||
if e == 0x1F:
|
||||
return ("nan",) if m else ("inf", sign)
|
||||
v = Fraction(m, 1 << 24) if e == 0 else Fraction(0x400 | m, 1 << 25) * (Fraction(2) ** e)
|
||||
return ("num", -v if sign else v, sign)
|
||||
|
||||
|
||||
def round_half(x: Fraction) -> int:
|
||||
"""x (not zero) rounded to F16, to nearest even; beyond the range infinity."""
|
||||
sign = 0x8000 if x < 0 else 0
|
||||
m = abs(x)
|
||||
e = m.numerator.bit_length() - m.denominator.bit_length()
|
||||
if Fraction(2) ** e > m:
|
||||
e -= 1
|
||||
elif Fraction(2) ** (e + 1) <= m:
|
||||
e += 1
|
||||
quantum = Fraction(2) ** (max(e, -14) - 10)
|
||||
q = m / quantum
|
||||
n = q.numerator // q.denominator
|
||||
rest = q - n
|
||||
if rest > Fraction(1, 2) or (rest == Fraction(1, 2) and n & 1):
|
||||
n += 1
|
||||
value = n * quantum
|
||||
if value >= 65536:
|
||||
return sign | 0x7C00
|
||||
return sign | int(np.array([float(value)], dtype=np.float64).astype(np.float16).view(np.uint16)[0])
|
||||
|
||||
|
||||
def fma_half(a: int, b: int, c: int) -> int:
|
||||
"""F16 a * b + c rounded once (IEEE 754 fusedMultiplyAdd, round to nearest even); a NaN result is 0x7fff."""
|
||||
va, vb, vc = half_value(a), half_value(b), half_value(c)
|
||||
if "nan" in (va[0], vb[0], vc[0]):
|
||||
return 0x7FFF
|
||||
zero = lambda v: v[0] == "num" and v[1] == 0
|
||||
if (va[0] == "inf" and zero(vb)) or (vb[0] == "inf" and zero(va)):
|
||||
return 0x7FFF
|
||||
psign = (a >> 15) ^ (b >> 15)
|
||||
if va[0] == "inf" or vb[0] == "inf":
|
||||
if vc[0] == "inf" and vc[1] != psign:
|
||||
return 0x7FFF
|
||||
return 0x7C00 | psign << 15
|
||||
if vc[0] == "inf":
|
||||
return 0x7C00 | vc[1] << 15
|
||||
p = va[1] * vb[1]
|
||||
s = p + vc[1]
|
||||
if s == 0: # exact zero: -0 only when both terms are -0
|
||||
return 0x8000 if (p == 0 and psign and vc[2]) else 0
|
||||
return round_half(s)
|
||||
|
||||
|
||||
def hfma2_model(a: int, b: int, c: int, relu: bool) -> int:
|
||||
"""HFMA2 (.RELU) on both halves (H1_H0 operands): RELU makes negative numbers and -0 +0, keeps NaN as 0x7fff."""
|
||||
res = 0
|
||||
for shift in (0, 16):
|
||||
h = fma_half((a >> shift) & 0xFFFF, (b >> shift) & 0xFFFF, (c >> shift) & 0xFFFF)
|
||||
if relu and h != 0x7FFF and h & 0x8000:
|
||||
h = 0
|
||||
res |= h << shift
|
||||
return res
|
||||
|
||||
|
||||
def hfma2_operands(case: dict, lane: int, b_imm: int | None = None):
|
||||
w0, w1, w2, _ = words_of(case["sm86"])
|
||||
regs = emu.load_regs(case["regs_in"])[lane]
|
||||
get = lambda n: 0 if n == RZ else regs[("R", n)]
|
||||
b = b_imm if b_imm is not None else get(w1 & 0xFF)
|
||||
return get((w0 >> 24) & 0xFF), b, get(w2 & 0xFF), bool((w2 >> 15) & 1)
|
||||
|
||||
|
||||
def ldsm_model(smem: bytes, addresses: list[int], count: int, transpose: bool, offset: int) -> list[list[int]]:
|
||||
"""LDSM per work/tensor-gt/README.md "Fragment layouts": out[lane][m] is lane's value of Rd + m."""
|
||||
out = [[0] * count for _ in range(LANES)]
|
||||
rd16 = lambda at: int.from_bytes(smem[at:at + 2], "little")
|
||||
for m in range(count):
|
||||
rows = [addresses[8 * m + r] + offset for r in range(8)]
|
||||
for t in range(LANES):
|
||||
if transpose:
|
||||
out[t][m] = rd16(rows[2 * (t % 4)] + 2 * (t // 4)) | rd16(rows[2 * (t % 4) + 1] + 2 * (t // 4)) << 16
|
||||
else:
|
||||
at = rows[t // 4] + 4 * (t % 4)
|
||||
out[t][m] = int.from_bytes(smem[at:at + 4], "little")
|
||||
return out
|
||||
|
||||
|
||||
def ldsm_fields(sm86):
|
||||
w0, w1, w2, _ = words_of(sm86)
|
||||
return (w0 >> 16) & 0xFF, (w0 >> 24) & 0xFF, 1 << ((w2 >> 8) & 3), bool((w2 >> 14) & 1), (w1 >> 8) & 0xFFFFFF
|
||||
|
||||
|
||||
def model_problems(cases: list[dict]) -> dict[str, list[str]]:
|
||||
"""Each model against every hardware case of its instruction (exact bits: the models write 0x7fff for NaN)."""
|
||||
found = {}
|
||||
for case in cases:
|
||||
w0 = words_of(case["sm86"])[0]
|
||||
op = w0 & 0xFFF
|
||||
expected = emu.load_regs(case["regs_out"])
|
||||
bad = []
|
||||
if op == 0x231:
|
||||
rd = (w0 >> 16) & 0xFF
|
||||
for lane in range(LANES):
|
||||
want = expected[lane][("R", rd)]
|
||||
got = hfma2_model(*hfma2_operands(case, lane))
|
||||
if got != want:
|
||||
bad.append(f"lane {lane}: model {got:08x}, GPU {want:08x}")
|
||||
elif op == 0x83B:
|
||||
rd, ra, count, transpose, offset = ldsm_fields(case["sm86"])
|
||||
regs = emu.load_regs(case["regs_in"])
|
||||
model = ldsm_model(shared_memory(case), [regs[t][("R", ra)] for t in range(LANES)], count, transpose, offset)
|
||||
for t in range(LANES):
|
||||
for m in range(count):
|
||||
if model[t][m] != expected[t][("R", rd + m)]:
|
||||
bad.append(f"lane {t} R{rd + m}: model {model[t][m]:08x}, GPU {expected[t][('R', rd + m)]:08x}")
|
||||
else:
|
||||
continue
|
||||
found[case["name"]] = bad
|
||||
return found
|
||||
|
||||
|
||||
def derived_cases(cases: list[dict]) -> list[dict]:
|
||||
by_name = {c["name"]: c for c in cases}
|
||||
out = []
|
||||
relu = by_name.get("hfma2_relu")
|
||||
if relu is not None:
|
||||
w0, _, w2, w3 = words_of(relu["sm86"])
|
||||
rb = words_of(relu["sm86"])[1] & 0xFF
|
||||
for imm in (0x3C003C00, 0x4000BC00): # (1, 1) as the game's networks use it; (-1, 2)
|
||||
regs_in = [{k: v for k, v in lane.items() if k != f"R{rb}"} for lane in relu["regs_in"]]
|
||||
sm86 = [(w0 & ~0xFFF) | 0x831, imm, w2, w3]
|
||||
rd = (w0 >> 16) & 0xFF
|
||||
regs_out = []
|
||||
for lane in range(LANES):
|
||||
a, b, c, r = hfma2_operands(relu, lane, imm)
|
||||
regs_out.append({f"R{rd}": f"{hfma2_model(a, b, c, r):08x}"})
|
||||
out.append({"name": f"derived_hfma2_relu_imm_{imm:08x}", "sm86": [f"{x:08x}" for x in sm86],
|
||||
"regs_in": regs_in, "regs_out": regs_out})
|
||||
src = by_name.get("ldsm_m88_x4")
|
||||
if src is not None:
|
||||
w0, _, w2, w3 = words_of(src["sm86"])
|
||||
regs = emu.load_regs(src["regs_in"])
|
||||
rd, ra = (w0 >> 16) & 0xFF, (w0 >> 24) & 0xFF
|
||||
addresses = [regs[t][("R", ra)] for t in range(LANES)]
|
||||
for transpose in (False, True):
|
||||
sm86 = [w0, 0x40 << 8, (w2 & ~0x4000) | (0x4000 if transpose else 0), w3]
|
||||
model = ldsm_model(shared_memory(src), addresses, 4, transpose, 0x40)
|
||||
regs_out = [{f"R{rd + m}": f"{model[t][m]:08x}" for m in range(4)} for t in range(LANES)]
|
||||
out.append(dict(src, name=f"derived_ldsm_{'mt88' if transpose else 'm88'}_x4_off40",
|
||||
sm86=[f"{x:08x}" for x in sm86], regs_out=regs_out))
|
||||
return out
|
||||
|
||||
|
||||
# ---- driver -------------------------------------------------------------------------------------------------------
|
||||
|
||||
def main(argv=None) -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
|
||||
ap.add_argument("vectors", nargs="?", type=Path, default=DEFAULT_VECTORS, help=f"vector file (default {DEFAULT_VECTORS})")
|
||||
ap.add_argument("--xlate", type=Path, default=emu.DEFAULT_XLATE, help="nx2navi_xlate executable")
|
||||
ap.add_argument("--case", action="append", help="only cases whose name contains this (repeatable)")
|
||||
ap.add_argument("-v", "--verbose", action="store_true", help="more differing values per failure")
|
||||
args = ap.parse_args(argv)
|
||||
if not args.vectors.exists():
|
||||
print(f"no vector file {args.vectors}")
|
||||
return 1
|
||||
if not args.xlate.exists():
|
||||
print(f"no translator {args.xlate} (python tools/build_xlate.py)")
|
||||
return 1
|
||||
cases = emu.load_cases(args.vectors)
|
||||
failed = 0
|
||||
|
||||
models = model_problems(cases)
|
||||
for name, bad in models.items():
|
||||
if bad:
|
||||
failed += 1
|
||||
print(f"FAIL model vs GPU on {name}: " + "; ".join(bad[:4]))
|
||||
model_ok = not any(models.values())
|
||||
print(f"reference models (HFMA2, LDSM) agree with the GPU on {sum(1 for b in models.values() if not b)}/{len(models)} cases")
|
||||
|
||||
todo = cases + (derived_cases(cases) if model_ok else [])
|
||||
passed = 0
|
||||
for case in todo:
|
||||
name = case["name"]
|
||||
if args.case and not any(c in name for c in args.case):
|
||||
continue
|
||||
try:
|
||||
out = check_case(case, args.xlate, args.verbose)
|
||||
except Exception as e: # one broken case does not stop the run
|
||||
out = Outcome()
|
||||
out.problems.append(f"error: {type(e).__name__}: {e}")
|
||||
kind = emu.result_kind(case["sm86"])
|
||||
notes = []
|
||||
if kind != "bits":
|
||||
notes.append("largest difference " + ("NaN against a number" if out.worst_ulp >= 1 << 32 else f"{out.worst_ulp} ulp"))
|
||||
if out.nan_matches:
|
||||
notes.append(f"{out.nan_matches} NaN values with another NaN encoding than the GPU's")
|
||||
ok = not out.problems
|
||||
passed += ok
|
||||
failed += not ok
|
||||
print(f"{'PASS' if ok else 'FAIL'} {name} ({out.runs} runs{', ' + ', '.join(notes) if notes else ''})")
|
||||
for p in out.problems:
|
||||
print(" " + p)
|
||||
print(f"{passed} passed, {failed} failed")
|
||||
return 0 if failed == 0 else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -99,8 +99,8 @@ def prepare_sources() -> Path:
|
||||
for junk in list(src.glob("src/*/obj")) + list(src.glob("src/*/bin")):
|
||||
shutil.rmtree(junk)
|
||||
patches = sorted((XLATE / "patches").glob("*.patch"))
|
||||
if len(patches) != 19:
|
||||
sys.exit(f"expected 19 patches, found {len(patches)}")
|
||||
if len(patches) != 20:
|
||||
sys.exit(f"expected 20 patches, found {len(patches)}")
|
||||
for patch in patches:
|
||||
# git apply works outside a repository when given a directory; --check first so a failure names the patch.
|
||||
for extra in (["--check"], []):
|
||||
|
||||
@@ -130,6 +130,7 @@ static class AmpereLowering
|
||||
int highest = HighestGpr(code);
|
||||
if (spillAt < 0 && highest >= UrLimit) throw new RegisterPressureException($"guest registers up to R{highest}");
|
||||
UrBase = spillAt >= 0 ? SpillFrom : (highest + 2) & ~1;
|
||||
LastUrBase = UrBase;
|
||||
}
|
||||
|
||||
// Spill mode: the temporary that holds guest register r for this instruction, loading its group on first use.
|
||||
@@ -168,13 +169,14 @@ static class AmpereLowering
|
||||
if (b is >= 0x140 and <= 0x15f) continue; // branches, calls, exits, barriers: no register fields
|
||||
see(i.Bits(16, 8)); see(i.Bits(24, 8));
|
||||
// Register ranges the base fields do not show: LDSM writes Rd..Rd+count-1; TXD reads Ra..Ra+2 and Rb..Rb+3
|
||||
// and writes the second destination pair; HMMA reads A, B and C fragments and writes the D pair.
|
||||
// and writes the second destination pair; HMMA reads A (K/4 registers), B (K/8) and C and writes D, where
|
||||
// C and D are a pair (.F16) or four registers (.F32, bit 76).
|
||||
if (op == 0x83b && i.Bits(16, 8) != RZ) see(i.Bits(16, 8) + (1u << (int)i.Bits(72, 2)) - 1);
|
||||
if (op == 0xb6c) { see(i.Bits(16, 8) + 1); see(i.Bits(24, 8) + 2); see(i.Bits(32, 8) + 3); see(i.Bits(64, 8) + 1); }
|
||||
if (op == 0x23c)
|
||||
{
|
||||
uint k16 = i.Bits(75, 1);
|
||||
see(i.Bits(16, 8) + 1); see(i.Bits(24, 8) + 1 + 2 * k16); see(i.Bits(32, 8) + k16); see(i.Bits(64, 8) + 1);
|
||||
uint k16 = i.Bits(75, 1), cd = i.Bit(76) ? 3u : 1u;
|
||||
see(i.Bits(16, 8) + cd); see(i.Bits(24, 8) + 1 + 2 * k16); see(i.Bits(32, 8) + k16); see(i.Bits(64, 8) + cd);
|
||||
}
|
||||
if (b < 0x080 || b is >= 0x160 and <= 0x16f) see(i.Bits(64, 8)); // C of ALU forms, TEX second destinations
|
||||
if (i.Bits(9, 3) == 1) see(i.Bits(32, 8));
|
||||
@@ -1507,10 +1509,13 @@ static class AmpereLowering
|
||||
}
|
||||
|
||||
// HADD2 / HMUL2 / HFMA2 with packed-half results: saturate 77, FTZ 80, bit 76 the FMZ precision of HMUL2/HFMA2.
|
||||
// HFMA2.RELU (79) clamps each lane at zero where the predicate in bits 87-90 holds; only PT is lowered: the sum
|
||||
// goes to TexOut, PRMT replicates each lane's sign bit over that lane (selector 0xbb99) and LOP.AND with the
|
||||
// inverted mask clears the negative lanes (negative NaN becomes 0 too). F32 results (78), the H0_NH1 source (86)
|
||||
// and absolute values where Maxwell has none are not lowered. HADD2's second operand is B, or C for forms 2/3.
|
||||
// HFMA2.RELU (79) clamps each lane at zero where the predicate in bits 87-90 holds; only PT is lowered. Measured
|
||||
// on an NVIDIA GPU (tests/tensor_lowering_check.py): a negative result or -0 becomes +0, +inf stays, -inf
|
||||
// becomes 0, a NaN result stays NaN (0x7fff). The sum goes to TexOut and integer operations build the mask of
|
||||
// the half lanes to clear. Maxwell PRMT would be the obvious tool, but the translator emits nothing for it
|
||||
// (Ryujinx 1.3.2, Instructions/InstEmit.cs, PrmtR only logs "not implemented").
|
||||
// F32 results (78), the H0_NH1 source (86) and absolute values where Maxwell has none are not lowered. HADD2's
|
||||
// second operand is B, or C for forms 2/3.
|
||||
void Half(ulong maxwell)
|
||||
{
|
||||
uint op = _i.Bits(0, 12) & 0x1ff;
|
||||
@@ -1548,11 +1553,20 @@ static class AmpereLowering
|
||||
| ((ulong)c.reg << 39);
|
||||
}
|
||||
if (!relu) { EmitG(w); return; }
|
||||
Emit(Unguarded(w));
|
||||
Mov32i(TexOut + 1, 0xbb99); // PRMT selector: bytes 0-1 the sign of byte 1, bytes 2-3 the sign of byte 3
|
||||
Emit(Unguarded(0x5bc0000000000000UL | ((ulong)RZ << 39) | ((ulong)(TexOut + 1) << 20) | ((ulong)TexOut << 8) | (uint)(TexOut + 2)));
|
||||
// LOP.AND Rd, sum, ~mask (InvB bit 40), no predicate output (PT at 48)
|
||||
EmitG(0x5c40000000000000UL | (7UL << 48) | (1UL << 40) | ((ulong)(TexOut + 2) << 20) | ((ulong)TexOut << 8) | (uint)Gpr((uint)Dst));
|
||||
// RELU on both half lanes of the sum x at once: each half h becomes |h| (its sign bit cleared), or 0 when h is
|
||||
// a negative number (sign set, not NaN). |h| > 0x7c00 means NaN, so |h| + 0x3ff carries into bit 15 of its
|
||||
// half exactly for a NaN, and never into the other half (|h| + 0x3ff < 0x10000). A NaN keeps the payload the
|
||||
// host computed, with the sign cleared (the hardware's is 0x7fff). Scratch writes unguarded, Rd guarded.
|
||||
const int x = TexOut, y = TexOut + 1, t = TexOut + 2;
|
||||
Emit(Unguarded(w)); // HFMA2 x
|
||||
Lop32i(y, x, 0x7fff7fff, 0); // y = |h| per half
|
||||
Emit(Unguarded(0x1c00000000000000UL | (0x03ff03ffUL << 20) | ((ulong)y << 8) | t)); // IADD32I t = y + 0x03ff03ff
|
||||
Emit(Unguarded(0x5c40000000000000UL | (7UL << 48) | (1UL << 40) | ((ulong)t << 20) | ((ulong)x << 8) | t)); // LOP.AND t = x & ~t
|
||||
Emit(Unguarded(0x3828000000000000UL | (15UL << 20) | ((ulong)t << 8) | t)); // SHR t >>= 15
|
||||
Lop32i(t, t, 0x00010001, 0); // bit 0 / 16: clear that half
|
||||
Emit(Unguarded(0x5c18000000000000UL | (16UL << 39) | (1UL << 48) | ((ulong)t << 20) | ((ulong)t << 8) | t)); // ISCADD t = (t << 16) - t
|
||||
// LOP.AND Rd = y & ~t (InvB bit 40), no predicate output (PT at 48)
|
||||
EmitG(0x5c40000000000000UL | (7UL << 48) | (1UL << 40) | ((ulong)t << 20) | ((ulong)y << 8) | (uint)Gpr((uint)Dst));
|
||||
}
|
||||
|
||||
// MOVM.16.MT88 Rd, Ra (0x23a): an 8x8 matrix of 16-bit elements spread over the warp (lane t holds row t/4,
|
||||
@@ -1973,33 +1987,39 @@ static class AmpereLowering
|
||||
}
|
||||
}
|
||||
|
||||
// HMMA.1688.F16 / HMMA.16816.F16 Rd, Ra, Rb, Rc: a warp-wide matrix multiply-accumulate D(16x8) = A(16xK) *
|
||||
// B(Kx8) + C(16x8) on F16 elements, K = 8, or 16 when bit 75 is set. Fields: D 16-23 (a register pair), A 24-31
|
||||
// (K/4 registers), B 32-39 (K/8 registers), C 64-71 (a pair; RZ means a zero accumulator). With g = lane / 4 and
|
||||
// q = lane % 4, each lane holds (the lower-numbered column or k in the low half of every register):
|
||||
// HMMA.1688 / HMMA.16816 Rd, Ra, Rb, Rc with F16 inputs: a warp-wide matrix multiply-accumulate D(16x8) =
|
||||
// A(16xK) * B(Kx8) + C(16x8), K = 8, or 16 when bit 75 is set. Bit 76 selects F32 accumulators and results
|
||||
// (.F32: C and D are four registers each) instead of F16 ones (.F16: a register pair). Fields: D 16-23, A 24-31
|
||||
// (K/4 registers), B 32-39 (K/8 registers), C 64-71 (RZ means a zero accumulator). With g = lane / 4 and
|
||||
// q = lane % 4, each lane holds (the lower-numbered column or k in the low half of every packed register):
|
||||
// A register 2j + r: row g + 8r, k 8j + 2q and 8j + 2q + 1 (j is 0 only when K = 8)
|
||||
// B register j: k 8j + 2q and 8j + 2q + 1, column g
|
||||
// C / D register r: row g + 8r, columns 2q and 2q + 1
|
||||
// C / D, .F16: register r holds row g + 8r, columns 2q and 2q + 1
|
||||
// C / D, .F32: register 2r + c holds row g + 8r, column 2q + c
|
||||
// Maxwell has no tensor instructions, so the product is expanded per lane. For each k pair (2p, 2p + 1) the lane
|
||||
// fetches the A word of its own rows from lane 4g + p % 4 and, for each of its two columns n = 2q + c, the B word
|
||||
// from lane 4n + p % 4 (SHFL.IDX), widens the halves to F32 (F2F.F32.F16, exact) and accumulates with FFMA (an
|
||||
// F16 x F16 product is exact in F32). The accumulators are rounded to F16 after every eight k (one 1688 step of
|
||||
// the hardware) and widened again when more follow; the final F16 values are packed as F2fp in this file does
|
||||
// (F2F.F16 writes the low half and zeroes the high half, ISCADD merges the column pair).
|
||||
// The work runs unguarded because every lane must supply its fragments to the shuffles; only the two result
|
||||
// writes take the instruction's guard. Scratch R239-R252 (this file's scratch range, see the header); the guest
|
||||
// operands are never there (guest and uniform registers end below R237, spill temporaries are R204-R235).
|
||||
// from lane 4n + p % 4 (SHFL.IDX), widens the halves to F32 (F2F.F32.F16, exact) and accumulates with FFMA in
|
||||
// F32 (an F16 x F16 product is exact in F32, so the step rounds once whether or not the host fuses it).
|
||||
// The .F16 form rounds to F16 once, after all K products, as the hardware does: it does not round between the
|
||||
// two eight-k halves of HMMA.16816. Measured on an NVIDIA GPU (tests/tensor_lowering_check.py compares this
|
||||
// lowering with those results): a first-half sum of 65536 that the second half cancels gives 0, which an F16
|
||||
// rounding after the first half turns into +inf, and 1 + 2^-11 + 2^-11 split over the halves gives 1 + 2^-10,
|
||||
// which that rounding turns into 1. The F16 results are then packed as F2fp in this file does (F2F.F16 writes
|
||||
// the low half and zeroes the high half, ISCADD merges the column pair).
|
||||
// The work runs unguarded because every lane must supply its fragments to the shuffles; only the result writes
|
||||
// take the instruction's guard. Scratch R239-R252 (this file's scratch range, see the header); the guest operands
|
||||
// are never there (guest and uniform registers end below R237, spill temporaries are R188-R235).
|
||||
void Hmma()
|
||||
{
|
||||
bool k16 = _i.Bit(75);
|
||||
bool k16 = _i.Bit(75), f32 = _i.Bit(76);
|
||||
uint d = (uint)Dst, a = _i.Bits(24, 8), b = _i.Bits(32, 8), c = _i.Bits(64, 8);
|
||||
int aRegs = k16 ? 4 : 2, bRegs = k16 ? 2 : 1;
|
||||
int aRegs = k16 ? 4 : 2, bRegs = k16 ? 2 : 1, cdRegs = f32 ? 4 : 2;
|
||||
// A fragment base aligned to its size and the whole fragment below RZ (RZ itself reads as zeros).
|
||||
bool fits(uint reg, int n) => reg == RZ || (reg % (uint)n == 0 && reg + (uint)n <= RZ);
|
||||
// Word 1 above B, word 2 other than C and the shape bit, and word 3 bits 96-104 / 126-127 select modes
|
||||
// (F32 accumulators and others) that are not lowered.
|
||||
if ((_i.W1 & ~0xffu) != 0 || (_i.W2 & ~0x8ffu) != 0 || ((uint)(_i.Hi >> 32) & 0xc00001ffu) != 0
|
||||
|| !fits(d, 2) || !fits(a, aRegs) || !fits(b, bRegs) || !fits(c, 2))
|
||||
// Word 1 above B, word 2 other than C, the shape (75) and the accumulator type (76), and word 3 bits 96-104 /
|
||||
// 126-127 select modes (bit 74 another shape, HMMA.1684; other input types) that are not lowered.
|
||||
if ((_i.W1 & ~0xffu) != 0 || (_i.W2 & ~0x18ffu) != 0 || ((uint)(_i.Hi >> 32) & 0xc00001ffu) != 0
|
||||
|| !fits(d, cdRegs) || !fits(a, aRegs) || !fits(b, bRegs) || !fits(c, cdRegs))
|
||||
throw Fail("HMMA mode");
|
||||
if (d == RZ) return; // no result and no other effect
|
||||
|
||||
@@ -2027,13 +2047,14 @@ static class AmpereLowering
|
||||
return lane;
|
||||
}
|
||||
|
||||
// Accumulators from C (both halves of each row register), or zero.
|
||||
for (int r = 0; r < 2; r++)
|
||||
for (int col = 0; col < 2; col++)
|
||||
{
|
||||
if (c == RZ) Mov32i(acc + 2 * r + col, 0);
|
||||
else widen(acc + 2 * r + col, part(c, r), col == 1);
|
||||
}
|
||||
// Accumulators from C, or zero: accumulator 2r + col is register 2r + col of an .F32 C, half col of register
|
||||
// r of an .F16 C.
|
||||
for (int k = 0; k < 4; k++)
|
||||
{
|
||||
if (c == RZ) Mov32i(acc + k, 0);
|
||||
else if (f32) Mov(acc + k, part(c, k), false);
|
||||
else widen(acc + k, part(c, k >> 1), (k & 1) == 1);
|
||||
}
|
||||
|
||||
// Lane bases: 8q first (it needs the raw lane id), then 4g.
|
||||
Put(0xf0c8000000000000UL | laneA, false); // S2R SR_LANEID
|
||||
@@ -2065,15 +2086,17 @@ static class AmpereLowering
|
||||
Emit(Unguarded(0x5980000000000000UL | ((ulong)t << 39) | ((ulong)(bw + h) << 20) | ((ulong)(aw + 2 * r + h) << 8) | (uint)t));
|
||||
}
|
||||
}
|
||||
if ((p & 3) == 3) // end of an eight-k step: round to F16, widen again if another step follows
|
||||
for (int t = acc; t < acc + 4; t++)
|
||||
{
|
||||
narrow(t);
|
||||
if (p + 1 < pairs) widen(t, t, false);
|
||||
}
|
||||
}
|
||||
|
||||
// D register r = column 2q + 1 in the high half, column 2q in the low half (ISCADD hi << 16 + lo).
|
||||
// .F32: D register 2r + col is accumulator 2r + col.
|
||||
if (f32)
|
||||
{
|
||||
for (int k = 0; k < 4; k++) Mov(part(d, k), acc + k);
|
||||
return;
|
||||
}
|
||||
// .F16: one rounding of each sum to F16, then D register r = column 2q + 1 in the high half, column 2q in the
|
||||
// low half (ISCADD hi << 16 + lo).
|
||||
for (int t = acc; t < acc + 4; t++) narrow(t);
|
||||
for (int r = 0; r < 2; r++)
|
||||
EmitG(0x5c18000000000000UL | (16UL << 39) | ((ulong)(acc + 2 * r) << 20) | ((ulong)(acc + 2 * r + 1) << 8) | (uint)part(d, r));
|
||||
}
|
||||
@@ -2328,6 +2351,10 @@ static class AmpereLowering
|
||||
// c1: the program's c1 constants (BRX jump tables); set before Lower by the caller.
|
||||
[ThreadStatic] public static byte[] C1Bytes;
|
||||
|
||||
// Diagnostics only (Program.cs LowerFile, the --lower mode): the register UR0 was mapped to by the last Lowerer
|
||||
// built on this thread (Lowerer.UrBase; SpillFrom in spill mode). Lowering never reads it.
|
||||
[ThreadStatic] public static int LastUrBase;
|
||||
|
||||
// localBytes: the local memory size the lowered program needs when that changed (spill mode), else 0. Graphics
|
||||
// programs carry it in their header already; compute and task/mesh programs need it from the translator's host
|
||||
// (job "local_mem").
|
||||
|
||||
@@ -6,6 +6,7 @@
|
||||
// nx2navi_xlate --serve one request per stdin line: {"job": {...}, "out": "<dir>"}; one reply line per request:
|
||||
// "ok <key>" or "fail <key> <message>"; a malformed line gets "fail ? <message>" and the loop continues.
|
||||
// nx2navi_xlate <jobs.json> <outdir> batch mode: a JSON array of jobs, prints a summary, exit 1 if any failed.
|
||||
// nx2navi_xlate --lower <file> lowers SM86 instructions given as hex words and prints the Maxwell words (LowerFile).
|
||||
// job: { "key", "stage": "vs|ps|cs", "code": path (graphics = 0x50-byte SPH + instructions, compute = instructions), "c1": hex,
|
||||
// "sm86": bool + "size": bytes (an NVN2 program, lowered by AmpereLowering.cs), "nvn2_table": [start, textures,
|
||||
// images] (NVN2 compute binding table), "lowered_out": path, "mesh": {"stage": 6|7, "threads", "maxv", "maxp",
|
||||
@@ -231,8 +232,56 @@ static class SwshXlate
|
||||
catch (Exception e) { if (Environment.GetEnvironmentVariable("DMSH_STACK") != null) Console.Error.WriteLine(e.StackTrace); return $"{e.GetType().Name}: {e.Message.Split('\n')[0]}"; }
|
||||
}
|
||||
|
||||
// --lower <file>: runs AmpereLowering on a few SM86 instructions, for executing single instructions on the CPU
|
||||
// (tests/maxwell_warp_emu.py). Each line of the file is one instruction as four hex 32-bit words "w0 w1 w2 w3"
|
||||
// (w0 = bits 0-31, optional 0x prefix; blank lines and '#' comments skipped). They are lowered as a compute program
|
||||
// (AmpereLowering.Lower with graphics false), with an SM86 EXIT appended unless the last instruction is an
|
||||
// unpredicated EXIT. Prints the lowered Maxwell instruction words in program order, one 16-digit hex word per line:
|
||||
// the scheduling words Lower puts first in every 32 bytes are left out, the closing EXIT Lower appends is kept. Then
|
||||
// "urbase N" (the register UR0 maps to, AmpereLowering.LastUrBase) and, in spill mode only, "local N" (the local
|
||||
// memory bytes the program needs). Errors go to stderr with exit code 1.
|
||||
static int LowerFile(string path)
|
||||
{
|
||||
try
|
||||
{
|
||||
var bytes = new List<byte>();
|
||||
uint[] last = null;
|
||||
foreach (string raw in File.ReadAllLines(path))
|
||||
{
|
||||
string line = raw.Split('#')[0].Trim();
|
||||
if (line.Length == 0) continue;
|
||||
string[] f = line.Split((char[])null, StringSplitOptions.RemoveEmptyEntries);
|
||||
if (f.Length != 4) throw new FormatException($"expected four hex words: \"{line}\"");
|
||||
last = f.Select(s => Convert.ToUInt32(s.StartsWith("0x", StringComparison.OrdinalIgnoreCase) ? s[2..] : s, 16)).ToArray();
|
||||
foreach (uint w in last) bytes.AddRange(BitConverter.GetBytes(w));
|
||||
}
|
||||
// SM86 EXIT: opcode 0x94d under PT (bits 12-15 = 7), second condition PT (bits 87-90 = 7); AmpereLowering Exit.
|
||||
bool exits = last != null && (last[0] & 0xffff) == 0x794d && ((last[2] >> 23) & 15) == 7 && ((last[2] >> 20) & 3) == 0;
|
||||
if (!exits) foreach (uint w in new uint[] { 0x0000794d, 0, 0x03800000, 0x000fea00 }) bytes.AddRange(BitConverter.GetBytes(w));
|
||||
byte[] code = bytes.ToArray();
|
||||
byte[] lowered = AmpereLowering.Lower(code, code.Length, false, out int local);
|
||||
// Compute layout (AmpereLowering.Lower): a scheduling word at every 32-byte boundary, instructions in the other
|
||||
// slots, zero padding after the closing EXIT. Instruction words are never 0, so the last nonzero one ends it.
|
||||
var words = new List<ulong>();
|
||||
int end = 0;
|
||||
for (int a = 0; a + 8 <= lowered.Length; a += 8)
|
||||
{
|
||||
if ((a & 0x1f) == 0) continue;
|
||||
ulong w = BitConverter.ToUInt64(lowered, a);
|
||||
words.Add(w);
|
||||
if (w != 0) end = words.Count;
|
||||
}
|
||||
foreach (ulong w in words.Take(end)) Console.WriteLine(w.ToString("x16"));
|
||||
Console.WriteLine($"urbase {AmpereLowering.LastUrBase}");
|
||||
if (local != 0) Console.WriteLine($"local {local}");
|
||||
return 0;
|
||||
}
|
||||
catch (Exception e) { Console.Error.WriteLine($"{e.GetType().Name}: {e.Message}"); return 1; }
|
||||
}
|
||||
|
||||
static int Main(string[] args)
|
||||
{
|
||||
if (args.Length == 2 && args[0] == "--lower") return LowerFile(args[1]);
|
||||
if (args.Length == 1 && args[0] == "--serve")
|
||||
{
|
||||
// One request per line: {"job": {...}, "out": "<dir>"}; one reply line "ok <key>" / "fail <key> <message>".
|
||||
|
||||
161
tools/xlate/patches/0021-prmt-byte-permute.patch
Normal file
161
tools/xlate/patches/0021-prmt-byte-permute.patch
Normal file
@@ -0,0 +1,161 @@
|
||||
diff --git a/src/Ryujinx.Graphics.Shader/Instructions/InstEmit.cs b/src/Ryujinx.Graphics.Shader/Instructions/InstEmit.cs
|
||||
index d77621d..462c0ca 100644
|
||||
--- a/src/Ryujinx.Graphics.Shader/Instructions/InstEmit.cs
|
||||
+++ b/src/Ryujinx.Graphics.Shader/Instructions/InstEmit.cs
|
||||
@@ -215,34 +215,6 @@ namespace Ryujinx.Graphics.Shader.Instructions
|
||||
context.TranslatorContext.GpuAccessor.Log("Shader instruction Pret is not implemented.");
|
||||
}
|
||||
|
||||
- public static void PrmtR(EmitterContext context)
|
||||
- {
|
||||
- context.GetOp<InstPrmtR>();
|
||||
-
|
||||
- context.TranslatorContext.GpuAccessor.Log("Shader instruction PrmtR is not implemented.");
|
||||
- }
|
||||
-
|
||||
- public static void PrmtI(EmitterContext context)
|
||||
- {
|
||||
- context.GetOp<InstPrmtI>();
|
||||
-
|
||||
- context.TranslatorContext.GpuAccessor.Log("Shader instruction PrmtI is not implemented.");
|
||||
- }
|
||||
-
|
||||
- public static void PrmtC(EmitterContext context)
|
||||
- {
|
||||
- context.GetOp<InstPrmtC>();
|
||||
-
|
||||
- context.TranslatorContext.GpuAccessor.Log("Shader instruction PrmtC is not implemented.");
|
||||
- }
|
||||
-
|
||||
- public static void PrmtRc(EmitterContext context)
|
||||
- {
|
||||
- context.GetOp<InstPrmtRc>();
|
||||
-
|
||||
- context.TranslatorContext.GpuAccessor.Log("Shader instruction PrmtRc is not implemented.");
|
||||
- }
|
||||
-
|
||||
public static void R2b(EmitterContext context)
|
||||
{
|
||||
context.GetOp<InstR2b>();
|
||||
diff --git a/src/Ryujinx.Graphics.Shader/Instructions/InstEmitPermute.cs b/src/Ryujinx.Graphics.Shader/Instructions/InstEmitPermute.cs
|
||||
new file mode 100644
|
||||
index 0000000..b79ad3e
|
||||
--- /dev/null
|
||||
+++ b/src/Ryujinx.Graphics.Shader/Instructions/InstEmitPermute.cs
|
||||
@@ -0,0 +1,116 @@
|
||||
+using Ryujinx.Graphics.Shader.Decoders;
|
||||
+using Ryujinx.Graphics.Shader.IntermediateRepresentation;
|
||||
+using Ryujinx.Graphics.Shader.Translation;
|
||||
+
|
||||
+using static Ryujinx.Graphics.Shader.Instructions.InstEmitHelper;
|
||||
+using static Ryujinx.Graphics.Shader.IntermediateRepresentation.OperandHelper;
|
||||
+
|
||||
+namespace Ryujinx.Graphics.Shader.Instructions
|
||||
+{
|
||||
+ // NX2NAVI: PRMT (byte permute). Upstream Ryujinx only logged it as not implemented, so the destination kept its old
|
||||
+ // value; the SM86 lowering (tools/xlate/AmpereLowering.cs Prmt, the HFMA2.RELU sign mask in Half) emits it for every
|
||||
+ // byte permute of the Switch 2 shaders (82 pixel, 303 vertex and 277 compute programs of Ocarina), which lost them.
|
||||
+ // Rd byte i is byte k of the 8 bytes {C, A} (A bytes 0-3, C bytes 4-7). In the default mode k is nibble i of B and
|
||||
+ // its bit 3 replaces the byte with its sign (0x00 or 0xff); the other modes take k from B bits 0-1 alone, as the PTX
|
||||
+ // prmt modes f4e, b4e, rc8, ecl, ecr and rc16 define.
|
||||
+ static partial class InstEmit
|
||||
+ {
|
||||
+ public static void PrmtR(EmitterContext context)
|
||||
+ {
|
||||
+ InstPrmtR op = context.GetOp<InstPrmtR>();
|
||||
+
|
||||
+ Operand srcA = GetSrcReg(context, op.SrcA);
|
||||
+ Operand srcB = GetSrcReg(context, op.SrcB);
|
||||
+ Operand srcC = GetSrcReg(context, op.SrcC);
|
||||
+
|
||||
+ context.Copy(GetDest(op.Dest), EmitPrmt(context, srcA, srcB, srcC, op.PMode));
|
||||
+ }
|
||||
+
|
||||
+ public static void PrmtI(EmitterContext context)
|
||||
+ {
|
||||
+ InstPrmtI op = context.GetOp<InstPrmtI>();
|
||||
+
|
||||
+ Operand srcA = GetSrcReg(context, op.SrcA);
|
||||
+ Operand srcB = Const(op.Imm20);
|
||||
+ Operand srcC = GetSrcReg(context, op.SrcC);
|
||||
+
|
||||
+ context.Copy(GetDest(op.Dest), EmitPrmt(context, srcA, srcB, srcC, op.PMode));
|
||||
+ }
|
||||
+
|
||||
+ public static void PrmtC(EmitterContext context)
|
||||
+ {
|
||||
+ InstPrmtC op = context.GetOp<InstPrmtC>();
|
||||
+
|
||||
+ Operand srcA = GetSrcReg(context, op.SrcA);
|
||||
+ Operand srcB = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset);
|
||||
+ Operand srcC = GetSrcReg(context, op.SrcC);
|
||||
+
|
||||
+ context.Copy(GetDest(op.Dest), EmitPrmt(context, srcA, srcB, srcC, op.PMode));
|
||||
+ }
|
||||
+
|
||||
+ public static void PrmtRc(EmitterContext context)
|
||||
+ {
|
||||
+ InstPrmtRc op = context.GetOp<InstPrmtRc>();
|
||||
+
|
||||
+ Operand srcA = GetSrcReg(context, op.SrcA);
|
||||
+ Operand srcB = GetSrcReg(context, op.SrcC);
|
||||
+ Operand srcC = GetSrcCbuf(context, op.CbufSlot, op.CbufOffset);
|
||||
+
|
||||
+ context.Copy(GetDest(op.Dest), EmitPrmt(context, srcA, srcB, srcC, op.PMode));
|
||||
+ }
|
||||
+
|
||||
+ private static Operand EmitPrmt(EmitterContext context, Operand srcA, Operand srcB, Operand srcC, PMode mode)
|
||||
+ {
|
||||
+ Operand result = Const(0);
|
||||
+ Operand low = context.BitwiseAnd(srcB, Const(3)); // the start byte of the special modes
|
||||
+
|
||||
+ for (int i = 0; i < 4; i++)
|
||||
+ {
|
||||
+ Operand index; // 0-7: the source byte of result byte i
|
||||
+ Operand sign = null; // non-zero: replicate that byte's sign bit (default mode only)
|
||||
+
|
||||
+ switch (mode)
|
||||
+ {
|
||||
+ case PMode.F4e:
|
||||
+ index = context.BitwiseAnd(context.IAdd(low, Const(i)), Const(7));
|
||||
+ break;
|
||||
+ case PMode.B4e:
|
||||
+ index = context.BitwiseAnd(context.IAdd(low, Const(8 - i)), Const(7));
|
||||
+ break;
|
||||
+ case PMode.Rc8:
|
||||
+ index = low;
|
||||
+ break;
|
||||
+ case PMode.Ecl:
|
||||
+ index = context.IMaximumU32(low, Const(i));
|
||||
+ break;
|
||||
+ case PMode.Ecr:
|
||||
+ index = context.IMinimumU32(low, Const(i));
|
||||
+ break;
|
||||
+ case PMode.Rc16:
|
||||
+ index = context.BitwiseOr(context.ShiftLeft(context.BitwiseAnd(low, Const(1)), Const(1)), Const(i & 1));
|
||||
+ break;
|
||||
+ default:
|
||||
+ Operand nibble = context.BitfieldExtractU32(srcB, Const(i * 4), Const(4));
|
||||
+ index = context.BitwiseAnd(nibble, Const(7));
|
||||
+ sign = context.BitwiseAnd(nibble, Const(8));
|
||||
+ break;
|
||||
+ }
|
||||
+
|
||||
+ Operand word = context.ConditionalSelect(context.ICompareLessUnsigned(index, Const(4)), srcA, srcC);
|
||||
+ Operand shift = context.ShiftLeft(context.BitwiseAnd(index, Const(3)), Const(3));
|
||||
+ Operand value = context.BitfieldExtractU32(word, shift, Const(8));
|
||||
+
|
||||
+ if (sign != null)
|
||||
+ {
|
||||
+ Operand negative = context.ICompareNotEqual(context.BitwiseAnd(value, Const(0x80)), Const(0));
|
||||
+ Operand replicated = context.ConditionalSelect(negative, Const(0xff), Const(0));
|
||||
+ value = context.ConditionalSelect(context.ICompareNotEqual(sign, Const(0)), replicated, value);
|
||||
+ }
|
||||
+
|
||||
+ result = context.BitwiseOr(result, context.ShiftLeft(value, Const(i * 8)));
|
||||
+ }
|
||||
+
|
||||
+ return result;
|
||||
+ }
|
||||
+ }
|
||||
+}
|
||||
Reference in New Issue
Block a user