diff --git a/docs/architecture.md b/docs/architecture.md index 6c0c9baa..eada8886 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -133,6 +133,28 @@ keeps per-core data without locks and relies on that, so the host models it (`ru ring of Vulkan `{x, y, z}` records filled on the GPU. The copy cannot be recorded inside a pass, so the first such draw of a pass fills the records of the mesh draws that follow it too, and the pass is not cut at each of them (`runtime/vk_exec_draw.c` `mesh_ring_entry`). +- Mesh fallback (`mesh_fallback`: `auto` on a device without the extension, `1` on any device for testing): the task + and mesh stages run as compute dispatches and the result is drawn as ordinary triangles. + - The modules come from the translations the shader cache already holds, not from the translator + (`runtime/mesh_fb.c`): a translated task or mesh module is the program as a compute function over shared memory + plus an entry function that hands that memory to the mesh pipeline, and the fallback replaces the entry function + by one that copies the memory into a storage buffer. The translator, its identity and every cache key stay as + they are, so a device with mesh shaders keeps its shaders and never sees these modules, and a device without + needs no other cache or prepared shader list. + - A draw is: its record (the workgroup count, copied on the GPU when the game's record holds it); the task + dispatch, which writes each task workgroup's payload into the task buffer; a scan (`shaders/meshfb_scan.comp`) + that lists the mesh workgroups the tasks started, in task order, and writes their number into the record; the + mesh dispatch, which writes each workgroup's vertex lines and primitive bytes into its slot of the mesh buffer; + and an instanced indirect draw, one instance per slot, whose generated vertex stage looks the vertex of each + primitive corner up in the slot (`runtime/vk_exec_draw.c` `mfb_draw`, `runtime/vk_mesh_fb.c`). + - Each step reads what the one before wrote, so full barriers stand between them and every such draw ends its pass: + a village frame's 45 mesh draws are 45 passes cut, 90 to 135 dispatches and about 270 barriers more than with mesh + shaders. The buffers (`mesh_fallback_mb`, 128 MiB) are reused by every draw, so they hold the largest draw, not + a frame; a draw that emits more loses what does not fit, and the log says how much it needed. + - A program's modules and compute pipelines are made on a thread of their own at its first draw, which is skipped + until they are ready; they are not recorded for the next start, where the pipeline cache makes them quick. + Not carried: the far shadow cascades' coarser shading rate, a draw's first workgroup other than 0, and more + workgroups in one draw than one compute dispatch may have. - Pipelines: `VK_EXT_graphics_pipeline_library` where the driver has it: stage libraries compiled ahead, a fast link at a new state, an optimised pipeline replacing it from a background thread. A draw whose pipeline is not ready is skipped when its pass is one the game renders frame after frame: one of its targets was rendered in the last few diff --git a/docs/building.md b/docs/building.md index 1841ab4e..4fb12077 100644 --- a/docs/building.md +++ b/docs/building.md @@ -484,6 +484,12 @@ Runtime probes, each a mode of `nx2navi.exe` (the list is at the top of `runtime part of `--probe-vkexec` translate shaders of the census below: they look for `build//shaders/jobs.json` of the exe's own version first, then of the other one. `--probe-vkmem` also checks CPU-backed texture uploads and rewrites, GPU-written texture aliases, and memory write watches. +`--probe-meshfb` (no game files) draws mesh programs of the runtime's own, with and without a task stage and with +direct and GPU-written workgroup counts, through `VK_EXT_mesh_shader` and through the mesh fallback, and compares the +pictures; on a device without mesh shaders only the fallback draws. A change to `runtime/mesh_fb.c`, +`runtime/vk_mesh_fb.c` or the fallback's part of `runtime/vk_exec_draw.c` passes it. With +`NX2NAVI_PROBE_MESHFB_DIR=` it converts the game's own task and mesh translations instead, for `spirv-val` +(the top of `runtime/mesh_fb_probe.c` says what the folder holds). A change to the texture detail (`runtime/vk_texture.c` "texture detail": `detail_skip`, `upload`, `astc_bl_regions`) passes it three times, with `NX2NAVI_TEXTURE_DETAIL` set to `full`, `reduced` and `low`: each time ASTC textures are uploaded and rewritten, and every level and layer of their images must hold the game's level it stands for. diff --git a/nx2navi.ini.example b/nx2navi.ini.example index 06850775..a1c45b65 100644 --- a/nx2navi.ini.example +++ b/nx2navi.ini.example @@ -876,6 +876,14 @@ # first one, so their draw pass is not ended and begun again at each of them (about 40 passes fewer a frame in a # village). 0: each such draw ends its pass and prepares its own count, as before. # mesh_ahead = 1 +# Graphics cards without mesh shaders (GTX 10 series, RX 500 and 5000 series, many integrated ones): the game's mesh +# draws, most of the world, are computed in compute passes and drawn as ordinary triangles. auto (default) does that +# only on such a card; 1 does it on every card (for testing: it is slower than mesh shaders); 0 never, and such a +# card then draws no world. +# mesh_fallback = auto +# Video memory in MB for what those compute passes hand to the draws. A draw that needs more loses the geometry that +# does not fit, and the log says so. +# mesh_fallback_mb = 128 # Windows, 1 (experimental): the game's heap is committed a 64 KiB run at a time where it is first used, instead of # all of it when the game sizes it. Windows counts committed memory against RAM plus the page file whether it is used # or not, and in Castle Town after Hyrule Field about 3.4 of the heap's 6.8 GiB had never been used. 0: committed up diff --git a/runtime/cfg.c b/runtime/cfg.c index a60af1e1..4d6dfefe 100644 --- a/runtime/cfg.c +++ b/runtime/cfg.c @@ -36,7 +36,7 @@ static int save_ok = 1; // the last cfg_save_to wrote (o static const char *const known[] = { "CORES", "TM_DISPATCH", "NVN2_CS_TABLE", "SKIP_SHADERS", "MOVIE_DEBAND", "MOVIE_FPS", "MOVIE_SKIP", "MOVIE_OVERLAY", "MOVIES", "NPLN", "NVN2_PROGRAMS", "NVN_TRAP", "OCARINA_REALTIME", "ONESHOT_BUDGET", "ONESHOT_WAIT_MS", "OCARINA_STEP", "PHYSICS_STEPS", "PICKUP_COUNT", "STEP_STEADY", "VI_HZ", "VI_VRR", "TIME_DUMP", "SHADER_DUMP", "SHOT_AFTER", "SHOT_DIR", "SHOT_EVERY", // Switch 2 port: movie skip (romfs.c), NVN2 proc traps (nvn_procs.c) "AB", "AMIIBO_DIR", "ANISO", "APP_POOL", "ASPECT", "ASPECT_PATCH", "ASSUME_FOCUS", "ASTC", "ASTC_BC", "ASTC_TIME", "ASYNC_PREEMPT", "AUDIO", "AUDIO_CHANNELS", "AUDIO_DEVICE", "AUDIO_GAME_PRIO", "AUDIO_OFF", "AUDIO_PREFILL_MS", "AUDIO_STATS", - "BARRIER_GAME", "MESH_AHEAD", + "BARRIER_GAME", "MESH_AHEAD", "MESH_FALLBACK", "MESH_FALLBACK_MB", "PROBE_MESHFB_DIR", "BACKGROUND_WINDOW", "BIND_A", "BIND_B", "BIND_DOWN", "BIND_L", "BIND_LCLICK", "BIND_LEFT", "BIND_LSTICK_DOWN", "BIND_LSTICK_LEFT", "BIND_LSTICK_RIGHT", "BIND_LSTICK_UP", "BIND_MINUS", "BIND_PLUS", "BIND_R", "BIND_RCLICK", "BIND_RIGHT", "BIND_RSTICK_DOWN", "BIND_RSTICK_LEFT", "BIND_RSTICK_RIGHT", "BIND_RSTICK_UP", @@ -141,7 +141,7 @@ static const struct { const char *key, *section; } sections[] = { {"VK_DYN_VERTEX", "performance"}, {"VK_DYN_A2C", "performance"}, {"VK_SUBGROUP_PIN", "performance"}, {"VK_RECONVERGE", "performance"}, {"VK_NAN_PRESERVE", "performance"}, {"VK_NAN_BITS", "performance"}, {"TRACE", "diagnostics"}, {"FLIGHT", "diagnostics"}, {"PASS_CRUMBS", "diagnostics"}, {"FLIGHT_SPIKE_MS", "diagnostics"}, - {"WB_FENCE_LIMIT", "diagnostics"}, {"WB_MERGE", "diagnostics"}, {"WB_EXACT", "diagnostics"}, {"MEM_CENSUS", "diagnostics"}, {"WB_NO_REUP", "diagnostics"}, {"WB_PROBE", "diagnostics"}, {"WB_SPLIT", "performance"}, {"SHADOW_BULK_WRITE", "performance"}, {"WB_LAZY", "performance"}, {"HEAP_LAZY", "performance"}, {"BARRIER_GAME", "performance"}, {"MESH_AHEAD", "performance"}, {"WB_LAZY_TOUCH", "diagnostics"}, {"WB_LAZY_CHUNK", "performance"}, {"FLUSH_UPLOAD", "diagnostics"}, {"GPA_TWIN_MAX_KB", "diagnostics"}, + {"WB_FENCE_LIMIT", "diagnostics"}, {"WB_MERGE", "diagnostics"}, {"WB_EXACT", "diagnostics"}, {"MEM_CENSUS", "diagnostics"}, {"WB_NO_REUP", "diagnostics"}, {"WB_PROBE", "diagnostics"}, {"WB_SPLIT", "performance"}, {"SHADOW_BULK_WRITE", "performance"}, {"WB_LAZY", "performance"}, {"HEAP_LAZY", "performance"}, {"BARRIER_GAME", "performance"}, {"MESH_AHEAD", "performance"}, {"MESH_FALLBACK", "performance"}, {"MESH_FALLBACK_MB", "performance"}, {"WB_LAZY_TOUCH", "diagnostics"}, {"WB_LAZY_CHUNK", "performance"}, {"FLUSH_UPLOAD", "diagnostics"}, {"GPA_TWIN_MAX_KB", "diagnostics"}, {"WB_MERGE_MAX", "diagnostics"}, {"MV_SURVEY", "diagnostics"}, {"TEXUP_LOG_MS", "diagnostics"}, {"VA_REPORT", "diagnostics"}, {"VMA_BLOCK_MB", "diagnostics"}, {"ASTC_BC_PROBE", "diagnostics"}, {"DEVICE_LOST_AT", "diagnostics"}, }; @@ -269,6 +269,9 @@ static const CfgMeta metas[] = { {"PIPE_FRAME_WAIT_MS", "50", "0..500", RESTART}, // vk_exec_draw.c frame_wait {"PIPE_SKIP_STEADY", "1", "0|1", RESTART}, // vk_exec.c steady_on {"PREPARE_PIPELINES", "mesh", "0|mesh|all", RESTART}, // vk_pipeline.c seed_pipelines + // vk_device.c: task and mesh programs through compute passes (vk_exec_draw.c "mesh fallback"); auto: on a device + // without mesh shaders. The megabytes are the buffers the passes write (vk_mesh_fb.c). + {"MESH_FALLBACK", "auto", "auto|0|1", RESTART}, {"MESH_FALLBACK_MB", "128", "16..4096", RESTART}, {"PIPE_AHEAD", "1", "0|1", RESTART}, // vk_exec.c vk_exec_ahead, vk_pipeline.c vk_pipeline_hint {"GPL_FEEDBACK", "1", "0|1", RESTART}, // vk_pipeline.c gpl_feedback {"PIPE_FAST_WORKERS", "1", "0|1", RESTART}, // vk_pipeline.c lib_worker, fast_take diff --git a/runtime/gfx.h b/runtime/gfx.h index bcff7188..21250f95 100644 --- a/runtime/gfx.h +++ b/runtime/gfx.h @@ -100,6 +100,7 @@ size_t gfx_shader_warm_list(GfxWarmEnt **out); // malloc'd; recorde const GfxShader *gfx_shader_load_private(const uint8_t iid[32], int *owned); void gfx_shader_release_private(const GfxShader *s); const uint32_t *gfx_shader_spirv(const GfxShader *s, uint64_t *words); +const char *gfx_shader_res_json(const GfxShader *s); // the translator's .res.json text int gfx_shader_write_code(const GfxShader *s, const char *path); // the guest program, from the stored inputs // A shader of the runtime's own (SPIR-V and a .res.json text as the translator writes them), outside the cache and // never freed; its key is the sha256 of both. NULL when the SPIR-V is malformed. diff --git a/runtime/gfx_shader.c b/runtime/gfx_shader.c index 132c395e..1653b3b7 100644 --- a/runtime/gfx_shader.c +++ b/runtime/gfx_shader.c @@ -2424,6 +2424,8 @@ const uint32_t *gfx_shader_spirv(const GfxShader *s, uint64_t *words) { return s->spirv; } +const char *gfx_shader_res_json(const GfxShader *s) { return s->reflection; } + // The guest program a shader was translated from, as its stored inputs hold it (the whole program: NVN2 code from its // first byte), written to path for a disassembler (NX2NAVI_DUMP's sh_.code), and next to it .job, the // translator request that translates it (one line for nx2navi_xlate --serve; its "out" names the shader cache, and diff --git a/runtime/host.h b/runtime/host.h index 37103c49..72e933a4 100644 --- a/runtime/host.h +++ b/runtime/host.h @@ -436,6 +436,7 @@ int probe_window(void); // gfx_window_probe.c int probe_shader(void); // gfx_shader.c int probe_census_dir(char *dir, size_t n); // gfx_shader_probe.c: build//shaders for the probes int probe_vkmem(void), probe_vkexec(void), probe_astc(void); // tex_probe.c, vk_exec_probe.c, astc_probe.c +int probe_meshfb(void); // mesh_fb_probe.c int probe_zstd(void); // ov_zstd.c int probe_gamepad(void); // input.c int probe_movie(void); // movie_probe.c: the host movie decoders on one RomFS movie diff --git a/runtime/main.c b/runtime/main.c index 14b123c3..9e376322 100644 --- a/runtime/main.c +++ b/runtime/main.c @@ -26,6 +26,8 @@ // --probe-ffx-cache one FSR 3.1 upscaling context and where its pipelines came from (the pipeline caches kept // across runs); skips without amd_fidelityfx_vk.dll, runs no game // --probe-astc the compute ASTC decoder against astc_decode_ldr; needs no game files +// --probe-meshfb the mesh shader fallback: mesh programs drawn through VK_EXT_mesh_shader and through compute +// passes, the pictures compared; needs no game files (mesh_fb_probe.c) // --probe-zstd host zstd streams (ov_zstd.c) on frames made here; needs no game files // --probe-input gamepad polls on a virtual gamepad, also while SDL's update takes long; needs no game files // --probe-movie the host movie decoders on one RomFS movie (VP9 movie::Decoder, AV1 movie::VideoDecoder); skips without game data @@ -535,7 +537,7 @@ static int usage(const char *argv0) { fprintf(stderr, "usage: %s [--exefs DIR] [--romfs DIR] [--profile DIR] [--audio-device VALUE] [--monitor VALUE] [--list-audio-devices | --list-monitors | --selftest | --probe-loader | " "--probe-cfg | --probe-dpi | --probe-mods | --probe-kernel | --probe-ipc | --probe-services | --probe-fs | --probe-gfx-abi | " "--probe-nvn | --probe-nvn-tex | --probe-window | --probe-shader | --probe-vkmem | " - "--probe-vkexec | --probe-astc | --probe-zstd | --probe-input | --probe-movie | --probe-crash-call | --probe-crash-null | --probe-discord | --probe-coop | " + "--probe-vkexec | --probe-meshfb | --probe-astc | --probe-zstd | --probe-input | --probe-movie | --probe-crash-call | --probe-crash-null | --probe-discord | --probe-coop | " "--prepare-shaders | --warm-shaders | --prepare-pipelines | --probe-prewarm | --probe-buildgate | --probe-ffx-cache | --probe-memneed | --probe-devlost | --boot |--seconds S [--shot PNG]]\n", argv0); return 2; } @@ -684,6 +686,7 @@ int main(int argc, char **argv) { if (!strcmp(mode, "--probe-shader")) return probe_shader(); if (!strcmp(mode, "--probe-vkmem")) return probe_vkmem(); if (!strcmp(mode, "--probe-vkexec")) return probe_vkexec(); + if (!strcmp(mode, "--probe-meshfb")) return probe_meshfb(); if (!strcmp(mode, "--probe-astc")) return probe_astc(); if (!strcmp(mode, "--probe-zstd")) return probe_zstd(); if (!strcmp(mode, "--probe-input")) return probe_gamepad(); diff --git a/runtime/mesh_fb.c b/runtime/mesh_fb.c new file mode 100644 index 00000000..985169b5 --- /dev/null +++ b/runtime/mesh_fb.c @@ -0,0 +1,638 @@ +// The mesh shader fallback's modules, for a GPU without VK_EXT_mesh_shader (vk_exec_draw.c "mesh fallback" draws with +// them; docs/architecture.md, "Mesh fallback"). +// +// The translator turns a task or mesh program into a compute module whose shared memory is the program's stage buffer +// and wraps it in a TaskEXT or MeshEXT entry function (tools/xlate/MeshSpirv.cs). The wrapping is undone here and +// replaced: the translator's main function stays as it is, the entry function that hands the stage buffer to the mesh +// pipeline goes, and a compute entry function copies the stage buffer into a storage buffer instead. +// task: main; the payload area into the task buffer at the workgroup's place. +// mesh: [the payload from the task buffer into the payload area] main; the vertex lines and the primitive area into +// the workgroup's slot of the mesh buffer. +// draw: a vertex module made here reads a slot per instance: vertex index / indices per primitive is the +// primitive, its index byte names the vertex, and the vertex's lines are the outputs. +// The module is made from the translation the shader cache already holds, so the translator and with it every cache +// key stay as they are: a card with mesh shaders never sees these modules, and one without needs no other cache. +// +// Mesh workgroups after a task stage. A task workgroup starts up to `threads` mesh workgroups, and how many is known +// only on the GPU. A pass between the two dispatches (shaders/meshfb_scan.comp) lists them in task order: entry g of +// the list names the task workgroup and the index the mesh program sees as its workgroup id. That id comes from the +// WorkgroupId built-in in the translated code; here the built-in counts list entries, so main reads a private +// variable the entry function fills from the list. +// +// Opcode and enum numbers are the SPIR-V 1.6 specification's and SPV_EXT_mesh_shader's. +#include "mesh_fb.h" + +#include +#include +#include +#include + +enum { + OpName = 5, OpExtension = 10, OpMemoryModel = 14, OpEntryPoint = 15, OpExecutionMode = 16, OpCapability = 17, + OpTypeVoid = 19, OpTypeBool = 20, OpTypeInt = 21, OpTypeFloat = 22, OpTypeVector = 23, OpTypeArray = 28, + OpTypeRuntimeArray = 29, OpTypeStruct = 30, OpTypePointer = 32, OpTypeFunction = 33, OpConstant = 43, + OpConstantComposite = 44, OpConstantNull = 46, OpFunction = 54, OpFunctionEnd = 56, OpFunctionCall = 57, + OpVariable = 59, OpLoad = 61, OpStore = 62, OpCopyMemory = 63, OpAccessChain = 65, OpInBoundsAccessChain = 66, + OpPtrAccessChain = 67, OpArrayLength = 68, OpDecorate = 71, OpMemberDecorate = 72, OpCompositeConstruct = 80, + OpCompositeExtract = 81, OpCopyObject = 83, OpBitcast = 124, OpIAdd = 128, OpISub = 130, OpIMul = 132, OpUDiv = 134, + OpSelect = 169, OpINotEqual = 171, OpULessThan = 176, OpULessThanEqual = 178, OpShiftRightLogical = 194, + OpBitwiseAnd = 199, OpControlBarrier = 224, OpPhi = 245, OpLoopMerge = 246, OpSelectionMerge = 247, OpLabel = 248, + OpBranch = 249, OpBranchConditional = 250, OpReturn = 253, OpDecorateId = 332, OpDecorateString = 5632, +}; +enum { ScInput = 1, ScOutput = 3, ScWorkgroup = 4, ScPrivate = 6, ScFunction = 7, ScStorageBuffer = 12, ScTaskPayload = 5402 }; +enum { DecBlock = 2, DecArrayStride = 6, DecBuiltIn = 11, DecInvariant = 18, DecNonWritable = 24, DecLocation = 30, + DecBinding = 33, DecDescriptorSet = 34, DecOffset = 35 }; +enum { BiPosition = 0, BiPointSize = 1, BiWorkgroupId = 26, BiLocalInvocationIndex = 29, BiVertexIndex = 42, + BiInstanceIndex = 43 }; +enum { CapShader = 1, CapMeshShading = 5283, ModelVertex = 0, ModelGLCompute = 5, ModelTask = 5364, ModelMesh = 5365, + ModeLocalSize = 17 }; + +// ---- the program's layout + +static void fail(char *err, size_t cap, const char *fmt, ...) { + if (!err || !cap) return; + va_list ap; + va_start(ap, fmt); + vsnprintf(err, cap, fmt, ap); + va_end(ap); +} + +int meshfb_info(const uint8_t *control, const uint8_t *header, int stage, MeshFbInfo *m, char *err, size_t cap) { + uint32_t w58, w5c, w60; + memset(m, 0, sizeof *m); + memcpy(&w58, control + 0x58, 4), memcpy(&w5c, control + 0x5c, 4), memcpy(&w60, control + 0x60, 4); + m->stage = stage; + m->threads = w58 & 0xffff ? w58 & 0xffff : 1; + m->payload_bytes = (w58 >> 24) * 256; + m->maxv = w60 & 0xffff, m->maxp = w60 >> 16; + uint32_t topo = w5c >> 24; + m->ipp = topo == 1 ? 1 : topo == 2 ? 2 : 3; + m->has_task = stage == 7 && (w5c >> 16 & 0xff) != 0; + // The payload as the translator sizes it (tools/xlate/Program.cs Translate): a size of 0 is one 256-byte unit, and + // the header of 0x20 bytes comes on top. + if (stage == 6 || m->has_task) m->payload_bytes = (m->payload_bytes ? m->payload_bytes : 256) + 0x20; + if (stage == 6) { + m->pay = 0, m->shared_bytes = m->payload_bytes; + return 1; + } + if (stage != 7) return fail(err, cap, "stage %d is neither task nor mesh", stage), 0; + if (!m->maxv || !m->maxp || m->maxv > 256) return fail(err, cap, "limits of %u vertices and %u primitives", m->maxv, m->maxp), 0; + // The output map of the program header: system values B at byte 0x35, one bit per component from 0x60; generic + // vectors from byte 0x36, a nibble each. System values A and C never appear in the game's mesh programs. + if (header[0x34] || header[0x48] || header[0x49]) return fail(err, cap, "output map with system values A or C"), 0; + for (int b = 0; b < 8; b++) + if (header[0x35] >> b & 1) m->lines[m->nlines++] = (uint16_t)(0x60 + 4 * b); + for (int g = 0; g < 32; g++) + for (int c = 0; c < 4; c++) + if (header[0x36 + g / 2] >> (4 * (g & 1) + c) & 1) m->lines[m->nlines++] = (uint16_t)(0x80 + 16 * g + 4 * c); + uint32_t groups = (m->maxv + 31) / 32; + m->attr = 0; + m->map = (m->attr + groups * m->nlines * 128 + 15) & ~15u; + m->pay = (m->map + 4 + m->ipp * m->maxp + 15) & ~15u; + m->shared_bytes = m->pay + (m->has_task ? m->payload_bytes : 0); + return 1; +} + +// ---- a growing list of SPIR-V words + +typedef struct { uint32_t *w; size_t n, cap; int oom; } Wb; + +static void wb_put(Wb *b, const uint32_t *w, size_t n) { + if (b->n + n > b->cap) { + size_t cap = (b->n + n) * 2 + 256; + uint32_t *g = realloc(b->w, cap * 4); + if (!g) { b->oom = 1; return; } + b->w = g, b->cap = cap; + } + memcpy(b->w + b->n, w, n * 4); + b->n += n; +} + +static void wb_ins(Wb *b, uint32_t op, const uint32_t *operands, size_t n) { + uint32_t head = (uint32_t)((n + 1) << 16) | op; + wb_put(b, &head, 1); + wb_put(b, operands, n); +} +#define INS(b, op, ...) wb_ins(b, op, (const uint32_t[]){__VA_ARGS__}, sizeof((const uint32_t[]){__VA_ARGS__}) / 4) + +// What a module under construction adds: decorations, then types, constants and variables, then one function. +typedef struct { + uint32_t bound; + Wb deco, glob, fn; + uint32_t t_void, t_bool, t_uint, t_fnvoid, t_v3uint; // existing or new + uint32_t p_fn_uint, p_sb_uint; + struct { uint32_t v, id; } k[96]; + int nk; +} Cx; + +static uint32_t nid(Cx *c) { return c->bound++; } + +static uint32_t K(Cx *c, uint32_t v) { // a 32-bit unsigned constant + for (int i = 0; i < c->nk; i++) + if (c->k[i].v == v) return c->k[i].id; + uint32_t id = nid(c); + INS(&c->glob, OpConstant, c->t_uint, id, v); + if (c->nk < 96) c->k[c->nk].v = v, c->k[c->nk].id = id, c->nk++; + return id; +} + +static uint32_t E_(Cx *c, uint32_t op, uint32_t type, const uint32_t *operands, size_t n) { // an instruction with a result + uint32_t id = nid(c), head[3] = {(uint32_t)((n + 3) << 16) | op, type, id}; + wb_put(&c->fn, head, 3); + wb_put(&c->fn, operands, n); + return id; +} +#define E(c, op, type, ...) E_(c, op, type, (const uint32_t[]){__VA_ARGS__}, sizeof((const uint32_t[]){__VA_ARGS__}) / 4) +#define DO(c, op, ...) INS(&(c)->fn, op, __VA_ARGS__) + +static uint32_t add(Cx *c, uint32_t a, uint32_t b) { return E(c, OpIAdd, c->t_uint, a, b); } +static uint32_t mul(Cx *c, uint32_t a, uint32_t b) { return E(c, OpIMul, c->t_uint, a, b); } +static uint32_t umin(Cx *c, uint32_t a, uint32_t b) { + return E(c, OpSelect, c->t_uint, E(c, OpULessThan, c->t_bool, a, b), a, b); +} + +// if (cond) { ...the instructions emitted until if_end } +static uint32_t if_begin(Cx *c, uint32_t cond) { + uint32_t then = nid(c), merge = nid(c); + DO(c, OpSelectionMerge, merge, 0); + DO(c, OpBranchConditional, cond, then, merge); + DO(c, OpLabel, then); + return merge; +} +static void if_end(Cx *c, uint32_t merge) { + DO(c, OpBranch, merge); + DO(c, OpLabel, merge); +} + +// for (i = start; i < limit; i += step) { ...the instructions emitted until loop_end }; counter: a Function variable +typedef struct { uint32_t head, cont, merge, i, counter, step; } Loop; +static Loop loop_begin(Cx *c, uint32_t counter, uint32_t start, uint32_t limit, uint32_t step) { + Loop l = {nid(c), nid(c), nid(c), 0, counter, step}; + uint32_t cond = nid(c), body = nid(c); + DO(c, OpStore, counter, start); + DO(c, OpBranch, l.head); + DO(c, OpLabel, l.head); + DO(c, OpLoopMerge, l.merge, l.cont, 0); + DO(c, OpBranch, cond); + DO(c, OpLabel, cond); + l.i = E(c, OpLoad, c->t_uint, counter); + DO(c, OpBranchConditional, E(c, OpULessThan, c->t_bool, l.i, limit), body, l.merge); + DO(c, OpLabel, body); + return l; +} +static void loop_end(Cx *c, const Loop *l) { + DO(c, OpBranch, l->cont); + DO(c, OpLabel, l->cont); + DO(c, OpStore, l->counter, add(c, l->i, l->step)); + DO(c, OpBranch, l->head); + DO(c, OpLabel, l->merge); +} + +// A storage buffer of 32-bit words at (set 0, binding): its variable. readonly: the module only reads it. +static uint32_t word_buffer(Cx *c, uint32_t binding, int readonly) { + uint32_t arr = nid(c), st = nid(c), ptr = nid(c), var = nid(c); + INS(&c->glob, OpTypeRuntimeArray, arr, c->t_uint); + INS(&c->glob, OpTypeStruct, st, arr); + INS(&c->glob, OpTypePointer, ptr, ScStorageBuffer, st); + INS(&c->glob, OpVariable, ptr, var, ScStorageBuffer); + INS(&c->deco, OpDecorate, arr, DecArrayStride, 4); + INS(&c->deco, OpDecorate, st, DecBlock); + INS(&c->deco, OpMemberDecorate, st, 0, DecOffset, 0); + if (readonly) INS(&c->deco, OpMemberDecorate, st, 0, DecNonWritable); + INS(&c->deco, OpDecorate, var, DecDescriptorSet, 0); + INS(&c->deco, OpDecorate, var, DecBinding, binding); + return var; +} +static uint32_t buf_ptr(Cx *c, uint32_t var, uint32_t index) { return E(c, OpAccessChain, c->p_sb_uint, var, K(c, 0), index); } +static uint32_t buf_load(Cx *c, uint32_t var, uint32_t index) { return E(c, OpLoad, c->t_uint, buf_ptr(c, var, index)); } + +// ---- a translated task or mesh module to its compute module + +typedef struct { uint32_t at, op, len; } In; // an instruction: its first word's index, opcode, word count + +static int id_in(const uint32_t *set, int n, uint32_t id) { + for (int i = 0; i < n; i++) + if (set[i] == id) return 1; + return 0; +} + +static int is_decl(uint32_t op) { // a type, constant or global variable declaration + return (op >= OpTypeVoid && op <= 39) || (op >= 41 && op <= 52) || op == OpVariable || op == 1; +} + +static uint32_t *convert(const uint32_t *w, uint64_t words, const MeshFbInfo *m, uint32_t binding, uint32_t task_words, + uint64_t *out_words, char *err, size_t cap) { + In *in = NULL; + uint32_t *removed = NULL, *gvars = NULL; + Wb out = {0}; + Cx c = {0}; +#define FAIL(...) do { fail(err, cap, __VA_ARGS__); goto bad; } while (0) + if (words < 6 || w[0] != 0x07230203) return fail(err, cap, "not a SPIR-V module"), NULL; + if (w[1] < 0x00010400) return fail(err, cap, "SPIR-V %u.%u, 1.4 expected", w[1] >> 16 & 0xff, w[1] >> 8 & 0xff), NULL; + size_t n = 0, ncap = 0; + for (uint64_t p = 5; p < words;) { + uint32_t len = w[p] >> 16, op = w[p] & 0xffff; + if (!len || p + len > words) return fail(err, cap, "malformed SPIR-V"), NULL; + if (n == ncap) { + ncap = ncap ? ncap * 2 : 4096; + In *g = realloc(in, ncap * sizeof *in); + if (!g) FAIL("out of memory"); + in = g; + } + in[n++] = (In){(uint32_t)p, op, len}; + p += len; + } + c.bound = w[3]; + int mesh = m->stage == 7, n_removed = 0, n_gvars = 0, n_entry = 0, n_shared = 0; + removed = malloc((n + 1) * 4), gvars = malloc((n + 16) * 4); + if (!removed || !gvars) FAIL("out of memory"); + uint32_t old_entry = 0, model = 0, shared_var = 0, shared_type = 0, wg_var = 0, lii_var = 0, first_fn = 0, first_decl = 0; + // pass 1: what the module has + for (size_t i = 0; i < n; i++) { + const uint32_t *a = w + in[i].at + 1; + uint32_t op = in[i].op, len = in[i].len; + if (op == OpEntryPoint) n_entry++, model = a[0], old_entry = a[1]; + else if (op == OpTypeVoid) c.t_void = a[0]; + else if (op == OpTypeBool) c.t_bool = a[0]; + else if (op == OpTypeInt && a[1] == 32 && a[2] == 0) c.t_uint = a[0]; + else if (op == OpTypeVector && a[1] == c.t_uint && c.t_uint && a[2] == 3) c.t_v3uint = a[0]; + else if (op == OpTypeFunction && len == 3 && a[1] == c.t_void && c.t_void) c.t_fnvoid = a[0]; + else if (op == OpTypePointer && (a[1] == ScOutput || a[1] == ScTaskPayload)) removed[n_removed++] = a[0]; + else if (op == OpVariable && !first_fn) { + if (a[2] == ScOutput || a[2] == ScTaskPayload) removed[n_removed++] = a[1]; + else gvars[n_gvars++] = a[1]; + if (a[2] == ScWorkgroup) n_shared++, shared_var = a[1], shared_type = a[0]; + } else if (op == OpFunction && !first_fn) first_fn = (uint32_t)i; + if (!first_decl && !first_fn && is_decl(op)) first_decl = (uint32_t)i; + } + if (n_entry != 1 || model != (uint32_t)(mesh ? ModelMesh : ModelTask)) FAIL("the entry point is not a %s one", mesh ? "mesh" : "task"); + if (n_shared != 1) FAIL("%d workgroup variables (one shared memory array expected)", n_shared); + if (!c.t_void || !c.t_bool || !c.t_uint || !c.t_fnvoid || !first_fn || !first_decl) FAIL("a basic type or the functions are missing"); + { // the shared memory array holds the layout this module is asked to copy + uint32_t arr = 0, len_id = 0, len = 0; + for (size_t i = 0; i < n; i++) { + const uint32_t *a = w + in[i].at + 1; + if (in[i].op == OpTypePointer && a[0] == shared_type) arr = a[2]; + } + for (size_t i = 0; i < n; i++) { + const uint32_t *a = w + in[i].at + 1; + if (in[i].op == OpTypeArray && a[0] == arr && a[1] == c.t_uint) len_id = a[2]; + } + for (size_t i = 0; i < n; i++) { + const uint32_t *a = w + in[i].at + 1; + if (in[i].op == OpConstant && a[1] == len_id && len_id) len = a[2]; + } + if (len * 4 < m->shared_bytes) FAIL("shared memory of %u bytes, the program's layout needs %u", len * 4, m->shared_bytes); + } + for (size_t i = 0; i < n; i++) { + const uint32_t *a = w + in[i].at + 1; + if (in[i].op != OpDecorate || in[i].len < 4 || a[1] != DecBuiltIn || id_in(removed, n_removed, a[0])) continue; + if (a[2] == BiWorkgroupId) wg_var = a[0]; + if (a[2] == BiLocalInvocationIndex) lii_var = a[0]; + } + // the old entry function, and the function it calls: the translator's main + size_t e0 = 0, e1 = 0; + uint32_t main_fn = 0, n_calls = 0; + for (size_t i = first_fn; i < n; i++) + if (in[i].op == OpFunction && w[in[i].at + 2] == old_entry) { + e0 = i; + for (e1 = i; e1 < n && in[e1].op != OpFunctionEnd; e1++) + if (in[e1].op == OpFunctionCall) n_calls++, main_fn = w[in[e1].at + 3]; + break; + } + if (!e0 || e1 >= n || n_calls != 1) FAIL("the entry function calls %u functions (one expected)", n_calls); + + // ---- new types, variables and the entry function + if (!c.t_v3uint) c.t_v3uint = nid(&c), INS(&c.glob, OpTypeVector, c.t_v3uint, c.t_uint, 3); + c.p_fn_uint = nid(&c), INS(&c.glob, OpTypePointer, c.p_fn_uint, ScFunction, c.t_uint); + c.p_sb_uint = nid(&c), INS(&c.glob, OpTypePointer, c.p_sb_uint, ScStorageBuffer, c.t_uint); + uint32_t p_shared = nid(&c); + INS(&c.glob, OpTypePointer, p_shared, ScWorkgroup, c.t_uint); + if (!lii_var) { + uint32_t pt = nid(&c); + lii_var = nid(&c); + INS(&c.glob, OpTypePointer, pt, ScInput, c.t_uint); + INS(&c.glob, OpVariable, pt, lii_var, ScInput); + INS(&c.deco, OpDecorate, lii_var, DecBuiltIn, BiLocalInvocationIndex); + gvars[n_gvars++] = lii_var; + } + if (!wg_var) { + uint32_t pt = nid(&c); + wg_var = nid(&c); + INS(&c.glob, OpTypePointer, pt, ScInput, c.t_v3uint); + INS(&c.glob, OpVariable, pt, wg_var, ScInput); + INS(&c.deco, OpDecorate, wg_var, DecBuiltIn, BiWorkgroupId); + gvars[n_gvars++] = wg_var; + } + uint32_t fake_wg = 0, p_priv_uint = 0; + if (mesh) { // the workgroup id main sees (see the top) + uint32_t pt = nid(&c); + fake_wg = nid(&c), p_priv_uint = nid(&c); + INS(&c.glob, OpTypePointer, pt, ScPrivate, c.t_v3uint); + INS(&c.glob, OpTypePointer, p_priv_uint, ScPrivate, c.t_uint); + INS(&c.glob, OpVariable, pt, fake_wg, ScPrivate); + gvars[n_gvars++] = fake_wg; + } + uint32_t task_buf = 0, mesh_buf = 0, list_buf = 0; + if (!mesh || m->has_task) task_buf = word_buffer(&c, binding, mesh), gvars[n_gvars++] = task_buf; + if (mesh) mesh_buf = word_buffer(&c, binding + 1, 0), gvars[n_gvars++] = mesh_buf; + if (mesh && m->has_task) list_buf = word_buffer(&c, binding + 2, 1), gvars[n_gvars++] = list_buf; + + uint32_t threads = K(&c, m->threads), k0 = K(&c, 0), k1 = K(&c, 1); + uint32_t entry = nid(&c); + DO(&c, OpFunction, c.t_void, entry, 0, c.t_fnvoid); + DO(&c, OpLabel, nid(&c)); + uint32_t counter = nid(&c); + DO(&c, OpVariable, c.p_fn_uint, counter, ScFunction); + uint32_t tid = E(&c, OpLoad, c.t_uint, lii_var); + uint32_t g = E(&c, OpCompositeExtract, c.t_uint, E(&c, OpLoad, c.t_v3uint, wg_var), 0); + uint32_t first = E(&c, OpULessThan, c.t_bool, tid, k1); // the workgroup's first thread +#define SHARED(index) E(&c, OpAccessChain, p_shared, shared_var, index) +#define BARRIER() DO(&c, OpControlBarrier, K(&c, 2), K(&c, 2), K(&c, 0x108)) /* Workgroup, AcquireRelease | WorkgroupMemory */ + if (!mesh) { + uint32_t pw = m->payload_bytes / 4; + uint32_t at = mul(&c, g, K(&c, pw)); + uint32_t room = E(&c, OpULessThanEqual, c.t_bool, add(&c, at, K(&c, pw)), E(&c, OpArrayLength, c.t_uint, task_buf, 0)); + // the mesh workgroup count starts at 0: a task that culls everything never writes it + uint32_t mg = if_begin(&c, first); + DO(&c, OpStore, SHARED(K(&c, (m->pay + 4) / 4)), k0); + if_end(&c, mg); + BARRIER(); + E(&c, OpFunctionCall, c.t_void, main_fn); + BARRIER(); + mg = if_begin(&c, room); + Loop l = loop_begin(&c, counter, tid, K(&c, pw), threads); + DO(&c, OpStore, buf_ptr(&c, task_buf, add(&c, at, l.i)), E(&c, OpLoad, c.t_uint, SHARED(add(&c, l.i, K(&c, m->pay / 4))))); + loop_end(&c, &l); + if_end(&c, mg); + } else { + uint32_t slot_words = meshfb_slot_words(m), count_word = K(&c, m->map / 4); + uint32_t j = g, task_at = 0; + if (m->has_task) { + uint32_t e = buf_load(&c, list_buf, g); + task_at = mul(&c, E(&c, OpBitwiseAnd, c.t_uint, e, K(&c, (1u << MESHFB_LIST_TASK_BITS) - 1)), K(&c, task_words)); + j = E(&c, OpShiftRightLogical, c.t_uint, e, K(&c, MESHFB_LIST_TASK_BITS)); + } + DO(&c, OpStore, fake_wg, E(&c, OpCompositeConstruct, c.t_v3uint, j, k0, k0)); + uint32_t at = mul(&c, g, K(&c, slot_words)); + uint32_t room = E(&c, OpULessThanEqual, c.t_bool, add(&c, at, K(&c, slot_words)), E(&c, OpArrayLength, c.t_uint, mesh_buf, 0)); + // the primitive count starts at 0: a mesh program that emits nothing never writes it + uint32_t mg = if_begin(&c, first); + DO(&c, OpStore, SHARED(count_word), k0); + if_end(&c, mg); + if (m->has_task) { + uint32_t pw = m->payload_bytes / 4; + if (pw > task_words) pw = task_words; + Loop l = loop_begin(&c, counter, tid, K(&c, pw), threads); + DO(&c, OpStore, SHARED(add(&c, l.i, K(&c, m->pay / 4))), buf_load(&c, task_buf, add(&c, task_at, l.i))); + loop_end(&c, &l); + } + BARRIER(); + E(&c, OpFunctionCall, c.t_void, main_fn); + BARRIER(); + uint32_t count = E(&c, OpLoad, c.t_uint, SHARED(count_word)); + mg = if_begin(&c, room); + { // a slot without primitives gets its count alone + uint32_t some = E(&c, OpINotEqual, c.t_bool, count, k0), then = nid(&c), other = nid(&c), merge = nid(&c); + DO(&c, OpSelectionMerge, merge, 0); + DO(&c, OpBranchConditional, some, then, other); + DO(&c, OpLabel, then); + Loop l = loop_begin(&c, counter, tid, K(&c, slot_words), threads); + DO(&c, OpStore, buf_ptr(&c, mesh_buf, add(&c, at, l.i)), E(&c, OpLoad, c.t_uint, SHARED(l.i))); + loop_end(&c, &l); + DO(&c, OpBranch, merge); + DO(&c, OpLabel, other); + uint32_t m2 = if_begin(&c, first); + DO(&c, OpStore, buf_ptr(&c, mesh_buf, add(&c, at, count_word)), k0); + if_end(&c, m2); + DO(&c, OpBranch, merge); + DO(&c, OpLabel, merge); + } + if_end(&c, mg); + } + wb_ins(&c.fn, OpReturn, NULL, 0); + wb_ins(&c.fn, OpFunctionEnd, NULL, 0); + + // ---- pass 2: the module again + uint32_t head[5] = {w[0], w[1], w[2], 0, 0}; + wb_put(&out, head, 5); + for (size_t i = 0; i < n; i++) { + const uint32_t *a = w + in[i].at + 1; + uint32_t op = in[i].op, len = in[i].len; + if (i == first_decl) wb_put(&out, c.deco.w, c.deco.n); + if (i == first_fn) wb_put(&out, c.glob.w, c.glob.n); + if (i >= e0 && i <= e1) continue; // the old entry function + if (op == OpCapability && a[0] == CapMeshShading) continue; + if (op == OpExtension && len >= 6 && !memcmp(a, "SPV_EXT_mesh_shader", 20)) continue; + if (op == OpExecutionMode && a[1] != ModeLocalSize) continue; + if ((op == OpName || op == OpDecorate || op == OpDecorateId || op == OpDecorateString) && id_in(removed, n_removed, a[0])) continue; + if ((op == OpTypePointer && id_in(removed, n_removed, a[0])) || (op == OpVariable && id_in(removed, n_removed, a[1]))) continue; + if (op == OpEntryPoint) { // every global variable is its interface (SPIR-V 1.4) + uint32_t ep[4] = {(uint32_t)((n_gvars + 5) << 16) | OpEntryPoint, ModelGLCompute, entry, 0x6e69616d /* "main" */}; + uint32_t zero = 0; + wb_put(&out, ep, 4), wb_put(&out, &zero, 1); + wb_put(&out, gvars, (size_t)n_gvars); + continue; + } + size_t at = out.n; + wb_put(&out, w + in[i].at, len); + if (out.oom) break; + uint32_t *o = out.w + at + 1; + if (op == OpExecutionMode) o[0] = entry; + if (mesh && i > first_fn) { // main and what it calls read the workgroup id from the private variable + if ((op == OpAccessChain || op == OpInBoundsAccessChain) && a[2] == wg_var) { + uint32_t pointee = 0; + for (size_t t = first_decl; t < first_fn; t++) + if (in[t].op == OpTypePointer && w[in[t].at + 1] == a[0]) pointee = w[in[t].at + 3]; + if (pointee != c.t_uint || len != 5) FAIL("the workgroup id is read in a way the fallback does not know"); + o[0] = p_priv_uint, o[2] = fake_wg; + } else if (op == OpLoad && a[2] == wg_var) { + o[2] = fake_wg; + } else if ((op == OpStore && a[0] == wg_var) || (op == OpCopyMemory && (a[0] == wg_var || a[1] == wg_var)) || + ((op == OpPtrAccessChain || op == OpCopyObject) && a[2] == wg_var)) { + FAIL("the workgroup id is read in a way the fallback does not know"); + } else if (op == OpFunctionCall || op == OpSelect || op == OpPhi) { + for (uint32_t k = 2; k + 1 < len; k++) + if (a[k] == wg_var) FAIL("the workgroup id is passed on as a pointer"); + } + } + } + wb_put(&out, c.fn.w, c.fn.n); + if (out.oom || c.deco.oom || c.glob.oom || c.fn.oom) FAIL("out of memory"); + out.w[3] = c.bound; + *out_words = out.n; + free(in), free(removed), free(gvars), free(c.deco.w), free(c.glob.w), free(c.fn.w); + return out.w; +bad: + free(in), free(removed), free(gvars), free(c.deco.w), free(c.glob.w), free(c.fn.w), free(out.w); + return NULL; +#undef FAIL +#undef SHARED +#undef BARRIER +} + +uint32_t *meshfb_task_cs(const uint32_t *spirv, uint64_t words, const MeshFbInfo *m, uint32_t binding, uint64_t *out_words, + char *err, size_t cap) { + if (m->stage != 6) return fail(err, cap, "not a task program"), NULL; + return convert(spirv, words, m, binding, 0, out_words, err, cap); +} + +uint32_t *meshfb_mesh_cs(const uint32_t *spirv, uint64_t words, const MeshFbInfo *m, uint32_t binding, + uint32_t task_payload_words, uint64_t *out_words, char *err, size_t cap) { + if (m->stage != 7) return fail(err, cap, "not a mesh program"), NULL; + if (m->has_task && !task_payload_words) return fail(err, cap, "a mesh program after a task stage without its payload size"), NULL; + return convert(spirv, words, m, binding, task_payload_words, out_words, err, cap); +} + +// ---- the vertex module + +uint32_t *meshfb_vs(const MeshFbInfo *m, uint32_t out_mask, const uint8_t out_type[32], uint64_t *out_words) { + Cx c = {.bound = 1}; + Wb out = {0}; + uint32_t iface[40], n_iface = 0; + c.t_void = nid(&c), INS(&c.glob, OpTypeVoid, c.t_void); + c.t_bool = nid(&c), INS(&c.glob, OpTypeBool, c.t_bool); + c.t_uint = nid(&c), INS(&c.glob, OpTypeInt, c.t_uint, 32, 0); + uint32_t t_int = nid(&c), t_float = nid(&c), t_v4[3], p_out[3], t_scalar[3]; // by GFX_TYPE_*: float, sint, uint + INS(&c.glob, OpTypeInt, t_int, 32, 1); + INS(&c.glob, OpTypeFloat, t_float, 32); + t_scalar[0] = t_float, t_scalar[1] = t_int, t_scalar[2] = c.t_uint; + for (int t = 0; t < 3; t++) { + t_v4[t] = nid(&c), INS(&c.glob, OpTypeVector, t_v4[t], t_scalar[t], 4); + p_out[t] = nid(&c), INS(&c.glob, OpTypePointer, p_out[t], ScOutput, t_v4[t]); + } + c.t_fnvoid = nid(&c), INS(&c.glob, OpTypeFunction, c.t_fnvoid, c.t_void); + c.p_sb_uint = nid(&c), INS(&c.glob, OpTypePointer, c.p_sb_uint, ScStorageBuffer, c.t_uint); + uint32_t p_in_int = nid(&c), p_out_float = nid(&c); + INS(&c.glob, OpTypePointer, p_in_int, ScInput, t_int); + INS(&c.glob, OpTypePointer, p_out_float, ScOutput, t_float); + uint32_t vi_var = nid(&c), ii_var = nid(&c), pos_var = nid(&c), size_var = 0, buf, loc_var[32] = {0}; + INS(&c.glob, OpVariable, p_in_int, vi_var, ScInput); + INS(&c.glob, OpVariable, p_in_int, ii_var, ScInput); + INS(&c.glob, OpVariable, p_out[0], pos_var, ScOutput); + INS(&c.deco, OpDecorate, vi_var, DecBuiltIn, BiVertexIndex); + INS(&c.deco, OpDecorate, ii_var, DecBuiltIn, BiInstanceIndex); + INS(&c.deco, OpDecorate, pos_var, DecBuiltIn, BiPosition); + INS(&c.deco, OpDecorate, pos_var, DecInvariant); + iface[n_iface++] = vi_var, iface[n_iface++] = ii_var, iface[n_iface++] = pos_var; + if (m->ipp == 1) { // a point list needs its size written + size_var = nid(&c); + INS(&c.glob, OpVariable, p_out_float, size_var, ScOutput); + INS(&c.deco, OpDecorate, size_var, DecBuiltIn, BiPointSize); + iface[n_iface++] = size_var; + } + for (int g = 0; g < 32; g++) { + if (!(out_mask >> g & 1)) continue; + int t = out_type[g] < 3 ? out_type[g] : 0; + loc_var[g] = nid(&c); + INS(&c.glob, OpVariable, p_out[t], loc_var[g], ScOutput); + INS(&c.deco, OpDecorate, loc_var[g], DecLocation, (uint32_t)g); + iface[n_iface++] = loc_var[g]; + } + buf = word_buffer(&c, 1, 1); + iface[n_iface++] = buf; + uint32_t f0 = nid(&c), f1 = nid(&c), fout = nid(&c), null_v4[3], away = nid(&c); + INS(&c.glob, OpConstant, t_float, f0, 0); + INS(&c.glob, OpConstant, t_float, f1, 0x3f800000); + INS(&c.glob, OpConstant, t_float, fout, 0xc1800000); // -16: outside the clip volume, for a point of no primitive + for (int t = 0; t < 3; t++) null_v4[t] = nid(&c), INS(&c.glob, OpConstantNull, t_v4[t], null_v4[t]); + INS(&c.glob, OpConstantComposite, t_v4[0], away, fout, fout, f0, f1); + + uint32_t slot_words = meshfb_slot_words(m), k0 = K(&c, 0), entry = nid(&c); + DO(&c, OpFunction, c.t_void, entry, 0, c.t_fnvoid); + DO(&c, OpLabel, nid(&c)); + uint32_t vi = E(&c, OpBitcast, c.t_uint, E(&c, OpLoad, t_int, vi_var)); + uint32_t slot = E(&c, OpBitcast, c.t_uint, E(&c, OpLoad, t_int, ii_var)); + uint32_t prim = m->ipp == 1 ? vi : E(&c, OpUDiv, c.t_uint, vi, K(&c, m->ipp)); + uint32_t corner = m->ipp == 1 ? k0 : E(&c, OpISub, c.t_uint, vi, mul(&c, prim, K(&c, m->ipp))); + uint32_t base = mul(&c, slot, K(&c, slot_words)); + // a slot past the buffer (more workgroups than it has room for) draws nothing + uint32_t valid = E(&c, OpULessThanEqual, c.t_bool, add(&c, base, K(&c, slot_words)), E(&c, OpArrayLength, c.t_uint, buf, 0)); + uint32_t cw = buf_load(&c, buf, E(&c, OpSelect, c.t_uint, valid, add(&c, base, K(&c, m->map / 4)), k0)); + uint32_t count = E(&c, OpSelect, c.t_uint, valid, umin(&c, cw, K(&c, m->maxp)), k0); + uint32_t live = E(&c, OpULessThan, c.t_bool, prim, count), then = nid(&c), other = nid(&c), merge = nid(&c); + DO(&c, OpSelectionMerge, merge, 0); + DO(&c, OpBranchConditional, live, then, other); + DO(&c, OpLabel, then); + { + // byte map + 4 + ipp * prim + corner of the slot is the vertex + uint32_t b = add(&c, add(&c, mul(&c, prim, K(&c, m->ipp)), corner), K(&c, m->map + 4)); + uint32_t word = buf_load(&c, buf, add(&c, base, E(&c, OpShiftRightLogical, c.t_uint, b, K(&c, 2)))); + uint32_t shift = mul(&c, E(&c, OpBitwiseAnd, c.t_uint, b, K(&c, 3)), K(&c, 8)); + uint32_t v = umin(&c, E(&c, OpBitwiseAnd, c.t_uint, E(&c, OpShiftRightLogical, c.t_uint, word, shift), K(&c, 0xff)), K(&c, m->maxv - 1)); + // word ((v >> 5) * lines + line) * 32 + (v & 31) of the vertex area + uint32_t at = add(&c, add(&c, mul(&c, E(&c, OpShiftRightLogical, c.t_uint, v, K(&c, 5)), K(&c, m->nlines * 32)), + E(&c, OpBitwiseAnd, c.t_uint, v, K(&c, 31))), add(&c, base, K(&c, m->attr / 4))); + uint32_t pos[4] = {f0, f0, f0, f1}, size = f1, gen[32][4]; + memset(gen, 0, sizeof gen); + for (uint32_t l = 0; l < m->nlines; l++) { + uint32_t a = m->lines[l]; + int g = a >= 0x80 ? (int)(a - 0x80) / 16 : -1; + if (a < 0x6c || (g >= 0 && !loc_var[g])) continue; // layer and viewport index are not carried; an output nothing reads + uint32_t word_l = buf_load(&c, buf, l ? add(&c, at, K(&c, l * 32)) : at); + if (g >= 0) gen[g][(a - 0x80) % 16 / 4] = word_l; + else if (a == 0x6c) size = E(&c, OpBitcast, t_float, word_l); + else pos[(a - 0x70) / 4] = E(&c, OpBitcast, t_float, word_l); + } + DO(&c, OpStore, pos_var, E(&c, OpCompositeConstruct, t_v4[0], pos[0], pos[1], pos[2], pos[3])); + if (size_var) DO(&c, OpStore, size_var, size); + for (int g = 0; g < 32; g++) { + if (!loc_var[g]) continue; + int t = out_type[g] < 3 ? out_type[g] : 0; + uint32_t comp[4]; + for (int k = 0; k < 4; k++) { + uint32_t word_k = gen[g][k] ? gen[g][k] : k0; + comp[k] = t == 2 ? word_k : E(&c, OpBitcast, t_scalar[t], word_k); + } + DO(&c, OpStore, loc_var[g], E(&c, OpCompositeConstruct, t_v4[t], comp[0], comp[1], comp[2], comp[3])); + } + } + DO(&c, OpBranch, merge); + DO(&c, OpLabel, other); + DO(&c, OpStore, pos_var, away); + if (size_var) DO(&c, OpStore, size_var, f1); + for (int g = 0; g < 32; g++) + if (loc_var[g]) DO(&c, OpStore, loc_var[g], null_v4[out_type[g] < 3 ? out_type[g] : 0]); + DO(&c, OpBranch, merge); + DO(&c, OpLabel, merge); + wb_ins(&c.fn, OpReturn, NULL, 0); + wb_ins(&c.fn, OpFunctionEnd, NULL, 0); + + uint32_t head[5] = {0x07230203, 0x00010400, 0, c.bound, 0}; + wb_put(&out, head, 5); + INS(&out, OpCapability, CapShader); + INS(&out, OpMemoryModel, 0, 1); // Logical, GLSL450 + uint32_t ep[5] = {(n_iface + 5) << 16 | OpEntryPoint, ModelVertex, entry, 0x6e69616d /* "main" */, 0}; + wb_put(&out, ep, 5); + wb_put(&out, iface, n_iface); + wb_put(&out, c.deco.w, c.deco.n); + wb_put(&out, c.glob.w, c.glob.n); + wb_put(&out, c.fn.w, c.fn.n); + int oom = out.oom || c.deco.oom || c.glob.oom || c.fn.oom; + free(c.deco.w), free(c.glob.w), free(c.fn.w); + if (oom) return free(out.w), NULL; + *out_words = out.n; + return out.w; +} + +// ---- reflection text + +char *meshfb_res_json(const char *res, uint32_t binding, const int *cb_slots, int n) { + size_t len = strlen(res); + char *out = malloc(len + (size_t)n * 96 + 1), *o = out; + if (!out) return NULL; + const char *sb = strstr(res, "\"sbufs\""), *open = sb ? strchr(sb, '[') : NULL; + if (!open) return free(out), NULL; + memcpy(o, res, (size_t)(open + 1 - res)), o += open + 1 - res; + const char *rest = open + 1; + while (*rest == ' ' || *rest == '\n' || *rest == '\r' || *rest == '\t') rest++; + for (int i = 0; i < n; i++) + o += sprintf(o, "{\"binding\": %u, \"slot\": -1, \"cb_slot\": %d, \"cb_offset\": 0, \"flags\": %d}%s", binding + (uint32_t)i, + cb_slots[i], cb_slots[i] == MESHFB_CB_LIST ? 0 : 1, i + 1 < n || *rest != ']' ? "," : ""); + strcpy(o, open + 1); + char *st = strstr(out, "\"stage\""); + st = st ? strchr(st + 7, '"') : NULL; + if (!st || (st[1] != 'm' && st[1] != 't') || st[2] != 's') return free(out), NULL; + st[1] = 'c'; + return out; +} diff --git a/runtime/mesh_fb.h b/runtime/mesh_fb.h new file mode 100644 index 00000000..c3b5188c --- /dev/null +++ b/runtime/mesh_fb.h @@ -0,0 +1,51 @@ +// The mesh shader fallback's modules (mesh_fb.c): what a task or mesh program computes, as compute modules that write +// it into storage buffers, and the vertex stage that draws from those buffers. No Vulkan calls here. +#pragma once +#include +#include + +#define MESHFB_MAX_LINES 136 // 8 system value components and 32 generic vectors of 4 + +// A task or mesh program's stage buffer, as tools/xlate/AmpereLowering.cs MeshInfo lays it out in the shared memory +// of the translated module. +typedef struct { + int stage; // 6 task, 7 mesh + uint32_t threads; // workgroup size + uint32_t maxv, maxp; // mesh: vertex and primitive limits + uint32_t ipp; // mesh: indices per primitive (1 points, 2 lines, 3 triangles) + uint32_t payload_bytes; // the task payload (task, or a mesh program a task feeds), with its header + int has_task; // mesh: a task stage feeds it + uint32_t nlines; // mesh: output components per vertex + uint16_t lines[MESHFB_MAX_LINES]; // their attribute addresses: 0x6c point size, 0x70.. position, 0x80 + 16 g + 4 c + uint32_t attr, map, pay, shared_bytes; // byte offsets in shared memory, and its size +} MeshFbInfo; + +// Where a module's extra storage buffers are bound: res.json "cb_slot" of their entries (vk_exec_draw.c ssbo_info). +enum { MESHFB_CB_TASK = 0x41, MESHFB_CB_MESH = 0x42, MESHFB_CB_LIST = 0x43 }; +// A list entry: the task workgroup in bits 0-23, the mesh workgroup's index among that task's in bits 24-31. +#define MESHFB_LIST_TASK_BITS 24 + +// From the program's control section and the first 0x80 bytes of its code (the header with the output map). 0 with +// err filled when the program uses what the fallback does not carry. +int meshfb_info(const uint8_t *control, const uint8_t *header, int stage, MeshFbInfo *out, char *err, size_t cap); +// Words one mesh workgroup's output takes in the mesh buffer: its vertex lines and its primitive area. +static inline uint32_t meshfb_slot_words(const MeshFbInfo *m) { return m->pay / 4; } + +// The compute module of a TaskEXT module (spirv, as the translator wrote it): the same program, then the payload of +// workgroup t into words [t * payload_bytes / 4, ..) of the task buffer, a storage buffer at `binding`. malloc'd; +// NULL with err filled. +uint32_t *meshfb_task_cs(const uint32_t *spirv, uint64_t words, const MeshFbInfo *m, uint32_t binding, uint64_t *out_words, + char *err, size_t cap); +// The compute module of a MeshEXT module. Workgroup g writes its stage buffer (meshfb_slot_words) into slot g of the +// mesh buffer at binding + 1. After a task stage (m->has_task) entry g of the list at binding + 2 names its task +// workgroup and its index among that task's mesh workgroups, and the payload comes from the task buffer at `binding`, +// task_payload_words apart. +uint32_t *meshfb_mesh_cs(const uint32_t *spirv, uint64_t words, const MeshFbInfo *m, uint32_t binding, + uint32_t task_payload_words, uint64_t *out_words, char *err, size_t cap); +// The vertex module that draws the mesh buffer (binding 1): instance = slot, maxp * ipp vertices an instance; vertex +// index / ipp is the primitive, and a primitive past the slot's count is degenerate. Outputs: position, point size for +// points, and a 4-component vector at every Location of out_mask, of out_type[location] (GFX_TYPE_*). +uint32_t *meshfb_vs(const MeshFbInfo *m, uint32_t out_mask, const uint8_t out_type[32], uint64_t *out_words); +// A copy of the translator's res.json text res for a compute module made above: its stage is "cs" and its storage +// buffers gain the entries of the extra buffers (n of them from `binding` on, cb_slots[i] each). malloc'd. +char *meshfb_res_json(const char *res, uint32_t binding, const int *cb_slots, int n); diff --git a/runtime/mesh_fb_probe.c b/runtime/mesh_fb_probe.c new file mode 100644 index 00000000..8b514787 --- /dev/null +++ b/runtime/mesh_fb_probe.c @@ -0,0 +1,432 @@ +// nx2navi.exe --probe-meshfb: the mesh shader fallback (mesh_fb.c, vk_mesh_fb.c, vk_exec_draw.c "mesh fallback"). +// +// Draw part, no game files: two mesh programs of the runtime's own in the shape the translator gives one +// (shaders/meshfb_probe_*.mesh, .task), one without a task stage and one fed by a task program, drawn through the +// replay into an RGBA8 target with direct and with indirect workgroup counts, and both in one pass after each other. +// Each picture is drawn twice, through VK_EXT_mesh_shader and through the fallback, and the two are compared texel by +// texel; without mesh shaders on the device only the fallback draws, and the probe says that nothing was compared. +// +// Module part, with NX2NAVI_PROBE_MESHFB_DIR=: every "" of /list.txt names /.in (a stored +// shader input, gfx_shader.c in_store: control section and code) and /.spv (its translation). The compute +// module and, for a mesh program, a vertex module are written to /out/.cs.spv and .vs.spv for spirv-val +// (work the caller does; tests of game shaders stay outside the repository). Where /.res.json (the +// translation's reflection) is there too, the compute pipeline of the module is built on this device, as its first +// draw would build it. +#include +#include +#include + +#include "cfg.h" +#include "gfx.h" +#include "host.h" +#include "mesh_fb.h" +#include "nvn_api.h" +#include "nvn_tex.h" +#include "vk.h" +#include "vk_exec.h" +#include "vk_mesh_fb.h" +#include "platform/platform.h" + +#include "meshfb_probe_f_spv.h" +#include "meshfb_probe_m_spv.h" +#include "meshfb_probe_t_spv.h" +#include "meshfb_probe_tm_spv.h" + +#define TEX_W 256 +#define TEX_H 144 +#define CHECK(cond, ...) \ + do { \ + if (!(cond)) { \ + printf("probe-meshfb: FAILED: "); \ + printf(__VA_ARGS__); \ + printf("\n"); \ + return 1; \ + } \ + } while (0) + +void vk_exec_probe_shader(const uint8_t *ctl, const GfxShader *s); // vk_exec_draw.c +void vk_exec_draw_shutdown(void); +void vk_exec_xfer_shutdown(void); + +// ---- module part + +static uint8_t *read_all(const char *path, size_t *size) { + FILE *f = fopen(path, "rb"); + if (!f) return NULL; + fseek(f, 0, SEEK_END); + long n = ftell(f); + fseek(f, 0, SEEK_SET); + uint8_t *b = malloc((size_t)n + 1); + if (b && fread(b, 1, (size_t)n, f) != (size_t)n) free(b), b = NULL; + fclose(f); + if (b) b[n] = 0, *size = (size_t)n; + return b; +} + +static int write_all(const char *path, const void *data, size_t n) { + FILE *f = fopen(path, "wb"); + if (!f) return 0; + int ok = fwrite(data, 1, n, f) == n; + fclose(f); + return ok; +} + +static int modules_part(const char *dir) { + char path[1024]; + size_t list_n = 0; + snprintf(path, sizeof path, "%s/list.txt", dir); + char *list = (char *)read_all(path, &list_n); + CHECK(list, "no %s", path); + int n_task = 0, n_mesh = 0, n_fail = 0, n_pipe = 0, n_pipe_fail = 0; + double pipe_ms = 0, pipe_worst = 0; + uint64_t pipe_worst_words = 0; + const GfxBackend *b = &gfx_vulkan_backend; + pool_t pool = {.magic = NVN_POOL_MAGIC, .flags = 0x22}; + int device = 0; + for (char *name = strtok(list, "\r\n"); name; name = strtok(NULL, "\r\n")) { + size_t in_n = 0, spv_n = 0; + snprintf(path, sizeof path, "%s/%s.in", dir, name); + uint8_t *in = read_all(path, &in_n); + snprintf(path, sizeof path, "%s/%s.spv", dir, name); + uint8_t *spv = read_all(path, &spv_n); + uint32_t h[6]; // magic, version, stage, code length, c1 length, size of the answers (gfx_shader.c InHeader) + if (!in || !spv || in_n < sizeof h) { + printf("probe-meshfb: %s: input or translation missing\n", name), n_fail++; + free(in), free(spv); + continue; + } + memcpy(h, in, sizeof h); + const uint8_t *control = in + sizeof h + h[5], *code = control + 0x880; + int stage = (int)h[2]; // GFX_STAGE_TS / GFX_STAGE_MS are the NVN stage numbers 6 and 7 + char err[200] = ""; + MeshFbInfo m; + uint64_t on = 0, vn = 0; + uint32_t *cs = NULL, *vs = NULL; + if (in_n < sizeof h + h[5] + 0x880 + 0x80 || (stage != 6 && stage != 7)) snprintf(err, sizeof err, "not a task or mesh input"); + else if (meshfb_info(control, code, stage, &m, err, sizeof err)) { + uint32_t top = 0; + const uint32_t *w = (const uint32_t *)spv; + for (size_t i = 5; i < spv_n / 4;) { + uint32_t op = w[i] & 0xffff, len = w[i] >> 16; + if (!len) break; + if (op == 71 && len == 4 && w[i + 2] == 33 && w[i + 3] > top) top = w[i + 3]; + i += len; + } + cs = stage == 6 ? meshfb_task_cs(w, spv_n / 4, &m, top + 1, &on, err, sizeof err) + : meshfb_mesh_cs(w, spv_n / 4, &m, top + 1, m.payload_bytes / 4, &on, err, sizeof err); + if (cs && stage == 7) { + uint8_t type[32] = {0}; + uint32_t mask = 0; + for (uint32_t l = 0; l < m.nlines; l++) + if (m.lines[l] >= 0x80) mask |= 1u << (m.lines[l] - 0x80) / 16; + for (int g = 0; g < 32; g++) type[g] = (uint8_t)(g % 3); // every output type gets made somewhere + vs = meshfb_vs(&m, mask, type, &vn); + if (!vs) snprintf(err, sizeof err, "no vertex module"); + } + } + if (cs && (stage == 6 || vs)) { + snprintf(path, sizeof path, "%s/out/%s.cs.spv", dir, name); + int ok = write_all(path, cs, on * 4); + if (vs) snprintf(path, sizeof path, "%s/out/%s.vs.spv", dir, name), ok = ok && write_all(path, vs, vn * 4); + if (!ok) printf("probe-meshfb: %s: cannot write under %s/out\n", name, dir), n_fail++; + else if (stage == 6) n_task++; + else n_mesh++; + size_t res_n = 0; + snprintf(path, sizeof path, "%s/%s.res.json", dir, name); + char *res = (char *)read_all(path, &res_n); + if (res) { // the pipeline, with the layout the reflection gives + if (!device) { // the first pool brings the device up + uint64_t heap = 0; + CHECK(mem_set_heap_size(16ull << 20, &heap) == 0 && heap, "SetHeapSize 16 MiB"); + pool.mem = pool.game_mem = (uint8_t *)(uintptr_t)heap + 0x100000, pool.size = 0x100000; + CHECK(b->pool_init(&pool), "pool_init of a CPU-visible pool"); + device = 1; + } + uint32_t top = 0; + const uint32_t *w = (const uint32_t *)spv; + for (size_t i = 5; i < spv_n / 4;) { + uint32_t op = w[i] & 0xffff, len = w[i] >> 16; + if (!len) break; + if (op == 71 && len == 4 && w[i + 2] == 33 && w[i + 3] > top) top = w[i + 3]; + i += len; + } + static const int task_slots[1] = {MESHFB_CB_TASK}, mesh_slots[3] = {MESHFB_CB_TASK, MESHFB_CB_MESH, MESHFB_CB_LIST}; + char *cres = stage == 6 ? meshfb_res_json(res, top + 1, task_slots, 1) + : m.has_task ? meshfb_res_json(res, top + 1, mesh_slots, 3) : meshfb_res_json(res, top + 2, mesh_slots + 1, 1); + const GfxShader *sh = cres ? gfx_shader_own(cs, on, cres) : NULL; + const VkPipeInfo *info = NULL; + uint64_t t0 = plat_now_ns(); + VkPipeline p = sh ? vk_pipeline_compute(sh, &info) : VK_NULL_HANDLE; + double ms = (double)(plat_now_ns() - t0) / 1e6; + if (p && info) { + n_pipe++, pipe_ms += ms; + if (ms > pipe_worst) pipe_worst = ms, pipe_worst_words = on; + } else { + n_pipe_fail++; + printf("probe-meshfb: %s (stage %d, %llu words): %s\n", name, stage, (unsigned long long)on, + sh ? "its compute pipeline was not built" : "its reflection could not be made"); + } + fflush(stdout); + free(cres), free(res); + } + } else { + printf("probe-meshfb: %s (stage %d): %s\n", name, stage, err), n_fail++; + } + free(cs), free(vs), free(in), free(spv); + } + free(list); + printf("probe-meshfb: module part: %d task and %d mesh programs converted, %d failed\n", n_task, n_mesh, n_fail); + if (device) { + printf("probe-meshfb: module part: %d compute pipelines built on this device (%.0f ms each, the longest %.0f ms for %llu " + "words), %d not built\n", n_pipe, n_pipe ? pipe_ms / n_pipe : 0.0, pipe_worst, (unsigned long long)pipe_worst_words, + n_pipe_fail); + b->pool_fini(&pool); + b->submit(NULL, 0); // the pool's buffers go when a submission after their release has completed + b->finish(); + vk_pipeline_shutdown(); + vk_exec_draw_shutdown(); + vk_exec_xfer_shutdown(); + vk_backend_shutdown(); + } + return n_fail != 0 || n_pipe_fail != 0; +} + +// ---- draw part + +static uint8_t *put(uint8_t *p, uint32_t op, const void *payload, uint32_t n) { + op_hdr h = {op, (uint32_t)((sizeof h + n + 7) & ~7u)}; + memset(p, 0, h.size); + memcpy(p, &h, sizeof h); + if (n) memcpy(p + sizeof h, payload, n); + return p + h.size; +} + +// An NVN2 control section of a task (6), mesh (7) or fragment (1) stage as vk_exec_draw.c and mesh_fb.c read it, and +// a code block whose header maps the outputs: position and generic vector 0. +static void fake_stage(NVNshaderData *sd, uint32_t stage, int fed_by_task) { + uint8_t *ctl = calloc(1, 0x880), *code = calloc(1, 0x100); + uint32_t magic = 0x98761234u, len = 0x100, st = stage << 16; + uint32_t w58 = 32u | 1u << 24; // 32 threads, a payload of one 256-byte unit + uint32_t w5c = 3u << 24 | (fed_by_task ? 1u << 16 : 0); // triangles + uint32_t w60 = 64u | 32u << 16; // 64 vertices, 32 primitives + memcpy(ctl, &magic, 4), memcpy(ctl + 0x38, &len, 4), memcpy(ctl + 0x54, &st, 4); + if (stage >= 6) memcpy(ctl + 0x58, &w58, 4); + if (stage == 7) memcpy(ctl + 0x5c, &w5c, 4), memcpy(ctl + 0x60, &w60, 4), code[0x35] = 0xf0, code[0x36] = 0x0f; + sd->control = ctl, sd->data = (uint64_t)(uintptr_t)code; +} + +static const char RES_MS[] = "{\"key\":\"probe\",\"stage\":\"ms\",\"notes\":[],\"cbufs\":[{\"binding\":1,\"slot\":3}]," + "\"sbufs\":[],\"textures\":[],\"images\":[]}"; +static const char RES_TS[] = "{\"key\":\"probe\",\"stage\":\"ts\",\"notes\":[],\"cbufs\":[{\"binding\":1,\"slot\":3}]," + "\"sbufs\":[],\"textures\":[],\"images\":[]}"; +static const char RES_FS[] = "{\"key\":\"probe\",\"stage\":\"ps\",\"notes\":[],\"cbufs\":[],\"sbufs\":[],\"textures\":[]," + "\"images\":[]}"; + +// The program of a mesh stage, its task stage if any, and the fragment stage. +static prog_t *make_prog(const GfxShader *ms, const GfxShader *ts, const GfxShader *fs) { + static uint64_t serial = 0x9000; + prog_t *p = calloc(1, sizeof *p); + p->magic = NVN_PROG_MAGIC, p->serial = ++serial; + fake_stage(&p->sd[p->nstage], 7, ts != NULL), vk_exec_probe_shader(p->sd[p->nstage++].control, ms); + if (ts) fake_stage(&p->sd[p->nstage], 6, 0), vk_exec_probe_shader(p->sd[p->nstage++].control, ts); + fake_stage(&p->sd[p->nstage], 1, 0), vk_exec_probe_shader(p->sd[p->nstage++].control, fs); + return p; +} + +typedef struct { prog_t *prog; uint32_t count; int indirect; uint32_t grid[2]; float tint[4]; } MeshDraw; + +// Uniform data in the pool, 0x1000 apart per draw: the mesh stage's block at 0x90000, the task stage's at 0x98000; the +// indirect records at 0xa0000. +static int draw_picture(const GfxBackend *b, pool_t *pool, tex_t *color, const MeshDraw *d, int n, uint32_t *px) { + static _Alignas(8) uint8_t stream[16384]; + uint8_t *q = put(stream, OP_BEGIN, NULL, 0); + p_set_rt rt; + memset(&rt, 0, sizeof rt); + rt.n = 1, rt.color[0] = color; + q = put(q, OP_SET_RT, &rt, sizeof rt); + const p_clear_color c0 = {0, 0xf, {0.2f, 0.4f, 0.6f, 1.0f}}; + q = put(q, OP_CLEAR_COLOR, &c0, sizeof c0); + p_viewports vp; + memset(&vp, 0, sizeof vp); + vp.n = 1, vp.v[0][2] = TEX_W, vp.v[0][3] = TEX_H, vp.v[0][5] = 1.0f; + q = put(q, OP_VIEWPORTS, &vp, sizeof vp); + const p_scissor full = {0, 0, 1 << 20, 1 << 20}; + q = put(q, OP_SCISSOR, &full, sizeof full); + for (int i = 0; i < n; i++) { + uint8_t *mu = pool->mem + 0x90000 + 0x1000 * i, *tu = pool->mem + 0x98000 + 0x1000 * i; + uint32_t *rec = (uint32_t *)(pool->mem + 0xa0000 + 0x100 * i); + uint32_t grid[4] = {d[i].grid[0], d[i].grid[1], 0, 0}, shift[4] = {1, 0, 0, 0}; + memcpy(mu, d[i].tint, 16), memcpy(mu + 16, grid, 16), memcpy(tu, shift, 16); + rec[0] = d[i].count, rec[1] = 0, rec[2] = 0; + const p_bind_program bp = {d[i].prog, 0xc2}; + q = put(q, OP_BIND_PROGRAM, &bp, sizeof bp); + const p_bind_buf bm = {7, 0, (uint64_t)(uintptr_t)mu, 256}, bt = {6, 0, (uint64_t)(uintptr_t)tu, 256}; + q = put(q, OP_BIND_UBO, &bm, sizeof bm); + q = put(q, OP_BIND_UBO, &bt, sizeof bt); + if (d[i].indirect) { + const p_draw_mesh_ind di = {(uint64_t)(uintptr_t)rec, 0, 0, 0, 0}; + q = put(q, OP_DRAW_MESH_IND, &di, sizeof di); + } else { + const p_draw_mesh dm = {0, d[i].count, 0}; + q = put(q, OP_DRAW_MESH, &dm, sizeof dm); + } + } + q = put(q, OP_END, NULL, 0); + CHECK(q <= stream + sizeof stream, "op stream overflow"); + uint64_t handle = (uint64_t)(uintptr_t)stream; + uint64_t f = b->submit(&handle, 1); + CHECK(b->fence_wait(f, 60000000000ull), "fence_wait of a mesh draw submission timed out"); + NVNcopyRegion r = {0, 0, 0, TEX_W, TEX_H, 1}; + b->read_texels(color, NULL, &r, (uint8_t *)px, TEX_W * 4, TEX_W * TEX_H * 4); + return 0; +} + +static int drawn_texels(const uint32_t *px) { + int n = 0; + for (int i = 0; i < TEX_W * TEX_H; i++) n += px[i] != px[0]; // texel 0 is in the margin no cell covers + return n; +} + +// Texels of a and b that differ, and the largest difference of a channel. +static int differing(const uint32_t *a, const uint32_t *b, int *worst) { + int n = 0; + *worst = 0; + for (int i = 0; i < TEX_W * TEX_H; i++) { + if (a[i] == b[i]) continue; + n++; + for (int c = 0; c < 4; c++) { + int d = (int)(a[i] >> 8 * c & 0xff) - (int)(b[i] >> 8 * c & 0xff); + if (d < 0) d = -d; + if (d > *worst) *worst = d; + } + } + return n; +} + +static int draw_part(const GfxBackend *b, uint8_t *guest) { + pool_t pool = {.magic = NVN_POOL_MAGIC, .flags = 0x22, .mem = guest + 0x100000, .size = 0xC0000, + .game_mem = guest + 0x100000}; + CHECK(b->pool_init(&pool), "pool_init of a CPU-visible pool"); + VkCtx *c = vk_ctx(); + CHECK(c, "no Vulkan device"); + static tex_t color; + memset(&color, 0, sizeof color); + color.magic = NVN_TEX_MAGIC; + color.d.target = NVN_TEXTURE_TARGET_2D, color.d.format = NVN_FORMAT_RGBA8, color.d.width = TEX_W, color.d.height = TEX_H; + color.d.depth = 1, color.d.levels = 1; + nvn_tex_layout(&color.d, &color.L); + CHECK(color.L.size <= 0x80000, "the target exceeds its pool slot"); + color.pool = &pool, color.pool_flags = (uint16_t)pool.flags, color.off = 0, color.mem = pool.mem, color.serial = 4000; + + const GfxShader *fs = gfx_shader_own(meshfb_probe_f_spv, sizeof meshfb_probe_f_spv / 4, RES_FS); + const GfxShader *ms = gfx_shader_own(meshfb_probe_m_spv, sizeof meshfb_probe_m_spv / 4, RES_MS); + const GfxShader *ts = gfx_shader_own(meshfb_probe_t_spv, sizeof meshfb_probe_t_spv / 4, RES_TS); + const GfxShader *tm = gfx_shader_own(meshfb_probe_tm_spv, sizeof meshfb_probe_tm_spv / 4, RES_MS); + CHECK(fs && ms && ts && tm, "a probe shader is malformed"); + prog_t *plain = make_prog(ms, NULL, fs), *tasked = make_prog(tm, ts, fs); + + // 24 cells of a 6 x 4 grid by 24 mesh workgroups; 40 cells of an 8 x 5 grid by 10 task workgroups that start 0 to + // 4 mesh workgroups each + const MeshDraw cases[][2] = { + {{plain, 24, 0, {6, 4}, {1, 1, 1, 1}}}, + {{plain, 24, 1, {6, 4}, {1, 1, 1, 1}}}, + {{tasked, 10, 0, {8, 5}, {1, 1, 1, 1}}}, + {{tasked, 10, 1, {8, 5}, {1, 1, 1, 1}}}, + {{plain, 24, 1, {6, 4}, {1, 1, 1, 1}}, {tasked, 10, 1, {8, 5}, {0.5f, 1, 0.25f, 1}}}, + {{tasked, 10, 0, {8, 5}, {1, 1, 1, 1}}, {plain, 24, 0, {6, 4}, {1, 0.5f, 0.5f, 1}}}, + }; + static const char *const names[] = {"mesh, direct", "mesh, indirect", "task + mesh, direct", "task + mesh, indirect", + "mesh then task + mesh in one pass, indirect", "task + mesh then mesh in one pass, direct"}; + enum { CASES = sizeof cases / sizeof cases[0] }; + static uint32_t px[2][CASES][TEX_W * TEX_H]; + int native = c->has_mesh_shader, was = c->mesh_fallback; + for (int way = native ? 0 : 1; way < 2; way++) { + c->mesh_fallback = way; + for (int k = 0; k < CASES; k++) + if (draw_picture(b, &pool, &color, cases[k], cases[k][1].prog ? 2 : 1, px[way][k])) return 1; + } + c->mesh_fallback = was; + for (int k = 0; k < CASES; k++) { + int fb = drawn_texels(px[1][k]); + CHECK(fb > TEX_W * TEX_H / 20, "%s: the fallback drew %d texels", names[k], fb); + if (!native) { + printf("probe-meshfb: %s: the fallback drew %d of %d texels (no mesh shaders here: not compared)\n", names[k], fb, + TEX_W * TEX_H); + continue; + } + int worst, diff = differing(px[0][k], px[1][k], &worst), nat = drawn_texels(px[0][k]); + CHECK(nat > TEX_W * TEX_H / 20, "%s: the mesh pipeline drew %d texels", names[k], nat); + CHECK(worst <= 1, "%s: %d texels differ between the mesh pipeline and the fallback, by up to %d of 255 (the mesh " + "pipeline drew %d texels, the fallback %d)", names[k], diff, worst, nat, fb); + printf("probe-meshfb: %s: %d texels drawn by the mesh pipeline, %d by the fallback, %d differ (by %d of 255 at most)\n", + names[k], nat, fb, diff, worst); + } + int w01, w23; + CHECK(!differing(px[1][0], px[1][1], &w01) && !differing(px[1][2], px[1][3], &w23), + "the fallback's direct and indirect pictures differ"); + CHECK(!vk_meshfb_short(), "%llu of the draws above did not fit the fallback's buffers", (unsigned long long)vk_meshfb_short()); + // More workgroups than the mesh buffer has slots for: what fits is drawn, the rest is dropped and counted. The + // cells past the grid are outside the picture, so it is the 24-cell picture again. + uint32_t slots = (uint32_t)(vk_meshfb_mesh_words() / 540), over = slots + 1000; + const MeshDraw big = {plain, over, 1, {6, 4}, {1, 1, 1, 1}}; + static uint32_t bpx[TEX_W * TEX_H]; + c->mesh_fallback = 1; + int bad = draw_picture(b, &pool, &color, &big, 1, bpx); + c->mesh_fallback = was; + if (bad) return 1; + b->finish(); + int wb, db = differing(bpx, px[1][0], &wb); + CHECK(!db, "a draw of %u workgroups, %u more than the mesh buffer holds: %d texels differ from the 24-cell picture", over, + over - slots, db); + CHECK(vk_meshfb_short() == 1, "a draw of %u workgroups with room for %u was counted as %llu draws that did not fit", over, slots, + (unsigned long long)vk_meshfb_short()); + printf("probe-meshfb: a draw of %u workgroups with room for %u: the picture of the first 24 is whole, the draw was counted\n", + over, slots); + // As the game runs it: pipelines built in the background, and a program's modules made on the fallback's own + // thread. New shader objects of the same programs are drawn until the picture is whole: the draws before that are + // skipped, and none may leave a partial picture behind for good. + const GfxShader *fs2 = gfx_shader_own(meshfb_probe_f_spv, sizeof meshfb_probe_f_spv / 4, RES_FS); + const GfxShader *ms2 = gfx_shader_own(meshfb_probe_m_spv, sizeof meshfb_probe_m_spv / 4, RES_MS); + const GfxShader *ts2 = gfx_shader_own(meshfb_probe_t_spv, sizeof meshfb_probe_t_spv / 4, RES_TS); + const GfxShader *tm2 = gfx_shader_own(meshfb_probe_tm_spv, sizeof meshfb_probe_tm_spv / 4, RES_MS); + CHECK(fs2 && ms2 && ts2 && tm2, "a probe shader is malformed"); + prog_t *plain2 = make_prog(ms2, NULL, fs2), *tasked2 = make_prog(tm2, ts2, fs2); + const MeshDraw again[2] = {{tasked2, 10, 0, {8, 5}, {1, 1, 1, 1}}, {plain2, 24, 0, {6, 4}, {1, 0.5f, 0.5f, 1}}}; + vk_pipeline_async_enable(); + c->mesh_fallback = 1; + int tries = 0, left = -1, wa; + while (left && tries < 500) { + if (tries++) plat_sleep_ms(10); + if (draw_picture(b, &pool, &color, again, 2, bpx)) return 1; + left = differing(bpx, px[1][5], &wa); + } + c->mesh_fallback = was; + CHECK(!left, "with background builds the picture still differs in %d texels after %d submissions", left, tries); + printf("probe-meshfb: with background builds the picture was whole at submission %d (the earlier ones skipped their draws)\n", + tries); + b->texture_release(&color); + b->pool_fini(&pool); + return 0; +} + +int probe_meshfb(void) { + const char *dir = cfg_get("NX2NAVI_PROBE_MESHFB_DIR"); + if (dir && *dir) return modules_part(dir); + vk_hdr_off(); + const GfxBackend *b = &gfx_vulkan_backend; + uint64_t heap = 0; + CHECK(mem_set_heap_size(16ull << 20, &heap) == 0 && heap, "SetHeapSize 16 MiB"); + int bad = draw_part(b, (uint8_t *)(uintptr_t)heap); + b->finish(); + vk_pipeline_shutdown(); + vk_exec_draw_shutdown(); + vk_exec_xfer_shutdown(); + vk_backend_shutdown(); + long errors = vk_validation_errors_total(); + if (bad || errors) { + printf("probe-meshfb: FAILED (%s, %ld validation errors)\n", bad ? "a draw check failed" : "checks passed", errors); + return 1; + } + printf("probe-meshfb: passed (0 validation errors)\n"); + return 0; +} diff --git a/runtime/shaders/meshfb_probe_f.frag b/runtime/shaders/meshfb_probe_f.frag new file mode 100644 index 00000000..cbedb894 --- /dev/null +++ b/runtime/shaders/meshfb_probe_f.frag @@ -0,0 +1,6 @@ +#version 450 +// --probe-meshfb (mesh_fb_probe.c): the fragment stage of the probe's mesh programs: the interpolated generic vector 0. +layout(location = 0) in vec4 i_color; +layout(location = 0) out vec4 o_color; + +void main() { o_color = i_color; } diff --git a/runtime/shaders/meshfb_probe_m.mesh b/runtime/shaders/meshfb_probe_m.mesh new file mode 100644 index 00000000..742b023f --- /dev/null +++ b/runtime/shaders/meshfb_probe_m.mesh @@ -0,0 +1,75 @@ +#version 460 +#extension GL_EXT_mesh_shader : require +// build_shaders: keep functions +// +// --probe-meshfb (mesh_fb_probe.c): a mesh program without a task stage, in the shape the translator gives one +// (tools/xlate/MeshSpirv.cs): the program itself is a function that writes its stage buffer into shared memory, and +// the entry function calls it once and hands the buffer to the mesh pipeline. mesh_fb.c makes its compute module from +// this one, as it does from a translated program. The layout is AmpereLowering.MeshInfo's for 64 vertices, 32 +// triangles and 8 output components (position, generic vector 0): vertex v's component l is word +// ((v >> 5) * 8 + l) * 32 + (v & 31), the primitive count word 512, the index bytes from byte 2052. +// +// Workgroup g draws cell g of a grid: nine vertices (28 to 36, across the two vertex groups) and up to eight +// triangles, none in every fifth cell. +layout(local_size_x = 32) in; +layout(triangles, max_vertices = 64, max_primitives = 32) out; +layout(location = 0) out vec4 o_color[]; +layout(set = 0, binding = 1) uniform Params { vec4 tint; uvec4 grid; } u; // grid: columns, rows +shared uint mem[540]; + +#define VWORD(v, l) ((((v) >> 5) * 8u + (l)) * 32u + ((v) & 31u)) +#define COUNT_WORD 512u +#define INDEX_BYTE 2052u + +void program() { + uint tid = gl_LocalInvocationIndex, cell = gl_WorkGroupID.x; + if (tid < 9u) { + uint v = tid + 28u, ix = tid % 3u, iy = tid / 3u; + vec2 p = -1.0 + (vec2(cell % u.grid.x, cell / u.grid.x) + 0.1 + 0.4 * vec2(ix, iy)) * (2.0 / vec2(u.grid.xy)); + vec4 c = vec4(float(ix) * 0.5, float(iy) * 0.5, float(cell % 4u) / 3.0, 1.0) * u.tint; + mem[VWORD(v, 0u)] = floatBitsToUint(p.x); + mem[VWORD(v, 1u)] = floatBitsToUint(p.y); + mem[VWORD(v, 2u)] = floatBitsToUint(0.5); + mem[VWORD(v, 3u)] = floatBitsToUint(1.0); + mem[VWORD(v, 4u)] = floatBitsToUint(c.x); + mem[VWORD(v, 5u)] = floatBitsToUint(c.y); + mem[VWORD(v, 6u)] = floatBitsToUint(c.z); + mem[VWORD(v, 7u)] = floatBitsToUint(c.w); + } + if (tid == 0u) { // one thread writes the index bytes: they share words + uint count = cell % 5u == 0u ? 0u : 2u + cell % 7u; + for (uint t = 0u; t < count; t++) { + uint q = t / 2u, a = (q / 2u) * 3u + q % 2u + 28u; + uvec3 tri = (t & 1u) == 0u ? uvec3(a, a + 1u, a + 3u) : uvec3(a + 1u, a + 4u, a + 3u); + for (uint k = 0u; k < 3u; k++) { + uint b = INDEX_BYTE + 3u * t + k, sh = (b & 3u) * 8u; + mem[b >> 2] = (mem[b >> 2] & ~(0xffu << sh)) | tri[k] << sh; + } + } + mem[COUNT_WORD] = count; + } +} + +void main() { + uint tid = gl_LocalInvocationIndex; + if (tid < 1u) mem[COUNT_WORD] = 0u; + barrier(); + program(); + barrier(); + uint count = min(mem[COUNT_WORD], 32u); + SetMeshOutputsEXT(64u, count); + for (uint v = tid; v < 64u; v += 32u) { + gl_MeshVerticesEXT[v].gl_Position = vec4(uintBitsToFloat(mem[VWORD(v, 0u)]), uintBitsToFloat(mem[VWORD(v, 1u)]), + uintBitsToFloat(mem[VWORD(v, 2u)]), uintBitsToFloat(mem[VWORD(v, 3u)])); + o_color[v] = vec4(uintBitsToFloat(mem[VWORD(v, 4u)]), uintBitsToFloat(mem[VWORD(v, 5u)]), + uintBitsToFloat(mem[VWORD(v, 6u)]), uintBitsToFloat(mem[VWORD(v, 7u)])); + } + for (uint p = tid; p < count; p += 32u) { + uvec3 tri; + for (uint k = 0u; k < 3u; k++) { + uint b = INDEX_BYTE + 3u * p + k; + tri[k] = min(mem[b >> 2] >> ((b & 3u) * 8u) & 0xffu, 63u); + } + gl_PrimitiveTriangleIndicesEXT[p] = tri; + } +} diff --git a/runtime/shaders/meshfb_probe_t.task b/runtime/shaders/meshfb_probe_t.task new file mode 100644 index 00000000..634a0bfa --- /dev/null +++ b/runtime/shaders/meshfb_probe_t.task @@ -0,0 +1,33 @@ +#version 460 +#extension GL_EXT_mesh_shader : require +// build_shaders: keep functions +// +// --probe-meshfb (mesh_fb_probe.c): a task program in the shape the translator gives one (tools/xlate/MeshSpirv.cs): +// the program writes its payload into shared memory, and the entry function copies that out and starts the mesh +// workgroups the program counted into word 1. The payload is 288 bytes: the header, then the program's data from +// word 4. +// +// Task workgroup t starts 0 to 4 mesh workgroups (the uniform block shifts which tasks start how many) and tells each +// the grid cell it draws; the last payload word carries a mark the mesh program checks. +layout(local_size_x = 32) in; +layout(set = 0, binding = 1) uniform Params { uvec4 shift; } u; +shared uint mem[72]; +taskPayloadSharedEXT uint payload[72]; + +void program() { + uint tid = gl_LocalInvocationIndex, t = gl_WorkGroupID.x; + uint count = (t * 3u + u.shift.x) % 5u; + if (tid == 0u) mem[1] = count; + if (tid < count) mem[4u + tid] = t * 4u + tid; + if (tid == 31u) mem[71] = 0x600df00du + t; +} + +void main() { + uint tid = gl_LocalInvocationIndex; + if (tid < 1u) mem[1] = 0u; + barrier(); + program(); + barrier(); + for (uint i = tid; i < 72u; i += 32u) payload[i] = mem[i]; + EmitMeshTasksEXT(min(mem[1], 32u), 1u, 1u); +} diff --git a/runtime/shaders/meshfb_probe_tm.mesh b/runtime/shaders/meshfb_probe_tm.mesh new file mode 100644 index 00000000..df1feb4f --- /dev/null +++ b/runtime/shaders/meshfb_probe_tm.mesh @@ -0,0 +1,74 @@ +#version 460 +#extension GL_EXT_mesh_shader : require +// build_shaders: keep functions +// +// --probe-meshfb (mesh_fb_probe.c): the mesh program that meshfb_probe_t.task feeds, in the translator's shape +// (meshfb_probe_m.mesh says which). Its shared memory holds the task payload behind the stage buffer, from word 540. +// The workgroup id is its index among its task's mesh workgroups: payload word 4 + id names the cell it draws, and +// the mark in the last payload word names its task, which the colour shows when either is wrong. +layout(local_size_x = 32) in; +layout(triangles, max_vertices = 64, max_primitives = 32) out; +layout(location = 0) out vec4 o_color[]; +layout(set = 0, binding = 1) uniform Params { vec4 tint; uvec4 grid; } u; // grid: columns, rows +shared uint mem[612]; +taskPayloadSharedEXT uint payload[72]; + +#define VWORD(v, l) ((((v) >> 5) * 8u + (l)) * 32u + ((v) & 31u)) +#define COUNT_WORD 512u +#define INDEX_BYTE 2052u +#define PAYLOAD 540u + +void program() { + uint tid = gl_LocalInvocationIndex, cell = mem[PAYLOAD + 4u + gl_WorkGroupID.x]; + bool mine = mem[PAYLOAD + 71u] == 0x600df00du + cell / 4u && cell % 4u == gl_WorkGroupID.x; + if (tid < 9u) { + uint v = tid + 28u, ix = tid % 3u, iy = tid / 3u; + vec2 p = -1.0 + (vec2(cell % u.grid.x, cell / u.grid.x) + 0.1 + 0.4 * vec2(ix, iy)) * (2.0 / vec2(u.grid.xy)); + vec4 c = mine ? vec4(float(ix) * 0.5, float(iy) * 0.5, float(cell % 4u) / 3.0, 1.0) * u.tint : vec4(1.0, 0.0, 1.0, 1.0); + mem[VWORD(v, 0u)] = floatBitsToUint(p.x); + mem[VWORD(v, 1u)] = floatBitsToUint(p.y); + mem[VWORD(v, 2u)] = floatBitsToUint(0.5); + mem[VWORD(v, 3u)] = floatBitsToUint(1.0); + mem[VWORD(v, 4u)] = floatBitsToUint(c.x); + mem[VWORD(v, 5u)] = floatBitsToUint(c.y); + mem[VWORD(v, 6u)] = floatBitsToUint(c.z); + mem[VWORD(v, 7u)] = floatBitsToUint(c.w); + } + if (tid == 0u) { // one thread writes the index bytes: they share words + uint count = 2u + cell % 7u; + for (uint t = 0u; t < count; t++) { + uint q = t / 2u, a = (q / 2u) * 3u + q % 2u + 28u; + uvec3 tri = (t & 1u) == 0u ? uvec3(a, a + 1u, a + 3u) : uvec3(a + 1u, a + 4u, a + 3u); + for (uint k = 0u; k < 3u; k++) { + uint b = INDEX_BYTE + 3u * t + k, sh = (b & 3u) * 8u; + mem[b >> 2] = (mem[b >> 2] & ~(0xffu << sh)) | tri[k] << sh; + } + } + mem[COUNT_WORD] = count; + } +} + +void main() { + uint tid = gl_LocalInvocationIndex; + if (tid < 1u) mem[COUNT_WORD] = 0u; + for (uint i = tid; i < 72u; i += 32u) mem[PAYLOAD + i] = payload[i]; + barrier(); + program(); + barrier(); + uint count = min(mem[COUNT_WORD], 32u); + SetMeshOutputsEXT(64u, count); + for (uint v = tid; v < 64u; v += 32u) { + gl_MeshVerticesEXT[v].gl_Position = vec4(uintBitsToFloat(mem[VWORD(v, 0u)]), uintBitsToFloat(mem[VWORD(v, 1u)]), + uintBitsToFloat(mem[VWORD(v, 2u)]), uintBitsToFloat(mem[VWORD(v, 3u)])); + o_color[v] = vec4(uintBitsToFloat(mem[VWORD(v, 4u)]), uintBitsToFloat(mem[VWORD(v, 5u)]), + uintBitsToFloat(mem[VWORD(v, 6u)]), uintBitsToFloat(mem[VWORD(v, 7u)])); + } + for (uint p = tid; p < count; p += 32u) { + uvec3 tri; + for (uint k = 0u; k < 3u; k++) { + uint b = INDEX_BYTE + 3u * p + k; + tri[k] = min(mem[b >> 2] >> ((b & 3u) * 8u) & 0xffu, 63u); + } + gl_PrimitiveTriangleIndicesEXT[p] = tri; + } +} diff --git a/runtime/shaders/meshfb_scan.comp b/runtime/shaders/meshfb_scan.comp new file mode 100644 index 00000000..eac0e3cd --- /dev/null +++ b/runtime/shaders/meshfb_scan.comp @@ -0,0 +1,47 @@ +#version 450 +// The mesh fallback's pass between a draw's task dispatch and its mesh dispatch (vk_mesh_fb.c vk_meshfb_scan, +// mesh_fb.c): every task workgroup wrote how many mesh workgroups it starts into its payload (word 1). This lists them +// in task order, so that mesh workgroup g finds its task and its index among that task's in list entry g, and writes +// the draw's record: how many mesh workgroups to dispatch and how many instances to draw. +// +// One thread walks the tasks: the order of the list is the order the primitives are drawn in, which has to be the +// same in every frame and in the depth and colour passes of one frame. +layout(local_size_x = 1) in; + +layout(push_constant) uniform Params { + uint rec; // the record's first word in the record buffer + uint task_words; // words of a task workgroup's payload in the task buffer + uint fan; // mesh workgroups a task workgroup starts at most + uint slot_words; // words of a mesh workgroup's output in the mesh buffer + uint verts; // vertices drawn per mesh workgroup + uint max_groups; // workgroups one dispatch may have (maxComputeWorkGroupCount) +} pc; + +// The record: words 0-2 the task dispatch (x written before this pass), 4-6 the mesh dispatch, 8-11 the draw (vertices, +// instances, first vertex, first instance), 12 the mesh workgroups the tasks asked for, 13 the task workgroups. +layout(set = 0, binding = 0) buffer Rec { uint w[]; } rec; +layout(set = 0, binding = 1) readonly buffer Task { uint w[]; } task; +layout(set = 0, binding = 2) writeonly buffer List { uint e[]; } list; +layout(set = 0, binding = 3) readonly buffer Mesh { uint w[]; } mesh; // read for its size alone + +void main() { + uint asked = rec.w[pc.rec]; + uint tasks = min(min(asked, uint(task.w.length()) / pc.task_words), 1u << 24); + uint room = min(min(uint(mesh.w.length()) / pc.slot_words, uint(list.e.length())), pc.max_groups); + uint fan = min(pc.fan, 255u); + uint m = 0u, want = 0u; + for (uint t = 0u; t < tasks; t++) { + uint c = min(task.w[t * pc.task_words + 1u], fan); + want += c; + for (uint j = 0u; j < c && m < room; j++) list.e[m++] = t | j << 24; + } + rec.w[pc.rec + 4u] = m; + rec.w[pc.rec + 5u] = 1u; + rec.w[pc.rec + 6u] = 1u; + rec.w[pc.rec + 8u] = pc.verts; + rec.w[pc.rec + 9u] = m; + rec.w[pc.rec + 10u] = 0u; + rec.w[pc.rec + 11u] = 0u; + rec.w[pc.rec + 12u] = want; + rec.w[pc.rec + 13u] = asked; +} diff --git a/runtime/vk.h b/runtime/vk.h index aad3ca73..7d8a705e 100644 --- a/runtime/vk.h +++ b/runtime/vk.h @@ -41,6 +41,8 @@ typedef struct VkCtx { has_maximal_reconvergence, // VK_KHR_shader_maximal_reconvergence (vk_pipeline.c spirv_reconverge) has_pipeline_binary, // VK_KHR_pipeline_binary with VK_KHR_maintenance5 (vk_pipeline.c pipe_create) has_mesh_queries; // meshShaderQueries, kept while NX2NAVI_GPU_PASSES or NX2NAVI_DUMP is set + int mesh_fallback; // task and mesh programs run as compute passes and are drawn from buffers + // (mesh_fallback: auto without has_mesh_shader; vk_exec_draw.c "mesh fallback") // Feedback loops (maxwell.md 1.9, vk_exec_draw.c feedback_prepare): a draw samples a target of its open pass that // an earlier draw of the pass wrote. NX2NAVI_FEEDBACK=0: VK_FEEDBACK_OFF (sampled in GENERAL without a barrier, the // old behaviour); VK_FEEDBACK_LAYOUT with VK_EXT_attachment_feedback_loop_layout and dynamicRenderingLocalRead diff --git a/runtime/vk_device.c b/runtime/vk_device.c index 1d0311cc..97efa84e 100644 --- a/runtime/vk_device.c +++ b/runtime/vk_device.c @@ -469,8 +469,17 @@ static int create_device(void) { g_vk.has_mesh_queries = want && c->mesh.meshShaderQueries && h->pipelineStatisticsQuery && c->f12.hostQueryReset; c->mesh.meshShaderQueries = g_vk.has_mesh_queries ? VK_TRUE : VK_FALSE; if (!c->mesh.meshShader || !c->mesh.taskShader) g_vk.has_mesh_shader = 0; - rt_log("vk: mesh shaders: %s", g_vk.has_mesh_shader ? "VK_EXT_mesh_shader" : "not supported (task/mesh draws are skipped)"); + rt_log("vk: mesh shaders: %s", g_vk.has_mesh_shader ? "VK_EXT_mesh_shader" : "the extension is there, its task or mesh feature is not"); } // vkGetDeviceFaultInfoEXT needs the feature, not just the extension + { // mesh_fallback: auto (on a device without mesh shaders), 1 (on every device, for testing) or 0 + const char *mf = cfg_value("NX2NAVI_MESH_FALLBACK"); + g_vk.mesh_fallback = !strcmp(mf, "1") || (strcmp(mf, "0") && !g_vk.has_mesh_shader); + if (g_vk.mesh_fallback) + rt_log("vk: mesh fallback: task and mesh programs run as compute passes and are drawn from their buffers (%s)", + g_vk.has_mesh_shader ? "mesh_fallback = 1" : "this device has no mesh shaders"); + else if (!g_vk.has_mesh_shader) + rt_log("vk: mesh shaders: not supported and mesh_fallback = 0 (task/mesh draws are skipped)"); + } if (g_vk.has_maximal_reconvergence && !c->mrc.shaderMaximalReconvergence) g_vk.has_maximal_reconvergence = 0; if (g_vk.has_pipeline_binary && !(c->pbin.pipelineBinaries && c->m5.maintenance5)) g_vk.has_pipeline_binary = 0; g_vk.has_buffer_device_address = c->f12.bufferDeviceAddress; diff --git a/runtime/vk_exec_draw.c b/runtime/vk_exec_draw.c index 584c1d54..102e4b9d 100644 --- a/runtime/vk_exec_draw.c +++ b/runtime/vk_exec_draw.c @@ -29,6 +29,8 @@ #endif #include "vk_exec.h" #include "vk_gpa.h" +#include "mesh_fb.h" +#include "vk_mesh_fb.h" #include "platform/platform.h" #include "vk_dlaa.h" @@ -446,7 +448,12 @@ static void twins_release(VkCtx *c); // Releases the stand-ins and the depth twins. The device must be idle; the game path leaves them to process exit like // the pipelines. +static void mesh_ring_release(VkCtx *c); +static void pool_marks_release(VkCtx *c); + void vk_exec_draw_shutdown(void) { + vk_meshfb_shutdown(); + if (vk_ctx()) mesh_ring_release(vk_ctx()), pool_marks_release(vk_ctx()); // made without the stand-ins below too if (!Z.ready) return; VkCtx *c = vk_ctx(); twins_release(c); @@ -948,6 +955,13 @@ static void answers_for(VkExec *ex, int st, const uint8_t *ctl, uint32_t mask, G // and *pending = 1 says the wait ended first. static int sf_mvprev; // shader_for asks for the vertex shader's last-frame variant (per-object motion) +// Probes only (mesh_fb_probe.c): the shader of the program stage whose control section is ctl, in place of a +// translation. The probes build their programs from modules made here, which no translator has an input for. +static struct { const uint8_t *ctl; const GfxShader *s; } probe_sh[16]; +static int n_probe_sh; +void vk_exec_probe_shader(const uint8_t *ctl, const GfxShader *s) { + if (n_probe_sh < 16) probe_sh[n_probe_sh].ctl = ctl, probe_sh[n_probe_sh].s = s, n_probe_sh++; +} static const GfxShader *shader_for(VkExec *ex, int st, int *present, int *pending, uint32_t await_ms) { const uint8_t *ctl, *code; uint32_t len; @@ -955,6 +969,8 @@ static const GfxShader *shader_for(VkExec *ex, int st, int *present, int *pendin if (pending) *pending = 0; if (!ex->prog || !find_stage(ex->prog, st, &ctl, &code, &len)) return NULL; *present = 1; + for (int i = 0; i < n_probe_sh; i++) + if (probe_sh[i].ctl == ctl) return probe_sh[i].s; GfxShaderAnswers a; uint64_t serial = ex->prog->serial; answers_for(ex, st, ctl, slot_mask_of(serial, st), &a); @@ -994,7 +1010,12 @@ static const GfxShader *shader_for(VkExec *ex, int st, int *present, int *pendin // ---- descriptors +// The mesh fallback (below, "mesh fallback") runs a program's task and mesh stages as compute modules: while it binds +// one's resources, a compute module's bindings are those of this NVN stage (6 task, 7 mesh), not of the compute stage. +static int mfb_stage; + static int nvn_stage_of(int gfx_stage) { + if (gfx_stage == GFX_STAGE_CS && mfb_stage) return mfb_stage; return gfx_stage == GFX_STAGE_VS ? NVN_STAGE_VS : gfx_stage == GFX_STAGE_FS ? NVN_STAGE_FS : gfx_stage == GFX_STAGE_TS ? NVN_STAGE_TASK : gfx_stage == GFX_STAGE_MS ? NVN_STAGE_MESH : NVN_STAGE_CS; } @@ -1373,6 +1394,8 @@ static VkDescriptorBufferInfo ssbo_info(VkExec *ex, int st, const GfxShaderRes * VkDescriptorBufferInfo t = vk_gpa_info(); return t.buffer ? t : (VkDescriptorBufferInfo){Z.zbuf, 0, VK_WHOLE_SIZE}; } + // The mesh fallback's own buffers (mesh_fb.c): no guest memory behind them + if (r->cbuf_slot >= MESHFB_CB_TASK && r->cbuf_slot <= MESHFB_CB_LIST) return vk_meshfb_buffer(r->cbuf_slot); if (!__atomic_load_n(&LIM.ubo_max, __ATOMIC_ACQUIRE)) limits_init(); uint64_t max = LIM.sbo_max, align = LIM.sbo_align; VkDescriptorBufferInfo bi; @@ -1908,6 +1931,15 @@ static void pool_marks_make(void) { __atomic_store_n(&PL.marks_ready, 1, __ATOMIC_RELEASE); // the recording thread reads the rest after this } +static void pool_marks_release(VkCtx *c) { // vk_exec_draw_shutdown: a probe ends with the device idle + if (!PL.marks) return; + __atomic_store_n(&PL.marks_ready, 0, __ATOMIC_RELEASE); + vmaDestroyBuffer(c->vma, PL.marks, PL.marks_alloc); + for (int i = 0; i < MARKS_RB; i++) + if (PL.rb[i].buf) vmaDestroyBuffer(c->vma, PL.rb[i].buf, PL.rb[i].alloc), PL.rb[i].buf = VK_NULL_HANDLE; + PL.marks = VK_NULL_HANDLE; +} + void vk_exec_pool_marks(VkExec *ex) { if (!__atomic_load_n(&PL.marks_ready, __ATOMIC_ACQUIRE)) return; VkCtx *c = vk_ctx(); @@ -3121,7 +3153,7 @@ static int pb_key(const VkExec *ex, const VkPipeBinding *b, uint32_t k, uint64_t return 1; // slot 1: the program's constants (pb_globals); others: the zero buffer } if (b->kind == GFX_RES_SSBO) { - if ((r->flags & 1) || r->ptr || r->cbuf_slot == VK_GPA_CB_SLOT) return 0; + if ((r->flags & 1) || r->ptr || r->cbuf_slot >= VK_GPA_CB_SLOT) return 0; // the table, the mesh fallback's buffers int ubo, slot = ssbo_slot(r, st, &ubo); if (slot < 0 || slot >= 16) return 0; const p_bind_buf *p = ubo ? &ex->ubo[st][slot] : &ex->ssbo[st][slot]; @@ -3860,8 +3892,13 @@ static void prog_lead(const prog_t *p, int mesh) { } // mesh: a mesh draw (op_draw_mesh): the program's mesh and task stages stand in for the vertex stage, and nothing -// vertex-related (vertex buffers, motion vector bookkeeping of vertex shaders) is bound or noted. +// vertex-related (vertex buffers, motion vector bookkeeping of vertex shaders) is bound or noted. 2: a mesh draw of +// the mesh fallback (mfb_draw): the vertex stage is mfb_cur_vs, in an ordinary pipeline without vertex input. +static const GfxShader *mfb_cur_vs; + static int prepare_draw_ex(VkExec *ex, int prim, int32_t count, int restart, int mesh) { + int fb = mesh == 2; // from here on mesh says what the draw is, and !fb that its pipeline is a mesh pipeline + mesh = mesh != 0; if (ex->render_disabled) return vk_cond_draws_skip += ex->has_render_cond, draw_skipped(ex, mesh, SK_DISABLED); cb_res(ex); dummies_init(); @@ -3892,8 +3929,9 @@ static int prepare_draw_ex(VkExec *ex, int prim, int32_t count, int restart, int } } else { SM.valid = 0; - vs = shader_for(ex, vst, &has_vs, &vs_wait, 0); - if (mesh) ts = shader_for(ex, NVN_STAGE_TASK, &has_ts, &ts_wait, 0); + if (fb) vs = mfb_cur_vs, has_vs = 1; + else vs = shader_for(ex, vst, &has_vs, &vs_wait, 0); + if (mesh && !fb) ts = shader_for(ex, NVN_STAGE_TASK, &has_ts, &ts_wait, 0); fs = shader_for(ex, NVN_STAGE_FS, &has_fs, &fs_wait, 0); if (memo && vs && !vs_wait && !fs_wait && (!has_fs || fs)) memo_key(ex, gen, &SM.k), SM.vs = vs, SM.fs = fs, SM.has_fs = has_fs, SM.valid = 1; @@ -3927,6 +3965,7 @@ static int prepare_draw_ex(VkExec *ex, int prim, int32_t count, int restart, int VkPipeState s = ex->pipe; s.vs = vs, s.fs = fs, s.ts = ts; if (mesh) foliage_shadow_cull(ex, &s); + if (fb) memset(s.vattr, 0, sizeof s.vattr), memset(s.vstream, 0, sizeof s.vstream); // whatever the last vertex draw left s.feedback = (uint8_t)((fbl & 0xff ? 1 : 0) | (fbl & 0x100 ? 2 : 0)); s.prim = (uint8_t)prim; s.depth_minus_w = (uint8_t)(nvn_depth_minus_w() != 0); @@ -3946,7 +3985,7 @@ static int prepare_draw_ex(VkExec *ex, int prim, int32_t count, int restart, int int fullscreen = 0; // the wait is a full-screen draw's if (oneshot > 0 && oneshot_budget_on()) { budget = frame_wait(); - wait_ms = oneshot_ms(budget, mesh, &s); + wait_ms = oneshot_ms(budget, mesh && !fb, &s); if (wait_ms) t0 = kern_now_ns(); } else if (oneshot > 0) { wait_ms = ONESHOT_WAIT_MS; @@ -3984,7 +4023,7 @@ static int prepare_draw_ex(VkExec *ex, int prim, int32_t count, int restart, int vk_exec_ahead(ex); // A mesh pipeline that is known to be compiling gets no fast link, so the few milliseconds a draw waits // for one are lost in every frame that meets it until the compile ends, 0.2 to 1.1 s later (MESH_WAIT_MS). - if (wait_ms && linking && mesh && vk_pipeline_building_ms(&s) > MESH_WAIT_MS) wait_ms = 0; + if (wait_ms && linking && mesh && !fb && vk_pipeline_building_ms(&s) > MESH_WAIT_MS) wait_ms = 0; if (wait_ms) p = vk_pipeline_graphics_async(&s, &info, &pending, wait_ms); } if (t0 && budget) { @@ -3999,7 +4038,7 @@ static int prepare_draw_ex(VkExec *ex, int prim, int32_t count, int restart, int if (!w0) w0 = kern_now_ns(); if (oneshot_budget_on()) { uint64_t *pool = frame_wait(), b0 = kern_now_ns(); - p = vk_pipeline_graphics_async(&s, &info, &pending, oneshot_ms(pool, mesh, &s)); + p = vk_pipeline_graphics_async(&s, &info, &pending, oneshot_ms(pool, mesh && !fb, &s)); pool_charge(pool, b0); } else { p = vk_pipeline_graphics_async(&s, &info, &pending, ONESHOT_WAIT_MS); @@ -4045,7 +4084,7 @@ static int prepare_draw_ex(VkExec *ex, int prim, int32_t count, int restart, int delta = cfg_bool("NX2NAVI_DYN_DELTA"); if (!delta) rt_log("vk_exec: every dynamic state recorded after each pipeline bind (NX2NAVI_DYN_DELTA=0)"); } - if (delta) vk_exec_frame.dyn_calls += (uint64_t)vk_pipeline_set_dynamic_delta(ex->cb, &s, &B.dyn, mesh, + if (delta) vk_exec_frame.dyn_calls += (uint64_t)vk_pipeline_set_dynamic_delta(ex->cb, &s, &B.dyn, mesh && !fb, &vk_exec_frame.dyn_left); else vk_pipeline_set_dynamic(ex->cb, &s); ex->recorded++; @@ -4068,7 +4107,7 @@ static int prepare_draw_ex(VkExec *ex, int prim, int32_t count, int restart, int if (fbl & ex->fb_dirty) feedback_barrier(ex); feedback_written(ex, &s, fbl); if (!mesh) bind_vertex_buffers(ex); - if (vk_dump_armed) vk_dump_draw(ex, &s, p, info, mesh); + if (vk_dump_armed) vk_dump_draw(ex, &s, p, info, mesh && !fb); if (vk_hdr_track) vk_hdr_draw(ex, info); vk_exec_oq_draw(ex); DRS.issued[mesh]++; @@ -4951,12 +4990,266 @@ static void mesh_shading_rate(VkExec *ex) { ex->recorded++; } +// ---- mesh fallback: a mesh draw on a device without VK_EXT_mesh_shader (mesh_fallback, vk_device.c), or on any +// device with mesh_fallback = 1. The program's task and mesh stages run as compute dispatches that write the +// fallback's buffers, and an ordinary draw reads them (mesh_fb.c makes the modules, vk_mesh_fb.c holds the buffers): +// the draw's record its workgroup count, copied on the GPU when the game's own record holds it +// [task dispatch the task module, with the task stage's bindings +// scan lists the mesh workgroups the tasks started and counts them into the record] +// mesh dispatch the mesh module, with the mesh stage's bindings +// draw one instance per mesh workgroup, with the generated vertex stage and the program's +// fragment stage, the instance count from the record +// Every step reads what the one before wrote, so a full barrier stands between them, and the dispatches cannot be +// recorded inside a pass: each such draw ends the open pass, and the next draw begins it again. The buffers are +// reused by every draw (vk_mesh_fb.c), which the barrier before a draw's first dispatch makes safe: the previous +// draw's vertex stage has read them by then. +// +// The modules of a program are made once, on a thread of their own, with their compute pipelines (a mesh program of +// the game is up to 900,000 SPIR-V words, and its pipeline compiles for as long as the mesh pipeline it replaces): +// until they are ready the program's draws are skipped, like draws whose pipeline is being built. Without background +// pipeline builds (the probes) they are made at the draw. +typedef struct MfbCs { + const GfxShader *ms, *ts; // the key: the translated mesh and task shaders (ts NULL: none) + MeshFbInfo mi, ti; + volatile int32_t state; // 0 queued or being made, 1 ready, -1 failed + const GfxShader *cs_mesh, *cs_task; // the compute modules (gfx_shader_own) + struct MfbCs *next; // the worker's queue +} MfbCs; +typedef struct { const GfxShader *ms; uint32_t mask; uint8_t type[32]; const GfxShader *vs; } MfbVs; +#define MFB_TAB 4096 +static MfbCs *mfb_cs_tab[MFB_TAB]; +static MfbVs mfb_vs_tab[MFB_TAB]; +static struct { + PlatMutex lock; + PlatCond wake; + PlatThread *th; + MfbCs *head, *tail; + uint64_t made, failed, pending_skips; +} MFB = {.lock = PLAT_MUTEX_INIT}; + +static uint32_t mfb_binding_base(const GfxShader *s) { // one past the highest binding the module declares + uint64_t n; + const uint32_t *w = gfx_shader_spirv(s, &n); + uint32_t top = 0; + for (uint64_t i = 5; i < n;) { + uint32_t op = w[i] & 0xffff, len = w[i] >> 16; + if (!len) break; + if (op == 71 && len == 4 && w[i + 2] == 33 && w[i + 3] > top) top = w[i + 3]; // OpDecorate Binding + i += len; + } + return top + 1; +} + +// The compute module of stage (6 task, 7 mesh) of e, as a shader of the runtime's own with the stage's resources and +// the fallback's buffers. +static const GfxShader *mfb_module(const MfbCs *e, int stage, char *err, size_t cap) { + const GfxShader *src = stage == 6 ? e->ts : e->ms; + uint64_t n, on = 0; + const uint32_t *w = gfx_shader_spirv(src, &n); + uint32_t base = mfb_binding_base(src); + uint32_t *out = stage == 6 ? meshfb_task_cs(w, n, &e->ti, base, &on, err, cap) + : meshfb_mesh_cs(w, n, &e->mi, base, e->ti.payload_bytes / 4, &on, err, cap); + if (!out) return NULL; + static const int task_slots[1] = {MESHFB_CB_TASK}, mesh_slots[3] = {MESHFB_CB_TASK, MESHFB_CB_MESH, MESHFB_CB_LIST}; + char *res = stage == 6 ? meshfb_res_json(gfx_shader_res_json(src), base, task_slots, 1) + : e->mi.has_task ? meshfb_res_json(gfx_shader_res_json(src), base, mesh_slots, 3) + : meshfb_res_json(gfx_shader_res_json(src), base + 1, mesh_slots + 1, 1); + const GfxShader *s = res ? gfx_shader_own(out, on, res) : NULL; + if (!s) snprintf(err, cap, "the converted module or its reflection is malformed"); + free(out), free(res); + return s; +} + +static void mfb_make(MfbCs *e) { + char err[200] = ""; + const VkPipeInfo *info; + int ok = (e->cs_mesh = mfb_module(e, 7, err, sizeof err)) != NULL; + if (ok && e->ts) ok = (e->cs_task = mfb_module(e, 6, err, sizeof err)) != NULL; + if (ok && !vk_pipeline_compute(e->cs_mesh, &info)) ok = 0, snprintf(err, sizeof err, "the mesh stage's compute pipeline failed"); + if (ok && e->ts && !vk_pipeline_compute(e->cs_task, &info)) ok = 0, snprintf(err, sizeof err, "the task stage's compute pipeline failed"); + plat_mutex_lock(&MFB.lock); + uint64_t nth = ok ? ++MFB.made : ++MFB.failed; + plat_mutex_unlock(&MFB.lock); + if (!ok && (nth <= 8 || !(nth & (nth - 1)))) + rt_log("vk: mesh fallback: a program's modules could not be made (%s): its draws are skipped (%llu such programs)", err, + (unsigned long long)nth); + __atomic_store_n(&e->state, ok ? 1 : -1, __ATOMIC_RELEASE); +} + +static void mfb_worker(void *arg) { + (void)arg; + plat_thread_priority(PLAT_PRIO_BELOW_NORMAL); // as the pipeline workers' optimised builds: the game's threads go first + for (;;) { + plat_mutex_lock(&MFB.lock); + while (!MFB.head) plat_cond_wait(&MFB.wake, &MFB.lock); + MfbCs *e = MFB.head; + if (!(MFB.head = e->next)) MFB.tail = NULL; + plat_mutex_unlock(&MFB.lock); + mfb_make(e); + } +} + +// The entry of (ms, ts), made and queued at its first draw; NULL (logged) when the program's layout is not carried. +static MfbCs *mfb_cs_get(VkExec *ex, const GfxShader *ms, const GfxShader *ts) { + uint32_t i = (uint32_t)((((uintptr_t)ms >> 4) * 0x9e3779b1u) ^ (((uintptr_t)ts >> 4) * 0x85ebca6bu)) & (MFB_TAB - 1); + for (uint32_t k = 0; k < MFB_TAB; k++, i = (i + 1) & (MFB_TAB - 1)) { + MfbCs *e = mfb_cs_tab[i]; + if (e && (e->ms != ms || e->ts != ts)) continue; + if (e) return e; + const uint8_t *ctl, *code; + uint32_t len; + char err[160] = ""; + e = calloc(1, sizeof *e); + if (!e) rt_fatal("vk: out of memory"); + e->ms = ms, e->ts = ts, e->state = -1; + mfb_cs_tab[i] = e; // a program that is refused is not looked at again + int ok = find_stage_scan(ex->prog, NVN_STAGE_MESH, &ctl, &code, &len) && len >= 0x80 && + meshfb_info(ctl, code, 7, &e->mi, err, sizeof err); + if (ok && ts) ok = find_stage_scan(ex->prog, NVN_STAGE_TASK, &ctl, &code, &len) && meshfb_info(ctl, code, 6, &e->ti, err, sizeof err); + if (ok && e->mi.has_task != (ts != NULL)) ok = 0, snprintf(err, sizeof err, "the mesh stage %s a task stage and the program has %s", + e->mi.has_task ? "expects" : "does not expect", ts ? "one" : "none"); + if (!ok) { + warn_once(W_MESH, "mesh fallback: a program is not carried (%s): its draws are skipped", *err ? err : "no stage data"); + return e; + } + e->state = 0; + if (!vk_pipeline_async_on()) { + mfb_make(e); + return e; + } + plat_mutex_lock(&MFB.lock); + if (!MFB.th) plat_cond_init(&MFB.wake), MFB.th = plat_thread_new(mfb_worker, NULL, "mesh fallback"); + if (MFB.tail) MFB.tail->next = e; + else MFB.head = e; + MFB.tail = e; + plat_cond_signal(&MFB.wake); + plat_mutex_unlock(&MFB.lock); + return e; + } + rt_fatal("vk: mesh fallback: more than %d task and mesh program pairs", MFB_TAB); +} + +// The vertex stage that draws e's mesh buffer slots for fragment shader fs: an output for every input fs reads, in +// its type. +static const GfxShader *mfb_vs_get(const MfbCs *e, const GfxShader *fs) { + GfxShaderIo fin[32]; + uint8_t type[32] = {0}; + uint32_t mask = fs ? gfx_shader_inputs(fs, fin) : 0; + for (int g = 0; g < 32; g++) + if (mask >> g & 1) type[g] = fin[g].type; + uint32_t i = (uint32_t)((((uintptr_t)e->ms >> 4) * 0x9e3779b1u) ^ (mask * 0x85ebca6bu)) & (MFB_TAB - 1); + for (uint32_t k = 0; k < MFB_TAB; k++, i = (i + 1) & (MFB_TAB - 1)) { + MfbVs *v = &mfb_vs_tab[i]; + if (v->ms && (v->ms != e->ms || v->mask != mask || memcmp(v->type, type, 32))) continue; + if (v->ms) return v->vs; + static const char res[] = "{\"key\":\"meshfb\",\"stage\":\"vs\",\"notes\":[],\"cbufs\":[],\"sbufs\":[{\"binding\": 1, " + "\"slot\": -1, \"cb_slot\": 66, \"cb_offset\": 0, \"flags\": 0}],\"textures\":[],\"images\":[]}"; + _Static_assert(MESHFB_CB_MESH == 66, "the vertex stage's reflection names the mesh buffer"); + uint64_t n = 0; + uint32_t *w = meshfb_vs(&e->mi, mask, type, &n); + v->ms = e->ms, v->mask = mask, memcpy(v->type, type, 32); + v->vs = w ? gfx_shader_own(w, n, res) : NULL; + free(w); + if (!v->vs) warn_once(W_MESH, "mesh fallback: a vertex stage could not be made: its draws are skipped"); + return v->vs; + } + rt_fatal("vk: mesh fallback: more than %d vertex stages", MFB_TAB); +} + +// One of the draw's compute dispatches: module cs with the bindings of NVN stage `stage`, its group counts read from +// the record at `at`. 0 when its pipeline is not there. +static int mfb_dispatch(VkExec *ex, const GfxShader *cs, int stage, VkBuffer rec, VkDeviceSize at) { + const VkPipeInfo *info = NULL; + VkPipeline p = vk_pipeline_compute(cs, &info); + if (!p || !info || (info->pool && !pool_on())) return 0; + bind_pipeline(ex, VK_PIPELINE_BIND_POINT_COMPUTE, p); + mfb_stage = stage; + bind_resources(ex, info); + mfb_stage = 0; + VkPassMark *m = dispatch_mark(ex, cs, 0, 0, 0); + vkCmdDispatchIndirect(ex->cb, rec, at); + vk_pass_mark_end(ex->cb, m); + ex->recorded++; + return 1; +} + +// A mesh draw of `count` workgroups, or of the count in the game's record at ind (4 bytes, GPU-written). +static void mfb_draw(VkExec *ex, uint32_t count, const VkMemSpan *ind) { + if (ex->render_disabled) { + vk_cond_draws_skip += ex->has_render_cond; + draw_skipped(ex, 1, SK_DISABLED); + return; + } + if (!ex->prog) { draw_skipped(ex, 1, SK_NO_PROGRAM); return; } + cb_res(ex); + dummies_init(); + int has_ms, has_ts, has_fs, w_ms = 0, w_ts = 0, w_fs = 0; + const GfxShader *ms = shader_for(ex, NVN_STAGE_MESH, &has_ms, &w_ms, 0); + const GfxShader *ts = shader_for(ex, NVN_STAGE_TASK, &has_ts, &w_ts, 0); + const GfxShader *fs = shader_for(ex, NVN_STAGE_FS, &has_fs, &w_fs, 0); + if (!has_ms) { draw_skipped(ex, 1, SK_NO_VERTEX_STAGE); return; } + if (w_ms || w_ts || w_fs) { + vk_exec_ahead(ex); + perf_count(PERF_CNT_SHADER_SKIP); + draw_skipped(ex, 1, SK_SHADER_PENDING); + return; + } + if (!ms || (has_ts && !ts) || (has_fs && !fs)) { draw_skipped(ex, 1, SK_SHADER_FAILED); return; } + MfbCs *c = mfb_cs_get(ex, ms, ts); + int state = __atomic_load_n(&c->state, __ATOMIC_ACQUIRE); + if (!state && !vk_exec_targets_recent(ex, pipe_skip_mode() >= 1)) { + // a target the game does not render every frame: the draw may not come again, so it waits for its modules + // from the frame's wait pool, as such a draw waits for a pipeline (prepare_draw_ex) + uint64_t *pool = frame_wait(), t0 = kern_now_ns(); + while (!state && pool_ms(pool, t0)) plat_sleep_ms(1), state = __atomic_load_n(&c->state, __ATOMIC_ACQUIRE); + pool_charge(pool, t0); + } + if (!state) { MFB.pending_skips++; draw_skipped(ex, 1, SK_PIPE_PENDING); return; } + const GfxShader *vs = state > 0 ? mfb_vs_get(c, fs) : NULL; + if (!vs || !vk_meshfb_init()) { draw_skipped(ex, 1, SK_PIPE_FAILED); return; } + uint32_t verts = c->mi.maxp * c->mi.ipp, slot_words = meshfb_slot_words(&c->mi); + uint32_t most = vk_ctx()->props.limits.maxComputeWorkGroupCount[0]; + if (count > most) count = most; // a count the GPU wrote is not looked at here: the dispatch reads it as it is + VkBuffer rec; + int cmds; + vk_exec_barrier(ex); // ends the pass; the culling's record and the last draw's reads come before what follows + VkDeviceSize at = vk_meshfb_record(ex->cb, ind ? ind->buffer : VK_NULL_HANDLE, ind ? ind->offset : 0, count, ts != NULL, + verts, slot_words, &rec, &cmds); + ex->recorded += (uint32_t)cmds; + vk_exec_barrier(ex); + if (ts) { + if (!mfb_dispatch(ex, c->cs_task, NVN_STAGE_TASK, rec, at)) { draw_skipped(ex, 1, SK_PIPE_FAILED); return; } + vk_exec_barrier(ex); + vk_meshfb_scan(ex->cb, at, c->ti.payload_bytes / 4, c->ti.threads, slot_words, verts); + B.pipe[1] = VK_NULL_HANDLE; // the scan bound its own + ex->recorded += 4; + vk_exec_barrier(ex); + } + if (!mfb_dispatch(ex, c->cs_mesh, NVN_STAGE_MESH, rec, at + 16)) { draw_skipped(ex, 1, SK_PIPE_FAILED); return; } + vk_exec_barrier(ex); + vk_meshfb_note(ex->cb, at); + ex->recorded++; + vk_exec_barrier(ex); + c0_binst = 0; + mfb_cur_vs = vs; + int prim = c->mi.ipp == 1 ? NVN_DRAW_PRIMITIVE_POINTS : c->mi.ipp == 2 ? NVN_DRAW_PRIMITIVE_LINES : NVN_DRAW_PRIMITIVE_TRIANGLES; + if (!prepare_draw_ex(ex, prim, 0x7fffffff, 0, 2)) return; + vk_exec_frame.draws++; + vkCmdDrawIndirect(ex->cb, rec, at + 32, 1, sizeof(VkDrawIndirectCommand)); + ex->recorded++; +} + static void op_draw_mesh(VkExec *ex, const p_draw_mesh *d) { - PFN_vkCmdDrawMeshTasksEXT fn = draw_mesh_fn(); - if (!fn) return warn_once(W_MESH, "mesh draws need VK_EXT_mesh_shader: skipped"); static int on = -1; if (on < 0) on = rt_knob("MESH", "1"); // NX2NAVI_MESH=0: mesh draws are skipped (diagnosis) if (!d->count || !on) return; + if (vk_ctx()->mesh_fallback) { + if (d->first) warn_once(W_MESH, "a mesh draw from workgroup %u: drawn from workgroup 0", d->first); + mfb_draw(ex, d->count, NULL); + return; + } + PFN_vkCmdDrawMeshTasksEXT fn = draw_mesh_fn(); + if (!fn) return warn_once(W_MESH, "mesh draws need VK_EXT_mesh_shader: skipped"); c0_binst = 0; if (!prepare_draw_ex(ex, NVN_DRAW_PRIMITIVE_TRIANGLES, 0x7fffffff, 0, 1)) return; if (d->first) warn_once(W_MESH, "a mesh draw from workgroup %u: drawn from workgroup 0", d->first); @@ -4980,13 +5273,21 @@ static void op_draw_mesh(VkExec *ex, const p_draw_mesh *d) { // MESH_RING entries outlast the frames in flight: an entry is rewritten only after a barrier behind its last draw. enum { MESH_RING = 4096 }; static VkBuffer mesh_ring; +static VmaAllocation mesh_ring_alloc; static uint32_t mesh_ring_next; // NX2NAVI_GPU_PASSES: the workgroup counts of the indirect mesh draws of the timed frames, copied to host memory. int vk_gpu_passes_capturing(void); enum { MESH_COUNTS = 4096 }; static VkBuffer mesh_count_buf; +static VmaAllocation mesh_count_alloc; static uint32_t *mesh_count_map, mesh_count_n, mesh_count_was; + +static void mesh_ring_release(VkCtx *c) { // vk_exec_draw_shutdown: a probe ends with the device idle + if (mesh_ring) vmaDestroyBuffer(c->vma, mesh_ring, mesh_ring_alloc); + if (mesh_count_buf) vmaDestroyBuffer(c->vma, mesh_count_buf, mesh_count_alloc); + mesh_ring = mesh_count_buf = VK_NULL_HANDLE, mesh_count_map = NULL; +} uint32_t vk_exec_mesh_counts(const uint32_t **out) { *out = mesh_count_map; return mesh_count_map ? mesh_count_n : 0; @@ -5001,8 +5302,7 @@ static void mesh_count_note(VkExec *ex, const VkMemSpan *src) { VmaAllocationCreateInfo ai = {.usage = VMA_MEMORY_USAGE_AUTO_PREFER_HOST, .flags = VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT | VMA_ALLOCATION_CREATE_MAPPED_BIT}; VmaAllocationInfo info; - VmaAllocation a; - if (vmaCreateBuffer(vk_ctx()->vma, &bi, &ai, &mesh_count_buf, &a, &info) != VK_SUCCESS) return; + if (vmaCreateBuffer(vk_ctx()->vma, &bi, &ai, &mesh_count_buf, &mesh_count_alloc, &info) != VK_SUCCESS) return; mesh_count_map = info.pMappedData; } if (mesh_count_n >= MESH_COUNTS) return; @@ -5053,8 +5353,7 @@ static int mesh_ring_entry(VkExec *ex, const p_draw_mesh_ind *d, const VkMemSpan VkBufferCreateInfo bi = {.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, .size = 12ull * MESH_RING, .usage = VK_BUFFER_USAGE_INDIRECT_BUFFER_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT}; VmaAllocationCreateInfo ai = {.usage = VMA_MEMORY_USAGE_AUTO_PREFER_DEVICE}; - VmaAllocation a; - if (vmaCreateBuffer(vk_ctx()->vma, &bi, &ai, &mesh_ring, &a, NULL) != VK_SUCCESS) { + if (vmaCreateBuffer(vk_ctx()->vma, &bi, &ai, &mesh_ring, &mesh_ring_alloc, NULL) != VK_SUCCESS) { mesh_ring = VK_NULL_HANDLE; return 0; } @@ -5107,7 +5406,8 @@ static void op_draw_mesh_ind(VkExec *ex, const p_draw_mesh_ind *d) { if (vk_ctx()->has_mesh_shader) fn = (PFN_vkCmdDrawMeshTasksIndirectEXT)vk_dpa(vk_ctx()->device, "vkCmdDrawMeshTasksIndirectEXT"); } - if (!fn) return warn_once(W_MESH, "mesh draws need VK_EXT_mesh_shader: skipped"); + int fallback = vk_ctx()->mesh_fallback; + if (!fn && !fallback) return warn_once(W_MESH, "mesh draws need VK_EXT_mesh_shader: skipped"); static int on = -1; if (on < 0) on = rt_knob("MESH", "1"); if (!on) return; @@ -5117,6 +5417,12 @@ static void op_draw_mesh_ind(VkExec *ex, const p_draw_mesh_ind *d) { (unsigned long long)d->ind); c0_binst = 0; VK_MEM_USE(ip); + if (fallback) { + mfb_draw(ex, 0, &ip); + vk_mem_count(ip.cls, VK_MUSE_IND, 4); + VK_MEM_BOUND(ip); + return; + } VkDeviceSize at; if (!mesh_ring_entry(ex, d, &ip, &at)) return warn_once(W_MESH, "the indirect mesh ring could not be created: skipped"); if (!prepare_draw_ex(ex, NVN_DRAW_PRIMITIVE_TRIANGLES, 0x7fffffff, 0, 1)) return; diff --git a/runtime/vk_mesh_fb.c b/runtime/vk_mesh_fb.c new file mode 100644 index 00000000..e0939f54 --- /dev/null +++ b/runtime/vk_mesh_fb.c @@ -0,0 +1,215 @@ +// The mesh fallback's buffers and records. A task or mesh program runs as a compute dispatch that writes a storage +// buffer (mesh_fb.c), and the draw reads it: +// task buffer a task workgroup's payload at its index, payload-size apart +// list per mesh workgroup after a task stage: its task and its index among that task's (meshfb_scan.comp) +// mesh buffer a mesh workgroup's vertex lines and primitive area at its slot +// records per draw 64 bytes: the indirect arguments of its task dispatch, its mesh dispatch and its draw +// The three buffers are reused by every draw: a draw's dispatches write them and its vertex stage has read them +// before the next draw's dispatches run, with a barrier in between (vk_exec_draw.c mfb_draw), so their size is what +// the largest draw needs, not what a frame's draws need together. mesh_fallback_mb sets it: an eighth for the task +// buffer, the rest for the mesh buffer. Workgroups past either buffer's end write nothing and draw nothing (the +// modules check the buffer's length); the draws that happened to are counted and the log names the size they needed. +// A record is used by one draw and written again only after MESHFB_RECS further draws. +#include +#include +#include + +#include "cfg.h" +#include "host.h" +#include "mesh_fb.h" +#include "vk.h" +#include "vk_mesh_fb.h" +#include "meshfb_scan_spv.h" + +enum { MESHFB_RECS = 4096, REC_BYTES = 64 }; + +static struct { + int tried, ok; + VkBuffer task, mesh, list, rec, stat; + VmaAllocation task_a, mesh_a, list_a, rec_a, stat_a; + VkDeviceSize task_bytes, mesh_bytes, list_bytes; // as one storage binding spans them + const uint32_t *stat_map; // per record: mesh workgroups and task workgroups it asked for + uint32_t cap_slots[MESHFB_RECS], slot_words[MESHFB_RECS], cap_tasks[MESHFB_RECS]; // what the record's draw had room for + uint8_t noted[MESHFB_RECS]; + uint32_t next; + uint64_t draws, short_draws, worst_bytes; + PFN_vkCmdPushDescriptorSetKHR push; + VkDescriptorSetLayout dsl; + VkPipelineLayout layout; + VkPipeline scan; +} M; + +static int make_buffer(VkCtx *c, VkDeviceSize size, VkBufferUsageFlags usage, int host, VkBuffer *b, VmaAllocation *a, void **map) { + VkBufferCreateInfo bi = {.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, .size = size, .usage = usage}; + VmaAllocationCreateInfo ai = {.usage = host ? VMA_MEMORY_USAGE_AUTO_PREFER_HOST : VMA_MEMORY_USAGE_AUTO_PREFER_DEVICE, + .flags = host ? VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT | VMA_ALLOCATION_CREATE_MAPPED_BIT : 0}; + VmaAllocationInfo info; + if (vmaCreateBuffer(c->vma, &bi, &ai, b, a, &info) != VK_SUCCESS) return *b = VK_NULL_HANDLE, 0; + if (map) *map = info.pMappedData; + return !host || info.pMappedData; +} + +int vk_meshfb_init(void) { + if (M.tried) return M.ok; + M.tried = 1; + VkCtx *c = vk_ctx(); + if (!c) return 0; + uint64_t mb = (uint64_t)cfg_int("NX2NAVI_MESH_FALLBACK_MB"); + if (mb < 16) mb = 16; + if (mb > 4096) mb = 4096; + VkDeviceSize range = c->props.limits.maxStorageBufferRange & ~(VkDeviceSize)3, task = (mb << 20) / 8, mesh = (mb << 20) - task; + VkDeviceSize list = mesh / 128; // an entry per slot of 512 bytes, and no mesh program's slot is smaller + void *map = NULL; + const VkBufferUsageFlags st = VK_BUFFER_USAGE_STORAGE_BUFFER_BIT; + if (!make_buffer(c, task, st, 0, &M.task, &M.task_a, NULL) || !make_buffer(c, mesh, st, 0, &M.mesh, &M.mesh_a, NULL) || + !make_buffer(c, list, st, 0, &M.list, &M.list_a, NULL) || + !make_buffer(c, (VkDeviceSize)MESHFB_RECS * REC_BYTES, + st | VK_BUFFER_USAGE_INDIRECT_BUFFER_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT | VK_BUFFER_USAGE_TRANSFER_SRC_BIT, 0, + &M.rec, &M.rec_a, NULL) || + !make_buffer(c, (VkDeviceSize)MESHFB_RECS * 8, VK_BUFFER_USAGE_TRANSFER_DST_BIT, 1, &M.stat, &M.stat_a, &map)) { + rt_log("vk: mesh fallback: its buffers (%llu MiB, mesh_fallback_mb) could not be allocated: task and mesh draws are skipped", + (unsigned long long)mb); + return 0; + } + M.stat_map = map; + memset(map, 0, (size_t)MESHFB_RECS * 8); + M.task_bytes = task < range ? task : range, M.mesh_bytes = mesh < range ? mesh : range, M.list_bytes = list < range ? list : range; + + M.push = (PFN_vkCmdPushDescriptorSetKHR)vk_dpa(c->device, "vkCmdPushDescriptorSetKHR"); + VkDescriptorSetLayoutBinding b[4]; + for (int i = 0; i < 4; i++) + b[i] = (VkDescriptorSetLayoutBinding){.binding = (uint32_t)i, .descriptorType = VK_DESCRIPTOR_TYPE_STORAGE_BUFFER, + .descriptorCount = 1, .stageFlags = VK_SHADER_STAGE_COMPUTE_BIT}; + VkDescriptorSetLayoutCreateInfo dci = {.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_SET_LAYOUT_CREATE_INFO, + .flags = VK_DESCRIPTOR_SET_LAYOUT_CREATE_PUSH_DESCRIPTOR_BIT_KHR, + .bindingCount = 4, .pBindings = b}; + VkPushConstantRange pr = {VK_SHADER_STAGE_COMPUTE_BIT, 0, 24}; + VkShaderModuleCreateInfo mi = {.sType = VK_STRUCTURE_TYPE_SHADER_MODULE_CREATE_INFO, .codeSize = sizeof meshfb_scan_spv, + .pCode = meshfb_scan_spv}; + VkShaderModule m = VK_NULL_HANDLE; + if (!M.push || vkCreateDescriptorSetLayout(c->device, &dci, NULL, &M.dsl) != VK_SUCCESS) goto fail; + VkPipelineLayoutCreateInfo li = {.sType = VK_STRUCTURE_TYPE_PIPELINE_LAYOUT_CREATE_INFO, .setLayoutCount = 1, + .pSetLayouts = &M.dsl, .pushConstantRangeCount = 1, .pPushConstantRanges = &pr}; + if (vkCreatePipelineLayout(c->device, &li, NULL, &M.layout) != VK_SUCCESS || + vkCreateShaderModule(c->device, &mi, NULL, &m) != VK_SUCCESS) + goto fail; + VkComputePipelineCreateInfo ci = {.sType = VK_STRUCTURE_TYPE_COMPUTE_PIPELINE_CREATE_INFO, + .stage = {.sType = VK_STRUCTURE_TYPE_PIPELINE_SHADER_STAGE_CREATE_INFO, + .stage = VK_SHADER_STAGE_COMPUTE_BIT, .module = m, .pName = "main"}, + .layout = M.layout}; + VkResult res = vkCreateComputePipelines(c->device, VK_NULL_HANDLE, 1, &ci, NULL, &M.scan); + vkDestroyShaderModule(c->device, m, NULL); + if (res != VK_SUCCESS) goto fail; + rt_log("vk: mesh fallback: %llu MiB for task payloads, %llu MiB for mesh output (mesh_fallback_mb = %llu)", + (unsigned long long)(M.task_bytes >> 20), (unsigned long long)(M.mesh_bytes >> 20), (unsigned long long)mb); + return M.ok = 1; +fail: + rt_log("vk: mesh fallback: the scan pipeline could not be created: task and mesh draws are skipped"); + return 0; +} + +VkDescriptorBufferInfo vk_meshfb_buffer(int cb_slot) { + if (cb_slot == MESHFB_CB_TASK) return (VkDescriptorBufferInfo){M.task, 0, M.task_bytes}; + if (cb_slot == MESHFB_CB_LIST) return (VkDescriptorBufferInfo){M.list, 0, M.list_bytes}; + return (VkDescriptorBufferInfo){M.mesh, 0, M.mesh_bytes}; +} + +uint64_t vk_meshfb_mesh_words(void) { return M.mesh_bytes / 4; } + +// The record about to be written again: what its last draw asked for against what it had room for. +static void record_check(uint32_t i) { + if (!M.noted[i]) return; + M.noted[i] = 0; + uint32_t want = M.stat_map[2 * i], tasks = M.stat_map[2 * i + 1]; + if (want <= M.cap_slots[i] && tasks <= M.cap_tasks[i]) return; + uint64_t need = (uint64_t)want * M.slot_words[i] * 4; + if (need > M.worst_bytes) M.worst_bytes = need; + uint64_t n = ++M.short_draws; + if (n <= 4 || !(n & (n - 1))) { + char from[96] = ""; + if (tasks) snprintf(from, sizeof from, " from %u task workgroups (room for %u)", tasks, M.cap_tasks[i]); + rt_log("vk: mesh fallback: a draw's programs emitted %u mesh workgroups of %u bytes%s, and the mesh buffer holds %u: " + "what did not fit was not drawn (%llu such draws of %llu; mesh_fallback_mb of %llu or more would hold the " + "largest so far)", want, M.slot_words[i] * 4, from, M.cap_slots[i], (unsigned long long)n, + (unsigned long long)M.draws, (unsigned long long)((M.worst_bytes * 8 / 7 >> 20) + 1)); + } +} + +VkDeviceSize vk_meshfb_record(VkCommandBuffer cb, VkBuffer src, VkDeviceSize src_off, uint32_t count, int has_task, + uint32_t verts, uint32_t slot_words, VkBuffer *rec, int *cmds) { + uint32_t i = M.next++ % MESHFB_RECS; + VkDeviceSize at = (VkDeviceSize)i * REC_BYTES; + record_check(i); + M.draws++; + M.slot_words[i] = slot_words, M.cap_slots[i] = (uint32_t)(M.mesh_bytes / 4 / (slot_words ? slot_words : 1)); + M.cap_tasks[i] = 0xffffffffu; // vk_meshfb_scan knows the payload size + if (has_task && M.cap_slots[i] > M.list_bytes / 4) M.cap_slots[i] = (uint32_t)(M.list_bytes / 4); + *rec = M.rec; + int n = 0; + vkCmdFillBuffer(cb, M.rec, at, REC_BYTES, 1), n++; + // where the workgroup count goes: the task dispatch, or the mesh dispatch, the draw's instances and the report + VkDeviceSize to[3] = {has_task ? at : at + 16, at + 36, at + 48}; + int nto = has_task ? 1 : 3; + if (!has_task) { + vkCmdFillBuffer(cb, M.rec, at + 32, 4, verts), n++; + vkCmdFillBuffer(cb, M.rec, at + 40, 8, 0), n++; + vkCmdFillBuffer(cb, M.rec, at + 52, 4, 0), n++; + } + if (src) { + VkBufferCopy cp[3]; + for (int k = 0; k < nto; k++) cp[k] = (VkBufferCopy){src_off, to[k], 4}; + vkCmdCopyBuffer(cb, src, M.rec, (uint32_t)nto, cp), n++; + } else { + for (int k = 0; k < nto; k++) vkCmdFillBuffer(cb, M.rec, to[k], 4, count), n++; + } + *cmds = n; + return at; +} + +void vk_meshfb_scan(VkCommandBuffer cb, VkDeviceSize at, uint32_t task_words, uint32_t fan, uint32_t slot_words, uint32_t verts) { + uint32_t i = (uint32_t)(at / REC_BYTES); + M.cap_tasks[i] = (uint32_t)(M.task_bytes / 4 / (task_words ? task_words : 1)); + VkDescriptorBufferInfo bi[4] = {{M.rec, 0, VK_WHOLE_SIZE}, {M.task, 0, M.task_bytes}, {M.list, 0, M.list_bytes}, + {M.mesh, 0, M.mesh_bytes}}; + VkWriteDescriptorSet w[4]; + for (int k = 0; k < 4; k++) + w[k] = (VkWriteDescriptorSet){.sType = VK_STRUCTURE_TYPE_WRITE_DESCRIPTOR_SET, .dstBinding = (uint32_t)k, + .descriptorCount = 1, .descriptorType = VK_DESCRIPTOR_TYPE_STORAGE_BUFFER, + .pBufferInfo = &bi[k]}; + uint32_t pc[6] = {(uint32_t)(at / 4), task_words, fan, slot_words, verts, vk_ctx()->props.limits.maxComputeWorkGroupCount[0]}; + vkCmdBindPipeline(cb, VK_PIPELINE_BIND_POINT_COMPUTE, M.scan); + rec_push(M.push, cb, VK_PIPELINE_BIND_POINT_COMPUTE, M.layout, 0, 4, w); + vkCmdPushConstants(cb, M.layout, VK_SHADER_STAGE_COMPUTE_BIT, 0, sizeof pc, pc); + vkCmdDispatch(cb, 1, 1, 1); +} + +void vk_meshfb_note(VkCommandBuffer cb, VkDeviceSize at) { + uint32_t i = (uint32_t)(at / REC_BYTES); + VkBufferCopy cp = {at + 48, (VkDeviceSize)i * 8, 8}; + vkCmdCopyBuffer(cb, M.rec, M.stat, 1, &cp); + M.noted[i] = 1; +} + +uint64_t vk_meshfb_short(void) { + if (!M.ok) return 0; + for (uint32_t i = 0; i < MESHFB_RECS; i++) record_check(i); + return M.short_draws; +} + +void vk_meshfb_shutdown(void) { + VkCtx *c = vk_ctx(); + if (!c || !M.tried) return; + if (vk_meshfb_short()) + rt_log("vk: mesh fallback: %llu of %llu draws emitted more than the buffers hold (mesh_fallback_mb of %llu or more would " + "hold the largest)", (unsigned long long)M.short_draws, (unsigned long long)M.draws, + (unsigned long long)((M.worst_bytes * 8 / 7 >> 20) + 1)); + if (M.scan) vkDestroyPipeline(c->device, M.scan, NULL); + if (M.layout) vkDestroyPipelineLayout(c->device, M.layout, NULL); + if (M.dsl) vkDestroyDescriptorSetLayout(c->device, M.dsl, NULL); + if (M.task) vmaDestroyBuffer(c->vma, M.task, M.task_a); + if (M.mesh) vmaDestroyBuffer(c->vma, M.mesh, M.mesh_a); + if (M.list) vmaDestroyBuffer(c->vma, M.list, M.list_a); + if (M.rec) vmaDestroyBuffer(c->vma, M.rec, M.rec_a); + if (M.stat) vmaDestroyBuffer(c->vma, M.stat, M.stat_a); + memset(&M, 0, sizeof M); +} diff --git a/runtime/vk_mesh_fb.h b/runtime/vk_mesh_fb.h new file mode 100644 index 00000000..8012c510 --- /dev/null +++ b/runtime/vk_mesh_fb.h @@ -0,0 +1,29 @@ +// The mesh fallback's buffers and its records (vk_mesh_fb.c); the modules are mesh_fb.c's, the draws +// vk_exec_draw.c's ("mesh fallback"). +#pragma once +#include "vk.h" + +// Creates the buffers and the scan pipeline on first use. 0 when that failed (logged once): the draws are skipped. +int vk_meshfb_init(void); +// The buffer a module's extra storage binding reads or writes (mesh_fb.h MESHFB_CB_*), whole or as much of it as one +// storage binding may span. +VkDescriptorBufferInfo vk_meshfb_buffer(int cb_slot); +uint64_t vk_meshfb_mesh_words(void); // the mesh buffer's size as a binding sees it, in words + +// A draw's record, 64 bytes of the record buffer (*rec) at the offset returned: the task dispatch at +0, the mesh +// dispatch at +16, the draw (VkDrawIndirectCommand) at +32. Records the fills and copies that start it, outside a +// pass: the workgroup count from 4 bytes at src_off of src, or `count` when src is VK_NULL_HANDLE. Without a task +// stage that count is the mesh dispatch and the draw's instance count; with one it is the task dispatch, and +// vk_meshfb_scan writes the rest. slot_words: of the draw's mesh program, for the report of a draw the mesh buffer +// was too small for. Returns the number of commands recorded in *cmds. +VkDeviceSize vk_meshfb_record(VkCommandBuffer cb, VkBuffer src, VkDeviceSize src_off, uint32_t count, int has_task, + uint32_t verts, uint32_t slot_words, VkBuffer *rec, int *cmds); +// The pass between the task and the mesh dispatch (shaders/meshfb_scan.comp): binds its own compute pipeline and set. +void vk_meshfb_scan(VkCommandBuffer cb, VkDeviceSize at, uint32_t task_words, uint32_t fan, uint32_t slot_words, uint32_t verts); +// After the record is complete (the copies or the scan, and a barrier): keeps what the draw asked for where the host +// reads it later. One copy command. +void vk_meshfb_note(VkCommandBuffer cb, VkDeviceSize at); +// Draws so far whose programs emitted more than the buffers hold, the records not looked at yet included: call it +// with the device idle (a probe, the shutdown). +uint64_t vk_meshfb_short(void); +void vk_meshfb_shutdown(void); // the device must be idle diff --git a/runtime/vk_pipeline.c b/runtime/vk_pipeline.c index 1b910638..2cfdacb0 100644 --- a/runtime/vk_pipeline.c +++ b/runtime/vk_pipeline.c @@ -3508,6 +3508,7 @@ static void keys_note(const VkPipeState *s) { static const uint8_t none[32]; if (s->fs && !memcmp(gfx_shader_input_id(s->fs), none, 32)) return; // the runtime's own shader (gfx_shader_own) if (s->ts && !memcmp(gfx_shader_input_id(s->ts), none, 32)) return; + if (!memcmp(gfx_shader_input_id(s->vs), none, 32)) return; // the mesh fallback's vertex stage (vk_exec_draw.c mfb_vs_get) KeyRec r; rec_fill(&r, s); if (!seen_add(rec_hash(&r))) return; diff --git a/runtime/vk_rec.c b/runtime/vk_rec.c index a06f0e97..97098330 100644 --- a/runtime/vk_rec.c +++ b/runtime/vk_rec.c @@ -49,7 +49,7 @@ enum { R_SetRasterizerDiscardEnable, R_SetScissorWithCount, R_SetStencilCompareMask, R_SetStencilOp, R_SetStencilReference, R_SetStencilTestEnable, R_SetStencilWriteMask, R_SetViewportWithCount, R_WriteTimestamp, R_WriteTimestamp2, R_PushTpl, R_Push, R_VertexInput, R_Bool, R_PolygonMode, R_BlendEnable, R_BlendEquation, R_WriteMask, R_EndCb, - R_PassMark, R_DrawMesh, R_DrawMeshInd, R_ShadingRate, + R_PassMark, R_DrawMesh, R_DrawMeshInd, R_ShadingRate, R_DrawIndirect, }; static uint64_t now_ns(void) { return plat_now_ns(); } @@ -347,6 +347,9 @@ FIXED(DrawIndexed, (VkCommandBuffer cb, uint32_t n, uint32_t i, uint32_t f, int3 #define PACK_DrawIndexedIndirect *a = (A_DrawIndexedIndirect){b, o, n, s} FIXED(DrawIndexedIndirect, (VkCommandBuffer cb, VkBuffer b, VkDeviceSize o, uint32_t n, uint32_t s), (cb, b, o, n, s), VkBuffer b; VkDeviceSize o; uint32_t n, s) +#define PACK_DrawIndirect *a = (A_DrawIndirect){b, o, n, s} +FIXED(DrawIndirect, (VkCommandBuffer cb, VkBuffer b, VkDeviceSize o, uint32_t n, uint32_t s), (cb, b, o, n, s), + VkBuffer b; VkDeviceSize o; uint32_t n, s) #define PACK_EndQuery *a = (A_EndQuery){p, q} FIXED(EndQuery, (VkCommandBuffer cb, VkQueryPool p, uint32_t q), (cb, p, q), VkQueryPool p; uint32_t q) #define PACK_EndRendering (void)a @@ -705,6 +708,7 @@ static void exec_packet(uint32_t op, void *p) { F(Draw, a->n, a->i, a->f, a->fi) F(DrawIndexed, a->n, a->i, a->f, a->v, a->fi) F(DrawIndexedIndirect, a->b, a->o, a->n, a->s) + F(DrawIndirect, a->b, a->o, a->n, a->s) F(EndQuery, a->p, a->q) F(FillBuffer, a->b, a->o, a->n, a->v) F(ResetQueryPool, a->p, a->f, a->n) diff --git a/runtime/vk_rec.h b/runtime/vk_rec.h index 9261ee16..62f8c440 100644 --- a/runtime/vk_rec.h +++ b/runtime/vk_rec.h @@ -80,6 +80,7 @@ void rec_pass_mark(VkCommandBuffer cb, uint32_t id, int end); X(Draw, (VkCommandBuffer cb, uint32_t n, uint32_t i, uint32_t f, uint32_t fi), (cb, n, i, f, fi)) \ X(DrawIndexed, (VkCommandBuffer cb, uint32_t n, uint32_t i, uint32_t f, int32_t v, uint32_t fi), (cb, n, i, f, v, fi)) \ X(DrawIndexedIndirect, (VkCommandBuffer cb, VkBuffer b, VkDeviceSize o, uint32_t n, uint32_t s), (cb, b, o, n, s)) \ + X(DrawIndirect, (VkCommandBuffer cb, VkBuffer b, VkDeviceSize o, uint32_t n, uint32_t s), (cb, b, o, n, s)) \ X(EndQuery, (VkCommandBuffer cb, VkQueryPool p, uint32_t q), (cb, p, q)) \ X(EndRendering, (VkCommandBuffer cb), (cb)) \ X(FillBuffer, (VkCommandBuffer cb, VkBuffer b, VkDeviceSize o, VkDeviceSize n, uint32_t v), (cb, b, o, n, v)) \ @@ -137,6 +138,7 @@ REC_CMDS(REC_DECL) #define vkCmdDraw rc_vkCmdDraw #define vkCmdDrawIndexed rc_vkCmdDrawIndexed #define vkCmdDrawIndexedIndirect rc_vkCmdDrawIndexedIndirect +#define vkCmdDrawIndirect rc_vkCmdDrawIndirect #define vkCmdEndQuery rc_vkCmdEndQuery #define vkCmdEndRendering rc_vkCmdEndRendering #define vkCmdFillBuffer rc_vkCmdFillBuffer diff --git a/tools/build_shaders.py b/tools/build_shaders.py index 69088412..d62b4148 100644 --- a/tools/build_shaders.py +++ b/tools/build_shaders.py @@ -5,6 +5,7 @@ The repository holds the shaders as source only. Each becomes runtime/_spv missing and the stale ones before it compiles the runtime. Three kinds of source: runtime/shaders/.comp, .vert, .frag GLSL, compiled with glslc (one name per shader) and checked with spirv-val + runtime/shaders/.mesh, .task the same for the mesh fallback probe's mesh and task shaders (SPIR-V 1.5) runtime/shaders/.spvasm SPIR-V assembly, assembled with spirv-as and checked with spirv-val tools/make_dlaa_mask.py, make_mvobj_fs.py shaders assembled in Python (ASSEMBLED); they need no other tool @@ -34,7 +35,8 @@ TOOLS = Path(__file__).resolve().parent ROOT = TOOLS.parent BIN = Path(r'C:\msys64\ucrt64\bin') # Windows; elsewhere, and without MSYS2 there, the tools come from PATH TARGET = 'vulkan1.1' -STAGES = {'.comp': 'compute', '.vert': 'vertex', '.frag': 'fragment'} +STAGES = {'.comp': 'compute', '.vert': 'vertex', '.frag': 'fragment', '.mesh': 'mesh', '.task': 'task'} +MESH_TARGET = 'vulkan1.2' # mesh and task shaders need SPIR-V 1.4 or later ASM = '.spvasm' ASM_TARGET = 'vulkan1.3' # header name -> the script beside this one that assembles it; the script's HEADER is the header's text @@ -138,7 +140,7 @@ def words_of(src): else: # "// build_shaders: keep functions" in a source: no optimisation, so its functions stay functions (the HDR # tonemap, which runtime/vk_hdr.c splices into the game's shader by function) - target = TARGET + target = MESH_TARGET if src.suffix in ('.mesh', '.task') else TARGET opt = [] if '// build_shaders: keep functions' in src.read_text(encoding='utf-8') else ['-O'] subprocess.run([tool('glslc'), *opt, f'--target-env={target}', f'-fshader-stage={STAGES[src.suffix]}', '-o', str(spv), str(src)], check=True)