Merge the mesh shader fallback: task and mesh programs drawn through compute passes and an ordinary draw where the card has no mesh shaders.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
chrg6
2026-11-11 06:01:39 -05:00
25 changed files with 1995 additions and 23 deletions

View File

@@ -133,6 +133,28 @@ keeps per-core data without locks and relies on that, so the host models it (`ru
ring of Vulkan `{x, y, z}` records filled on the GPU. The copy cannot be recorded inside a pass, so the first such
draw of a pass fills the records of the mesh draws that follow it too, and the pass is not cut at each of them
(`runtime/vk_exec_draw.c` `mesh_ring_entry`).
- Mesh fallback (`mesh_fallback`: `auto` on a device without the extension, `1` on any device for testing): the task
and mesh stages run as compute dispatches and the result is drawn as ordinary triangles.
- The modules come from the translations the shader cache already holds, not from the translator
(`runtime/mesh_fb.c`): a translated task or mesh module is the program as a compute function over shared memory
plus an entry function that hands that memory to the mesh pipeline, and the fallback replaces the entry function
by one that copies the memory into a storage buffer. The translator, its identity and every cache key stay as
they are, so a device with mesh shaders keeps its shaders and never sees these modules, and a device without
needs no other cache or prepared shader list.
- A draw is: its record (the workgroup count, copied on the GPU when the game's record holds it); the task
dispatch, which writes each task workgroup's payload into the task buffer; a scan (`shaders/meshfb_scan.comp`)
that lists the mesh workgroups the tasks started, in task order, and writes their number into the record; the
mesh dispatch, which writes each workgroup's vertex lines and primitive bytes into its slot of the mesh buffer;
and an instanced indirect draw, one instance per slot, whose generated vertex stage looks the vertex of each
primitive corner up in the slot (`runtime/vk_exec_draw.c` `mfb_draw`, `runtime/vk_mesh_fb.c`).
- Each step reads what the one before wrote, so full barriers stand between them and every such draw ends its pass:
a village frame's 45 mesh draws are 45 passes cut, 90 to 135 dispatches and about 270 barriers more than with mesh
shaders. The buffers (`mesh_fallback_mb`, 128 MiB) are reused by every draw, so they hold the largest draw, not
a frame; a draw that emits more loses what does not fit, and the log says how much it needed.
- A program's modules and compute pipelines are made on a thread of their own at its first draw, which is skipped
until they are ready; they are not recorded for the next start, where the pipeline cache makes them quick.
Not carried: the far shadow cascades' coarser shading rate, a draw's first workgroup other than 0, and more
workgroups in one draw than one compute dispatch may have.
- Pipelines: `VK_EXT_graphics_pipeline_library` where the driver has it: stage libraries compiled ahead, a fast link
at a new state, an optimised pipeline replacing it from a background thread. A draw whose pipeline is not ready is
skipped when its pass is one the game renders frame after frame: one of its targets was rendered in the last few

View File

@@ -484,6 +484,12 @@ Runtime probes, each a mode of `nx2navi.exe` (the list is at the top of `runtime
part of `--probe-vkexec` translate shaders of the census below: they look for `build/<ver>/shaders/jobs.json` of the
exe's own version first, then of the other one.
`--probe-vkmem` also checks CPU-backed texture uploads and rewrites, GPU-written texture aliases, and memory write watches.
`--probe-meshfb` (no game files) draws mesh programs of the runtime's own, with and without a task stage and with
direct and GPU-written workgroup counts, through `VK_EXT_mesh_shader` and through the mesh fallback, and compares the
pictures; on a device without mesh shaders only the fallback draws. A change to `runtime/mesh_fb.c`,
`runtime/vk_mesh_fb.c` or the fallback's part of `runtime/vk_exec_draw.c` passes it. With
`NX2NAVI_PROBE_MESHFB_DIR=<folder>` it converts the game's own task and mesh translations instead, for `spirv-val`
(the top of `runtime/mesh_fb_probe.c` says what the folder holds).
A change to the texture detail (`runtime/vk_texture.c` "texture detail": `detail_skip`, `upload`, `astc_bl_regions`)
passes it three times, with `NX2NAVI_TEXTURE_DETAIL` set to `full`, `reduced` and `low`: each time ASTC textures are
uploaded and rewritten, and every level and layer of their images must hold the game's level it stands for.

View File

@@ -876,6 +876,14 @@
# first one, so their draw pass is not ended and begun again at each of them (about 40 passes fewer a frame in a
# village). 0: each such draw ends its pass and prepares its own count, as before.
# mesh_ahead = 1
# Graphics cards without mesh shaders (GTX 10 series, RX 500 and 5000 series, many integrated ones): the game's mesh
# draws, most of the world, are computed in compute passes and drawn as ordinary triangles. auto (default) does that
# only on such a card; 1 does it on every card (for testing: it is slower than mesh shaders); 0 never, and such a
# card then draws no world.
# mesh_fallback = auto
# Video memory in MB for what those compute passes hand to the draws. A draw that needs more loses the geometry that
# does not fit, and the log says so.
# mesh_fallback_mb = 128
# Windows, 1 (experimental): the game's heap is committed a 64 KiB run at a time where it is first used, instead of
# all of it when the game sizes it. Windows counts committed memory against RAM plus the page file whether it is used
# or not, and in Castle Town after Hyrule Field about 3.4 of the heap's 6.8 GiB had never been used. 0: committed up

View File

@@ -36,7 +36,7 @@ static int save_ok = 1; // the last cfg_save_to wrote (o
static const char *const known[] = {
"CORES", "TM_DISPATCH", "NVN2_CS_TABLE", "SKIP_SHADERS", "MOVIE_DEBAND", "MOVIE_FPS", "MOVIE_SKIP", "MOVIE_OVERLAY", "MOVIES", "NPLN", "NVN2_PROGRAMS", "NVN_TRAP", "OCARINA_REALTIME", "ONESHOT_BUDGET", "ONESHOT_WAIT_MS", "OCARINA_STEP", "PHYSICS_STEPS", "PICKUP_COUNT", "STEP_STEADY", "VI_HZ", "VI_VRR", "TIME_DUMP", "SHADER_DUMP", "SHOT_AFTER", "SHOT_DIR", "SHOT_EVERY", // Switch 2 port: movie skip (romfs.c), NVN2 proc traps (nvn_procs.c)
"AB", "AMIIBO_DIR", "ANISO", "APP_POOL", "ASPECT", "ASPECT_PATCH", "ASSUME_FOCUS", "ASTC", "ASTC_BC", "ASTC_TIME", "ASYNC_PREEMPT", "AUDIO", "AUDIO_CHANNELS", "AUDIO_DEVICE", "AUDIO_GAME_PRIO", "AUDIO_OFF", "AUDIO_PREFILL_MS", "AUDIO_STATS",
"BARRIER_GAME", "MESH_AHEAD",
"BARRIER_GAME", "MESH_AHEAD", "MESH_FALLBACK", "MESH_FALLBACK_MB", "PROBE_MESHFB_DIR",
"BACKGROUND_WINDOW", "BIND_A", "BIND_B", "BIND_DOWN", "BIND_L", "BIND_LCLICK", "BIND_LEFT", "BIND_LSTICK_DOWN", "BIND_LSTICK_LEFT",
"BIND_LSTICK_RIGHT", "BIND_LSTICK_UP", "BIND_MINUS",
"BIND_PLUS", "BIND_R", "BIND_RCLICK", "BIND_RIGHT", "BIND_RSTICK_DOWN", "BIND_RSTICK_LEFT", "BIND_RSTICK_RIGHT", "BIND_RSTICK_UP",
@@ -141,7 +141,7 @@ static const struct { const char *key, *section; } sections[] = {
{"VK_DYN_VERTEX", "performance"}, {"VK_DYN_A2C", "performance"}, {"VK_SUBGROUP_PIN", "performance"}, {"VK_RECONVERGE", "performance"},
{"VK_NAN_PRESERVE", "performance"}, {"VK_NAN_BITS", "performance"},
{"TRACE", "diagnostics"}, {"FLIGHT", "diagnostics"}, {"PASS_CRUMBS", "diagnostics"}, {"FLIGHT_SPIKE_MS", "diagnostics"},
{"WB_FENCE_LIMIT", "diagnostics"}, {"WB_MERGE", "diagnostics"}, {"WB_EXACT", "diagnostics"}, {"MEM_CENSUS", "diagnostics"}, {"WB_NO_REUP", "diagnostics"}, {"WB_PROBE", "diagnostics"}, {"WB_SPLIT", "performance"}, {"SHADOW_BULK_WRITE", "performance"}, {"WB_LAZY", "performance"}, {"HEAP_LAZY", "performance"}, {"BARRIER_GAME", "performance"}, {"MESH_AHEAD", "performance"}, {"WB_LAZY_TOUCH", "diagnostics"}, {"WB_LAZY_CHUNK", "performance"}, {"FLUSH_UPLOAD", "diagnostics"}, {"GPA_TWIN_MAX_KB", "diagnostics"},
{"WB_FENCE_LIMIT", "diagnostics"}, {"WB_MERGE", "diagnostics"}, {"WB_EXACT", "diagnostics"}, {"MEM_CENSUS", "diagnostics"}, {"WB_NO_REUP", "diagnostics"}, {"WB_PROBE", "diagnostics"}, {"WB_SPLIT", "performance"}, {"SHADOW_BULK_WRITE", "performance"}, {"WB_LAZY", "performance"}, {"HEAP_LAZY", "performance"}, {"BARRIER_GAME", "performance"}, {"MESH_AHEAD", "performance"}, {"MESH_FALLBACK", "performance"}, {"MESH_FALLBACK_MB", "performance"}, {"WB_LAZY_TOUCH", "diagnostics"}, {"WB_LAZY_CHUNK", "performance"}, {"FLUSH_UPLOAD", "diagnostics"}, {"GPA_TWIN_MAX_KB", "diagnostics"},
{"WB_MERGE_MAX", "diagnostics"}, {"MV_SURVEY", "diagnostics"}, {"TEXUP_LOG_MS", "diagnostics"}, {"VA_REPORT", "diagnostics"},
{"VMA_BLOCK_MB", "diagnostics"}, {"ASTC_BC_PROBE", "diagnostics"}, {"DEVICE_LOST_AT", "diagnostics"},
};
@@ -269,6 +269,9 @@ static const CfgMeta metas[] = {
{"PIPE_FRAME_WAIT_MS", "50", "0..500", RESTART}, // vk_exec_draw.c frame_wait
{"PIPE_SKIP_STEADY", "1", "0|1", RESTART}, // vk_exec.c steady_on
{"PREPARE_PIPELINES", "mesh", "0|mesh|all", RESTART}, // vk_pipeline.c seed_pipelines
// vk_device.c: task and mesh programs through compute passes (vk_exec_draw.c "mesh fallback"); auto: on a device
// without mesh shaders. The megabytes are the buffers the passes write (vk_mesh_fb.c).
{"MESH_FALLBACK", "auto", "auto|0|1", RESTART}, {"MESH_FALLBACK_MB", "128", "16..4096", RESTART},
{"PIPE_AHEAD", "1", "0|1", RESTART}, // vk_exec.c vk_exec_ahead, vk_pipeline.c vk_pipeline_hint
{"GPL_FEEDBACK", "1", "0|1", RESTART}, // vk_pipeline.c gpl_feedback
{"PIPE_FAST_WORKERS", "1", "0|1", RESTART}, // vk_pipeline.c lib_worker, fast_take

View File

@@ -100,6 +100,7 @@ size_t gfx_shader_warm_list(GfxWarmEnt **out); // malloc'd; recorde
const GfxShader *gfx_shader_load_private(const uint8_t iid[32], int *owned);
void gfx_shader_release_private(const GfxShader *s);
const uint32_t *gfx_shader_spirv(const GfxShader *s, uint64_t *words);
const char *gfx_shader_res_json(const GfxShader *s); // the translator's .res.json text
int gfx_shader_write_code(const GfxShader *s, const char *path); // the guest program, from the stored inputs
// A shader of the runtime's own (SPIR-V and a .res.json text as the translator writes them), outside the cache and
// never freed; its key is the sha256 of both. NULL when the SPIR-V is malformed.

View File

@@ -2424,6 +2424,8 @@ const uint32_t *gfx_shader_spirv(const GfxShader *s, uint64_t *words) {
return s->spirv;
}
const char *gfx_shader_res_json(const GfxShader *s) { return s->reflection; }
// The guest program a shader was translated from, as its stored inputs hold it (the whole program: NVN2 code from its
// first byte), written to path for a disassembler (NX2NAVI_DUMP's sh_<hash>.code), and next to it <path>.job, the
// translator request that translates it (one line for nx2navi_xlate --serve; its "out" names the shader cache, and

View File

@@ -436,6 +436,7 @@ int probe_window(void); // gfx_window_probe.c
int probe_shader(void); // gfx_shader.c
int probe_census_dir(char *dir, size_t n); // gfx_shader_probe.c: build/<version>/shaders for the probes
int probe_vkmem(void), probe_vkexec(void), probe_astc(void); // tex_probe.c, vk_exec_probe.c, astc_probe.c
int probe_meshfb(void); // mesh_fb_probe.c
int probe_zstd(void); // ov_zstd.c
int probe_gamepad(void); // input.c
int probe_movie(void); // movie_probe.c: the host movie decoders on one RomFS movie

View File

@@ -26,6 +26,8 @@
// --probe-ffx-cache one FSR 3.1 upscaling context and where its pipelines came from (the pipeline caches kept
// across runs); skips without amd_fidelityfx_vk.dll, runs no game
// --probe-astc the compute ASTC decoder against astc_decode_ldr; needs no game files
// --probe-meshfb the mesh shader fallback: mesh programs drawn through VK_EXT_mesh_shader and through compute
// passes, the pictures compared; needs no game files (mesh_fb_probe.c)
// --probe-zstd host zstd streams (ov_zstd.c) on frames made here; needs no game files
// --probe-input gamepad polls on a virtual gamepad, also while SDL's update takes long; needs no game files
// --probe-movie the host movie decoders on one RomFS movie (VP9 movie::Decoder, AV1 movie::VideoDecoder); skips without game data
@@ -535,7 +537,7 @@ static int usage(const char *argv0) {
fprintf(stderr, "usage: %s [--exefs DIR] [--romfs DIR] [--profile DIR] [--audio-device VALUE] [--monitor VALUE] [--list-audio-devices | --list-monitors | --selftest | --probe-loader | "
"--probe-cfg | --probe-dpi | --probe-mods | --probe-kernel | --probe-ipc | --probe-services | --probe-fs | --probe-gfx-abi | "
"--probe-nvn | --probe-nvn-tex | --probe-window | --probe-shader | --probe-vkmem | "
"--probe-vkexec | --probe-astc | --probe-zstd | --probe-input | --probe-movie | --probe-crash-call | --probe-crash-null | --probe-discord | --probe-coop | "
"--probe-vkexec | --probe-meshfb | --probe-astc | --probe-zstd | --probe-input | --probe-movie | --probe-crash-call | --probe-crash-null | --probe-discord | --probe-coop | "
"--prepare-shaders | --warm-shaders | --prepare-pipelines | --probe-prewarm | --probe-buildgate | --probe-ffx-cache | --probe-memneed | --probe-devlost | --boot |--seconds S [--shot PNG]]\n", argv0);
return 2;
}
@@ -684,6 +686,7 @@ int main(int argc, char **argv) {
if (!strcmp(mode, "--probe-shader")) return probe_shader();
if (!strcmp(mode, "--probe-vkmem")) return probe_vkmem();
if (!strcmp(mode, "--probe-vkexec")) return probe_vkexec();
if (!strcmp(mode, "--probe-meshfb")) return probe_meshfb();
if (!strcmp(mode, "--probe-astc")) return probe_astc();
if (!strcmp(mode, "--probe-zstd")) return probe_zstd();
if (!strcmp(mode, "--probe-input")) return probe_gamepad();

638
runtime/mesh_fb.c Normal file
View File

@@ -0,0 +1,638 @@
// The mesh shader fallback's modules, for a GPU without VK_EXT_mesh_shader (vk_exec_draw.c "mesh fallback" draws with
// them; docs/architecture.md, "Mesh fallback").
//
// The translator turns a task or mesh program into a compute module whose shared memory is the program's stage buffer
// and wraps it in a TaskEXT or MeshEXT entry function (tools/xlate/MeshSpirv.cs). The wrapping is undone here and
// replaced: the translator's main function stays as it is, the entry function that hands the stage buffer to the mesh
// pipeline goes, and a compute entry function copies the stage buffer into a storage buffer instead.
// task: main; the payload area into the task buffer at the workgroup's place.
// mesh: [the payload from the task buffer into the payload area] main; the vertex lines and the primitive area into
// the workgroup's slot of the mesh buffer.
// draw: a vertex module made here reads a slot per instance: vertex index / indices per primitive is the
// primitive, its index byte names the vertex, and the vertex's lines are the outputs.
// The module is made from the translation the shader cache already holds, so the translator and with it every cache
// key stay as they are: a card with mesh shaders never sees these modules, and one without needs no other cache.
//
// Mesh workgroups after a task stage. A task workgroup starts up to `threads` mesh workgroups, and how many is known
// only on the GPU. A pass between the two dispatches (shaders/meshfb_scan.comp) lists them in task order: entry g of
// the list names the task workgroup and the index the mesh program sees as its workgroup id. That id comes from the
// WorkgroupId built-in in the translated code; here the built-in counts list entries, so main reads a private
// variable the entry function fills from the list.
//
// Opcode and enum numbers are the SPIR-V 1.6 specification's and SPV_EXT_mesh_shader's.
#include "mesh_fb.h"
#include <stdarg.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
enum {
OpName = 5, OpExtension = 10, OpMemoryModel = 14, OpEntryPoint = 15, OpExecutionMode = 16, OpCapability = 17,
OpTypeVoid = 19, OpTypeBool = 20, OpTypeInt = 21, OpTypeFloat = 22, OpTypeVector = 23, OpTypeArray = 28,
OpTypeRuntimeArray = 29, OpTypeStruct = 30, OpTypePointer = 32, OpTypeFunction = 33, OpConstant = 43,
OpConstantComposite = 44, OpConstantNull = 46, OpFunction = 54, OpFunctionEnd = 56, OpFunctionCall = 57,
OpVariable = 59, OpLoad = 61, OpStore = 62, OpCopyMemory = 63, OpAccessChain = 65, OpInBoundsAccessChain = 66,
OpPtrAccessChain = 67, OpArrayLength = 68, OpDecorate = 71, OpMemberDecorate = 72, OpCompositeConstruct = 80,
OpCompositeExtract = 81, OpCopyObject = 83, OpBitcast = 124, OpIAdd = 128, OpISub = 130, OpIMul = 132, OpUDiv = 134,
OpSelect = 169, OpINotEqual = 171, OpULessThan = 176, OpULessThanEqual = 178, OpShiftRightLogical = 194,
OpBitwiseAnd = 199, OpControlBarrier = 224, OpPhi = 245, OpLoopMerge = 246, OpSelectionMerge = 247, OpLabel = 248,
OpBranch = 249, OpBranchConditional = 250, OpReturn = 253, OpDecorateId = 332, OpDecorateString = 5632,
};
enum { ScInput = 1, ScOutput = 3, ScWorkgroup = 4, ScPrivate = 6, ScFunction = 7, ScStorageBuffer = 12, ScTaskPayload = 5402 };
enum { DecBlock = 2, DecArrayStride = 6, DecBuiltIn = 11, DecInvariant = 18, DecNonWritable = 24, DecLocation = 30,
DecBinding = 33, DecDescriptorSet = 34, DecOffset = 35 };
enum { BiPosition = 0, BiPointSize = 1, BiWorkgroupId = 26, BiLocalInvocationIndex = 29, BiVertexIndex = 42,
BiInstanceIndex = 43 };
enum { CapShader = 1, CapMeshShading = 5283, ModelVertex = 0, ModelGLCompute = 5, ModelTask = 5364, ModelMesh = 5365,
ModeLocalSize = 17 };
// ---- the program's layout
static void fail(char *err, size_t cap, const char *fmt, ...) {
if (!err || !cap) return;
va_list ap;
va_start(ap, fmt);
vsnprintf(err, cap, fmt, ap);
va_end(ap);
}
int meshfb_info(const uint8_t *control, const uint8_t *header, int stage, MeshFbInfo *m, char *err, size_t cap) {
uint32_t w58, w5c, w60;
memset(m, 0, sizeof *m);
memcpy(&w58, control + 0x58, 4), memcpy(&w5c, control + 0x5c, 4), memcpy(&w60, control + 0x60, 4);
m->stage = stage;
m->threads = w58 & 0xffff ? w58 & 0xffff : 1;
m->payload_bytes = (w58 >> 24) * 256;
m->maxv = w60 & 0xffff, m->maxp = w60 >> 16;
uint32_t topo = w5c >> 24;
m->ipp = topo == 1 ? 1 : topo == 2 ? 2 : 3;
m->has_task = stage == 7 && (w5c >> 16 & 0xff) != 0;
// The payload as the translator sizes it (tools/xlate/Program.cs Translate): a size of 0 is one 256-byte unit, and
// the header of 0x20 bytes comes on top.
if (stage == 6 || m->has_task) m->payload_bytes = (m->payload_bytes ? m->payload_bytes : 256) + 0x20;
if (stage == 6) {
m->pay = 0, m->shared_bytes = m->payload_bytes;
return 1;
}
if (stage != 7) return fail(err, cap, "stage %d is neither task nor mesh", stage), 0;
if (!m->maxv || !m->maxp || m->maxv > 256) return fail(err, cap, "limits of %u vertices and %u primitives", m->maxv, m->maxp), 0;
// The output map of the program header: system values B at byte 0x35, one bit per component from 0x60; generic
// vectors from byte 0x36, a nibble each. System values A and C never appear in the game's mesh programs.
if (header[0x34] || header[0x48] || header[0x49]) return fail(err, cap, "output map with system values A or C"), 0;
for (int b = 0; b < 8; b++)
if (header[0x35] >> b & 1) m->lines[m->nlines++] = (uint16_t)(0x60 + 4 * b);
for (int g = 0; g < 32; g++)
for (int c = 0; c < 4; c++)
if (header[0x36 + g / 2] >> (4 * (g & 1) + c) & 1) m->lines[m->nlines++] = (uint16_t)(0x80 + 16 * g + 4 * c);
uint32_t groups = (m->maxv + 31) / 32;
m->attr = 0;
m->map = (m->attr + groups * m->nlines * 128 + 15) & ~15u;
m->pay = (m->map + 4 + m->ipp * m->maxp + 15) & ~15u;
m->shared_bytes = m->pay + (m->has_task ? m->payload_bytes : 0);
return 1;
}
// ---- a growing list of SPIR-V words
typedef struct { uint32_t *w; size_t n, cap; int oom; } Wb;
static void wb_put(Wb *b, const uint32_t *w, size_t n) {
if (b->n + n > b->cap) {
size_t cap = (b->n + n) * 2 + 256;
uint32_t *g = realloc(b->w, cap * 4);
if (!g) { b->oom = 1; return; }
b->w = g, b->cap = cap;
}
memcpy(b->w + b->n, w, n * 4);
b->n += n;
}
static void wb_ins(Wb *b, uint32_t op, const uint32_t *operands, size_t n) {
uint32_t head = (uint32_t)((n + 1) << 16) | op;
wb_put(b, &head, 1);
wb_put(b, operands, n);
}
#define INS(b, op, ...) wb_ins(b, op, (const uint32_t[]){__VA_ARGS__}, sizeof((const uint32_t[]){__VA_ARGS__}) / 4)
// What a module under construction adds: decorations, then types, constants and variables, then one function.
typedef struct {
uint32_t bound;
Wb deco, glob, fn;
uint32_t t_void, t_bool, t_uint, t_fnvoid, t_v3uint; // existing or new
uint32_t p_fn_uint, p_sb_uint;
struct { uint32_t v, id; } k[96];
int nk;
} Cx;
static uint32_t nid(Cx *c) { return c->bound++; }
static uint32_t K(Cx *c, uint32_t v) { // a 32-bit unsigned constant
for (int i = 0; i < c->nk; i++)
if (c->k[i].v == v) return c->k[i].id;
uint32_t id = nid(c);
INS(&c->glob, OpConstant, c->t_uint, id, v);
if (c->nk < 96) c->k[c->nk].v = v, c->k[c->nk].id = id, c->nk++;
return id;
}
static uint32_t E_(Cx *c, uint32_t op, uint32_t type, const uint32_t *operands, size_t n) { // an instruction with a result
uint32_t id = nid(c), head[3] = {(uint32_t)((n + 3) << 16) | op, type, id};
wb_put(&c->fn, head, 3);
wb_put(&c->fn, operands, n);
return id;
}
#define E(c, op, type, ...) E_(c, op, type, (const uint32_t[]){__VA_ARGS__}, sizeof((const uint32_t[]){__VA_ARGS__}) / 4)
#define DO(c, op, ...) INS(&(c)->fn, op, __VA_ARGS__)
static uint32_t add(Cx *c, uint32_t a, uint32_t b) { return E(c, OpIAdd, c->t_uint, a, b); }
static uint32_t mul(Cx *c, uint32_t a, uint32_t b) { return E(c, OpIMul, c->t_uint, a, b); }
static uint32_t umin(Cx *c, uint32_t a, uint32_t b) {
return E(c, OpSelect, c->t_uint, E(c, OpULessThan, c->t_bool, a, b), a, b);
}
// if (cond) { ...the instructions emitted until if_end }
static uint32_t if_begin(Cx *c, uint32_t cond) {
uint32_t then = nid(c), merge = nid(c);
DO(c, OpSelectionMerge, merge, 0);
DO(c, OpBranchConditional, cond, then, merge);
DO(c, OpLabel, then);
return merge;
}
static void if_end(Cx *c, uint32_t merge) {
DO(c, OpBranch, merge);
DO(c, OpLabel, merge);
}
// for (i = start; i < limit; i += step) { ...the instructions emitted until loop_end }; counter: a Function variable
typedef struct { uint32_t head, cont, merge, i, counter, step; } Loop;
static Loop loop_begin(Cx *c, uint32_t counter, uint32_t start, uint32_t limit, uint32_t step) {
Loop l = {nid(c), nid(c), nid(c), 0, counter, step};
uint32_t cond = nid(c), body = nid(c);
DO(c, OpStore, counter, start);
DO(c, OpBranch, l.head);
DO(c, OpLabel, l.head);
DO(c, OpLoopMerge, l.merge, l.cont, 0);
DO(c, OpBranch, cond);
DO(c, OpLabel, cond);
l.i = E(c, OpLoad, c->t_uint, counter);
DO(c, OpBranchConditional, E(c, OpULessThan, c->t_bool, l.i, limit), body, l.merge);
DO(c, OpLabel, body);
return l;
}
static void loop_end(Cx *c, const Loop *l) {
DO(c, OpBranch, l->cont);
DO(c, OpLabel, l->cont);
DO(c, OpStore, l->counter, add(c, l->i, l->step));
DO(c, OpBranch, l->head);
DO(c, OpLabel, l->merge);
}
// A storage buffer of 32-bit words at (set 0, binding): its variable. readonly: the module only reads it.
static uint32_t word_buffer(Cx *c, uint32_t binding, int readonly) {
uint32_t arr = nid(c), st = nid(c), ptr = nid(c), var = nid(c);
INS(&c->glob, OpTypeRuntimeArray, arr, c->t_uint);
INS(&c->glob, OpTypeStruct, st, arr);
INS(&c->glob, OpTypePointer, ptr, ScStorageBuffer, st);
INS(&c->glob, OpVariable, ptr, var, ScStorageBuffer);
INS(&c->deco, OpDecorate, arr, DecArrayStride, 4);
INS(&c->deco, OpDecorate, st, DecBlock);
INS(&c->deco, OpMemberDecorate, st, 0, DecOffset, 0);
if (readonly) INS(&c->deco, OpMemberDecorate, st, 0, DecNonWritable);
INS(&c->deco, OpDecorate, var, DecDescriptorSet, 0);
INS(&c->deco, OpDecorate, var, DecBinding, binding);
return var;
}
static uint32_t buf_ptr(Cx *c, uint32_t var, uint32_t index) { return E(c, OpAccessChain, c->p_sb_uint, var, K(c, 0), index); }
static uint32_t buf_load(Cx *c, uint32_t var, uint32_t index) { return E(c, OpLoad, c->t_uint, buf_ptr(c, var, index)); }
// ---- a translated task or mesh module to its compute module
typedef struct { uint32_t at, op, len; } In; // an instruction: its first word's index, opcode, word count
static int id_in(const uint32_t *set, int n, uint32_t id) {
for (int i = 0; i < n; i++)
if (set[i] == id) return 1;
return 0;
}
static int is_decl(uint32_t op) { // a type, constant or global variable declaration
return (op >= OpTypeVoid && op <= 39) || (op >= 41 && op <= 52) || op == OpVariable || op == 1;
}
static uint32_t *convert(const uint32_t *w, uint64_t words, const MeshFbInfo *m, uint32_t binding, uint32_t task_words,
uint64_t *out_words, char *err, size_t cap) {
In *in = NULL;
uint32_t *removed = NULL, *gvars = NULL;
Wb out = {0};
Cx c = {0};
#define FAIL(...) do { fail(err, cap, __VA_ARGS__); goto bad; } while (0)
if (words < 6 || w[0] != 0x07230203) return fail(err, cap, "not a SPIR-V module"), NULL;
if (w[1] < 0x00010400) return fail(err, cap, "SPIR-V %u.%u, 1.4 expected", w[1] >> 16 & 0xff, w[1] >> 8 & 0xff), NULL;
size_t n = 0, ncap = 0;
for (uint64_t p = 5; p < words;) {
uint32_t len = w[p] >> 16, op = w[p] & 0xffff;
if (!len || p + len > words) return fail(err, cap, "malformed SPIR-V"), NULL;
if (n == ncap) {
ncap = ncap ? ncap * 2 : 4096;
In *g = realloc(in, ncap * sizeof *in);
if (!g) FAIL("out of memory");
in = g;
}
in[n++] = (In){(uint32_t)p, op, len};
p += len;
}
c.bound = w[3];
int mesh = m->stage == 7, n_removed = 0, n_gvars = 0, n_entry = 0, n_shared = 0;
removed = malloc((n + 1) * 4), gvars = malloc((n + 16) * 4);
if (!removed || !gvars) FAIL("out of memory");
uint32_t old_entry = 0, model = 0, shared_var = 0, shared_type = 0, wg_var = 0, lii_var = 0, first_fn = 0, first_decl = 0;
// pass 1: what the module has
for (size_t i = 0; i < n; i++) {
const uint32_t *a = w + in[i].at + 1;
uint32_t op = in[i].op, len = in[i].len;
if (op == OpEntryPoint) n_entry++, model = a[0], old_entry = a[1];
else if (op == OpTypeVoid) c.t_void = a[0];
else if (op == OpTypeBool) c.t_bool = a[0];
else if (op == OpTypeInt && a[1] == 32 && a[2] == 0) c.t_uint = a[0];
else if (op == OpTypeVector && a[1] == c.t_uint && c.t_uint && a[2] == 3) c.t_v3uint = a[0];
else if (op == OpTypeFunction && len == 3 && a[1] == c.t_void && c.t_void) c.t_fnvoid = a[0];
else if (op == OpTypePointer && (a[1] == ScOutput || a[1] == ScTaskPayload)) removed[n_removed++] = a[0];
else if (op == OpVariable && !first_fn) {
if (a[2] == ScOutput || a[2] == ScTaskPayload) removed[n_removed++] = a[1];
else gvars[n_gvars++] = a[1];
if (a[2] == ScWorkgroup) n_shared++, shared_var = a[1], shared_type = a[0];
} else if (op == OpFunction && !first_fn) first_fn = (uint32_t)i;
if (!first_decl && !first_fn && is_decl(op)) first_decl = (uint32_t)i;
}
if (n_entry != 1 || model != (uint32_t)(mesh ? ModelMesh : ModelTask)) FAIL("the entry point is not a %s one", mesh ? "mesh" : "task");
if (n_shared != 1) FAIL("%d workgroup variables (one shared memory array expected)", n_shared);
if (!c.t_void || !c.t_bool || !c.t_uint || !c.t_fnvoid || !first_fn || !first_decl) FAIL("a basic type or the functions are missing");
{ // the shared memory array holds the layout this module is asked to copy
uint32_t arr = 0, len_id = 0, len = 0;
for (size_t i = 0; i < n; i++) {
const uint32_t *a = w + in[i].at + 1;
if (in[i].op == OpTypePointer && a[0] == shared_type) arr = a[2];
}
for (size_t i = 0; i < n; i++) {
const uint32_t *a = w + in[i].at + 1;
if (in[i].op == OpTypeArray && a[0] == arr && a[1] == c.t_uint) len_id = a[2];
}
for (size_t i = 0; i < n; i++) {
const uint32_t *a = w + in[i].at + 1;
if (in[i].op == OpConstant && a[1] == len_id && len_id) len = a[2];
}
if (len * 4 < m->shared_bytes) FAIL("shared memory of %u bytes, the program's layout needs %u", len * 4, m->shared_bytes);
}
for (size_t i = 0; i < n; i++) {
const uint32_t *a = w + in[i].at + 1;
if (in[i].op != OpDecorate || in[i].len < 4 || a[1] != DecBuiltIn || id_in(removed, n_removed, a[0])) continue;
if (a[2] == BiWorkgroupId) wg_var = a[0];
if (a[2] == BiLocalInvocationIndex) lii_var = a[0];
}
// the old entry function, and the function it calls: the translator's main
size_t e0 = 0, e1 = 0;
uint32_t main_fn = 0, n_calls = 0;
for (size_t i = first_fn; i < n; i++)
if (in[i].op == OpFunction && w[in[i].at + 2] == old_entry) {
e0 = i;
for (e1 = i; e1 < n && in[e1].op != OpFunctionEnd; e1++)
if (in[e1].op == OpFunctionCall) n_calls++, main_fn = w[in[e1].at + 3];
break;
}
if (!e0 || e1 >= n || n_calls != 1) FAIL("the entry function calls %u functions (one expected)", n_calls);
// ---- new types, variables and the entry function
if (!c.t_v3uint) c.t_v3uint = nid(&c), INS(&c.glob, OpTypeVector, c.t_v3uint, c.t_uint, 3);
c.p_fn_uint = nid(&c), INS(&c.glob, OpTypePointer, c.p_fn_uint, ScFunction, c.t_uint);
c.p_sb_uint = nid(&c), INS(&c.glob, OpTypePointer, c.p_sb_uint, ScStorageBuffer, c.t_uint);
uint32_t p_shared = nid(&c);
INS(&c.glob, OpTypePointer, p_shared, ScWorkgroup, c.t_uint);
if (!lii_var) {
uint32_t pt = nid(&c);
lii_var = nid(&c);
INS(&c.glob, OpTypePointer, pt, ScInput, c.t_uint);
INS(&c.glob, OpVariable, pt, lii_var, ScInput);
INS(&c.deco, OpDecorate, lii_var, DecBuiltIn, BiLocalInvocationIndex);
gvars[n_gvars++] = lii_var;
}
if (!wg_var) {
uint32_t pt = nid(&c);
wg_var = nid(&c);
INS(&c.glob, OpTypePointer, pt, ScInput, c.t_v3uint);
INS(&c.glob, OpVariable, pt, wg_var, ScInput);
INS(&c.deco, OpDecorate, wg_var, DecBuiltIn, BiWorkgroupId);
gvars[n_gvars++] = wg_var;
}
uint32_t fake_wg = 0, p_priv_uint = 0;
if (mesh) { // the workgroup id main sees (see the top)
uint32_t pt = nid(&c);
fake_wg = nid(&c), p_priv_uint = nid(&c);
INS(&c.glob, OpTypePointer, pt, ScPrivate, c.t_v3uint);
INS(&c.glob, OpTypePointer, p_priv_uint, ScPrivate, c.t_uint);
INS(&c.glob, OpVariable, pt, fake_wg, ScPrivate);
gvars[n_gvars++] = fake_wg;
}
uint32_t task_buf = 0, mesh_buf = 0, list_buf = 0;
if (!mesh || m->has_task) task_buf = word_buffer(&c, binding, mesh), gvars[n_gvars++] = task_buf;
if (mesh) mesh_buf = word_buffer(&c, binding + 1, 0), gvars[n_gvars++] = mesh_buf;
if (mesh && m->has_task) list_buf = word_buffer(&c, binding + 2, 1), gvars[n_gvars++] = list_buf;
uint32_t threads = K(&c, m->threads), k0 = K(&c, 0), k1 = K(&c, 1);
uint32_t entry = nid(&c);
DO(&c, OpFunction, c.t_void, entry, 0, c.t_fnvoid);
DO(&c, OpLabel, nid(&c));
uint32_t counter = nid(&c);
DO(&c, OpVariable, c.p_fn_uint, counter, ScFunction);
uint32_t tid = E(&c, OpLoad, c.t_uint, lii_var);
uint32_t g = E(&c, OpCompositeExtract, c.t_uint, E(&c, OpLoad, c.t_v3uint, wg_var), 0);
uint32_t first = E(&c, OpULessThan, c.t_bool, tid, k1); // the workgroup's first thread
#define SHARED(index) E(&c, OpAccessChain, p_shared, shared_var, index)
#define BARRIER() DO(&c, OpControlBarrier, K(&c, 2), K(&c, 2), K(&c, 0x108)) /* Workgroup, AcquireRelease | WorkgroupMemory */
if (!mesh) {
uint32_t pw = m->payload_bytes / 4;
uint32_t at = mul(&c, g, K(&c, pw));
uint32_t room = E(&c, OpULessThanEqual, c.t_bool, add(&c, at, K(&c, pw)), E(&c, OpArrayLength, c.t_uint, task_buf, 0));
// the mesh workgroup count starts at 0: a task that culls everything never writes it
uint32_t mg = if_begin(&c, first);
DO(&c, OpStore, SHARED(K(&c, (m->pay + 4) / 4)), k0);
if_end(&c, mg);
BARRIER();
E(&c, OpFunctionCall, c.t_void, main_fn);
BARRIER();
mg = if_begin(&c, room);
Loop l = loop_begin(&c, counter, tid, K(&c, pw), threads);
DO(&c, OpStore, buf_ptr(&c, task_buf, add(&c, at, l.i)), E(&c, OpLoad, c.t_uint, SHARED(add(&c, l.i, K(&c, m->pay / 4)))));
loop_end(&c, &l);
if_end(&c, mg);
} else {
uint32_t slot_words = meshfb_slot_words(m), count_word = K(&c, m->map / 4);
uint32_t j = g, task_at = 0;
if (m->has_task) {
uint32_t e = buf_load(&c, list_buf, g);
task_at = mul(&c, E(&c, OpBitwiseAnd, c.t_uint, e, K(&c, (1u << MESHFB_LIST_TASK_BITS) - 1)), K(&c, task_words));
j = E(&c, OpShiftRightLogical, c.t_uint, e, K(&c, MESHFB_LIST_TASK_BITS));
}
DO(&c, OpStore, fake_wg, E(&c, OpCompositeConstruct, c.t_v3uint, j, k0, k0));
uint32_t at = mul(&c, g, K(&c, slot_words));
uint32_t room = E(&c, OpULessThanEqual, c.t_bool, add(&c, at, K(&c, slot_words)), E(&c, OpArrayLength, c.t_uint, mesh_buf, 0));
// the primitive count starts at 0: a mesh program that emits nothing never writes it
uint32_t mg = if_begin(&c, first);
DO(&c, OpStore, SHARED(count_word), k0);
if_end(&c, mg);
if (m->has_task) {
uint32_t pw = m->payload_bytes / 4;
if (pw > task_words) pw = task_words;
Loop l = loop_begin(&c, counter, tid, K(&c, pw), threads);
DO(&c, OpStore, SHARED(add(&c, l.i, K(&c, m->pay / 4))), buf_load(&c, task_buf, add(&c, task_at, l.i)));
loop_end(&c, &l);
}
BARRIER();
E(&c, OpFunctionCall, c.t_void, main_fn);
BARRIER();
uint32_t count = E(&c, OpLoad, c.t_uint, SHARED(count_word));
mg = if_begin(&c, room);
{ // a slot without primitives gets its count alone
uint32_t some = E(&c, OpINotEqual, c.t_bool, count, k0), then = nid(&c), other = nid(&c), merge = nid(&c);
DO(&c, OpSelectionMerge, merge, 0);
DO(&c, OpBranchConditional, some, then, other);
DO(&c, OpLabel, then);
Loop l = loop_begin(&c, counter, tid, K(&c, slot_words), threads);
DO(&c, OpStore, buf_ptr(&c, mesh_buf, add(&c, at, l.i)), E(&c, OpLoad, c.t_uint, SHARED(l.i)));
loop_end(&c, &l);
DO(&c, OpBranch, merge);
DO(&c, OpLabel, other);
uint32_t m2 = if_begin(&c, first);
DO(&c, OpStore, buf_ptr(&c, mesh_buf, add(&c, at, count_word)), k0);
if_end(&c, m2);
DO(&c, OpBranch, merge);
DO(&c, OpLabel, merge);
}
if_end(&c, mg);
}
wb_ins(&c.fn, OpReturn, NULL, 0);
wb_ins(&c.fn, OpFunctionEnd, NULL, 0);
// ---- pass 2: the module again
uint32_t head[5] = {w[0], w[1], w[2], 0, 0};
wb_put(&out, head, 5);
for (size_t i = 0; i < n; i++) {
const uint32_t *a = w + in[i].at + 1;
uint32_t op = in[i].op, len = in[i].len;
if (i == first_decl) wb_put(&out, c.deco.w, c.deco.n);
if (i == first_fn) wb_put(&out, c.glob.w, c.glob.n);
if (i >= e0 && i <= e1) continue; // the old entry function
if (op == OpCapability && a[0] == CapMeshShading) continue;
if (op == OpExtension && len >= 6 && !memcmp(a, "SPV_EXT_mesh_shader", 20)) continue;
if (op == OpExecutionMode && a[1] != ModeLocalSize) continue;
if ((op == OpName || op == OpDecorate || op == OpDecorateId || op == OpDecorateString) && id_in(removed, n_removed, a[0])) continue;
if ((op == OpTypePointer && id_in(removed, n_removed, a[0])) || (op == OpVariable && id_in(removed, n_removed, a[1]))) continue;
if (op == OpEntryPoint) { // every global variable is its interface (SPIR-V 1.4)
uint32_t ep[4] = {(uint32_t)((n_gvars + 5) << 16) | OpEntryPoint, ModelGLCompute, entry, 0x6e69616d /* "main" */};
uint32_t zero = 0;
wb_put(&out, ep, 4), wb_put(&out, &zero, 1);
wb_put(&out, gvars, (size_t)n_gvars);
continue;
}
size_t at = out.n;
wb_put(&out, w + in[i].at, len);
if (out.oom) break;
uint32_t *o = out.w + at + 1;
if (op == OpExecutionMode) o[0] = entry;
if (mesh && i > first_fn) { // main and what it calls read the workgroup id from the private variable
if ((op == OpAccessChain || op == OpInBoundsAccessChain) && a[2] == wg_var) {
uint32_t pointee = 0;
for (size_t t = first_decl; t < first_fn; t++)
if (in[t].op == OpTypePointer && w[in[t].at + 1] == a[0]) pointee = w[in[t].at + 3];
if (pointee != c.t_uint || len != 5) FAIL("the workgroup id is read in a way the fallback does not know");
o[0] = p_priv_uint, o[2] = fake_wg;
} else if (op == OpLoad && a[2] == wg_var) {
o[2] = fake_wg;
} else if ((op == OpStore && a[0] == wg_var) || (op == OpCopyMemory && (a[0] == wg_var || a[1] == wg_var)) ||
((op == OpPtrAccessChain || op == OpCopyObject) && a[2] == wg_var)) {
FAIL("the workgroup id is read in a way the fallback does not know");
} else if (op == OpFunctionCall || op == OpSelect || op == OpPhi) {
for (uint32_t k = 2; k + 1 < len; k++)
if (a[k] == wg_var) FAIL("the workgroup id is passed on as a pointer");
}
}
}
wb_put(&out, c.fn.w, c.fn.n);
if (out.oom || c.deco.oom || c.glob.oom || c.fn.oom) FAIL("out of memory");
out.w[3] = c.bound;
*out_words = out.n;
free(in), free(removed), free(gvars), free(c.deco.w), free(c.glob.w), free(c.fn.w);
return out.w;
bad:
free(in), free(removed), free(gvars), free(c.deco.w), free(c.glob.w), free(c.fn.w), free(out.w);
return NULL;
#undef FAIL
#undef SHARED
#undef BARRIER
}
uint32_t *meshfb_task_cs(const uint32_t *spirv, uint64_t words, const MeshFbInfo *m, uint32_t binding, uint64_t *out_words,
char *err, size_t cap) {
if (m->stage != 6) return fail(err, cap, "not a task program"), NULL;
return convert(spirv, words, m, binding, 0, out_words, err, cap);
}
uint32_t *meshfb_mesh_cs(const uint32_t *spirv, uint64_t words, const MeshFbInfo *m, uint32_t binding,
uint32_t task_payload_words, uint64_t *out_words, char *err, size_t cap) {
if (m->stage != 7) return fail(err, cap, "not a mesh program"), NULL;
if (m->has_task && !task_payload_words) return fail(err, cap, "a mesh program after a task stage without its payload size"), NULL;
return convert(spirv, words, m, binding, task_payload_words, out_words, err, cap);
}
// ---- the vertex module
uint32_t *meshfb_vs(const MeshFbInfo *m, uint32_t out_mask, const uint8_t out_type[32], uint64_t *out_words) {
Cx c = {.bound = 1};
Wb out = {0};
uint32_t iface[40], n_iface = 0;
c.t_void = nid(&c), INS(&c.glob, OpTypeVoid, c.t_void);
c.t_bool = nid(&c), INS(&c.glob, OpTypeBool, c.t_bool);
c.t_uint = nid(&c), INS(&c.glob, OpTypeInt, c.t_uint, 32, 0);
uint32_t t_int = nid(&c), t_float = nid(&c), t_v4[3], p_out[3], t_scalar[3]; // by GFX_TYPE_*: float, sint, uint
INS(&c.glob, OpTypeInt, t_int, 32, 1);
INS(&c.glob, OpTypeFloat, t_float, 32);
t_scalar[0] = t_float, t_scalar[1] = t_int, t_scalar[2] = c.t_uint;
for (int t = 0; t < 3; t++) {
t_v4[t] = nid(&c), INS(&c.glob, OpTypeVector, t_v4[t], t_scalar[t], 4);
p_out[t] = nid(&c), INS(&c.glob, OpTypePointer, p_out[t], ScOutput, t_v4[t]);
}
c.t_fnvoid = nid(&c), INS(&c.glob, OpTypeFunction, c.t_fnvoid, c.t_void);
c.p_sb_uint = nid(&c), INS(&c.glob, OpTypePointer, c.p_sb_uint, ScStorageBuffer, c.t_uint);
uint32_t p_in_int = nid(&c), p_out_float = nid(&c);
INS(&c.glob, OpTypePointer, p_in_int, ScInput, t_int);
INS(&c.glob, OpTypePointer, p_out_float, ScOutput, t_float);
uint32_t vi_var = nid(&c), ii_var = nid(&c), pos_var = nid(&c), size_var = 0, buf, loc_var[32] = {0};
INS(&c.glob, OpVariable, p_in_int, vi_var, ScInput);
INS(&c.glob, OpVariable, p_in_int, ii_var, ScInput);
INS(&c.glob, OpVariable, p_out[0], pos_var, ScOutput);
INS(&c.deco, OpDecorate, vi_var, DecBuiltIn, BiVertexIndex);
INS(&c.deco, OpDecorate, ii_var, DecBuiltIn, BiInstanceIndex);
INS(&c.deco, OpDecorate, pos_var, DecBuiltIn, BiPosition);
INS(&c.deco, OpDecorate, pos_var, DecInvariant);
iface[n_iface++] = vi_var, iface[n_iface++] = ii_var, iface[n_iface++] = pos_var;
if (m->ipp == 1) { // a point list needs its size written
size_var = nid(&c);
INS(&c.glob, OpVariable, p_out_float, size_var, ScOutput);
INS(&c.deco, OpDecorate, size_var, DecBuiltIn, BiPointSize);
iface[n_iface++] = size_var;
}
for (int g = 0; g < 32; g++) {
if (!(out_mask >> g & 1)) continue;
int t = out_type[g] < 3 ? out_type[g] : 0;
loc_var[g] = nid(&c);
INS(&c.glob, OpVariable, p_out[t], loc_var[g], ScOutput);
INS(&c.deco, OpDecorate, loc_var[g], DecLocation, (uint32_t)g);
iface[n_iface++] = loc_var[g];
}
buf = word_buffer(&c, 1, 1);
iface[n_iface++] = buf;
uint32_t f0 = nid(&c), f1 = nid(&c), fout = nid(&c), null_v4[3], away = nid(&c);
INS(&c.glob, OpConstant, t_float, f0, 0);
INS(&c.glob, OpConstant, t_float, f1, 0x3f800000);
INS(&c.glob, OpConstant, t_float, fout, 0xc1800000); // -16: outside the clip volume, for a point of no primitive
for (int t = 0; t < 3; t++) null_v4[t] = nid(&c), INS(&c.glob, OpConstantNull, t_v4[t], null_v4[t]);
INS(&c.glob, OpConstantComposite, t_v4[0], away, fout, fout, f0, f1);
uint32_t slot_words = meshfb_slot_words(m), k0 = K(&c, 0), entry = nid(&c);
DO(&c, OpFunction, c.t_void, entry, 0, c.t_fnvoid);
DO(&c, OpLabel, nid(&c));
uint32_t vi = E(&c, OpBitcast, c.t_uint, E(&c, OpLoad, t_int, vi_var));
uint32_t slot = E(&c, OpBitcast, c.t_uint, E(&c, OpLoad, t_int, ii_var));
uint32_t prim = m->ipp == 1 ? vi : E(&c, OpUDiv, c.t_uint, vi, K(&c, m->ipp));
uint32_t corner = m->ipp == 1 ? k0 : E(&c, OpISub, c.t_uint, vi, mul(&c, prim, K(&c, m->ipp)));
uint32_t base = mul(&c, slot, K(&c, slot_words));
// a slot past the buffer (more workgroups than it has room for) draws nothing
uint32_t valid = E(&c, OpULessThanEqual, c.t_bool, add(&c, base, K(&c, slot_words)), E(&c, OpArrayLength, c.t_uint, buf, 0));
uint32_t cw = buf_load(&c, buf, E(&c, OpSelect, c.t_uint, valid, add(&c, base, K(&c, m->map / 4)), k0));
uint32_t count = E(&c, OpSelect, c.t_uint, valid, umin(&c, cw, K(&c, m->maxp)), k0);
uint32_t live = E(&c, OpULessThan, c.t_bool, prim, count), then = nid(&c), other = nid(&c), merge = nid(&c);
DO(&c, OpSelectionMerge, merge, 0);
DO(&c, OpBranchConditional, live, then, other);
DO(&c, OpLabel, then);
{
// byte map + 4 + ipp * prim + corner of the slot is the vertex
uint32_t b = add(&c, add(&c, mul(&c, prim, K(&c, m->ipp)), corner), K(&c, m->map + 4));
uint32_t word = buf_load(&c, buf, add(&c, base, E(&c, OpShiftRightLogical, c.t_uint, b, K(&c, 2))));
uint32_t shift = mul(&c, E(&c, OpBitwiseAnd, c.t_uint, b, K(&c, 3)), K(&c, 8));
uint32_t v = umin(&c, E(&c, OpBitwiseAnd, c.t_uint, E(&c, OpShiftRightLogical, c.t_uint, word, shift), K(&c, 0xff)), K(&c, m->maxv - 1));
// word ((v >> 5) * lines + line) * 32 + (v & 31) of the vertex area
uint32_t at = add(&c, add(&c, mul(&c, E(&c, OpShiftRightLogical, c.t_uint, v, K(&c, 5)), K(&c, m->nlines * 32)),
E(&c, OpBitwiseAnd, c.t_uint, v, K(&c, 31))), add(&c, base, K(&c, m->attr / 4)));
uint32_t pos[4] = {f0, f0, f0, f1}, size = f1, gen[32][4];
memset(gen, 0, sizeof gen);
for (uint32_t l = 0; l < m->nlines; l++) {
uint32_t a = m->lines[l];
int g = a >= 0x80 ? (int)(a - 0x80) / 16 : -1;
if (a < 0x6c || (g >= 0 && !loc_var[g])) continue; // layer and viewport index are not carried; an output nothing reads
uint32_t word_l = buf_load(&c, buf, l ? add(&c, at, K(&c, l * 32)) : at);
if (g >= 0) gen[g][(a - 0x80) % 16 / 4] = word_l;
else if (a == 0x6c) size = E(&c, OpBitcast, t_float, word_l);
else pos[(a - 0x70) / 4] = E(&c, OpBitcast, t_float, word_l);
}
DO(&c, OpStore, pos_var, E(&c, OpCompositeConstruct, t_v4[0], pos[0], pos[1], pos[2], pos[3]));
if (size_var) DO(&c, OpStore, size_var, size);
for (int g = 0; g < 32; g++) {
if (!loc_var[g]) continue;
int t = out_type[g] < 3 ? out_type[g] : 0;
uint32_t comp[4];
for (int k = 0; k < 4; k++) {
uint32_t word_k = gen[g][k] ? gen[g][k] : k0;
comp[k] = t == 2 ? word_k : E(&c, OpBitcast, t_scalar[t], word_k);
}
DO(&c, OpStore, loc_var[g], E(&c, OpCompositeConstruct, t_v4[t], comp[0], comp[1], comp[2], comp[3]));
}
}
DO(&c, OpBranch, merge);
DO(&c, OpLabel, other);
DO(&c, OpStore, pos_var, away);
if (size_var) DO(&c, OpStore, size_var, f1);
for (int g = 0; g < 32; g++)
if (loc_var[g]) DO(&c, OpStore, loc_var[g], null_v4[out_type[g] < 3 ? out_type[g] : 0]);
DO(&c, OpBranch, merge);
DO(&c, OpLabel, merge);
wb_ins(&c.fn, OpReturn, NULL, 0);
wb_ins(&c.fn, OpFunctionEnd, NULL, 0);
uint32_t head[5] = {0x07230203, 0x00010400, 0, c.bound, 0};
wb_put(&out, head, 5);
INS(&out, OpCapability, CapShader);
INS(&out, OpMemoryModel, 0, 1); // Logical, GLSL450
uint32_t ep[5] = {(n_iface + 5) << 16 | OpEntryPoint, ModelVertex, entry, 0x6e69616d /* "main" */, 0};
wb_put(&out, ep, 5);
wb_put(&out, iface, n_iface);
wb_put(&out, c.deco.w, c.deco.n);
wb_put(&out, c.glob.w, c.glob.n);
wb_put(&out, c.fn.w, c.fn.n);
int oom = out.oom || c.deco.oom || c.glob.oom || c.fn.oom;
free(c.deco.w), free(c.glob.w), free(c.fn.w);
if (oom) return free(out.w), NULL;
*out_words = out.n;
return out.w;
}
// ---- reflection text
char *meshfb_res_json(const char *res, uint32_t binding, const int *cb_slots, int n) {
size_t len = strlen(res);
char *out = malloc(len + (size_t)n * 96 + 1), *o = out;
if (!out) return NULL;
const char *sb = strstr(res, "\"sbufs\""), *open = sb ? strchr(sb, '[') : NULL;
if (!open) return free(out), NULL;
memcpy(o, res, (size_t)(open + 1 - res)), o += open + 1 - res;
const char *rest = open + 1;
while (*rest == ' ' || *rest == '\n' || *rest == '\r' || *rest == '\t') rest++;
for (int i = 0; i < n; i++)
o += sprintf(o, "{\"binding\": %u, \"slot\": -1, \"cb_slot\": %d, \"cb_offset\": 0, \"flags\": %d}%s", binding + (uint32_t)i,
cb_slots[i], cb_slots[i] == MESHFB_CB_LIST ? 0 : 1, i + 1 < n || *rest != ']' ? "," : "");
strcpy(o, open + 1);
char *st = strstr(out, "\"stage\"");
st = st ? strchr(st + 7, '"') : NULL;
if (!st || (st[1] != 'm' && st[1] != 't') || st[2] != 's') return free(out), NULL;
st[1] = 'c';
return out;
}

51
runtime/mesh_fb.h Normal file
View File

@@ -0,0 +1,51 @@
// The mesh shader fallback's modules (mesh_fb.c): what a task or mesh program computes, as compute modules that write
// it into storage buffers, and the vertex stage that draws from those buffers. No Vulkan calls here.
#pragma once
#include <stddef.h>
#include <stdint.h>
#define MESHFB_MAX_LINES 136 // 8 system value components and 32 generic vectors of 4
// A task or mesh program's stage buffer, as tools/xlate/AmpereLowering.cs MeshInfo lays it out in the shared memory
// of the translated module.
typedef struct {
int stage; // 6 task, 7 mesh
uint32_t threads; // workgroup size
uint32_t maxv, maxp; // mesh: vertex and primitive limits
uint32_t ipp; // mesh: indices per primitive (1 points, 2 lines, 3 triangles)
uint32_t payload_bytes; // the task payload (task, or a mesh program a task feeds), with its header
int has_task; // mesh: a task stage feeds it
uint32_t nlines; // mesh: output components per vertex
uint16_t lines[MESHFB_MAX_LINES]; // their attribute addresses: 0x6c point size, 0x70.. position, 0x80 + 16 g + 4 c
uint32_t attr, map, pay, shared_bytes; // byte offsets in shared memory, and its size
} MeshFbInfo;
// Where a module's extra storage buffers are bound: res.json "cb_slot" of their entries (vk_exec_draw.c ssbo_info).
enum { MESHFB_CB_TASK = 0x41, MESHFB_CB_MESH = 0x42, MESHFB_CB_LIST = 0x43 };
// A list entry: the task workgroup in bits 0-23, the mesh workgroup's index among that task's in bits 24-31.
#define MESHFB_LIST_TASK_BITS 24
// From the program's control section and the first 0x80 bytes of its code (the header with the output map). 0 with
// err filled when the program uses what the fallback does not carry.
int meshfb_info(const uint8_t *control, const uint8_t *header, int stage, MeshFbInfo *out, char *err, size_t cap);
// Words one mesh workgroup's output takes in the mesh buffer: its vertex lines and its primitive area.
static inline uint32_t meshfb_slot_words(const MeshFbInfo *m) { return m->pay / 4; }
// The compute module of a TaskEXT module (spirv, as the translator wrote it): the same program, then the payload of
// workgroup t into words [t * payload_bytes / 4, ..) of the task buffer, a storage buffer at `binding`. malloc'd;
// NULL with err filled.
uint32_t *meshfb_task_cs(const uint32_t *spirv, uint64_t words, const MeshFbInfo *m, uint32_t binding, uint64_t *out_words,
char *err, size_t cap);
// The compute module of a MeshEXT module. Workgroup g writes its stage buffer (meshfb_slot_words) into slot g of the
// mesh buffer at binding + 1. After a task stage (m->has_task) entry g of the list at binding + 2 names its task
// workgroup and its index among that task's mesh workgroups, and the payload comes from the task buffer at `binding`,
// task_payload_words apart.
uint32_t *meshfb_mesh_cs(const uint32_t *spirv, uint64_t words, const MeshFbInfo *m, uint32_t binding,
uint32_t task_payload_words, uint64_t *out_words, char *err, size_t cap);
// The vertex module that draws the mesh buffer (binding 1): instance = slot, maxp * ipp vertices an instance; vertex
// index / ipp is the primitive, and a primitive past the slot's count is degenerate. Outputs: position, point size for
// points, and a 4-component vector at every Location of out_mask, of out_type[location] (GFX_TYPE_*).
uint32_t *meshfb_vs(const MeshFbInfo *m, uint32_t out_mask, const uint8_t out_type[32], uint64_t *out_words);
// A copy of the translator's res.json text res for a compute module made above: its stage is "cs" and its storage
// buffers gain the entries of the extra buffers (n of them from `binding` on, cb_slots[i] each). malloc'd.
char *meshfb_res_json(const char *res, uint32_t binding, const int *cb_slots, int n);

432
runtime/mesh_fb_probe.c Normal file
View File

@@ -0,0 +1,432 @@
// nx2navi.exe --probe-meshfb: the mesh shader fallback (mesh_fb.c, vk_mesh_fb.c, vk_exec_draw.c "mesh fallback").
//
// Draw part, no game files: two mesh programs of the runtime's own in the shape the translator gives one
// (shaders/meshfb_probe_*.mesh, .task), one without a task stage and one fed by a task program, drawn through the
// replay into an RGBA8 target with direct and with indirect workgroup counts, and both in one pass after each other.
// Each picture is drawn twice, through VK_EXT_mesh_shader and through the fallback, and the two are compared texel by
// texel; without mesh shaders on the device only the fallback draws, and the probe says that nothing was compared.
//
// Module part, with NX2NAVI_PROBE_MESHFB_DIR=<dir>: every "<name>" of <dir>/list.txt names <dir>/<name>.in (a stored
// shader input, gfx_shader.c in_store: control section and code) and <dir>/<name>.spv (its translation). The compute
// module and, for a mesh program, a vertex module are written to <dir>/out/<name>.cs.spv and .vs.spv for spirv-val
// (work the caller does; tests of game shaders stay outside the repository). Where <dir>/<name>.res.json (the
// translation's reflection) is there too, the compute pipeline of the module is built on this device, as its first
// draw would build it.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "cfg.h"
#include "gfx.h"
#include "host.h"
#include "mesh_fb.h"
#include "nvn_api.h"
#include "nvn_tex.h"
#include "vk.h"
#include "vk_exec.h"
#include "vk_mesh_fb.h"
#include "platform/platform.h"
#include "meshfb_probe_f_spv.h"
#include "meshfb_probe_m_spv.h"
#include "meshfb_probe_t_spv.h"
#include "meshfb_probe_tm_spv.h"
#define TEX_W 256
#define TEX_H 144
#define CHECK(cond, ...) \
do { \
if (!(cond)) { \
printf("probe-meshfb: FAILED: "); \
printf(__VA_ARGS__); \
printf("\n"); \
return 1; \
} \
} while (0)
void vk_exec_probe_shader(const uint8_t *ctl, const GfxShader *s); // vk_exec_draw.c
void vk_exec_draw_shutdown(void);
void vk_exec_xfer_shutdown(void);
// ---- module part
static uint8_t *read_all(const char *path, size_t *size) {
FILE *f = fopen(path, "rb");
if (!f) return NULL;
fseek(f, 0, SEEK_END);
long n = ftell(f);
fseek(f, 0, SEEK_SET);
uint8_t *b = malloc((size_t)n + 1);
if (b && fread(b, 1, (size_t)n, f) != (size_t)n) free(b), b = NULL;
fclose(f);
if (b) b[n] = 0, *size = (size_t)n;
return b;
}
static int write_all(const char *path, const void *data, size_t n) {
FILE *f = fopen(path, "wb");
if (!f) return 0;
int ok = fwrite(data, 1, n, f) == n;
fclose(f);
return ok;
}
static int modules_part(const char *dir) {
char path[1024];
size_t list_n = 0;
snprintf(path, sizeof path, "%s/list.txt", dir);
char *list = (char *)read_all(path, &list_n);
CHECK(list, "no %s", path);
int n_task = 0, n_mesh = 0, n_fail = 0, n_pipe = 0, n_pipe_fail = 0;
double pipe_ms = 0, pipe_worst = 0;
uint64_t pipe_worst_words = 0;
const GfxBackend *b = &gfx_vulkan_backend;
pool_t pool = {.magic = NVN_POOL_MAGIC, .flags = 0x22};
int device = 0;
for (char *name = strtok(list, "\r\n"); name; name = strtok(NULL, "\r\n")) {
size_t in_n = 0, spv_n = 0;
snprintf(path, sizeof path, "%s/%s.in", dir, name);
uint8_t *in = read_all(path, &in_n);
snprintf(path, sizeof path, "%s/%s.spv", dir, name);
uint8_t *spv = read_all(path, &spv_n);
uint32_t h[6]; // magic, version, stage, code length, c1 length, size of the answers (gfx_shader.c InHeader)
if (!in || !spv || in_n < sizeof h) {
printf("probe-meshfb: %s: input or translation missing\n", name), n_fail++;
free(in), free(spv);
continue;
}
memcpy(h, in, sizeof h);
const uint8_t *control = in + sizeof h + h[5], *code = control + 0x880;
int stage = (int)h[2]; // GFX_STAGE_TS / GFX_STAGE_MS are the NVN stage numbers 6 and 7
char err[200] = "";
MeshFbInfo m;
uint64_t on = 0, vn = 0;
uint32_t *cs = NULL, *vs = NULL;
if (in_n < sizeof h + h[5] + 0x880 + 0x80 || (stage != 6 && stage != 7)) snprintf(err, sizeof err, "not a task or mesh input");
else if (meshfb_info(control, code, stage, &m, err, sizeof err)) {
uint32_t top = 0;
const uint32_t *w = (const uint32_t *)spv;
for (size_t i = 5; i < spv_n / 4;) {
uint32_t op = w[i] & 0xffff, len = w[i] >> 16;
if (!len) break;
if (op == 71 && len == 4 && w[i + 2] == 33 && w[i + 3] > top) top = w[i + 3];
i += len;
}
cs = stage == 6 ? meshfb_task_cs(w, spv_n / 4, &m, top + 1, &on, err, sizeof err)
: meshfb_mesh_cs(w, spv_n / 4, &m, top + 1, m.payload_bytes / 4, &on, err, sizeof err);
if (cs && stage == 7) {
uint8_t type[32] = {0};
uint32_t mask = 0;
for (uint32_t l = 0; l < m.nlines; l++)
if (m.lines[l] >= 0x80) mask |= 1u << (m.lines[l] - 0x80) / 16;
for (int g = 0; g < 32; g++) type[g] = (uint8_t)(g % 3); // every output type gets made somewhere
vs = meshfb_vs(&m, mask, type, &vn);
if (!vs) snprintf(err, sizeof err, "no vertex module");
}
}
if (cs && (stage == 6 || vs)) {
snprintf(path, sizeof path, "%s/out/%s.cs.spv", dir, name);
int ok = write_all(path, cs, on * 4);
if (vs) snprintf(path, sizeof path, "%s/out/%s.vs.spv", dir, name), ok = ok && write_all(path, vs, vn * 4);
if (!ok) printf("probe-meshfb: %s: cannot write under %s/out\n", name, dir), n_fail++;
else if (stage == 6) n_task++;
else n_mesh++;
size_t res_n = 0;
snprintf(path, sizeof path, "%s/%s.res.json", dir, name);
char *res = (char *)read_all(path, &res_n);
if (res) { // the pipeline, with the layout the reflection gives
if (!device) { // the first pool brings the device up
uint64_t heap = 0;
CHECK(mem_set_heap_size(16ull << 20, &heap) == 0 && heap, "SetHeapSize 16 MiB");
pool.mem = pool.game_mem = (uint8_t *)(uintptr_t)heap + 0x100000, pool.size = 0x100000;
CHECK(b->pool_init(&pool), "pool_init of a CPU-visible pool");
device = 1;
}
uint32_t top = 0;
const uint32_t *w = (const uint32_t *)spv;
for (size_t i = 5; i < spv_n / 4;) {
uint32_t op = w[i] & 0xffff, len = w[i] >> 16;
if (!len) break;
if (op == 71 && len == 4 && w[i + 2] == 33 && w[i + 3] > top) top = w[i + 3];
i += len;
}
static const int task_slots[1] = {MESHFB_CB_TASK}, mesh_slots[3] = {MESHFB_CB_TASK, MESHFB_CB_MESH, MESHFB_CB_LIST};
char *cres = stage == 6 ? meshfb_res_json(res, top + 1, task_slots, 1)
: m.has_task ? meshfb_res_json(res, top + 1, mesh_slots, 3) : meshfb_res_json(res, top + 2, mesh_slots + 1, 1);
const GfxShader *sh = cres ? gfx_shader_own(cs, on, cres) : NULL;
const VkPipeInfo *info = NULL;
uint64_t t0 = plat_now_ns();
VkPipeline p = sh ? vk_pipeline_compute(sh, &info) : VK_NULL_HANDLE;
double ms = (double)(plat_now_ns() - t0) / 1e6;
if (p && info) {
n_pipe++, pipe_ms += ms;
if (ms > pipe_worst) pipe_worst = ms, pipe_worst_words = on;
} else {
n_pipe_fail++;
printf("probe-meshfb: %s (stage %d, %llu words): %s\n", name, stage, (unsigned long long)on,
sh ? "its compute pipeline was not built" : "its reflection could not be made");
}
fflush(stdout);
free(cres), free(res);
}
} else {
printf("probe-meshfb: %s (stage %d): %s\n", name, stage, err), n_fail++;
}
free(cs), free(vs), free(in), free(spv);
}
free(list);
printf("probe-meshfb: module part: %d task and %d mesh programs converted, %d failed\n", n_task, n_mesh, n_fail);
if (device) {
printf("probe-meshfb: module part: %d compute pipelines built on this device (%.0f ms each, the longest %.0f ms for %llu "
"words), %d not built\n", n_pipe, n_pipe ? pipe_ms / n_pipe : 0.0, pipe_worst, (unsigned long long)pipe_worst_words,
n_pipe_fail);
b->pool_fini(&pool);
b->submit(NULL, 0); // the pool's buffers go when a submission after their release has completed
b->finish();
vk_pipeline_shutdown();
vk_exec_draw_shutdown();
vk_exec_xfer_shutdown();
vk_backend_shutdown();
}
return n_fail != 0 || n_pipe_fail != 0;
}
// ---- draw part
static uint8_t *put(uint8_t *p, uint32_t op, const void *payload, uint32_t n) {
op_hdr h = {op, (uint32_t)((sizeof h + n + 7) & ~7u)};
memset(p, 0, h.size);
memcpy(p, &h, sizeof h);
if (n) memcpy(p + sizeof h, payload, n);
return p + h.size;
}
// An NVN2 control section of a task (6), mesh (7) or fragment (1) stage as vk_exec_draw.c and mesh_fb.c read it, and
// a code block whose header maps the outputs: position and generic vector 0.
static void fake_stage(NVNshaderData *sd, uint32_t stage, int fed_by_task) {
uint8_t *ctl = calloc(1, 0x880), *code = calloc(1, 0x100);
uint32_t magic = 0x98761234u, len = 0x100, st = stage << 16;
uint32_t w58 = 32u | 1u << 24; // 32 threads, a payload of one 256-byte unit
uint32_t w5c = 3u << 24 | (fed_by_task ? 1u << 16 : 0); // triangles
uint32_t w60 = 64u | 32u << 16; // 64 vertices, 32 primitives
memcpy(ctl, &magic, 4), memcpy(ctl + 0x38, &len, 4), memcpy(ctl + 0x54, &st, 4);
if (stage >= 6) memcpy(ctl + 0x58, &w58, 4);
if (stage == 7) memcpy(ctl + 0x5c, &w5c, 4), memcpy(ctl + 0x60, &w60, 4), code[0x35] = 0xf0, code[0x36] = 0x0f;
sd->control = ctl, sd->data = (uint64_t)(uintptr_t)code;
}
static const char RES_MS[] = "{\"key\":\"probe\",\"stage\":\"ms\",\"notes\":[],\"cbufs\":[{\"binding\":1,\"slot\":3}],"
"\"sbufs\":[],\"textures\":[],\"images\":[]}";
static const char RES_TS[] = "{\"key\":\"probe\",\"stage\":\"ts\",\"notes\":[],\"cbufs\":[{\"binding\":1,\"slot\":3}],"
"\"sbufs\":[],\"textures\":[],\"images\":[]}";
static const char RES_FS[] = "{\"key\":\"probe\",\"stage\":\"ps\",\"notes\":[],\"cbufs\":[],\"sbufs\":[],\"textures\":[],"
"\"images\":[]}";
// The program of a mesh stage, its task stage if any, and the fragment stage.
static prog_t *make_prog(const GfxShader *ms, const GfxShader *ts, const GfxShader *fs) {
static uint64_t serial = 0x9000;
prog_t *p = calloc(1, sizeof *p);
p->magic = NVN_PROG_MAGIC, p->serial = ++serial;
fake_stage(&p->sd[p->nstage], 7, ts != NULL), vk_exec_probe_shader(p->sd[p->nstage++].control, ms);
if (ts) fake_stage(&p->sd[p->nstage], 6, 0), vk_exec_probe_shader(p->sd[p->nstage++].control, ts);
fake_stage(&p->sd[p->nstage], 1, 0), vk_exec_probe_shader(p->sd[p->nstage++].control, fs);
return p;
}
typedef struct { prog_t *prog; uint32_t count; int indirect; uint32_t grid[2]; float tint[4]; } MeshDraw;
// Uniform data in the pool, 0x1000 apart per draw: the mesh stage's block at 0x90000, the task stage's at 0x98000; the
// indirect records at 0xa0000.
static int draw_picture(const GfxBackend *b, pool_t *pool, tex_t *color, const MeshDraw *d, int n, uint32_t *px) {
static _Alignas(8) uint8_t stream[16384];
uint8_t *q = put(stream, OP_BEGIN, NULL, 0);
p_set_rt rt;
memset(&rt, 0, sizeof rt);
rt.n = 1, rt.color[0] = color;
q = put(q, OP_SET_RT, &rt, sizeof rt);
const p_clear_color c0 = {0, 0xf, {0.2f, 0.4f, 0.6f, 1.0f}};
q = put(q, OP_CLEAR_COLOR, &c0, sizeof c0);
p_viewports vp;
memset(&vp, 0, sizeof vp);
vp.n = 1, vp.v[0][2] = TEX_W, vp.v[0][3] = TEX_H, vp.v[0][5] = 1.0f;
q = put(q, OP_VIEWPORTS, &vp, sizeof vp);
const p_scissor full = {0, 0, 1 << 20, 1 << 20};
q = put(q, OP_SCISSOR, &full, sizeof full);
for (int i = 0; i < n; i++) {
uint8_t *mu = pool->mem + 0x90000 + 0x1000 * i, *tu = pool->mem + 0x98000 + 0x1000 * i;
uint32_t *rec = (uint32_t *)(pool->mem + 0xa0000 + 0x100 * i);
uint32_t grid[4] = {d[i].grid[0], d[i].grid[1], 0, 0}, shift[4] = {1, 0, 0, 0};
memcpy(mu, d[i].tint, 16), memcpy(mu + 16, grid, 16), memcpy(tu, shift, 16);
rec[0] = d[i].count, rec[1] = 0, rec[2] = 0;
const p_bind_program bp = {d[i].prog, 0xc2};
q = put(q, OP_BIND_PROGRAM, &bp, sizeof bp);
const p_bind_buf bm = {7, 0, (uint64_t)(uintptr_t)mu, 256}, bt = {6, 0, (uint64_t)(uintptr_t)tu, 256};
q = put(q, OP_BIND_UBO, &bm, sizeof bm);
q = put(q, OP_BIND_UBO, &bt, sizeof bt);
if (d[i].indirect) {
const p_draw_mesh_ind di = {(uint64_t)(uintptr_t)rec, 0, 0, 0, 0};
q = put(q, OP_DRAW_MESH_IND, &di, sizeof di);
} else {
const p_draw_mesh dm = {0, d[i].count, 0};
q = put(q, OP_DRAW_MESH, &dm, sizeof dm);
}
}
q = put(q, OP_END, NULL, 0);
CHECK(q <= stream + sizeof stream, "op stream overflow");
uint64_t handle = (uint64_t)(uintptr_t)stream;
uint64_t f = b->submit(&handle, 1);
CHECK(b->fence_wait(f, 60000000000ull), "fence_wait of a mesh draw submission timed out");
NVNcopyRegion r = {0, 0, 0, TEX_W, TEX_H, 1};
b->read_texels(color, NULL, &r, (uint8_t *)px, TEX_W * 4, TEX_W * TEX_H * 4);
return 0;
}
static int drawn_texels(const uint32_t *px) {
int n = 0;
for (int i = 0; i < TEX_W * TEX_H; i++) n += px[i] != px[0]; // texel 0 is in the margin no cell covers
return n;
}
// Texels of a and b that differ, and the largest difference of a channel.
static int differing(const uint32_t *a, const uint32_t *b, int *worst) {
int n = 0;
*worst = 0;
for (int i = 0; i < TEX_W * TEX_H; i++) {
if (a[i] == b[i]) continue;
n++;
for (int c = 0; c < 4; c++) {
int d = (int)(a[i] >> 8 * c & 0xff) - (int)(b[i] >> 8 * c & 0xff);
if (d < 0) d = -d;
if (d > *worst) *worst = d;
}
}
return n;
}
static int draw_part(const GfxBackend *b, uint8_t *guest) {
pool_t pool = {.magic = NVN_POOL_MAGIC, .flags = 0x22, .mem = guest + 0x100000, .size = 0xC0000,
.game_mem = guest + 0x100000};
CHECK(b->pool_init(&pool), "pool_init of a CPU-visible pool");
VkCtx *c = vk_ctx();
CHECK(c, "no Vulkan device");
static tex_t color;
memset(&color, 0, sizeof color);
color.magic = NVN_TEX_MAGIC;
color.d.target = NVN_TEXTURE_TARGET_2D, color.d.format = NVN_FORMAT_RGBA8, color.d.width = TEX_W, color.d.height = TEX_H;
color.d.depth = 1, color.d.levels = 1;
nvn_tex_layout(&color.d, &color.L);
CHECK(color.L.size <= 0x80000, "the target exceeds its pool slot");
color.pool = &pool, color.pool_flags = (uint16_t)pool.flags, color.off = 0, color.mem = pool.mem, color.serial = 4000;
const GfxShader *fs = gfx_shader_own(meshfb_probe_f_spv, sizeof meshfb_probe_f_spv / 4, RES_FS);
const GfxShader *ms = gfx_shader_own(meshfb_probe_m_spv, sizeof meshfb_probe_m_spv / 4, RES_MS);
const GfxShader *ts = gfx_shader_own(meshfb_probe_t_spv, sizeof meshfb_probe_t_spv / 4, RES_TS);
const GfxShader *tm = gfx_shader_own(meshfb_probe_tm_spv, sizeof meshfb_probe_tm_spv / 4, RES_MS);
CHECK(fs && ms && ts && tm, "a probe shader is malformed");
prog_t *plain = make_prog(ms, NULL, fs), *tasked = make_prog(tm, ts, fs);
// 24 cells of a 6 x 4 grid by 24 mesh workgroups; 40 cells of an 8 x 5 grid by 10 task workgroups that start 0 to
// 4 mesh workgroups each
const MeshDraw cases[][2] = {
{{plain, 24, 0, {6, 4}, {1, 1, 1, 1}}},
{{plain, 24, 1, {6, 4}, {1, 1, 1, 1}}},
{{tasked, 10, 0, {8, 5}, {1, 1, 1, 1}}},
{{tasked, 10, 1, {8, 5}, {1, 1, 1, 1}}},
{{plain, 24, 1, {6, 4}, {1, 1, 1, 1}}, {tasked, 10, 1, {8, 5}, {0.5f, 1, 0.25f, 1}}},
{{tasked, 10, 0, {8, 5}, {1, 1, 1, 1}}, {plain, 24, 0, {6, 4}, {1, 0.5f, 0.5f, 1}}},
};
static const char *const names[] = {"mesh, direct", "mesh, indirect", "task + mesh, direct", "task + mesh, indirect",
"mesh then task + mesh in one pass, indirect", "task + mesh then mesh in one pass, direct"};
enum { CASES = sizeof cases / sizeof cases[0] };
static uint32_t px[2][CASES][TEX_W * TEX_H];
int native = c->has_mesh_shader, was = c->mesh_fallback;
for (int way = native ? 0 : 1; way < 2; way++) {
c->mesh_fallback = way;
for (int k = 0; k < CASES; k++)
if (draw_picture(b, &pool, &color, cases[k], cases[k][1].prog ? 2 : 1, px[way][k])) return 1;
}
c->mesh_fallback = was;
for (int k = 0; k < CASES; k++) {
int fb = drawn_texels(px[1][k]);
CHECK(fb > TEX_W * TEX_H / 20, "%s: the fallback drew %d texels", names[k], fb);
if (!native) {
printf("probe-meshfb: %s: the fallback drew %d of %d texels (no mesh shaders here: not compared)\n", names[k], fb,
TEX_W * TEX_H);
continue;
}
int worst, diff = differing(px[0][k], px[1][k], &worst), nat = drawn_texels(px[0][k]);
CHECK(nat > TEX_W * TEX_H / 20, "%s: the mesh pipeline drew %d texels", names[k], nat);
CHECK(worst <= 1, "%s: %d texels differ between the mesh pipeline and the fallback, by up to %d of 255 (the mesh "
"pipeline drew %d texels, the fallback %d)", names[k], diff, worst, nat, fb);
printf("probe-meshfb: %s: %d texels drawn by the mesh pipeline, %d by the fallback, %d differ (by %d of 255 at most)\n",
names[k], nat, fb, diff, worst);
}
int w01, w23;
CHECK(!differing(px[1][0], px[1][1], &w01) && !differing(px[1][2], px[1][3], &w23),
"the fallback's direct and indirect pictures differ");
CHECK(!vk_meshfb_short(), "%llu of the draws above did not fit the fallback's buffers", (unsigned long long)vk_meshfb_short());
// More workgroups than the mesh buffer has slots for: what fits is drawn, the rest is dropped and counted. The
// cells past the grid are outside the picture, so it is the 24-cell picture again.
uint32_t slots = (uint32_t)(vk_meshfb_mesh_words() / 540), over = slots + 1000;
const MeshDraw big = {plain, over, 1, {6, 4}, {1, 1, 1, 1}};
static uint32_t bpx[TEX_W * TEX_H];
c->mesh_fallback = 1;
int bad = draw_picture(b, &pool, &color, &big, 1, bpx);
c->mesh_fallback = was;
if (bad) return 1;
b->finish();
int wb, db = differing(bpx, px[1][0], &wb);
CHECK(!db, "a draw of %u workgroups, %u more than the mesh buffer holds: %d texels differ from the 24-cell picture", over,
over - slots, db);
CHECK(vk_meshfb_short() == 1, "a draw of %u workgroups with room for %u was counted as %llu draws that did not fit", over, slots,
(unsigned long long)vk_meshfb_short());
printf("probe-meshfb: a draw of %u workgroups with room for %u: the picture of the first 24 is whole, the draw was counted\n",
over, slots);
// As the game runs it: pipelines built in the background, and a program's modules made on the fallback's own
// thread. New shader objects of the same programs are drawn until the picture is whole: the draws before that are
// skipped, and none may leave a partial picture behind for good.
const GfxShader *fs2 = gfx_shader_own(meshfb_probe_f_spv, sizeof meshfb_probe_f_spv / 4, RES_FS);
const GfxShader *ms2 = gfx_shader_own(meshfb_probe_m_spv, sizeof meshfb_probe_m_spv / 4, RES_MS);
const GfxShader *ts2 = gfx_shader_own(meshfb_probe_t_spv, sizeof meshfb_probe_t_spv / 4, RES_TS);
const GfxShader *tm2 = gfx_shader_own(meshfb_probe_tm_spv, sizeof meshfb_probe_tm_spv / 4, RES_MS);
CHECK(fs2 && ms2 && ts2 && tm2, "a probe shader is malformed");
prog_t *plain2 = make_prog(ms2, NULL, fs2), *tasked2 = make_prog(tm2, ts2, fs2);
const MeshDraw again[2] = {{tasked2, 10, 0, {8, 5}, {1, 1, 1, 1}}, {plain2, 24, 0, {6, 4}, {1, 0.5f, 0.5f, 1}}};
vk_pipeline_async_enable();
c->mesh_fallback = 1;
int tries = 0, left = -1, wa;
while (left && tries < 500) {
if (tries++) plat_sleep_ms(10);
if (draw_picture(b, &pool, &color, again, 2, bpx)) return 1;
left = differing(bpx, px[1][5], &wa);
}
c->mesh_fallback = was;
CHECK(!left, "with background builds the picture still differs in %d texels after %d submissions", left, tries);
printf("probe-meshfb: with background builds the picture was whole at submission %d (the earlier ones skipped their draws)\n",
tries);
b->texture_release(&color);
b->pool_fini(&pool);
return 0;
}
int probe_meshfb(void) {
const char *dir = cfg_get("NX2NAVI_PROBE_MESHFB_DIR");
if (dir && *dir) return modules_part(dir);
vk_hdr_off();
const GfxBackend *b = &gfx_vulkan_backend;
uint64_t heap = 0;
CHECK(mem_set_heap_size(16ull << 20, &heap) == 0 && heap, "SetHeapSize 16 MiB");
int bad = draw_part(b, (uint8_t *)(uintptr_t)heap);
b->finish();
vk_pipeline_shutdown();
vk_exec_draw_shutdown();
vk_exec_xfer_shutdown();
vk_backend_shutdown();
long errors = vk_validation_errors_total();
if (bad || errors) {
printf("probe-meshfb: FAILED (%s, %ld validation errors)\n", bad ? "a draw check failed" : "checks passed", errors);
return 1;
}
printf("probe-meshfb: passed (0 validation errors)\n");
return 0;
}

View File

@@ -0,0 +1,6 @@
#version 450
// --probe-meshfb (mesh_fb_probe.c): the fragment stage of the probe's mesh programs: the interpolated generic vector 0.
layout(location = 0) in vec4 i_color;
layout(location = 0) out vec4 o_color;
void main() { o_color = i_color; }

View File

@@ -0,0 +1,75 @@
#version 460
#extension GL_EXT_mesh_shader : require
// build_shaders: keep functions
//
// --probe-meshfb (mesh_fb_probe.c): a mesh program without a task stage, in the shape the translator gives one
// (tools/xlate/MeshSpirv.cs): the program itself is a function that writes its stage buffer into shared memory, and
// the entry function calls it once and hands the buffer to the mesh pipeline. mesh_fb.c makes its compute module from
// this one, as it does from a translated program. The layout is AmpereLowering.MeshInfo's for 64 vertices, 32
// triangles and 8 output components (position, generic vector 0): vertex v's component l is word
// ((v >> 5) * 8 + l) * 32 + (v & 31), the primitive count word 512, the index bytes from byte 2052.
//
// Workgroup g draws cell g of a grid: nine vertices (28 to 36, across the two vertex groups) and up to eight
// triangles, none in every fifth cell.
layout(local_size_x = 32) in;
layout(triangles, max_vertices = 64, max_primitives = 32) out;
layout(location = 0) out vec4 o_color[];
layout(set = 0, binding = 1) uniform Params { vec4 tint; uvec4 grid; } u; // grid: columns, rows
shared uint mem[540];
#define VWORD(v, l) ((((v) >> 5) * 8u + (l)) * 32u + ((v) & 31u))
#define COUNT_WORD 512u
#define INDEX_BYTE 2052u
void program() {
uint tid = gl_LocalInvocationIndex, cell = gl_WorkGroupID.x;
if (tid < 9u) {
uint v = tid + 28u, ix = tid % 3u, iy = tid / 3u;
vec2 p = -1.0 + (vec2(cell % u.grid.x, cell / u.grid.x) + 0.1 + 0.4 * vec2(ix, iy)) * (2.0 / vec2(u.grid.xy));
vec4 c = vec4(float(ix) * 0.5, float(iy) * 0.5, float(cell % 4u) / 3.0, 1.0) * u.tint;
mem[VWORD(v, 0u)] = floatBitsToUint(p.x);
mem[VWORD(v, 1u)] = floatBitsToUint(p.y);
mem[VWORD(v, 2u)] = floatBitsToUint(0.5);
mem[VWORD(v, 3u)] = floatBitsToUint(1.0);
mem[VWORD(v, 4u)] = floatBitsToUint(c.x);
mem[VWORD(v, 5u)] = floatBitsToUint(c.y);
mem[VWORD(v, 6u)] = floatBitsToUint(c.z);
mem[VWORD(v, 7u)] = floatBitsToUint(c.w);
}
if (tid == 0u) { // one thread writes the index bytes: they share words
uint count = cell % 5u == 0u ? 0u : 2u + cell % 7u;
for (uint t = 0u; t < count; t++) {
uint q = t / 2u, a = (q / 2u) * 3u + q % 2u + 28u;
uvec3 tri = (t & 1u) == 0u ? uvec3(a, a + 1u, a + 3u) : uvec3(a + 1u, a + 4u, a + 3u);
for (uint k = 0u; k < 3u; k++) {
uint b = INDEX_BYTE + 3u * t + k, sh = (b & 3u) * 8u;
mem[b >> 2] = (mem[b >> 2] & ~(0xffu << sh)) | tri[k] << sh;
}
}
mem[COUNT_WORD] = count;
}
}
void main() {
uint tid = gl_LocalInvocationIndex;
if (tid < 1u) mem[COUNT_WORD] = 0u;
barrier();
program();
barrier();
uint count = min(mem[COUNT_WORD], 32u);
SetMeshOutputsEXT(64u, count);
for (uint v = tid; v < 64u; v += 32u) {
gl_MeshVerticesEXT[v].gl_Position = vec4(uintBitsToFloat(mem[VWORD(v, 0u)]), uintBitsToFloat(mem[VWORD(v, 1u)]),
uintBitsToFloat(mem[VWORD(v, 2u)]), uintBitsToFloat(mem[VWORD(v, 3u)]));
o_color[v] = vec4(uintBitsToFloat(mem[VWORD(v, 4u)]), uintBitsToFloat(mem[VWORD(v, 5u)]),
uintBitsToFloat(mem[VWORD(v, 6u)]), uintBitsToFloat(mem[VWORD(v, 7u)]));
}
for (uint p = tid; p < count; p += 32u) {
uvec3 tri;
for (uint k = 0u; k < 3u; k++) {
uint b = INDEX_BYTE + 3u * p + k;
tri[k] = min(mem[b >> 2] >> ((b & 3u) * 8u) & 0xffu, 63u);
}
gl_PrimitiveTriangleIndicesEXT[p] = tri;
}
}

View File

@@ -0,0 +1,33 @@
#version 460
#extension GL_EXT_mesh_shader : require
// build_shaders: keep functions
//
// --probe-meshfb (mesh_fb_probe.c): a task program in the shape the translator gives one (tools/xlate/MeshSpirv.cs):
// the program writes its payload into shared memory, and the entry function copies that out and starts the mesh
// workgroups the program counted into word 1. The payload is 288 bytes: the header, then the program's data from
// word 4.
//
// Task workgroup t starts 0 to 4 mesh workgroups (the uniform block shifts which tasks start how many) and tells each
// the grid cell it draws; the last payload word carries a mark the mesh program checks.
layout(local_size_x = 32) in;
layout(set = 0, binding = 1) uniform Params { uvec4 shift; } u;
shared uint mem[72];
taskPayloadSharedEXT uint payload[72];
void program() {
uint tid = gl_LocalInvocationIndex, t = gl_WorkGroupID.x;
uint count = (t * 3u + u.shift.x) % 5u;
if (tid == 0u) mem[1] = count;
if (tid < count) mem[4u + tid] = t * 4u + tid;
if (tid == 31u) mem[71] = 0x600df00du + t;
}
void main() {
uint tid = gl_LocalInvocationIndex;
if (tid < 1u) mem[1] = 0u;
barrier();
program();
barrier();
for (uint i = tid; i < 72u; i += 32u) payload[i] = mem[i];
EmitMeshTasksEXT(min(mem[1], 32u), 1u, 1u);
}

View File

@@ -0,0 +1,74 @@
#version 460
#extension GL_EXT_mesh_shader : require
// build_shaders: keep functions
//
// --probe-meshfb (mesh_fb_probe.c): the mesh program that meshfb_probe_t.task feeds, in the translator's shape
// (meshfb_probe_m.mesh says which). Its shared memory holds the task payload behind the stage buffer, from word 540.
// The workgroup id is its index among its task's mesh workgroups: payload word 4 + id names the cell it draws, and
// the mark in the last payload word names its task, which the colour shows when either is wrong.
layout(local_size_x = 32) in;
layout(triangles, max_vertices = 64, max_primitives = 32) out;
layout(location = 0) out vec4 o_color[];
layout(set = 0, binding = 1) uniform Params { vec4 tint; uvec4 grid; } u; // grid: columns, rows
shared uint mem[612];
taskPayloadSharedEXT uint payload[72];
#define VWORD(v, l) ((((v) >> 5) * 8u + (l)) * 32u + ((v) & 31u))
#define COUNT_WORD 512u
#define INDEX_BYTE 2052u
#define PAYLOAD 540u
void program() {
uint tid = gl_LocalInvocationIndex, cell = mem[PAYLOAD + 4u + gl_WorkGroupID.x];
bool mine = mem[PAYLOAD + 71u] == 0x600df00du + cell / 4u && cell % 4u == gl_WorkGroupID.x;
if (tid < 9u) {
uint v = tid + 28u, ix = tid % 3u, iy = tid / 3u;
vec2 p = -1.0 + (vec2(cell % u.grid.x, cell / u.grid.x) + 0.1 + 0.4 * vec2(ix, iy)) * (2.0 / vec2(u.grid.xy));
vec4 c = mine ? vec4(float(ix) * 0.5, float(iy) * 0.5, float(cell % 4u) / 3.0, 1.0) * u.tint : vec4(1.0, 0.0, 1.0, 1.0);
mem[VWORD(v, 0u)] = floatBitsToUint(p.x);
mem[VWORD(v, 1u)] = floatBitsToUint(p.y);
mem[VWORD(v, 2u)] = floatBitsToUint(0.5);
mem[VWORD(v, 3u)] = floatBitsToUint(1.0);
mem[VWORD(v, 4u)] = floatBitsToUint(c.x);
mem[VWORD(v, 5u)] = floatBitsToUint(c.y);
mem[VWORD(v, 6u)] = floatBitsToUint(c.z);
mem[VWORD(v, 7u)] = floatBitsToUint(c.w);
}
if (tid == 0u) { // one thread writes the index bytes: they share words
uint count = 2u + cell % 7u;
for (uint t = 0u; t < count; t++) {
uint q = t / 2u, a = (q / 2u) * 3u + q % 2u + 28u;
uvec3 tri = (t & 1u) == 0u ? uvec3(a, a + 1u, a + 3u) : uvec3(a + 1u, a + 4u, a + 3u);
for (uint k = 0u; k < 3u; k++) {
uint b = INDEX_BYTE + 3u * t + k, sh = (b & 3u) * 8u;
mem[b >> 2] = (mem[b >> 2] & ~(0xffu << sh)) | tri[k] << sh;
}
}
mem[COUNT_WORD] = count;
}
}
void main() {
uint tid = gl_LocalInvocationIndex;
if (tid < 1u) mem[COUNT_WORD] = 0u;
for (uint i = tid; i < 72u; i += 32u) mem[PAYLOAD + i] = payload[i];
barrier();
program();
barrier();
uint count = min(mem[COUNT_WORD], 32u);
SetMeshOutputsEXT(64u, count);
for (uint v = tid; v < 64u; v += 32u) {
gl_MeshVerticesEXT[v].gl_Position = vec4(uintBitsToFloat(mem[VWORD(v, 0u)]), uintBitsToFloat(mem[VWORD(v, 1u)]),
uintBitsToFloat(mem[VWORD(v, 2u)]), uintBitsToFloat(mem[VWORD(v, 3u)]));
o_color[v] = vec4(uintBitsToFloat(mem[VWORD(v, 4u)]), uintBitsToFloat(mem[VWORD(v, 5u)]),
uintBitsToFloat(mem[VWORD(v, 6u)]), uintBitsToFloat(mem[VWORD(v, 7u)]));
}
for (uint p = tid; p < count; p += 32u) {
uvec3 tri;
for (uint k = 0u; k < 3u; k++) {
uint b = INDEX_BYTE + 3u * p + k;
tri[k] = min(mem[b >> 2] >> ((b & 3u) * 8u) & 0xffu, 63u);
}
gl_PrimitiveTriangleIndicesEXT[p] = tri;
}
}

View File

@@ -0,0 +1,47 @@
#version 450
// The mesh fallback's pass between a draw's task dispatch and its mesh dispatch (vk_mesh_fb.c vk_meshfb_scan,
// mesh_fb.c): every task workgroup wrote how many mesh workgroups it starts into its payload (word 1). This lists them
// in task order, so that mesh workgroup g finds its task and its index among that task's in list entry g, and writes
// the draw's record: how many mesh workgroups to dispatch and how many instances to draw.
//
// One thread walks the tasks: the order of the list is the order the primitives are drawn in, which has to be the
// same in every frame and in the depth and colour passes of one frame.
layout(local_size_x = 1) in;
layout(push_constant) uniform Params {
uint rec; // the record's first word in the record buffer
uint task_words; // words of a task workgroup's payload in the task buffer
uint fan; // mesh workgroups a task workgroup starts at most
uint slot_words; // words of a mesh workgroup's output in the mesh buffer
uint verts; // vertices drawn per mesh workgroup
uint max_groups; // workgroups one dispatch may have (maxComputeWorkGroupCount)
} pc;
// The record: words 0-2 the task dispatch (x written before this pass), 4-6 the mesh dispatch, 8-11 the draw (vertices,
// instances, first vertex, first instance), 12 the mesh workgroups the tasks asked for, 13 the task workgroups.
layout(set = 0, binding = 0) buffer Rec { uint w[]; } rec;
layout(set = 0, binding = 1) readonly buffer Task { uint w[]; } task;
layout(set = 0, binding = 2) writeonly buffer List { uint e[]; } list;
layout(set = 0, binding = 3) readonly buffer Mesh { uint w[]; } mesh; // read for its size alone
void main() {
uint asked = rec.w[pc.rec];
uint tasks = min(min(asked, uint(task.w.length()) / pc.task_words), 1u << 24);
uint room = min(min(uint(mesh.w.length()) / pc.slot_words, uint(list.e.length())), pc.max_groups);
uint fan = min(pc.fan, 255u);
uint m = 0u, want = 0u;
for (uint t = 0u; t < tasks; t++) {
uint c = min(task.w[t * pc.task_words + 1u], fan);
want += c;
for (uint j = 0u; j < c && m < room; j++) list.e[m++] = t | j << 24;
}
rec.w[pc.rec + 4u] = m;
rec.w[pc.rec + 5u] = 1u;
rec.w[pc.rec + 6u] = 1u;
rec.w[pc.rec + 8u] = pc.verts;
rec.w[pc.rec + 9u] = m;
rec.w[pc.rec + 10u] = 0u;
rec.w[pc.rec + 11u] = 0u;
rec.w[pc.rec + 12u] = want;
rec.w[pc.rec + 13u] = asked;
}

View File

@@ -41,6 +41,8 @@ typedef struct VkCtx {
has_maximal_reconvergence, // VK_KHR_shader_maximal_reconvergence (vk_pipeline.c spirv_reconverge)
has_pipeline_binary, // VK_KHR_pipeline_binary with VK_KHR_maintenance5 (vk_pipeline.c pipe_create)
has_mesh_queries; // meshShaderQueries, kept while NX2NAVI_GPU_PASSES or NX2NAVI_DUMP is set
int mesh_fallback; // task and mesh programs run as compute passes and are drawn from buffers
// (mesh_fallback: auto without has_mesh_shader; vk_exec_draw.c "mesh fallback")
// Feedback loops (maxwell.md 1.9, vk_exec_draw.c feedback_prepare): a draw samples a target of its open pass that
// an earlier draw of the pass wrote. NX2NAVI_FEEDBACK=0: VK_FEEDBACK_OFF (sampled in GENERAL without a barrier, the
// old behaviour); VK_FEEDBACK_LAYOUT with VK_EXT_attachment_feedback_loop_layout and dynamicRenderingLocalRead

View File

@@ -469,8 +469,17 @@ static int create_device(void) {
g_vk.has_mesh_queries = want && c->mesh.meshShaderQueries && h->pipelineStatisticsQuery && c->f12.hostQueryReset;
c->mesh.meshShaderQueries = g_vk.has_mesh_queries ? VK_TRUE : VK_FALSE;
if (!c->mesh.meshShader || !c->mesh.taskShader) g_vk.has_mesh_shader = 0;
rt_log("vk: mesh shaders: %s", g_vk.has_mesh_shader ? "VK_EXT_mesh_shader" : "not supported (task/mesh draws are skipped)");
rt_log("vk: mesh shaders: %s", g_vk.has_mesh_shader ? "VK_EXT_mesh_shader" : "the extension is there, its task or mesh feature is not");
} // vkGetDeviceFaultInfoEXT needs the feature, not just the extension
{ // mesh_fallback: auto (on a device without mesh shaders), 1 (on every device, for testing) or 0
const char *mf = cfg_value("NX2NAVI_MESH_FALLBACK");
g_vk.mesh_fallback = !strcmp(mf, "1") || (strcmp(mf, "0") && !g_vk.has_mesh_shader);
if (g_vk.mesh_fallback)
rt_log("vk: mesh fallback: task and mesh programs run as compute passes and are drawn from their buffers (%s)",
g_vk.has_mesh_shader ? "mesh_fallback = 1" : "this device has no mesh shaders");
else if (!g_vk.has_mesh_shader)
rt_log("vk: mesh shaders: not supported and mesh_fallback = 0 (task/mesh draws are skipped)");
}
if (g_vk.has_maximal_reconvergence && !c->mrc.shaderMaximalReconvergence) g_vk.has_maximal_reconvergence = 0;
if (g_vk.has_pipeline_binary && !(c->pbin.pipelineBinaries && c->m5.maintenance5)) g_vk.has_pipeline_binary = 0;
g_vk.has_buffer_device_address = c->f12.bufferDeviceAddress;

View File

@@ -29,6 +29,8 @@
#endif
#include "vk_exec.h"
#include "vk_gpa.h"
#include "mesh_fb.h"
#include "vk_mesh_fb.h"
#include "platform/platform.h"
#include "vk_dlaa.h"
@@ -446,7 +448,12 @@ static void twins_release(VkCtx *c);
// Releases the stand-ins and the depth twins. The device must be idle; the game path leaves them to process exit like
// the pipelines.
static void mesh_ring_release(VkCtx *c);
static void pool_marks_release(VkCtx *c);
void vk_exec_draw_shutdown(void) {
vk_meshfb_shutdown();
if (vk_ctx()) mesh_ring_release(vk_ctx()), pool_marks_release(vk_ctx()); // made without the stand-ins below too
if (!Z.ready) return;
VkCtx *c = vk_ctx();
twins_release(c);
@@ -948,6 +955,13 @@ static void answers_for(VkExec *ex, int st, const uint8_t *ctl, uint32_t mask, G
// and *pending = 1 says the wait ended first.
static int sf_mvprev; // shader_for asks for the vertex shader's last-frame variant (per-object motion)
// Probes only (mesh_fb_probe.c): the shader of the program stage whose control section is ctl, in place of a
// translation. The probes build their programs from modules made here, which no translator has an input for.
static struct { const uint8_t *ctl; const GfxShader *s; } probe_sh[16];
static int n_probe_sh;
void vk_exec_probe_shader(const uint8_t *ctl, const GfxShader *s) {
if (n_probe_sh < 16) probe_sh[n_probe_sh].ctl = ctl, probe_sh[n_probe_sh].s = s, n_probe_sh++;
}
static const GfxShader *shader_for(VkExec *ex, int st, int *present, int *pending, uint32_t await_ms) {
const uint8_t *ctl, *code;
uint32_t len;
@@ -955,6 +969,8 @@ static const GfxShader *shader_for(VkExec *ex, int st, int *present, int *pendin
if (pending) *pending = 0;
if (!ex->prog || !find_stage(ex->prog, st, &ctl, &code, &len)) return NULL;
*present = 1;
for (int i = 0; i < n_probe_sh; i++)
if (probe_sh[i].ctl == ctl) return probe_sh[i].s;
GfxShaderAnswers a;
uint64_t serial = ex->prog->serial;
answers_for(ex, st, ctl, slot_mask_of(serial, st), &a);
@@ -994,7 +1010,12 @@ static const GfxShader *shader_for(VkExec *ex, int st, int *present, int *pendin
// ---- descriptors
// The mesh fallback (below, "mesh fallback") runs a program's task and mesh stages as compute modules: while it binds
// one's resources, a compute module's bindings are those of this NVN stage (6 task, 7 mesh), not of the compute stage.
static int mfb_stage;
static int nvn_stage_of(int gfx_stage) {
if (gfx_stage == GFX_STAGE_CS && mfb_stage) return mfb_stage;
return gfx_stage == GFX_STAGE_VS ? NVN_STAGE_VS : gfx_stage == GFX_STAGE_FS ? NVN_STAGE_FS
: gfx_stage == GFX_STAGE_TS ? NVN_STAGE_TASK : gfx_stage == GFX_STAGE_MS ? NVN_STAGE_MESH : NVN_STAGE_CS;
}
@@ -1373,6 +1394,8 @@ static VkDescriptorBufferInfo ssbo_info(VkExec *ex, int st, const GfxShaderRes *
VkDescriptorBufferInfo t = vk_gpa_info();
return t.buffer ? t : (VkDescriptorBufferInfo){Z.zbuf, 0, VK_WHOLE_SIZE};
}
// The mesh fallback's own buffers (mesh_fb.c): no guest memory behind them
if (r->cbuf_slot >= MESHFB_CB_TASK && r->cbuf_slot <= MESHFB_CB_LIST) return vk_meshfb_buffer(r->cbuf_slot);
if (!__atomic_load_n(&LIM.ubo_max, __ATOMIC_ACQUIRE)) limits_init();
uint64_t max = LIM.sbo_max, align = LIM.sbo_align;
VkDescriptorBufferInfo bi;
@@ -1908,6 +1931,15 @@ static void pool_marks_make(void) {
__atomic_store_n(&PL.marks_ready, 1, __ATOMIC_RELEASE); // the recording thread reads the rest after this
}
static void pool_marks_release(VkCtx *c) { // vk_exec_draw_shutdown: a probe ends with the device idle
if (!PL.marks) return;
__atomic_store_n(&PL.marks_ready, 0, __ATOMIC_RELEASE);
vmaDestroyBuffer(c->vma, PL.marks, PL.marks_alloc);
for (int i = 0; i < MARKS_RB; i++)
if (PL.rb[i].buf) vmaDestroyBuffer(c->vma, PL.rb[i].buf, PL.rb[i].alloc), PL.rb[i].buf = VK_NULL_HANDLE;
PL.marks = VK_NULL_HANDLE;
}
void vk_exec_pool_marks(VkExec *ex) {
if (!__atomic_load_n(&PL.marks_ready, __ATOMIC_ACQUIRE)) return;
VkCtx *c = vk_ctx();
@@ -3121,7 +3153,7 @@ static int pb_key(const VkExec *ex, const VkPipeBinding *b, uint32_t k, uint64_t
return 1; // slot 1: the program's constants (pb_globals); others: the zero buffer
}
if (b->kind == GFX_RES_SSBO) {
if ((r->flags & 1) || r->ptr || r->cbuf_slot == VK_GPA_CB_SLOT) return 0;
if ((r->flags & 1) || r->ptr || r->cbuf_slot >= VK_GPA_CB_SLOT) return 0; // the table, the mesh fallback's buffers
int ubo, slot = ssbo_slot(r, st, &ubo);
if (slot < 0 || slot >= 16) return 0;
const p_bind_buf *p = ubo ? &ex->ubo[st][slot] : &ex->ssbo[st][slot];
@@ -3860,8 +3892,13 @@ static void prog_lead(const prog_t *p, int mesh) {
}
// mesh: a mesh draw (op_draw_mesh): the program's mesh and task stages stand in for the vertex stage, and nothing
// vertex-related (vertex buffers, motion vector bookkeeping of vertex shaders) is bound or noted.
// vertex-related (vertex buffers, motion vector bookkeeping of vertex shaders) is bound or noted. 2: a mesh draw of
// the mesh fallback (mfb_draw): the vertex stage is mfb_cur_vs, in an ordinary pipeline without vertex input.
static const GfxShader *mfb_cur_vs;
static int prepare_draw_ex(VkExec *ex, int prim, int32_t count, int restart, int mesh) {
int fb = mesh == 2; // from here on mesh says what the draw is, and !fb that its pipeline is a mesh pipeline
mesh = mesh != 0;
if (ex->render_disabled) return vk_cond_draws_skip += ex->has_render_cond, draw_skipped(ex, mesh, SK_DISABLED);
cb_res(ex);
dummies_init();
@@ -3892,8 +3929,9 @@ static int prepare_draw_ex(VkExec *ex, int prim, int32_t count, int restart, int
}
} else {
SM.valid = 0;
vs = shader_for(ex, vst, &has_vs, &vs_wait, 0);
if (mesh) ts = shader_for(ex, NVN_STAGE_TASK, &has_ts, &ts_wait, 0);
if (fb) vs = mfb_cur_vs, has_vs = 1;
else vs = shader_for(ex, vst, &has_vs, &vs_wait, 0);
if (mesh && !fb) ts = shader_for(ex, NVN_STAGE_TASK, &has_ts, &ts_wait, 0);
fs = shader_for(ex, NVN_STAGE_FS, &has_fs, &fs_wait, 0);
if (memo && vs && !vs_wait && !fs_wait && (!has_fs || fs))
memo_key(ex, gen, &SM.k), SM.vs = vs, SM.fs = fs, SM.has_fs = has_fs, SM.valid = 1;
@@ -3927,6 +3965,7 @@ static int prepare_draw_ex(VkExec *ex, int prim, int32_t count, int restart, int
VkPipeState s = ex->pipe;
s.vs = vs, s.fs = fs, s.ts = ts;
if (mesh) foliage_shadow_cull(ex, &s);
if (fb) memset(s.vattr, 0, sizeof s.vattr), memset(s.vstream, 0, sizeof s.vstream); // whatever the last vertex draw left
s.feedback = (uint8_t)((fbl & 0xff ? 1 : 0) | (fbl & 0x100 ? 2 : 0));
s.prim = (uint8_t)prim;
s.depth_minus_w = (uint8_t)(nvn_depth_minus_w() != 0);
@@ -3946,7 +3985,7 @@ static int prepare_draw_ex(VkExec *ex, int prim, int32_t count, int restart, int
int fullscreen = 0; // the wait is a full-screen draw's
if (oneshot > 0 && oneshot_budget_on()) {
budget = frame_wait();
wait_ms = oneshot_ms(budget, mesh, &s);
wait_ms = oneshot_ms(budget, mesh && !fb, &s);
if (wait_ms) t0 = kern_now_ns();
} else if (oneshot > 0) {
wait_ms = ONESHOT_WAIT_MS;
@@ -3984,7 +4023,7 @@ static int prepare_draw_ex(VkExec *ex, int prim, int32_t count, int restart, int
vk_exec_ahead(ex);
// A mesh pipeline that is known to be compiling gets no fast link, so the few milliseconds a draw waits
// for one are lost in every frame that meets it until the compile ends, 0.2 to 1.1 s later (MESH_WAIT_MS).
if (wait_ms && linking && mesh && vk_pipeline_building_ms(&s) > MESH_WAIT_MS) wait_ms = 0;
if (wait_ms && linking && mesh && !fb && vk_pipeline_building_ms(&s) > MESH_WAIT_MS) wait_ms = 0;
if (wait_ms) p = vk_pipeline_graphics_async(&s, &info, &pending, wait_ms);
}
if (t0 && budget) {
@@ -3999,7 +4038,7 @@ static int prepare_draw_ex(VkExec *ex, int prim, int32_t count, int restart, int
if (!w0) w0 = kern_now_ns();
if (oneshot_budget_on()) {
uint64_t *pool = frame_wait(), b0 = kern_now_ns();
p = vk_pipeline_graphics_async(&s, &info, &pending, oneshot_ms(pool, mesh, &s));
p = vk_pipeline_graphics_async(&s, &info, &pending, oneshot_ms(pool, mesh && !fb, &s));
pool_charge(pool, b0);
} else {
p = vk_pipeline_graphics_async(&s, &info, &pending, ONESHOT_WAIT_MS);
@@ -4045,7 +4084,7 @@ static int prepare_draw_ex(VkExec *ex, int prim, int32_t count, int restart, int
delta = cfg_bool("NX2NAVI_DYN_DELTA");
if (!delta) rt_log("vk_exec: every dynamic state recorded after each pipeline bind (NX2NAVI_DYN_DELTA=0)");
}
if (delta) vk_exec_frame.dyn_calls += (uint64_t)vk_pipeline_set_dynamic_delta(ex->cb, &s, &B.dyn, mesh,
if (delta) vk_exec_frame.dyn_calls += (uint64_t)vk_pipeline_set_dynamic_delta(ex->cb, &s, &B.dyn, mesh && !fb,
&vk_exec_frame.dyn_left);
else vk_pipeline_set_dynamic(ex->cb, &s);
ex->recorded++;
@@ -4068,7 +4107,7 @@ static int prepare_draw_ex(VkExec *ex, int prim, int32_t count, int restart, int
if (fbl & ex->fb_dirty) feedback_barrier(ex);
feedback_written(ex, &s, fbl);
if (!mesh) bind_vertex_buffers(ex);
if (vk_dump_armed) vk_dump_draw(ex, &s, p, info, mesh);
if (vk_dump_armed) vk_dump_draw(ex, &s, p, info, mesh && !fb);
if (vk_hdr_track) vk_hdr_draw(ex, info);
vk_exec_oq_draw(ex);
DRS.issued[mesh]++;
@@ -4951,12 +4990,266 @@ static void mesh_shading_rate(VkExec *ex) {
ex->recorded++;
}
// ---- mesh fallback: a mesh draw on a device without VK_EXT_mesh_shader (mesh_fallback, vk_device.c), or on any
// device with mesh_fallback = 1. The program's task and mesh stages run as compute dispatches that write the
// fallback's buffers, and an ordinary draw reads them (mesh_fb.c makes the modules, vk_mesh_fb.c holds the buffers):
// the draw's record its workgroup count, copied on the GPU when the game's own record holds it
// [task dispatch the task module, with the task stage's bindings
// scan lists the mesh workgroups the tasks started and counts them into the record]
// mesh dispatch the mesh module, with the mesh stage's bindings
// draw one instance per mesh workgroup, with the generated vertex stage and the program's
// fragment stage, the instance count from the record
// Every step reads what the one before wrote, so a full barrier stands between them, and the dispatches cannot be
// recorded inside a pass: each such draw ends the open pass, and the next draw begins it again. The buffers are
// reused by every draw (vk_mesh_fb.c), which the barrier before a draw's first dispatch makes safe: the previous
// draw's vertex stage has read them by then.
//
// The modules of a program are made once, on a thread of their own, with their compute pipelines (a mesh program of
// the game is up to 900,000 SPIR-V words, and its pipeline compiles for as long as the mesh pipeline it replaces):
// until they are ready the program's draws are skipped, like draws whose pipeline is being built. Without background
// pipeline builds (the probes) they are made at the draw.
typedef struct MfbCs {
const GfxShader *ms, *ts; // the key: the translated mesh and task shaders (ts NULL: none)
MeshFbInfo mi, ti;
volatile int32_t state; // 0 queued or being made, 1 ready, -1 failed
const GfxShader *cs_mesh, *cs_task; // the compute modules (gfx_shader_own)
struct MfbCs *next; // the worker's queue
} MfbCs;
typedef struct { const GfxShader *ms; uint32_t mask; uint8_t type[32]; const GfxShader *vs; } MfbVs;
#define MFB_TAB 4096
static MfbCs *mfb_cs_tab[MFB_TAB];
static MfbVs mfb_vs_tab[MFB_TAB];
static struct {
PlatMutex lock;
PlatCond wake;
PlatThread *th;
MfbCs *head, *tail;
uint64_t made, failed, pending_skips;
} MFB = {.lock = PLAT_MUTEX_INIT};
static uint32_t mfb_binding_base(const GfxShader *s) { // one past the highest binding the module declares
uint64_t n;
const uint32_t *w = gfx_shader_spirv(s, &n);
uint32_t top = 0;
for (uint64_t i = 5; i < n;) {
uint32_t op = w[i] & 0xffff, len = w[i] >> 16;
if (!len) break;
if (op == 71 && len == 4 && w[i + 2] == 33 && w[i + 3] > top) top = w[i + 3]; // OpDecorate Binding
i += len;
}
return top + 1;
}
// The compute module of stage (6 task, 7 mesh) of e, as a shader of the runtime's own with the stage's resources and
// the fallback's buffers.
static const GfxShader *mfb_module(const MfbCs *e, int stage, char *err, size_t cap) {
const GfxShader *src = stage == 6 ? e->ts : e->ms;
uint64_t n, on = 0;
const uint32_t *w = gfx_shader_spirv(src, &n);
uint32_t base = mfb_binding_base(src);
uint32_t *out = stage == 6 ? meshfb_task_cs(w, n, &e->ti, base, &on, err, cap)
: meshfb_mesh_cs(w, n, &e->mi, base, e->ti.payload_bytes / 4, &on, err, cap);
if (!out) return NULL;
static const int task_slots[1] = {MESHFB_CB_TASK}, mesh_slots[3] = {MESHFB_CB_TASK, MESHFB_CB_MESH, MESHFB_CB_LIST};
char *res = stage == 6 ? meshfb_res_json(gfx_shader_res_json(src), base, task_slots, 1)
: e->mi.has_task ? meshfb_res_json(gfx_shader_res_json(src), base, mesh_slots, 3)
: meshfb_res_json(gfx_shader_res_json(src), base + 1, mesh_slots + 1, 1);
const GfxShader *s = res ? gfx_shader_own(out, on, res) : NULL;
if (!s) snprintf(err, cap, "the converted module or its reflection is malformed");
free(out), free(res);
return s;
}
static void mfb_make(MfbCs *e) {
char err[200] = "";
const VkPipeInfo *info;
int ok = (e->cs_mesh = mfb_module(e, 7, err, sizeof err)) != NULL;
if (ok && e->ts) ok = (e->cs_task = mfb_module(e, 6, err, sizeof err)) != NULL;
if (ok && !vk_pipeline_compute(e->cs_mesh, &info)) ok = 0, snprintf(err, sizeof err, "the mesh stage's compute pipeline failed");
if (ok && e->ts && !vk_pipeline_compute(e->cs_task, &info)) ok = 0, snprintf(err, sizeof err, "the task stage's compute pipeline failed");
plat_mutex_lock(&MFB.lock);
uint64_t nth = ok ? ++MFB.made : ++MFB.failed;
plat_mutex_unlock(&MFB.lock);
if (!ok && (nth <= 8 || !(nth & (nth - 1))))
rt_log("vk: mesh fallback: a program's modules could not be made (%s): its draws are skipped (%llu such programs)", err,
(unsigned long long)nth);
__atomic_store_n(&e->state, ok ? 1 : -1, __ATOMIC_RELEASE);
}
static void mfb_worker(void *arg) {
(void)arg;
plat_thread_priority(PLAT_PRIO_BELOW_NORMAL); // as the pipeline workers' optimised builds: the game's threads go first
for (;;) {
plat_mutex_lock(&MFB.lock);
while (!MFB.head) plat_cond_wait(&MFB.wake, &MFB.lock);
MfbCs *e = MFB.head;
if (!(MFB.head = e->next)) MFB.tail = NULL;
plat_mutex_unlock(&MFB.lock);
mfb_make(e);
}
}
// The entry of (ms, ts), made and queued at its first draw; NULL (logged) when the program's layout is not carried.
static MfbCs *mfb_cs_get(VkExec *ex, const GfxShader *ms, const GfxShader *ts) {
uint32_t i = (uint32_t)((((uintptr_t)ms >> 4) * 0x9e3779b1u) ^ (((uintptr_t)ts >> 4) * 0x85ebca6bu)) & (MFB_TAB - 1);
for (uint32_t k = 0; k < MFB_TAB; k++, i = (i + 1) & (MFB_TAB - 1)) {
MfbCs *e = mfb_cs_tab[i];
if (e && (e->ms != ms || e->ts != ts)) continue;
if (e) return e;
const uint8_t *ctl, *code;
uint32_t len;
char err[160] = "";
e = calloc(1, sizeof *e);
if (!e) rt_fatal("vk: out of memory");
e->ms = ms, e->ts = ts, e->state = -1;
mfb_cs_tab[i] = e; // a program that is refused is not looked at again
int ok = find_stage_scan(ex->prog, NVN_STAGE_MESH, &ctl, &code, &len) && len >= 0x80 &&
meshfb_info(ctl, code, 7, &e->mi, err, sizeof err);
if (ok && ts) ok = find_stage_scan(ex->prog, NVN_STAGE_TASK, &ctl, &code, &len) && meshfb_info(ctl, code, 6, &e->ti, err, sizeof err);
if (ok && e->mi.has_task != (ts != NULL)) ok = 0, snprintf(err, sizeof err, "the mesh stage %s a task stage and the program has %s",
e->mi.has_task ? "expects" : "does not expect", ts ? "one" : "none");
if (!ok) {
warn_once(W_MESH, "mesh fallback: a program is not carried (%s): its draws are skipped", *err ? err : "no stage data");
return e;
}
e->state = 0;
if (!vk_pipeline_async_on()) {
mfb_make(e);
return e;
}
plat_mutex_lock(&MFB.lock);
if (!MFB.th) plat_cond_init(&MFB.wake), MFB.th = plat_thread_new(mfb_worker, NULL, "mesh fallback");
if (MFB.tail) MFB.tail->next = e;
else MFB.head = e;
MFB.tail = e;
plat_cond_signal(&MFB.wake);
plat_mutex_unlock(&MFB.lock);
return e;
}
rt_fatal("vk: mesh fallback: more than %d task and mesh program pairs", MFB_TAB);
}
// The vertex stage that draws e's mesh buffer slots for fragment shader fs: an output for every input fs reads, in
// its type.
static const GfxShader *mfb_vs_get(const MfbCs *e, const GfxShader *fs) {
GfxShaderIo fin[32];
uint8_t type[32] = {0};
uint32_t mask = fs ? gfx_shader_inputs(fs, fin) : 0;
for (int g = 0; g < 32; g++)
if (mask >> g & 1) type[g] = fin[g].type;
uint32_t i = (uint32_t)((((uintptr_t)e->ms >> 4) * 0x9e3779b1u) ^ (mask * 0x85ebca6bu)) & (MFB_TAB - 1);
for (uint32_t k = 0; k < MFB_TAB; k++, i = (i + 1) & (MFB_TAB - 1)) {
MfbVs *v = &mfb_vs_tab[i];
if (v->ms && (v->ms != e->ms || v->mask != mask || memcmp(v->type, type, 32))) continue;
if (v->ms) return v->vs;
static const char res[] = "{\"key\":\"meshfb\",\"stage\":\"vs\",\"notes\":[],\"cbufs\":[],\"sbufs\":[{\"binding\": 1, "
"\"slot\": -1, \"cb_slot\": 66, \"cb_offset\": 0, \"flags\": 0}],\"textures\":[],\"images\":[]}";
_Static_assert(MESHFB_CB_MESH == 66, "the vertex stage's reflection names the mesh buffer");
uint64_t n = 0;
uint32_t *w = meshfb_vs(&e->mi, mask, type, &n);
v->ms = e->ms, v->mask = mask, memcpy(v->type, type, 32);
v->vs = w ? gfx_shader_own(w, n, res) : NULL;
free(w);
if (!v->vs) warn_once(W_MESH, "mesh fallback: a vertex stage could not be made: its draws are skipped");
return v->vs;
}
rt_fatal("vk: mesh fallback: more than %d vertex stages", MFB_TAB);
}
// One of the draw's compute dispatches: module cs with the bindings of NVN stage `stage`, its group counts read from
// the record at `at`. 0 when its pipeline is not there.
static int mfb_dispatch(VkExec *ex, const GfxShader *cs, int stage, VkBuffer rec, VkDeviceSize at) {
const VkPipeInfo *info = NULL;
VkPipeline p = vk_pipeline_compute(cs, &info);
if (!p || !info || (info->pool && !pool_on())) return 0;
bind_pipeline(ex, VK_PIPELINE_BIND_POINT_COMPUTE, p);
mfb_stage = stage;
bind_resources(ex, info);
mfb_stage = 0;
VkPassMark *m = dispatch_mark(ex, cs, 0, 0, 0);
vkCmdDispatchIndirect(ex->cb, rec, at);
vk_pass_mark_end(ex->cb, m);
ex->recorded++;
return 1;
}
// A mesh draw of `count` workgroups, or of the count in the game's record at ind (4 bytes, GPU-written).
static void mfb_draw(VkExec *ex, uint32_t count, const VkMemSpan *ind) {
if (ex->render_disabled) {
vk_cond_draws_skip += ex->has_render_cond;
draw_skipped(ex, 1, SK_DISABLED);
return;
}
if (!ex->prog) { draw_skipped(ex, 1, SK_NO_PROGRAM); return; }
cb_res(ex);
dummies_init();
int has_ms, has_ts, has_fs, w_ms = 0, w_ts = 0, w_fs = 0;
const GfxShader *ms = shader_for(ex, NVN_STAGE_MESH, &has_ms, &w_ms, 0);
const GfxShader *ts = shader_for(ex, NVN_STAGE_TASK, &has_ts, &w_ts, 0);
const GfxShader *fs = shader_for(ex, NVN_STAGE_FS, &has_fs, &w_fs, 0);
if (!has_ms) { draw_skipped(ex, 1, SK_NO_VERTEX_STAGE); return; }
if (w_ms || w_ts || w_fs) {
vk_exec_ahead(ex);
perf_count(PERF_CNT_SHADER_SKIP);
draw_skipped(ex, 1, SK_SHADER_PENDING);
return;
}
if (!ms || (has_ts && !ts) || (has_fs && !fs)) { draw_skipped(ex, 1, SK_SHADER_FAILED); return; }
MfbCs *c = mfb_cs_get(ex, ms, ts);
int state = __atomic_load_n(&c->state, __ATOMIC_ACQUIRE);
if (!state && !vk_exec_targets_recent(ex, pipe_skip_mode() >= 1)) {
// a target the game does not render every frame: the draw may not come again, so it waits for its modules
// from the frame's wait pool, as such a draw waits for a pipeline (prepare_draw_ex)
uint64_t *pool = frame_wait(), t0 = kern_now_ns();
while (!state && pool_ms(pool, t0)) plat_sleep_ms(1), state = __atomic_load_n(&c->state, __ATOMIC_ACQUIRE);
pool_charge(pool, t0);
}
if (!state) { MFB.pending_skips++; draw_skipped(ex, 1, SK_PIPE_PENDING); return; }
const GfxShader *vs = state > 0 ? mfb_vs_get(c, fs) : NULL;
if (!vs || !vk_meshfb_init()) { draw_skipped(ex, 1, SK_PIPE_FAILED); return; }
uint32_t verts = c->mi.maxp * c->mi.ipp, slot_words = meshfb_slot_words(&c->mi);
uint32_t most = vk_ctx()->props.limits.maxComputeWorkGroupCount[0];
if (count > most) count = most; // a count the GPU wrote is not looked at here: the dispatch reads it as it is
VkBuffer rec;
int cmds;
vk_exec_barrier(ex); // ends the pass; the culling's record and the last draw's reads come before what follows
VkDeviceSize at = vk_meshfb_record(ex->cb, ind ? ind->buffer : VK_NULL_HANDLE, ind ? ind->offset : 0, count, ts != NULL,
verts, slot_words, &rec, &cmds);
ex->recorded += (uint32_t)cmds;
vk_exec_barrier(ex);
if (ts) {
if (!mfb_dispatch(ex, c->cs_task, NVN_STAGE_TASK, rec, at)) { draw_skipped(ex, 1, SK_PIPE_FAILED); return; }
vk_exec_barrier(ex);
vk_meshfb_scan(ex->cb, at, c->ti.payload_bytes / 4, c->ti.threads, slot_words, verts);
B.pipe[1] = VK_NULL_HANDLE; // the scan bound its own
ex->recorded += 4;
vk_exec_barrier(ex);
}
if (!mfb_dispatch(ex, c->cs_mesh, NVN_STAGE_MESH, rec, at + 16)) { draw_skipped(ex, 1, SK_PIPE_FAILED); return; }
vk_exec_barrier(ex);
vk_meshfb_note(ex->cb, at);
ex->recorded++;
vk_exec_barrier(ex);
c0_binst = 0;
mfb_cur_vs = vs;
int prim = c->mi.ipp == 1 ? NVN_DRAW_PRIMITIVE_POINTS : c->mi.ipp == 2 ? NVN_DRAW_PRIMITIVE_LINES : NVN_DRAW_PRIMITIVE_TRIANGLES;
if (!prepare_draw_ex(ex, prim, 0x7fffffff, 0, 2)) return;
vk_exec_frame.draws++;
vkCmdDrawIndirect(ex->cb, rec, at + 32, 1, sizeof(VkDrawIndirectCommand));
ex->recorded++;
}
static void op_draw_mesh(VkExec *ex, const p_draw_mesh *d) {
PFN_vkCmdDrawMeshTasksEXT fn = draw_mesh_fn();
if (!fn) return warn_once(W_MESH, "mesh draws need VK_EXT_mesh_shader: skipped");
static int on = -1;
if (on < 0) on = rt_knob("MESH", "1"); // NX2NAVI_MESH=0: mesh draws are skipped (diagnosis)
if (!d->count || !on) return;
if (vk_ctx()->mesh_fallback) {
if (d->first) warn_once(W_MESH, "a mesh draw from workgroup %u: drawn from workgroup 0", d->first);
mfb_draw(ex, d->count, NULL);
return;
}
PFN_vkCmdDrawMeshTasksEXT fn = draw_mesh_fn();
if (!fn) return warn_once(W_MESH, "mesh draws need VK_EXT_mesh_shader: skipped");
c0_binst = 0;
if (!prepare_draw_ex(ex, NVN_DRAW_PRIMITIVE_TRIANGLES, 0x7fffffff, 0, 1)) return;
if (d->first) warn_once(W_MESH, "a mesh draw from workgroup %u: drawn from workgroup 0", d->first);
@@ -4980,13 +5273,21 @@ static void op_draw_mesh(VkExec *ex, const p_draw_mesh *d) {
// MESH_RING entries outlast the frames in flight: an entry is rewritten only after a barrier behind its last draw.
enum { MESH_RING = 4096 };
static VkBuffer mesh_ring;
static VmaAllocation mesh_ring_alloc;
static uint32_t mesh_ring_next;
// NX2NAVI_GPU_PASSES: the workgroup counts of the indirect mesh draws of the timed frames, copied to host memory.
int vk_gpu_passes_capturing(void);
enum { MESH_COUNTS = 4096 };
static VkBuffer mesh_count_buf;
static VmaAllocation mesh_count_alloc;
static uint32_t *mesh_count_map, mesh_count_n, mesh_count_was;
static void mesh_ring_release(VkCtx *c) { // vk_exec_draw_shutdown: a probe ends with the device idle
if (mesh_ring) vmaDestroyBuffer(c->vma, mesh_ring, mesh_ring_alloc);
if (mesh_count_buf) vmaDestroyBuffer(c->vma, mesh_count_buf, mesh_count_alloc);
mesh_ring = mesh_count_buf = VK_NULL_HANDLE, mesh_count_map = NULL;
}
uint32_t vk_exec_mesh_counts(const uint32_t **out) {
*out = mesh_count_map;
return mesh_count_map ? mesh_count_n : 0;
@@ -5001,8 +5302,7 @@ static void mesh_count_note(VkExec *ex, const VkMemSpan *src) {
VmaAllocationCreateInfo ai = {.usage = VMA_MEMORY_USAGE_AUTO_PREFER_HOST,
.flags = VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT | VMA_ALLOCATION_CREATE_MAPPED_BIT};
VmaAllocationInfo info;
VmaAllocation a;
if (vmaCreateBuffer(vk_ctx()->vma, &bi, &ai, &mesh_count_buf, &a, &info) != VK_SUCCESS) return;
if (vmaCreateBuffer(vk_ctx()->vma, &bi, &ai, &mesh_count_buf, &mesh_count_alloc, &info) != VK_SUCCESS) return;
mesh_count_map = info.pMappedData;
}
if (mesh_count_n >= MESH_COUNTS) return;
@@ -5053,8 +5353,7 @@ static int mesh_ring_entry(VkExec *ex, const p_draw_mesh_ind *d, const VkMemSpan
VkBufferCreateInfo bi = {.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, .size = 12ull * MESH_RING,
.usage = VK_BUFFER_USAGE_INDIRECT_BUFFER_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT};
VmaAllocationCreateInfo ai = {.usage = VMA_MEMORY_USAGE_AUTO_PREFER_DEVICE};
VmaAllocation a;
if (vmaCreateBuffer(vk_ctx()->vma, &bi, &ai, &mesh_ring, &a, NULL) != VK_SUCCESS) {
if (vmaCreateBuffer(vk_ctx()->vma, &bi, &ai, &mesh_ring, &mesh_ring_alloc, NULL) != VK_SUCCESS) {
mesh_ring = VK_NULL_HANDLE;
return 0;
}
@@ -5107,7 +5406,8 @@ static void op_draw_mesh_ind(VkExec *ex, const p_draw_mesh_ind *d) {
if (vk_ctx()->has_mesh_shader)
fn = (PFN_vkCmdDrawMeshTasksIndirectEXT)vk_dpa(vk_ctx()->device, "vkCmdDrawMeshTasksIndirectEXT");
}
if (!fn) return warn_once(W_MESH, "mesh draws need VK_EXT_mesh_shader: skipped");
int fallback = vk_ctx()->mesh_fallback;
if (!fn && !fallback) return warn_once(W_MESH, "mesh draws need VK_EXT_mesh_shader: skipped");
static int on = -1;
if (on < 0) on = rt_knob("MESH", "1");
if (!on) return;
@@ -5117,6 +5417,12 @@ static void op_draw_mesh_ind(VkExec *ex, const p_draw_mesh_ind *d) {
(unsigned long long)d->ind);
c0_binst = 0;
VK_MEM_USE(ip);
if (fallback) {
mfb_draw(ex, 0, &ip);
vk_mem_count(ip.cls, VK_MUSE_IND, 4);
VK_MEM_BOUND(ip);
return;
}
VkDeviceSize at;
if (!mesh_ring_entry(ex, d, &ip, &at)) return warn_once(W_MESH, "the indirect mesh ring could not be created: skipped");
if (!prepare_draw_ex(ex, NVN_DRAW_PRIMITIVE_TRIANGLES, 0x7fffffff, 0, 1)) return;

215
runtime/vk_mesh_fb.c Normal file
View File

@@ -0,0 +1,215 @@
// The mesh fallback's buffers and records. A task or mesh program runs as a compute dispatch that writes a storage
// buffer (mesh_fb.c), and the draw reads it:
// task buffer a task workgroup's payload at its index, payload-size apart
// list per mesh workgroup after a task stage: its task and its index among that task's (meshfb_scan.comp)
// mesh buffer a mesh workgroup's vertex lines and primitive area at its slot
// records per draw 64 bytes: the indirect arguments of its task dispatch, its mesh dispatch and its draw
// The three buffers are reused by every draw: a draw's dispatches write them and its vertex stage has read them
// before the next draw's dispatches run, with a barrier in between (vk_exec_draw.c mfb_draw), so their size is what
// the largest draw needs, not what a frame's draws need together. mesh_fallback_mb sets it: an eighth for the task
// buffer, the rest for the mesh buffer. Workgroups past either buffer's end write nothing and draw nothing (the
// modules check the buffer's length); the draws that happened to are counted and the log names the size they needed.
// A record is used by one draw and written again only after MESHFB_RECS further draws.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "cfg.h"
#include "host.h"
#include "mesh_fb.h"
#include "vk.h"
#include "vk_mesh_fb.h"
#include "meshfb_scan_spv.h"
enum { MESHFB_RECS = 4096, REC_BYTES = 64 };
static struct {
int tried, ok;
VkBuffer task, mesh, list, rec, stat;
VmaAllocation task_a, mesh_a, list_a, rec_a, stat_a;
VkDeviceSize task_bytes, mesh_bytes, list_bytes; // as one storage binding spans them
const uint32_t *stat_map; // per record: mesh workgroups and task workgroups it asked for
uint32_t cap_slots[MESHFB_RECS], slot_words[MESHFB_RECS], cap_tasks[MESHFB_RECS]; // what the record's draw had room for
uint8_t noted[MESHFB_RECS];
uint32_t next;
uint64_t draws, short_draws, worst_bytes;
PFN_vkCmdPushDescriptorSetKHR push;
VkDescriptorSetLayout dsl;
VkPipelineLayout layout;
VkPipeline scan;
} M;
static int make_buffer(VkCtx *c, VkDeviceSize size, VkBufferUsageFlags usage, int host, VkBuffer *b, VmaAllocation *a, void **map) {
VkBufferCreateInfo bi = {.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, .size = size, .usage = usage};
VmaAllocationCreateInfo ai = {.usage = host ? VMA_MEMORY_USAGE_AUTO_PREFER_HOST : VMA_MEMORY_USAGE_AUTO_PREFER_DEVICE,
.flags = host ? VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT | VMA_ALLOCATION_CREATE_MAPPED_BIT : 0};
VmaAllocationInfo info;
if (vmaCreateBuffer(c->vma, &bi, &ai, b, a, &info) != VK_SUCCESS) return *b = VK_NULL_HANDLE, 0;
if (map) *map = info.pMappedData;
return !host || info.pMappedData;
}
int vk_meshfb_init(void) {
if (M.tried) return M.ok;
M.tried = 1;
VkCtx *c = vk_ctx();
if (!c) return 0;
uint64_t mb = (uint64_t)cfg_int("NX2NAVI_MESH_FALLBACK_MB");
if (mb < 16) mb = 16;
if (mb > 4096) mb = 4096;
VkDeviceSize range = c->props.limits.maxStorageBufferRange & ~(VkDeviceSize)3, task = (mb << 20) / 8, mesh = (mb << 20) - task;
VkDeviceSize list = mesh / 128; // an entry per slot of 512 bytes, and no mesh program's slot is smaller
void *map = NULL;
const VkBufferUsageFlags st = VK_BUFFER_USAGE_STORAGE_BUFFER_BIT;
if (!make_buffer(c, task, st, 0, &M.task, &M.task_a, NULL) || !make_buffer(c, mesh, st, 0, &M.mesh, &M.mesh_a, NULL) ||
!make_buffer(c, list, st, 0, &M.list, &M.list_a, NULL) ||
!make_buffer(c, (VkDeviceSize)MESHFB_RECS * REC_BYTES,
st | VK_BUFFER_USAGE_INDIRECT_BUFFER_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT | VK_BUFFER_USAGE_TRANSFER_SRC_BIT, 0,
&M.rec, &M.rec_a, NULL) ||
!make_buffer(c, (VkDeviceSize)MESHFB_RECS * 8, VK_BUFFER_USAGE_TRANSFER_DST_BIT, 1, &M.stat, &M.stat_a, &map)) {
rt_log("vk: mesh fallback: its buffers (%llu MiB, mesh_fallback_mb) could not be allocated: task and mesh draws are skipped",
(unsigned long long)mb);
return 0;
}
M.stat_map = map;
memset(map, 0, (size_t)MESHFB_RECS * 8);
M.task_bytes = task < range ? task : range, M.mesh_bytes = mesh < range ? mesh : range, M.list_bytes = list < range ? list : range;
M.push = (PFN_vkCmdPushDescriptorSetKHR)vk_dpa(c->device, "vkCmdPushDescriptorSetKHR");
VkDescriptorSetLayoutBinding b[4];
for (int i = 0; i < 4; i++)
b[i] = (VkDescriptorSetLayoutBinding){.binding = (uint32_t)i, .descriptorType = VK_DESCRIPTOR_TYPE_STORAGE_BUFFER,
.descriptorCount = 1, .stageFlags = VK_SHADER_STAGE_COMPUTE_BIT};
VkDescriptorSetLayoutCreateInfo dci = {.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_SET_LAYOUT_CREATE_INFO,
.flags = VK_DESCRIPTOR_SET_LAYOUT_CREATE_PUSH_DESCRIPTOR_BIT_KHR,
.bindingCount = 4, .pBindings = b};
VkPushConstantRange pr = {VK_SHADER_STAGE_COMPUTE_BIT, 0, 24};
VkShaderModuleCreateInfo mi = {.sType = VK_STRUCTURE_TYPE_SHADER_MODULE_CREATE_INFO, .codeSize = sizeof meshfb_scan_spv,
.pCode = meshfb_scan_spv};
VkShaderModule m = VK_NULL_HANDLE;
if (!M.push || vkCreateDescriptorSetLayout(c->device, &dci, NULL, &M.dsl) != VK_SUCCESS) goto fail;
VkPipelineLayoutCreateInfo li = {.sType = VK_STRUCTURE_TYPE_PIPELINE_LAYOUT_CREATE_INFO, .setLayoutCount = 1,
.pSetLayouts = &M.dsl, .pushConstantRangeCount = 1, .pPushConstantRanges = &pr};
if (vkCreatePipelineLayout(c->device, &li, NULL, &M.layout) != VK_SUCCESS ||
vkCreateShaderModule(c->device, &mi, NULL, &m) != VK_SUCCESS)
goto fail;
VkComputePipelineCreateInfo ci = {.sType = VK_STRUCTURE_TYPE_COMPUTE_PIPELINE_CREATE_INFO,
.stage = {.sType = VK_STRUCTURE_TYPE_PIPELINE_SHADER_STAGE_CREATE_INFO,
.stage = VK_SHADER_STAGE_COMPUTE_BIT, .module = m, .pName = "main"},
.layout = M.layout};
VkResult res = vkCreateComputePipelines(c->device, VK_NULL_HANDLE, 1, &ci, NULL, &M.scan);
vkDestroyShaderModule(c->device, m, NULL);
if (res != VK_SUCCESS) goto fail;
rt_log("vk: mesh fallback: %llu MiB for task payloads, %llu MiB for mesh output (mesh_fallback_mb = %llu)",
(unsigned long long)(M.task_bytes >> 20), (unsigned long long)(M.mesh_bytes >> 20), (unsigned long long)mb);
return M.ok = 1;
fail:
rt_log("vk: mesh fallback: the scan pipeline could not be created: task and mesh draws are skipped");
return 0;
}
VkDescriptorBufferInfo vk_meshfb_buffer(int cb_slot) {
if (cb_slot == MESHFB_CB_TASK) return (VkDescriptorBufferInfo){M.task, 0, M.task_bytes};
if (cb_slot == MESHFB_CB_LIST) return (VkDescriptorBufferInfo){M.list, 0, M.list_bytes};
return (VkDescriptorBufferInfo){M.mesh, 0, M.mesh_bytes};
}
uint64_t vk_meshfb_mesh_words(void) { return M.mesh_bytes / 4; }
// The record about to be written again: what its last draw asked for against what it had room for.
static void record_check(uint32_t i) {
if (!M.noted[i]) return;
M.noted[i] = 0;
uint32_t want = M.stat_map[2 * i], tasks = M.stat_map[2 * i + 1];
if (want <= M.cap_slots[i] && tasks <= M.cap_tasks[i]) return;
uint64_t need = (uint64_t)want * M.slot_words[i] * 4;
if (need > M.worst_bytes) M.worst_bytes = need;
uint64_t n = ++M.short_draws;
if (n <= 4 || !(n & (n - 1))) {
char from[96] = "";
if (tasks) snprintf(from, sizeof from, " from %u task workgroups (room for %u)", tasks, M.cap_tasks[i]);
rt_log("vk: mesh fallback: a draw's programs emitted %u mesh workgroups of %u bytes%s, and the mesh buffer holds %u: "
"what did not fit was not drawn (%llu such draws of %llu; mesh_fallback_mb of %llu or more would hold the "
"largest so far)", want, M.slot_words[i] * 4, from, M.cap_slots[i], (unsigned long long)n,
(unsigned long long)M.draws, (unsigned long long)((M.worst_bytes * 8 / 7 >> 20) + 1));
}
}
VkDeviceSize vk_meshfb_record(VkCommandBuffer cb, VkBuffer src, VkDeviceSize src_off, uint32_t count, int has_task,
uint32_t verts, uint32_t slot_words, VkBuffer *rec, int *cmds) {
uint32_t i = M.next++ % MESHFB_RECS;
VkDeviceSize at = (VkDeviceSize)i * REC_BYTES;
record_check(i);
M.draws++;
M.slot_words[i] = slot_words, M.cap_slots[i] = (uint32_t)(M.mesh_bytes / 4 / (slot_words ? slot_words : 1));
M.cap_tasks[i] = 0xffffffffu; // vk_meshfb_scan knows the payload size
if (has_task && M.cap_slots[i] > M.list_bytes / 4) M.cap_slots[i] = (uint32_t)(M.list_bytes / 4);
*rec = M.rec;
int n = 0;
vkCmdFillBuffer(cb, M.rec, at, REC_BYTES, 1), n++;
// where the workgroup count goes: the task dispatch, or the mesh dispatch, the draw's instances and the report
VkDeviceSize to[3] = {has_task ? at : at + 16, at + 36, at + 48};
int nto = has_task ? 1 : 3;
if (!has_task) {
vkCmdFillBuffer(cb, M.rec, at + 32, 4, verts), n++;
vkCmdFillBuffer(cb, M.rec, at + 40, 8, 0), n++;
vkCmdFillBuffer(cb, M.rec, at + 52, 4, 0), n++;
}
if (src) {
VkBufferCopy cp[3];
for (int k = 0; k < nto; k++) cp[k] = (VkBufferCopy){src_off, to[k], 4};
vkCmdCopyBuffer(cb, src, M.rec, (uint32_t)nto, cp), n++;
} else {
for (int k = 0; k < nto; k++) vkCmdFillBuffer(cb, M.rec, to[k], 4, count), n++;
}
*cmds = n;
return at;
}
void vk_meshfb_scan(VkCommandBuffer cb, VkDeviceSize at, uint32_t task_words, uint32_t fan, uint32_t slot_words, uint32_t verts) {
uint32_t i = (uint32_t)(at / REC_BYTES);
M.cap_tasks[i] = (uint32_t)(M.task_bytes / 4 / (task_words ? task_words : 1));
VkDescriptorBufferInfo bi[4] = {{M.rec, 0, VK_WHOLE_SIZE}, {M.task, 0, M.task_bytes}, {M.list, 0, M.list_bytes},
{M.mesh, 0, M.mesh_bytes}};
VkWriteDescriptorSet w[4];
for (int k = 0; k < 4; k++)
w[k] = (VkWriteDescriptorSet){.sType = VK_STRUCTURE_TYPE_WRITE_DESCRIPTOR_SET, .dstBinding = (uint32_t)k,
.descriptorCount = 1, .descriptorType = VK_DESCRIPTOR_TYPE_STORAGE_BUFFER,
.pBufferInfo = &bi[k]};
uint32_t pc[6] = {(uint32_t)(at / 4), task_words, fan, slot_words, verts, vk_ctx()->props.limits.maxComputeWorkGroupCount[0]};
vkCmdBindPipeline(cb, VK_PIPELINE_BIND_POINT_COMPUTE, M.scan);
rec_push(M.push, cb, VK_PIPELINE_BIND_POINT_COMPUTE, M.layout, 0, 4, w);
vkCmdPushConstants(cb, M.layout, VK_SHADER_STAGE_COMPUTE_BIT, 0, sizeof pc, pc);
vkCmdDispatch(cb, 1, 1, 1);
}
void vk_meshfb_note(VkCommandBuffer cb, VkDeviceSize at) {
uint32_t i = (uint32_t)(at / REC_BYTES);
VkBufferCopy cp = {at + 48, (VkDeviceSize)i * 8, 8};
vkCmdCopyBuffer(cb, M.rec, M.stat, 1, &cp);
M.noted[i] = 1;
}
uint64_t vk_meshfb_short(void) {
if (!M.ok) return 0;
for (uint32_t i = 0; i < MESHFB_RECS; i++) record_check(i);
return M.short_draws;
}
void vk_meshfb_shutdown(void) {
VkCtx *c = vk_ctx();
if (!c || !M.tried) return;
if (vk_meshfb_short())
rt_log("vk: mesh fallback: %llu of %llu draws emitted more than the buffers hold (mesh_fallback_mb of %llu or more would "
"hold the largest)", (unsigned long long)M.short_draws, (unsigned long long)M.draws,
(unsigned long long)((M.worst_bytes * 8 / 7 >> 20) + 1));
if (M.scan) vkDestroyPipeline(c->device, M.scan, NULL);
if (M.layout) vkDestroyPipelineLayout(c->device, M.layout, NULL);
if (M.dsl) vkDestroyDescriptorSetLayout(c->device, M.dsl, NULL);
if (M.task) vmaDestroyBuffer(c->vma, M.task, M.task_a);
if (M.mesh) vmaDestroyBuffer(c->vma, M.mesh, M.mesh_a);
if (M.list) vmaDestroyBuffer(c->vma, M.list, M.list_a);
if (M.rec) vmaDestroyBuffer(c->vma, M.rec, M.rec_a);
if (M.stat) vmaDestroyBuffer(c->vma, M.stat, M.stat_a);
memset(&M, 0, sizeof M);
}

29
runtime/vk_mesh_fb.h Normal file
View File

@@ -0,0 +1,29 @@
// The mesh fallback's buffers and its records (vk_mesh_fb.c); the modules are mesh_fb.c's, the draws
// vk_exec_draw.c's ("mesh fallback").
#pragma once
#include "vk.h"
// Creates the buffers and the scan pipeline on first use. 0 when that failed (logged once): the draws are skipped.
int vk_meshfb_init(void);
// The buffer a module's extra storage binding reads or writes (mesh_fb.h MESHFB_CB_*), whole or as much of it as one
// storage binding may span.
VkDescriptorBufferInfo vk_meshfb_buffer(int cb_slot);
uint64_t vk_meshfb_mesh_words(void); // the mesh buffer's size as a binding sees it, in words
// A draw's record, 64 bytes of the record buffer (*rec) at the offset returned: the task dispatch at +0, the mesh
// dispatch at +16, the draw (VkDrawIndirectCommand) at +32. Records the fills and copies that start it, outside a
// pass: the workgroup count from 4 bytes at src_off of src, or `count` when src is VK_NULL_HANDLE. Without a task
// stage that count is the mesh dispatch and the draw's instance count; with one it is the task dispatch, and
// vk_meshfb_scan writes the rest. slot_words: of the draw's mesh program, for the report of a draw the mesh buffer
// was too small for. Returns the number of commands recorded in *cmds.
VkDeviceSize vk_meshfb_record(VkCommandBuffer cb, VkBuffer src, VkDeviceSize src_off, uint32_t count, int has_task,
uint32_t verts, uint32_t slot_words, VkBuffer *rec, int *cmds);
// The pass between the task and the mesh dispatch (shaders/meshfb_scan.comp): binds its own compute pipeline and set.
void vk_meshfb_scan(VkCommandBuffer cb, VkDeviceSize at, uint32_t task_words, uint32_t fan, uint32_t slot_words, uint32_t verts);
// After the record is complete (the copies or the scan, and a barrier): keeps what the draw asked for where the host
// reads it later. One copy command.
void vk_meshfb_note(VkCommandBuffer cb, VkDeviceSize at);
// Draws so far whose programs emitted more than the buffers hold, the records not looked at yet included: call it
// with the device idle (a probe, the shutdown).
uint64_t vk_meshfb_short(void);
void vk_meshfb_shutdown(void); // the device must be idle

View File

@@ -3508,6 +3508,7 @@ static void keys_note(const VkPipeState *s) {
static const uint8_t none[32];
if (s->fs && !memcmp(gfx_shader_input_id(s->fs), none, 32)) return; // the runtime's own shader (gfx_shader_own)
if (s->ts && !memcmp(gfx_shader_input_id(s->ts), none, 32)) return;
if (!memcmp(gfx_shader_input_id(s->vs), none, 32)) return; // the mesh fallback's vertex stage (vk_exec_draw.c mfb_vs_get)
KeyRec r;
rec_fill(&r, s);
if (!seen_add(rec_hash(&r))) return;

View File

@@ -49,7 +49,7 @@ enum {
R_SetRasterizerDiscardEnable, R_SetScissorWithCount, R_SetStencilCompareMask, R_SetStencilOp, R_SetStencilReference,
R_SetStencilTestEnable, R_SetStencilWriteMask, R_SetViewportWithCount, R_WriteTimestamp, R_WriteTimestamp2,
R_PushTpl, R_Push, R_VertexInput, R_Bool, R_PolygonMode, R_BlendEnable, R_BlendEquation, R_WriteMask, R_EndCb,
R_PassMark, R_DrawMesh, R_DrawMeshInd, R_ShadingRate,
R_PassMark, R_DrawMesh, R_DrawMeshInd, R_ShadingRate, R_DrawIndirect,
};
static uint64_t now_ns(void) { return plat_now_ns(); }
@@ -347,6 +347,9 @@ FIXED(DrawIndexed, (VkCommandBuffer cb, uint32_t n, uint32_t i, uint32_t f, int3
#define PACK_DrawIndexedIndirect *a = (A_DrawIndexedIndirect){b, o, n, s}
FIXED(DrawIndexedIndirect, (VkCommandBuffer cb, VkBuffer b, VkDeviceSize o, uint32_t n, uint32_t s), (cb, b, o, n, s),
VkBuffer b; VkDeviceSize o; uint32_t n, s)
#define PACK_DrawIndirect *a = (A_DrawIndirect){b, o, n, s}
FIXED(DrawIndirect, (VkCommandBuffer cb, VkBuffer b, VkDeviceSize o, uint32_t n, uint32_t s), (cb, b, o, n, s),
VkBuffer b; VkDeviceSize o; uint32_t n, s)
#define PACK_EndQuery *a = (A_EndQuery){p, q}
FIXED(EndQuery, (VkCommandBuffer cb, VkQueryPool p, uint32_t q), (cb, p, q), VkQueryPool p; uint32_t q)
#define PACK_EndRendering (void)a
@@ -705,6 +708,7 @@ static void exec_packet(uint32_t op, void *p) {
F(Draw, a->n, a->i, a->f, a->fi)
F(DrawIndexed, a->n, a->i, a->f, a->v, a->fi)
F(DrawIndexedIndirect, a->b, a->o, a->n, a->s)
F(DrawIndirect, a->b, a->o, a->n, a->s)
F(EndQuery, a->p, a->q)
F(FillBuffer, a->b, a->o, a->n, a->v)
F(ResetQueryPool, a->p, a->f, a->n)

View File

@@ -80,6 +80,7 @@ void rec_pass_mark(VkCommandBuffer cb, uint32_t id, int end);
X(Draw, (VkCommandBuffer cb, uint32_t n, uint32_t i, uint32_t f, uint32_t fi), (cb, n, i, f, fi)) \
X(DrawIndexed, (VkCommandBuffer cb, uint32_t n, uint32_t i, uint32_t f, int32_t v, uint32_t fi), (cb, n, i, f, v, fi)) \
X(DrawIndexedIndirect, (VkCommandBuffer cb, VkBuffer b, VkDeviceSize o, uint32_t n, uint32_t s), (cb, b, o, n, s)) \
X(DrawIndirect, (VkCommandBuffer cb, VkBuffer b, VkDeviceSize o, uint32_t n, uint32_t s), (cb, b, o, n, s)) \
X(EndQuery, (VkCommandBuffer cb, VkQueryPool p, uint32_t q), (cb, p, q)) \
X(EndRendering, (VkCommandBuffer cb), (cb)) \
X(FillBuffer, (VkCommandBuffer cb, VkBuffer b, VkDeviceSize o, VkDeviceSize n, uint32_t v), (cb, b, o, n, v)) \
@@ -137,6 +138,7 @@ REC_CMDS(REC_DECL)
#define vkCmdDraw rc_vkCmdDraw
#define vkCmdDrawIndexed rc_vkCmdDrawIndexed
#define vkCmdDrawIndexedIndirect rc_vkCmdDrawIndexedIndirect
#define vkCmdDrawIndirect rc_vkCmdDrawIndirect
#define vkCmdEndQuery rc_vkCmdEndQuery
#define vkCmdEndRendering rc_vkCmdEndRendering
#define vkCmdFillBuffer rc_vkCmdFillBuffer

View File

@@ -5,6 +5,7 @@ The repository holds the shaders as source only. Each becomes runtime/<name>_spv
missing and the stale ones before it compiles the runtime. Three kinds of source:
runtime/shaders/<name>.comp, .vert, .frag GLSL, compiled with glslc (one name per shader) and checked with spirv-val
runtime/shaders/<name>.mesh, .task the same for the mesh fallback probe's mesh and task shaders (SPIR-V 1.5)
runtime/shaders/<name>.spvasm SPIR-V assembly, assembled with spirv-as and checked with spirv-val
tools/make_dlaa_mask.py, make_mvobj_fs.py shaders assembled in Python (ASSEMBLED); they need no other tool
@@ -34,7 +35,8 @@ TOOLS = Path(__file__).resolve().parent
ROOT = TOOLS.parent
BIN = Path(r'C:\msys64\ucrt64\bin') # Windows; elsewhere, and without MSYS2 there, the tools come from PATH
TARGET = 'vulkan1.1'
STAGES = {'.comp': 'compute', '.vert': 'vertex', '.frag': 'fragment'}
STAGES = {'.comp': 'compute', '.vert': 'vertex', '.frag': 'fragment', '.mesh': 'mesh', '.task': 'task'}
MESH_TARGET = 'vulkan1.2' # mesh and task shaders need SPIR-V 1.4 or later
ASM = '.spvasm'
ASM_TARGET = 'vulkan1.3'
# header name -> the script beside this one that assembles it; the script's HEADER is the header's text
@@ -138,7 +140,7 @@ def words_of(src):
else:
# "// build_shaders: keep functions" in a source: no optimisation, so its functions stay functions (the HDR
# tonemap, which runtime/vk_hdr.c splices into the game's shader by function)
target = TARGET
target = MESH_TARGET if src.suffix in ('.mesh', '.task') else TARGET
opt = [] if '// build_shaders: keep functions' in src.read_text(encoding='utf-8') else ['-O']
subprocess.run([tool('glslc'), *opt, f'--target-env={target}', f'-fshader-stage={STAGES[src.suffix]}',
'-o', str(spv), str(src)], check=True)