mirror of
https://github.com/tangerineturtle1/NX2navi.git
synced 2026-11-18 21:49:57 -04:00
Merge the mesh shader fallback: task and mesh programs drawn through compute passes and an ordinary draw where the card has no mesh shaders.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
@@ -133,6 +133,28 @@ keeps per-core data without locks and relies on that, so the host models it (`ru
|
||||
ring of Vulkan `{x, y, z}` records filled on the GPU. The copy cannot be recorded inside a pass, so the first such
|
||||
draw of a pass fills the records of the mesh draws that follow it too, and the pass is not cut at each of them
|
||||
(`runtime/vk_exec_draw.c` `mesh_ring_entry`).
|
||||
- Mesh fallback (`mesh_fallback`: `auto` on a device without the extension, `1` on any device for testing): the task
|
||||
and mesh stages run as compute dispatches and the result is drawn as ordinary triangles.
|
||||
- The modules come from the translations the shader cache already holds, not from the translator
|
||||
(`runtime/mesh_fb.c`): a translated task or mesh module is the program as a compute function over shared memory
|
||||
plus an entry function that hands that memory to the mesh pipeline, and the fallback replaces the entry function
|
||||
by one that copies the memory into a storage buffer. The translator, its identity and every cache key stay as
|
||||
they are, so a device with mesh shaders keeps its shaders and never sees these modules, and a device without
|
||||
needs no other cache or prepared shader list.
|
||||
- A draw is: its record (the workgroup count, copied on the GPU when the game's record holds it); the task
|
||||
dispatch, which writes each task workgroup's payload into the task buffer; a scan (`shaders/meshfb_scan.comp`)
|
||||
that lists the mesh workgroups the tasks started, in task order, and writes their number into the record; the
|
||||
mesh dispatch, which writes each workgroup's vertex lines and primitive bytes into its slot of the mesh buffer;
|
||||
and an instanced indirect draw, one instance per slot, whose generated vertex stage looks the vertex of each
|
||||
primitive corner up in the slot (`runtime/vk_exec_draw.c` `mfb_draw`, `runtime/vk_mesh_fb.c`).
|
||||
- Each step reads what the one before wrote, so full barriers stand between them and every such draw ends its pass:
|
||||
a village frame's 45 mesh draws are 45 passes cut, 90 to 135 dispatches and about 270 barriers more than with mesh
|
||||
shaders. The buffers (`mesh_fallback_mb`, 128 MiB) are reused by every draw, so they hold the largest draw, not
|
||||
a frame; a draw that emits more loses what does not fit, and the log says how much it needed.
|
||||
- A program's modules and compute pipelines are made on a thread of their own at its first draw, which is skipped
|
||||
until they are ready; they are not recorded for the next start, where the pipeline cache makes them quick.
|
||||
Not carried: the far shadow cascades' coarser shading rate, a draw's first workgroup other than 0, and more
|
||||
workgroups in one draw than one compute dispatch may have.
|
||||
- Pipelines: `VK_EXT_graphics_pipeline_library` where the driver has it: stage libraries compiled ahead, a fast link
|
||||
at a new state, an optimised pipeline replacing it from a background thread. A draw whose pipeline is not ready is
|
||||
skipped when its pass is one the game renders frame after frame: one of its targets was rendered in the last few
|
||||
|
||||
@@ -484,6 +484,12 @@ Runtime probes, each a mode of `nx2navi.exe` (the list is at the top of `runtime
|
||||
part of `--probe-vkexec` translate shaders of the census below: they look for `build/<ver>/shaders/jobs.json` of the
|
||||
exe's own version first, then of the other one.
|
||||
`--probe-vkmem` also checks CPU-backed texture uploads and rewrites, GPU-written texture aliases, and memory write watches.
|
||||
`--probe-meshfb` (no game files) draws mesh programs of the runtime's own, with and without a task stage and with
|
||||
direct and GPU-written workgroup counts, through `VK_EXT_mesh_shader` and through the mesh fallback, and compares the
|
||||
pictures; on a device without mesh shaders only the fallback draws. A change to `runtime/mesh_fb.c`,
|
||||
`runtime/vk_mesh_fb.c` or the fallback's part of `runtime/vk_exec_draw.c` passes it. With
|
||||
`NX2NAVI_PROBE_MESHFB_DIR=<folder>` it converts the game's own task and mesh translations instead, for `spirv-val`
|
||||
(the top of `runtime/mesh_fb_probe.c` says what the folder holds).
|
||||
A change to the texture detail (`runtime/vk_texture.c` "texture detail": `detail_skip`, `upload`, `astc_bl_regions`)
|
||||
passes it three times, with `NX2NAVI_TEXTURE_DETAIL` set to `full`, `reduced` and `low`: each time ASTC textures are
|
||||
uploaded and rewritten, and every level and layer of their images must hold the game's level it stands for.
|
||||
|
||||
@@ -876,6 +876,14 @@
|
||||
# first one, so their draw pass is not ended and begun again at each of them (about 40 passes fewer a frame in a
|
||||
# village). 0: each such draw ends its pass and prepares its own count, as before.
|
||||
# mesh_ahead = 1
|
||||
# Graphics cards without mesh shaders (GTX 10 series, RX 500 and 5000 series, many integrated ones): the game's mesh
|
||||
# draws, most of the world, are computed in compute passes and drawn as ordinary triangles. auto (default) does that
|
||||
# only on such a card; 1 does it on every card (for testing: it is slower than mesh shaders); 0 never, and such a
|
||||
# card then draws no world.
|
||||
# mesh_fallback = auto
|
||||
# Video memory in MB for what those compute passes hand to the draws. A draw that needs more loses the geometry that
|
||||
# does not fit, and the log says so.
|
||||
# mesh_fallback_mb = 128
|
||||
# Windows, 1 (experimental): the game's heap is committed a 64 KiB run at a time where it is first used, instead of
|
||||
# all of it when the game sizes it. Windows counts committed memory against RAM plus the page file whether it is used
|
||||
# or not, and in Castle Town after Hyrule Field about 3.4 of the heap's 6.8 GiB had never been used. 0: committed up
|
||||
|
||||
@@ -36,7 +36,7 @@ static int save_ok = 1; // the last cfg_save_to wrote (o
|
||||
static const char *const known[] = {
|
||||
"CORES", "TM_DISPATCH", "NVN2_CS_TABLE", "SKIP_SHADERS", "MOVIE_DEBAND", "MOVIE_FPS", "MOVIE_SKIP", "MOVIE_OVERLAY", "MOVIES", "NPLN", "NVN2_PROGRAMS", "NVN_TRAP", "OCARINA_REALTIME", "ONESHOT_BUDGET", "ONESHOT_WAIT_MS", "OCARINA_STEP", "PHYSICS_STEPS", "PICKUP_COUNT", "STEP_STEADY", "VI_HZ", "VI_VRR", "TIME_DUMP", "SHADER_DUMP", "SHOT_AFTER", "SHOT_DIR", "SHOT_EVERY", // Switch 2 port: movie skip (romfs.c), NVN2 proc traps (nvn_procs.c)
|
||||
"AB", "AMIIBO_DIR", "ANISO", "APP_POOL", "ASPECT", "ASPECT_PATCH", "ASSUME_FOCUS", "ASTC", "ASTC_BC", "ASTC_TIME", "ASYNC_PREEMPT", "AUDIO", "AUDIO_CHANNELS", "AUDIO_DEVICE", "AUDIO_GAME_PRIO", "AUDIO_OFF", "AUDIO_PREFILL_MS", "AUDIO_STATS",
|
||||
"BARRIER_GAME", "MESH_AHEAD",
|
||||
"BARRIER_GAME", "MESH_AHEAD", "MESH_FALLBACK", "MESH_FALLBACK_MB", "PROBE_MESHFB_DIR",
|
||||
"BACKGROUND_WINDOW", "BIND_A", "BIND_B", "BIND_DOWN", "BIND_L", "BIND_LCLICK", "BIND_LEFT", "BIND_LSTICK_DOWN", "BIND_LSTICK_LEFT",
|
||||
"BIND_LSTICK_RIGHT", "BIND_LSTICK_UP", "BIND_MINUS",
|
||||
"BIND_PLUS", "BIND_R", "BIND_RCLICK", "BIND_RIGHT", "BIND_RSTICK_DOWN", "BIND_RSTICK_LEFT", "BIND_RSTICK_RIGHT", "BIND_RSTICK_UP",
|
||||
@@ -141,7 +141,7 @@ static const struct { const char *key, *section; } sections[] = {
|
||||
{"VK_DYN_VERTEX", "performance"}, {"VK_DYN_A2C", "performance"}, {"VK_SUBGROUP_PIN", "performance"}, {"VK_RECONVERGE", "performance"},
|
||||
{"VK_NAN_PRESERVE", "performance"}, {"VK_NAN_BITS", "performance"},
|
||||
{"TRACE", "diagnostics"}, {"FLIGHT", "diagnostics"}, {"PASS_CRUMBS", "diagnostics"}, {"FLIGHT_SPIKE_MS", "diagnostics"},
|
||||
{"WB_FENCE_LIMIT", "diagnostics"}, {"WB_MERGE", "diagnostics"}, {"WB_EXACT", "diagnostics"}, {"MEM_CENSUS", "diagnostics"}, {"WB_NO_REUP", "diagnostics"}, {"WB_PROBE", "diagnostics"}, {"WB_SPLIT", "performance"}, {"SHADOW_BULK_WRITE", "performance"}, {"WB_LAZY", "performance"}, {"HEAP_LAZY", "performance"}, {"BARRIER_GAME", "performance"}, {"MESH_AHEAD", "performance"}, {"WB_LAZY_TOUCH", "diagnostics"}, {"WB_LAZY_CHUNK", "performance"}, {"FLUSH_UPLOAD", "diagnostics"}, {"GPA_TWIN_MAX_KB", "diagnostics"},
|
||||
{"WB_FENCE_LIMIT", "diagnostics"}, {"WB_MERGE", "diagnostics"}, {"WB_EXACT", "diagnostics"}, {"MEM_CENSUS", "diagnostics"}, {"WB_NO_REUP", "diagnostics"}, {"WB_PROBE", "diagnostics"}, {"WB_SPLIT", "performance"}, {"SHADOW_BULK_WRITE", "performance"}, {"WB_LAZY", "performance"}, {"HEAP_LAZY", "performance"}, {"BARRIER_GAME", "performance"}, {"MESH_AHEAD", "performance"}, {"MESH_FALLBACK", "performance"}, {"MESH_FALLBACK_MB", "performance"}, {"WB_LAZY_TOUCH", "diagnostics"}, {"WB_LAZY_CHUNK", "performance"}, {"FLUSH_UPLOAD", "diagnostics"}, {"GPA_TWIN_MAX_KB", "diagnostics"},
|
||||
{"WB_MERGE_MAX", "diagnostics"}, {"MV_SURVEY", "diagnostics"}, {"TEXUP_LOG_MS", "diagnostics"}, {"VA_REPORT", "diagnostics"},
|
||||
{"VMA_BLOCK_MB", "diagnostics"}, {"ASTC_BC_PROBE", "diagnostics"}, {"DEVICE_LOST_AT", "diagnostics"},
|
||||
};
|
||||
@@ -269,6 +269,9 @@ static const CfgMeta metas[] = {
|
||||
{"PIPE_FRAME_WAIT_MS", "50", "0..500", RESTART}, // vk_exec_draw.c frame_wait
|
||||
{"PIPE_SKIP_STEADY", "1", "0|1", RESTART}, // vk_exec.c steady_on
|
||||
{"PREPARE_PIPELINES", "mesh", "0|mesh|all", RESTART}, // vk_pipeline.c seed_pipelines
|
||||
// vk_device.c: task and mesh programs through compute passes (vk_exec_draw.c "mesh fallback"); auto: on a device
|
||||
// without mesh shaders. The megabytes are the buffers the passes write (vk_mesh_fb.c).
|
||||
{"MESH_FALLBACK", "auto", "auto|0|1", RESTART}, {"MESH_FALLBACK_MB", "128", "16..4096", RESTART},
|
||||
{"PIPE_AHEAD", "1", "0|1", RESTART}, // vk_exec.c vk_exec_ahead, vk_pipeline.c vk_pipeline_hint
|
||||
{"GPL_FEEDBACK", "1", "0|1", RESTART}, // vk_pipeline.c gpl_feedback
|
||||
{"PIPE_FAST_WORKERS", "1", "0|1", RESTART}, // vk_pipeline.c lib_worker, fast_take
|
||||
|
||||
@@ -100,6 +100,7 @@ size_t gfx_shader_warm_list(GfxWarmEnt **out); // malloc'd; recorde
|
||||
const GfxShader *gfx_shader_load_private(const uint8_t iid[32], int *owned);
|
||||
void gfx_shader_release_private(const GfxShader *s);
|
||||
const uint32_t *gfx_shader_spirv(const GfxShader *s, uint64_t *words);
|
||||
const char *gfx_shader_res_json(const GfxShader *s); // the translator's .res.json text
|
||||
int gfx_shader_write_code(const GfxShader *s, const char *path); // the guest program, from the stored inputs
|
||||
// A shader of the runtime's own (SPIR-V and a .res.json text as the translator writes them), outside the cache and
|
||||
// never freed; its key is the sha256 of both. NULL when the SPIR-V is malformed.
|
||||
|
||||
@@ -2424,6 +2424,8 @@ const uint32_t *gfx_shader_spirv(const GfxShader *s, uint64_t *words) {
|
||||
return s->spirv;
|
||||
}
|
||||
|
||||
const char *gfx_shader_res_json(const GfxShader *s) { return s->reflection; }
|
||||
|
||||
// The guest program a shader was translated from, as its stored inputs hold it (the whole program: NVN2 code from its
|
||||
// first byte), written to path for a disassembler (NX2NAVI_DUMP's sh_<hash>.code), and next to it <path>.job, the
|
||||
// translator request that translates it (one line for nx2navi_xlate --serve; its "out" names the shader cache, and
|
||||
|
||||
@@ -436,6 +436,7 @@ int probe_window(void); // gfx_window_probe.c
|
||||
int probe_shader(void); // gfx_shader.c
|
||||
int probe_census_dir(char *dir, size_t n); // gfx_shader_probe.c: build/<version>/shaders for the probes
|
||||
int probe_vkmem(void), probe_vkexec(void), probe_astc(void); // tex_probe.c, vk_exec_probe.c, astc_probe.c
|
||||
int probe_meshfb(void); // mesh_fb_probe.c
|
||||
int probe_zstd(void); // ov_zstd.c
|
||||
int probe_gamepad(void); // input.c
|
||||
int probe_movie(void); // movie_probe.c: the host movie decoders on one RomFS movie
|
||||
|
||||
@@ -26,6 +26,8 @@
|
||||
// --probe-ffx-cache one FSR 3.1 upscaling context and where its pipelines came from (the pipeline caches kept
|
||||
// across runs); skips without amd_fidelityfx_vk.dll, runs no game
|
||||
// --probe-astc the compute ASTC decoder against astc_decode_ldr; needs no game files
|
||||
// --probe-meshfb the mesh shader fallback: mesh programs drawn through VK_EXT_mesh_shader and through compute
|
||||
// passes, the pictures compared; needs no game files (mesh_fb_probe.c)
|
||||
// --probe-zstd host zstd streams (ov_zstd.c) on frames made here; needs no game files
|
||||
// --probe-input gamepad polls on a virtual gamepad, also while SDL's update takes long; needs no game files
|
||||
// --probe-movie the host movie decoders on one RomFS movie (VP9 movie::Decoder, AV1 movie::VideoDecoder); skips without game data
|
||||
@@ -535,7 +537,7 @@ static int usage(const char *argv0) {
|
||||
fprintf(stderr, "usage: %s [--exefs DIR] [--romfs DIR] [--profile DIR] [--audio-device VALUE] [--monitor VALUE] [--list-audio-devices | --list-monitors | --selftest | --probe-loader | "
|
||||
"--probe-cfg | --probe-dpi | --probe-mods | --probe-kernel | --probe-ipc | --probe-services | --probe-fs | --probe-gfx-abi | "
|
||||
"--probe-nvn | --probe-nvn-tex | --probe-window | --probe-shader | --probe-vkmem | "
|
||||
"--probe-vkexec | --probe-astc | --probe-zstd | --probe-input | --probe-movie | --probe-crash-call | --probe-crash-null | --probe-discord | --probe-coop | "
|
||||
"--probe-vkexec | --probe-meshfb | --probe-astc | --probe-zstd | --probe-input | --probe-movie | --probe-crash-call | --probe-crash-null | --probe-discord | --probe-coop | "
|
||||
"--prepare-shaders | --warm-shaders | --prepare-pipelines | --probe-prewarm | --probe-buildgate | --probe-ffx-cache | --probe-memneed | --probe-devlost | --boot |--seconds S [--shot PNG]]\n", argv0);
|
||||
return 2;
|
||||
}
|
||||
@@ -684,6 +686,7 @@ int main(int argc, char **argv) {
|
||||
if (!strcmp(mode, "--probe-shader")) return probe_shader();
|
||||
if (!strcmp(mode, "--probe-vkmem")) return probe_vkmem();
|
||||
if (!strcmp(mode, "--probe-vkexec")) return probe_vkexec();
|
||||
if (!strcmp(mode, "--probe-meshfb")) return probe_meshfb();
|
||||
if (!strcmp(mode, "--probe-astc")) return probe_astc();
|
||||
if (!strcmp(mode, "--probe-zstd")) return probe_zstd();
|
||||
if (!strcmp(mode, "--probe-input")) return probe_gamepad();
|
||||
|
||||
638
runtime/mesh_fb.c
Normal file
638
runtime/mesh_fb.c
Normal file
@@ -0,0 +1,638 @@
|
||||
// The mesh shader fallback's modules, for a GPU without VK_EXT_mesh_shader (vk_exec_draw.c "mesh fallback" draws with
|
||||
// them; docs/architecture.md, "Mesh fallback").
|
||||
//
|
||||
// The translator turns a task or mesh program into a compute module whose shared memory is the program's stage buffer
|
||||
// and wraps it in a TaskEXT or MeshEXT entry function (tools/xlate/MeshSpirv.cs). The wrapping is undone here and
|
||||
// replaced: the translator's main function stays as it is, the entry function that hands the stage buffer to the mesh
|
||||
// pipeline goes, and a compute entry function copies the stage buffer into a storage buffer instead.
|
||||
// task: main; the payload area into the task buffer at the workgroup's place.
|
||||
// mesh: [the payload from the task buffer into the payload area] main; the vertex lines and the primitive area into
|
||||
// the workgroup's slot of the mesh buffer.
|
||||
// draw: a vertex module made here reads a slot per instance: vertex index / indices per primitive is the
|
||||
// primitive, its index byte names the vertex, and the vertex's lines are the outputs.
|
||||
// The module is made from the translation the shader cache already holds, so the translator and with it every cache
|
||||
// key stay as they are: a card with mesh shaders never sees these modules, and one without needs no other cache.
|
||||
//
|
||||
// Mesh workgroups after a task stage. A task workgroup starts up to `threads` mesh workgroups, and how many is known
|
||||
// only on the GPU. A pass between the two dispatches (shaders/meshfb_scan.comp) lists them in task order: entry g of
|
||||
// the list names the task workgroup and the index the mesh program sees as its workgroup id. That id comes from the
|
||||
// WorkgroupId built-in in the translated code; here the built-in counts list entries, so main reads a private
|
||||
// variable the entry function fills from the list.
|
||||
//
|
||||
// Opcode and enum numbers are the SPIR-V 1.6 specification's and SPV_EXT_mesh_shader's.
|
||||
#include "mesh_fb.h"
|
||||
|
||||
#include <stdarg.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
|
||||
enum {
|
||||
OpName = 5, OpExtension = 10, OpMemoryModel = 14, OpEntryPoint = 15, OpExecutionMode = 16, OpCapability = 17,
|
||||
OpTypeVoid = 19, OpTypeBool = 20, OpTypeInt = 21, OpTypeFloat = 22, OpTypeVector = 23, OpTypeArray = 28,
|
||||
OpTypeRuntimeArray = 29, OpTypeStruct = 30, OpTypePointer = 32, OpTypeFunction = 33, OpConstant = 43,
|
||||
OpConstantComposite = 44, OpConstantNull = 46, OpFunction = 54, OpFunctionEnd = 56, OpFunctionCall = 57,
|
||||
OpVariable = 59, OpLoad = 61, OpStore = 62, OpCopyMemory = 63, OpAccessChain = 65, OpInBoundsAccessChain = 66,
|
||||
OpPtrAccessChain = 67, OpArrayLength = 68, OpDecorate = 71, OpMemberDecorate = 72, OpCompositeConstruct = 80,
|
||||
OpCompositeExtract = 81, OpCopyObject = 83, OpBitcast = 124, OpIAdd = 128, OpISub = 130, OpIMul = 132, OpUDiv = 134,
|
||||
OpSelect = 169, OpINotEqual = 171, OpULessThan = 176, OpULessThanEqual = 178, OpShiftRightLogical = 194,
|
||||
OpBitwiseAnd = 199, OpControlBarrier = 224, OpPhi = 245, OpLoopMerge = 246, OpSelectionMerge = 247, OpLabel = 248,
|
||||
OpBranch = 249, OpBranchConditional = 250, OpReturn = 253, OpDecorateId = 332, OpDecorateString = 5632,
|
||||
};
|
||||
enum { ScInput = 1, ScOutput = 3, ScWorkgroup = 4, ScPrivate = 6, ScFunction = 7, ScStorageBuffer = 12, ScTaskPayload = 5402 };
|
||||
enum { DecBlock = 2, DecArrayStride = 6, DecBuiltIn = 11, DecInvariant = 18, DecNonWritable = 24, DecLocation = 30,
|
||||
DecBinding = 33, DecDescriptorSet = 34, DecOffset = 35 };
|
||||
enum { BiPosition = 0, BiPointSize = 1, BiWorkgroupId = 26, BiLocalInvocationIndex = 29, BiVertexIndex = 42,
|
||||
BiInstanceIndex = 43 };
|
||||
enum { CapShader = 1, CapMeshShading = 5283, ModelVertex = 0, ModelGLCompute = 5, ModelTask = 5364, ModelMesh = 5365,
|
||||
ModeLocalSize = 17 };
|
||||
|
||||
// ---- the program's layout
|
||||
|
||||
static void fail(char *err, size_t cap, const char *fmt, ...) {
|
||||
if (!err || !cap) return;
|
||||
va_list ap;
|
||||
va_start(ap, fmt);
|
||||
vsnprintf(err, cap, fmt, ap);
|
||||
va_end(ap);
|
||||
}
|
||||
|
||||
int meshfb_info(const uint8_t *control, const uint8_t *header, int stage, MeshFbInfo *m, char *err, size_t cap) {
|
||||
uint32_t w58, w5c, w60;
|
||||
memset(m, 0, sizeof *m);
|
||||
memcpy(&w58, control + 0x58, 4), memcpy(&w5c, control + 0x5c, 4), memcpy(&w60, control + 0x60, 4);
|
||||
m->stage = stage;
|
||||
m->threads = w58 & 0xffff ? w58 & 0xffff : 1;
|
||||
m->payload_bytes = (w58 >> 24) * 256;
|
||||
m->maxv = w60 & 0xffff, m->maxp = w60 >> 16;
|
||||
uint32_t topo = w5c >> 24;
|
||||
m->ipp = topo == 1 ? 1 : topo == 2 ? 2 : 3;
|
||||
m->has_task = stage == 7 && (w5c >> 16 & 0xff) != 0;
|
||||
// The payload as the translator sizes it (tools/xlate/Program.cs Translate): a size of 0 is one 256-byte unit, and
|
||||
// the header of 0x20 bytes comes on top.
|
||||
if (stage == 6 || m->has_task) m->payload_bytes = (m->payload_bytes ? m->payload_bytes : 256) + 0x20;
|
||||
if (stage == 6) {
|
||||
m->pay = 0, m->shared_bytes = m->payload_bytes;
|
||||
return 1;
|
||||
}
|
||||
if (stage != 7) return fail(err, cap, "stage %d is neither task nor mesh", stage), 0;
|
||||
if (!m->maxv || !m->maxp || m->maxv > 256) return fail(err, cap, "limits of %u vertices and %u primitives", m->maxv, m->maxp), 0;
|
||||
// The output map of the program header: system values B at byte 0x35, one bit per component from 0x60; generic
|
||||
// vectors from byte 0x36, a nibble each. System values A and C never appear in the game's mesh programs.
|
||||
if (header[0x34] || header[0x48] || header[0x49]) return fail(err, cap, "output map with system values A or C"), 0;
|
||||
for (int b = 0; b < 8; b++)
|
||||
if (header[0x35] >> b & 1) m->lines[m->nlines++] = (uint16_t)(0x60 + 4 * b);
|
||||
for (int g = 0; g < 32; g++)
|
||||
for (int c = 0; c < 4; c++)
|
||||
if (header[0x36 + g / 2] >> (4 * (g & 1) + c) & 1) m->lines[m->nlines++] = (uint16_t)(0x80 + 16 * g + 4 * c);
|
||||
uint32_t groups = (m->maxv + 31) / 32;
|
||||
m->attr = 0;
|
||||
m->map = (m->attr + groups * m->nlines * 128 + 15) & ~15u;
|
||||
m->pay = (m->map + 4 + m->ipp * m->maxp + 15) & ~15u;
|
||||
m->shared_bytes = m->pay + (m->has_task ? m->payload_bytes : 0);
|
||||
return 1;
|
||||
}
|
||||
|
||||
// ---- a growing list of SPIR-V words
|
||||
|
||||
typedef struct { uint32_t *w; size_t n, cap; int oom; } Wb;
|
||||
|
||||
static void wb_put(Wb *b, const uint32_t *w, size_t n) {
|
||||
if (b->n + n > b->cap) {
|
||||
size_t cap = (b->n + n) * 2 + 256;
|
||||
uint32_t *g = realloc(b->w, cap * 4);
|
||||
if (!g) { b->oom = 1; return; }
|
||||
b->w = g, b->cap = cap;
|
||||
}
|
||||
memcpy(b->w + b->n, w, n * 4);
|
||||
b->n += n;
|
||||
}
|
||||
|
||||
static void wb_ins(Wb *b, uint32_t op, const uint32_t *operands, size_t n) {
|
||||
uint32_t head = (uint32_t)((n + 1) << 16) | op;
|
||||
wb_put(b, &head, 1);
|
||||
wb_put(b, operands, n);
|
||||
}
|
||||
#define INS(b, op, ...) wb_ins(b, op, (const uint32_t[]){__VA_ARGS__}, sizeof((const uint32_t[]){__VA_ARGS__}) / 4)
|
||||
|
||||
// What a module under construction adds: decorations, then types, constants and variables, then one function.
|
||||
typedef struct {
|
||||
uint32_t bound;
|
||||
Wb deco, glob, fn;
|
||||
uint32_t t_void, t_bool, t_uint, t_fnvoid, t_v3uint; // existing or new
|
||||
uint32_t p_fn_uint, p_sb_uint;
|
||||
struct { uint32_t v, id; } k[96];
|
||||
int nk;
|
||||
} Cx;
|
||||
|
||||
static uint32_t nid(Cx *c) { return c->bound++; }
|
||||
|
||||
static uint32_t K(Cx *c, uint32_t v) { // a 32-bit unsigned constant
|
||||
for (int i = 0; i < c->nk; i++)
|
||||
if (c->k[i].v == v) return c->k[i].id;
|
||||
uint32_t id = nid(c);
|
||||
INS(&c->glob, OpConstant, c->t_uint, id, v);
|
||||
if (c->nk < 96) c->k[c->nk].v = v, c->k[c->nk].id = id, c->nk++;
|
||||
return id;
|
||||
}
|
||||
|
||||
static uint32_t E_(Cx *c, uint32_t op, uint32_t type, const uint32_t *operands, size_t n) { // an instruction with a result
|
||||
uint32_t id = nid(c), head[3] = {(uint32_t)((n + 3) << 16) | op, type, id};
|
||||
wb_put(&c->fn, head, 3);
|
||||
wb_put(&c->fn, operands, n);
|
||||
return id;
|
||||
}
|
||||
#define E(c, op, type, ...) E_(c, op, type, (const uint32_t[]){__VA_ARGS__}, sizeof((const uint32_t[]){__VA_ARGS__}) / 4)
|
||||
#define DO(c, op, ...) INS(&(c)->fn, op, __VA_ARGS__)
|
||||
|
||||
static uint32_t add(Cx *c, uint32_t a, uint32_t b) { return E(c, OpIAdd, c->t_uint, a, b); }
|
||||
static uint32_t mul(Cx *c, uint32_t a, uint32_t b) { return E(c, OpIMul, c->t_uint, a, b); }
|
||||
static uint32_t umin(Cx *c, uint32_t a, uint32_t b) {
|
||||
return E(c, OpSelect, c->t_uint, E(c, OpULessThan, c->t_bool, a, b), a, b);
|
||||
}
|
||||
|
||||
// if (cond) { ...the instructions emitted until if_end }
|
||||
static uint32_t if_begin(Cx *c, uint32_t cond) {
|
||||
uint32_t then = nid(c), merge = nid(c);
|
||||
DO(c, OpSelectionMerge, merge, 0);
|
||||
DO(c, OpBranchConditional, cond, then, merge);
|
||||
DO(c, OpLabel, then);
|
||||
return merge;
|
||||
}
|
||||
static void if_end(Cx *c, uint32_t merge) {
|
||||
DO(c, OpBranch, merge);
|
||||
DO(c, OpLabel, merge);
|
||||
}
|
||||
|
||||
// for (i = start; i < limit; i += step) { ...the instructions emitted until loop_end }; counter: a Function variable
|
||||
typedef struct { uint32_t head, cont, merge, i, counter, step; } Loop;
|
||||
static Loop loop_begin(Cx *c, uint32_t counter, uint32_t start, uint32_t limit, uint32_t step) {
|
||||
Loop l = {nid(c), nid(c), nid(c), 0, counter, step};
|
||||
uint32_t cond = nid(c), body = nid(c);
|
||||
DO(c, OpStore, counter, start);
|
||||
DO(c, OpBranch, l.head);
|
||||
DO(c, OpLabel, l.head);
|
||||
DO(c, OpLoopMerge, l.merge, l.cont, 0);
|
||||
DO(c, OpBranch, cond);
|
||||
DO(c, OpLabel, cond);
|
||||
l.i = E(c, OpLoad, c->t_uint, counter);
|
||||
DO(c, OpBranchConditional, E(c, OpULessThan, c->t_bool, l.i, limit), body, l.merge);
|
||||
DO(c, OpLabel, body);
|
||||
return l;
|
||||
}
|
||||
static void loop_end(Cx *c, const Loop *l) {
|
||||
DO(c, OpBranch, l->cont);
|
||||
DO(c, OpLabel, l->cont);
|
||||
DO(c, OpStore, l->counter, add(c, l->i, l->step));
|
||||
DO(c, OpBranch, l->head);
|
||||
DO(c, OpLabel, l->merge);
|
||||
}
|
||||
|
||||
// A storage buffer of 32-bit words at (set 0, binding): its variable. readonly: the module only reads it.
|
||||
static uint32_t word_buffer(Cx *c, uint32_t binding, int readonly) {
|
||||
uint32_t arr = nid(c), st = nid(c), ptr = nid(c), var = nid(c);
|
||||
INS(&c->glob, OpTypeRuntimeArray, arr, c->t_uint);
|
||||
INS(&c->glob, OpTypeStruct, st, arr);
|
||||
INS(&c->glob, OpTypePointer, ptr, ScStorageBuffer, st);
|
||||
INS(&c->glob, OpVariable, ptr, var, ScStorageBuffer);
|
||||
INS(&c->deco, OpDecorate, arr, DecArrayStride, 4);
|
||||
INS(&c->deco, OpDecorate, st, DecBlock);
|
||||
INS(&c->deco, OpMemberDecorate, st, 0, DecOffset, 0);
|
||||
if (readonly) INS(&c->deco, OpMemberDecorate, st, 0, DecNonWritable);
|
||||
INS(&c->deco, OpDecorate, var, DecDescriptorSet, 0);
|
||||
INS(&c->deco, OpDecorate, var, DecBinding, binding);
|
||||
return var;
|
||||
}
|
||||
static uint32_t buf_ptr(Cx *c, uint32_t var, uint32_t index) { return E(c, OpAccessChain, c->p_sb_uint, var, K(c, 0), index); }
|
||||
static uint32_t buf_load(Cx *c, uint32_t var, uint32_t index) { return E(c, OpLoad, c->t_uint, buf_ptr(c, var, index)); }
|
||||
|
||||
// ---- a translated task or mesh module to its compute module
|
||||
|
||||
typedef struct { uint32_t at, op, len; } In; // an instruction: its first word's index, opcode, word count
|
||||
|
||||
static int id_in(const uint32_t *set, int n, uint32_t id) {
|
||||
for (int i = 0; i < n; i++)
|
||||
if (set[i] == id) return 1;
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int is_decl(uint32_t op) { // a type, constant or global variable declaration
|
||||
return (op >= OpTypeVoid && op <= 39) || (op >= 41 && op <= 52) || op == OpVariable || op == 1;
|
||||
}
|
||||
|
||||
static uint32_t *convert(const uint32_t *w, uint64_t words, const MeshFbInfo *m, uint32_t binding, uint32_t task_words,
|
||||
uint64_t *out_words, char *err, size_t cap) {
|
||||
In *in = NULL;
|
||||
uint32_t *removed = NULL, *gvars = NULL;
|
||||
Wb out = {0};
|
||||
Cx c = {0};
|
||||
#define FAIL(...) do { fail(err, cap, __VA_ARGS__); goto bad; } while (0)
|
||||
if (words < 6 || w[0] != 0x07230203) return fail(err, cap, "not a SPIR-V module"), NULL;
|
||||
if (w[1] < 0x00010400) return fail(err, cap, "SPIR-V %u.%u, 1.4 expected", w[1] >> 16 & 0xff, w[1] >> 8 & 0xff), NULL;
|
||||
size_t n = 0, ncap = 0;
|
||||
for (uint64_t p = 5; p < words;) {
|
||||
uint32_t len = w[p] >> 16, op = w[p] & 0xffff;
|
||||
if (!len || p + len > words) return fail(err, cap, "malformed SPIR-V"), NULL;
|
||||
if (n == ncap) {
|
||||
ncap = ncap ? ncap * 2 : 4096;
|
||||
In *g = realloc(in, ncap * sizeof *in);
|
||||
if (!g) FAIL("out of memory");
|
||||
in = g;
|
||||
}
|
||||
in[n++] = (In){(uint32_t)p, op, len};
|
||||
p += len;
|
||||
}
|
||||
c.bound = w[3];
|
||||
int mesh = m->stage == 7, n_removed = 0, n_gvars = 0, n_entry = 0, n_shared = 0;
|
||||
removed = malloc((n + 1) * 4), gvars = malloc((n + 16) * 4);
|
||||
if (!removed || !gvars) FAIL("out of memory");
|
||||
uint32_t old_entry = 0, model = 0, shared_var = 0, shared_type = 0, wg_var = 0, lii_var = 0, first_fn = 0, first_decl = 0;
|
||||
// pass 1: what the module has
|
||||
for (size_t i = 0; i < n; i++) {
|
||||
const uint32_t *a = w + in[i].at + 1;
|
||||
uint32_t op = in[i].op, len = in[i].len;
|
||||
if (op == OpEntryPoint) n_entry++, model = a[0], old_entry = a[1];
|
||||
else if (op == OpTypeVoid) c.t_void = a[0];
|
||||
else if (op == OpTypeBool) c.t_bool = a[0];
|
||||
else if (op == OpTypeInt && a[1] == 32 && a[2] == 0) c.t_uint = a[0];
|
||||
else if (op == OpTypeVector && a[1] == c.t_uint && c.t_uint && a[2] == 3) c.t_v3uint = a[0];
|
||||
else if (op == OpTypeFunction && len == 3 && a[1] == c.t_void && c.t_void) c.t_fnvoid = a[0];
|
||||
else if (op == OpTypePointer && (a[1] == ScOutput || a[1] == ScTaskPayload)) removed[n_removed++] = a[0];
|
||||
else if (op == OpVariable && !first_fn) {
|
||||
if (a[2] == ScOutput || a[2] == ScTaskPayload) removed[n_removed++] = a[1];
|
||||
else gvars[n_gvars++] = a[1];
|
||||
if (a[2] == ScWorkgroup) n_shared++, shared_var = a[1], shared_type = a[0];
|
||||
} else if (op == OpFunction && !first_fn) first_fn = (uint32_t)i;
|
||||
if (!first_decl && !first_fn && is_decl(op)) first_decl = (uint32_t)i;
|
||||
}
|
||||
if (n_entry != 1 || model != (uint32_t)(mesh ? ModelMesh : ModelTask)) FAIL("the entry point is not a %s one", mesh ? "mesh" : "task");
|
||||
if (n_shared != 1) FAIL("%d workgroup variables (one shared memory array expected)", n_shared);
|
||||
if (!c.t_void || !c.t_bool || !c.t_uint || !c.t_fnvoid || !first_fn || !first_decl) FAIL("a basic type or the functions are missing");
|
||||
{ // the shared memory array holds the layout this module is asked to copy
|
||||
uint32_t arr = 0, len_id = 0, len = 0;
|
||||
for (size_t i = 0; i < n; i++) {
|
||||
const uint32_t *a = w + in[i].at + 1;
|
||||
if (in[i].op == OpTypePointer && a[0] == shared_type) arr = a[2];
|
||||
}
|
||||
for (size_t i = 0; i < n; i++) {
|
||||
const uint32_t *a = w + in[i].at + 1;
|
||||
if (in[i].op == OpTypeArray && a[0] == arr && a[1] == c.t_uint) len_id = a[2];
|
||||
}
|
||||
for (size_t i = 0; i < n; i++) {
|
||||
const uint32_t *a = w + in[i].at + 1;
|
||||
if (in[i].op == OpConstant && a[1] == len_id && len_id) len = a[2];
|
||||
}
|
||||
if (len * 4 < m->shared_bytes) FAIL("shared memory of %u bytes, the program's layout needs %u", len * 4, m->shared_bytes);
|
||||
}
|
||||
for (size_t i = 0; i < n; i++) {
|
||||
const uint32_t *a = w + in[i].at + 1;
|
||||
if (in[i].op != OpDecorate || in[i].len < 4 || a[1] != DecBuiltIn || id_in(removed, n_removed, a[0])) continue;
|
||||
if (a[2] == BiWorkgroupId) wg_var = a[0];
|
||||
if (a[2] == BiLocalInvocationIndex) lii_var = a[0];
|
||||
}
|
||||
// the old entry function, and the function it calls: the translator's main
|
||||
size_t e0 = 0, e1 = 0;
|
||||
uint32_t main_fn = 0, n_calls = 0;
|
||||
for (size_t i = first_fn; i < n; i++)
|
||||
if (in[i].op == OpFunction && w[in[i].at + 2] == old_entry) {
|
||||
e0 = i;
|
||||
for (e1 = i; e1 < n && in[e1].op != OpFunctionEnd; e1++)
|
||||
if (in[e1].op == OpFunctionCall) n_calls++, main_fn = w[in[e1].at + 3];
|
||||
break;
|
||||
}
|
||||
if (!e0 || e1 >= n || n_calls != 1) FAIL("the entry function calls %u functions (one expected)", n_calls);
|
||||
|
||||
// ---- new types, variables and the entry function
|
||||
if (!c.t_v3uint) c.t_v3uint = nid(&c), INS(&c.glob, OpTypeVector, c.t_v3uint, c.t_uint, 3);
|
||||
c.p_fn_uint = nid(&c), INS(&c.glob, OpTypePointer, c.p_fn_uint, ScFunction, c.t_uint);
|
||||
c.p_sb_uint = nid(&c), INS(&c.glob, OpTypePointer, c.p_sb_uint, ScStorageBuffer, c.t_uint);
|
||||
uint32_t p_shared = nid(&c);
|
||||
INS(&c.glob, OpTypePointer, p_shared, ScWorkgroup, c.t_uint);
|
||||
if (!lii_var) {
|
||||
uint32_t pt = nid(&c);
|
||||
lii_var = nid(&c);
|
||||
INS(&c.glob, OpTypePointer, pt, ScInput, c.t_uint);
|
||||
INS(&c.glob, OpVariable, pt, lii_var, ScInput);
|
||||
INS(&c.deco, OpDecorate, lii_var, DecBuiltIn, BiLocalInvocationIndex);
|
||||
gvars[n_gvars++] = lii_var;
|
||||
}
|
||||
if (!wg_var) {
|
||||
uint32_t pt = nid(&c);
|
||||
wg_var = nid(&c);
|
||||
INS(&c.glob, OpTypePointer, pt, ScInput, c.t_v3uint);
|
||||
INS(&c.glob, OpVariable, pt, wg_var, ScInput);
|
||||
INS(&c.deco, OpDecorate, wg_var, DecBuiltIn, BiWorkgroupId);
|
||||
gvars[n_gvars++] = wg_var;
|
||||
}
|
||||
uint32_t fake_wg = 0, p_priv_uint = 0;
|
||||
if (mesh) { // the workgroup id main sees (see the top)
|
||||
uint32_t pt = nid(&c);
|
||||
fake_wg = nid(&c), p_priv_uint = nid(&c);
|
||||
INS(&c.glob, OpTypePointer, pt, ScPrivate, c.t_v3uint);
|
||||
INS(&c.glob, OpTypePointer, p_priv_uint, ScPrivate, c.t_uint);
|
||||
INS(&c.glob, OpVariable, pt, fake_wg, ScPrivate);
|
||||
gvars[n_gvars++] = fake_wg;
|
||||
}
|
||||
uint32_t task_buf = 0, mesh_buf = 0, list_buf = 0;
|
||||
if (!mesh || m->has_task) task_buf = word_buffer(&c, binding, mesh), gvars[n_gvars++] = task_buf;
|
||||
if (mesh) mesh_buf = word_buffer(&c, binding + 1, 0), gvars[n_gvars++] = mesh_buf;
|
||||
if (mesh && m->has_task) list_buf = word_buffer(&c, binding + 2, 1), gvars[n_gvars++] = list_buf;
|
||||
|
||||
uint32_t threads = K(&c, m->threads), k0 = K(&c, 0), k1 = K(&c, 1);
|
||||
uint32_t entry = nid(&c);
|
||||
DO(&c, OpFunction, c.t_void, entry, 0, c.t_fnvoid);
|
||||
DO(&c, OpLabel, nid(&c));
|
||||
uint32_t counter = nid(&c);
|
||||
DO(&c, OpVariable, c.p_fn_uint, counter, ScFunction);
|
||||
uint32_t tid = E(&c, OpLoad, c.t_uint, lii_var);
|
||||
uint32_t g = E(&c, OpCompositeExtract, c.t_uint, E(&c, OpLoad, c.t_v3uint, wg_var), 0);
|
||||
uint32_t first = E(&c, OpULessThan, c.t_bool, tid, k1); // the workgroup's first thread
|
||||
#define SHARED(index) E(&c, OpAccessChain, p_shared, shared_var, index)
|
||||
#define BARRIER() DO(&c, OpControlBarrier, K(&c, 2), K(&c, 2), K(&c, 0x108)) /* Workgroup, AcquireRelease | WorkgroupMemory */
|
||||
if (!mesh) {
|
||||
uint32_t pw = m->payload_bytes / 4;
|
||||
uint32_t at = mul(&c, g, K(&c, pw));
|
||||
uint32_t room = E(&c, OpULessThanEqual, c.t_bool, add(&c, at, K(&c, pw)), E(&c, OpArrayLength, c.t_uint, task_buf, 0));
|
||||
// the mesh workgroup count starts at 0: a task that culls everything never writes it
|
||||
uint32_t mg = if_begin(&c, first);
|
||||
DO(&c, OpStore, SHARED(K(&c, (m->pay + 4) / 4)), k0);
|
||||
if_end(&c, mg);
|
||||
BARRIER();
|
||||
E(&c, OpFunctionCall, c.t_void, main_fn);
|
||||
BARRIER();
|
||||
mg = if_begin(&c, room);
|
||||
Loop l = loop_begin(&c, counter, tid, K(&c, pw), threads);
|
||||
DO(&c, OpStore, buf_ptr(&c, task_buf, add(&c, at, l.i)), E(&c, OpLoad, c.t_uint, SHARED(add(&c, l.i, K(&c, m->pay / 4)))));
|
||||
loop_end(&c, &l);
|
||||
if_end(&c, mg);
|
||||
} else {
|
||||
uint32_t slot_words = meshfb_slot_words(m), count_word = K(&c, m->map / 4);
|
||||
uint32_t j = g, task_at = 0;
|
||||
if (m->has_task) {
|
||||
uint32_t e = buf_load(&c, list_buf, g);
|
||||
task_at = mul(&c, E(&c, OpBitwiseAnd, c.t_uint, e, K(&c, (1u << MESHFB_LIST_TASK_BITS) - 1)), K(&c, task_words));
|
||||
j = E(&c, OpShiftRightLogical, c.t_uint, e, K(&c, MESHFB_LIST_TASK_BITS));
|
||||
}
|
||||
DO(&c, OpStore, fake_wg, E(&c, OpCompositeConstruct, c.t_v3uint, j, k0, k0));
|
||||
uint32_t at = mul(&c, g, K(&c, slot_words));
|
||||
uint32_t room = E(&c, OpULessThanEqual, c.t_bool, add(&c, at, K(&c, slot_words)), E(&c, OpArrayLength, c.t_uint, mesh_buf, 0));
|
||||
// the primitive count starts at 0: a mesh program that emits nothing never writes it
|
||||
uint32_t mg = if_begin(&c, first);
|
||||
DO(&c, OpStore, SHARED(count_word), k0);
|
||||
if_end(&c, mg);
|
||||
if (m->has_task) {
|
||||
uint32_t pw = m->payload_bytes / 4;
|
||||
if (pw > task_words) pw = task_words;
|
||||
Loop l = loop_begin(&c, counter, tid, K(&c, pw), threads);
|
||||
DO(&c, OpStore, SHARED(add(&c, l.i, K(&c, m->pay / 4))), buf_load(&c, task_buf, add(&c, task_at, l.i)));
|
||||
loop_end(&c, &l);
|
||||
}
|
||||
BARRIER();
|
||||
E(&c, OpFunctionCall, c.t_void, main_fn);
|
||||
BARRIER();
|
||||
uint32_t count = E(&c, OpLoad, c.t_uint, SHARED(count_word));
|
||||
mg = if_begin(&c, room);
|
||||
{ // a slot without primitives gets its count alone
|
||||
uint32_t some = E(&c, OpINotEqual, c.t_bool, count, k0), then = nid(&c), other = nid(&c), merge = nid(&c);
|
||||
DO(&c, OpSelectionMerge, merge, 0);
|
||||
DO(&c, OpBranchConditional, some, then, other);
|
||||
DO(&c, OpLabel, then);
|
||||
Loop l = loop_begin(&c, counter, tid, K(&c, slot_words), threads);
|
||||
DO(&c, OpStore, buf_ptr(&c, mesh_buf, add(&c, at, l.i)), E(&c, OpLoad, c.t_uint, SHARED(l.i)));
|
||||
loop_end(&c, &l);
|
||||
DO(&c, OpBranch, merge);
|
||||
DO(&c, OpLabel, other);
|
||||
uint32_t m2 = if_begin(&c, first);
|
||||
DO(&c, OpStore, buf_ptr(&c, mesh_buf, add(&c, at, count_word)), k0);
|
||||
if_end(&c, m2);
|
||||
DO(&c, OpBranch, merge);
|
||||
DO(&c, OpLabel, merge);
|
||||
}
|
||||
if_end(&c, mg);
|
||||
}
|
||||
wb_ins(&c.fn, OpReturn, NULL, 0);
|
||||
wb_ins(&c.fn, OpFunctionEnd, NULL, 0);
|
||||
|
||||
// ---- pass 2: the module again
|
||||
uint32_t head[5] = {w[0], w[1], w[2], 0, 0};
|
||||
wb_put(&out, head, 5);
|
||||
for (size_t i = 0; i < n; i++) {
|
||||
const uint32_t *a = w + in[i].at + 1;
|
||||
uint32_t op = in[i].op, len = in[i].len;
|
||||
if (i == first_decl) wb_put(&out, c.deco.w, c.deco.n);
|
||||
if (i == first_fn) wb_put(&out, c.glob.w, c.glob.n);
|
||||
if (i >= e0 && i <= e1) continue; // the old entry function
|
||||
if (op == OpCapability && a[0] == CapMeshShading) continue;
|
||||
if (op == OpExtension && len >= 6 && !memcmp(a, "SPV_EXT_mesh_shader", 20)) continue;
|
||||
if (op == OpExecutionMode && a[1] != ModeLocalSize) continue;
|
||||
if ((op == OpName || op == OpDecorate || op == OpDecorateId || op == OpDecorateString) && id_in(removed, n_removed, a[0])) continue;
|
||||
if ((op == OpTypePointer && id_in(removed, n_removed, a[0])) || (op == OpVariable && id_in(removed, n_removed, a[1]))) continue;
|
||||
if (op == OpEntryPoint) { // every global variable is its interface (SPIR-V 1.4)
|
||||
uint32_t ep[4] = {(uint32_t)((n_gvars + 5) << 16) | OpEntryPoint, ModelGLCompute, entry, 0x6e69616d /* "main" */};
|
||||
uint32_t zero = 0;
|
||||
wb_put(&out, ep, 4), wb_put(&out, &zero, 1);
|
||||
wb_put(&out, gvars, (size_t)n_gvars);
|
||||
continue;
|
||||
}
|
||||
size_t at = out.n;
|
||||
wb_put(&out, w + in[i].at, len);
|
||||
if (out.oom) break;
|
||||
uint32_t *o = out.w + at + 1;
|
||||
if (op == OpExecutionMode) o[0] = entry;
|
||||
if (mesh && i > first_fn) { // main and what it calls read the workgroup id from the private variable
|
||||
if ((op == OpAccessChain || op == OpInBoundsAccessChain) && a[2] == wg_var) {
|
||||
uint32_t pointee = 0;
|
||||
for (size_t t = first_decl; t < first_fn; t++)
|
||||
if (in[t].op == OpTypePointer && w[in[t].at + 1] == a[0]) pointee = w[in[t].at + 3];
|
||||
if (pointee != c.t_uint || len != 5) FAIL("the workgroup id is read in a way the fallback does not know");
|
||||
o[0] = p_priv_uint, o[2] = fake_wg;
|
||||
} else if (op == OpLoad && a[2] == wg_var) {
|
||||
o[2] = fake_wg;
|
||||
} else if ((op == OpStore && a[0] == wg_var) || (op == OpCopyMemory && (a[0] == wg_var || a[1] == wg_var)) ||
|
||||
((op == OpPtrAccessChain || op == OpCopyObject) && a[2] == wg_var)) {
|
||||
FAIL("the workgroup id is read in a way the fallback does not know");
|
||||
} else if (op == OpFunctionCall || op == OpSelect || op == OpPhi) {
|
||||
for (uint32_t k = 2; k + 1 < len; k++)
|
||||
if (a[k] == wg_var) FAIL("the workgroup id is passed on as a pointer");
|
||||
}
|
||||
}
|
||||
}
|
||||
wb_put(&out, c.fn.w, c.fn.n);
|
||||
if (out.oom || c.deco.oom || c.glob.oom || c.fn.oom) FAIL("out of memory");
|
||||
out.w[3] = c.bound;
|
||||
*out_words = out.n;
|
||||
free(in), free(removed), free(gvars), free(c.deco.w), free(c.glob.w), free(c.fn.w);
|
||||
return out.w;
|
||||
bad:
|
||||
free(in), free(removed), free(gvars), free(c.deco.w), free(c.glob.w), free(c.fn.w), free(out.w);
|
||||
return NULL;
|
||||
#undef FAIL
|
||||
#undef SHARED
|
||||
#undef BARRIER
|
||||
}
|
||||
|
||||
uint32_t *meshfb_task_cs(const uint32_t *spirv, uint64_t words, const MeshFbInfo *m, uint32_t binding, uint64_t *out_words,
|
||||
char *err, size_t cap) {
|
||||
if (m->stage != 6) return fail(err, cap, "not a task program"), NULL;
|
||||
return convert(spirv, words, m, binding, 0, out_words, err, cap);
|
||||
}
|
||||
|
||||
uint32_t *meshfb_mesh_cs(const uint32_t *spirv, uint64_t words, const MeshFbInfo *m, uint32_t binding,
|
||||
uint32_t task_payload_words, uint64_t *out_words, char *err, size_t cap) {
|
||||
if (m->stage != 7) return fail(err, cap, "not a mesh program"), NULL;
|
||||
if (m->has_task && !task_payload_words) return fail(err, cap, "a mesh program after a task stage without its payload size"), NULL;
|
||||
return convert(spirv, words, m, binding, task_payload_words, out_words, err, cap);
|
||||
}
|
||||
|
||||
// ---- the vertex module
|
||||
|
||||
uint32_t *meshfb_vs(const MeshFbInfo *m, uint32_t out_mask, const uint8_t out_type[32], uint64_t *out_words) {
|
||||
Cx c = {.bound = 1};
|
||||
Wb out = {0};
|
||||
uint32_t iface[40], n_iface = 0;
|
||||
c.t_void = nid(&c), INS(&c.glob, OpTypeVoid, c.t_void);
|
||||
c.t_bool = nid(&c), INS(&c.glob, OpTypeBool, c.t_bool);
|
||||
c.t_uint = nid(&c), INS(&c.glob, OpTypeInt, c.t_uint, 32, 0);
|
||||
uint32_t t_int = nid(&c), t_float = nid(&c), t_v4[3], p_out[3], t_scalar[3]; // by GFX_TYPE_*: float, sint, uint
|
||||
INS(&c.glob, OpTypeInt, t_int, 32, 1);
|
||||
INS(&c.glob, OpTypeFloat, t_float, 32);
|
||||
t_scalar[0] = t_float, t_scalar[1] = t_int, t_scalar[2] = c.t_uint;
|
||||
for (int t = 0; t < 3; t++) {
|
||||
t_v4[t] = nid(&c), INS(&c.glob, OpTypeVector, t_v4[t], t_scalar[t], 4);
|
||||
p_out[t] = nid(&c), INS(&c.glob, OpTypePointer, p_out[t], ScOutput, t_v4[t]);
|
||||
}
|
||||
c.t_fnvoid = nid(&c), INS(&c.glob, OpTypeFunction, c.t_fnvoid, c.t_void);
|
||||
c.p_sb_uint = nid(&c), INS(&c.glob, OpTypePointer, c.p_sb_uint, ScStorageBuffer, c.t_uint);
|
||||
uint32_t p_in_int = nid(&c), p_out_float = nid(&c);
|
||||
INS(&c.glob, OpTypePointer, p_in_int, ScInput, t_int);
|
||||
INS(&c.glob, OpTypePointer, p_out_float, ScOutput, t_float);
|
||||
uint32_t vi_var = nid(&c), ii_var = nid(&c), pos_var = nid(&c), size_var = 0, buf, loc_var[32] = {0};
|
||||
INS(&c.glob, OpVariable, p_in_int, vi_var, ScInput);
|
||||
INS(&c.glob, OpVariable, p_in_int, ii_var, ScInput);
|
||||
INS(&c.glob, OpVariable, p_out[0], pos_var, ScOutput);
|
||||
INS(&c.deco, OpDecorate, vi_var, DecBuiltIn, BiVertexIndex);
|
||||
INS(&c.deco, OpDecorate, ii_var, DecBuiltIn, BiInstanceIndex);
|
||||
INS(&c.deco, OpDecorate, pos_var, DecBuiltIn, BiPosition);
|
||||
INS(&c.deco, OpDecorate, pos_var, DecInvariant);
|
||||
iface[n_iface++] = vi_var, iface[n_iface++] = ii_var, iface[n_iface++] = pos_var;
|
||||
if (m->ipp == 1) { // a point list needs its size written
|
||||
size_var = nid(&c);
|
||||
INS(&c.glob, OpVariable, p_out_float, size_var, ScOutput);
|
||||
INS(&c.deco, OpDecorate, size_var, DecBuiltIn, BiPointSize);
|
||||
iface[n_iface++] = size_var;
|
||||
}
|
||||
for (int g = 0; g < 32; g++) {
|
||||
if (!(out_mask >> g & 1)) continue;
|
||||
int t = out_type[g] < 3 ? out_type[g] : 0;
|
||||
loc_var[g] = nid(&c);
|
||||
INS(&c.glob, OpVariable, p_out[t], loc_var[g], ScOutput);
|
||||
INS(&c.deco, OpDecorate, loc_var[g], DecLocation, (uint32_t)g);
|
||||
iface[n_iface++] = loc_var[g];
|
||||
}
|
||||
buf = word_buffer(&c, 1, 1);
|
||||
iface[n_iface++] = buf;
|
||||
uint32_t f0 = nid(&c), f1 = nid(&c), fout = nid(&c), null_v4[3], away = nid(&c);
|
||||
INS(&c.glob, OpConstant, t_float, f0, 0);
|
||||
INS(&c.glob, OpConstant, t_float, f1, 0x3f800000);
|
||||
INS(&c.glob, OpConstant, t_float, fout, 0xc1800000); // -16: outside the clip volume, for a point of no primitive
|
||||
for (int t = 0; t < 3; t++) null_v4[t] = nid(&c), INS(&c.glob, OpConstantNull, t_v4[t], null_v4[t]);
|
||||
INS(&c.glob, OpConstantComposite, t_v4[0], away, fout, fout, f0, f1);
|
||||
|
||||
uint32_t slot_words = meshfb_slot_words(m), k0 = K(&c, 0), entry = nid(&c);
|
||||
DO(&c, OpFunction, c.t_void, entry, 0, c.t_fnvoid);
|
||||
DO(&c, OpLabel, nid(&c));
|
||||
uint32_t vi = E(&c, OpBitcast, c.t_uint, E(&c, OpLoad, t_int, vi_var));
|
||||
uint32_t slot = E(&c, OpBitcast, c.t_uint, E(&c, OpLoad, t_int, ii_var));
|
||||
uint32_t prim = m->ipp == 1 ? vi : E(&c, OpUDiv, c.t_uint, vi, K(&c, m->ipp));
|
||||
uint32_t corner = m->ipp == 1 ? k0 : E(&c, OpISub, c.t_uint, vi, mul(&c, prim, K(&c, m->ipp)));
|
||||
uint32_t base = mul(&c, slot, K(&c, slot_words));
|
||||
// a slot past the buffer (more workgroups than it has room for) draws nothing
|
||||
uint32_t valid = E(&c, OpULessThanEqual, c.t_bool, add(&c, base, K(&c, slot_words)), E(&c, OpArrayLength, c.t_uint, buf, 0));
|
||||
uint32_t cw = buf_load(&c, buf, E(&c, OpSelect, c.t_uint, valid, add(&c, base, K(&c, m->map / 4)), k0));
|
||||
uint32_t count = E(&c, OpSelect, c.t_uint, valid, umin(&c, cw, K(&c, m->maxp)), k0);
|
||||
uint32_t live = E(&c, OpULessThan, c.t_bool, prim, count), then = nid(&c), other = nid(&c), merge = nid(&c);
|
||||
DO(&c, OpSelectionMerge, merge, 0);
|
||||
DO(&c, OpBranchConditional, live, then, other);
|
||||
DO(&c, OpLabel, then);
|
||||
{
|
||||
// byte map + 4 + ipp * prim + corner of the slot is the vertex
|
||||
uint32_t b = add(&c, add(&c, mul(&c, prim, K(&c, m->ipp)), corner), K(&c, m->map + 4));
|
||||
uint32_t word = buf_load(&c, buf, add(&c, base, E(&c, OpShiftRightLogical, c.t_uint, b, K(&c, 2))));
|
||||
uint32_t shift = mul(&c, E(&c, OpBitwiseAnd, c.t_uint, b, K(&c, 3)), K(&c, 8));
|
||||
uint32_t v = umin(&c, E(&c, OpBitwiseAnd, c.t_uint, E(&c, OpShiftRightLogical, c.t_uint, word, shift), K(&c, 0xff)), K(&c, m->maxv - 1));
|
||||
// word ((v >> 5) * lines + line) * 32 + (v & 31) of the vertex area
|
||||
uint32_t at = add(&c, add(&c, mul(&c, E(&c, OpShiftRightLogical, c.t_uint, v, K(&c, 5)), K(&c, m->nlines * 32)),
|
||||
E(&c, OpBitwiseAnd, c.t_uint, v, K(&c, 31))), add(&c, base, K(&c, m->attr / 4)));
|
||||
uint32_t pos[4] = {f0, f0, f0, f1}, size = f1, gen[32][4];
|
||||
memset(gen, 0, sizeof gen);
|
||||
for (uint32_t l = 0; l < m->nlines; l++) {
|
||||
uint32_t a = m->lines[l];
|
||||
int g = a >= 0x80 ? (int)(a - 0x80) / 16 : -1;
|
||||
if (a < 0x6c || (g >= 0 && !loc_var[g])) continue; // layer and viewport index are not carried; an output nothing reads
|
||||
uint32_t word_l = buf_load(&c, buf, l ? add(&c, at, K(&c, l * 32)) : at);
|
||||
if (g >= 0) gen[g][(a - 0x80) % 16 / 4] = word_l;
|
||||
else if (a == 0x6c) size = E(&c, OpBitcast, t_float, word_l);
|
||||
else pos[(a - 0x70) / 4] = E(&c, OpBitcast, t_float, word_l);
|
||||
}
|
||||
DO(&c, OpStore, pos_var, E(&c, OpCompositeConstruct, t_v4[0], pos[0], pos[1], pos[2], pos[3]));
|
||||
if (size_var) DO(&c, OpStore, size_var, size);
|
||||
for (int g = 0; g < 32; g++) {
|
||||
if (!loc_var[g]) continue;
|
||||
int t = out_type[g] < 3 ? out_type[g] : 0;
|
||||
uint32_t comp[4];
|
||||
for (int k = 0; k < 4; k++) {
|
||||
uint32_t word_k = gen[g][k] ? gen[g][k] : k0;
|
||||
comp[k] = t == 2 ? word_k : E(&c, OpBitcast, t_scalar[t], word_k);
|
||||
}
|
||||
DO(&c, OpStore, loc_var[g], E(&c, OpCompositeConstruct, t_v4[t], comp[0], comp[1], comp[2], comp[3]));
|
||||
}
|
||||
}
|
||||
DO(&c, OpBranch, merge);
|
||||
DO(&c, OpLabel, other);
|
||||
DO(&c, OpStore, pos_var, away);
|
||||
if (size_var) DO(&c, OpStore, size_var, f1);
|
||||
for (int g = 0; g < 32; g++)
|
||||
if (loc_var[g]) DO(&c, OpStore, loc_var[g], null_v4[out_type[g] < 3 ? out_type[g] : 0]);
|
||||
DO(&c, OpBranch, merge);
|
||||
DO(&c, OpLabel, merge);
|
||||
wb_ins(&c.fn, OpReturn, NULL, 0);
|
||||
wb_ins(&c.fn, OpFunctionEnd, NULL, 0);
|
||||
|
||||
uint32_t head[5] = {0x07230203, 0x00010400, 0, c.bound, 0};
|
||||
wb_put(&out, head, 5);
|
||||
INS(&out, OpCapability, CapShader);
|
||||
INS(&out, OpMemoryModel, 0, 1); // Logical, GLSL450
|
||||
uint32_t ep[5] = {(n_iface + 5) << 16 | OpEntryPoint, ModelVertex, entry, 0x6e69616d /* "main" */, 0};
|
||||
wb_put(&out, ep, 5);
|
||||
wb_put(&out, iface, n_iface);
|
||||
wb_put(&out, c.deco.w, c.deco.n);
|
||||
wb_put(&out, c.glob.w, c.glob.n);
|
||||
wb_put(&out, c.fn.w, c.fn.n);
|
||||
int oom = out.oom || c.deco.oom || c.glob.oom || c.fn.oom;
|
||||
free(c.deco.w), free(c.glob.w), free(c.fn.w);
|
||||
if (oom) return free(out.w), NULL;
|
||||
*out_words = out.n;
|
||||
return out.w;
|
||||
}
|
||||
|
||||
// ---- reflection text
|
||||
|
||||
char *meshfb_res_json(const char *res, uint32_t binding, const int *cb_slots, int n) {
|
||||
size_t len = strlen(res);
|
||||
char *out = malloc(len + (size_t)n * 96 + 1), *o = out;
|
||||
if (!out) return NULL;
|
||||
const char *sb = strstr(res, "\"sbufs\""), *open = sb ? strchr(sb, '[') : NULL;
|
||||
if (!open) return free(out), NULL;
|
||||
memcpy(o, res, (size_t)(open + 1 - res)), o += open + 1 - res;
|
||||
const char *rest = open + 1;
|
||||
while (*rest == ' ' || *rest == '\n' || *rest == '\r' || *rest == '\t') rest++;
|
||||
for (int i = 0; i < n; i++)
|
||||
o += sprintf(o, "{\"binding\": %u, \"slot\": -1, \"cb_slot\": %d, \"cb_offset\": 0, \"flags\": %d}%s", binding + (uint32_t)i,
|
||||
cb_slots[i], cb_slots[i] == MESHFB_CB_LIST ? 0 : 1, i + 1 < n || *rest != ']' ? "," : "");
|
||||
strcpy(o, open + 1);
|
||||
char *st = strstr(out, "\"stage\"");
|
||||
st = st ? strchr(st + 7, '"') : NULL;
|
||||
if (!st || (st[1] != 'm' && st[1] != 't') || st[2] != 's') return free(out), NULL;
|
||||
st[1] = 'c';
|
||||
return out;
|
||||
}
|
||||
51
runtime/mesh_fb.h
Normal file
51
runtime/mesh_fb.h
Normal file
@@ -0,0 +1,51 @@
|
||||
// The mesh shader fallback's modules (mesh_fb.c): what a task or mesh program computes, as compute modules that write
|
||||
// it into storage buffers, and the vertex stage that draws from those buffers. No Vulkan calls here.
|
||||
#pragma once
|
||||
#include <stddef.h>
|
||||
#include <stdint.h>
|
||||
|
||||
#define MESHFB_MAX_LINES 136 // 8 system value components and 32 generic vectors of 4
|
||||
|
||||
// A task or mesh program's stage buffer, as tools/xlate/AmpereLowering.cs MeshInfo lays it out in the shared memory
|
||||
// of the translated module.
|
||||
typedef struct {
|
||||
int stage; // 6 task, 7 mesh
|
||||
uint32_t threads; // workgroup size
|
||||
uint32_t maxv, maxp; // mesh: vertex and primitive limits
|
||||
uint32_t ipp; // mesh: indices per primitive (1 points, 2 lines, 3 triangles)
|
||||
uint32_t payload_bytes; // the task payload (task, or a mesh program a task feeds), with its header
|
||||
int has_task; // mesh: a task stage feeds it
|
||||
uint32_t nlines; // mesh: output components per vertex
|
||||
uint16_t lines[MESHFB_MAX_LINES]; // their attribute addresses: 0x6c point size, 0x70.. position, 0x80 + 16 g + 4 c
|
||||
uint32_t attr, map, pay, shared_bytes; // byte offsets in shared memory, and its size
|
||||
} MeshFbInfo;
|
||||
|
||||
// Where a module's extra storage buffers are bound: res.json "cb_slot" of their entries (vk_exec_draw.c ssbo_info).
|
||||
enum { MESHFB_CB_TASK = 0x41, MESHFB_CB_MESH = 0x42, MESHFB_CB_LIST = 0x43 };
|
||||
// A list entry: the task workgroup in bits 0-23, the mesh workgroup's index among that task's in bits 24-31.
|
||||
#define MESHFB_LIST_TASK_BITS 24
|
||||
|
||||
// From the program's control section and the first 0x80 bytes of its code (the header with the output map). 0 with
|
||||
// err filled when the program uses what the fallback does not carry.
|
||||
int meshfb_info(const uint8_t *control, const uint8_t *header, int stage, MeshFbInfo *out, char *err, size_t cap);
|
||||
// Words one mesh workgroup's output takes in the mesh buffer: its vertex lines and its primitive area.
|
||||
static inline uint32_t meshfb_slot_words(const MeshFbInfo *m) { return m->pay / 4; }
|
||||
|
||||
// The compute module of a TaskEXT module (spirv, as the translator wrote it): the same program, then the payload of
|
||||
// workgroup t into words [t * payload_bytes / 4, ..) of the task buffer, a storage buffer at `binding`. malloc'd;
|
||||
// NULL with err filled.
|
||||
uint32_t *meshfb_task_cs(const uint32_t *spirv, uint64_t words, const MeshFbInfo *m, uint32_t binding, uint64_t *out_words,
|
||||
char *err, size_t cap);
|
||||
// The compute module of a MeshEXT module. Workgroup g writes its stage buffer (meshfb_slot_words) into slot g of the
|
||||
// mesh buffer at binding + 1. After a task stage (m->has_task) entry g of the list at binding + 2 names its task
|
||||
// workgroup and its index among that task's mesh workgroups, and the payload comes from the task buffer at `binding`,
|
||||
// task_payload_words apart.
|
||||
uint32_t *meshfb_mesh_cs(const uint32_t *spirv, uint64_t words, const MeshFbInfo *m, uint32_t binding,
|
||||
uint32_t task_payload_words, uint64_t *out_words, char *err, size_t cap);
|
||||
// The vertex module that draws the mesh buffer (binding 1): instance = slot, maxp * ipp vertices an instance; vertex
|
||||
// index / ipp is the primitive, and a primitive past the slot's count is degenerate. Outputs: position, point size for
|
||||
// points, and a 4-component vector at every Location of out_mask, of out_type[location] (GFX_TYPE_*).
|
||||
uint32_t *meshfb_vs(const MeshFbInfo *m, uint32_t out_mask, const uint8_t out_type[32], uint64_t *out_words);
|
||||
// A copy of the translator's res.json text res for a compute module made above: its stage is "cs" and its storage
|
||||
// buffers gain the entries of the extra buffers (n of them from `binding` on, cb_slots[i] each). malloc'd.
|
||||
char *meshfb_res_json(const char *res, uint32_t binding, const int *cb_slots, int n);
|
||||
432
runtime/mesh_fb_probe.c
Normal file
432
runtime/mesh_fb_probe.c
Normal file
@@ -0,0 +1,432 @@
|
||||
// nx2navi.exe --probe-meshfb: the mesh shader fallback (mesh_fb.c, vk_mesh_fb.c, vk_exec_draw.c "mesh fallback").
|
||||
//
|
||||
// Draw part, no game files: two mesh programs of the runtime's own in the shape the translator gives one
|
||||
// (shaders/meshfb_probe_*.mesh, .task), one without a task stage and one fed by a task program, drawn through the
|
||||
// replay into an RGBA8 target with direct and with indirect workgroup counts, and both in one pass after each other.
|
||||
// Each picture is drawn twice, through VK_EXT_mesh_shader and through the fallback, and the two are compared texel by
|
||||
// texel; without mesh shaders on the device only the fallback draws, and the probe says that nothing was compared.
|
||||
//
|
||||
// Module part, with NX2NAVI_PROBE_MESHFB_DIR=<dir>: every "<name>" of <dir>/list.txt names <dir>/<name>.in (a stored
|
||||
// shader input, gfx_shader.c in_store: control section and code) and <dir>/<name>.spv (its translation). The compute
|
||||
// module and, for a mesh program, a vertex module are written to <dir>/out/<name>.cs.spv and .vs.spv for spirv-val
|
||||
// (work the caller does; tests of game shaders stay outside the repository). Where <dir>/<name>.res.json (the
|
||||
// translation's reflection) is there too, the compute pipeline of the module is built on this device, as its first
|
||||
// draw would build it.
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
|
||||
#include "cfg.h"
|
||||
#include "gfx.h"
|
||||
#include "host.h"
|
||||
#include "mesh_fb.h"
|
||||
#include "nvn_api.h"
|
||||
#include "nvn_tex.h"
|
||||
#include "vk.h"
|
||||
#include "vk_exec.h"
|
||||
#include "vk_mesh_fb.h"
|
||||
#include "platform/platform.h"
|
||||
|
||||
#include "meshfb_probe_f_spv.h"
|
||||
#include "meshfb_probe_m_spv.h"
|
||||
#include "meshfb_probe_t_spv.h"
|
||||
#include "meshfb_probe_tm_spv.h"
|
||||
|
||||
#define TEX_W 256
|
||||
#define TEX_H 144
|
||||
#define CHECK(cond, ...) \
|
||||
do { \
|
||||
if (!(cond)) { \
|
||||
printf("probe-meshfb: FAILED: "); \
|
||||
printf(__VA_ARGS__); \
|
||||
printf("\n"); \
|
||||
return 1; \
|
||||
} \
|
||||
} while (0)
|
||||
|
||||
void vk_exec_probe_shader(const uint8_t *ctl, const GfxShader *s); // vk_exec_draw.c
|
||||
void vk_exec_draw_shutdown(void);
|
||||
void vk_exec_xfer_shutdown(void);
|
||||
|
||||
// ---- module part
|
||||
|
||||
static uint8_t *read_all(const char *path, size_t *size) {
|
||||
FILE *f = fopen(path, "rb");
|
||||
if (!f) return NULL;
|
||||
fseek(f, 0, SEEK_END);
|
||||
long n = ftell(f);
|
||||
fseek(f, 0, SEEK_SET);
|
||||
uint8_t *b = malloc((size_t)n + 1);
|
||||
if (b && fread(b, 1, (size_t)n, f) != (size_t)n) free(b), b = NULL;
|
||||
fclose(f);
|
||||
if (b) b[n] = 0, *size = (size_t)n;
|
||||
return b;
|
||||
}
|
||||
|
||||
static int write_all(const char *path, const void *data, size_t n) {
|
||||
FILE *f = fopen(path, "wb");
|
||||
if (!f) return 0;
|
||||
int ok = fwrite(data, 1, n, f) == n;
|
||||
fclose(f);
|
||||
return ok;
|
||||
}
|
||||
|
||||
static int modules_part(const char *dir) {
|
||||
char path[1024];
|
||||
size_t list_n = 0;
|
||||
snprintf(path, sizeof path, "%s/list.txt", dir);
|
||||
char *list = (char *)read_all(path, &list_n);
|
||||
CHECK(list, "no %s", path);
|
||||
int n_task = 0, n_mesh = 0, n_fail = 0, n_pipe = 0, n_pipe_fail = 0;
|
||||
double pipe_ms = 0, pipe_worst = 0;
|
||||
uint64_t pipe_worst_words = 0;
|
||||
const GfxBackend *b = &gfx_vulkan_backend;
|
||||
pool_t pool = {.magic = NVN_POOL_MAGIC, .flags = 0x22};
|
||||
int device = 0;
|
||||
for (char *name = strtok(list, "\r\n"); name; name = strtok(NULL, "\r\n")) {
|
||||
size_t in_n = 0, spv_n = 0;
|
||||
snprintf(path, sizeof path, "%s/%s.in", dir, name);
|
||||
uint8_t *in = read_all(path, &in_n);
|
||||
snprintf(path, sizeof path, "%s/%s.spv", dir, name);
|
||||
uint8_t *spv = read_all(path, &spv_n);
|
||||
uint32_t h[6]; // magic, version, stage, code length, c1 length, size of the answers (gfx_shader.c InHeader)
|
||||
if (!in || !spv || in_n < sizeof h) {
|
||||
printf("probe-meshfb: %s: input or translation missing\n", name), n_fail++;
|
||||
free(in), free(spv);
|
||||
continue;
|
||||
}
|
||||
memcpy(h, in, sizeof h);
|
||||
const uint8_t *control = in + sizeof h + h[5], *code = control + 0x880;
|
||||
int stage = (int)h[2]; // GFX_STAGE_TS / GFX_STAGE_MS are the NVN stage numbers 6 and 7
|
||||
char err[200] = "";
|
||||
MeshFbInfo m;
|
||||
uint64_t on = 0, vn = 0;
|
||||
uint32_t *cs = NULL, *vs = NULL;
|
||||
if (in_n < sizeof h + h[5] + 0x880 + 0x80 || (stage != 6 && stage != 7)) snprintf(err, sizeof err, "not a task or mesh input");
|
||||
else if (meshfb_info(control, code, stage, &m, err, sizeof err)) {
|
||||
uint32_t top = 0;
|
||||
const uint32_t *w = (const uint32_t *)spv;
|
||||
for (size_t i = 5; i < spv_n / 4;) {
|
||||
uint32_t op = w[i] & 0xffff, len = w[i] >> 16;
|
||||
if (!len) break;
|
||||
if (op == 71 && len == 4 && w[i + 2] == 33 && w[i + 3] > top) top = w[i + 3];
|
||||
i += len;
|
||||
}
|
||||
cs = stage == 6 ? meshfb_task_cs(w, spv_n / 4, &m, top + 1, &on, err, sizeof err)
|
||||
: meshfb_mesh_cs(w, spv_n / 4, &m, top + 1, m.payload_bytes / 4, &on, err, sizeof err);
|
||||
if (cs && stage == 7) {
|
||||
uint8_t type[32] = {0};
|
||||
uint32_t mask = 0;
|
||||
for (uint32_t l = 0; l < m.nlines; l++)
|
||||
if (m.lines[l] >= 0x80) mask |= 1u << (m.lines[l] - 0x80) / 16;
|
||||
for (int g = 0; g < 32; g++) type[g] = (uint8_t)(g % 3); // every output type gets made somewhere
|
||||
vs = meshfb_vs(&m, mask, type, &vn);
|
||||
if (!vs) snprintf(err, sizeof err, "no vertex module");
|
||||
}
|
||||
}
|
||||
if (cs && (stage == 6 || vs)) {
|
||||
snprintf(path, sizeof path, "%s/out/%s.cs.spv", dir, name);
|
||||
int ok = write_all(path, cs, on * 4);
|
||||
if (vs) snprintf(path, sizeof path, "%s/out/%s.vs.spv", dir, name), ok = ok && write_all(path, vs, vn * 4);
|
||||
if (!ok) printf("probe-meshfb: %s: cannot write under %s/out\n", name, dir), n_fail++;
|
||||
else if (stage == 6) n_task++;
|
||||
else n_mesh++;
|
||||
size_t res_n = 0;
|
||||
snprintf(path, sizeof path, "%s/%s.res.json", dir, name);
|
||||
char *res = (char *)read_all(path, &res_n);
|
||||
if (res) { // the pipeline, with the layout the reflection gives
|
||||
if (!device) { // the first pool brings the device up
|
||||
uint64_t heap = 0;
|
||||
CHECK(mem_set_heap_size(16ull << 20, &heap) == 0 && heap, "SetHeapSize 16 MiB");
|
||||
pool.mem = pool.game_mem = (uint8_t *)(uintptr_t)heap + 0x100000, pool.size = 0x100000;
|
||||
CHECK(b->pool_init(&pool), "pool_init of a CPU-visible pool");
|
||||
device = 1;
|
||||
}
|
||||
uint32_t top = 0;
|
||||
const uint32_t *w = (const uint32_t *)spv;
|
||||
for (size_t i = 5; i < spv_n / 4;) {
|
||||
uint32_t op = w[i] & 0xffff, len = w[i] >> 16;
|
||||
if (!len) break;
|
||||
if (op == 71 && len == 4 && w[i + 2] == 33 && w[i + 3] > top) top = w[i + 3];
|
||||
i += len;
|
||||
}
|
||||
static const int task_slots[1] = {MESHFB_CB_TASK}, mesh_slots[3] = {MESHFB_CB_TASK, MESHFB_CB_MESH, MESHFB_CB_LIST};
|
||||
char *cres = stage == 6 ? meshfb_res_json(res, top + 1, task_slots, 1)
|
||||
: m.has_task ? meshfb_res_json(res, top + 1, mesh_slots, 3) : meshfb_res_json(res, top + 2, mesh_slots + 1, 1);
|
||||
const GfxShader *sh = cres ? gfx_shader_own(cs, on, cres) : NULL;
|
||||
const VkPipeInfo *info = NULL;
|
||||
uint64_t t0 = plat_now_ns();
|
||||
VkPipeline p = sh ? vk_pipeline_compute(sh, &info) : VK_NULL_HANDLE;
|
||||
double ms = (double)(plat_now_ns() - t0) / 1e6;
|
||||
if (p && info) {
|
||||
n_pipe++, pipe_ms += ms;
|
||||
if (ms > pipe_worst) pipe_worst = ms, pipe_worst_words = on;
|
||||
} else {
|
||||
n_pipe_fail++;
|
||||
printf("probe-meshfb: %s (stage %d, %llu words): %s\n", name, stage, (unsigned long long)on,
|
||||
sh ? "its compute pipeline was not built" : "its reflection could not be made");
|
||||
}
|
||||
fflush(stdout);
|
||||
free(cres), free(res);
|
||||
}
|
||||
} else {
|
||||
printf("probe-meshfb: %s (stage %d): %s\n", name, stage, err), n_fail++;
|
||||
}
|
||||
free(cs), free(vs), free(in), free(spv);
|
||||
}
|
||||
free(list);
|
||||
printf("probe-meshfb: module part: %d task and %d mesh programs converted, %d failed\n", n_task, n_mesh, n_fail);
|
||||
if (device) {
|
||||
printf("probe-meshfb: module part: %d compute pipelines built on this device (%.0f ms each, the longest %.0f ms for %llu "
|
||||
"words), %d not built\n", n_pipe, n_pipe ? pipe_ms / n_pipe : 0.0, pipe_worst, (unsigned long long)pipe_worst_words,
|
||||
n_pipe_fail);
|
||||
b->pool_fini(&pool);
|
||||
b->submit(NULL, 0); // the pool's buffers go when a submission after their release has completed
|
||||
b->finish();
|
||||
vk_pipeline_shutdown();
|
||||
vk_exec_draw_shutdown();
|
||||
vk_exec_xfer_shutdown();
|
||||
vk_backend_shutdown();
|
||||
}
|
||||
return n_fail != 0 || n_pipe_fail != 0;
|
||||
}
|
||||
|
||||
// ---- draw part
|
||||
|
||||
static uint8_t *put(uint8_t *p, uint32_t op, const void *payload, uint32_t n) {
|
||||
op_hdr h = {op, (uint32_t)((sizeof h + n + 7) & ~7u)};
|
||||
memset(p, 0, h.size);
|
||||
memcpy(p, &h, sizeof h);
|
||||
if (n) memcpy(p + sizeof h, payload, n);
|
||||
return p + h.size;
|
||||
}
|
||||
|
||||
// An NVN2 control section of a task (6), mesh (7) or fragment (1) stage as vk_exec_draw.c and mesh_fb.c read it, and
|
||||
// a code block whose header maps the outputs: position and generic vector 0.
|
||||
static void fake_stage(NVNshaderData *sd, uint32_t stage, int fed_by_task) {
|
||||
uint8_t *ctl = calloc(1, 0x880), *code = calloc(1, 0x100);
|
||||
uint32_t magic = 0x98761234u, len = 0x100, st = stage << 16;
|
||||
uint32_t w58 = 32u | 1u << 24; // 32 threads, a payload of one 256-byte unit
|
||||
uint32_t w5c = 3u << 24 | (fed_by_task ? 1u << 16 : 0); // triangles
|
||||
uint32_t w60 = 64u | 32u << 16; // 64 vertices, 32 primitives
|
||||
memcpy(ctl, &magic, 4), memcpy(ctl + 0x38, &len, 4), memcpy(ctl + 0x54, &st, 4);
|
||||
if (stage >= 6) memcpy(ctl + 0x58, &w58, 4);
|
||||
if (stage == 7) memcpy(ctl + 0x5c, &w5c, 4), memcpy(ctl + 0x60, &w60, 4), code[0x35] = 0xf0, code[0x36] = 0x0f;
|
||||
sd->control = ctl, sd->data = (uint64_t)(uintptr_t)code;
|
||||
}
|
||||
|
||||
static const char RES_MS[] = "{\"key\":\"probe\",\"stage\":\"ms\",\"notes\":[],\"cbufs\":[{\"binding\":1,\"slot\":3}],"
|
||||
"\"sbufs\":[],\"textures\":[],\"images\":[]}";
|
||||
static const char RES_TS[] = "{\"key\":\"probe\",\"stage\":\"ts\",\"notes\":[],\"cbufs\":[{\"binding\":1,\"slot\":3}],"
|
||||
"\"sbufs\":[],\"textures\":[],\"images\":[]}";
|
||||
static const char RES_FS[] = "{\"key\":\"probe\",\"stage\":\"ps\",\"notes\":[],\"cbufs\":[],\"sbufs\":[],\"textures\":[],"
|
||||
"\"images\":[]}";
|
||||
|
||||
// The program of a mesh stage, its task stage if any, and the fragment stage.
|
||||
static prog_t *make_prog(const GfxShader *ms, const GfxShader *ts, const GfxShader *fs) {
|
||||
static uint64_t serial = 0x9000;
|
||||
prog_t *p = calloc(1, sizeof *p);
|
||||
p->magic = NVN_PROG_MAGIC, p->serial = ++serial;
|
||||
fake_stage(&p->sd[p->nstage], 7, ts != NULL), vk_exec_probe_shader(p->sd[p->nstage++].control, ms);
|
||||
if (ts) fake_stage(&p->sd[p->nstage], 6, 0), vk_exec_probe_shader(p->sd[p->nstage++].control, ts);
|
||||
fake_stage(&p->sd[p->nstage], 1, 0), vk_exec_probe_shader(p->sd[p->nstage++].control, fs);
|
||||
return p;
|
||||
}
|
||||
|
||||
typedef struct { prog_t *prog; uint32_t count; int indirect; uint32_t grid[2]; float tint[4]; } MeshDraw;
|
||||
|
||||
// Uniform data in the pool, 0x1000 apart per draw: the mesh stage's block at 0x90000, the task stage's at 0x98000; the
|
||||
// indirect records at 0xa0000.
|
||||
static int draw_picture(const GfxBackend *b, pool_t *pool, tex_t *color, const MeshDraw *d, int n, uint32_t *px) {
|
||||
static _Alignas(8) uint8_t stream[16384];
|
||||
uint8_t *q = put(stream, OP_BEGIN, NULL, 0);
|
||||
p_set_rt rt;
|
||||
memset(&rt, 0, sizeof rt);
|
||||
rt.n = 1, rt.color[0] = color;
|
||||
q = put(q, OP_SET_RT, &rt, sizeof rt);
|
||||
const p_clear_color c0 = {0, 0xf, {0.2f, 0.4f, 0.6f, 1.0f}};
|
||||
q = put(q, OP_CLEAR_COLOR, &c0, sizeof c0);
|
||||
p_viewports vp;
|
||||
memset(&vp, 0, sizeof vp);
|
||||
vp.n = 1, vp.v[0][2] = TEX_W, vp.v[0][3] = TEX_H, vp.v[0][5] = 1.0f;
|
||||
q = put(q, OP_VIEWPORTS, &vp, sizeof vp);
|
||||
const p_scissor full = {0, 0, 1 << 20, 1 << 20};
|
||||
q = put(q, OP_SCISSOR, &full, sizeof full);
|
||||
for (int i = 0; i < n; i++) {
|
||||
uint8_t *mu = pool->mem + 0x90000 + 0x1000 * i, *tu = pool->mem + 0x98000 + 0x1000 * i;
|
||||
uint32_t *rec = (uint32_t *)(pool->mem + 0xa0000 + 0x100 * i);
|
||||
uint32_t grid[4] = {d[i].grid[0], d[i].grid[1], 0, 0}, shift[4] = {1, 0, 0, 0};
|
||||
memcpy(mu, d[i].tint, 16), memcpy(mu + 16, grid, 16), memcpy(tu, shift, 16);
|
||||
rec[0] = d[i].count, rec[1] = 0, rec[2] = 0;
|
||||
const p_bind_program bp = {d[i].prog, 0xc2};
|
||||
q = put(q, OP_BIND_PROGRAM, &bp, sizeof bp);
|
||||
const p_bind_buf bm = {7, 0, (uint64_t)(uintptr_t)mu, 256}, bt = {6, 0, (uint64_t)(uintptr_t)tu, 256};
|
||||
q = put(q, OP_BIND_UBO, &bm, sizeof bm);
|
||||
q = put(q, OP_BIND_UBO, &bt, sizeof bt);
|
||||
if (d[i].indirect) {
|
||||
const p_draw_mesh_ind di = {(uint64_t)(uintptr_t)rec, 0, 0, 0, 0};
|
||||
q = put(q, OP_DRAW_MESH_IND, &di, sizeof di);
|
||||
} else {
|
||||
const p_draw_mesh dm = {0, d[i].count, 0};
|
||||
q = put(q, OP_DRAW_MESH, &dm, sizeof dm);
|
||||
}
|
||||
}
|
||||
q = put(q, OP_END, NULL, 0);
|
||||
CHECK(q <= stream + sizeof stream, "op stream overflow");
|
||||
uint64_t handle = (uint64_t)(uintptr_t)stream;
|
||||
uint64_t f = b->submit(&handle, 1);
|
||||
CHECK(b->fence_wait(f, 60000000000ull), "fence_wait of a mesh draw submission timed out");
|
||||
NVNcopyRegion r = {0, 0, 0, TEX_W, TEX_H, 1};
|
||||
b->read_texels(color, NULL, &r, (uint8_t *)px, TEX_W * 4, TEX_W * TEX_H * 4);
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int drawn_texels(const uint32_t *px) {
|
||||
int n = 0;
|
||||
for (int i = 0; i < TEX_W * TEX_H; i++) n += px[i] != px[0]; // texel 0 is in the margin no cell covers
|
||||
return n;
|
||||
}
|
||||
|
||||
// Texels of a and b that differ, and the largest difference of a channel.
|
||||
static int differing(const uint32_t *a, const uint32_t *b, int *worst) {
|
||||
int n = 0;
|
||||
*worst = 0;
|
||||
for (int i = 0; i < TEX_W * TEX_H; i++) {
|
||||
if (a[i] == b[i]) continue;
|
||||
n++;
|
||||
for (int c = 0; c < 4; c++) {
|
||||
int d = (int)(a[i] >> 8 * c & 0xff) - (int)(b[i] >> 8 * c & 0xff);
|
||||
if (d < 0) d = -d;
|
||||
if (d > *worst) *worst = d;
|
||||
}
|
||||
}
|
||||
return n;
|
||||
}
|
||||
|
||||
static int draw_part(const GfxBackend *b, uint8_t *guest) {
|
||||
pool_t pool = {.magic = NVN_POOL_MAGIC, .flags = 0x22, .mem = guest + 0x100000, .size = 0xC0000,
|
||||
.game_mem = guest + 0x100000};
|
||||
CHECK(b->pool_init(&pool), "pool_init of a CPU-visible pool");
|
||||
VkCtx *c = vk_ctx();
|
||||
CHECK(c, "no Vulkan device");
|
||||
static tex_t color;
|
||||
memset(&color, 0, sizeof color);
|
||||
color.magic = NVN_TEX_MAGIC;
|
||||
color.d.target = NVN_TEXTURE_TARGET_2D, color.d.format = NVN_FORMAT_RGBA8, color.d.width = TEX_W, color.d.height = TEX_H;
|
||||
color.d.depth = 1, color.d.levels = 1;
|
||||
nvn_tex_layout(&color.d, &color.L);
|
||||
CHECK(color.L.size <= 0x80000, "the target exceeds its pool slot");
|
||||
color.pool = &pool, color.pool_flags = (uint16_t)pool.flags, color.off = 0, color.mem = pool.mem, color.serial = 4000;
|
||||
|
||||
const GfxShader *fs = gfx_shader_own(meshfb_probe_f_spv, sizeof meshfb_probe_f_spv / 4, RES_FS);
|
||||
const GfxShader *ms = gfx_shader_own(meshfb_probe_m_spv, sizeof meshfb_probe_m_spv / 4, RES_MS);
|
||||
const GfxShader *ts = gfx_shader_own(meshfb_probe_t_spv, sizeof meshfb_probe_t_spv / 4, RES_TS);
|
||||
const GfxShader *tm = gfx_shader_own(meshfb_probe_tm_spv, sizeof meshfb_probe_tm_spv / 4, RES_MS);
|
||||
CHECK(fs && ms && ts && tm, "a probe shader is malformed");
|
||||
prog_t *plain = make_prog(ms, NULL, fs), *tasked = make_prog(tm, ts, fs);
|
||||
|
||||
// 24 cells of a 6 x 4 grid by 24 mesh workgroups; 40 cells of an 8 x 5 grid by 10 task workgroups that start 0 to
|
||||
// 4 mesh workgroups each
|
||||
const MeshDraw cases[][2] = {
|
||||
{{plain, 24, 0, {6, 4}, {1, 1, 1, 1}}},
|
||||
{{plain, 24, 1, {6, 4}, {1, 1, 1, 1}}},
|
||||
{{tasked, 10, 0, {8, 5}, {1, 1, 1, 1}}},
|
||||
{{tasked, 10, 1, {8, 5}, {1, 1, 1, 1}}},
|
||||
{{plain, 24, 1, {6, 4}, {1, 1, 1, 1}}, {tasked, 10, 1, {8, 5}, {0.5f, 1, 0.25f, 1}}},
|
||||
{{tasked, 10, 0, {8, 5}, {1, 1, 1, 1}}, {plain, 24, 0, {6, 4}, {1, 0.5f, 0.5f, 1}}},
|
||||
};
|
||||
static const char *const names[] = {"mesh, direct", "mesh, indirect", "task + mesh, direct", "task + mesh, indirect",
|
||||
"mesh then task + mesh in one pass, indirect", "task + mesh then mesh in one pass, direct"};
|
||||
enum { CASES = sizeof cases / sizeof cases[0] };
|
||||
static uint32_t px[2][CASES][TEX_W * TEX_H];
|
||||
int native = c->has_mesh_shader, was = c->mesh_fallback;
|
||||
for (int way = native ? 0 : 1; way < 2; way++) {
|
||||
c->mesh_fallback = way;
|
||||
for (int k = 0; k < CASES; k++)
|
||||
if (draw_picture(b, &pool, &color, cases[k], cases[k][1].prog ? 2 : 1, px[way][k])) return 1;
|
||||
}
|
||||
c->mesh_fallback = was;
|
||||
for (int k = 0; k < CASES; k++) {
|
||||
int fb = drawn_texels(px[1][k]);
|
||||
CHECK(fb > TEX_W * TEX_H / 20, "%s: the fallback drew %d texels", names[k], fb);
|
||||
if (!native) {
|
||||
printf("probe-meshfb: %s: the fallback drew %d of %d texels (no mesh shaders here: not compared)\n", names[k], fb,
|
||||
TEX_W * TEX_H);
|
||||
continue;
|
||||
}
|
||||
int worst, diff = differing(px[0][k], px[1][k], &worst), nat = drawn_texels(px[0][k]);
|
||||
CHECK(nat > TEX_W * TEX_H / 20, "%s: the mesh pipeline drew %d texels", names[k], nat);
|
||||
CHECK(worst <= 1, "%s: %d texels differ between the mesh pipeline and the fallback, by up to %d of 255 (the mesh "
|
||||
"pipeline drew %d texels, the fallback %d)", names[k], diff, worst, nat, fb);
|
||||
printf("probe-meshfb: %s: %d texels drawn by the mesh pipeline, %d by the fallback, %d differ (by %d of 255 at most)\n",
|
||||
names[k], nat, fb, diff, worst);
|
||||
}
|
||||
int w01, w23;
|
||||
CHECK(!differing(px[1][0], px[1][1], &w01) && !differing(px[1][2], px[1][3], &w23),
|
||||
"the fallback's direct and indirect pictures differ");
|
||||
CHECK(!vk_meshfb_short(), "%llu of the draws above did not fit the fallback's buffers", (unsigned long long)vk_meshfb_short());
|
||||
// More workgroups than the mesh buffer has slots for: what fits is drawn, the rest is dropped and counted. The
|
||||
// cells past the grid are outside the picture, so it is the 24-cell picture again.
|
||||
uint32_t slots = (uint32_t)(vk_meshfb_mesh_words() / 540), over = slots + 1000;
|
||||
const MeshDraw big = {plain, over, 1, {6, 4}, {1, 1, 1, 1}};
|
||||
static uint32_t bpx[TEX_W * TEX_H];
|
||||
c->mesh_fallback = 1;
|
||||
int bad = draw_picture(b, &pool, &color, &big, 1, bpx);
|
||||
c->mesh_fallback = was;
|
||||
if (bad) return 1;
|
||||
b->finish();
|
||||
int wb, db = differing(bpx, px[1][0], &wb);
|
||||
CHECK(!db, "a draw of %u workgroups, %u more than the mesh buffer holds: %d texels differ from the 24-cell picture", over,
|
||||
over - slots, db);
|
||||
CHECK(vk_meshfb_short() == 1, "a draw of %u workgroups with room for %u was counted as %llu draws that did not fit", over, slots,
|
||||
(unsigned long long)vk_meshfb_short());
|
||||
printf("probe-meshfb: a draw of %u workgroups with room for %u: the picture of the first 24 is whole, the draw was counted\n",
|
||||
over, slots);
|
||||
// As the game runs it: pipelines built in the background, and a program's modules made on the fallback's own
|
||||
// thread. New shader objects of the same programs are drawn until the picture is whole: the draws before that are
|
||||
// skipped, and none may leave a partial picture behind for good.
|
||||
const GfxShader *fs2 = gfx_shader_own(meshfb_probe_f_spv, sizeof meshfb_probe_f_spv / 4, RES_FS);
|
||||
const GfxShader *ms2 = gfx_shader_own(meshfb_probe_m_spv, sizeof meshfb_probe_m_spv / 4, RES_MS);
|
||||
const GfxShader *ts2 = gfx_shader_own(meshfb_probe_t_spv, sizeof meshfb_probe_t_spv / 4, RES_TS);
|
||||
const GfxShader *tm2 = gfx_shader_own(meshfb_probe_tm_spv, sizeof meshfb_probe_tm_spv / 4, RES_MS);
|
||||
CHECK(fs2 && ms2 && ts2 && tm2, "a probe shader is malformed");
|
||||
prog_t *plain2 = make_prog(ms2, NULL, fs2), *tasked2 = make_prog(tm2, ts2, fs2);
|
||||
const MeshDraw again[2] = {{tasked2, 10, 0, {8, 5}, {1, 1, 1, 1}}, {plain2, 24, 0, {6, 4}, {1, 0.5f, 0.5f, 1}}};
|
||||
vk_pipeline_async_enable();
|
||||
c->mesh_fallback = 1;
|
||||
int tries = 0, left = -1, wa;
|
||||
while (left && tries < 500) {
|
||||
if (tries++) plat_sleep_ms(10);
|
||||
if (draw_picture(b, &pool, &color, again, 2, bpx)) return 1;
|
||||
left = differing(bpx, px[1][5], &wa);
|
||||
}
|
||||
c->mesh_fallback = was;
|
||||
CHECK(!left, "with background builds the picture still differs in %d texels after %d submissions", left, tries);
|
||||
printf("probe-meshfb: with background builds the picture was whole at submission %d (the earlier ones skipped their draws)\n",
|
||||
tries);
|
||||
b->texture_release(&color);
|
||||
b->pool_fini(&pool);
|
||||
return 0;
|
||||
}
|
||||
|
||||
int probe_meshfb(void) {
|
||||
const char *dir = cfg_get("NX2NAVI_PROBE_MESHFB_DIR");
|
||||
if (dir && *dir) return modules_part(dir);
|
||||
vk_hdr_off();
|
||||
const GfxBackend *b = &gfx_vulkan_backend;
|
||||
uint64_t heap = 0;
|
||||
CHECK(mem_set_heap_size(16ull << 20, &heap) == 0 && heap, "SetHeapSize 16 MiB");
|
||||
int bad = draw_part(b, (uint8_t *)(uintptr_t)heap);
|
||||
b->finish();
|
||||
vk_pipeline_shutdown();
|
||||
vk_exec_draw_shutdown();
|
||||
vk_exec_xfer_shutdown();
|
||||
vk_backend_shutdown();
|
||||
long errors = vk_validation_errors_total();
|
||||
if (bad || errors) {
|
||||
printf("probe-meshfb: FAILED (%s, %ld validation errors)\n", bad ? "a draw check failed" : "checks passed", errors);
|
||||
return 1;
|
||||
}
|
||||
printf("probe-meshfb: passed (0 validation errors)\n");
|
||||
return 0;
|
||||
}
|
||||
6
runtime/shaders/meshfb_probe_f.frag
Normal file
6
runtime/shaders/meshfb_probe_f.frag
Normal file
@@ -0,0 +1,6 @@
|
||||
#version 450
|
||||
// --probe-meshfb (mesh_fb_probe.c): the fragment stage of the probe's mesh programs: the interpolated generic vector 0.
|
||||
layout(location = 0) in vec4 i_color;
|
||||
layout(location = 0) out vec4 o_color;
|
||||
|
||||
void main() { o_color = i_color; }
|
||||
75
runtime/shaders/meshfb_probe_m.mesh
Normal file
75
runtime/shaders/meshfb_probe_m.mesh
Normal file
@@ -0,0 +1,75 @@
|
||||
#version 460
|
||||
#extension GL_EXT_mesh_shader : require
|
||||
// build_shaders: keep functions
|
||||
//
|
||||
// --probe-meshfb (mesh_fb_probe.c): a mesh program without a task stage, in the shape the translator gives one
|
||||
// (tools/xlate/MeshSpirv.cs): the program itself is a function that writes its stage buffer into shared memory, and
|
||||
// the entry function calls it once and hands the buffer to the mesh pipeline. mesh_fb.c makes its compute module from
|
||||
// this one, as it does from a translated program. The layout is AmpereLowering.MeshInfo's for 64 vertices, 32
|
||||
// triangles and 8 output components (position, generic vector 0): vertex v's component l is word
|
||||
// ((v >> 5) * 8 + l) * 32 + (v & 31), the primitive count word 512, the index bytes from byte 2052.
|
||||
//
|
||||
// Workgroup g draws cell g of a grid: nine vertices (28 to 36, across the two vertex groups) and up to eight
|
||||
// triangles, none in every fifth cell.
|
||||
layout(local_size_x = 32) in;
|
||||
layout(triangles, max_vertices = 64, max_primitives = 32) out;
|
||||
layout(location = 0) out vec4 o_color[];
|
||||
layout(set = 0, binding = 1) uniform Params { vec4 tint; uvec4 grid; } u; // grid: columns, rows
|
||||
shared uint mem[540];
|
||||
|
||||
#define VWORD(v, l) ((((v) >> 5) * 8u + (l)) * 32u + ((v) & 31u))
|
||||
#define COUNT_WORD 512u
|
||||
#define INDEX_BYTE 2052u
|
||||
|
||||
void program() {
|
||||
uint tid = gl_LocalInvocationIndex, cell = gl_WorkGroupID.x;
|
||||
if (tid < 9u) {
|
||||
uint v = tid + 28u, ix = tid % 3u, iy = tid / 3u;
|
||||
vec2 p = -1.0 + (vec2(cell % u.grid.x, cell / u.grid.x) + 0.1 + 0.4 * vec2(ix, iy)) * (2.0 / vec2(u.grid.xy));
|
||||
vec4 c = vec4(float(ix) * 0.5, float(iy) * 0.5, float(cell % 4u) / 3.0, 1.0) * u.tint;
|
||||
mem[VWORD(v, 0u)] = floatBitsToUint(p.x);
|
||||
mem[VWORD(v, 1u)] = floatBitsToUint(p.y);
|
||||
mem[VWORD(v, 2u)] = floatBitsToUint(0.5);
|
||||
mem[VWORD(v, 3u)] = floatBitsToUint(1.0);
|
||||
mem[VWORD(v, 4u)] = floatBitsToUint(c.x);
|
||||
mem[VWORD(v, 5u)] = floatBitsToUint(c.y);
|
||||
mem[VWORD(v, 6u)] = floatBitsToUint(c.z);
|
||||
mem[VWORD(v, 7u)] = floatBitsToUint(c.w);
|
||||
}
|
||||
if (tid == 0u) { // one thread writes the index bytes: they share words
|
||||
uint count = cell % 5u == 0u ? 0u : 2u + cell % 7u;
|
||||
for (uint t = 0u; t < count; t++) {
|
||||
uint q = t / 2u, a = (q / 2u) * 3u + q % 2u + 28u;
|
||||
uvec3 tri = (t & 1u) == 0u ? uvec3(a, a + 1u, a + 3u) : uvec3(a + 1u, a + 4u, a + 3u);
|
||||
for (uint k = 0u; k < 3u; k++) {
|
||||
uint b = INDEX_BYTE + 3u * t + k, sh = (b & 3u) * 8u;
|
||||
mem[b >> 2] = (mem[b >> 2] & ~(0xffu << sh)) | tri[k] << sh;
|
||||
}
|
||||
}
|
||||
mem[COUNT_WORD] = count;
|
||||
}
|
||||
}
|
||||
|
||||
void main() {
|
||||
uint tid = gl_LocalInvocationIndex;
|
||||
if (tid < 1u) mem[COUNT_WORD] = 0u;
|
||||
barrier();
|
||||
program();
|
||||
barrier();
|
||||
uint count = min(mem[COUNT_WORD], 32u);
|
||||
SetMeshOutputsEXT(64u, count);
|
||||
for (uint v = tid; v < 64u; v += 32u) {
|
||||
gl_MeshVerticesEXT[v].gl_Position = vec4(uintBitsToFloat(mem[VWORD(v, 0u)]), uintBitsToFloat(mem[VWORD(v, 1u)]),
|
||||
uintBitsToFloat(mem[VWORD(v, 2u)]), uintBitsToFloat(mem[VWORD(v, 3u)]));
|
||||
o_color[v] = vec4(uintBitsToFloat(mem[VWORD(v, 4u)]), uintBitsToFloat(mem[VWORD(v, 5u)]),
|
||||
uintBitsToFloat(mem[VWORD(v, 6u)]), uintBitsToFloat(mem[VWORD(v, 7u)]));
|
||||
}
|
||||
for (uint p = tid; p < count; p += 32u) {
|
||||
uvec3 tri;
|
||||
for (uint k = 0u; k < 3u; k++) {
|
||||
uint b = INDEX_BYTE + 3u * p + k;
|
||||
tri[k] = min(mem[b >> 2] >> ((b & 3u) * 8u) & 0xffu, 63u);
|
||||
}
|
||||
gl_PrimitiveTriangleIndicesEXT[p] = tri;
|
||||
}
|
||||
}
|
||||
33
runtime/shaders/meshfb_probe_t.task
Normal file
33
runtime/shaders/meshfb_probe_t.task
Normal file
@@ -0,0 +1,33 @@
|
||||
#version 460
|
||||
#extension GL_EXT_mesh_shader : require
|
||||
// build_shaders: keep functions
|
||||
//
|
||||
// --probe-meshfb (mesh_fb_probe.c): a task program in the shape the translator gives one (tools/xlate/MeshSpirv.cs):
|
||||
// the program writes its payload into shared memory, and the entry function copies that out and starts the mesh
|
||||
// workgroups the program counted into word 1. The payload is 288 bytes: the header, then the program's data from
|
||||
// word 4.
|
||||
//
|
||||
// Task workgroup t starts 0 to 4 mesh workgroups (the uniform block shifts which tasks start how many) and tells each
|
||||
// the grid cell it draws; the last payload word carries a mark the mesh program checks.
|
||||
layout(local_size_x = 32) in;
|
||||
layout(set = 0, binding = 1) uniform Params { uvec4 shift; } u;
|
||||
shared uint mem[72];
|
||||
taskPayloadSharedEXT uint payload[72];
|
||||
|
||||
void program() {
|
||||
uint tid = gl_LocalInvocationIndex, t = gl_WorkGroupID.x;
|
||||
uint count = (t * 3u + u.shift.x) % 5u;
|
||||
if (tid == 0u) mem[1] = count;
|
||||
if (tid < count) mem[4u + tid] = t * 4u + tid;
|
||||
if (tid == 31u) mem[71] = 0x600df00du + t;
|
||||
}
|
||||
|
||||
void main() {
|
||||
uint tid = gl_LocalInvocationIndex;
|
||||
if (tid < 1u) mem[1] = 0u;
|
||||
barrier();
|
||||
program();
|
||||
barrier();
|
||||
for (uint i = tid; i < 72u; i += 32u) payload[i] = mem[i];
|
||||
EmitMeshTasksEXT(min(mem[1], 32u), 1u, 1u);
|
||||
}
|
||||
74
runtime/shaders/meshfb_probe_tm.mesh
Normal file
74
runtime/shaders/meshfb_probe_tm.mesh
Normal file
@@ -0,0 +1,74 @@
|
||||
#version 460
|
||||
#extension GL_EXT_mesh_shader : require
|
||||
// build_shaders: keep functions
|
||||
//
|
||||
// --probe-meshfb (mesh_fb_probe.c): the mesh program that meshfb_probe_t.task feeds, in the translator's shape
|
||||
// (meshfb_probe_m.mesh says which). Its shared memory holds the task payload behind the stage buffer, from word 540.
|
||||
// The workgroup id is its index among its task's mesh workgroups: payload word 4 + id names the cell it draws, and
|
||||
// the mark in the last payload word names its task, which the colour shows when either is wrong.
|
||||
layout(local_size_x = 32) in;
|
||||
layout(triangles, max_vertices = 64, max_primitives = 32) out;
|
||||
layout(location = 0) out vec4 o_color[];
|
||||
layout(set = 0, binding = 1) uniform Params { vec4 tint; uvec4 grid; } u; // grid: columns, rows
|
||||
shared uint mem[612];
|
||||
taskPayloadSharedEXT uint payload[72];
|
||||
|
||||
#define VWORD(v, l) ((((v) >> 5) * 8u + (l)) * 32u + ((v) & 31u))
|
||||
#define COUNT_WORD 512u
|
||||
#define INDEX_BYTE 2052u
|
||||
#define PAYLOAD 540u
|
||||
|
||||
void program() {
|
||||
uint tid = gl_LocalInvocationIndex, cell = mem[PAYLOAD + 4u + gl_WorkGroupID.x];
|
||||
bool mine = mem[PAYLOAD + 71u] == 0x600df00du + cell / 4u && cell % 4u == gl_WorkGroupID.x;
|
||||
if (tid < 9u) {
|
||||
uint v = tid + 28u, ix = tid % 3u, iy = tid / 3u;
|
||||
vec2 p = -1.0 + (vec2(cell % u.grid.x, cell / u.grid.x) + 0.1 + 0.4 * vec2(ix, iy)) * (2.0 / vec2(u.grid.xy));
|
||||
vec4 c = mine ? vec4(float(ix) * 0.5, float(iy) * 0.5, float(cell % 4u) / 3.0, 1.0) * u.tint : vec4(1.0, 0.0, 1.0, 1.0);
|
||||
mem[VWORD(v, 0u)] = floatBitsToUint(p.x);
|
||||
mem[VWORD(v, 1u)] = floatBitsToUint(p.y);
|
||||
mem[VWORD(v, 2u)] = floatBitsToUint(0.5);
|
||||
mem[VWORD(v, 3u)] = floatBitsToUint(1.0);
|
||||
mem[VWORD(v, 4u)] = floatBitsToUint(c.x);
|
||||
mem[VWORD(v, 5u)] = floatBitsToUint(c.y);
|
||||
mem[VWORD(v, 6u)] = floatBitsToUint(c.z);
|
||||
mem[VWORD(v, 7u)] = floatBitsToUint(c.w);
|
||||
}
|
||||
if (tid == 0u) { // one thread writes the index bytes: they share words
|
||||
uint count = 2u + cell % 7u;
|
||||
for (uint t = 0u; t < count; t++) {
|
||||
uint q = t / 2u, a = (q / 2u) * 3u + q % 2u + 28u;
|
||||
uvec3 tri = (t & 1u) == 0u ? uvec3(a, a + 1u, a + 3u) : uvec3(a + 1u, a + 4u, a + 3u);
|
||||
for (uint k = 0u; k < 3u; k++) {
|
||||
uint b = INDEX_BYTE + 3u * t + k, sh = (b & 3u) * 8u;
|
||||
mem[b >> 2] = (mem[b >> 2] & ~(0xffu << sh)) | tri[k] << sh;
|
||||
}
|
||||
}
|
||||
mem[COUNT_WORD] = count;
|
||||
}
|
||||
}
|
||||
|
||||
void main() {
|
||||
uint tid = gl_LocalInvocationIndex;
|
||||
if (tid < 1u) mem[COUNT_WORD] = 0u;
|
||||
for (uint i = tid; i < 72u; i += 32u) mem[PAYLOAD + i] = payload[i];
|
||||
barrier();
|
||||
program();
|
||||
barrier();
|
||||
uint count = min(mem[COUNT_WORD], 32u);
|
||||
SetMeshOutputsEXT(64u, count);
|
||||
for (uint v = tid; v < 64u; v += 32u) {
|
||||
gl_MeshVerticesEXT[v].gl_Position = vec4(uintBitsToFloat(mem[VWORD(v, 0u)]), uintBitsToFloat(mem[VWORD(v, 1u)]),
|
||||
uintBitsToFloat(mem[VWORD(v, 2u)]), uintBitsToFloat(mem[VWORD(v, 3u)]));
|
||||
o_color[v] = vec4(uintBitsToFloat(mem[VWORD(v, 4u)]), uintBitsToFloat(mem[VWORD(v, 5u)]),
|
||||
uintBitsToFloat(mem[VWORD(v, 6u)]), uintBitsToFloat(mem[VWORD(v, 7u)]));
|
||||
}
|
||||
for (uint p = tid; p < count; p += 32u) {
|
||||
uvec3 tri;
|
||||
for (uint k = 0u; k < 3u; k++) {
|
||||
uint b = INDEX_BYTE + 3u * p + k;
|
||||
tri[k] = min(mem[b >> 2] >> ((b & 3u) * 8u) & 0xffu, 63u);
|
||||
}
|
||||
gl_PrimitiveTriangleIndicesEXT[p] = tri;
|
||||
}
|
||||
}
|
||||
47
runtime/shaders/meshfb_scan.comp
Normal file
47
runtime/shaders/meshfb_scan.comp
Normal file
@@ -0,0 +1,47 @@
|
||||
#version 450
|
||||
// The mesh fallback's pass between a draw's task dispatch and its mesh dispatch (vk_mesh_fb.c vk_meshfb_scan,
|
||||
// mesh_fb.c): every task workgroup wrote how many mesh workgroups it starts into its payload (word 1). This lists them
|
||||
// in task order, so that mesh workgroup g finds its task and its index among that task's in list entry g, and writes
|
||||
// the draw's record: how many mesh workgroups to dispatch and how many instances to draw.
|
||||
//
|
||||
// One thread walks the tasks: the order of the list is the order the primitives are drawn in, which has to be the
|
||||
// same in every frame and in the depth and colour passes of one frame.
|
||||
layout(local_size_x = 1) in;
|
||||
|
||||
layout(push_constant) uniform Params {
|
||||
uint rec; // the record's first word in the record buffer
|
||||
uint task_words; // words of a task workgroup's payload in the task buffer
|
||||
uint fan; // mesh workgroups a task workgroup starts at most
|
||||
uint slot_words; // words of a mesh workgroup's output in the mesh buffer
|
||||
uint verts; // vertices drawn per mesh workgroup
|
||||
uint max_groups; // workgroups one dispatch may have (maxComputeWorkGroupCount)
|
||||
} pc;
|
||||
|
||||
// The record: words 0-2 the task dispatch (x written before this pass), 4-6 the mesh dispatch, 8-11 the draw (vertices,
|
||||
// instances, first vertex, first instance), 12 the mesh workgroups the tasks asked for, 13 the task workgroups.
|
||||
layout(set = 0, binding = 0) buffer Rec { uint w[]; } rec;
|
||||
layout(set = 0, binding = 1) readonly buffer Task { uint w[]; } task;
|
||||
layout(set = 0, binding = 2) writeonly buffer List { uint e[]; } list;
|
||||
layout(set = 0, binding = 3) readonly buffer Mesh { uint w[]; } mesh; // read for its size alone
|
||||
|
||||
void main() {
|
||||
uint asked = rec.w[pc.rec];
|
||||
uint tasks = min(min(asked, uint(task.w.length()) / pc.task_words), 1u << 24);
|
||||
uint room = min(min(uint(mesh.w.length()) / pc.slot_words, uint(list.e.length())), pc.max_groups);
|
||||
uint fan = min(pc.fan, 255u);
|
||||
uint m = 0u, want = 0u;
|
||||
for (uint t = 0u; t < tasks; t++) {
|
||||
uint c = min(task.w[t * pc.task_words + 1u], fan);
|
||||
want += c;
|
||||
for (uint j = 0u; j < c && m < room; j++) list.e[m++] = t | j << 24;
|
||||
}
|
||||
rec.w[pc.rec + 4u] = m;
|
||||
rec.w[pc.rec + 5u] = 1u;
|
||||
rec.w[pc.rec + 6u] = 1u;
|
||||
rec.w[pc.rec + 8u] = pc.verts;
|
||||
rec.w[pc.rec + 9u] = m;
|
||||
rec.w[pc.rec + 10u] = 0u;
|
||||
rec.w[pc.rec + 11u] = 0u;
|
||||
rec.w[pc.rec + 12u] = want;
|
||||
rec.w[pc.rec + 13u] = asked;
|
||||
}
|
||||
@@ -41,6 +41,8 @@ typedef struct VkCtx {
|
||||
has_maximal_reconvergence, // VK_KHR_shader_maximal_reconvergence (vk_pipeline.c spirv_reconverge)
|
||||
has_pipeline_binary, // VK_KHR_pipeline_binary with VK_KHR_maintenance5 (vk_pipeline.c pipe_create)
|
||||
has_mesh_queries; // meshShaderQueries, kept while NX2NAVI_GPU_PASSES or NX2NAVI_DUMP is set
|
||||
int mesh_fallback; // task and mesh programs run as compute passes and are drawn from buffers
|
||||
// (mesh_fallback: auto without has_mesh_shader; vk_exec_draw.c "mesh fallback")
|
||||
// Feedback loops (maxwell.md 1.9, vk_exec_draw.c feedback_prepare): a draw samples a target of its open pass that
|
||||
// an earlier draw of the pass wrote. NX2NAVI_FEEDBACK=0: VK_FEEDBACK_OFF (sampled in GENERAL without a barrier, the
|
||||
// old behaviour); VK_FEEDBACK_LAYOUT with VK_EXT_attachment_feedback_loop_layout and dynamicRenderingLocalRead
|
||||
|
||||
@@ -469,8 +469,17 @@ static int create_device(void) {
|
||||
g_vk.has_mesh_queries = want && c->mesh.meshShaderQueries && h->pipelineStatisticsQuery && c->f12.hostQueryReset;
|
||||
c->mesh.meshShaderQueries = g_vk.has_mesh_queries ? VK_TRUE : VK_FALSE;
|
||||
if (!c->mesh.meshShader || !c->mesh.taskShader) g_vk.has_mesh_shader = 0;
|
||||
rt_log("vk: mesh shaders: %s", g_vk.has_mesh_shader ? "VK_EXT_mesh_shader" : "not supported (task/mesh draws are skipped)");
|
||||
rt_log("vk: mesh shaders: %s", g_vk.has_mesh_shader ? "VK_EXT_mesh_shader" : "the extension is there, its task or mesh feature is not");
|
||||
} // vkGetDeviceFaultInfoEXT needs the feature, not just the extension
|
||||
{ // mesh_fallback: auto (on a device without mesh shaders), 1 (on every device, for testing) or 0
|
||||
const char *mf = cfg_value("NX2NAVI_MESH_FALLBACK");
|
||||
g_vk.mesh_fallback = !strcmp(mf, "1") || (strcmp(mf, "0") && !g_vk.has_mesh_shader);
|
||||
if (g_vk.mesh_fallback)
|
||||
rt_log("vk: mesh fallback: task and mesh programs run as compute passes and are drawn from their buffers (%s)",
|
||||
g_vk.has_mesh_shader ? "mesh_fallback = 1" : "this device has no mesh shaders");
|
||||
else if (!g_vk.has_mesh_shader)
|
||||
rt_log("vk: mesh shaders: not supported and mesh_fallback = 0 (task/mesh draws are skipped)");
|
||||
}
|
||||
if (g_vk.has_maximal_reconvergence && !c->mrc.shaderMaximalReconvergence) g_vk.has_maximal_reconvergence = 0;
|
||||
if (g_vk.has_pipeline_binary && !(c->pbin.pipelineBinaries && c->m5.maintenance5)) g_vk.has_pipeline_binary = 0;
|
||||
g_vk.has_buffer_device_address = c->f12.bufferDeviceAddress;
|
||||
|
||||
@@ -29,6 +29,8 @@
|
||||
#endif
|
||||
#include "vk_exec.h"
|
||||
#include "vk_gpa.h"
|
||||
#include "mesh_fb.h"
|
||||
#include "vk_mesh_fb.h"
|
||||
#include "platform/platform.h"
|
||||
#include "vk_dlaa.h"
|
||||
|
||||
@@ -446,7 +448,12 @@ static void twins_release(VkCtx *c);
|
||||
|
||||
// Releases the stand-ins and the depth twins. The device must be idle; the game path leaves them to process exit like
|
||||
// the pipelines.
|
||||
static void mesh_ring_release(VkCtx *c);
|
||||
static void pool_marks_release(VkCtx *c);
|
||||
|
||||
void vk_exec_draw_shutdown(void) {
|
||||
vk_meshfb_shutdown();
|
||||
if (vk_ctx()) mesh_ring_release(vk_ctx()), pool_marks_release(vk_ctx()); // made without the stand-ins below too
|
||||
if (!Z.ready) return;
|
||||
VkCtx *c = vk_ctx();
|
||||
twins_release(c);
|
||||
@@ -948,6 +955,13 @@ static void answers_for(VkExec *ex, int st, const uint8_t *ctl, uint32_t mask, G
|
||||
// and *pending = 1 says the wait ended first.
|
||||
static int sf_mvprev; // shader_for asks for the vertex shader's last-frame variant (per-object motion)
|
||||
|
||||
// Probes only (mesh_fb_probe.c): the shader of the program stage whose control section is ctl, in place of a
|
||||
// translation. The probes build their programs from modules made here, which no translator has an input for.
|
||||
static struct { const uint8_t *ctl; const GfxShader *s; } probe_sh[16];
|
||||
static int n_probe_sh;
|
||||
void vk_exec_probe_shader(const uint8_t *ctl, const GfxShader *s) {
|
||||
if (n_probe_sh < 16) probe_sh[n_probe_sh].ctl = ctl, probe_sh[n_probe_sh].s = s, n_probe_sh++;
|
||||
}
|
||||
static const GfxShader *shader_for(VkExec *ex, int st, int *present, int *pending, uint32_t await_ms) {
|
||||
const uint8_t *ctl, *code;
|
||||
uint32_t len;
|
||||
@@ -955,6 +969,8 @@ static const GfxShader *shader_for(VkExec *ex, int st, int *present, int *pendin
|
||||
if (pending) *pending = 0;
|
||||
if (!ex->prog || !find_stage(ex->prog, st, &ctl, &code, &len)) return NULL;
|
||||
*present = 1;
|
||||
for (int i = 0; i < n_probe_sh; i++)
|
||||
if (probe_sh[i].ctl == ctl) return probe_sh[i].s;
|
||||
GfxShaderAnswers a;
|
||||
uint64_t serial = ex->prog->serial;
|
||||
answers_for(ex, st, ctl, slot_mask_of(serial, st), &a);
|
||||
@@ -994,7 +1010,12 @@ static const GfxShader *shader_for(VkExec *ex, int st, int *present, int *pendin
|
||||
|
||||
// ---- descriptors
|
||||
|
||||
// The mesh fallback (below, "mesh fallback") runs a program's task and mesh stages as compute modules: while it binds
|
||||
// one's resources, a compute module's bindings are those of this NVN stage (6 task, 7 mesh), not of the compute stage.
|
||||
static int mfb_stage;
|
||||
|
||||
static int nvn_stage_of(int gfx_stage) {
|
||||
if (gfx_stage == GFX_STAGE_CS && mfb_stage) return mfb_stage;
|
||||
return gfx_stage == GFX_STAGE_VS ? NVN_STAGE_VS : gfx_stage == GFX_STAGE_FS ? NVN_STAGE_FS
|
||||
: gfx_stage == GFX_STAGE_TS ? NVN_STAGE_TASK : gfx_stage == GFX_STAGE_MS ? NVN_STAGE_MESH : NVN_STAGE_CS;
|
||||
}
|
||||
@@ -1373,6 +1394,8 @@ static VkDescriptorBufferInfo ssbo_info(VkExec *ex, int st, const GfxShaderRes *
|
||||
VkDescriptorBufferInfo t = vk_gpa_info();
|
||||
return t.buffer ? t : (VkDescriptorBufferInfo){Z.zbuf, 0, VK_WHOLE_SIZE};
|
||||
}
|
||||
// The mesh fallback's own buffers (mesh_fb.c): no guest memory behind them
|
||||
if (r->cbuf_slot >= MESHFB_CB_TASK && r->cbuf_slot <= MESHFB_CB_LIST) return vk_meshfb_buffer(r->cbuf_slot);
|
||||
if (!__atomic_load_n(&LIM.ubo_max, __ATOMIC_ACQUIRE)) limits_init();
|
||||
uint64_t max = LIM.sbo_max, align = LIM.sbo_align;
|
||||
VkDescriptorBufferInfo bi;
|
||||
@@ -1908,6 +1931,15 @@ static void pool_marks_make(void) {
|
||||
__atomic_store_n(&PL.marks_ready, 1, __ATOMIC_RELEASE); // the recording thread reads the rest after this
|
||||
}
|
||||
|
||||
static void pool_marks_release(VkCtx *c) { // vk_exec_draw_shutdown: a probe ends with the device idle
|
||||
if (!PL.marks) return;
|
||||
__atomic_store_n(&PL.marks_ready, 0, __ATOMIC_RELEASE);
|
||||
vmaDestroyBuffer(c->vma, PL.marks, PL.marks_alloc);
|
||||
for (int i = 0; i < MARKS_RB; i++)
|
||||
if (PL.rb[i].buf) vmaDestroyBuffer(c->vma, PL.rb[i].buf, PL.rb[i].alloc), PL.rb[i].buf = VK_NULL_HANDLE;
|
||||
PL.marks = VK_NULL_HANDLE;
|
||||
}
|
||||
|
||||
void vk_exec_pool_marks(VkExec *ex) {
|
||||
if (!__atomic_load_n(&PL.marks_ready, __ATOMIC_ACQUIRE)) return;
|
||||
VkCtx *c = vk_ctx();
|
||||
@@ -3121,7 +3153,7 @@ static int pb_key(const VkExec *ex, const VkPipeBinding *b, uint32_t k, uint64_t
|
||||
return 1; // slot 1: the program's constants (pb_globals); others: the zero buffer
|
||||
}
|
||||
if (b->kind == GFX_RES_SSBO) {
|
||||
if ((r->flags & 1) || r->ptr || r->cbuf_slot == VK_GPA_CB_SLOT) return 0;
|
||||
if ((r->flags & 1) || r->ptr || r->cbuf_slot >= VK_GPA_CB_SLOT) return 0; // the table, the mesh fallback's buffers
|
||||
int ubo, slot = ssbo_slot(r, st, &ubo);
|
||||
if (slot < 0 || slot >= 16) return 0;
|
||||
const p_bind_buf *p = ubo ? &ex->ubo[st][slot] : &ex->ssbo[st][slot];
|
||||
@@ -3860,8 +3892,13 @@ static void prog_lead(const prog_t *p, int mesh) {
|
||||
}
|
||||
|
||||
// mesh: a mesh draw (op_draw_mesh): the program's mesh and task stages stand in for the vertex stage, and nothing
|
||||
// vertex-related (vertex buffers, motion vector bookkeeping of vertex shaders) is bound or noted.
|
||||
// vertex-related (vertex buffers, motion vector bookkeeping of vertex shaders) is bound or noted. 2: a mesh draw of
|
||||
// the mesh fallback (mfb_draw): the vertex stage is mfb_cur_vs, in an ordinary pipeline without vertex input.
|
||||
static const GfxShader *mfb_cur_vs;
|
||||
|
||||
static int prepare_draw_ex(VkExec *ex, int prim, int32_t count, int restart, int mesh) {
|
||||
int fb = mesh == 2; // from here on mesh says what the draw is, and !fb that its pipeline is a mesh pipeline
|
||||
mesh = mesh != 0;
|
||||
if (ex->render_disabled) return vk_cond_draws_skip += ex->has_render_cond, draw_skipped(ex, mesh, SK_DISABLED);
|
||||
cb_res(ex);
|
||||
dummies_init();
|
||||
@@ -3892,8 +3929,9 @@ static int prepare_draw_ex(VkExec *ex, int prim, int32_t count, int restart, int
|
||||
}
|
||||
} else {
|
||||
SM.valid = 0;
|
||||
vs = shader_for(ex, vst, &has_vs, &vs_wait, 0);
|
||||
if (mesh) ts = shader_for(ex, NVN_STAGE_TASK, &has_ts, &ts_wait, 0);
|
||||
if (fb) vs = mfb_cur_vs, has_vs = 1;
|
||||
else vs = shader_for(ex, vst, &has_vs, &vs_wait, 0);
|
||||
if (mesh && !fb) ts = shader_for(ex, NVN_STAGE_TASK, &has_ts, &ts_wait, 0);
|
||||
fs = shader_for(ex, NVN_STAGE_FS, &has_fs, &fs_wait, 0);
|
||||
if (memo && vs && !vs_wait && !fs_wait && (!has_fs || fs))
|
||||
memo_key(ex, gen, &SM.k), SM.vs = vs, SM.fs = fs, SM.has_fs = has_fs, SM.valid = 1;
|
||||
@@ -3927,6 +3965,7 @@ static int prepare_draw_ex(VkExec *ex, int prim, int32_t count, int restart, int
|
||||
VkPipeState s = ex->pipe;
|
||||
s.vs = vs, s.fs = fs, s.ts = ts;
|
||||
if (mesh) foliage_shadow_cull(ex, &s);
|
||||
if (fb) memset(s.vattr, 0, sizeof s.vattr), memset(s.vstream, 0, sizeof s.vstream); // whatever the last vertex draw left
|
||||
s.feedback = (uint8_t)((fbl & 0xff ? 1 : 0) | (fbl & 0x100 ? 2 : 0));
|
||||
s.prim = (uint8_t)prim;
|
||||
s.depth_minus_w = (uint8_t)(nvn_depth_minus_w() != 0);
|
||||
@@ -3946,7 +3985,7 @@ static int prepare_draw_ex(VkExec *ex, int prim, int32_t count, int restart, int
|
||||
int fullscreen = 0; // the wait is a full-screen draw's
|
||||
if (oneshot > 0 && oneshot_budget_on()) {
|
||||
budget = frame_wait();
|
||||
wait_ms = oneshot_ms(budget, mesh, &s);
|
||||
wait_ms = oneshot_ms(budget, mesh && !fb, &s);
|
||||
if (wait_ms) t0 = kern_now_ns();
|
||||
} else if (oneshot > 0) {
|
||||
wait_ms = ONESHOT_WAIT_MS;
|
||||
@@ -3984,7 +4023,7 @@ static int prepare_draw_ex(VkExec *ex, int prim, int32_t count, int restart, int
|
||||
vk_exec_ahead(ex);
|
||||
// A mesh pipeline that is known to be compiling gets no fast link, so the few milliseconds a draw waits
|
||||
// for one are lost in every frame that meets it until the compile ends, 0.2 to 1.1 s later (MESH_WAIT_MS).
|
||||
if (wait_ms && linking && mesh && vk_pipeline_building_ms(&s) > MESH_WAIT_MS) wait_ms = 0;
|
||||
if (wait_ms && linking && mesh && !fb && vk_pipeline_building_ms(&s) > MESH_WAIT_MS) wait_ms = 0;
|
||||
if (wait_ms) p = vk_pipeline_graphics_async(&s, &info, &pending, wait_ms);
|
||||
}
|
||||
if (t0 && budget) {
|
||||
@@ -3999,7 +4038,7 @@ static int prepare_draw_ex(VkExec *ex, int prim, int32_t count, int restart, int
|
||||
if (!w0) w0 = kern_now_ns();
|
||||
if (oneshot_budget_on()) {
|
||||
uint64_t *pool = frame_wait(), b0 = kern_now_ns();
|
||||
p = vk_pipeline_graphics_async(&s, &info, &pending, oneshot_ms(pool, mesh, &s));
|
||||
p = vk_pipeline_graphics_async(&s, &info, &pending, oneshot_ms(pool, mesh && !fb, &s));
|
||||
pool_charge(pool, b0);
|
||||
} else {
|
||||
p = vk_pipeline_graphics_async(&s, &info, &pending, ONESHOT_WAIT_MS);
|
||||
@@ -4045,7 +4084,7 @@ static int prepare_draw_ex(VkExec *ex, int prim, int32_t count, int restart, int
|
||||
delta = cfg_bool("NX2NAVI_DYN_DELTA");
|
||||
if (!delta) rt_log("vk_exec: every dynamic state recorded after each pipeline bind (NX2NAVI_DYN_DELTA=0)");
|
||||
}
|
||||
if (delta) vk_exec_frame.dyn_calls += (uint64_t)vk_pipeline_set_dynamic_delta(ex->cb, &s, &B.dyn, mesh,
|
||||
if (delta) vk_exec_frame.dyn_calls += (uint64_t)vk_pipeline_set_dynamic_delta(ex->cb, &s, &B.dyn, mesh && !fb,
|
||||
&vk_exec_frame.dyn_left);
|
||||
else vk_pipeline_set_dynamic(ex->cb, &s);
|
||||
ex->recorded++;
|
||||
@@ -4068,7 +4107,7 @@ static int prepare_draw_ex(VkExec *ex, int prim, int32_t count, int restart, int
|
||||
if (fbl & ex->fb_dirty) feedback_barrier(ex);
|
||||
feedback_written(ex, &s, fbl);
|
||||
if (!mesh) bind_vertex_buffers(ex);
|
||||
if (vk_dump_armed) vk_dump_draw(ex, &s, p, info, mesh);
|
||||
if (vk_dump_armed) vk_dump_draw(ex, &s, p, info, mesh && !fb);
|
||||
if (vk_hdr_track) vk_hdr_draw(ex, info);
|
||||
vk_exec_oq_draw(ex);
|
||||
DRS.issued[mesh]++;
|
||||
@@ -4951,12 +4990,266 @@ static void mesh_shading_rate(VkExec *ex) {
|
||||
ex->recorded++;
|
||||
}
|
||||
|
||||
// ---- mesh fallback: a mesh draw on a device without VK_EXT_mesh_shader (mesh_fallback, vk_device.c), or on any
|
||||
// device with mesh_fallback = 1. The program's task and mesh stages run as compute dispatches that write the
|
||||
// fallback's buffers, and an ordinary draw reads them (mesh_fb.c makes the modules, vk_mesh_fb.c holds the buffers):
|
||||
// the draw's record its workgroup count, copied on the GPU when the game's own record holds it
|
||||
// [task dispatch the task module, with the task stage's bindings
|
||||
// scan lists the mesh workgroups the tasks started and counts them into the record]
|
||||
// mesh dispatch the mesh module, with the mesh stage's bindings
|
||||
// draw one instance per mesh workgroup, with the generated vertex stage and the program's
|
||||
// fragment stage, the instance count from the record
|
||||
// Every step reads what the one before wrote, so a full barrier stands between them, and the dispatches cannot be
|
||||
// recorded inside a pass: each such draw ends the open pass, and the next draw begins it again. The buffers are
|
||||
// reused by every draw (vk_mesh_fb.c), which the barrier before a draw's first dispatch makes safe: the previous
|
||||
// draw's vertex stage has read them by then.
|
||||
//
|
||||
// The modules of a program are made once, on a thread of their own, with their compute pipelines (a mesh program of
|
||||
// the game is up to 900,000 SPIR-V words, and its pipeline compiles for as long as the mesh pipeline it replaces):
|
||||
// until they are ready the program's draws are skipped, like draws whose pipeline is being built. Without background
|
||||
// pipeline builds (the probes) they are made at the draw.
|
||||
typedef struct MfbCs {
|
||||
const GfxShader *ms, *ts; // the key: the translated mesh and task shaders (ts NULL: none)
|
||||
MeshFbInfo mi, ti;
|
||||
volatile int32_t state; // 0 queued or being made, 1 ready, -1 failed
|
||||
const GfxShader *cs_mesh, *cs_task; // the compute modules (gfx_shader_own)
|
||||
struct MfbCs *next; // the worker's queue
|
||||
} MfbCs;
|
||||
typedef struct { const GfxShader *ms; uint32_t mask; uint8_t type[32]; const GfxShader *vs; } MfbVs;
|
||||
#define MFB_TAB 4096
|
||||
static MfbCs *mfb_cs_tab[MFB_TAB];
|
||||
static MfbVs mfb_vs_tab[MFB_TAB];
|
||||
static struct {
|
||||
PlatMutex lock;
|
||||
PlatCond wake;
|
||||
PlatThread *th;
|
||||
MfbCs *head, *tail;
|
||||
uint64_t made, failed, pending_skips;
|
||||
} MFB = {.lock = PLAT_MUTEX_INIT};
|
||||
|
||||
static uint32_t mfb_binding_base(const GfxShader *s) { // one past the highest binding the module declares
|
||||
uint64_t n;
|
||||
const uint32_t *w = gfx_shader_spirv(s, &n);
|
||||
uint32_t top = 0;
|
||||
for (uint64_t i = 5; i < n;) {
|
||||
uint32_t op = w[i] & 0xffff, len = w[i] >> 16;
|
||||
if (!len) break;
|
||||
if (op == 71 && len == 4 && w[i + 2] == 33 && w[i + 3] > top) top = w[i + 3]; // OpDecorate Binding
|
||||
i += len;
|
||||
}
|
||||
return top + 1;
|
||||
}
|
||||
|
||||
// The compute module of stage (6 task, 7 mesh) of e, as a shader of the runtime's own with the stage's resources and
|
||||
// the fallback's buffers.
|
||||
static const GfxShader *mfb_module(const MfbCs *e, int stage, char *err, size_t cap) {
|
||||
const GfxShader *src = stage == 6 ? e->ts : e->ms;
|
||||
uint64_t n, on = 0;
|
||||
const uint32_t *w = gfx_shader_spirv(src, &n);
|
||||
uint32_t base = mfb_binding_base(src);
|
||||
uint32_t *out = stage == 6 ? meshfb_task_cs(w, n, &e->ti, base, &on, err, cap)
|
||||
: meshfb_mesh_cs(w, n, &e->mi, base, e->ti.payload_bytes / 4, &on, err, cap);
|
||||
if (!out) return NULL;
|
||||
static const int task_slots[1] = {MESHFB_CB_TASK}, mesh_slots[3] = {MESHFB_CB_TASK, MESHFB_CB_MESH, MESHFB_CB_LIST};
|
||||
char *res = stage == 6 ? meshfb_res_json(gfx_shader_res_json(src), base, task_slots, 1)
|
||||
: e->mi.has_task ? meshfb_res_json(gfx_shader_res_json(src), base, mesh_slots, 3)
|
||||
: meshfb_res_json(gfx_shader_res_json(src), base + 1, mesh_slots + 1, 1);
|
||||
const GfxShader *s = res ? gfx_shader_own(out, on, res) : NULL;
|
||||
if (!s) snprintf(err, cap, "the converted module or its reflection is malformed");
|
||||
free(out), free(res);
|
||||
return s;
|
||||
}
|
||||
|
||||
static void mfb_make(MfbCs *e) {
|
||||
char err[200] = "";
|
||||
const VkPipeInfo *info;
|
||||
int ok = (e->cs_mesh = mfb_module(e, 7, err, sizeof err)) != NULL;
|
||||
if (ok && e->ts) ok = (e->cs_task = mfb_module(e, 6, err, sizeof err)) != NULL;
|
||||
if (ok && !vk_pipeline_compute(e->cs_mesh, &info)) ok = 0, snprintf(err, sizeof err, "the mesh stage's compute pipeline failed");
|
||||
if (ok && e->ts && !vk_pipeline_compute(e->cs_task, &info)) ok = 0, snprintf(err, sizeof err, "the task stage's compute pipeline failed");
|
||||
plat_mutex_lock(&MFB.lock);
|
||||
uint64_t nth = ok ? ++MFB.made : ++MFB.failed;
|
||||
plat_mutex_unlock(&MFB.lock);
|
||||
if (!ok && (nth <= 8 || !(nth & (nth - 1))))
|
||||
rt_log("vk: mesh fallback: a program's modules could not be made (%s): its draws are skipped (%llu such programs)", err,
|
||||
(unsigned long long)nth);
|
||||
__atomic_store_n(&e->state, ok ? 1 : -1, __ATOMIC_RELEASE);
|
||||
}
|
||||
|
||||
static void mfb_worker(void *arg) {
|
||||
(void)arg;
|
||||
plat_thread_priority(PLAT_PRIO_BELOW_NORMAL); // as the pipeline workers' optimised builds: the game's threads go first
|
||||
for (;;) {
|
||||
plat_mutex_lock(&MFB.lock);
|
||||
while (!MFB.head) plat_cond_wait(&MFB.wake, &MFB.lock);
|
||||
MfbCs *e = MFB.head;
|
||||
if (!(MFB.head = e->next)) MFB.tail = NULL;
|
||||
plat_mutex_unlock(&MFB.lock);
|
||||
mfb_make(e);
|
||||
}
|
||||
}
|
||||
|
||||
// The entry of (ms, ts), made and queued at its first draw; NULL (logged) when the program's layout is not carried.
|
||||
static MfbCs *mfb_cs_get(VkExec *ex, const GfxShader *ms, const GfxShader *ts) {
|
||||
uint32_t i = (uint32_t)((((uintptr_t)ms >> 4) * 0x9e3779b1u) ^ (((uintptr_t)ts >> 4) * 0x85ebca6bu)) & (MFB_TAB - 1);
|
||||
for (uint32_t k = 0; k < MFB_TAB; k++, i = (i + 1) & (MFB_TAB - 1)) {
|
||||
MfbCs *e = mfb_cs_tab[i];
|
||||
if (e && (e->ms != ms || e->ts != ts)) continue;
|
||||
if (e) return e;
|
||||
const uint8_t *ctl, *code;
|
||||
uint32_t len;
|
||||
char err[160] = "";
|
||||
e = calloc(1, sizeof *e);
|
||||
if (!e) rt_fatal("vk: out of memory");
|
||||
e->ms = ms, e->ts = ts, e->state = -1;
|
||||
mfb_cs_tab[i] = e; // a program that is refused is not looked at again
|
||||
int ok = find_stage_scan(ex->prog, NVN_STAGE_MESH, &ctl, &code, &len) && len >= 0x80 &&
|
||||
meshfb_info(ctl, code, 7, &e->mi, err, sizeof err);
|
||||
if (ok && ts) ok = find_stage_scan(ex->prog, NVN_STAGE_TASK, &ctl, &code, &len) && meshfb_info(ctl, code, 6, &e->ti, err, sizeof err);
|
||||
if (ok && e->mi.has_task != (ts != NULL)) ok = 0, snprintf(err, sizeof err, "the mesh stage %s a task stage and the program has %s",
|
||||
e->mi.has_task ? "expects" : "does not expect", ts ? "one" : "none");
|
||||
if (!ok) {
|
||||
warn_once(W_MESH, "mesh fallback: a program is not carried (%s): its draws are skipped", *err ? err : "no stage data");
|
||||
return e;
|
||||
}
|
||||
e->state = 0;
|
||||
if (!vk_pipeline_async_on()) {
|
||||
mfb_make(e);
|
||||
return e;
|
||||
}
|
||||
plat_mutex_lock(&MFB.lock);
|
||||
if (!MFB.th) plat_cond_init(&MFB.wake), MFB.th = plat_thread_new(mfb_worker, NULL, "mesh fallback");
|
||||
if (MFB.tail) MFB.tail->next = e;
|
||||
else MFB.head = e;
|
||||
MFB.tail = e;
|
||||
plat_cond_signal(&MFB.wake);
|
||||
plat_mutex_unlock(&MFB.lock);
|
||||
return e;
|
||||
}
|
||||
rt_fatal("vk: mesh fallback: more than %d task and mesh program pairs", MFB_TAB);
|
||||
}
|
||||
|
||||
// The vertex stage that draws e's mesh buffer slots for fragment shader fs: an output for every input fs reads, in
|
||||
// its type.
|
||||
static const GfxShader *mfb_vs_get(const MfbCs *e, const GfxShader *fs) {
|
||||
GfxShaderIo fin[32];
|
||||
uint8_t type[32] = {0};
|
||||
uint32_t mask = fs ? gfx_shader_inputs(fs, fin) : 0;
|
||||
for (int g = 0; g < 32; g++)
|
||||
if (mask >> g & 1) type[g] = fin[g].type;
|
||||
uint32_t i = (uint32_t)((((uintptr_t)e->ms >> 4) * 0x9e3779b1u) ^ (mask * 0x85ebca6bu)) & (MFB_TAB - 1);
|
||||
for (uint32_t k = 0; k < MFB_TAB; k++, i = (i + 1) & (MFB_TAB - 1)) {
|
||||
MfbVs *v = &mfb_vs_tab[i];
|
||||
if (v->ms && (v->ms != e->ms || v->mask != mask || memcmp(v->type, type, 32))) continue;
|
||||
if (v->ms) return v->vs;
|
||||
static const char res[] = "{\"key\":\"meshfb\",\"stage\":\"vs\",\"notes\":[],\"cbufs\":[],\"sbufs\":[{\"binding\": 1, "
|
||||
"\"slot\": -1, \"cb_slot\": 66, \"cb_offset\": 0, \"flags\": 0}],\"textures\":[],\"images\":[]}";
|
||||
_Static_assert(MESHFB_CB_MESH == 66, "the vertex stage's reflection names the mesh buffer");
|
||||
uint64_t n = 0;
|
||||
uint32_t *w = meshfb_vs(&e->mi, mask, type, &n);
|
||||
v->ms = e->ms, v->mask = mask, memcpy(v->type, type, 32);
|
||||
v->vs = w ? gfx_shader_own(w, n, res) : NULL;
|
||||
free(w);
|
||||
if (!v->vs) warn_once(W_MESH, "mesh fallback: a vertex stage could not be made: its draws are skipped");
|
||||
return v->vs;
|
||||
}
|
||||
rt_fatal("vk: mesh fallback: more than %d vertex stages", MFB_TAB);
|
||||
}
|
||||
|
||||
// One of the draw's compute dispatches: module cs with the bindings of NVN stage `stage`, its group counts read from
|
||||
// the record at `at`. 0 when its pipeline is not there.
|
||||
static int mfb_dispatch(VkExec *ex, const GfxShader *cs, int stage, VkBuffer rec, VkDeviceSize at) {
|
||||
const VkPipeInfo *info = NULL;
|
||||
VkPipeline p = vk_pipeline_compute(cs, &info);
|
||||
if (!p || !info || (info->pool && !pool_on())) return 0;
|
||||
bind_pipeline(ex, VK_PIPELINE_BIND_POINT_COMPUTE, p);
|
||||
mfb_stage = stage;
|
||||
bind_resources(ex, info);
|
||||
mfb_stage = 0;
|
||||
VkPassMark *m = dispatch_mark(ex, cs, 0, 0, 0);
|
||||
vkCmdDispatchIndirect(ex->cb, rec, at);
|
||||
vk_pass_mark_end(ex->cb, m);
|
||||
ex->recorded++;
|
||||
return 1;
|
||||
}
|
||||
|
||||
// A mesh draw of `count` workgroups, or of the count in the game's record at ind (4 bytes, GPU-written).
|
||||
static void mfb_draw(VkExec *ex, uint32_t count, const VkMemSpan *ind) {
|
||||
if (ex->render_disabled) {
|
||||
vk_cond_draws_skip += ex->has_render_cond;
|
||||
draw_skipped(ex, 1, SK_DISABLED);
|
||||
return;
|
||||
}
|
||||
if (!ex->prog) { draw_skipped(ex, 1, SK_NO_PROGRAM); return; }
|
||||
cb_res(ex);
|
||||
dummies_init();
|
||||
int has_ms, has_ts, has_fs, w_ms = 0, w_ts = 0, w_fs = 0;
|
||||
const GfxShader *ms = shader_for(ex, NVN_STAGE_MESH, &has_ms, &w_ms, 0);
|
||||
const GfxShader *ts = shader_for(ex, NVN_STAGE_TASK, &has_ts, &w_ts, 0);
|
||||
const GfxShader *fs = shader_for(ex, NVN_STAGE_FS, &has_fs, &w_fs, 0);
|
||||
if (!has_ms) { draw_skipped(ex, 1, SK_NO_VERTEX_STAGE); return; }
|
||||
if (w_ms || w_ts || w_fs) {
|
||||
vk_exec_ahead(ex);
|
||||
perf_count(PERF_CNT_SHADER_SKIP);
|
||||
draw_skipped(ex, 1, SK_SHADER_PENDING);
|
||||
return;
|
||||
}
|
||||
if (!ms || (has_ts && !ts) || (has_fs && !fs)) { draw_skipped(ex, 1, SK_SHADER_FAILED); return; }
|
||||
MfbCs *c = mfb_cs_get(ex, ms, ts);
|
||||
int state = __atomic_load_n(&c->state, __ATOMIC_ACQUIRE);
|
||||
if (!state && !vk_exec_targets_recent(ex, pipe_skip_mode() >= 1)) {
|
||||
// a target the game does not render every frame: the draw may not come again, so it waits for its modules
|
||||
// from the frame's wait pool, as such a draw waits for a pipeline (prepare_draw_ex)
|
||||
uint64_t *pool = frame_wait(), t0 = kern_now_ns();
|
||||
while (!state && pool_ms(pool, t0)) plat_sleep_ms(1), state = __atomic_load_n(&c->state, __ATOMIC_ACQUIRE);
|
||||
pool_charge(pool, t0);
|
||||
}
|
||||
if (!state) { MFB.pending_skips++; draw_skipped(ex, 1, SK_PIPE_PENDING); return; }
|
||||
const GfxShader *vs = state > 0 ? mfb_vs_get(c, fs) : NULL;
|
||||
if (!vs || !vk_meshfb_init()) { draw_skipped(ex, 1, SK_PIPE_FAILED); return; }
|
||||
uint32_t verts = c->mi.maxp * c->mi.ipp, slot_words = meshfb_slot_words(&c->mi);
|
||||
uint32_t most = vk_ctx()->props.limits.maxComputeWorkGroupCount[0];
|
||||
if (count > most) count = most; // a count the GPU wrote is not looked at here: the dispatch reads it as it is
|
||||
VkBuffer rec;
|
||||
int cmds;
|
||||
vk_exec_barrier(ex); // ends the pass; the culling's record and the last draw's reads come before what follows
|
||||
VkDeviceSize at = vk_meshfb_record(ex->cb, ind ? ind->buffer : VK_NULL_HANDLE, ind ? ind->offset : 0, count, ts != NULL,
|
||||
verts, slot_words, &rec, &cmds);
|
||||
ex->recorded += (uint32_t)cmds;
|
||||
vk_exec_barrier(ex);
|
||||
if (ts) {
|
||||
if (!mfb_dispatch(ex, c->cs_task, NVN_STAGE_TASK, rec, at)) { draw_skipped(ex, 1, SK_PIPE_FAILED); return; }
|
||||
vk_exec_barrier(ex);
|
||||
vk_meshfb_scan(ex->cb, at, c->ti.payload_bytes / 4, c->ti.threads, slot_words, verts);
|
||||
B.pipe[1] = VK_NULL_HANDLE; // the scan bound its own
|
||||
ex->recorded += 4;
|
||||
vk_exec_barrier(ex);
|
||||
}
|
||||
if (!mfb_dispatch(ex, c->cs_mesh, NVN_STAGE_MESH, rec, at + 16)) { draw_skipped(ex, 1, SK_PIPE_FAILED); return; }
|
||||
vk_exec_barrier(ex);
|
||||
vk_meshfb_note(ex->cb, at);
|
||||
ex->recorded++;
|
||||
vk_exec_barrier(ex);
|
||||
c0_binst = 0;
|
||||
mfb_cur_vs = vs;
|
||||
int prim = c->mi.ipp == 1 ? NVN_DRAW_PRIMITIVE_POINTS : c->mi.ipp == 2 ? NVN_DRAW_PRIMITIVE_LINES : NVN_DRAW_PRIMITIVE_TRIANGLES;
|
||||
if (!prepare_draw_ex(ex, prim, 0x7fffffff, 0, 2)) return;
|
||||
vk_exec_frame.draws++;
|
||||
vkCmdDrawIndirect(ex->cb, rec, at + 32, 1, sizeof(VkDrawIndirectCommand));
|
||||
ex->recorded++;
|
||||
}
|
||||
|
||||
static void op_draw_mesh(VkExec *ex, const p_draw_mesh *d) {
|
||||
PFN_vkCmdDrawMeshTasksEXT fn = draw_mesh_fn();
|
||||
if (!fn) return warn_once(W_MESH, "mesh draws need VK_EXT_mesh_shader: skipped");
|
||||
static int on = -1;
|
||||
if (on < 0) on = rt_knob("MESH", "1"); // NX2NAVI_MESH=0: mesh draws are skipped (diagnosis)
|
||||
if (!d->count || !on) return;
|
||||
if (vk_ctx()->mesh_fallback) {
|
||||
if (d->first) warn_once(W_MESH, "a mesh draw from workgroup %u: drawn from workgroup 0", d->first);
|
||||
mfb_draw(ex, d->count, NULL);
|
||||
return;
|
||||
}
|
||||
PFN_vkCmdDrawMeshTasksEXT fn = draw_mesh_fn();
|
||||
if (!fn) return warn_once(W_MESH, "mesh draws need VK_EXT_mesh_shader: skipped");
|
||||
c0_binst = 0;
|
||||
if (!prepare_draw_ex(ex, NVN_DRAW_PRIMITIVE_TRIANGLES, 0x7fffffff, 0, 1)) return;
|
||||
if (d->first) warn_once(W_MESH, "a mesh draw from workgroup %u: drawn from workgroup 0", d->first);
|
||||
@@ -4980,13 +5273,21 @@ static void op_draw_mesh(VkExec *ex, const p_draw_mesh *d) {
|
||||
// MESH_RING entries outlast the frames in flight: an entry is rewritten only after a barrier behind its last draw.
|
||||
enum { MESH_RING = 4096 };
|
||||
static VkBuffer mesh_ring;
|
||||
static VmaAllocation mesh_ring_alloc;
|
||||
static uint32_t mesh_ring_next;
|
||||
|
||||
// NX2NAVI_GPU_PASSES: the workgroup counts of the indirect mesh draws of the timed frames, copied to host memory.
|
||||
int vk_gpu_passes_capturing(void);
|
||||
enum { MESH_COUNTS = 4096 };
|
||||
static VkBuffer mesh_count_buf;
|
||||
static VmaAllocation mesh_count_alloc;
|
||||
static uint32_t *mesh_count_map, mesh_count_n, mesh_count_was;
|
||||
|
||||
static void mesh_ring_release(VkCtx *c) { // vk_exec_draw_shutdown: a probe ends with the device idle
|
||||
if (mesh_ring) vmaDestroyBuffer(c->vma, mesh_ring, mesh_ring_alloc);
|
||||
if (mesh_count_buf) vmaDestroyBuffer(c->vma, mesh_count_buf, mesh_count_alloc);
|
||||
mesh_ring = mesh_count_buf = VK_NULL_HANDLE, mesh_count_map = NULL;
|
||||
}
|
||||
uint32_t vk_exec_mesh_counts(const uint32_t **out) {
|
||||
*out = mesh_count_map;
|
||||
return mesh_count_map ? mesh_count_n : 0;
|
||||
@@ -5001,8 +5302,7 @@ static void mesh_count_note(VkExec *ex, const VkMemSpan *src) {
|
||||
VmaAllocationCreateInfo ai = {.usage = VMA_MEMORY_USAGE_AUTO_PREFER_HOST,
|
||||
.flags = VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT | VMA_ALLOCATION_CREATE_MAPPED_BIT};
|
||||
VmaAllocationInfo info;
|
||||
VmaAllocation a;
|
||||
if (vmaCreateBuffer(vk_ctx()->vma, &bi, &ai, &mesh_count_buf, &a, &info) != VK_SUCCESS) return;
|
||||
if (vmaCreateBuffer(vk_ctx()->vma, &bi, &ai, &mesh_count_buf, &mesh_count_alloc, &info) != VK_SUCCESS) return;
|
||||
mesh_count_map = info.pMappedData;
|
||||
}
|
||||
if (mesh_count_n >= MESH_COUNTS) return;
|
||||
@@ -5053,8 +5353,7 @@ static int mesh_ring_entry(VkExec *ex, const p_draw_mesh_ind *d, const VkMemSpan
|
||||
VkBufferCreateInfo bi = {.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, .size = 12ull * MESH_RING,
|
||||
.usage = VK_BUFFER_USAGE_INDIRECT_BUFFER_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT};
|
||||
VmaAllocationCreateInfo ai = {.usage = VMA_MEMORY_USAGE_AUTO_PREFER_DEVICE};
|
||||
VmaAllocation a;
|
||||
if (vmaCreateBuffer(vk_ctx()->vma, &bi, &ai, &mesh_ring, &a, NULL) != VK_SUCCESS) {
|
||||
if (vmaCreateBuffer(vk_ctx()->vma, &bi, &ai, &mesh_ring, &mesh_ring_alloc, NULL) != VK_SUCCESS) {
|
||||
mesh_ring = VK_NULL_HANDLE;
|
||||
return 0;
|
||||
}
|
||||
@@ -5107,7 +5406,8 @@ static void op_draw_mesh_ind(VkExec *ex, const p_draw_mesh_ind *d) {
|
||||
if (vk_ctx()->has_mesh_shader)
|
||||
fn = (PFN_vkCmdDrawMeshTasksIndirectEXT)vk_dpa(vk_ctx()->device, "vkCmdDrawMeshTasksIndirectEXT");
|
||||
}
|
||||
if (!fn) return warn_once(W_MESH, "mesh draws need VK_EXT_mesh_shader: skipped");
|
||||
int fallback = vk_ctx()->mesh_fallback;
|
||||
if (!fn && !fallback) return warn_once(W_MESH, "mesh draws need VK_EXT_mesh_shader: skipped");
|
||||
static int on = -1;
|
||||
if (on < 0) on = rt_knob("MESH", "1");
|
||||
if (!on) return;
|
||||
@@ -5117,6 +5417,12 @@ static void op_draw_mesh_ind(VkExec *ex, const p_draw_mesh_ind *d) {
|
||||
(unsigned long long)d->ind);
|
||||
c0_binst = 0;
|
||||
VK_MEM_USE(ip);
|
||||
if (fallback) {
|
||||
mfb_draw(ex, 0, &ip);
|
||||
vk_mem_count(ip.cls, VK_MUSE_IND, 4);
|
||||
VK_MEM_BOUND(ip);
|
||||
return;
|
||||
}
|
||||
VkDeviceSize at;
|
||||
if (!mesh_ring_entry(ex, d, &ip, &at)) return warn_once(W_MESH, "the indirect mesh ring could not be created: skipped");
|
||||
if (!prepare_draw_ex(ex, NVN_DRAW_PRIMITIVE_TRIANGLES, 0x7fffffff, 0, 1)) return;
|
||||
|
||||
215
runtime/vk_mesh_fb.c
Normal file
215
runtime/vk_mesh_fb.c
Normal file
@@ -0,0 +1,215 @@
|
||||
// The mesh fallback's buffers and records. A task or mesh program runs as a compute dispatch that writes a storage
|
||||
// buffer (mesh_fb.c), and the draw reads it:
|
||||
// task buffer a task workgroup's payload at its index, payload-size apart
|
||||
// list per mesh workgroup after a task stage: its task and its index among that task's (meshfb_scan.comp)
|
||||
// mesh buffer a mesh workgroup's vertex lines and primitive area at its slot
|
||||
// records per draw 64 bytes: the indirect arguments of its task dispatch, its mesh dispatch and its draw
|
||||
// The three buffers are reused by every draw: a draw's dispatches write them and its vertex stage has read them
|
||||
// before the next draw's dispatches run, with a barrier in between (vk_exec_draw.c mfb_draw), so their size is what
|
||||
// the largest draw needs, not what a frame's draws need together. mesh_fallback_mb sets it: an eighth for the task
|
||||
// buffer, the rest for the mesh buffer. Workgroups past either buffer's end write nothing and draw nothing (the
|
||||
// modules check the buffer's length); the draws that happened to are counted and the log names the size they needed.
|
||||
// A record is used by one draw and written again only after MESHFB_RECS further draws.
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
|
||||
#include "cfg.h"
|
||||
#include "host.h"
|
||||
#include "mesh_fb.h"
|
||||
#include "vk.h"
|
||||
#include "vk_mesh_fb.h"
|
||||
#include "meshfb_scan_spv.h"
|
||||
|
||||
enum { MESHFB_RECS = 4096, REC_BYTES = 64 };
|
||||
|
||||
static struct {
|
||||
int tried, ok;
|
||||
VkBuffer task, mesh, list, rec, stat;
|
||||
VmaAllocation task_a, mesh_a, list_a, rec_a, stat_a;
|
||||
VkDeviceSize task_bytes, mesh_bytes, list_bytes; // as one storage binding spans them
|
||||
const uint32_t *stat_map; // per record: mesh workgroups and task workgroups it asked for
|
||||
uint32_t cap_slots[MESHFB_RECS], slot_words[MESHFB_RECS], cap_tasks[MESHFB_RECS]; // what the record's draw had room for
|
||||
uint8_t noted[MESHFB_RECS];
|
||||
uint32_t next;
|
||||
uint64_t draws, short_draws, worst_bytes;
|
||||
PFN_vkCmdPushDescriptorSetKHR push;
|
||||
VkDescriptorSetLayout dsl;
|
||||
VkPipelineLayout layout;
|
||||
VkPipeline scan;
|
||||
} M;
|
||||
|
||||
static int make_buffer(VkCtx *c, VkDeviceSize size, VkBufferUsageFlags usage, int host, VkBuffer *b, VmaAllocation *a, void **map) {
|
||||
VkBufferCreateInfo bi = {.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, .size = size, .usage = usage};
|
||||
VmaAllocationCreateInfo ai = {.usage = host ? VMA_MEMORY_USAGE_AUTO_PREFER_HOST : VMA_MEMORY_USAGE_AUTO_PREFER_DEVICE,
|
||||
.flags = host ? VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT | VMA_ALLOCATION_CREATE_MAPPED_BIT : 0};
|
||||
VmaAllocationInfo info;
|
||||
if (vmaCreateBuffer(c->vma, &bi, &ai, b, a, &info) != VK_SUCCESS) return *b = VK_NULL_HANDLE, 0;
|
||||
if (map) *map = info.pMappedData;
|
||||
return !host || info.pMappedData;
|
||||
}
|
||||
|
||||
int vk_meshfb_init(void) {
|
||||
if (M.tried) return M.ok;
|
||||
M.tried = 1;
|
||||
VkCtx *c = vk_ctx();
|
||||
if (!c) return 0;
|
||||
uint64_t mb = (uint64_t)cfg_int("NX2NAVI_MESH_FALLBACK_MB");
|
||||
if (mb < 16) mb = 16;
|
||||
if (mb > 4096) mb = 4096;
|
||||
VkDeviceSize range = c->props.limits.maxStorageBufferRange & ~(VkDeviceSize)3, task = (mb << 20) / 8, mesh = (mb << 20) - task;
|
||||
VkDeviceSize list = mesh / 128; // an entry per slot of 512 bytes, and no mesh program's slot is smaller
|
||||
void *map = NULL;
|
||||
const VkBufferUsageFlags st = VK_BUFFER_USAGE_STORAGE_BUFFER_BIT;
|
||||
if (!make_buffer(c, task, st, 0, &M.task, &M.task_a, NULL) || !make_buffer(c, mesh, st, 0, &M.mesh, &M.mesh_a, NULL) ||
|
||||
!make_buffer(c, list, st, 0, &M.list, &M.list_a, NULL) ||
|
||||
!make_buffer(c, (VkDeviceSize)MESHFB_RECS * REC_BYTES,
|
||||
st | VK_BUFFER_USAGE_INDIRECT_BUFFER_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT | VK_BUFFER_USAGE_TRANSFER_SRC_BIT, 0,
|
||||
&M.rec, &M.rec_a, NULL) ||
|
||||
!make_buffer(c, (VkDeviceSize)MESHFB_RECS * 8, VK_BUFFER_USAGE_TRANSFER_DST_BIT, 1, &M.stat, &M.stat_a, &map)) {
|
||||
rt_log("vk: mesh fallback: its buffers (%llu MiB, mesh_fallback_mb) could not be allocated: task and mesh draws are skipped",
|
||||
(unsigned long long)mb);
|
||||
return 0;
|
||||
}
|
||||
M.stat_map = map;
|
||||
memset(map, 0, (size_t)MESHFB_RECS * 8);
|
||||
M.task_bytes = task < range ? task : range, M.mesh_bytes = mesh < range ? mesh : range, M.list_bytes = list < range ? list : range;
|
||||
|
||||
M.push = (PFN_vkCmdPushDescriptorSetKHR)vk_dpa(c->device, "vkCmdPushDescriptorSetKHR");
|
||||
VkDescriptorSetLayoutBinding b[4];
|
||||
for (int i = 0; i < 4; i++)
|
||||
b[i] = (VkDescriptorSetLayoutBinding){.binding = (uint32_t)i, .descriptorType = VK_DESCRIPTOR_TYPE_STORAGE_BUFFER,
|
||||
.descriptorCount = 1, .stageFlags = VK_SHADER_STAGE_COMPUTE_BIT};
|
||||
VkDescriptorSetLayoutCreateInfo dci = {.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_SET_LAYOUT_CREATE_INFO,
|
||||
.flags = VK_DESCRIPTOR_SET_LAYOUT_CREATE_PUSH_DESCRIPTOR_BIT_KHR,
|
||||
.bindingCount = 4, .pBindings = b};
|
||||
VkPushConstantRange pr = {VK_SHADER_STAGE_COMPUTE_BIT, 0, 24};
|
||||
VkShaderModuleCreateInfo mi = {.sType = VK_STRUCTURE_TYPE_SHADER_MODULE_CREATE_INFO, .codeSize = sizeof meshfb_scan_spv,
|
||||
.pCode = meshfb_scan_spv};
|
||||
VkShaderModule m = VK_NULL_HANDLE;
|
||||
if (!M.push || vkCreateDescriptorSetLayout(c->device, &dci, NULL, &M.dsl) != VK_SUCCESS) goto fail;
|
||||
VkPipelineLayoutCreateInfo li = {.sType = VK_STRUCTURE_TYPE_PIPELINE_LAYOUT_CREATE_INFO, .setLayoutCount = 1,
|
||||
.pSetLayouts = &M.dsl, .pushConstantRangeCount = 1, .pPushConstantRanges = &pr};
|
||||
if (vkCreatePipelineLayout(c->device, &li, NULL, &M.layout) != VK_SUCCESS ||
|
||||
vkCreateShaderModule(c->device, &mi, NULL, &m) != VK_SUCCESS)
|
||||
goto fail;
|
||||
VkComputePipelineCreateInfo ci = {.sType = VK_STRUCTURE_TYPE_COMPUTE_PIPELINE_CREATE_INFO,
|
||||
.stage = {.sType = VK_STRUCTURE_TYPE_PIPELINE_SHADER_STAGE_CREATE_INFO,
|
||||
.stage = VK_SHADER_STAGE_COMPUTE_BIT, .module = m, .pName = "main"},
|
||||
.layout = M.layout};
|
||||
VkResult res = vkCreateComputePipelines(c->device, VK_NULL_HANDLE, 1, &ci, NULL, &M.scan);
|
||||
vkDestroyShaderModule(c->device, m, NULL);
|
||||
if (res != VK_SUCCESS) goto fail;
|
||||
rt_log("vk: mesh fallback: %llu MiB for task payloads, %llu MiB for mesh output (mesh_fallback_mb = %llu)",
|
||||
(unsigned long long)(M.task_bytes >> 20), (unsigned long long)(M.mesh_bytes >> 20), (unsigned long long)mb);
|
||||
return M.ok = 1;
|
||||
fail:
|
||||
rt_log("vk: mesh fallback: the scan pipeline could not be created: task and mesh draws are skipped");
|
||||
return 0;
|
||||
}
|
||||
|
||||
VkDescriptorBufferInfo vk_meshfb_buffer(int cb_slot) {
|
||||
if (cb_slot == MESHFB_CB_TASK) return (VkDescriptorBufferInfo){M.task, 0, M.task_bytes};
|
||||
if (cb_slot == MESHFB_CB_LIST) return (VkDescriptorBufferInfo){M.list, 0, M.list_bytes};
|
||||
return (VkDescriptorBufferInfo){M.mesh, 0, M.mesh_bytes};
|
||||
}
|
||||
|
||||
uint64_t vk_meshfb_mesh_words(void) { return M.mesh_bytes / 4; }
|
||||
|
||||
// The record about to be written again: what its last draw asked for against what it had room for.
|
||||
static void record_check(uint32_t i) {
|
||||
if (!M.noted[i]) return;
|
||||
M.noted[i] = 0;
|
||||
uint32_t want = M.stat_map[2 * i], tasks = M.stat_map[2 * i + 1];
|
||||
if (want <= M.cap_slots[i] && tasks <= M.cap_tasks[i]) return;
|
||||
uint64_t need = (uint64_t)want * M.slot_words[i] * 4;
|
||||
if (need > M.worst_bytes) M.worst_bytes = need;
|
||||
uint64_t n = ++M.short_draws;
|
||||
if (n <= 4 || !(n & (n - 1))) {
|
||||
char from[96] = "";
|
||||
if (tasks) snprintf(from, sizeof from, " from %u task workgroups (room for %u)", tasks, M.cap_tasks[i]);
|
||||
rt_log("vk: mesh fallback: a draw's programs emitted %u mesh workgroups of %u bytes%s, and the mesh buffer holds %u: "
|
||||
"what did not fit was not drawn (%llu such draws of %llu; mesh_fallback_mb of %llu or more would hold the "
|
||||
"largest so far)", want, M.slot_words[i] * 4, from, M.cap_slots[i], (unsigned long long)n,
|
||||
(unsigned long long)M.draws, (unsigned long long)((M.worst_bytes * 8 / 7 >> 20) + 1));
|
||||
}
|
||||
}
|
||||
|
||||
VkDeviceSize vk_meshfb_record(VkCommandBuffer cb, VkBuffer src, VkDeviceSize src_off, uint32_t count, int has_task,
|
||||
uint32_t verts, uint32_t slot_words, VkBuffer *rec, int *cmds) {
|
||||
uint32_t i = M.next++ % MESHFB_RECS;
|
||||
VkDeviceSize at = (VkDeviceSize)i * REC_BYTES;
|
||||
record_check(i);
|
||||
M.draws++;
|
||||
M.slot_words[i] = slot_words, M.cap_slots[i] = (uint32_t)(M.mesh_bytes / 4 / (slot_words ? slot_words : 1));
|
||||
M.cap_tasks[i] = 0xffffffffu; // vk_meshfb_scan knows the payload size
|
||||
if (has_task && M.cap_slots[i] > M.list_bytes / 4) M.cap_slots[i] = (uint32_t)(M.list_bytes / 4);
|
||||
*rec = M.rec;
|
||||
int n = 0;
|
||||
vkCmdFillBuffer(cb, M.rec, at, REC_BYTES, 1), n++;
|
||||
// where the workgroup count goes: the task dispatch, or the mesh dispatch, the draw's instances and the report
|
||||
VkDeviceSize to[3] = {has_task ? at : at + 16, at + 36, at + 48};
|
||||
int nto = has_task ? 1 : 3;
|
||||
if (!has_task) {
|
||||
vkCmdFillBuffer(cb, M.rec, at + 32, 4, verts), n++;
|
||||
vkCmdFillBuffer(cb, M.rec, at + 40, 8, 0), n++;
|
||||
vkCmdFillBuffer(cb, M.rec, at + 52, 4, 0), n++;
|
||||
}
|
||||
if (src) {
|
||||
VkBufferCopy cp[3];
|
||||
for (int k = 0; k < nto; k++) cp[k] = (VkBufferCopy){src_off, to[k], 4};
|
||||
vkCmdCopyBuffer(cb, src, M.rec, (uint32_t)nto, cp), n++;
|
||||
} else {
|
||||
for (int k = 0; k < nto; k++) vkCmdFillBuffer(cb, M.rec, to[k], 4, count), n++;
|
||||
}
|
||||
*cmds = n;
|
||||
return at;
|
||||
}
|
||||
|
||||
void vk_meshfb_scan(VkCommandBuffer cb, VkDeviceSize at, uint32_t task_words, uint32_t fan, uint32_t slot_words, uint32_t verts) {
|
||||
uint32_t i = (uint32_t)(at / REC_BYTES);
|
||||
M.cap_tasks[i] = (uint32_t)(M.task_bytes / 4 / (task_words ? task_words : 1));
|
||||
VkDescriptorBufferInfo bi[4] = {{M.rec, 0, VK_WHOLE_SIZE}, {M.task, 0, M.task_bytes}, {M.list, 0, M.list_bytes},
|
||||
{M.mesh, 0, M.mesh_bytes}};
|
||||
VkWriteDescriptorSet w[4];
|
||||
for (int k = 0; k < 4; k++)
|
||||
w[k] = (VkWriteDescriptorSet){.sType = VK_STRUCTURE_TYPE_WRITE_DESCRIPTOR_SET, .dstBinding = (uint32_t)k,
|
||||
.descriptorCount = 1, .descriptorType = VK_DESCRIPTOR_TYPE_STORAGE_BUFFER,
|
||||
.pBufferInfo = &bi[k]};
|
||||
uint32_t pc[6] = {(uint32_t)(at / 4), task_words, fan, slot_words, verts, vk_ctx()->props.limits.maxComputeWorkGroupCount[0]};
|
||||
vkCmdBindPipeline(cb, VK_PIPELINE_BIND_POINT_COMPUTE, M.scan);
|
||||
rec_push(M.push, cb, VK_PIPELINE_BIND_POINT_COMPUTE, M.layout, 0, 4, w);
|
||||
vkCmdPushConstants(cb, M.layout, VK_SHADER_STAGE_COMPUTE_BIT, 0, sizeof pc, pc);
|
||||
vkCmdDispatch(cb, 1, 1, 1);
|
||||
}
|
||||
|
||||
void vk_meshfb_note(VkCommandBuffer cb, VkDeviceSize at) {
|
||||
uint32_t i = (uint32_t)(at / REC_BYTES);
|
||||
VkBufferCopy cp = {at + 48, (VkDeviceSize)i * 8, 8};
|
||||
vkCmdCopyBuffer(cb, M.rec, M.stat, 1, &cp);
|
||||
M.noted[i] = 1;
|
||||
}
|
||||
|
||||
uint64_t vk_meshfb_short(void) {
|
||||
if (!M.ok) return 0;
|
||||
for (uint32_t i = 0; i < MESHFB_RECS; i++) record_check(i);
|
||||
return M.short_draws;
|
||||
}
|
||||
|
||||
void vk_meshfb_shutdown(void) {
|
||||
VkCtx *c = vk_ctx();
|
||||
if (!c || !M.tried) return;
|
||||
if (vk_meshfb_short())
|
||||
rt_log("vk: mesh fallback: %llu of %llu draws emitted more than the buffers hold (mesh_fallback_mb of %llu or more would "
|
||||
"hold the largest)", (unsigned long long)M.short_draws, (unsigned long long)M.draws,
|
||||
(unsigned long long)((M.worst_bytes * 8 / 7 >> 20) + 1));
|
||||
if (M.scan) vkDestroyPipeline(c->device, M.scan, NULL);
|
||||
if (M.layout) vkDestroyPipelineLayout(c->device, M.layout, NULL);
|
||||
if (M.dsl) vkDestroyDescriptorSetLayout(c->device, M.dsl, NULL);
|
||||
if (M.task) vmaDestroyBuffer(c->vma, M.task, M.task_a);
|
||||
if (M.mesh) vmaDestroyBuffer(c->vma, M.mesh, M.mesh_a);
|
||||
if (M.list) vmaDestroyBuffer(c->vma, M.list, M.list_a);
|
||||
if (M.rec) vmaDestroyBuffer(c->vma, M.rec, M.rec_a);
|
||||
if (M.stat) vmaDestroyBuffer(c->vma, M.stat, M.stat_a);
|
||||
memset(&M, 0, sizeof M);
|
||||
}
|
||||
29
runtime/vk_mesh_fb.h
Normal file
29
runtime/vk_mesh_fb.h
Normal file
@@ -0,0 +1,29 @@
|
||||
// The mesh fallback's buffers and its records (vk_mesh_fb.c); the modules are mesh_fb.c's, the draws
|
||||
// vk_exec_draw.c's ("mesh fallback").
|
||||
#pragma once
|
||||
#include "vk.h"
|
||||
|
||||
// Creates the buffers and the scan pipeline on first use. 0 when that failed (logged once): the draws are skipped.
|
||||
int vk_meshfb_init(void);
|
||||
// The buffer a module's extra storage binding reads or writes (mesh_fb.h MESHFB_CB_*), whole or as much of it as one
|
||||
// storage binding may span.
|
||||
VkDescriptorBufferInfo vk_meshfb_buffer(int cb_slot);
|
||||
uint64_t vk_meshfb_mesh_words(void); // the mesh buffer's size as a binding sees it, in words
|
||||
|
||||
// A draw's record, 64 bytes of the record buffer (*rec) at the offset returned: the task dispatch at +0, the mesh
|
||||
// dispatch at +16, the draw (VkDrawIndirectCommand) at +32. Records the fills and copies that start it, outside a
|
||||
// pass: the workgroup count from 4 bytes at src_off of src, or `count` when src is VK_NULL_HANDLE. Without a task
|
||||
// stage that count is the mesh dispatch and the draw's instance count; with one it is the task dispatch, and
|
||||
// vk_meshfb_scan writes the rest. slot_words: of the draw's mesh program, for the report of a draw the mesh buffer
|
||||
// was too small for. Returns the number of commands recorded in *cmds.
|
||||
VkDeviceSize vk_meshfb_record(VkCommandBuffer cb, VkBuffer src, VkDeviceSize src_off, uint32_t count, int has_task,
|
||||
uint32_t verts, uint32_t slot_words, VkBuffer *rec, int *cmds);
|
||||
// The pass between the task and the mesh dispatch (shaders/meshfb_scan.comp): binds its own compute pipeline and set.
|
||||
void vk_meshfb_scan(VkCommandBuffer cb, VkDeviceSize at, uint32_t task_words, uint32_t fan, uint32_t slot_words, uint32_t verts);
|
||||
// After the record is complete (the copies or the scan, and a barrier): keeps what the draw asked for where the host
|
||||
// reads it later. One copy command.
|
||||
void vk_meshfb_note(VkCommandBuffer cb, VkDeviceSize at);
|
||||
// Draws so far whose programs emitted more than the buffers hold, the records not looked at yet included: call it
|
||||
// with the device idle (a probe, the shutdown).
|
||||
uint64_t vk_meshfb_short(void);
|
||||
void vk_meshfb_shutdown(void); // the device must be idle
|
||||
@@ -3508,6 +3508,7 @@ static void keys_note(const VkPipeState *s) {
|
||||
static const uint8_t none[32];
|
||||
if (s->fs && !memcmp(gfx_shader_input_id(s->fs), none, 32)) return; // the runtime's own shader (gfx_shader_own)
|
||||
if (s->ts && !memcmp(gfx_shader_input_id(s->ts), none, 32)) return;
|
||||
if (!memcmp(gfx_shader_input_id(s->vs), none, 32)) return; // the mesh fallback's vertex stage (vk_exec_draw.c mfb_vs_get)
|
||||
KeyRec r;
|
||||
rec_fill(&r, s);
|
||||
if (!seen_add(rec_hash(&r))) return;
|
||||
|
||||
@@ -49,7 +49,7 @@ enum {
|
||||
R_SetRasterizerDiscardEnable, R_SetScissorWithCount, R_SetStencilCompareMask, R_SetStencilOp, R_SetStencilReference,
|
||||
R_SetStencilTestEnable, R_SetStencilWriteMask, R_SetViewportWithCount, R_WriteTimestamp, R_WriteTimestamp2,
|
||||
R_PushTpl, R_Push, R_VertexInput, R_Bool, R_PolygonMode, R_BlendEnable, R_BlendEquation, R_WriteMask, R_EndCb,
|
||||
R_PassMark, R_DrawMesh, R_DrawMeshInd, R_ShadingRate,
|
||||
R_PassMark, R_DrawMesh, R_DrawMeshInd, R_ShadingRate, R_DrawIndirect,
|
||||
};
|
||||
|
||||
static uint64_t now_ns(void) { return plat_now_ns(); }
|
||||
@@ -347,6 +347,9 @@ FIXED(DrawIndexed, (VkCommandBuffer cb, uint32_t n, uint32_t i, uint32_t f, int3
|
||||
#define PACK_DrawIndexedIndirect *a = (A_DrawIndexedIndirect){b, o, n, s}
|
||||
FIXED(DrawIndexedIndirect, (VkCommandBuffer cb, VkBuffer b, VkDeviceSize o, uint32_t n, uint32_t s), (cb, b, o, n, s),
|
||||
VkBuffer b; VkDeviceSize o; uint32_t n, s)
|
||||
#define PACK_DrawIndirect *a = (A_DrawIndirect){b, o, n, s}
|
||||
FIXED(DrawIndirect, (VkCommandBuffer cb, VkBuffer b, VkDeviceSize o, uint32_t n, uint32_t s), (cb, b, o, n, s),
|
||||
VkBuffer b; VkDeviceSize o; uint32_t n, s)
|
||||
#define PACK_EndQuery *a = (A_EndQuery){p, q}
|
||||
FIXED(EndQuery, (VkCommandBuffer cb, VkQueryPool p, uint32_t q), (cb, p, q), VkQueryPool p; uint32_t q)
|
||||
#define PACK_EndRendering (void)a
|
||||
@@ -705,6 +708,7 @@ static void exec_packet(uint32_t op, void *p) {
|
||||
F(Draw, a->n, a->i, a->f, a->fi)
|
||||
F(DrawIndexed, a->n, a->i, a->f, a->v, a->fi)
|
||||
F(DrawIndexedIndirect, a->b, a->o, a->n, a->s)
|
||||
F(DrawIndirect, a->b, a->o, a->n, a->s)
|
||||
F(EndQuery, a->p, a->q)
|
||||
F(FillBuffer, a->b, a->o, a->n, a->v)
|
||||
F(ResetQueryPool, a->p, a->f, a->n)
|
||||
|
||||
@@ -80,6 +80,7 @@ void rec_pass_mark(VkCommandBuffer cb, uint32_t id, int end);
|
||||
X(Draw, (VkCommandBuffer cb, uint32_t n, uint32_t i, uint32_t f, uint32_t fi), (cb, n, i, f, fi)) \
|
||||
X(DrawIndexed, (VkCommandBuffer cb, uint32_t n, uint32_t i, uint32_t f, int32_t v, uint32_t fi), (cb, n, i, f, v, fi)) \
|
||||
X(DrawIndexedIndirect, (VkCommandBuffer cb, VkBuffer b, VkDeviceSize o, uint32_t n, uint32_t s), (cb, b, o, n, s)) \
|
||||
X(DrawIndirect, (VkCommandBuffer cb, VkBuffer b, VkDeviceSize o, uint32_t n, uint32_t s), (cb, b, o, n, s)) \
|
||||
X(EndQuery, (VkCommandBuffer cb, VkQueryPool p, uint32_t q), (cb, p, q)) \
|
||||
X(EndRendering, (VkCommandBuffer cb), (cb)) \
|
||||
X(FillBuffer, (VkCommandBuffer cb, VkBuffer b, VkDeviceSize o, VkDeviceSize n, uint32_t v), (cb, b, o, n, v)) \
|
||||
@@ -137,6 +138,7 @@ REC_CMDS(REC_DECL)
|
||||
#define vkCmdDraw rc_vkCmdDraw
|
||||
#define vkCmdDrawIndexed rc_vkCmdDrawIndexed
|
||||
#define vkCmdDrawIndexedIndirect rc_vkCmdDrawIndexedIndirect
|
||||
#define vkCmdDrawIndirect rc_vkCmdDrawIndirect
|
||||
#define vkCmdEndQuery rc_vkCmdEndQuery
|
||||
#define vkCmdEndRendering rc_vkCmdEndRendering
|
||||
#define vkCmdFillBuffer rc_vkCmdFillBuffer
|
||||
|
||||
@@ -5,6 +5,7 @@ The repository holds the shaders as source only. Each becomes runtime/<name>_spv
|
||||
missing and the stale ones before it compiles the runtime. Three kinds of source:
|
||||
|
||||
runtime/shaders/<name>.comp, .vert, .frag GLSL, compiled with glslc (one name per shader) and checked with spirv-val
|
||||
runtime/shaders/<name>.mesh, .task the same for the mesh fallback probe's mesh and task shaders (SPIR-V 1.5)
|
||||
runtime/shaders/<name>.spvasm SPIR-V assembly, assembled with spirv-as and checked with spirv-val
|
||||
tools/make_dlaa_mask.py, make_mvobj_fs.py shaders assembled in Python (ASSEMBLED); they need no other tool
|
||||
|
||||
@@ -34,7 +35,8 @@ TOOLS = Path(__file__).resolve().parent
|
||||
ROOT = TOOLS.parent
|
||||
BIN = Path(r'C:\msys64\ucrt64\bin') # Windows; elsewhere, and without MSYS2 there, the tools come from PATH
|
||||
TARGET = 'vulkan1.1'
|
||||
STAGES = {'.comp': 'compute', '.vert': 'vertex', '.frag': 'fragment'}
|
||||
STAGES = {'.comp': 'compute', '.vert': 'vertex', '.frag': 'fragment', '.mesh': 'mesh', '.task': 'task'}
|
||||
MESH_TARGET = 'vulkan1.2' # mesh and task shaders need SPIR-V 1.4 or later
|
||||
ASM = '.spvasm'
|
||||
ASM_TARGET = 'vulkan1.3'
|
||||
# header name -> the script beside this one that assembles it; the script's HEADER is the header's text
|
||||
@@ -138,7 +140,7 @@ def words_of(src):
|
||||
else:
|
||||
# "// build_shaders: keep functions" in a source: no optimisation, so its functions stay functions (the HDR
|
||||
# tonemap, which runtime/vk_hdr.c splices into the game's shader by function)
|
||||
target = TARGET
|
||||
target = MESH_TARGET if src.suffix in ('.mesh', '.task') else TARGET
|
||||
opt = [] if '// build_shaders: keep functions' in src.read_text(encoding='utf-8') else ['-O']
|
||||
subprocess.run([tool('glslc'), *opt, f'--target-env={target}', f'-fshader-stage={STAGES[src.suffix]}',
|
||||
'-o', str(spv), str(src)], check=True)
|
||||
|
||||
Reference in New Issue
Block a user