mirror of
https://github.com/tangerineturtle1/NX2navi.git
synced 2026-11-20 14:39:58 -04:00
Wait 8 ms once for a pipeline met for the first time, and carry the compute variants in the pipeline list.
A draw that met a new pipeline waited 3 ms for its fast link, in every frame, which is just under what the link takes with the stage libraries in the driver's cache: in a first-visit session on a new install 66 of 87 such draws gave up at the end of the wait and their frame was held back. The wait is now 8 ms for each pipeline, counted from its first miss and shared by all its draws, and 16 ms for a frame, so a pipeline the driver has to compile costs one frame the wait and the frames after it nothing. The pipeline list tool also takes the compute variants a game folder dispatched, which no pipeline record names and which were translated in the middle of play, and reads a list made before as one of its inputs. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
@@ -161,10 +161,14 @@ of them.
|
|||||||
What this step leaves for play, and the shared pipeline list. The step translates every shader's default variant
|
What this step leaves for play, and the shared pipeline list. The step translates every shader's default variant
|
||||||
and the shader list's variants and builds each one's stage library or compute pipeline. A whole graphics pipeline
|
and the shader list's variants and builds each one's stage library or compute pipeline. A whole graphics pipeline
|
||||||
is built ahead at the start of a game folder that has drawn it before (`pipeline-keys.bin`) or whose release
|
is built ahead at the start of a game folder that has drawn it before (`pipeline-keys.bin`) or whose release
|
||||||
listed it (below); a pipeline met for the first time is linked from its stage libraries in 2 to 6 ms, except a mesh
|
listed it (below); a pipeline met for the first time is linked from its stage libraries in 2 to 8 ms, which its
|
||||||
|
first draw waits for (`pipe_link_draw_ms`, 8 ms from the pipeline's first miss; `runtime/vk_exec_draw.c`
|
||||||
|
`link_budget`), except a mesh
|
||||||
pipeline, which has no libraries and compiles whole (0.2 to 1.1 s, its draws left out of the picture meanwhile),
|
pipeline, which has no libraries and compiles whole (0.2 to 1.1 s, its draws left out of the picture meanwhile),
|
||||||
and many of them on a first visit add up to hitches. `tools/pipeline_list.py merge` folds the `pipeline-keys.bin`
|
and many of them on a first visit add up to hitches. `tools/pipeline_list.py merge` folds the `pipeline-keys.bin`
|
||||||
and `shader-answers.bin` of game folders that were played into a pipeline list; `prepare_shaders.py` reads it from
|
and `shader-answers.bin` of game folders that were played, and lists made before, into a pipeline list, with the
|
||||||
|
compute variants those folders dispatched (no pipeline record names a compute shader, and a variant the shader list
|
||||||
|
lacks is translated on the frame that first dispatches it); `prepare_shaders.py` reads it from
|
||||||
`tools\data\pipeline_list.bin` (`data\pipeline_list.bin` in a release folder, or `--pipeline-list`), adds the
|
`tools\data\pipeline_list.bin` (`data\pipeline_list.bin` in a release folder, or `--pipeline-list`), adds the
|
||||||
shader variants it names to the seed and writes its records next to the game as `seed-pipeline-keys.bin`, and
|
shader variants it names to the seed and writes its records next to the game as `seed-pipeline-keys.bin`, and
|
||||||
`--prepare-shaders` builds every one of them into the pipeline cache and the driver's (`runtime/vk_pipeline.c`
|
`--prepare-shaders` builds every one of them into the pipeline cache and the driver's (`runtime/vk_pipeline.c`
|
||||||
|
|||||||
@@ -18,12 +18,22 @@ for the launcher; a GPU with Vulkan 1.3 and `VK_EXT_external_memory_host` (curre
|
|||||||
## Faster
|
## Faster
|
||||||
|
|
||||||
- **A new install no longer builds its pipelines while you play.** A release now carries a list of the graphics
|
- **A new install no longer builds its pipelines while you play.** A release now carries a list of the graphics
|
||||||
pipelines the game is known to draw, 8,715 of them in this one. The install builds all of them while it prepares the
|
pipelines the game is known to draw, 8,849 of them in this one. The install builds all of them while it prepares the
|
||||||
shaders, and every start builds them before play, as it already did for the pipelines your own install had drawn.
|
shaders, and every start builds them before play, as it already did for the pipelines your own install had drawn.
|
||||||
Before, a new install met each pipeline in play first: in Hyrule Field one ten-second stretch of a first visit had
|
Before, a new install met each pipeline in play first: in Hyrule Field one ten-second stretch of a first visit had
|
||||||
1,769 draws wait for their pipelines, 461 ms in all, the longest frame 88 ms. In a first walk through Kakariko
|
1,769 draws wait for their pipelines, 461 ms in all, the longest frame 88 ms. In a first walk through Kakariko
|
||||||
Village on fresh caches no draw waited for a pipeline with the list, where 634 pipelines were missed without it, and
|
Village on fresh caches no draw waited for a pipeline with the list, where 634 pipelines were missed without it, and
|
||||||
the first minute had 10 frames over 25 ms instead of 137. `pipe_seed = 0` in nx2navi.ini leaves the list out.
|
the first minute had 10 frames over 25 ms instead of 137. `pipe_seed = 0` in nx2navi.ini leaves the list out.
|
||||||
|
- **A pipeline the list does not have costs its frame less.** Such a pipeline is linked from the shader libraries the
|
||||||
|
install prepared, and its first draw waits for that. The wait was 3 ms for each draw, just under what the link
|
||||||
|
takes: in a first-visit session on a new install 66 of 87 such draws gave up at the end of it, and their frame was
|
||||||
|
then not shown, so the picture before it stayed for one frame more. Of the 67 pipelines that session met outside
|
||||||
|
the list, 61 were ready within 8 ms. The wait is now 8 ms, counted once for each pipeline from the first draw that
|
||||||
|
needed it, so a pipeline the driver has to compile costs one frame that much and the frames after it nothing
|
||||||
|
(`pipe_link_draw_ms`, and `pipe_link_ms` for all the pipelines of one frame, now 16).
|
||||||
|
- **Compute shader variants are prepared too.** The list now carries the 629 compute shader variants met in play. No
|
||||||
|
pipeline record names a compute shader, so a variant the shader list lacked was translated on the frame that first
|
||||||
|
used it, 30 and 38 ms in the two cases of that session.
|
||||||
- **The draw replay runs on several CPU cores by default** (`replay_workers = 3`; contributed by epicmeme546): a
|
- **The draw replay runs on several CPU cores by default** (`replay_workers = 3`; contributed by epicmeme546): a
|
||||||
frame's large groups of draws are recorded on three threads beside the one that submits them. On an RX 7900 XT in
|
frame's large groups of draws are recorded on three threads beside the one that submits them. On an RX 7900 XT in
|
||||||
Hyrule Field it gave 51.7 instead of 43.6 fps standing. `replay_workers = 0` records on one thread as before; try
|
Hyrule Field it gave 51.7 instead of 43.6 fps standing. `replay_workers = 0` records on one thread as before; try
|
||||||
|
|||||||
@@ -126,7 +126,11 @@
|
|||||||
# pipe_skip_steady = 1
|
# pipe_skip_steady = 1
|
||||||
# How many milliseconds the draws of one frame may wait, in all, for a missing pipeline to be linked from the shader
|
# How many milliseconds the draws of one frame may wait, in all, for a missing pipeline to be linked from the shader
|
||||||
# libraries in the graphics driver's cache before they are dropped (0 to 100; 0: dropped at once, as before).
|
# libraries in the graphics driver's cache before they are dropped (0 to 100; 0: dropped at once, as before).
|
||||||
# pipe_link_ms = 6
|
# pipe_link_ms = 16
|
||||||
|
# How many milliseconds one missing pipeline is waited for, counted from the first draw that needed it (1 to 50).
|
||||||
|
# With its shader libraries in the driver's cache a pipeline takes 3 to 8 ms; one that takes longer is being
|
||||||
|
# compiled, and its draws are left out of the frames until it is ready.
|
||||||
|
# pipe_link_draw_ms = 8
|
||||||
# When a draw finds its pipeline missing, the rest of the commands the game submitted with it are read ahead and the
|
# When a draw finds its pipeline missing, the rest of the commands the game submitted with it are read ahead and the
|
||||||
# shader libraries of every other pipeline they will need are built at once on background threads, so the later draws
|
# shader libraries of every other pipeline they will need are built at once on background threads, so the later draws
|
||||||
# find theirs ready instead of waiting one after another or being left out of a frame. 0: off, as before.
|
# find theirs ready instead of waiting one after another or being left out of a frame. 0: off, as before.
|
||||||
|
|||||||
@@ -261,8 +261,8 @@ static const CfgMeta metas[] = {
|
|||||||
{"CACHE_DIE", "1", "0|1|ab", RESTART}, {"ONE_PER_CORE", "0", "0|1|ab", RESTART}, // perf.c cpu_placement
|
{"CACHE_DIE", "1", "0|1|ab", RESTART}, {"ONE_PER_CORE", "0", "0|1|ab", RESTART}, // perf.c cpu_placement
|
||||||
// shaders and pipelines
|
// shaders and pipelines
|
||||||
{"PIPE_WAIT", "default", "default|draw", RESTART}, {"PIPE_SKIP", "any", "strict|any|fullscreen", RESTART},
|
{"PIPE_WAIT", "default", "default|draw", RESTART}, {"PIPE_SKIP", "any", "strict|any|fullscreen", RESTART},
|
||||||
{"PIPE_LINK_MS", "6", "0..100", RESTART}, // vk_exec_draw.c link_budget
|
{"PIPE_LINK_MS", "16", "0..100", RESTART}, // vk_exec_draw.c link_budget
|
||||||
{"PIPE_LINK_DRAW_MS", "3", "1..50", RESTART}, // vk_exec_draw.c link_draw_ms
|
{"PIPE_LINK_DRAW_MS", "8", "1..50", RESTART}, // vk_exec_draw.c link_draw_ms
|
||||||
{"PIPE_FRAME_WAIT_MS", "50", "0..500", RESTART}, // vk_exec_draw.c frame_wait
|
{"PIPE_FRAME_WAIT_MS", "50", "0..500", RESTART}, // vk_exec_draw.c frame_wait
|
||||||
{"PIPE_SKIP_STEADY", "1", "0|1", RESTART}, // vk_exec.c steady_on
|
{"PIPE_SKIP_STEADY", "1", "0|1", RESTART}, // vk_exec.c steady_on
|
||||||
{"PREPARE_PIPELINES", "all", "0|mesh|all", RESTART}, // vk_pipeline.c seed_pipelines
|
{"PREPARE_PIPELINES", "all", "0|mesh|all", RESTART}, // vk_pipeline.c seed_pipelines
|
||||||
|
|||||||
@@ -3829,7 +3829,7 @@ static int pipe_skip_mode(void) {
|
|||||||
return mode;
|
return mode;
|
||||||
}
|
}
|
||||||
|
|
||||||
// NX2NAVI_PIPE_LINK_MS (default 6; 0 = off): how long the draws of one presented frame may wait, in all, for the fast
|
// NX2NAVI_PIPE_LINK_MS (default 16; 0 = off): how long the draws of one presented frame may wait, in all, for the fast
|
||||||
// link of a pipeline that is missing, LINK_DRAW_MS at most each. A draw into a per-frame pass used to be skipped at
|
// link of a pipeline that is missing, LINK_DRAW_MS at most each. A draw into a per-frame pass used to be skipped at
|
||||||
// once and for as long as its pipeline built, also when its stage libraries were in the driver's cache and the link
|
// once and for as long as its pipeline built, also when its stage libraries were in the driver's cache and the link
|
||||||
// took under a millisecond: geometry seen for the first time (ground, a wall, a character) was missing for a frame or
|
// took under a millisecond: geometry seen for the first time (ground, a wall, a character) was missing for a frame or
|
||||||
@@ -3837,6 +3837,13 @@ static int pipe_skip_mode(void) {
|
|||||||
// cold it runs out after a few draws and the rest are skipped as before, so a burst of cold pipelines costs a frame
|
// cold it runs out after a few draws and the rest are skipped as before, so a burst of cold pipelines costs a frame
|
||||||
// this much and no more. Measured on an RX 7900 XT, first walk through a village with no pipeline recorded: 2,349
|
// this much and no more. Measured on an RX 7900 XT, first walk through a village with no pipeline recorded: 2,349
|
||||||
// draws skipped without the budget, with the libraries warm in both runs.
|
// draws skipped without the budget, with the libraries warm in both runs.
|
||||||
|
// NX2NAVI_PIPE_LINK_DRAW_MS (default 8): the wait for one pipeline, counted from its first miss and shared by all its
|
||||||
|
// draws (prepare_draw_ex), so a pipeline whose libraries the driver has to compile costs its first frame this much and
|
||||||
|
// the frames after it nothing. It was 3 ms for each draw, in every frame, 6 ms a frame: just under what a pipeline
|
||||||
|
// takes with its two shader libraries in the driver's cache. On a new install's first visits (RTX 5090, the
|
||||||
|
// libraries prepared, 67 pipelines the install's list did not hold) 44 were usable under 4 ms after their miss, 17
|
||||||
|
// after 4 to 8 ms and 6 later, and of 87 draws that waited for a fast link 66 were skipped at the end of their 3 ms:
|
||||||
|
// each of those frames paid the wait and was then not shown, the previous picture staying for one frame more.
|
||||||
// NX2NAVI_PIPE_FRAME_WAIT_MS (default 50; 0 to 500): the wait pool of one presented frame. Its full-screen draws and
|
// NX2NAVI_PIPE_FRAME_WAIT_MS (default 50; 0 to 500): the wait pool of one presented frame. Its full-screen draws and
|
||||||
// its one-shot draws (NX2NAVI_ONESHOT_BUDGET) wait for their shaders and pipelines from it, each for what the ones
|
// its one-shot draws (NX2NAVI_ONESHOT_BUDGET) wait for their shaders and pipelines from it, each for what the ones
|
||||||
// before left, and are skipped once it is spent, so the pipelines of one frame hold the replay thread this long in
|
// before left, and are skipped once it is spent, so the pipelines of one frame hold the replay thread this long in
|
||||||
@@ -4239,9 +4246,16 @@ static int prepare_draw_ex(VkExec *ex, int prim, int32_t count, int restart, int
|
|||||||
if (linking) t0 = kern_now_ns();
|
if (linking) t0 = kern_now_ns();
|
||||||
else w0 = t0 ? t0 : wait_ms ? kern_now_ns() : 0;
|
else w0 = t0 ? t0 : wait_ms ? kern_now_ns() : 0;
|
||||||
vk_exec_ahead(ex);
|
vk_exec_ahead(ex);
|
||||||
|
// The fast link's wait is one for each pipeline, counted from its first miss (link_budget): what that left
|
||||||
|
// unbuilt is a compile, and its later draws, in this frame and the next ones, wait for nothing.
|
||||||
|
double building = wait_ms && linking ? vk_pipeline_building_ms(&s) : 0;
|
||||||
|
if (wait_ms && linking) {
|
||||||
|
uint32_t left = building >= LINK_DRAW_MS ? 0 : LINK_DRAW_MS - (uint32_t)building;
|
||||||
|
if (wait_ms > left) wait_ms = left;
|
||||||
|
}
|
||||||
// A mesh pipeline that is known to be compiling gets no fast link, so the few milliseconds a draw waits
|
// A mesh pipeline that is known to be compiling gets no fast link, so the few milliseconds a draw waits
|
||||||
// for one are lost in every frame that meets it until the compile ends, 0.2 to 1.1 s later (MESH_WAIT_MS).
|
// for one are lost in every frame that meets it until the compile ends, 0.2 to 1.1 s later (MESH_WAIT_MS).
|
||||||
if (wait_ms && linking && mesh && !fb && vk_pipeline_building_ms(&s) > MESH_WAIT_MS) wait_ms = 0;
|
if (wait_ms && linking && mesh && !fb && building > MESH_WAIT_MS) wait_ms = 0;
|
||||||
if (wait_ms) p = vk_pipeline_graphics_async(&s, &info, &pending, wait_ms);
|
if (wait_ms) p = vk_pipeline_graphics_async(&s, &info, &pending, wait_ms);
|
||||||
}
|
}
|
||||||
if (t0 && budget) {
|
if (t0 && budget) {
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
"""tools/pipeline_list.py on made-up game folders: records folded once, a key whose shader the folder's answers do not
|
"""tools/pipeline_list.py on made-up game folders: records folded once, a key whose shader the folder's answers do not
|
||||||
name left out, --mesh-only, a folder of another record layout left out, the file read back, and what
|
name left out, the compute variants taken, a list made before as an input, --mesh-only, a folder of another record
|
||||||
|
layout left out, the file read back, and what
|
||||||
tools/prepare_shaders.py takes from a list for its seed. Also tools/check_content.py refusing such files in a commit.
|
tools/prepare_shaders.py takes from a list for its seed. Also tools/check_content.py refusing such files in a commit.
|
||||||
No game files; stdlib only.
|
No game files; stdlib only.
|
||||||
|
|
||||||
@@ -18,7 +19,7 @@ import check_content # noqa: E402
|
|||||||
import pipeline_list as pl # noqa: E402
|
import pipeline_list as pl # noqa: E402
|
||||||
|
|
||||||
REC = 464 # a KeyRec of the runtime that wrote the test's files: three ids, two flags, the state
|
REC = 464 # a KeyRec of the runtime that wrote the test's files: three ids, two flags, the state
|
||||||
VS, FS, TS, MS = 0, 4, 6, 7
|
VS, FS, CS, TS, MS = 0, 4, 5, 6, 7
|
||||||
|
|
||||||
|
|
||||||
def h(seed):
|
def h(seed):
|
||||||
@@ -73,6 +74,24 @@ class PipelineList(unittest.TestCase):
|
|||||||
self.assertEqual(keys, [key('m1', 'f2', 't1'), key('m1', None)])
|
self.assertEqual(keys, [key('m1', 'f2', 't1'), key('m1', None)])
|
||||||
self.assertEqual({x[36:68] for x in answers}, {h('id ' + s) for s in ('m1', 'f2', 't1')})
|
self.assertEqual({x[36:68] for x in answers}, {h('id ' + s) for s in ('m1', 'f2', 't1')})
|
||||||
|
|
||||||
|
def test_compute_variants_and_a_list_as_input(self):
|
||||||
|
a = folder(self.root, 'a', [key('v1', 'f1')], [answer('v1', VS), answer('f1', FS), answer('c1', CS), answer('f9', FS)])
|
||||||
|
header, keys, answers = self.merge([a])
|
||||||
|
self.assertEqual({x[36:68] for x in answers}, {h('id ' + s) for s in ('v1', 'f1', 'c1')}) # f9: no key names it
|
||||||
|
self.assertIn('1 compute variants new', self.log[0])
|
||||||
|
lst = self.root / 'first.bin'
|
||||||
|
lst.write_bytes(pl.pack(header, keys, answers))
|
||||||
|
b = folder(self.root, 'b', [key('v1', 'f1'), key('v2', 'f1')],
|
||||||
|
[answer('v2', VS), answer('f1', FS), answer('c1', CS), answer('c2', CS)]) # v1's answers: the list's
|
||||||
|
header, keys, answers = self.merge([lst, b])
|
||||||
|
self.assertEqual(keys, [key('v1', 'f1'), key('v2', 'f1')])
|
||||||
|
self.assertEqual({x[36:68] for x in answers}, {h('id ' + s) for s in ('v1', 'f1', 'c1', 'v2', 'c2')})
|
||||||
|
self.assertIn('1 key records, 1 new to the list', self.log[1])
|
||||||
|
self.assertIn('2 key records, 1 new to the list, 0 without', self.log[2])
|
||||||
|
self.assertIn('1 compute variants new', self.log[2])
|
||||||
|
_, _, answers = self.merge([a], mesh_only=True)
|
||||||
|
self.assertEqual(answers, [])
|
||||||
|
|
||||||
def test_other_layout_left_out(self):
|
def test_other_layout_left_out(self):
|
||||||
a = folder(self.root, 'a', [key('v1', 'f1')], [answer('v1', VS), answer('f1', FS)])
|
a = folder(self.root, 'a', [key('v1', 'f1')], [answer('v1', VS), answer('f1', FS)])
|
||||||
b = folder(self.root, 'b', [key('v2', 'f1')], [answer('v2', VS), answer('f1', FS)], version=5)
|
b = folder(self.root, 'b', [key('v2', 'f1')], [answer('v2', VS), answer('f1', FS)], version=5)
|
||||||
|
|||||||
@@ -7,7 +7,11 @@ folder builds its recorded pipelines ahead of play; another install, or the same
|
|||||||
meets each of them in play first. A mesh pipeline met that way has no stage libraries to be linked from, so its draws
|
meets each of them in play first. A mesh pipeline met that way has no stage libraries to be linked from, so its draws
|
||||||
are left out of the picture for its whole compile, 0.2 to 1.1 s.
|
are left out of the picture for its whole compile, 0.2 to 1.1 s.
|
||||||
|
|
||||||
This tool folds the two files of any number of game folders into one list file. tools/prepare_shaders.py reads the
|
This tool folds the two files of any number of game folders into one list file; a list file made before can stand
|
||||||
|
in for a folder, so a list grows by the folders played since. Beside the shader variants the pipelines name, the list
|
||||||
|
takes the compute variants each folder dispatched: no pipeline record names a compute shader, and a variant with
|
||||||
|
other answers than the shader list's default one was translated in the middle of play, 30 to 40 ms on the frame that
|
||||||
|
first dispatched it. tools/prepare_shaders.py reads the
|
||||||
list beside the shared shader list: the shader variants it names go into the seed, so the preparation translates them,
|
list beside the shared shader list: the shader variants it names go into the seed, so the preparation translates them,
|
||||||
and its pipeline records go next to the game as seed-pipeline-keys.bin, which every start of the game counts as
|
and its pipeline records go next to the game as seed-pipeline-keys.bin, which every start of the game counts as
|
||||||
recorded pipelines beside its own (runtime/vk_pipeline.c keys_open_seed), so a new install builds them before play,
|
recorded pipelines beside its own (runtime/vk_pipeline.c keys_open_seed), so a new install builds them before play,
|
||||||
@@ -27,11 +31,11 @@ changes either, and the runtime ignores one that does not match.
|
|||||||
The list is derived from game data: it stays out of the repository (tools/check_content.py refuses it) and travels
|
The list is derived from game data: it stays out of the repository (tools/check_content.py refuses it) and travels
|
||||||
with the installer archive as data/pipeline_list.bin, as the shader list does.
|
with the installer archive as data/pipeline_list.bin, as the shader list does.
|
||||||
|
|
||||||
python tools/pipeline_list.py merge OUT.bin GAME_DIR [GAME_DIR ...] [--mesh-only]
|
python tools/pipeline_list.py merge OUT.bin GAME_DIR_OR_LIST [GAME_DIR_OR_LIST ...] [--mesh-only]
|
||||||
python tools/pipeline_list.py info FILE_OR_GAME_DIR [...]
|
python tools/pipeline_list.py info FILE_OR_GAME_DIR [...]
|
||||||
|
|
||||||
File: "NX2NPLST", version u32, answers bytes u32, keys bytes u32, then a shader-answers.bin image (header and the
|
File: "NX2NPLST", version u32, answers bytes u32, keys bytes u32, then a shader-answers.bin image (header, the
|
||||||
records the keys name, input id kept) and a pipeline-keys.bin image (header and records).
|
records the keys name and the compute variants', input id kept) and a pipeline-keys.bin image (header and records).
|
||||||
"""
|
"""
|
||||||
import argparse
|
import argparse
|
||||||
import struct
|
import struct
|
||||||
@@ -44,7 +48,7 @@ ANS_MAGIC, ANS_REC = b'NX2NSANS', 4 + 32 + 32 + 147 + 1 # gfx_shader.c: stage,
|
|||||||
KEYS_MAGIC = b'NX2NPKEY'
|
KEYS_MAGIC = b'NX2NPKEY'
|
||||||
ID = 32
|
ID = 32
|
||||||
STAGE_NAMES = {0: 'vertex', 4: 'fragment', 5: 'compute', 6: 'task', 7: 'mesh'} # runtime/gfx.h GFX_STAGE_*
|
STAGE_NAMES = {0: 'vertex', 4: 'fragment', 5: 'compute', 6: 'task', 7: 'mesh'} # runtime/gfx.h GFX_STAGE_*
|
||||||
STAGE_MESH = 7
|
STAGE_COMPUTE, STAGE_MESH = 5, 7
|
||||||
|
|
||||||
|
|
||||||
def read_answers(data):
|
def read_answers(data):
|
||||||
@@ -82,15 +86,19 @@ def key_ids(rec):
|
|||||||
|
|
||||||
|
|
||||||
def read_dir(d):
|
def read_dir(d):
|
||||||
"""A game folder's recorded keys and answers; a missing file gives no records."""
|
"""A game folder's recorded keys and answers; a missing file gives no records. A list file gives what it holds."""
|
||||||
d = Path(d)
|
d = Path(d)
|
||||||
|
if d.is_file():
|
||||||
|
got = unpack(d.read_bytes())
|
||||||
|
return (read_keys(got[1]), got[0]) if got else ((b'', 0, 0, []), [])
|
||||||
kp, ap = d / 'pipeline-keys.bin', d / 'shader-answers.bin'
|
kp, ap = d / 'pipeline-keys.bin', d / 'shader-answers.bin'
|
||||||
keys = read_keys(kp.read_bytes()) if kp.is_file() else (b'', 0, 0, [])
|
keys = read_keys(kp.read_bytes()) if kp.is_file() else (b'', 0, 0, [])
|
||||||
return keys, read_answers(ap.read_bytes()) if ap.is_file() else []
|
return keys, read_answers(ap.read_bytes()) if ap.is_file() else []
|
||||||
|
|
||||||
|
|
||||||
def merge(dirs, mesh_only=False, log=print):
|
def merge(dirs, mesh_only=False, log=print):
|
||||||
"""One list from the folders' files: each key record once, and the answers record of every shader they name."""
|
"""One list from the folders' files (or earlier lists): each key record once, the answers record of every shader
|
||||||
|
they name and, unless mesh_only, of every compute variant a folder dispatched."""
|
||||||
header, keys, seen, answers = None, [], set(), {}
|
header, keys, seen, answers = None, [], set(), {}
|
||||||
for d in dirs:
|
for d in dirs:
|
||||||
(magic, version, rec, recs), ans = read_dir(d)
|
(magic, version, rec, recs), ans = read_dir(d)
|
||||||
@@ -119,7 +127,13 @@ def merge(dirs, mesh_only=False, log=print):
|
|||||||
added += 1
|
added += 1
|
||||||
for i in ids:
|
for i in ids:
|
||||||
answers.setdefault(i, by_id.get(i) or answers[i])
|
answers.setdefault(i, by_id.get(i) or answers[i])
|
||||||
log(f'{d}: {len(recs)} key records, {added} new to the list, {lacking} without their shaders\' answers')
|
compute = 0
|
||||||
|
for a in ans if not mesh_only else ():
|
||||||
|
if a[0:4] == struct.pack('<I', STAGE_COMPUTE) and a[36:68] not in answers:
|
||||||
|
answers[a[36:68]] = a
|
||||||
|
compute += 1
|
||||||
|
log(f'{d}: {len(recs)} key records, {added} new to the list, {lacking} without their shaders\' answers; '
|
||||||
|
f'{compute} compute variants new to the list')
|
||||||
if header is None:
|
if header is None:
|
||||||
return None
|
return None
|
||||||
return header, keys, list(answers.values())
|
return header, keys, list(answers.values())
|
||||||
@@ -152,16 +166,17 @@ def describe(name, keys_image, answers):
|
|||||||
for i in named:
|
for i in named:
|
||||||
s = STAGE_NAMES.get(stage.get(i), 'not in the answers')
|
s = STAGE_NAMES.get(stage.get(i), 'not in the answers')
|
||||||
by[s] = by.get(s, 0) + 1
|
by[s] = by.get(s, 0) + 1
|
||||||
|
compute = sum(1 for a in answers if a[0:4] == struct.pack('<I', STAGE_COMPUTE))
|
||||||
print(f'{name}: {len(recs)} pipelines ({mesh} mesh, {task} of them with a task stage), layout {magic.decode("ascii", "replace")} '
|
print(f'{name}: {len(recs)} pipelines ({mesh} mesh, {task} of them with a task stage), layout {magic.decode("ascii", "replace")} '
|
||||||
f'version {version}, {rec} bytes a record; {len(named)} shaders named: '
|
f'version {version}, {rec} bytes a record; {len(named)} shaders named: '
|
||||||
+ ', '.join(f'{v} {k}' for k, v in sorted(by.items())))
|
+ ', '.join(f'{v} {k}' for k, v in sorted(by.items())) + f'; {compute} compute variants')
|
||||||
return len(recs), mesh
|
return len(recs), mesh
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||||
sub = ap.add_subparsers(dest='cmd', required=True)
|
sub = ap.add_subparsers(dest='cmd', required=True)
|
||||||
m = sub.add_parser('merge', help='fold game folders into a list file')
|
m = sub.add_parser('merge', help='fold game folders, and lists made before, into a list file')
|
||||||
m.add_argument('out', type=Path)
|
m.add_argument('out', type=Path)
|
||||||
m.add_argument('dirs', nargs='+', type=Path)
|
m.add_argument('dirs', nargs='+', type=Path)
|
||||||
m.add_argument('--mesh-only', action='store_true', help='keep the mesh pipelines only')
|
m.add_argument('--mesh-only', action='store_true', help='keep the mesh pipelines only')
|
||||||
|
|||||||
@@ -291,8 +291,9 @@ def main():
|
|||||||
recs += [(LIST_PRIORITY, r) for r in extra]
|
recs += [(LIST_PRIORITY, r) for r in extra]
|
||||||
(out / SEED_KEYS).write_bytes(plist[1])
|
(out / SEED_KEYS).write_bytes(plist[1])
|
||||||
n_keys = len(pipeline_list.read_keys(plist[1])[3])
|
n_keys = len(pipeline_list.read_keys(plist[1])[3])
|
||||||
print(f'pipeline list: {n_keys} pipelines; of the {len(plist[0])} shader variants they name, {len(extra)} join the '
|
print(f'pipeline list: {n_keys} pipelines; of its {len(plist[0])} shader variants (the ones they name and the '
|
||||||
f'seed, {absent} are of shaders {a.version} does not have, the others are in it already', flush=True)
|
f'compute variants), {len(extra)} join the seed, {absent} are of shaders {a.version} does not have, the '
|
||||||
|
'others are in it already', flush=True)
|
||||||
recs.sort(key=lambda r: r[0]) # stable: list order within a priority
|
recs.sort(key=lambda r: r[0]) # stable: list order within a priority
|
||||||
with open(out / 'seed-programs.bin', 'wb') as f:
|
with open(out / 'seed-programs.bin', 'wb') as f:
|
||||||
f.write(b'NX2NSPRG' + struct.pack('<II', 1, 0))
|
f.write(b'NX2NSPRG' + struct.pack('<II', 1, 0))
|
||||||
|
|||||||
Reference in New Issue
Block a user