Triforce v0.2.0

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
ruckus7382sfgsgdf
2026-11-10 19:32:48 -06:00
commit 34b3683441
1730 changed files with 554853 additions and 0 deletions

View File

@@ -0,0 +1,29 @@
# Screenshot dimension validation
`RX_VK_SHOT_W` and `RX_VK_SHOT_H` accept positive decimal integers. Missing variables
retain the1280x720 default. Empty, signed, whitespace-containing, partially numeric,
zero and overflowing values disable screenshots for that process and report the
invalid request once. They do not fall back to a different requested capture size.
Before any screenshot allocation, the dimensions must fit the device's
`maxImageDimension2D`, the RGBA readback's CPU addressable size, and the existing
PNG encoder. That encoder writes one IDAT with stored deflate blocks: the checked
size includes its zlib header, five bytes per block and Adler32. The
[PNG chunk limit](https://www.w3.org/TR/png-3/#5Chunk-layout) is2^31-1 bytes.
This is a format/capability bound; it does not add a memory budget or guarantee
that a valid request can be allocated on a particular GPU.
Rejected requests never enter the capture producer. Window presentation continues;
headless presentation keeps its existing offscreen policy and produces no capture
for an invalid size. The fixed-size binary frame logger is independent. Valid
standard, ultrawide, odd and small dimensions retain their requested sizes.
`tests/test_capture_dimensions.py` compiles production setup, the real capture
entry/presentation conditions, and the PNG writer. A Vulkan allocation model checks
rejection before calls, reports once, and preserves window/headless routing. A
small real PNG validates chunk CRCs, decoded scanlines and stored-block sizing.
Large dimension/format boundaries use handles only, never large allocations.
Resource readiness in #1472 remains separate. In a combined source tree its sliced
setup test must include the actual dimension guard and initialized physical-device
limits before exercising allocation failures; it must not stub the guard away.

35
docs/capture-readback.md Normal file
View File

@@ -0,0 +1,35 @@
# Capture copies and CPU visibility
Normal screenshots, generated-frame screenshots and binary frame-log slots are
GPU-written mapped staging buffers. Their contents reach CPU consumers through
three distinct requirements:
1. Record a copy-write to host-read dependency for the copied buffer range.
2. Wait for its owning GPU submission or presentation-slot fence.
3. Invalidate that allocation range before reading a non-coherent host cache.
`capture_host_read_barrier` uses COPY/TRANSFER_WRITE to HOST/HOST_READ, offset0
and the complete copied byte count, with no queue ownership transfer. Each
capture producer records it immediately after its copy. Existing timeline/fence
waits remain in place. `invalidate_mapped` delegates cache atom alignment and
coherent-memory handling to VMA.
`shot_pixels` owns normal screenshot bytes after the timeline wait. A pending
screenshot already copied to CPU memory keeps using that snapshot if its parity
slot is reused; it must not invalidate or reread the reused GPU slot. Generated
screenshots invalidate their own allocation after the same completed value.
Frame-log collection invalidates only a used slot after its fence wait and
preserves timestamp, sequence and kind metadata.
The source-sliced regression executes the actual three copy/dependency tails
and the CPU consumers against authored device bytes, host-domain availability
and coherent/non-coherent caches. Four phases (normal, owned snapshot, made,
binary log), two cache modes and two host-visibility modes give16 scenarios.
Permissive host visibility remains a positive control; explicit visibility
detects a missing producer dependency even on coherent memory. These models
verify protocol and ownership, not physical non-coherent GPU memory or driver
behavior. PNG encoding and generic device-loss/wait-error handling are outside
this test; no performance or intended visual change is claimed.
References: [Khronos host readback](https://docs.vulkan.org/guide/latest/synchronization_examples.html#cpu-read-back-of-data-written-by-a-compute-shader)
and [VMA cache management](https://gpuopen-librariesandsdks.github.io/VulkanMemoryAllocator/html/memory_mapping.html).

View File

@@ -0,0 +1,33 @@
# Pending screenshot slot ownership
Normal debug screenshots use two readback buffers, selected by present parity.
A pending capture may outlive the next non-shot present when the GPU is behind.
Another shot two presents later selects the same buffer. Submitting that new copy
before reading the old capture lets a fast GPU replace the pixels, even though
the later read waits for the older timeline value. The filename/flash analysis
then identifies the old frame while consuming the new frame's bytes.
Before recording a new copy into a pending capture's slot, snapshot that capture
into CPU-owned pixels. Its existing consumer moves the snapshot instead of reading
the reused buffer. Another slot still follows the asynchronous path, and no
pending capture adds no wait. Re-arming starts with no saved snapshot, so the old
pixels cannot be attached to the new frame. The generated-shot path and capture
format/metadata remain unchanged.
`do_present` is serialized by the replay job consumer; synchronous replay calls
hold `sync_order_mu`. The pending state remains in that producer/consumer flow.
There is at most one pending snapshot, with the same pixel allocation the normal
consumer already makes, moved earlier only when slot reuse requires it.
`tests/test_capture_slot_ownership.py` compiles the actual copy-selection, snapshot,
read and pending-consume statements with a queued GPU model. It covers a deferred
capture across a non-shot frame, same/opposite parity, fast/late new completion,
no-pending captures and re-arming across sparse intervals. It checks pixels and
metadata together, and verifies that different slots do not wait before submit.
This models legal GPU ordering; it does not claim to reproduce a hardware race.
This change concerns buffer lifetime. Host-cache visibility is the separate
`invalidate_mapped` contract tracked in PR1502; if that change lands first, retain
its normal-shot invalidation inside `shot_pixels`, after the wait and before the
copy into CPU-owned storage. PR1025's on-demand path still needs the same lifetime
rule when integrated; request paths do not change readback ownership.

View File

@@ -0,0 +1,19 @@
# Compiler memory accounting
The build memory gate counts a live compiler driver and its descendants, rather
than just the small driver process. `rss_mb` returns zero for a process not yet
attached, unavailable counters, or a root whose `Popen.poll()` already reports
completion. Retaining a Popen/OS handle must not let stale process-tree counters
or a subsequently reused PID contribute memory after that root has ended.
This check precedes the platform counter query. It does not change compiler peak
estimates, worker floors, reserves, lease release or live child accounting. A
process can still exit during a later process-tree snapshot; this is not an atomic
snapshot or a complete PID-ancestry race solution.
`tests/test_rss_liveness.py` retains an actual completed Popen and models a stale
positive backend sample. The known exit takes precedence without querying that
backend; live-tree sampling and unavailable-counter fallback are also checked.
`test_mem_gate_children.py` separately checks a real child holding300MiB, and the
existing memory-gate/lease regressions cover scheduling behavior. These tests need
no compiler, game dump or game execution.

265
docs/design.md Normal file
View File

@@ -0,0 +1,265 @@
# Hookshot (oot-recomp): design
A native x86-64 Windows and Linux build of the Switch 2 *The Legend of Zelda: Ocarina of Time* remake: the game's AArch64
code translated to C ahead of time from the user's own decrypted dump and compiled with clang, the Horizon OS and
NVN2 implemented natively, and the GPU work replayed on Vulkan. No CPU or GPU emulation at any point: the same
model as mkw-recomp (Mario Kart World), which this project forks.
This document is the plan. The numbers in it that depend on the binaries are filled in from the dump (section 9);
until it exists they are marked *dump*.
## 1. Short answer
**The game is a Switch 2 title, so the base is mkw-recomp, not TOTKPort.** Public reports before its release
(2026-11-05) describe a Switch 2 exclusive, by Nintendo, 1080p to 1440p docked and 720p handheld, with the ocarina
playable by humming into the console's microphone. A Switch 2 executable means:
| | Switch 2 (this game, expected) | Switch 1 (TOTK, TOTKPort's base) |
|---|---|---|
| CPU code | ARMv8.2+ (Cortex-A78C): PAC returns, LSE atomics, CAS, LDAPR | ARMv8.0 (Cortex-A57) |
| Relocations | DT_RELR packed + RELA | RELA |
| GPU API | NVN2 (API 231.x): native images, DLSS, mesh/task programs | NVN 55.x |
| Shaders | Ampere SASS (SM86) in `Nin_Ounce_NVN2` packages | Maxwell SASS in `Nin_NX_NVN` packages |
| Services | nx2 gs/gpu/vi2 sessions, no nvdrv in the npdm | nvdrv, vi:u |
mkw-recomp already handles every row of the left column, from the lifter (`nx2/insn.py`, `nx2/nso2.py`) to the
Vulkan back end with Eden's SM86 recompiler (`gpu/vk`), and its standalone runtime plays MKW races at 45-55 fps.
TOTKPort's NVN1 layer and Maxwell path would have to be rebuilt for Switch 2 first.
`nx2/dump.py` checks this on the real dump: six independent signals (program id prefix, ACID layout, nvdrv in the
services, DT_RELR, ARMv8.1+ instructions, NVN2 proc names) must all say Switch 2. If they say Switch 1, section 3
changes: TOTKPort's `hos/nvn_native.c` (NVN 55.15) would sit under this repository's Vulkan back end instead.
## 2. What is reused (checked 2026-11-04)
| From | What | Where here |
|---|---|---|
| **mkw-recomp** `7336f0a` (`..\mkw-recomp`, local) | lift driver and its fixes, Switch 2 forms, DT_RELR, Eden's image layout, 64-bit guest window, loop-header polls, stable chunks, Unicorn tests, build; the standalone runtime; the native NVN2 layer; the Vulkan back end | `nx2/`, `host/`, `gpu/`, `third_party/` (commit `86871a1`, byte-identical) |
| **a64-kit** (Rexcomp's AArch64 kit, TOTKPort `60de030`, via mkw-recomp) | NSO loader, image builder, lifter, suyu's translator, instruction harness | `third_party/a64-kit` (unmodified) |
| **Eden's shader recompiler** (Eglantine build 181, via mkw-recomp) | SM86 -> IR -> SPIR-V, task/mesh programs | `gpu/vk/eden` |
| **TOTKPort** (`Desktop\Ports\TOTK\TOTKPort`) | the original HLE runtime and NVN layer that mkw-recomp's are forks of | reference only |
| **Eglantine** (`..\Eglantine`, read and run only) | Switch 2 service protocols, the reference renderer, an A/B oracle | reference only; its CPM cache supplies the Vulkan/SPIR-V headers, VMA, FFmpeg |
Keeping the fork's paths identical to mkw-recomp's is deliberate: MKW is the reference port and gets fixes every
day (the queue/event ordering and pool captures of 2026-11-04 among them). A later mkw-recomp commit comes over with
```
git -C ../mkw-recomp diff 7336f0a <new> -- nx2 host gpu third_party > mkw.patch
git apply -3 mkw.patch
```
which works because the base blobs are in this repository. Each sync names its new mkw-recomp base in its commit
message. OoT-specific code goes into new files where it can, so the shared files stay easy to patch.
## 3. Architecture
```
exefs/{rtld,main,subsdk*,sdk} + main.npdm (the user's dump, read in place: work/dump.txt)
│ a64-kit tools/nso.py + nx2/nso2.py (DT_RELR)
▼
modules: LZ4 segments, MOD0, dynamic section, symbols, RELA + RELR, .eh_frame_hdr starts
│ nx2/image.py (Eden's layout from 0x80000000: rtld, main, subsdk*, sdk; we are rtld)
▼
image: relocated, imports bound across modules; the HLE set hooked (nx2/standalone.py)
│ a64-kit lift/a64_lift.py + nx2/insn.py + nx2/lift_fixes.py + nx2/poll.py + nx2/chunks.py
▼
one C function per guest function (work/sa/gen, ~1 GB, local), 64-bit guest window (nx2/rtgen.py)
│ clang (llvm-mingw, emsdk or the system's) -O1, the hot functions -O2 with the play profile, lld (nx2/build.py --standalone)
▼
hookshot.exe = lifted game + host/ (Horizon: threads, SVCs, memory, fs, hid, audio, vi, gs, NVN2)
hookshot_vk.dll = the Vulkan back end (gpu/vk), loaded by host/rx_backend.c unless RX_BACKEND names another renderer
```
- **Standalone from the start.** MKW went emulator-hosted first (Eglantine with a recompiled CPU backend), then
standalone. That detour is not needed any more: the standalone runtime exists and is the user's rule anyway
("100% native"). Eglantine stays the reference to compare against, never part of the process.
- **The SDK is replaced at the nn:: boundary** (TOTKPort's model): `HLE(<mangled name>)` functions in `host/` stand
in for the SDK's OS-facing functions; the rest of the SDK (libc, nn::util, nn::mem, the NVN2 driver's front end)
runs as lifted code. Unimplemented OS-facing imports get generated stubs that log their first call.
- **Memory.** A 512 GB reservation; the image at Eden's `0x80000000` (the lift's addresses), the heap from 64 GB,
address regions growing down from 512 GB; committed as handed out or on first touch (`host/hos.h`).
- **GPU.** `host/nvn_native.c` implements the NVN2 procs over guest memory in place: objects live where the game
puts them, command buffers record an op stream (`gpu/render/rx_nvn.h`), queue submits hand it to the back end.
`gpu/vk` replays it on Vulkan 1.3: pools imported as host memory, SM86 programs through Eden's recompiler to
SPIR-V, mesh/task programs with `VK_EXT_mesh_shader`. The WebGPU renderer (`gpu/render`) stays in the Windows exe as the
fallback when Vulkan is missing, as in mkw-recomp, when the build has wgpu-native (`WGPU_NATIVE`).
## 4. The lessons from MKW, as rules here
Each of these cost days in mkw-recomp; the fork has the fixes, and these are the checks that keep them true for a
second game.
1. **Cross-queue `nvnCommandBufferWaitEvent` is honoured.** MKW's queues pass stages back and forth with events;
replaying them in arrival order on one Vulkan queue flashed. `gpu/vk/vk_replay.cpp` `submit_job` replays each
submission as far as its waits allow, with per-queue state (`RX_VK_EVENT_WAITS=0` turns it off, for A/B only).
*Check early:* log OoT's queues and event waits on the first boot (which queues, how many waits per frame).
2. **Nothing the replay reads comes from live guest memory.** Games write per-frame data ahead of the GPU: every
read is from submit-time captures, and pools are captured from their creation, not their first GPU use
(`gpu/vk/vk_memory.cpp`, `RX_VK_CAPTURE_POOL_MB`). *Check:* the mirror sync log's live-sync count stays ~0.
3. **Texture and sampler registrations are queued in order with the submits**, never applied out of band.
4. **GPU timestamps drive dynamic resolution.** OoT is reported at 1080p-1440p docked, so it has a resolution
controller fed by timestamp reports: the fake GPU time (the counter reports the replay writes, `gpu/vk/vk_replay.cpp`
`NOP_REPORT_COUNTER`: the 31.25 MHz tick clock since the frame's present, times `RX_VK_TS_SCALE`, 0.1 by default) must be
scaled so the controller settles where the PC is, not at its floor. *Check:* the render size it picks over time.
5. **Clears after draws, and draws that sample a target their pass already wrote (or write one an earlier draw of the
pass sampled), get passes of their own** (`vk_draw.cpp` `clear_after_draws`, `draw_hazards`). A draw that samples
a target it writes itself stays in the pass.
6. **2D views of 3D textures need `VK_EXT_image_2d_view_of_3d`** (the RTX 4060 Ti has it).
7. **The CPU saturates first.** Ryzen 5 7600X, 6 cores / 12 threads: the game's threads, the replay, captures and
pipeline compiles share it, and extra helper threads have lowered fps before. Thread counts are measured, never
raised by default (`HOS_VCORES`, `RX_VK_CAPTURE_THREADS`, `HOS_THREAD_TIMES=<seconds>`, the `HOS_SAMPLE` profiler
with `scripts/prof.py`).
Also inherited, and game-specific in MKW: MKW's frame-pacer hooks at fixed guest addresses are gated on MKW's build
id (`hos_game_is_mkw`); OoT gets its own once its pacer is found, never by address coincidence.
## 5. What OoT needs that MKW did not (checked on the dump, 2026-11-04)
| | Found | Plan |
|---|---|---|
| Dump format | one `.nup` package (base v65536 + update v131072, each an exefs/ and romfs/ tree, data stored as is) | `nx2/nup.py` extracts the newest title to `work/dump` |
| NSO compression | `main` is ZBIC (NSO flags bit 7, firmware 22.0.0+: zstd with its own magic and BIC-coded tables); rtld, sdk, subsdk0 are LZ4 | `nx2/unpack.py` rewrites ZBIC NSOs as plain ones (hashes checked), decoder `third_party/zbic` |
| SDK / NVN2 version | 22.2.8 (MKW: 21.4.0); same 46 services and 49 SVCs as MKW | 26 NVN proc names MKW never asked for, 14 not in `host/nvn_native.c` yet (stubs that stop the game only when called; DLSS parameter procs, window acquire timeout, HDR metadata, `nvnProgramBuilderGetShaders`, ...); the device limits in `nvn_native.c` are MKW sdk's (`devinfo[]`) and may need OoT's |
| Instructions | 4 UDOT (FEAT_DotProd) sites, the only forms suyu's translator lacked | `nx2/insn.py` `dot()` |
| Engine | romfs laid out like TOTK's (agl, gsys, Phive, Banc, Bake), Havok 2025.2, NintendoWare G3d2/Vfx3/Ui, SMAA, `.zs` (zstd) files | romfs read from `work/dump/romfs` |
| Microphone (the ocarina by humming) | to check at runtime (nn::audio input; `audsmx:u`, `auddmg:u` are in the services, as in MKW) | first a stub that hears silence; then a host microphone (WASAPI) or a key that plays the note |
| Cutscene video / voice | `Movie/` in the romfs | MKW's paths; `HOS_NO_MOVIES=1` to skip while booting |
| DLSS | NVN2 DLSS procs present (66 DLSS / mesh-task names) | a bilinear stand-in first (as Eglantine), FSR or NVIDIA's own DLSS later |
## 6. Stages and milestones
| | What | Done when |
|---|---|---|
| **S0** | the dump located and characterized (`nx2/dump.py --set`) | the six signals agree; program id, build ids, module sizes, SDK and NVN versions, romfs size recorded in section 9 |
| **S1** | loader + RELR, image in Eden's layout, discovery, coverage, census; instruction test (every form the game uses) and function tests (self-contained and call trees) against Unicorn | 100 % of discovered instructions translated (UDF aside), no BR without a jump table, every form bit-exact, every comparable function case identical |
| **S2** | the standalone lift (`lift.py --standalone --write`) and a full build (`build.py --standalone`): hookshot.exe and hookshot_vk.dll link | the boot runs `_init_libc*`, `nnosInitialize`, `nninitStartup`, the modules' DT_INIT, `nnMain` and reaches `nvnBootstrapLoader` |
| **S3** | NVN2 for this game: the procs it asks for, device limits, its window; the Vulkan replay | **the first frame**: a `run-vk.ps1` screenshot shows what the game draws first (a logo or loading screen) |
| **S4** | title screen and menus: romfs, save data, hid, audio (muted in tests), movies, the stubs it hits | the title screen and the file select are reachable by scripted input (`HOS_INPUT_SCRIPT` / `HOS_INPUT_LIVE`) |
| **S5** | gameplay: correctness (the flash detector `RX_VK_FLASH=1`, frame dumps) and speed (the profiler) | Kokiri Forest plays without flashes at the game's own frame rate |
| **S6** | PC: resolution, frame rate, controls, the microphone, DLSS/FSR | the user's call |
## 7. Testing
- **Instruction forms** (`nx2/test_insn.py`): every form the game's code contains, random states, Unicorn's
ARMv8.x "max" CPU against the lifted C: X0-X30, SP, NZCV, V0-V31, FPSR and memory compared.
- **Whole functions** (`nx2/test_leaf.py`, `--calls` for call trees): random functions of every module on the real
relocated image, natively and in Unicorn, on the 64-bit window with polls firing; registers, V registers and
every changed page compared.
- **The production build** (`nx2/test_dll.py`, when a `hookshot_game.dll` is built).
- **In the game**: `scripts/run-vk.ps1` screenshots, the flash detector, frame dumps (`RX_VK_DUMP*`), the sampled
profiler (`HOS_SAMPLE` + `scripts/prof.py`), and Eglantine as the A/B reference for what a frame should look like.
## 8. Rules
- No game files in the repository: decompressed modules, the image, lifted C, builds, romfs copies, dumps and keys
stay in `work/` (ignored) or in the dump's own folder. Nothing is pushed anywhere.
- Commits end with `Assisted-by: Claude Opus 5.5` and `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`.
- `..\Eglantine` is read and run, never modified; mkw-recomp and the other ports are read and copied from, never
edited.
- One game run at a time on this PC (mkw-recomp's runs take ~10 GB): `run-vk.ps1` refuses while `mkw.exe` or
another `hookshot.exe` runs, and while Rainbow Six Siege is open; no heavy builds while Siege is open either. Runs are
muted (`HOS_AUDIO=0`), at the priority `-Priority` gives (High by default), and screenshot folders are deleted once looked at.
- The lifted C and every build are GPL (suyu's translator is GPL-2.0-or-later; Eden's code is GPL-3.0).
## 9. Status
2026-11-04 (before the dump): fork base committed (`86871a1` = mkw-recomp `7336f0a`), the tools made to take any
dump, the runtime made game-neutral, the stage-1 tools checked on MKW's dump as a stand-in.
2026-11-04, the dump (`gamefile/...[0400486024dec000][v131072].nup`, 25.6 GB; extracted to `work/dump`, 26.2 GB):
- **S0 done.** Switch 2 by all seven signals. Program id 0400486024DEC000, update v131072 (base v65536).
SDK and NVN2 22.2.8. 39-bit address space, main thread priority 44 core 0, 1 MB stack, system resource 48 MB.
| module | build id | .text | .rodata | .data | .bss | eh starts |
|---|---|---|---|---|---|---|
| rtld | 8e213f228d12c7b0f8c21c3ce81b95e214417369 | 10,048 | 2,532 | 49,160 | 65,528 | 42 |
| main | e0f3ccfaf50065ae4ea01eb5aeb13b1a45da3348 | 63,201,744 | 12,603,520 | 5,217,152 | 1,414,272 | 247,520 |
| subsdk0 | b0110087a7b339ef74588d3bd0b4dd5d5c7abd0f | 542,576 | 234,548 | 69,648 | 65,520 | 1,039 |
| sdk | 1643a838814f9c10c696f930ed681e7fa3fb9af8 | 8,906,832 | 6,093,948 | 435,448 | 1,862,408 | 35,786 |
romfs: 186,583 files, 24.4 GiB (Tex 8.8, TerrainArc 5.6, Model 2.6, Sound 2.0 GiB), 10,154 `Nin_Ounce_NVN2`
shader files.
- **S1:** the lift: 295,178 functions, 17,668,191 instructions, 100.0000 % translated (after UDOT), 1,475 jump
tables, no BR without one. Instruction test: 742 forms, 44,172 cases, all pass.
- **S2:** the standalone lift: 294,450 functions in 57 chunks (1.4 GB of C), 485 HLE functions, 243 OS imports
stubbed, 22 sdk NVN functions routed to `host/nvn2.c`.
The build links (`hookshot.exe`, 426 MB; ~25 min for a full build with 4 jobs, a few minutes after a relift).
- **S3 done (2026-11-04, boot13).** The title screen renders at 60 fps: black for ~23 s (~1,400 presents) while it
loads, then the logo, smoke and the menu (New Game / Options / ZELDA NOTES Setting), through the Vulkan back end.
It runs the 150 s out with no crash. What it took, in boot order:
- TLS: the owner handle at TLR+0x110 too (SDK 22). NVN API version 231.40 (main 0x8348c0f0 checks minor >= 40).
- svcGetInfo 20 (UserExceptionContextAddress); the address arbiter (svc 0x34/0x35, WaitIfEqual64 too).
- StandardSteadyClock::now returns in x0. ADR / ADRP+ADD code addresses as entries, but only outside the
functions .eh_frame covers (all of them: 1.68 M duplicated instructions and a 122 MB chunk; now 79 entries),
for the formatter callback 0x824c5560 that nothing else reaches.
- SDK 22 overloads left to the guest SDK, which reached kernel calls the runtime does not have:
CreateThread(..., prio, idealCore), ReleaseLightSemaphore(sem, count), MountRom(name),
GetDirectoryEntryCount.
- nn::os fibers on Win32 fibers, each with its own guest context (nn::async's FiberTaskCoordinator on the
nn.npln threads); a fiber function's NULL return goes back to the thread.
- Offline answers: ConnectToNamedPort gives a session whose requests are answered empty (sm:);
nsd::GetEnvironmentIdentifier "lp1"; socket::detail::Poll nothing ready (gRPC calls it through a pointer);
EnsureNetworkServiceAccountIdTokenCacheAsync fails.
- nvnQueueWaitSync (an ordered no-op), SetCopyImageStride; no virtual core held through host file I/O
(mkw-recomp 16bc3bc: the ARes loaders' 250 ms stalls are gone, longest wait now 0.7 ms).
- **Open at S3:** nvnCommandBufferPushComputeConstants is not implemented; one NVN2 program (e409733c45530d75,
stage 5) does not lower (an unsupported native opcode at +0x4330); 70 imports answered by the generic stub
(audio effects, prepo, nifm, socket/ssl setup, vi refresh rate, hid gyro drift mode, pl::RequestLoadNxFont).
Next: input to get past the title (S4: New Game into Kokiri Forest), then the open items above.
- **S4 (2026-11-05): gameplay.** New Game → the movies (AV1 through Media Foundation) → Kokiri Forest, with the
HUD, at 30-36 fps. What it took (commits through 7e88949 and the ones after):
- Push constants (nvnCommandBufferPushComputeConstants) in c0 after the grid; ClearBuffer; prepo stubs.
- setjmp/longjmp: the lifter wraps every call of a function that calls setjmp in a host setjmp of its own
(host/hos_setjmp.c, assembly, returns_twice: clang's `__builtin_longjmp` restored rbp to the frame address,
128 bytes off Win64's frame pointer), and longjmp's BR comes back through a64_ijump_hook.
- HMMA on the host's tensor cores: HMMA_nx2 → IR MatrixMulAddF16 → OpCooperativeMatrixMulAddKHR 16x8x16 f16.
gpu/vk/vk_coopmat.cpp measures the device's element order (NVIDIA: A's registers 1 and 2 swapped) and checks
a multiply. OoT's networks (the movie upscaler, the gameplay TAAU) run at full speed (the movie at 60 fps).
- Write-backs of GPU-written textures keep the pages the CPU wrote since (SceneLowPriority read texels as
pointers). Storage buffer tables: GetCbuf(i, x*16 + base) addresses get a 16-entry descriptor array that the
global memory fallbacks search (the froxel fog); the 32-bit global atomics got fallbacks too.
- Program variants remember the storage images' integer-ness too: OoT writes R11G11B10 and RGBA8 targets
through R32UI views, and a variant made while the slot held a float texture wrote packed words as floats
(the yellow cast: NaN scene before the TAAU).
- Extended registers: programs using ~249 GPRs (vertex and mesh) keep their high GPRs in IR registers 256 + n
(REGMAP_nx2 prefixes, lower_to_maxwell.h RegisterMapPrefix; R246+ and R239 renamed to markers first).
222 of the 240 dumped failing programs lower now (the offline harness: `nx2/lower_test.py work/lowfail`).
- **Open at S4:** an intermittent Vulkan device loss (nvlddmkm event 153) shortly after the first gameplay frames,
never with RX_VK_CHECKPOINTS=1 (five runs) and every time without (four); mesh output clamps did not change it.
Six programs still do not lower (CALL linkage 0x944, a texture gather mode, an LDG mode). Gameplay is CPU-bound
at 30-36 fps (the replay thread ~9 ms a frame, the GPU ~10 ms): the game's threads next.
## 10. Findings, 2026-11-06
- White flash: one NaN in the scene target goes through the bloom's averages to the whole picture for a frame, the HUD over it.
`GuardFragmentOutputs` keeps float colour outputs finite (RX_VK_NAN_GUARD=0: off). Where the NaN comes from is not known.
- Discards: a nested early return after a demote wrote its colours (`structured_control_flow.cpp`, from Lichii). That was the box
around Link (`1e26755d`, the per-character shadow projector, drawn as a box that discards outside) and most of the milky look.
- The OoT fog (0.25) and exposure (0.65) factors are still needed: with them off the fog is far too thick. Fog's cause is open.
- Fps is CPU-bound: RX_VK_NO_WRITEBACK cut GPU time from 10 to 3.6 ms and left fps alone. HOS_PIN_BY_CORE 35 fps, HOS_SPIN_US=200
slower, narrow barriers no change, RX_VK_WAIT_GPU no help.
- Tools: RX_VK_FRAMELOG + scripts/framelog.py (every frame shown), RX_VK_PROBE=2 (the passes at speed), RX_VK_SLOW_LOG, HOS_WATCH.
- A release build (`RX_RELEASE=1` at build time) precompiles before the game starts; test builds do not unless RX_VK_PRECOMPILE=1.
hookshot_programs.bin (with hookshot_programs.idx, the list of its records), hookshot_pipelines.bin and mkw_vk_pipelines.bin go
beside the executable.
## 11. Findings, 2026-11-06, day
- Builds: `nx2/split_hot.py` puts code_hot.c in eight files that compile side by side, 79 minutes to about 6. `RX_PGO=gen` counts branches
(host/pgo_rt.c writes LLVM raw format 11, the toolchain has no compiler-rt profile library), `HOS_PGO=<path>` with `<path>.reset` and
`<path>.dump` files drives a run, `nx2/pgo_pieces.py` fits the merged profile to the pieces, `RX_PGO=<file>` uses it. `RX_VK_ONLY=1`: the renderer alone.
- Dark frames, open: lit surfaces come out black for one frame, geometry and depth intact. 5% with `RX_VK_FRAMEGEN=0`, 2.9% with the menu's Off, about 0.1% at 4x.
Not the cause: render conditions, clear flags, ClearBuffer order, GPU lag, write-back, the real-time step, the ambient cubemap.
`HOS_COND_RECORD` and `RX_VK_COND_ALWAYS` look like fixes by count and are not: the picture goes white. Tools: the frame log's dark trap dumps probes, passes and conditions; `RX_VK_PROBE_MIN`, `RX_VK_PROBE_UBO_HEX`.
Also not the cause: colour write masks (`wm` in the pass trace: f on every lit pass), constants (`RX_VK_PROBE_UBO_HEX`), queue order (`RX_VK_EVENT_WAITS=0`), and the capture in
synchronous mode (`RX_VK_SYNC_CAPTURE=1`, which does take 380,000 pages per 600 submits). `RX_VK_QUEUE_STATE=0` stops the game at the Load menu, and `RX_VK_MIRROR=0` is 8 fps: neither says anything.
What holds: geometry and depth are right (black shapes against the fog), the prepass and the lit passes run with the same draw counts, and the lit passes output black.
The lit materials' four storage buffers (c0+0x510..0x540) keep their addresses, sizes and non-zero counts in dark frames as seen in guest memory; the 720 KB one is all zeros in good
frames too, so it is no cause, though its compute shader (4b03b304) runs every frame. Guest memory is not what the GPU reads, so these need comparing on the GPU's side.
Next: their mirrors' contents read back on the GPU's timeline, and the bound textures past the centre texel. `RX_VK_SUBMIT_DELAY_US=3000` shows none, but at 33 fps, where they never show:
repeat with a delay that keeps ~45 fps. The MKW session's mirror page skip fired 0 times here and was removed.
Not an instrument's doing: with nothing attached, photographed from outside the window (37 s, ~2200 photographs): 16 with `RX_VK_FRAMEGEN=0`, none at the user's 4x. With
`RX_VK_SUBMIT_DELAY_US`: 7 at 300, 0 at 1000; a repeat at 1000 with shots still had 7 events against 50-60. So it is a race with the game's threads: less overlap, fewer. The delay is a test, not a fix.
- The builder's first full build: 43 minutes at below-normal priority, 0 failures; the folder starts from a double click and reaches the title at 60 fps.
- Render conditions here gate single lights and read reports earlier in the same stream, so they cannot be read at record time.

221
docs/development.md Normal file
View File

@@ -0,0 +1,221 @@
# Hookshot (oot-recomp)
**Hookshot** is the port's name (as Skybound is TOTKPort's and BlueShell is mkw-recomp's); the maintainer's repository
folder stays `oot-recomp`. The game builds as `hookshot.exe` + `hookshot_vk.dll` (`hookshot` + `hookshot_vk.so` on Linux).
A static recompilation of the Switch 2 *The Legend of Zelda: Ocarina of Time* remake to native x86-64 Windows and Linux:
the game's code lifted to C (NotRexed's a64-kit, Rexcomp's AArch64 kit), the Horizon OS and NVN2 implemented
natively, the GPU work replayed on Vulkan. It is a fork of mkw-recomp (`..\mkw-recomp`, Mario Kart World), whose
paths it keeps so MKW's fixes apply here.
**No game files are in this repository.** The dump is read where it lies (`work/dump.txt`); everything made from it
(modules, image, lifted C, builds, runs) goes to `work/`, which git ignores.
Status: the game plays; the README's [status table](../readme.md#project-status) says how far. The design, its stages and
the lessons carried over from MKW: [design.md](design.md).
## Use
Requirements: Python 3 with `pip install -r requirements.txt` (`lz4`, `numpy`, `capstone`, `unicorn` 2.1,
`zstandard`, and the button-prompt packages); a compiler: on Windows llvm-mingw
(`HOOKSHOT_MINGW`, or an `llvm-mingw` folder in the clone), else emsdk's clang at `C:\emsdk` (or `EMSDK`) with Visual
Studio's C++ tools; on Linux clang and lld. For the Vulkan back end: distro Vulkan / SPIR-V / VMA headers under
`/usr`, or `HOOKSHOT_VULKAN_HEADERS` / `HOOKSHOT_SPIRV_HEADERS` / `HOOKSHOT_VMA_HEADERS`, or a CPM package cache
(`RX_CPM`, else `cpm` in the clone, else the old caches in mkw-recomp's `work/eg` or `..\Eglantine\.cache\cpm`).
Without those the standalone build stops. [CONTRIBUTING.md](../CONTRIBUTING.md#development-setup) has the rest.
Run everything with `python -I` from this folder.
| Command | What |
|---|---|
| `python -I nx2/nup.py <file.nup> --extract work/dump` | the `.nup` dump package to `work/dump/exefs` + `romfs` (newest title; ZBIC NSOs made plain by `nx2/unpack.py`) |
| `python -I nx2/dump.py --dump <folder with exefs/ and romfs/> --set` | what the dump is (Switch 1 or 2, ids, modules, SDK, NVN); `--set` records it in `work/dump.txt` |
| `python -I nx2/lift.py [--write]` | image + discovery + coverage (`work/image/coverage.txt`); `--write` emits the C to `work/gen` |
| `python -I nx2/census.py --nx2` | instruction census per module, after the Switch 2 forms |
| `python -I nx2/test_insn.py [--per 6 --states 12]` | every instruction form the game uses, Unicorn vs lifted C |
| `python -I nx2/test_leaf.py [--count 3000 --states 4] [--calls]` | random whole functions (or call trees), Unicorn vs recompiled native code on the real image |
| `python -I nx2/lift.py --standalone --write` then `python -I nx2/build.py --standalone --jobs 4` | the standalone game: `work/sa/build/hookshot.exe` + `hookshot_vk.dll` |
| `powershell -File scripts/run-vk.ps1 [-Window] [-Seconds 120] [-Name x]` | a muted test run on Vulkan, with screenshots in `work/runs/<name>/shots` |
| `python -I scripts/prof.py <stacks file>` | a profile sampled with `HOS_PROFILE=<seconds>` + `HOS_SAMPLE=<thread>` + `HOS_SAMPLE_FILE=<stacks file>` (the sampler runs only with `HOS_PROFILE` set), named from the linker maps |
The dump can also come from `OOT_DUMP` (its folder) or `OOT_EXEFS`; `OOT_WORK` moves the work folder.
The built game's own arguments (`image.bin`, `--romfs`, `--user`, `--trace`, `--prepare-shaders`) are in the README's
[Running the game from the command line](../readme.md#running-the-game-from-the-command-line).
Scripted test input (`HOS_INPUT_SCRIPT` and `HOS_INPUT_LIVE`) is applied before the window-focus check. A log saying
the game window is not in front does not mean those presses were discarded. `HOS_HEADLESS` and
`HOS_WINDOW_NOACTIVATE` change window creation, not this input routing. The game must still reach a state that reads
and responds to the supplied controller input.
Physical keyboard and mouse input stays with the foreground window. Physical controllers can also play in the
background when the existing "Controller in background" option or `HOS_INPUT_BACKGROUND=1` enables them;
`HOS_INPUT_BACKGROUND=0` overrides that option. With `HOS_INPUT_EXCLUSIVE` set (to any value, `0` too) the script
alone drives the game until 5 s after its last press; without it the player's devices add to the script's presses. It
is not a focus override.
## Switches for testing and debugging
The running game reads its switches from the environment or from `hookshot_env.txt` beside `image.bin` (one
`NAME=VALUE` a line, `#` for notes; `HOS_ENV_FILE=<path>` reads that file instead). The build reads its own from the
environment. The README has the ones players are asked to try (`RX_VK_ASTC_RGBA8`, `RX_VK_LOW_SPEC`,
`RX_VK_MOVIE_UPSCALE`, `RX_VK_SKIP_CODE`, `HOS_OOT_CHEATS`). There are several hundred more, most of them one-off
experiments: `grep -rn 'getenv("' host gpu/vk` lists them, and the comment where each is read says what it does.
"Set" below means any value, `0` included.
`RX_VK_MESH_NOCULL=<prefix,...>` turns off the culling in the task shaders whose native code key starts with one of
the prefixes. On a Steam Deck, `RX_VK_MESH_NOCULL=2c56dc1006942f74` avoids the same hang without leaving anything out:
that task shader walks a coarse depth grid cell by cell for every meshlet and ends with
`visible = (result & 1) | (c3[2320] & 4)`, bit 4 being the game's own "draw everything". The switch makes that answer
true, and dead code elimination removes the grid walk, so every meshlet is drawn and the rasterizer clips what is off
screen. It costs some GPU time on cards that cull quickly, so it is off unless set. The log says
"culling off, every meshlet drawn" for each task shader it changed.
### Running a test
| Switch | What it does |
|---|---|
| `HOS_HEADLESS` (set) | No window: the renderer draws offscreen, for screenshots and frame dumps |
| `HOS_WINDOW_SIZE=<w>x<h>` | The window's size in pixels (1280x720 by default, wider for an ultrawide picture) |
| `HOS_WINDOW_NOACTIVATE` (set) | The window opens behind the others and does not take the focus (Windows) |
| `HOS_FULLSCREEN=1` | Starts in fullscreen; the Insert menu's Fullscreen row is then left alone |
| `HOS_AUDIO=0` | No sound device is opened (the mixer still runs); `scripts/run-vk.ps1` sets it unless `-Sound` |
| `HOS_LANGUAGE=<code>` | The game's language for this run, a code from `language.txt` |
| `HOS_OUTPUT_RES=<0..3>`, `HOS_HANDHELD=1`, `HOS_ASPECT=<ratio>` | Over the options file: the output resolution (0 2048x1152, 1 2560x1440, 2 3840x2160, 3 1920x1080), the handheld profile, a picture wider than 16:9 (2.3889 for 3440x1440) |
| `HOS_DYNRES=0`, `HOS_REALTIME_STEP=0`, `HOS_TEXT_SPEED=<0..3>` | Over the options file: Resolution 100% (non-zero: Auto), Game speed Per frame (1: Real time), Text speed |
| `RX_VK_FRAMEGEN=0`, `RX_VK_FRAMEGEN_MULT=<2..8>`, `RX_VK_PACE=0`, `RX_VK_UPSCALE=<0..2>` | No frame generation whatever the menu says; over the options file: its multiplier, frame pacing, the upscaling filter |
| `RX_VK_DEVICE=<n>` | The graphics card listed as `vk: device <n>` in the log, over the menu's GPU device |
| `HOOKSHOT_NO_UPDATE_CHECK=1` | No look for a newer release at the start |
| `HOS_TRACE=1` | Every Horizon call logged, as `--trace` does |
| `RX_BACKEND=<name>` | Unset or `vulkan`: `hookshot_vk.dll`; any other value but `eden`: the WebGPU renderer, when the exe was built with one |
### Input for scripted runs
| Switch | What it does |
|---|---|
| `HOS_INPUT_SCRIPT=<presses>` | Presses timed from the game's first controller read |
| `HOS_INPUT_LIVE=<file>` | A file the test appends lines of presses to, read every 30 ms; each line's times count from when it is read. A line `!cheat <n>` sends a Cheats page action, `!synclog` logs the NVN syncs (`!dump`, `!shots`, `!rdoc` and `!flasharm` are the WebGPU renderer's) |
| `HOS_INPUT_FRAME_CLOCK=1` | The script's time runs at 60 game frames a second instead of the wall clock |
| `HOS_INPUT_EXCLUSIVE`, `HOS_INPUT_BACKGROUND`, `HOS_INPUT_SCRIPT_BACKGROUND` (set) | As above; the last lets every device count while the window is behind another one |
| `RX_VK_MENU_SCRIPT=<seconds>:<key>,...` | Timed Insert menu keys: `INS`, `UP`, `DOWN`, `LEFT`, `RIGHT`, `ESC`, `ENTER`, `TABL`, `TABR` |
| `RX_VK_UI_PROMPTS=pad` or `keys` | The menu's button prompts for a controller or the keys, whatever was used last (screenshots) |
| `RX_VK_UI_GAME_FONT=0` | The menu, the counter and the start-up screen in the included Libertinus, not the game's own font |
The presses are `<seconds>:<button>[+<button>...][/<ms>]`, separated by commas, for example `2:A,5.5:LUP/1500,9:ZL+R`.
The buttons are `A B X Y L R ZL ZR PLUS MINUS LS RS LEFT UP RIGHT DOWN`, and the sticks pushed to their edge `LUP
LDOWN LLEFT LRIGHT RUP RDOWN RLEFT RRIGHT`. A press is held for 200 ms unless `/<ms>` says otherwise. The names are
case-sensitive, and an unknown one is ignored.
### Pictures and frames
| Switch | What it does |
|---|---|
| `RX_VK_SHOTS=<dir>` | A PNG every `RX_VK_SHOT_EVERY` presents (300) as `vk_<present>.png`, at `RX_VK_SHOT_W` x `RX_VK_SHOT_H` (1280x720) |
| `RX_VK_FLASH=1` | The flash detector, with `RX_VK_SHOTS` and `RX_VK_SHOT_EVERY=1`: writes the frames around a flash and dumps the next one |
| `RX_VK_FRAMELOG=<file>` | Every frame shown, at 320x180, while `<file>.go` exists; `python -I scripts/framelog.py <file>` finds the flashes |
| `RX_VK_DUMP=<present>[+<count>][,...]` | Every step of the frame after that present as PNGs, in `dump_<present>` under `RX_VK_DUMP_DIR` (or `RX_VK_SHOTS`, or the current folder); `RX_VK_DUMP_STEPS=<lo>-<hi>` and `RX_VK_DUMP_PROGS=<key>,...` narrow it |
| `RX_VK_PASS_TRACE=1` | The passes of the last frames, logged at a flash or a dark frame (`RX_VK_PASS_TRACE_AT=<n>`: every n-th present) |
| `RX_VK_PROBE=1` or `2` | The centre texel after every pass: `probe_<frame>.txt` at a flash (1), or `probe_all_<n>.txt` at speed (2); `python -I scripts/probe.py` reads them |
| `RX_VK_DUMP_FAILED=<dir>` | The native code of every program that does not translate, for `nx2/lower_test.py` (game data: keep it local) |
### Time and profiles
| Switch | What it does |
|---|---|
| `HOS_PROFILE=<seconds>` | Every interval, the busiest threads in the log; it also starts the samplers below |
| `HOS_SAMPLE=<thread>` + `HOS_SAMPLE_FILE=<file>` | One thread's stacks, sampled every millisecond (needs `HOS_PROFILE`); `python -I scripts/prof.py <file>` reads them |
| `HOS_SAMPLE_ALL=<file>` | Every game and renderer thread every 2 ms while `<file>.go` exists (needs `HOS_PROFILE`); `scripts/crit.py`, `scripts/vkprof.py` and `scripts/hotlist.py` read it (on Linux with `RX_MAPS=<game folder>`: they name frames from its `hookshot` and `hookshot_vk.so`) |
| `HOS_THREAD_TIMES=<seconds>` | Every interval, the threads that used over 5% of a core |
| `HOS_FPS_LOG` (set), `HOS_PRESENT_TRACE=<csv>` | The game's frame limiter, VRR flags and nominal rate every 300 presents; a CSV line per game present |
| `RX_VK_FRAME_LOG=<csv>` | A CSV line per present: frame, seconds, frame time in ms |
| `RX_VK_PASS_TIMING=1` or `2` | GPU time per pass (2: per program too), logged every 600 presents |
| `RX_VK_SLOW_LOG=<ms>` | The replay's times for each frame that took longer than this |
| `HOS_PGO=<file>` | In a run of an `RX_PGO=gen` build: `<file>.reset` clears the counters, `<file>.dump` writes them to `<file>`. It is read before `hookshot_env.txt`, so set it in the environment |
### The GPU
| Switch | What it does |
|---|---|
| `RX_VK_VALIDATION` (set) | The Khronos validation layer |
| `RX_VK_CHECKPOINTS=0` | No NVIDIA checkpoints (on by default; after a lost device they name the draw it stopped at) |
| `RX_VK_HANG_DUMP=<file>` | Every thread's stack to the file once no frame was shown for `RX_VK_HANG_SEC` seconds (20) (Windows) |
| `RX_VK_NOOPT` (set) | Pipelines built without the driver's optimisations |
| `RX_VK_SKIP_PROG=<prefix>,...` | Leaves out the draws of these programs (default `53509044401d78d2`; `-` leaves out none) |
| `RX_VK_PIPE_SKIP=0` | A draw whose pipeline is not ready waits for it, instead of being left out when its targets hold an earlier frame |
| `RX_VK_PRECOMPILE=0` or `1` | The shader preparation before the title off or on (on in builds made with `RX_RELEASE=1`, as the builders' are) |
| `RX_VK_NO_PROGRAM_CACHE` (set) | Neither reads nor writes `hookshot_programs.bin` and the pipeline notes |
| `RX_VK_MIRROR=0`, `RX_VK_MIRROR_MB=<MB>` | No VRAM mirrors of the game's memory pools (on for discrete cards); their budget |
| `RX_VK_VRAM_MB=<MB>` | The texture cache's budget (default 70% of what the driver allows, at most 6 GB) |
| `RX_VK_COMPACT=1`, `RX_VK_COMPACT_PCT=<50..90>`, `RX_VK_COMPACT_MB=<MB>` | VRAM compaction (off by default): the textures that are not render targets in a pool of their own, moved out of sparsely used blocks a few a present so that the blocks go back to the driver, and the texture budget holding the driver's count at that share of what it allows (85); the free space that starts a run (512; 192 while the count is over 90%) |
| `RX_VK_NO_WRITEBACK` (set) | GPU writes to mirrored pools are not copied back to guest memory (an experiment, for A/B) |
| `RX_VK_WB_DIFF=0` | Write-backs copy every byte again instead of only what changed against a VRAM shadow (the diff is on by default; v2.0.9 and v2.0.10 had it off, issue 1161) |
| `RX_VK_DLSS=0/1/2`, `RX_VK_DLSS_PRESET=<letter>`, `RX_VK_DLSS_LOG=1`, `RX_VK_REFLEX`, `RX_VK_REFLEX_LOG=1`, `RX_VK_ANTILAG=0/1` | NVIDIA DLSS and Reflex, and AMD Anti-Lag 2, for one run (`=-` on Reflex or Anti-Lag 2: its extensions not even enabled): the readme's NVIDIA DLSS section. `RX_VK_DLSS_TRACE=<frames>` logs one frame's large draws every that many frames, `RX_VK_DLSS_DEBUG=motion` shows the motion vectors, `=alternate` the game's own upscale every other frame |
| `RX_VK_FSR4=0/1/2`, `RX_VK_FSR_DIR=<folder>` | AMD FSR 4 for one run (0 never, 1 FSR 4, 2 its Native AA), and another folder for AMD's DLLs (not from `hookshot_env.txt`): the readme's AMD FSR 4 section. `RX_VK_FSR4=1/2` and `RX_VK_DLSS=1/2` go over the Upscaler row, a saved FSR 3.1 too where the engine they ask for runs; `RX_VK_FSR=1` goes over them (one upscaler at a time). The DLSS trace and debug switches above apply to FSR 4 too, on the frame chain they share |
| `RX_VK_ASYNC=1` | The replay on a thread of its own instead of the game's submitting thread |
| `HOS_VCORES=0`, `HOS_CORES=<1..8>`, `HOS_RESERVE_CORE=0` | No virtual cores; the cores the game sees (6); no core kept for the replay (one is, with 8 or more logical processors) |
### Building
A build reads and writes tens of gigabytes. When the game folder or the build files folder is on a hard drive,
`make_game.py` warns that an SSD is much faster and builds anyway (the builders ask before they start).
| Switch | What it does |
|---|---|
| `HOOKSHOT_JOBS=<n>` (or `--jobs`) | Compilers at once (default one per CPU thread, each started when the free memory holds it; `HOOKSHOT_MEM_GATE=0` turns that check off) |
| `HOOKSHOT_LIFT_JOBS=<n>` | Processes that write the lifted C (default the cores, at most 16) |
| `HOOKSHOT_KEEP_BUILD_FILES=1` | Keeps the lifted C and the objects after a build, so that the next one compiles only what changed |
| `HOOKSHOT_MINGW=<dir>`, `HOOKSHOT_MSVC=1` | The llvm-mingw to build with; the Visual Studio route instead |
| `RX_CPM=<dir>`, `WGPU_NATIVE=<dir>` | The CPM cache (Vulkan and SPIR-V headers, VMA, FFmpeg) and wgpu-native |
| `RX_TARGET=windows` | On Linux: the Windows game, with llvm-mingw's Linux build in `HOOKSHOT_MINGW` |
| `RX_RELEASE=1` | A release build (`make_game.py` sets it): no console window, the shaders prepared before the title |
| `RX_PGO=gen` or `RX_PGO=<file>` | The hot code counting its branches for a profile, or built with one (`make_game.py` passes the shipped profile) |
| `RX_MARCH=<level>` | The CPU level to build for (default this PC's: `x86-64-v3`, else `x86-64-v2`, else `x86-64`) |
| `RX_VK_ONLY=1` | Only the Vulkan back end, with no lift |
| `OOT_WORK`, `OOT_DUMP`, `OOT_EXEFS`, `OOT_ROMFS` | The work folder and the dump's folders, over `work/dump.txt` |
## The options file
`hookshot_options.ini`, in the game folder, holds the Insert menu's rows as `<key>=<number>`: the row's name in lower
case with `_` for spaces (`frame_generation=2`, `invert_camera_x=1`), except Output resolution, which is `output_res`.
`gpu/vk/vk_menu.cpp` writes the whole file at every change (`save`), with `options_version=4`,
`microphone_device=<name>` when a microphone is chosen by name, and the controls (`pad_<button>` and `key_<button>`:
the README's [Controls](../readme.md#controls)). A value outside a row's range is clamped when it is read. Where the
number is not the obvious one:
| Key | Values |
|---|---|
| `quality_preset` | 0 Custom, 1 Minimum (4GB), 2 Low (6GB), 3 Balanced (8GB), 4 High (12GB), 5 Ultra (16GB+) |
| `gpu_device` | 0 Auto, n the n-th card with Vulkan 1.3 as the row lists them |
| `frame_generation` | 0 Off, 2 to 8 the multiplier |
| `resolution` | 0 Auto, 1 100%, 2 75%, 3 67%, 4 50% |
| `output_res` | 0 2048x1152, 1 2560x1440, 2 3840x2160, 3 1920x1080, 4 5120x1440, 5 5120x2880 |
| `upscaling` | 0 Off, 1 Catmull-Rom, 2 Lanczos |
| `sharpness`, `rumble_strength`, `motion_sensitivity_x`, `motion_sensitivity_y` | percent |
| `mouse_sensitivity` | hundredths: 100 is 1.00 |
| `robust_buffers` | 0 Auto (on), 1 Off, 2 On |
| `game_speed` | 0 Per frame, 1 Real time |
| `text_speed` | 0 Normal, 1 2x, 2 4x, 3 Instant |
| `microphone` | 0 Off, 1 Default, 2 and up a microphone (the name in `microphone_device` decides which) |
| `mouse_camera` | 0 As the stick, 1 Native |
| `layout_preset` | 0 Nintendo, 1 Xbox / PC, 2 Swap Face, 3 Custom |
The other rows are 0 Off and 1 On. Some are read at the start, before the menu exists, by the code that needs them:
`handheld_profile` and `output_res` (`host/nn_applet.c`), `ultrawide` (`host/hos_main.c`), `robust_buffers`
(`gpu/vk/vk_main.cpp`) and the controls (`host/nn_hid.c`).
## Layout
| Path | What |
|---|---|
| `docs/design.md` | why mkw-recomp's pipeline, the architecture, MKW's lessons as rules, stages, testing |
| `nx2/` | the lift pipeline and its tests (mkw-recomp's, made game-neutral; `kit.py` has the paths), and `make_game.py`, which the builders run |
| `host/` | the standalone Horizon runtime (TOTKPort's `hos/` as adapted by mkw-recomp), with the native NVN2 layer `nvn_native.c`; `host/posix` has the Win32 calls on Linux |
| `gpu/vk/` | the Vulkan back end (`hookshot_vk.dll`, `hookshot_vk.so`), with Eden's shader recompiler and sirit |
| `gpu/render/`, `gpu/third_party/` | the WebGPU renderer the Windows exe falls back to when it was built with wgpu-native, its op stream (`rx_nvn.h`), suyu's decoders |
| `builder/` | Hookshot Builder: the Windows app (.NET 8 WPF), its setup and MSI, and the Linux builder and launcher scripts |
| `scripts/` | test runs, the profile and log readers, the save copier (`oot_saves.py`), release tooling, `scripts/linux` (dependencies, AppImage) |
| `tests/` | the game-free checks, all run by `tests/run_all.py` |
| `third_party/` | a64-kit (Hookshot's few changes marked `(hookshot)`), libopus, zbic (Atmosphere's zstd with ZBIC, for unpacking only), SDL2's licence, Dear ImGui and its Vulkan backend, zstd's decoder, the Libertinus fonts |
License: GPL-3.0-or-later, as mkw-recomp: a64-kit's translator is suyu's (GPL-2.0-or-later) and Eden's code is
GPL-3.0.

View File

@@ -0,0 +1,26 @@
# Draw-stream visibility
A recording's CPU-written uniform, vertex/index conversion and buffer-copy streams
are published through VMA before their commands reach the GPU. Host-visible
sequential-write allocations need not be coherent; queue submission alone does not
flush their CPU caches.
The flush names the recording slot, including slots filled by worker threads. It
covers each stream used before growth and the current used prefix. Earlier stream
capacities are flushed in full (including unused padding); later retained streams
from an earlier recording are left alone. Existing VMA cache-range alignment and
coherent-memory handling apply. No per-draw flush or new allocation policy is added.
Both ordinary/batched submissions and successful parallel segments publish their
streams. Failed segments keep their existing no-submit policy. Descriptor lifetime,
slot completion/reuse and batching behavior remain unchanged. The internal draw
helper does not change the host/renderer API.
`tests/test_draw_stream_visibility.py` runs production growth/reservation/copy,
recording reset and complete serial submit code, plus the actual parallel segment
submission tail, with independent expected GPU input bytes and separate host
caches. Worker-to-main slot identity, retained-stream reuse, failed growth, empty
recordings, batching and skipped segments are covered. Vulkan queue/fence behavior
is modeled; this is not physical noncoherent hardware, a concurrency sanitizer,
shader execution or campaign coverage. Global fallback buffers and mixed CPU/GPU
argument rings have separate lifetimes and are outside this stream contract.

38
docs/dump-readback.md Normal file
View File

@@ -0,0 +1,38 @@
# Frame dump readback visibility
Frame dumps are diagnostic inputs: their statistics, images and raw float data
must describe the completed GPU copy. `vk_dump.cpp` allocates mapped staging
buffers with `create_buffer`, which does not require a host-coherent memory type.
Waiting for completion alone does not establish every memory-domain/cache step.
Each dump copy records a COPY/TRANSFER_WRITE to HOST/HOST_READ dependency. The
existing replay caller flushes/waits its submission outside an open burst before
calling `dump_write`. The consumer then invalidates the copied allocation range
before reading pixels or storage-buffer words. VMA handles non-coherent atom
alignment and makes invalidation a no-op where host coherence permits it.
Typed color copies contain RGBA32 float, unsigned or signed integer texels.
Depth and RGB9E5 copies retain their original dimensions and two/four-byte packed
texels until invalidation; depth conversion may then downscale or replace the
mapped pointer with temporary float output. Invalidating after that conversion
would already be too late and could use the wrong range. Buffer dumps invalidate
their actual capped, word-aligned copied size, including the final word.
This changes only the opt-in dump path. Existing GPU barriers, submission waits,
allocation policy, file layout, numeric conversions and cleanup are preserved.
It does not make PNG/raw files a single atomic transaction or change generic
device-loss/wait-error handling. The readback contract assumes successful GPU
completion, as the existing dump caller does.
`tests/test_dump_readback.py` compiles the actual dump translation unit, actual
full-barrier implementation and dump consumer call into an authored CPU model.
Its 56 scenarios cover coherent/non-coherent caches, explicit/permissive host
visibility, typed and packed formats, downscaled depth, buffer tail, raw float
data, statistics-only output, open-burst deferral and cleanup. Copy data, waits,
cache behavior and Vulkan calls are modeled; PNG encoding is outside this test.
This is protocol/numeric regression coverage, not proof of physical non-coherent
hardware, driver behavior, campaign rendering or a performance improvement.
References: [Khronos readback example](https://docs.vulkan.org/guide/latest/synchronization_examples.html#cpu-read-back-of-data-written-by-a-compute-shader),
[Vulkan memory domains](https://docs.vulkan.org/spec/latest/appendices/memorymodel.html),
[VMA mapped-memory cache management](https://gpuopen-librariesandsdks.github.io/VulkanMemoryAllocator/html/memory_mapping.html).

22
docs/flash-sampling.md Normal file
View File

@@ -0,0 +1,22 @@
# PNG flash sampling
`RX_VK_FLASH` compares a normal screenshot against its two neighbors using a64x36
RGB grid. Each cell covers its normalized source rectangle, including remainder
rows and columns when the capture dimensions do not divide the grid evenly.
Sampling retains the existing four-pixel stride. Its mean divides by the actual
number of visited samples, so a partial stride cannot brighten or wrap a channel.
Captures smaller than the grid repeat a valid source pixel for an otherwise empty
cell. Valid capture dimensions are positive; this does not change allocation or
dimension-validation policy.
The cell/region detector, thresholds and output filenames are unchanged. This is
a correction to diagnostic brightness, not a change to game rendering, FOV or HUD.
For example, a constant white3440x1440 capture used to become18 in the first grid
cell after overflow into uint8; its sampled channels now stay255. A below-threshold
brightness variation must not become a flash through that wrap.
`tests/test_flash_sampling.py` compiles the actual sampling and downstream detector
code with synthetic RGBA images. It checks constant channels at standard, wide,
odd and small sizes, labels source regions independently to detect partition drift,
and exercises stable images, a moving edge, a real returning brightness pulse and
a below-threshold negative. No game, GPU or image library is needed.

View File

@@ -0,0 +1,31 @@
# Frame interpolation setup failures
Frame interpolation is optional. A failed pipeline, sampler, image or image-view
creation must leave it disabled for the session, with its unused resources released
and ordinary presentation still available. An unchecked sampler result must never
be advertised as successful initialization.
`vk_framegen.cpp` constructs each pipeline and image locally, publishing the complete
object only on success. Failure releases its intermediate resources. Interpolation
initialization also releases earlier successful pipelines/samplers when a later
stage fails. The separate window-scaling path uses the same pipeline constructor
but retains its own state.
Resize keeps the existing display-job drain and queue/device-idle sequence before
replacing images. If replacement fails, all newly allocated interpolation images,
its setup objects and the published size/multiplier/history are cleared. Size and
multiplier are committed only after every required image/view exists. Interpolation
keeps the existing permanent-failure policy; this adds no per-frame retry loop,
new setting, allocation budget or change to interpolation quality.
The disabled flag is atomic: the replay thread writes it, while Reflex's
game-thread eligibility check can read it through `framegen_on()`.
`tests/test_framegen_resources.py` compiles the real constructors and lifecycle
functions with CPU Vulkan stubs and checks each creation failure, ownership order,
no duplicate destruction, stale metadata, resize draining and the direct-presentation
gate. It covers fixed2x and max8x image sets and the independent scale initialization.
It needs Vulkan headers but no GPU, SDK vendor runtime or game assets.
Injected failures in these tests do not prove actual driver OOM recovery or
device-loss handling. A successful game smoke validates integration, not a fault
injection into the player's GPU.

View File

@@ -0,0 +1,25 @@
# Frame-log sink failures
The optional binary frame log is ready only after its image, three mapped readback
buffers, output file and writer thread are available. A failed startup disables
logging for that process, reports the failing stage and releases the newly created
resources and file. It does not announce successful recording or retry every frame.
The worker checks both record writes and flushes. A short write or failed flush
disables further capture, discards queued CPU records, reports the error and closes
the file once. Enqueue checks the same state under the queue lock, so a producer
arriving after failure cannot leave records behind a worker that has exited.
Readbacks already in flight retain their GPU resources for the existing process
lifetime; they are not destroyed underneath submitted work.
Successful output keeps the existing record format, per-record flush and512record
queue bound. After a failed write the last record may be incomplete: the strict
reader rejects truncation rather than treating the file as a complete comparison.
This does not add renderer shutdown/draining or guarantee durability at process
exit; the healthy writer retains its existing process-lifetime behavior.
`tests/test_framelog_sink.py` executes production startup/cleanup, worker and enqueue
code. Only GPU/IO calls and the startup thread factory are stand-ins. Cases cover
allocation and missing-map failures, file/thread creation failure, repeated ready
queries, real CPU-thread short-write/flush failures, late producers, unchanged
record bytes and queue bounds. Tests require no game or GPU.

View File

@@ -0,0 +1,18 @@
# Sysroot library aliases in Linux update parts
The AppImage's selected system libraries are copied by value into its sysroot.
Some host distributions make a development alias such as libmvec.so an absolute
symlink. Keeping that target would either depend on the player's host filesystem
or be rejected by a strict updater. Dereferencing the
source alias while copying preserves its bytes and attributes inside the part.
The internal relative lib -> usr/lib sysroot layout remains a link. Existing
parts carrying absolute links are not retroactively rewritten. For legacy
sysroot parts only, the updater rebases absolute links to relative links when
their targets exist and resolve inside that sysroot. Absolute links in other
parts, missing or circular targets, and paths that escape the sysroot remain
errors.
Tests execute the production copy loop with synthetic library inputs, then
package and unpack the resulting sysroot, alongside legacy-link and
hostile-archive regression cases.

View File

@@ -0,0 +1,27 @@
# Pipeline cache save failures
`pipeline_cache_save` writes a temporary file before replacing the existing cache. Allocation failures in its
background worker must stay in that worker, release its busy flag and leave later saves possible. Starting the
worker can fail too. The synchronous prepare-only save follows the same exception handling.
Run the game-free check with a C++20 compiler and its runtime libraries available:
```sh
python -I -B tests/test_pipeline_cache_save.py --cxx clang++
```
The fixture compiles the actual function with Vulkan API spies, real C++ containers/threads, and authored temporary
file bytes. It injects allocation failures into the vector, temporary path and thread startup; an oversized vector
checks `length_error`. Each fault case checks that the old file survives and a subsequent save completes. Success,
the 32-pipeline threshold, a single forced-waiter scheduling smoke, Vulkan empty/error returns and a failed write
are covered too. The waiter smoke is not a deterministic concurrency proof.
`--baseline-source /path/to/original/vk_draw.cpp` additionally compares the original success path and runs six
original-failure negatives in separate, bounded processes. Uncaught worker exceptions terminate those children;
an independent authored thread control identifies the exact MSVC fail-fast result where its custom terminate
handler is bypassed. Fixed children must always exit successfully.
These checks use no driver or real cache and do not reproduce issue #1022's reported session. They do not establish
its root cause, validate saving a multi-gigabyte driver cache, force an OS thread-resource failure, or change cache
size/retry policy. The existing wait-then-store protocol for concurrent forced callers is unchanged; the current
force call is in prepare-only startup. Driver/native faults and thread-detach failures are outside this change.

34
docs/png-publication.md Normal file
View File

@@ -0,0 +1,34 @@
# Complete PNG publication
Screenshot, flash and frame-dump writers can run concurrently. The PNG writer
previously built a shared CRC table without synchronization, ignored write/close
errors, and opened the destination directly. A failed capture could replace an
older complete file, while the normal capture still announced success.
The CRC table is now immutable after thread-safe local initialization. PNG sizes
are checked before encoding. The encoded data is written to an exclusively
created temporary in the destination directory; every write and the final close
must succeed before replacement. Failure preserves the prior destination, closes
the owned handle and attempts to remove only this call's temporary. Temporary
name collisions are skipped without truncating other files. Diagnostics identify
the phase, error code and path, with a bounded report count. A later capture can
retry normally. Only successful normal captures announce that they were written.
Each PNG is published independently; a flash triplet is not a multi-file
transaction. Same-directory replacement targets complete-file visibility on
supported local filesystems, not power-loss durability or network-share
transaction guarantees. The encoder's pixels, alpha omission, compression and
capture scheduling are unchanged. Host/renderer ABI is unchanged.
The game-free test compiles the real CRC/writer and normal success-announcement
body. Authored CRT/file replacement failures cover each write, close, publication,
cleanup and open; old destinations and unrelated colliding temporaries must
survive. Concurrent first use and competing writers produce independently decoded
PNG chunks, CRCs, zlib data and RGB pixels, including multiple stored blocks.
The existing capture dimension/PNG consumer check is retained with its new return
type and required headers. Synthetic failures are not a physical disk-fault claim.
References: [PNG specification](https://www.w3.org/TR/png-3/),
[exclusive fopen mode](https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/fopen-wfopen?view=msvc-170),
[Windows replacement API](https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-movefileexa).
Windows narrow paths continue to use the game's existing UTF-8 manifest policy.

View File

@@ -0,0 +1,26 @@
# Device loss during native presentation
The replay already treats `VK_ERROR_DEVICE_LOST` as terminal: `device_lost()`
writes its checkpoint report once, then `replay_fatal()` flushes the host log
before exiting. Continuing cannot complete GPU work and can retain captures and
staging allocations indefinitely.
Native presentation uses the same policy for the frame-fence wait/reset,
command-pool reset, command-buffer begin/end, image acquisition, blit submission
and presentation. The check precedes dependent work: a failed fence wait cannot
lead to readback, a failed submission cannot enqueue a present waiting for its
unsignaled semaphore, and a failed present cannot be counted as completed by
Reflex. Both the shared and dedicated presentation queues are covered.
This does not recover a lost device, change successful presentation, or replace
the existing out-of-date/suboptimal/Wayland-timeout handling. Other error classes
retain their existing behavior, including allocation errors; this is not a
general submission-recovery mechanism.
`tests/test_present_device_loss.py` executes the actual call/guard sites with
injected return values. It covers nine sites (two presentation queues), checks
that loss reports before the site continues, and supplies non-loss controls.
It tests routing rather than simulating an entire Vulkan implementation. The
existing `test_device_lost_exit.py` separately exercises the real concurrent
report-once and host-log-flush/exit contract. Hardware-reset recovery is not
claimed, and no physical driver reset is needed to run these tests.

View File

@@ -0,0 +1,36 @@
# Effective presentation geometry diagnostics
Set `RX_VK_PRESENT_LOG=1` before launching to log presentation geometry changes.
Unset it or set it to `0` to disable it. This writes ordinary diagnostic lines;
it does not enable frame-log image readback or change rendering/settings.
Each sample reports texture size, supplied crop request, clipped content size,
swapchain output size, the actual destination rectangle, observed host aspect,
sampled-view availability, and the route selected by presentation:
- `direct`: the source blit was selected, including fallback when filtering returned no image.
- `scaled`: the scaling function returned an image and its result was selected.
- `clear`: presentation had no source and selected its black clear.
- `hdr`: the HDR conversion function returned an image selected for presentation.
A failed converter falls back to the scaled/direct route. Native PQ input bypasses conversion.
Filter, sharpness and frame-generation values are labeled `*-request`: configuration
does not establish that a feature ran successfully. The route is a CPU command-recording
decision, not a successful GPU present result. The host-aspect field reuses the exact
aspect passed to the destination fit, without a second callback read. No-source frames
report it unavailable, since fitting deliberately does not query the callback then.
This diagnostic does not alter aspect policy.
An unavailable/invalid aspect is explicit; invalid values are normalized for stable deduplication.
Requests accompany geometry/route changes but do not themselves trigger a line, so dragging
a sharpness slider cannot consume the geometry-change budget while the effective route stays the same.
Repeated unchanged samples are suppressed independently for game, interpolated (`made`)
and startup/clear frames, so alternating ordinary/interpolated sources does not spam a
line per frame. Up to 256 geometry changes are logged per process, followed by one
suppression notice. Restart to reset the limit. Continuously changing dynamic resolution
can reach that limit; this is a bounded diagnostic, not a full timeline capture.
`tests/test_present_geometry_trace.py` executes the production presentation-route and
trace blocks with CPU command spies. It checks effective crop/rectangle/output/options,
resize, direct fallback/scaled/clear routes, independent frame kinds, disabled/zero
configuration, invalid aspects and the output bound. No GPU or game data is required.

View File

@@ -0,0 +1,55 @@
# Presentation geometry
The texture allocation, its active crop, the display aspect of that picture and
the swapchain size are separate inputs. `gpu/vk/vk_present_rect.h` is the pure
geometry contract used by the Vulkan presentation path.
- Without a valid `PresentAspect`, fit the clamped crop's pixel aspect, preserving
the existing integer rounding and invalid-source fallback.
- A positive, finite `PresentAspect` describes the **presented picture after
crop**, not the texture allocation. It can differ from the allocation's aspect:
a wide camera picture in a 16:9 texture must still fill its intended wide window.
- Fit that display aspect into the current destination and center any bars. Round
the logical fit to the nearest pixel; a float camera ratio must not introduce a
one-pixel bar when the destination already has its intended shape. A nonempty
destination retains at least one pixel per axis at extreme ratios.
- Zero, negative, NaN and infinite overrides fall back to the physical content
aspect. The callback is read once per present; no host/backend ABI field is added.
For example, a 43:18 picture fills 1720x720. In a 1920x1009 destination it occupies
1920x804, with 102 pixels above and 103 below, even if the physical source texture
is 1920x1080. Changing the destination does not change the camera or render targets.
Normal and interpolated frames share `present_image`. Interpolation materializes
the crop into a separate image, but keeps the same display aspect. The scaling
filter receives the content rectangle's dimensions; the blit uses that same
rectangle and the existing clear path covers its bars. The Insert overlay remains
in swapchain coordinates. Internal screenshots retain their separate capture
dimensions and are not post-presentation evidence of letterboxing.
Game-free checks:
```sh
python -I -B tests/run_all.py present_rect test_present_aspect.py output_resolution --strict
```
`present_rect_test.cpp` covers fit, crop, invalid values, float rounding and bounded
centered rectangles across extreme sizes. `test_present_aspect.py` executes the
production geometry/blit-selection block with recording command stubs: host aspect,
normal/generated images, filtered/direct blits, bar clears and restore. It does not
execute Vulkan; backend compilation and a synthetic GPU/window integration run are
separate validation layers.
An optional GPU check (Vulkan 1.3 loader/device and `HOOKSHOT_VULKAN_HEADERS` needed):
```sh
python -I -B tests/run_all.py present_aspect_vulkan --strict
```
It executes the production transfer/present-selection block offscreen, reads the
pixels back and checks content and black bars against fixed expected rectangles.
Cases include matching wide output, maximize/restore geometry, a padded active crop
and invalid aspect fallback. The padded-crop check excludes only the scaled
half-source-texel footprint of linear filtering at content edges; every bar pixel
is checked. It does not validate swapchain acquisition, a window/DPI transition,
the interpolation shader or the upscaler. Ordinary CI skips this hardware check.

View File

@@ -0,0 +1,25 @@
# Presentation source crops
Direct presentation, capture, scaling and interpolation must sample the same valid
source region. `present_source_rect` is the common contract, independent of Vulkan:
- A null crop, or either nonpositive crop extent, selects the entire image, preserving
the existing convention even when the unused origin fields are negative.
- An active crop with a negative or out-of-image origin is empty.
- Positive extents are clipped to the remaining source area by subtraction before
addition. `x + width` in guest input must not overflow a signed integer.
- Empty sources and dimensions beyond signed 32-bit image offsets are rejected.
The content-aspect calculation uses the same clipped region. Direct/capture blits
clear their destination to opaque black for an empty source region instead of issuing
an invalid Vulkan blit or copying undefined pixels. Zero destination extent records
nothing. Scaling/interpolation decline an invalid crop before setup or allocation;
ordinary presentation supplies the black fallback. Valid crops retain their offsets,
filter and destination rectangle.
`tests/test_present_source_crop.py` executes the production blit, capture wrapper and
complete interpolation recorder with CPU API spies. It checks the command offsets,
interpolation copy push constants, clipped allocation size, invalid-input clear/skip,
full-image convention and thousands of deterministic edge-crossing crops. It needs
Vulkan headers and a C++20 compiler, but no GPU or game data. The independent scaler
fixture separately executes the production scaling function.

View File

@@ -0,0 +1,45 @@
# Per-recording allocation admission and retry
Recording streams carry CPU-authored shader banks, copied UBOs and generated
indices. Their buffer, VMA allocation, mapping and ownership entry must all exist
before a reservation can authorize a CPU write or return a binding.
`stream_insert` commits a complete mapped stream to its recording's container.
Failed creation, missing mapping or container allocation releases only the new
resource. Initial stream creation is independent of descriptor-pool retry, so a
failed pool cannot append another primary stream on every recording. A failed
primary stream remains absent and can be retried on the slot's next recording.
The active stream is explicitly empty until that succeeds; reservations reject
it rather than treating its requested capacity as allocated memory.
Growth preserves earlier streams because already-recorded bindings reference
them. Its cursor, ownership and counters change only after the new entry commits.
Reservation and MiB rounding check representability before arithmetic can wrap.
Successful growth sizes, cached-stream reuse, initial sizes, retention and the
opt-out policy are preserved.
Driver-bank cache generations and grid metadata commit after a successful
upload, or after a confirmed identical-byte cache hit. Failed reservation keeps
the previous generation attached to its previous bytes, so retry rebuilds the
changed bank instead of incorrectly treating stale bytes as current.
The descriptor allocation fallback rejects a missing pool. A new extra pool is
owned only after its first set allocation and the ownership entry both succeed.
Failure destroys that new pool, including any set allocated from it, before any
descriptor update/bind occurs. Existing pools remain valid for retry; a missing
primary pool can be retried by the next recording. Push-descriptor behavior and
the renderer interface are unchanged.
`tests/test_recording_admission.py` executes actual allocation/reservation/reset
functions and the descriptor allocation tail against authored VMA/Vulkan calls.
Only the container boundary is adapted to inject allocation exceptions. Cases
cover healthy byte copies, initial/growth failures, absent mappings, commit
failure, pool/set failure, retry, old binding/pool preservation, parallel slots,
aligned bounds, overflow, disabled growth and retention. Global defaults are
seeded outside this per-recording contract. The tests model resource failures;
they do not prove native driver OOM recovery, campaign behavior or performance.
Global default images/buffers, device loss and other resource families retain
their existing behavior. No cache visibility policy is changed here.
References: [VMA allocation ownership](https://gpuopen-librariesandsdks.github.io/VulkanMemoryAllocator/html/group__group__alloc.html)
and [Vulkan descriptor allocation](https://docs.vulkan.org/refpages/latest/refpages/source/vkAllocateDescriptorSets.html).

49
docs/sampler-admission.md Normal file
View File

@@ -0,0 +1,49 @@
# Sampler cache admission and retry
NVN samplers pass through an ID/generation table, a thread-local parameter/handle
cache and a global owning cache. Ignoring `vkCreateSampler` failure could admit
an undefined error-output handle, or retain null in those caches indefinitely.
Checking only the factory result would still let the outer ID table retain null.
Only successful handles now enter the owning or acceleration caches. The ID
table does not mark a failed creation resolved for that registration generation.
Failure of one key is bounded to one attempt per present; later presents can
retry the same ID, including on a fresh thread. Successful hits and unrelated
keys remain usable. Failure keys are cleared at a later present. If failure
tracking itself cannot allocate, new keys wait until the next present, while
already-owned successful handles still work.
A newly created sampler whose global cache insertion throws is destroyed before
returning null. An error-output handle is never returned or destroyed. The
parameter and thread-local handle tables are acceleration caches: allocation
failure there does not discard registered parameters or destroy a valid global
handle. Existing registration/version checks, immutable successful sampler
lifetimes, filter/LOD/compare/anisotropy/border/reduction policies and the
renderer API are unchanged. Existing replay locking serializes the shared cache
and failure tracking; no new worker or GPU wait is added.
Before either pushing descriptors or allocating/updating a descriptor set,
`bind_descriptors` rejects a combined-image-sampler array containing a failed
sampler. It leaves the pushed-state cache untouched and returns failure to
the existing draw, compute and mesh-fallback callers. A later rebuild can retry;
no replacement sampler changes the requested filtering policy. Storage images
continue to accept their unused null sampler.
`tests/test_sampler_admission.py` executes the complete actual sampler factory,
ID-cache consumer and parameter policy helpers, using the public NVN parameter
layout. Vulkan, registration/environment/hash inputs and the replay-lock boundary
are modeled. Standard containers use tagged throwing allocators to target each
cache operation. The owning-cache fault is armed after successful resource
creation, independent of an STL's sentinel/bucket allocation choices. The 108
cases cover failure/retry/cache chains, ownership and allocation rollback,
independent keys, registration changes, threads and unchanged sampler policies.
The actual complete descriptor-binding function and fingerprint helpers run
against modeled Vulkan calls too: cold and warm failures in an array's second
element, graphics/compute, pooled sets, both pushed-state layouts, full pushes,
next-present recovery and storage images. Rejected writes must produce no
allocation, update, bind or push and must preserve the prior fingerprint.
Run `python -I -B tests/run_all.py sampler_admission --strict`, or invoke the
test with `--headers <Vulkan include folder> --cxx <compiler>`; `--sanitize` is
supported on Linux. Game-free failure models do not prove native OOM/device-loss
recovery, image-quality changes, a full campaign or performance improvements.

120
docs/saves.md Normal file
View File

@@ -0,0 +1,120 @@
# Using a save from an emulator
Hookshot and the emulators and ports that run the Switch 2 Ocarina of Time on a PC all keep the game's saves as the
**same files**. You can carry on in Hookshot from a save made in Lichii (or Eglantine, Orchid, NX2navi, ...), or take
your Hookshot save to one of them, at the same spot: nothing is converted, the files are copied across.
`scripts/oot_saves.py` finds the folders and does the copy safely.
You need your own copy of the game. Nothing here
comes with game files.
## The files
Every one of them keeps one folder per user profile, holding:
```
account.sav
option.sav (the game's own settings)
slot_00/ ... slot_NN/ (one per save slot)
progress.sav
caption.sav
```
Only the folder they are in differs.
## Where the saves are
| Program | Linux | Windows | How sure |
|---|---|---|---|
| Hookshot | `~/.local/share/hookshot/save/<user id>/` | `<game folder>\user\save\<user id>\` | Hookshot's source; Linux tested |
| [Lichii](https://github.com/zortos293/Lichii) | `~/.local/share/eden/` + Eden's save path below | `%APPDATA%\eden\` + Eden's save path | Lichii's source; Linux tested, Windows not |
| [Frizti (Fritchii)](https://github.com/Frizti/Lichii), a Lichii fork | (Windows only) | `%APPDATA%\eden\` + Eden's save path | its source (`fritchii-v5.5`); not tried |
| [Orchid](https://github.com/MythosOfs/Orchid) | `~/.local/share/eden/` + Eden's save path | `%APPDATA%\eden\` + Eden's save path | its source; not tried |
| [PokeDinner](https://git.eden-emu.dev/Franseven/PokeDinner-a-Eglantine-Fork) | `~/.local/share/eden/` + Eden's save path | `%APPDATA%\eden\` + Eden's save path | its source; not tried |
| [Eglantine](https://github.com/Djihads80/Eglantine) | `~/.local/share/eglantine/` + Eden's save path | `%APPDATA%\eglantine\` + Eden's save path | its source; not tried |
| [NX2navi](https://github.com/tangerineturtle1/NX2navi) | `<game folder>/profile/save/<account>/` (its AppImage builds into `~/Games/Ocarina`) | `<game folder>\profile\save\<account>\` (the launcher's game folder is `%LOCALAPPDATA%\Programs\NX2navi\game` unless you chose another) | its documentation (`docs/launcher.md`, `docs/installer.md`, `docs/README-BuildKit.md`); not tried |
Eden's save path, under that data folder: `nand/user/save/0000000000000000/<profile id>/0400486024DEC000/`.
- `0400486024DEC000` is the game's title id. Eden's `<profile id>` is the Switch user profile you play with;
Hookshot's `<user id>` and NX2navi's `<account>` are one folder with a 32-character hex name.
- **Lichii, Frizti, Orchid and PokeDinner share one folder**, Eden's (`eden`): a save there is the same save in all of
them. Eglantine has its own (`eglantine`).
- The emulators use a `user` folder beside the program (Linux: in the folder you start it from) instead of the paths
above when that folder exists (their "portable" mode). Point the script at it with `--from`/`--to`.
- Newer Eden code also reads a save from `nand/user/save/account/<profile>/0400486024DEC000/0/` when that folder
exists, and then uses it first. The script looks there too.
- NX2navi writes a save to `profile/save-tx` first and moves it into place after. While something is waiting there,
that is the save the game loads next, so the script refuses to copy until NX2navi has been started and quit (or
`--force`). NX2navi's launcher also has its own **Import Saves**, which takes an emulator's save folder.
- `$XDG_DATA_HOME` replaces `~/.local/share` when it is set. Hookshot's `--user <folder>` puts its saves in
`<folder>/save/<user id>/` ([command line arguments](../readme.md#running-the-game-from-the-command-line)).
- The emulator paths come from each one's `src/common/fs/fs_paths.h`, `path_util.cpp` and
`src/core/file_sys/savedata_factory.cpp`. Official Eden does not run Switch 2 games; its name is accepted because
the forks above use its folder.
- Not listed: Ryujinx and its forks (Ryubing, ...) do not run Switch 2 games. Any other program that keeps these files
works through `--from`/`--to` with the folder that holds them.
## Copy with the script
`scripts/oot_saves.py` (Python 3, nothing to install) copies the slots, `account.sav` and `option.sav` one way, after
backing up the side it overwrites. From the Hookshot folder (or a clone):
```sh
python -I scripts/oot_saves.py list # every save it finds, its slots and when each was saved
python -I scripts/oot_saves.py import lichii # Lichii -> Hookshot
python -I scripts/oot_saves.py export lichii # Hookshot -> Lichii
python -I scripts/oot_saves.py import --from ~/Downloads/my-save # any folder holding a save -> Hookshot
python -I scripts/oot_saves.py export --to ~/Lichii/user # Hookshot -> a portable emulator
```
The names are `lichii`, `frizti` (or `fritchii`), `orchid`, `pokedinner`, `eden`, `eglantine` and `nx2navi`.
`to-lichii` and `to-hookshot` still work: they are `export lichii` and `import lichii`.
On Windows Hookshot's folder is your game folder, which the script cannot guess, so name it:
```powershell
python -I scripts\oot_saves.py import lichii --hookshot "C:\Games\Hookshot"
python -I scripts\oot_saves.py export nx2navi --hookshot "C:\Games\Hookshot"
```
| Option | What it does |
|---|---|
| `--dry-run` | Says what it would copy and where, and changes nothing |
| `--keep-settings` | Leaves the target's `option.sav` alone, so each program keeps its own settings |
| `--profile <id>` | The emulator's profile to use. Needed when more than one profile has saves: the script lists them and stops rather than pick one |
| `--from <folder>`, `--to <folder>` | The emulator's side: the save folder itself, or a folder above it to search (a data folder, a portable `user` folder, NX2navi's game folder). With a name, it replaces where that emulator is looked for; without one, any folder works |
| `--hookshot <folder>` | Hookshot's data folder (Windows: the game folder), or its `save\<user id>` folder |
| `--backup-dir <folder>` | Where the backup goes, outside the target save folder |
| `--force` | Copies even if a game seems to be running, or NX2navi has a save waiting |
The script refuses to run while it sees Hookshot or any of the programs above running. The side being overwritten is
copied to `oot-save-backups/<import or export>-<name>-<date>-<time>/` first: in `~/.local/share` on Linux, in
`%LOCALAPPDATA%\Hookshot` on Windows. To undo a copy, put the backup's contents back.
## Copy by hand
Close both programs, then copy the **contents** of the folder you are leaving into the folder you are going to (see
the table above). Keep a copy of the target first. Delete the target's old `slot_*` folders before copying, so a slot
that only exists on that side does not stay mixed in with the new save.
## Before you copy
- **Close both programs first.** A game that is running can write its save again on exit and undo the copy.
- **Never run two of them on the same save at the same time.** Pointing two at one folder with a symlink may work,
since the files are the same, but it has not been tried.
- **Settings follow the save.** `option.sav` holds the game's own settings (camera, text speed, ...) and is copied too.
Use `--keep-settings` (or skip that file by hand) if you want different settings in each.
- **Profiles.** If your emulator has several user profiles, make sure you copy to and from the one you play on.
- **Backups.** The script keeps one for every copy. Delete old ones when you no longer need them.
- Start each program once and save in it before the first copy, so that its save folder exists.
## What was tested
- Linux (Arch-based), Hookshot v2.0.5 and Lichii built from source on the same day.
- A Hookshot save copied to Lichii loads in Lichii at the same spot (Kokiri Forest, the spawn by the flower patch).
- A Lichii save copied to Hookshot arrives identical, but has not been played through on the Hookshot side.
- The script's discovery for every program above, its copies and backups are covered by `tests/test_oot_saves.py`, on
temporary folders.
- **Not tested:** Windows, and every program but Lichii. Their folders come from their source or, for NX2navi, its
documentation, as the table says.

View File

@@ -0,0 +1,55 @@
# Failed swapchain replacement
Passing a non-null `oldSwapchain` to `vkCreateSwapchainKHR` retires it even if
creation fails. It cannot be acquired again or passed as a non-retired old chain
on the next attempt. See the [Vulkan swapchain creation contract](https://docs.vulkan.org/refpages/latest/refpages/source/VkSwapchainCreateInfoKHR.html).
The same restriction applies when creation succeeds but enumerating the new
images or creating their presentation semaphores fails. Leaving the retired
handle in `dev.swapchain` makes the following present use an invalid chain.
Replacement now drains outstanding work with the existing device/present-queue
idle helper, invalidates Reflex's cached handle, and releases the retired chain
and its per-image semaphores after the creation attempt. New images and
semaphores are prepared before publishing the replacement. An unsuccessful
image-data query, including `VK_INCOMPLETE`, discards the incomplete replacement.
At the start of presentation, failed recreation returns before advancing the
frame slot, waiting/resetting its fence, or acquiring an image. The requested
size is retained for another attempt unless the window has already published a
newer size. A missing swapchain also triggers recreation without requiring a
new window event. Failures before the creation attempt leave the existing chain
owned but skip that presentation and retry the request.
## Regression coverage
`python -I -B tests/test_present_semaphores.py --cxx clang++` compiles extracted
creation, presentation-prefix, semaphore-selection and Reflex-invalidation code
with CPU Vulkan stubs. It covers:
- Four swapchain images across three frame slots, followed by two/five-image replacements.
- Retirement even when `vkCreateSwapchainKHR` fails.
- Semaphore allocation failure, image-count error/zero, and incomplete image data.
- Release of partial resources and recovery on a later attempt.
- A failed preflight retaining the current chain while skipping acquisition.
- Retrying the requested size while preserving a newer resize during recreation.
- Recreation requested by Reflex and recovery without a resize notification.
- Invalidation of Reflex's cached handle before its swapchain is destroyed.
The test requires a C++20 compiler, with no game, SDK runtime, GPU or display.
It is discovered by the existing source-regression runner. CPU stubs do not
establish driver behavior, Vulkan validation-layer results, or timing under
concurrent game rendering. Native WSI and game integration remain separate checks.
Reflex consumers check eligibility while holding their state mutex. A latency
sleep captures its swapchain generation and timeline target before releasing
that mutex for the CPU wait. After the wait, it checks the same generation,
latest target and eligibility before marking a new frame. Retired or replaced
chains, including a replacement reusing the same handle, cannot receive stale
markers. `tests/test_reflex_swapchain_lifetime.py` compiles the actual consumers
and exercises retirement before locking and during the wait, handle reuse,
overlapping sleep targets, disabling Reflex/turning on FG, and later recovery.
This handles replacement failure rather than device-loss or surface-loss
recovery. It preserves the existing sizing policy, per-image presentation
semaphores, queue locking, and frame-generation presentation path.

View File

@@ -0,0 +1,30 @@
# Tested configurations
What has actually been run, where, and how it went: one row per test run, newest first. The README's
[status table](../readme.md#project-status) sums this up by vendor; this page keeps the runs themselves, so a
regression can be traced to a build and a driver, and so the next tester knows what is already covered.
Add a row for your own runs (a PR, or paste it in an issue and a maintainer adds it). A run that worked is as useful as
one that did not. Keep it to facts you saw: the build (`hookshot vX.Y.Z-N-gHASH` from the first line of
`hookshot_log.txt`), the hardware and driver, the settings that matter, where in the game, and what you measured.
Frame rates come from `hookshot_log.txt` ("N presents, X fps") at a named spot, standing still unless the row says
otherwise.
## Runs
| Date | Build | OS / desktop | GPU / driver | CPU / RAM | Settings | Where | Result |
|---|---|---|---|---|---|---|---|
| 2026-11-10 | host `v2.0.11-438-gcd544a7` (source `8bdd7a54`); renderer main `8bac8aa` + #1502 + the host-clock capture change | Windows 11 Home N, 26300 | RX 9070 XT 16 GB, 32.0.32015.12012 | Ryzen 7 5700X3D, 64 GB | 2560x720 window, 32:9 camera, serial, FG off, dense capture, core validation; fresh prepared shader bank | Boot, title, save load; gameplay not reached | 96 s: 1,476 consecutive original capture records; all 1,461 records covered by the flushed native trace align one-to-one with native returns. Movement window rejected: no filled gameplay interval. No new validation IDs; existing `02251`, and two `08737` errors during full preparation. No gameplay speed or performance claim |
| 2026-11-10 | v2.0.11-447-gcbe49eb, sampler admission branch (main 8bac8aa + sampler checks + staging fixture repair) | Windows 11, windowed | RTX 5070 Ti, 621.01 | Ryzen 7 9800X3D, 32 GB | 3440x1440 picture, native upscaling, fixed FG 2x, 1 MB recording streams; serial and parallel replay (2 workers, batch 1) | Kokiri Forest, inside Link's house; copied checkpoint | Both runs rendered the scene, maximized/restored (client 1444x720 to 1440x829 and back), and exited normally. Six representative PNGs decoded at 3440x1440; original checkpoint hashes unchanged. Healthy-path validation only; no FPS comparison or native OOM recovery claim |
| 2026-11-10 | main b2bb214 (+ #1447 #1451 #1520 #1660 #1619 #1448 #1654 #1612 #1600, unmerged then) | Ubuntu 26.04.1, KDE Plasma 6.6.6 Wayland | RTX 4070 Ti 12 GB, 595.99.02 | Ryzen 7 9800X3D, 32 GB | 3840x2160 output, DLSS Quality, no mods | Hyrule Castle road (child) | 48-50 fps. Same spot at 2560x1440 output: 56.3 fps. No device lost, saves written |
| 2026-11-10 | main b2bb214 + #1459 (Dear ImGui menu) | as above | as above | as above | as above, windowed | Title, Hyrule Castle road | F1 menu, help box, prompts, the game's font and the FPS badge all correct; 60 fps |
| 2026-11-10 | main b2bb214 | as above | as above | as above | HDR on | start-up | KDE offers an HDR10 surface (`vk: HDR: an HDR10 surface (format 58)`) |
| 2026-11-09 | main 7eac5bf | as above | as above | as above | 3840x2160, DLSS Quality, no mods | Hyrule Castle road | 52.8 fps (replay submit 17.5 ms, memory sync 4.8 ms, GPU 12.8 ms) |
| 2026-11-09 | main 7eac5bf | as above | as above | as above | fullscreen (`HOS_FULLSCREEN=1`) | Hyrule Castle road | fullscreen works, no device lost |
| 2026-11-09 | v2.0.11-142 (7dbdcdc) | as above | as above | as above | 3840x2160, DLSS Quality | Water Temple 1F, Hyrule Castle road, Deku Tree 2F | Water Temple walls missing with the 23.3 GB dump, drawn with stand-in textures (#1308) or a complete dump. Slingshot and bow aiming: no fps change (51-52 fps, #1305). Pause / unpause: no freeze |
## Limits of the original RTX 4070 Ti runs
- Real mouse aiming with Mouse camera = Native after #1462 (the camera states were traced, but not tested with a real mouse).
- Frame generation at 120 Hz or above (the displays here top out at 75 Hz).
- HDR output on an HDR monitor (the surface is offered, the picture was not judged on one).

View File

@@ -0,0 +1,31 @@
# Texture staging cache publication
Texture staging arenas use mapped sequential-write host memory. That memory is
not required to be coherent: writing the pointer alone does not publish encoded
ASTC blocks or CPU-converted texels to the GPU.
Both staging consumers now flush the completed allocation-relative slice before
its GPU use. Staging carries its VMA allocation alongside the buffer/offset, and
slice starts respect `nonCoherentAtomSize` as well as the consumer alignment.
This keeps a rounded cache flush from sharing an atom with another slice already
being read by a submitted parallel segment. VMA handles allocation offsets, cache
rounding and coherent-memory no-ops. No additional queue wait is introduced.
CPU uploads publish after conversion workers join, before the slice is added to
the pending image copy. ASTC uploads publish after gathering their layers, before
the GPU decoder is recorded. This also covers uploads redirected into an earlier
parallel init command buffer, without depending on a later worker's submit hook.
Arena sizes, allocation fallback, lifetime/reuse, format conversion, shader paths,
descriptor policy and the renderer ABI remain unchanged.
The game-free fixture executes actual staging allocation and the actual ASTC
gather/publication prefix and CPU publication/copy-description region. Authored
CPU/device caches distinguish coherent and noncoherent memory; it does not execute
the full CPU converters or ASTC shader. Cases cover nonzero offsets, multiple
slices/arenas, completion-based reuse, 1024-byte atoms, failed allocation and empty
flushes. A source-order guard keeps CPU publication after worker joins. The range
helper itself is extracted from production. These checks do not establish a
physical noncoherent GPU reproduction or campaign/performance coverage.
Run `tests/test_texture_staging_publication.py --headers <Vulkan include> --cxx <compiler>`.
Linux additionally supports `--sanitize` for AddressSanitizer/UndefinedBehaviorSanitizer.

19
docs/ui-texture-upload.md Normal file
View File

@@ -0,0 +1,19 @@
# UI texture staging visibility
The custom ImGui texture uploader packs the requested rectangles into mapped
staging memory and flushes that allocation after its final pixel write, before GPU
use. Sequential-write host-visible memory need not be coherent. VMA's flush helper
handles cache-line alignment and treats coherent allocations appropriately.
This covers new font/logo textures, individual dirty rectangles and the fallback
update rectangle, for RGBA and Alpha8 expansion. The existing image layouts,
descriptor publication, retry behavior and staging retirement remain unchanged.
The bundled ImGui backend maintains its own vertex/index upload caches; this
corrects Hookshot's separate texture-upload path.
`tests/test_ui_texture_upload.py` executes production pixels/upload/retirement code
with the pinned ImGui/Vulkan types. A CPU/GPU cache model covers coherent and
noncoherent memory, full and partial uploads, untouched pixels, failure/retry and
deferred destruction. GPU copies execute after recording returns, so flushing
after recording but before submission remains a valid control. HDR color
conversion and physical noncoherent hardware behavior are not modeled.

191
docs/versioning.md Normal file
View File

@@ -0,0 +1,191 @@
# Versioning and releases
Hookshot follows [Semantic Versioning 2.0.0](https://semver.org/) and the conventions of
[semantic-release](https://semantic-release.org/). The current version is **0.1.0**.
The git tag is the only source of truth. No file carries a version that has to be edited for a release: the game reads
the tag through `nx2/version.py`, and the builder and the AppImage are stamped from the same tag when they are
packaged.
## The version number
`MAJOR.MINOR.PATCH`, with an optional prerelease suffix: `0.1.0`, `0.2.0`, `0.2.0-rc.1`. A release is tagged
`v0.1.0`: the `v` belongs to the tag, not to the version. Build metadata is not used in tags.
One version covers the whole project: the game (`hookshot.exe` / `hookshot`), the Vulkan back end, the lift pipeline
and its scripts, Hookshot Builder and the Linux AppImage. They are built from the same commit and released together,
so they never carry different versions.
The "public" surface the rules protect is the command line of `make_game.py` and the builder, the `RX_` and `HOS_`
switches, the files a build writes into a game folder (`hookshot_options.ini`, `hookshot_env.txt`, `language.txt`,
`hookshot_version.txt`), saves, and the format of `hookshot_log.txt`. The C sources and internal headers are not an
API and may change in any release.
## Commits decide the version
Releases are computed from the commits on the release branch since the last tag, so commit subjects must be
[Conventional Commits](https://www.conventionalcommits.org/):
```
type(scope): description
type(scope)!: description (a breaking change)
```
`scope` is optional. A `!` before the colon, or `BREAKING CHANGE:` / `BREAKING-CHANGE:` in the commit body, marks a
breaking change. The description is written as an imperative ("Stop scanning...", "Fix the..."), because it becomes
the changelog line.
| Commit | Next version |
|---|---|
| any commit with `!` or `BREAKING CHANGE:` | major (`0.1.0` -> `1.0.0`) |
| `feat` | minor (`0.1.0` -> `0.2.0`) |
| `fix` | patch (`0.1.0` -> `0.1.1`) |
| `perf` | patch |
| `revert` | patch |
| `docs`, `refactor`, `style`, `test`, `build`, `ci`, `chore` | no release |
`git revert` writes `Revert "..."`, which is not a Conventional Commit: use `git revert --edit` and write the subject
as `revert: ...`, or the revert makes no release.
A release takes the highest bump among its commits. Merge commits are ignored, and a commit with `[skip release]` or
`[release skip]` in its message is left out of the analysis. `chore(release): 0.2.0 [skip ci] [skip release]`, the
commit the release itself makes, is not a release (the notes in its body can mention a breaking change).
There is no manual version bump. If a change should not make a release, use one of the types in the last row; if it
should, use `feat` or `fix`.
## 0.x and 1.0.0
semantic-release does not treat major version zero specially: a breaking change is a major bump even from `0.1.0`.
This is deliberate. Its documentation says that SemVer's different rules for `0.x` are out of scope, and it recommends
`1.0.0` as the first release.
Hookshot starts at `0.1.0` anyway, because the first versioned release is not stable. That tag is made once by hand
(see below); after it, the release workflow is the only thing that creates tags. Consequence: the first breaking change
merged to `main` releases `1.0.0`, not `0.2.0`.
`1.0.0` means the public surface above is considered stable. Until then, keep breaking changes on the `next` branch
and release them there as `1.0.0-rc.N`, or accept that the release that carries one becomes `1.0.0`.
## Branches and the release workflow
| Branch | Purpose | Version |
|---|---|---|
| `main` | stable releases | `vX.Y.Z` |
| `next` | release candidates | `vX.Y.Z-rc.N` |
Every push to `main` or `next` runs the [Release workflow](../.github/workflows/release.yml). It:
1. finds the last release tag and the commits since it;
2. takes the highest bump in the table above;
3. runs `scripts/changelog.py` to put the new entry at the top of `changelog.txt`;
4. commits `changelog.txt` as `chore(release): <version> [skip ci] [skip release]`;
5. tags the release commit. The tag starts the Linux AppImage and Windows builder workflows: the first to finish makes
the GitHub release, and each attaches its files (the AppImage; `HookshotBuilder.exe`, the setup, and
`hookshot-setup.txt`, with the new parts in the `builder-parts` release). `scripts/release_upload.sh` is shared by both.
A merge with no releasable commit (only docs, chores, and so on) makes no release. A release is never edited by hand,
never re-tagged and never deleted. Require the *Game-free source regressions* workflow on `main` and `next` (branch
protection): the release then runs for a commit whose tests have passed.
The workflow pushes back to the repository, so it needs a token with `contents: write` that may bypass `main`'s branch
protection. Use a `RELEASE_TOKEN` secret (a fine-grained personal access token) rather than the built-in
`GITHUB_TOKEN`: a tag pushed with `GITHUB_TOKEN` does not start another workflow, and the Linux AppImage and Windows
builder workflows publish the release from the tag. With only `GITHUB_TOKEN`, build both with `workflow_dispatch` and
create the release by hand.
## The first tag: 0.1.0
semantic-release's first release is `1.0.0` when it finds no tag. To make `0.1.0` the first version, tag the merge
commit that introduces this scheme once:
```sh
git tag -a v0.1.0 -m "Hookshot 0.1.0"
git push origin v0.1.0
```
The Release workflow does nothing while no version tag exists, so the merge cannot release `1.0.0` first. The pushed
tag starts the Linux AppImage workflow, which creates the `0.1.0` GitHub release with the AppImage attached; create the
release by hand if you do not want to wait for it. The `0.1.0` entry in `changelog.txt` is written for that tag. After
it, do not tag by hand.
## Pre-releases
A release candidate is released from `next` with the same commands. semantic-release numbers it `v0.2.0-rc.1`,
`v0.2.0-rc.2`, and so on; the GitHub release is marked as a prerelease and never becomes "latest". A prerelease is not
written to `changelog.txt`: Factorio's parser accepts only `major.minor.patch`, and two release candidates share a base
version. Its notes are in the GitHub prerelease. Merge `next` into `main` to release the final `0.2.0`, whose entry
lists every commit since the last stable tag.
## Hotfixes
Branch from the release tag, not from `main`, when a fix must not include `main`'s unreleased features:
```sh
git switch -c fix/crash v0.1.0
```
Commit with `fix: ...` and open a pull request to `main`. If `main` has no unreleased feature commits, the release is a
patch. The project does not keep maintenance branches for older minor versions: only the newest release is supported.
## Where the version appears
| Place | Where it comes from |
|---|---|
| git tag | semantic-release; the source of truth |
| `changelog.txt` | `scripts/changelog.py` during the release |
| `hookshot_log.txt` and the Insert menu | `nx2/version.py`: `HOOKSHOT_VERSION`, then `VERSION`/`version.txt`, then `git describe --tags --match "v[0-9]*"` |
| `hookshot_version.txt` in a build | the `VERSION` file written when the tools are packaged (`nx2/make_game.py`) |
| AppImage file name | `git describe --tags --match "v[0-9]*"` (`scripts/linux/appimage.sh`) |
| `HookshotBuilder.exe` | the MSBuild `Version` passed at publish time (`dotnet publish -p:Version=<version>`, the Windows builder workflow); the `csproj`'s `0.1.0` is only the default for a local build |
`nx2/version.py` also appends a fingerprint of the source files, `(source 1a2b3c4d)`. It is not a version: it tells
two builds of the same release apart, and it is the only identifier a build from a modified checkout has.
`package.json` and `package-lock.json` (if present) are the release tooling's, not the project's: `package.json` keeps
the placeholder `0.0.0-development` on purpose, and the tag remains the version. The game never reads them.
## The changelog
`changelog.txt` follows [Factorio's changelog format](https://lua-api.factorio.com/latest/auxiliary/changelog-format.html).
Newest first, with a 99-dash separator before each version:
```
---------------------------------------------------------------------------------------------------
Version: 0.1.0
Date: 07. 11. 2026
Features:
- Something new.
Bugfixes:
- Something fixed.
```
The date is day. month. year (`07. 11. 2026`), as in Factorio's own changelog. Section names are indented two
spaces, bullets four, there are no tabs or trailing spaces, and a section is omitted when it is empty. The version is
always three numbers: a prerelease has no entry (see above).
| Commit | Section |
|---|---|
| `feat` | Features |
| `fix`, `revert` | Bugfixes |
| `perf` | Optimizations |
| `refactor` | Changes |
| breaking change | Changes, as `Breaking: ...` |
| `docs`, `style`, `test`, `build`, `ci`, `chore` | not listed |
A stable release writes the entry. Do not renumber, reorder or rewrite a released entry; a correction is a new commit
and a new release. The `0.1.0` entry was written by hand because the history before this scheme does not use
Conventional Commits.
## History
Tags from before this scheme (`Release`, `amd`, `text-hotfix`, and so on) are snapshots of the old workflow. They stay
in the repository for history but are not releases: the tooling matches only `v[0-9]*` tags. `v0.1.0` is the first
versioned release.
## Rules
- Do not edit a version number in a file, move a tag or reuse a version.
- Do not tag a branch other than `main` (or `next` for a release candidate).
- Do not release from a dirty working tree; the release commit contains `changelog.txt` and nothing else.
- Do not put game files, keys, builds or anything made from a dump in a release.

118
docs/workflow.md Normal file
View File

@@ -0,0 +1,118 @@
# Workflow: from report to release
How work moves through Hookshot, and how to decide what to work on next. It is written for people and for AI agents
alike (agents: [AGENTS.md](../AGENTS.md) sends you here). [CONTRIBUTING.md](../CONTRIBUTING.md) still holds the rules
that never bend, above all that no game files or anything made from them leave your PC.
The goal of this page is the order of work: with dozens of contributors and agents working at once, the project moves
fastest when everyone picks from the same priorities, claims before starting, and leaves a trail the next person can
follow.
## 1. Where work comes from
| Source | What lands there |
|---|---|
| GitHub issues (game problem, build problem, idea) | the record: version, hardware, place in the game, `hookshot_log.txt` |
| Tracking issues (`[Tracking] ...`) | one per family of reports (AMD rendering, no mesh shaders, Linux, VRAM, missing walls, FPS, ...) |
| Discord #bug-reporting (forum) | first reports, often without a log; turn the reproducible ones into issues |
| Discord #dev-work | claims, progress, review requests |
| Discord #dev-stuff | security scan results (Syntak): handled before anything else |
| CI | a red check on `main` is everyone's first problem |
## 2. Priority: what to work on first
Score a problem as **severity x reach x confidence** and work from the top. Write the three parts in the issue or the
claim, so others can see why it was picked. The scored, sorted list of open work is **[docs/worklist.md](worklist.md)**:
pick from its top, and add new problems there with their three parts.
### Severity
| Level | Weight | What it means |
|---|---|---|
| S0 | 8 | Security issue; crash, hang, device lost or a PC freeze; lost or corrupted saves; the builder cannot build |
| S1 | 5 | Cannot progress or cannot start: black screen, missing walls that block a dungeon, softlock, menus unusable |
| S2 | 3 | Wrong but playable: visual corruption, missing objects, audio out of sync, a large frame-rate loss (over 20%) |
| S3 | 1 | Polish: small visual issues, small frame-rate gains, settings, features |
### Reach: the systems people run
Reach is the share of players on the hardware the problem affects (1.0 when it affects everyone). Two numbers feed it,
and both change, so read them again rather than trusting old ones:
- **Who reports problems:** `python -I scripts/hardware_report.py` counts the GPU vendor, card class and OS in the
issue forms (it needs the `gh` CLI). As of 2026-11-10, of 399 issues that named their hardware: AMD 51%, NVIDIA 35%,
Intel 3.5%; by class RX 6000 16%, AMD APUs and handhelds 12%, RX 9000 10%, RTX 40 9%, RTX 50 8.5%, RTX 30 7%, cards
without mesh shaders 14.5%; Windows 57%, Linux and SteamOS 17%. This over-counts hardware where things break.
- **Who plays:** the [Steam Hardware Survey](https://store.steampowered.com/hwsurvey/) is the best public picture of
PC gaming hardware (vendor, card, OS). Use its current numbers for the installed base.
A problem that hits a large installed base, or that dominates the reports, ranks high either way.
### Confidence
| | Weight |
|---|---|
| Reproduced by you, or by a maintainer, with a log | 1.0 |
| Two or more reporters with logs | 0.7 |
| One report, or no log | 0.4 (ask for the log first) |
### Hardware tiers: stability across machines
A change must not break a higher tier to help a lower one.
| Tier | Hardware | What it means |
|---|---|---|
| 1 | NVIDIA RTX 20-50 and AMD RX 6000-9000, on Windows; the same on Linux (NVIDIA driver, Mesa RADV); Steam Deck | Must keep working: test here before merge, regressions are S0/S1 |
| 2 | AMD APUs and handhelds (ROG Ally, Legion Go, 780M/890M/8060S), Intel Arc | Fixed when reported, tested when someone with the hardware can |
| 3 | Cards without mesh shaders (GTX 10/16, RX 5000 and older) | Best effort through the fallback; never at a Tier 1 cost |
Today Tier 1 testing is mostly NVIDIA; AMD files half the reports. AMD stability is the largest gap, so an AMD Tier 1
problem with a log outranks an NVIDIA one of the same severity.
## 3. The lifecycle
1. **Triage.** Reproduce if you can. Find the tracking issue it belongs to, or open the issue (with the log). Score it
(section 2).
2. **Claim.** Before starting, say so in Discord #dev-work and on the issue: what, why (the score), and on what
hardware you can test. Search open PRs first: two people fixing the same thing is the most common waste here. Drop
a claim out loud when you stop.
3. **Fix.** A branch from current `main`, one topic. Find the cause before the fix: a log, a trace, a measurement.
4. **Test.**
- A game-free test that fails without the change (see AGENTS.md: tests read the source).
- `python -I -B tests/run_all.py`, all of it.
- In the game, on your hardware: what you did and what you saw, with numbers from `hookshot_log.txt` for performance.
- Add a row to [tested configurations](tested-configurations.md) for each run you measured.
- Say what you could not test (another OS, another vendor, real input devices).
5. **Pull request.** Open the description with why the PR is needed; then what changed, how it was tested, what was
not tested, and screenshots when it shows on screen. Link the issue and the tracking issue. Post the link in
#dev-work.
6. **Review.** Answer every comment in the PR. Rebase onto `main` before asking for review again: a PR that does not
merge cleanly waits.
7. **Merge and release.** Release branches are merged back into `main` right after a release, so that `main`'s builds
carry the release's fixes and the right version string (`git describe` only sees tags on `main`'s history).
8. **Verify.** Ask the reporter to retest on the release, then close the issue, or keep it open on the tracking issue
with what is left.
## 4. The security lane
Security comes first: a scan result in #dev-stuff is triaged before other work.
- Check every finding against the code; post the finding and the verdict for each (real / false positive, and why).
- Fix the real ones in a PR, marked as security in the title.
- Record false positives in the scanner as not exploitable, with the reason, so the next scan shows only new ones.
- Never paste secrets, tokens or the scanner's private links in public channels.
## 5. Keeping the board readable
Suggested labels (a maintainer creates them): `priority:P0` to `priority:P3` (from the score), `hw:nvidia`, `hw:amd`,
`hw:intel`, `hw:no-mesh-shaders`, `os:linux`, `os:steamdeck`, `needs-log`, `reproduced`, `claimed`. With them, the
next thing to work on is the top of `is:open label:reproduced -label:claimed sort:priority`.
## For agents
- Start here, then AGENTS.md, then CONTRIBUTING.md.
- Pick from the top of [docs/worklist.md](worklist.md); add a scored row there for a new problem you find.
- Prefer problems you can reproduce on the hardware you run on.
- Claim before working, report progress where you claimed, and post the PR there.
- Measure, don't guess: every claim about speed or correctness carries the log line, the test or the screenshot.
- Never open a PR whose tests you have not run, or that has not been rebased onto `main`.

57
docs/worklist.md Normal file
View File

@@ -0,0 +1,57 @@
# Work list
The prioritized list of open work, scored as [docs/workflow.md](workflow.md) describes: **score = severity weight x
reach x confidence** (S0 8, S1 5, S2 3, S3 1; reach 0-1 from `scripts/hardware_report.py` and the Steam survey;
confidence 1.0 reproduced with a log, 0.7 two or more reporters with logs, 0.4 one report or no log). Work from the
top of **Open**. A GitHub issue stays the full record; this list is the order.
How to use it:
- **Pick:** take the highest-scored row with no owner that you can reproduce on your hardware, claim it in Discord
#dev-work, and put your name in *Owner* in the same PR that starts the work (or a one-line PR).
- **Add:** a new report becomes a row in **Open** with its three parts filled in, so the score can be checked. Link the
issue or the Discord thread. One row per problem: if a row already covers it, add the link to that row and raise its
confidence instead of adding a second row.
- **Move:** when a PR merges, move the row to **Fixed, waiting for a release** with the PR; when a release ships it,
delete the row (the release notes are the record). A row nobody can reproduce for two weeks, with logs asked for,
goes to **Parked**.
- Keep each table sorted by score, highest first. Re-score when new logs or hardware numbers arrive.
Agents may add and re-score rows; a person reviews the change like any other PR.
## Open
| Score | Problem | S | Reach | Conf. | Owner | Links |
|---:|---|---|---:|---:|---|---|
| 3.0 | Shader programs that still do not translate (TLD multisample bit 78, LDG descriptor without `.E`, programs using the lowering scratch registers, uniform-addressed global loads) | S2 | 1.0 | 1.0 | | `--prepare-shaders` log; Legion Go 2 log in #bug-reporting |
| 2.1 | Ground and outdoor textures garble after a while of play and come back after a restart or going to the title; not the 4-8 GB over-budget case: on a 9070 XT 16 GB the textures sit at 5.3 of their 6.1 GB budget (VRAM 9.6 of 15.4 GB) with 287 texture retirements in the log; `RX_VK_ASTC_RGBA8=1` took effect (texture arrays 4x larger) and did not help. 9070 XT x2 (logs, v2.0.13), RTX 4080 (no log), RTX 5060 Ti (v2.0.5 log). Next: test whether retiring and re-making a texture brings back stale or wrong contents (force retirements with a small `RX_VK_VRAM_MB`) | S2 | 1.0 | 0.7 | | #993; #bug-reporting "texture glitch on 9070 XT", "Ground design is not loading properly", "Dodongo Cavern, floor texture glitch" |
| 2.1 | Camera turns in Kokiri Forest stutter: 46 fps minimum, dips to 38 (14700KF); goal is a 60 fps minimum | S2 | 1.0 | 0.7 | Joey-Magnus | Reddit report; #1762 (+13.5% in the village view; 57 of 60); next: the capture wait on the main thread |
| 2.0 | Antivirus flags HookshotBuilder.exe (Defender `Wacatac.H!ml` / `Wacatac.B!ml`, Chrome on v2.0.12): new unsigned exe, no reputation. Waiting on the SignPath open-source signing application; then sign in `windows-builder.yml` and publish the signed files | S1 | 0.57 | 0.7 | NotRexed (SignPath) | #1003, #525, #1620; checksums: #1307 |
| 1.6 | Stops after the first cutscene (Navi, Deku Tree): black screen, or the PC freezes or crashes, with no error in the logs; AMD (RX 9060 XT, RX 6700 XT, RX 6600, 5700G, Z1 Extreme) and one NVIDIA card (1080 Ti) reported; Linux too | S0 | 0.51 | 0.4 | | #bug-reporting "black screen after a first cutscene", "Crashes entire computer" (one log is from v2.0.7, the other not read yet: confidence goes up with a current log) |
| 1.2 | Link keeps playing a jump animation in dialogue after the Gohma fight (gameplay fine): cutscenes and dialogue run at the frame rate above a 60 cap since #1612 (v2.0.13); work-around: cap at 60 | S2 | 1.0 | 0.4 | | #1742 (#1726 closed into it); #bug-reporting "My link is jumping forever" |
| 1.2 | Ground looks smeared, without detail: Kokiri Forest / Lost Woods, and also Dodongo's Cave and Hyrule Field on RX 9070 XT (driver 26.10.2, 1440p, no mods; log received, clears after a restart) | S2 | 1.0 | 0.4 | | #bug-reporting "Ground design is not loading properly" |
| 1.2 | Trees float above the ground (Kakariko, Hyrule Field): reported as already fixed, check which change and that it shipped | S2 | 1.0 | 0.4 | | tracker #994 |
| 1.0 | Source checks fail after stream/descriptor admission merges: missing production stream_insert dependency and descriptor-pool fixture environment. CI-first; reproduced on main | S3 | 1.0 | 1.0 | Arkex-ai | [#1813](https://github.com/NotRexed/Hookshot/issues/1813), [claim](https://discord.com/channels/1568810484818378804/1569019786766188665/1569763504758464609) |
| 0.6 | Zora's Domain textures missing in the adult timeline (points to VRAM pressure, not the dump) | S2 | 0.3 | 0.7 | | #bug-reporting "Zora Domain textures missing?" |
| 0.5 | Sacred Forest Meadow fairy fountain grotto: floor and walls invisible (RTX 2070 Super) | S2 | 0.4 | 0.4 | | #bug-reporting thread |
| 0.4 | Voices in the wrong language | S3 | 1.0 | 0.4 | | #1460 (needs a save past Gohma) |
| 0.29 | Amd RX 580 (Polaris, no mesh shaders): hair renders white and bright, and the game crashes when Link moves; a second RX 580 8 GB user reports the same | S1 | 0.145 | 0.4 | | #bug-reporting "Amd rx 580 crashes" (no log yet) |
| 0.24 | Handheld with no keyboard (Asus ROG Xbox Ally X, controller only): fullscreen and the F1 options menu cannot be reached; no log yet | S1 | 0.12 | 0.4 | | #bug-reporting "Xinput devices cannot enter fullscreen or menu" |
| 0.12 | AMD RX 9070 XT: texture glitch that appears after some play (since v2.0.5); same as the tracked AMD texture artifact | S2 | 0.10 | 0.4 | | #bug-reporting "texture glitch on 9070 XT aris"; #57, #1006 |
| 0.11 | Intel GPU: black screen after loading a save (or on the Deku Tree), sound only, very low fps; three reports, no logs yet; tracked in the GitHub issue linked in #bug-reporting replies (#999) | S0 | 0.035 | 0.4 | | #bug-reporting "Intel G3E. Title and intro loads...", "intel arc b580 problems", "game gets stuck on a black screen when loading a save (intel gpu... gulp...)" |
| 0.11 | Crash at the door after the owl intro (Navi wakes Link, 2.0.13): access violation in `igvk64.dll` at start-up; GPU not given yet, reach uses Intel's 3.5% until it is | S0 | 0.035 | 0.4 | | #bug-reporting "2.0.13 Crash after you see the owl during intro cutscene" (5 logs) |
## Fixed, waiting for a release
| Problem | Fixed by | Notes |
|---|---|---|
| 1D surface loads (SULD.1D) stop 16-17 compute programs from translating; fits the green glow on a Legion Go 2 (890M) | #1447 | in `main`, not in v2.0.13 (log: #bug-reporting "Bugs 2.0.13 Dodaza") |
| Integer min/max with a wide immediate (IMNMX) does not translate | #1451 | in `main`, not in v2.0.13 (log: #bug-reporting "Bugs 2.0.13 Dodaza") |
| Insert menu purple with HDR10 on (v2.0.13 GDI overlay drawn into the HDR swapchain unconverted) | `main`'s ImGui overlay (`hdr10()`) | needs the release branch merged back into `main` |
## Parked
| Problem | Why parked | Links |
|---|---|---|
| | | |