mirror of
https://github.com/tangerineturtle1/NX2navi.git
synced 2026-11-18 04:49:57 -04:00
Land the shader and pipeline stutter pass, and the fixes it needed.
Large compute shaders that no cache holds are built on one background thread after the boot prebuild, smallest first, so their first dispatch no longer compiles them on the replay thread. NX2NAVI_BIG_BUILD_MAX_MB stops that builder once the process has grown by that much: on a 31 GB machine one of these shaders took the process up by 12.8 GB and the game stuttered while Windows paged. The builder takes the queue under its lock and copies each entry out before compiling it, so a shader queued meanwhile cannot free the arrays under it. The same change reports what pipeline-cache.bin gives back, and starts over when the file grows past NX2NAVI_PIPELINE_CACHE_MAX_MB. A tensor-core program is now tested before its shader is asked for, in both dispatch paths. It runs the stand-in while NX2NAVI_NATIVE_TENSOR is off, and translating it anyway recorded four AI-upscaler shaders of 0.87 to 4.4 million SPIR-V words that this game never dispatches natively. Process memory went from 29.3 GB to 24 GB and the multi-gigabyte builds during play are gone. The frame-time summary says which part a slow frame went to, and with NX2NAVI_REPLAY_TIME how the replay splits between draws, binds and transfers; NX2NAVI_SVC_SLOW_MS logs the kernel services that block, with the thread they blocked on. prof_report.py reads the map this build writes and adds the image base to a runtime Rip, so a profile resolves to function names instead of question marks. build_xlate.py applies the patches with GIT_CEILING_DIRECTORIES set to the build folder's parent. Inside a checkout git read 0021-prmt-byte-permute.patch's paths from the repository's top, skipped every file of it and still exited 0, so PRMT was missing from locally built translators and every byte permute was a no-op. A Skipped patch line now fails the build. The note in AmpereLowering.cs that said the translator emits nothing for PRMT is corrected.
This commit is contained in:
60
docs/protection-research.md
Normal file
60
docs/protection-research.md
Normal file
@@ -0,0 +1,60 @@
|
||||
# Can NX2navi be protected from copying, including by AI tools?
|
||||
|
||||
Research note, 2026-11-07. Not legal advice.
|
||||
|
||||
**Short answer.** Nothing stops a person with read access from copying code or pasting it into an AI tool. You can limit who gets access, set legal terms, and opt out of cooperative crawlers. Ideas are free to reuse; only patents cover them, and they are impractical here.
|
||||
|
||||
## Facts in this repo that change the answer
|
||||
|
||||
- `LICENSE` names **Valiantsin Dzerakh**, not chrg6. The local clone is shallow; its root commit is by that author, and the remote is `valentderah/totk-pc-recomp`, which the GitHub page showed as **public, MIT, about 5.5k stars and 459 forks**. Copies already made under MIT stay MIT. Only the author of a piece of code can relicense it, so chrg6 can relicense their own commits (23 of 24 here), not the base code.
|
||||
- `AGENTS.md` says "This repository is public"; update it once the decision is made.
|
||||
- `THIRD_PARTY_NOTICES.md` lists about 40 components under MIT, BSD, zlib, ISC and Apache-2.0. 44 of 112 `runtime/*.c` files carry copyright lines.
|
||||
|
||||
## 1. Technical limits
|
||||
|
||||
| Measure | What it does | Limit |
|
||||
|---|---|---|
|
||||
| Private repo | Strongest control. [GitHub's terms](https://docs.github.com/en/site-policy/github-terms/github-terms-of-service) treat private content as confidential, with narrow staff access | Anyone invited can clone and paste it anywhere |
|
||||
| GitHub AI settings | Since 24 Apr 2026, Copilot Free/Pro/Pro+ interaction data trains models unless you opt out at `github.com/settings/copilot/features`. Private content at rest is reportedly not used; code processed while Copilot is active in it is ([GitHub blog](https://github.blog/news-insights/company-news/updates-to-github-copilot-interaction-data-usage-policy/); details from secondary reports) | Copilot only. For public repos the terms (D.4) license your content to GitHub for AI training, and the opt-out excludes public repo content (J.3) |
|
||||
| robots.txt | You cannot edit github.com's. [GitHub's](https://github.com/robots.txt) lets GPTBot, ClaudeBot and Claude-User crawl with path limits such as `/copilot/`; only Bytespider is blocked. No per-repo opt-out | Only for sites you host. Voluntary |
|
||||
| Crawler tokens | [Anthropic](https://support.claude.com/en/articles/8896518): ClaudeBot (training), Claude-SearchBot, Claude-User; [OpenAI](https://developers.openai.com/api/docs/bots): GPTBot (training), OAI-SearchBot, ChatGPT-User (robots.txt "may not apply"); [Google-Extended](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers): Gemini training and grounding, not Search | Forward-looking; each bot named |
|
||||
|
||||
**Cannot be prevented:** a clone pasted into an AI tool, a local assistant reading files, a collaborator's own settings, a user-triggered fetch of a public URL, or any scraper that ignores robots.txt. Notices in `AGENTS.md` or file headers are signals, not controls.
|
||||
|
||||
## 2. Legal options
|
||||
|
||||
[17 U.S.C. 102(b)](https://www.law.cornell.edu/uscode/text/17/102): copyright never extends to "any idea, procedure, process, system, method of operation, concept, principle, or discovery". A clean rewrite of our approach is legal.
|
||||
|
||||
| Licence | Effect | Enforceability |
|
||||
|---|---|---|
|
||||
| MIT (now) | Anyone may copy, sell, relicense; must keep the notice | Nothing to enforce beyond the notice |
|
||||
| GPL / AGPL | Copiers must publish derived source under the same terms; AGPL covers network use. Does not stop copying | Enforceable by the holder; fits only if you want sharing |
|
||||
| PolyForm Noncommercial | Source-available; bars commercial use | Works as a copyright licence; not open source |
|
||||
| BSL 1.1 | Limits production use, then [converts to GPL-compatible](https://mariadb.com/bsl11/) within 4 years; "not an Open Source license" | Built for companies; poor fit |
|
||||
| All rights reserved | Default under copyright; fine while private | Strong in law, but no one else may use or contribute |
|
||||
| "No AI training" clause | Custom wording, no standard text; breaks the [Open Source Definition](https://opensource.org/osd) rule against field-of-use limits | Untested. US district courts found AI training fair use on the facts in 2025 (Bartz v. Anthropic, Kadrey v. Meta; [review](https://copyrightalliance.org/ai-copyright-case-developments-2025/)), and in [Doe v. GitHub](https://www.gibsondunn.com/ninth-circuit-clarifies-limits-of-dmca-liability-for-ai-generated-code/) (9th Cir., Sept 2026) only the contract claims survive. I found no appeals ruling on fair use |
|
||||
|
||||
Enforcement means finding a copy, proving copying and paying a lawyer. A DMCA takedown is the cheap tool for copies on GitHub.
|
||||
|
||||
**Patents** are the only protection for ideas. Reported costs: USPTO provisional $65 to $325, software drafting and filing about $8,000 to $15,000, the full path $15,000 to $30,000 or more. They need filing before public disclosure, and published upstream work is prior art. From general knowledge, not checked here: software eligibility after *Alice* is shaky. Not worth it.
|
||||
|
||||
## 3. What can and cannot be relicensed
|
||||
|
||||
**Cannot:** every row of `THIRD_PARTY_NOTICES.md`. This covers `tools/vendor/` (unmodified), Ryujinx's shader translator, `tools/xlate/` (its `LICENSE` is MIT, jhaizhou) and files adapted from MacNativeGames, LASRecomp and others. They stay MIT, zlib or BSD with their notices. zlib code (`runtime/nvn_tex.c`) must be marked if altered. Stricter terms can cover only our own additions; anyone can still take the MIT parts from their upstream projects.
|
||||
|
||||
**Translator:** pinned MIT Ryujinx code plus patches. Patch lines we wrote (nine of ten commits touching `tools/xlate/patches/` are chrg6's; per-patch authorship is not recorded) can carry our terms. The patched translator still ships under MIT with Ryujinx's notice, and the upstream stays free to everyone.
|
||||
|
||||
**Can, if the author agrees or it is chrg6's alone:** new files such as `runtime/widescreen.c` and `recomp/patches-v111.txt`, and the Ocarina port docs.
|
||||
|
||||
**Gap:** the TOTK-Ultrawide row is GPL with no URL or version; keep it data-only and record the source.
|
||||
|
||||
## 4. Recommendations, ranked
|
||||
|
||||
1. **Keep the repo private**, invite only trusted people, turn off the Copilot training opt-in in your account and ask collaborators to do the same. Free; the only real control.
|
||||
2. **Settle ownership with Dzerakh in writing**: who owns what, and whether a restrictive licence on the additions is acceptable. Without it you can relicense only your own work.
|
||||
3. **Choose a licence for our additions** (all rights reserved while private; PolyForm Noncommercial if shared), via per-file SPDX headers and a `LICENSE` that states the split. One short lawyer review.
|
||||
4. **Add a notice** in `README.md` reserving rights and stating that AI training is not permitted. A legal signal, not protection.
|
||||
5. **Add robots.txt** blocks for ClaudeBot, GPTBot and Google-Extended on any site you host.
|
||||
6. **Skip patents.** The practical moat is staying ahead.
|
||||
|
||||
Expect deterrence and a clean legal footing, not prevention.
|
||||
57
docs/shader-delivery-research.md
Normal file
57
docs/shader-delivery-research.md
Normal file
@@ -0,0 +1,57 @@
|
||||
# Shader delivery: Advanced Shader Delivery and Vulkan equivalents
|
||||
|
||||
Research only, 2026-11-07. No code was changed or built. Question: can Microsoft's Advanced Shader Delivery (ASD) cut first-run shader compile cost here?
|
||||
|
||||
**Verdict: no.** ASD is a Direct3D 12 feature, fed by a store, and its inputs are game-derived. A Vulkan equivalent already exists in this codebase. The remaining gap is cheap to close locally.
|
||||
|
||||
## 1. What ASD is
|
||||
|
||||
- A "Windows 11 Direct3D 12 technology" with three stages: create a State Object Database (SODB), test it by compiling a Precompiled Shader Database (PSDB) and registering it locally, deploy the SODB to a storefront ([overview](https://learn.microsoft.com/en-us/windows/win32/direct3d12/advanced-shader-delivery)).
|
||||
- The SODB is a SQLite file holding serialized PSO and state-object descriptions plus their DXIL bytecode, recorded from `CreateGraphicsPipelineState`, `CreateStateObject` and similar calls ([spec](https://microsoft.github.io/DirectX-Specs/d3d/StateObjectDatabase.html)). Recording uses `d3dconfig.exe` capture or a programmatic API; capture "adds significant overhead" ([Agility SDK 1.618](https://devblogs.microsoft.com/directx/agility-sdk-1-618/)).
|
||||
- The store compiles the SODB with each GPU vendor's compiler plugin (`D3D12StateObjectCompiler`) into a per-GPU PSDB and ships it with the download; Windows refreshes it after driver updates ([intro](https://devblogs.microsoft.com/directx/introducing-advanced-shader-delivery/)). The D3D12 runtime finds it inside the game process by executable path through the [registration API](https://microsoft.github.io/DirectX-Specs/d3d/ShaderCacheRegistrationAPI.html). A PSDB must match the adapter family and compiler ABI.
|
||||
- Requirements: Windows 11 24H2 (build 26100.1000+) and Agility SDK 1.619.5+ ([adoption guide](https://github.com/microsoft/DirectX-Graphics-Samples/blob/master/Samples/Desktop/D3D12AdvancedShaderDelivery/readme.md)). Submission is through Xbox Partner Center with an application identity ([GDC 2026](https://devblogs.microsoft.com/directx/advanced-shader-delivery-whats-new-at-gdc-2026/)).
|
||||
- Hardware per the [21 Oct 2026 post](https://devblogs.microsoft.com/directx/advanced-shader-delivery-is-available-now-across-hardware): AMD RDNA 1-4, Intel Arc B-series and Core Ultra 2/3, NVIDIA RTX, Qualcomm Snapdragon X2, each with a minimum driver. Delivery is through the Microsoft Store and Xbox PC app; Steam and Epic are "coming soon", undated.
|
||||
- No Microsoft page I read mentions Vulkan.
|
||||
|
||||
## 2. Can it apply here?
|
||||
|
||||
**No. ASD needs D3D12.** The SODB records D3D12 create calls and DXIL; the PSDB is looked up by the D3D12 runtime. A Vulkan process never touches either.
|
||||
|
||||
Current flow (from the top comments of `runtime/gfx_shader.c` and `runtime/vk_pipeline.c`):
|
||||
- An NVN program (control section, code, c1, translator answers) goes to `nx2navi_xlate.exe --serve` children and comes back as SPIR-V plus reflection, cached in memory and under `shader-cache/` (packed into `shader-cache.pack`). The key includes the translator identity, so any translator change invalidates every entry. `in/<input id>.in` files allow retranslation without guest memory.
|
||||
- Play records `shader-answers.bin` and `shader-programs.bin`. `pipeline-keys.bin` records every built pipeline, shaders named by input id, independent of the translator.
|
||||
- Pipelines use `VK_EXT_graphics_pipeline_library`: stage libraries, a fast link on a miss, an optimised pipeline built on a background thread. `pipeline-cache.bin` is dropped when vendor, device or cache UUID differ.
|
||||
- Boot (`vk_pipeline_prewarm`) retranslates and rebuilds what the key file names. The installer runs `nx2navi.exe --prepare-shaders`, which warms the driver's disk cache.
|
||||
- A comment in `runtime/boot.c` records a cold start (new translator or driver): 2,132 shaders translated and 1,579 pipelines built in 9.8 s. The guest waits at most `NX2NAVI_PREWARM_WAIT_MS` (default 3000), then the work continues in the background.
|
||||
- Compute shaders are translated and built synchronously on the replay thread, because "a skipped dispatch can change what the game computes". Graphics misses are asynchronous and the draw is skipped. So wrong lighting early in a cold start more plausibly comes from skipped graphics draws than from compute. I did not reproduce or measure it.
|
||||
|
||||
**A D3D12 back end** would replace about 25,800 lines (28 `vk_*.c` files plus headers) using push descriptors, mesh shaders, pipeline libraries and dynamic state 3, and the translator would have to emit DXIL, not SPIR-V. My estimate is several person-months. It would still not reach ASD, because delivery needs a store submission (section 3). The local "compile a SODB, register the PSDB" test path runs the same vendor compiler on the player's machine, so it gains nothing over the existing prepare pass.
|
||||
|
||||
## 3. Content boundary
|
||||
|
||||
`AGENTS.md` forbids anything derived from game data in the repository. Shaders, SPIR-V, DXIL, SODBs, PSDBs and `pipeline-cache.bin` are all derived from game code. `check_content.py` also blocks `.pack` files.
|
||||
|
||||
- **Ruled out:** shipping or uploading any shader database, SODB, PSDB, SPIR-V or driver cache. ASD's deploy stage is exactly an upload to a store.
|
||||
- **Still possible:** anything built on the player's machine from the player's own files. The installer's prepare step does this today, and everything stays in ignored folders.
|
||||
- **Existing precedent:** the installer bundles a variant list (shader hash plus translator answers, no code), outside the repository. Keep any new shared data to that level. Whether a hash counts as "derived" is the owner's call.
|
||||
|
||||
## 4. Vulkan-side options
|
||||
|
||||
| Option | State here | Verdict |
|
||||
|---|---|---|
|
||||
| `VK_EXT_graphics_pipeline_library` | In use (`vk_device.c`, `vk_pipeline.c`, `NX2NAVI_GPL`) | Done |
|
||||
| Driver disk cache | Used; a library built again on AMD takes 0.1 ms instead of 55 ms (`vk_pipeline.c`) | Done |
|
||||
| Fossilize-style capture | `pipeline-keys.bin`, `in/*.in` and the answer files play that role, with the translator in front | Done. [Fossilize](https://github.com/ValveSoftware/Fossilize) replays on any device. [Steam pre-caching](https://www.phoronix.com/news/Steam-Vulkan-Shader-Pre-Cache) needs Steam and uploaded data: unusable |
|
||||
| [`VK_KHR_pipeline_binary`](https://www.khronos.org/blog/bringing-explicit-pipeline-caching-control-to-vulkan) | Not used | Per-pipeline blobs with your own eviction, still per-driver. 52.8% of Windows devices expose it ([database](https://vulkan.gpuinfo.org/listextensions.php?platform=windows&extension=VK_KHR_pipeline_binary)); Mesa has it ([RADV](https://www.phoronix.com/news/RADV-Pipeline-Binary-Merged), [ANV](https://phoronix.com/news/Intel-ANV-Pipeline-Binary)). Per-vendor Windows support unconfirmed. Optional |
|
||||
| [`VK_EXT_shader_object`](https://docs.vulkan.org/features/latest/features/proposals/VK_EXT_shader_object.html) | Not used | Skip: replaces pipelines, conflicts with the library design, binaries are device-bound |
|
||||
| First-run "prepare shaders" pass | Installer runs it; boot prewarm capped at 3 s | The gap to close |
|
||||
|
||||
After a translator change the dominant cost is likely translation (CPU, translator children), which neither pipeline binaries nor ASD-style delivery can speed up. The driver compile only repeats for shaders whose SPIR-V changed. Measure the split first; the prewarm logs report both counts.
|
||||
|
||||
## 5. Recommendation
|
||||
|
||||
ASD is not feasible: D3D12-only, store-gated, built from game-derived data. The practical steps:
|
||||
|
||||
1. **Blocking "Preparing shaders" step on a cold cache (1-3 days).** When the translator identity, GPU or driver changes, run the existing prewarm to completion before the guest starts (the `NX2NAVI_PREWARM_WAIT_MS=0` behaviour), with progress shown. The `boot.c` figure is about 10 s, once. This removes the wrong-looking start.
|
||||
2. **Re-run the library prepare pass after a driver change (2-4 days).** `lb_warm_prepared` and `--warm-shaders` exist, and the skip list already keys on vendor:device:driverVersion. Triggering the pass when `cache_load` drops a stale cache is the local counterpart of ASD's driver-update handling. I found no such trigger in `runtime/`; confirm before building.
|
||||
3. **Measure, then decide on `VK_KHR_pipeline_binary` (1-2 weeks, low priority).** Prototype it only if the measurement points at the pipeline or driver cache. It could let libraries persist without the roughly 240 MB cache growth `vk_pipeline.c` avoids today.
|
||||
Reference in New Issue
Block a user