diff --git a/docs/launcher.md b/docs/launcher.md index 5cc12655..05470cc5 100644 --- a/docs/launcher.md +++ b/docs/launcher.md @@ -55,6 +55,18 @@ Customize Launcher (on the home screen) makes the look the player's own: it is silent until the player's game has been read. Moving the cursor makes no sound. They are played from memory (PlaySound on Windows; `pw-play`, `paplay` or `aplay` elsewhere). +- Add to Steam (Windows): the game in the player's Steam library as *The Legend of Zelda Ocarina of Time*, started + from there without the launcher's window (`launcher/steam.py`). Its pictures are made from the player's game when + the row is selected (`launcher/gameart.py`): Navi as the game's HUD draws her for the icon, the title screen's logo, + and a frame of the first title movie (the moonlit field) as the background, the cover and the wide tile, the movie + decoded with the toolchain's `dav1d.exe` and the logo with the runtime's own ASTC decoder built with its clang. They + go into each signed-in account's `config/grid`, the shortcut into its `shortcuts.vdf` (other shortcuts kept as they + are, ours updated when it is there; the file before the first change kept as `shortcuts.vdf.nx2navi-backup`). + Steam reads that file only when it starts and writes it back when it closes, so a running Steam is asked to close + first and started again after. The shortcut runs the launcher's `pythonw.exe` on `data/steam_play.pyw`, which + starts the launcher of `app/current.txt` with `--play`: the game starts as Play starts it, and the process waits + for it, so Steam counts the game as running while it is. + The choices are `theme`, `background`, `background_dim`, `animations` and `sounds` in `data/launcher.json`. Navi (`art.NAVI`, 22 x 21 pixels, drawn in whole pixels at every size) is also the icon of `NX2navi.exe`, @@ -66,7 +78,7 @@ Screens (`launcher/ui/app.py`): first run (dump, install folder, Desktop Shortcu Update Channel, Begin Installation); the steps of an install, update or rebuild with their progress, the build log's last lines and Cancel, and after a failure Try Again and Copy Report; home (Play, Updates, Game Folder, Saves, Rebuild the Game, Customize Launcher, Settings); customize (Color Scheme, Background, with a picture set Background -Dimming and Remove Background, Animations, Sounds, Back); saves (Export Save, Add Log and Settings, Export To, Import a Save, Import a Save Folder, +Dimming and Remove Background, Animations, Sounds, Add to Steam on Windows, Back); saves (Export Save, Add Log and Settings, Export To, Import a Save, Import a Save Folder, Open Saves Folder, Back); settings (update check, Install Updates Automatically, Hide Launcher While Playing, Keep Build Cache, Update Channel, the folders, Open Logs Folder, Uninstall, Back); uninstall (keep or remove the built game). A screen with more rows than the window holds draws them closer together (`fit_rows`). diff --git a/docs/maxwell.md b/docs/maxwell.md index 8c0a4b01..4a0abaeb 100644 --- a/docs/maxwell.md +++ b/docs/maxwell.md @@ -75,7 +75,7 @@ Source shorthands: | 3.10 | **BFE / BFI fields past bit 31** (as PTX `bfe` / `bfi`): a field that runs past bit 31 is cut there; one that starts past it extracts 0 (signed: bit 31 repeated) and inserts nothing. Terrain vertex shaders (42 variants) have taken the position from a register, `((2y + x) * 8) & 0xff` with 5-bit x and y, so up to 248. | `OpBitField*Extract` / `OpBitFieldInsert` undefined when offset or offset + count exceeds 32; AMD's bitfield instructions take offset and width modulo 32. | The translator passes the fields through. Terrain walls and ground went missing on the Steam Deck and a Linux RX 570 (RADV). Ours: clamped where the field is not a constant inside the word (translator patch 0012). RX 9070 on Windows drew the terrain the same before and after. | PTX ISA `bfe`, `bfi` | | 3.11 | **SHF by 0** gives the unshifted word (C for `SHF.L`, A for `SHF.R`). | The translator's funnel shifts the other word by 32: undefined in SPIR-V; AMD shifts take the low 5 bits and give A \| C. | Gives terrain gaps at a distance. Ours: a select on a zero amount (translator patch 0013). In the shaders checked, every register SHF pairs its word with 0, so the result was right already. | — | | 3.12 | **Registers read before any write** on a path (paths the structurizer adds included) have no defined value in the game's code; the NVIDIA driver has been reported to give 0. | An `OpVariable` without an initializer loads an undefined value; Mesa folds undefined values freely. | Ours: every function-local variable starts at 0 (`OpConstantNull`, job flag `zero_locals`, translator patch 0016) on every driver; `NX2NAVI_ZERO_LOCALS=0` turns it off. On NVIDIA an undefined read takes another warp lane's value: Ocarina's clustered-light loop lit its floors in 4x8-pixel tiles (one warp each) and darkened Link. It was off while DLAA was on, because zeroing moves the SPIR-V hash of the game's FXAA shader that `vk_dlaa.c` `vk_dlaa_skip_draw` matches (`FXAA_FS` holds the zeroed hash too); with it off the tiles showed under DLSS and DLAA, and the shader cache keys, which include the flag, changed with the upscaler, so shaders prepared with one setting missed under the other. Checked on an RTX 5090: zeroed, DLAA keeps its camera motion (`NX2NAVI_MV_CHECK`) and the floor is clean. | — | -| 3.13 | **NaN, infinity and signed zero follow IEEE 754**, and the game's code leans on them: its compiler writes an unordered comparison as `x > c \|\| isnan(x)`, a normalisation of a zero-length vector gives an infinity that a later test catches, and integer masks (0xffffffff) pass through float moves and selects. | SPIR-V without the `SignedZeroInfNanPreserve` execution mode lets the driver assume that no float is NaN or infinite. AMD's compilers act on it; NVIDIA's driver keeps the IEEE results. | The isnan tests of the game and of the translator (patches 0027, 0031) are only safe with the mode. On an RX 9070 XT a select between 1 and the bits 0xffffffff, converted as a signed integer, gave 2^32 instead of -1 and blew shadow casters up (issue #156); RX 6000 cards drew black and white blocks on Navi and on Link next to her (#41, #110); AMD GPUs drew Navi's sparkles without her orb. Ours: `vk_pipeline.c` `spirv_nan_preserve` adds the mode for 32-bit floats (16 and 64 where the module declares them and the driver can) when the module is created, on every driver except NVIDIA's (`NX2NAVI_VK_NAN_PRESERVE`). | VKSPEC (SPIR-V environment, precision and operation of SPIR-V instructions) | +| 3.13 | **NaN, infinity and signed zero follow IEEE 754**, and the game's code leans on them: its compiler writes an unordered comparison as `x > c \|\| isnan(x)`, a normalisation of a zero-length vector gives an infinity that a later test catches, and integer masks (0xffffffff) pass through float moves and selects. | SPIR-V without the `SignedZeroInfNanPreserve` execution mode lets the driver assume that no float is NaN or infinite. AMD's compilers act on it; NVIDIA's driver keeps the IEEE results. | The isnan tests of the game and of the translator (patches 0027, 0031) are only safe with the mode. On an RX 9070 XT a select between 1 and the bits 0xffffffff, converted as a signed integer, gave 2^32 instead of -1 and blew shadow casters up (issue #156); RX 6000 cards drew black and white blocks on Navi and on Link next to her (#41, #110); AMD GPUs drew Navi's sparkles without her orb. Ours: `vk_pipeline.c` `spirv_nan_preserve` adds the mode for 32-bit floats (16 and 64 where the module declares them and the driver can) when the module is created, on every driver except NVIDIA's (`NX2NAVI_VK_NAN_PRESERVE`). On AMD's Windows driver for RDNA 1 and 2 ("AMD proprietary shader compiler", `vk_caps.amd_sc`), where the black and white blocks were reported, `spirv_nan_bits` also turns every 32-bit `OpIsNan` into an integer test of its bits and a float select with a NaN-bits constant into a select of the bits, which no float assumption can change (`NX2NAVI_VK_NAN_BITS`). | VKSPEC (SPIR-V environment, precision and operation of SPIR-V instructions) | ## 4. Memory model diff --git a/docs/releases/v0.7.2.md b/docs/releases/v0.7.2.md new file mode 100644 index 00000000..23311b26 --- /dev/null +++ b/docs/releases/v0.7.2.md @@ -0,0 +1,35 @@ +Draft notes for the release after 0.7.1. The version number is not decided yet. + +## Faster + +- **AMD cards: about a fifth more frames in open areas, and more even frame times.** The game asks the video card + which objects are visible, about a hundred questions a frame in Hyrule Field, and waits for a frame to be finished + on the card before it lets the next one start. The answers were fetched one call at a time, on the thread that also + reports "finished", and on AMD's Windows driver each of those calls goes into the kernel: every frame was reported + finished about 4 ms late, with every thread of the game waiting. They are now fetched a few hundred at a time. + Standing in Hyrule Field on an RX 7900 XT (driver 26.10.1, 3440x1440, FSR): 53 fps instead of 45, and the longest + frame of a minute 46 ms instead of 55 (`oq_batch = 0` in `nx2navi.ini` brings the old readback back). +- **No more frames drawn one present late.** The game signals from the CPU when a frame's data is written, and its + render thread submits the frame's commands without waiting for that signal. When the signal came a millisecond or + two after the submit call, in about every other frame in Hyrule Field, the frame was set aside and drawn at its + present, and frames of 23 and 15 ms took turns. The submit call now waits for the signal, 10 ms at most + (`gate_wait_ms`): 54.5 fps instead of 53 in the same place and the longest frame 33 ms instead of 46; with + `vk_recorder = 1`, which AMD cards do not have on by default, 58.6 fps. + +## New + +- **Add to Steam** in Customize Launcher (Windows): the game joins your Steam library as *The Legend of Zelda Ocarina + of Time*, with Navi as its icon, the title screen's moonlit field as its background and cover, and the game's logo, + all made from your own game files when you select the row. Playing it from Steam starts the game straight away, + without the launcher. Steam closes for a moment while the shortcut is written and starts again with it. + +## Fixed + +- Black and white blocks on AMD RX 5000 and RX 6000 cards: on Navi and on Link next to her, on bright floors near her, + and on the bright outdoors seen through a door or window from inside a house. Every float colour a shader writes is + checked for NaN, so that one bad pixel cannot spread through the blur and bloom passes; those checks, and the + game's own NaN tests in nearly nine of ten shaders, are float comparisons that a shader compiler may decide can + never be true. The Windows driver for these cards uses AMD's older shader compiler (the log reads "AMD proprietary + shader compiler"); with it each NaN test is now made on the bits of the number instead, which no compiler can drop. + It is on by default only with that driver (`vk_nan_bits`); the first start after the update compiles the pipelines + again. Untested on those cards before this release: please report whether the blocks are gone. diff --git a/launcher/gameart.py b/launcher/gameart.py new file mode 100644 index 00000000..bb3c1623 --- /dev/null +++ b/launcher/gameart.py @@ -0,0 +1,358 @@ +"""The pictures of the game's Steam shortcut (launcher/steam.py), made from the player's own game: nothing ships. + + icon Navi as the game's HUD draws her (HUDNaviIcon_Body_00 over the glow HUDNaviIcon_Flare_00), 128 x 128 + logo the title screen's logo with the Master Sword and the Hylian Shield (Logo_USEU_00), cut to its outline + hero the title screen's moonlit field, a frame of its first movie (Event/Movie/Dm_Title_0005.webm), 1920 x 620 + portrait that frame with the logo over it, 600 x 900 (the library's cover) + wide the same at 920 x 430 (the library's wide tile) + +The textures come from the interface archive UI/Layout2Archive/Title...bntx.zs as runtime/menu_assets.c reads them: +a BNTX's BRTI records give format, size, block height and data; the level is block linear in the Switch 2 GOB layout +(runtime/tex_deswizzle.c tex_bl_copy); BC4 masks become white with the mask as alpha; ASTC 4x4 goes through the +runtime's own decoder, runtime/tex_astc.c, built here into a library with the toolchain's clang (the launcher builds +the game with it, so it is on every PC that has a game to add). The movie is WebM with AV1 video: the frames from the +key frame before TITLE_AT up to it go into an IVF file for the toolchain's dav1d.exe, whose last picture (8-bit 4:2:0, +limited range BT.709) becomes the background. Every picture is written as PNG with art.png_bytes; nothing here needs +an imaging package. +""" +import ctypes +import hashlib +import os +import struct +import subprocess +import tempfile +from pathlib import Path + +import numpy as np + +from . import gamefiles + +TITLE_ARCHIVE = 'UI/Layout2Archive/Title.Product.100.Nin_Ounce_NVN2.bntx.zs' +COMMON_ARCHIVE = 'UI/Layout2Archive/Common.Product.100.Nin_Ounce_NVN2.bntx.zs' +LOGO, NAVI, NAVI_GLOW = 'Logo_USEU_00', 'HUDNaviIcon_Body_00', 'HUDNaviIcon_Flare_00' +TITLE_MOVIE = 'Event/Movie/Dm_Title_0005.webm' +TITLE_AT = 12.0 # seconds into it: the field under the full moon, as the title screen shows it first +KIND_BC4, KIND_ASTC4 = 0x1d, 0x2d +NO_WINDOW = 0x08000000 if os.name == 'nt' else 0 + + +class ArtError(Exception): + pass + + +# ---------------------------------------------------------------- BNTX + +def _u16(b, at): return struct.unpack_from(' b + for i in range(6): + pal[..., 2 + i] = np.where(big, ((6 - i) * a + (i + 1) * b) // 7, 0) + for i in range(4): + pal[..., 2 + i] = np.where(big, pal[..., 2 + i], ((4 - i) * a + (i + 1) * b) // 5) + pal[..., 6] = np.where(big, pal[..., 6], 0) + pal[..., 7] = np.where(big, pal[..., 7], 255) + bits = np.zeros((hb, wb), dtype=np.uint64) + for i in range(6): + bits |= blocks[..., 2 + i].astype(np.uint64) << np.uint64(8 * i) + alpha = np.zeros((hb * 4, wb * 4), dtype=np.uint8) + for i in range(16): + sel = ((bits >> np.uint64(3 * i)) & np.uint64(7)).astype(np.intp) + alpha[i // 4::4, i % 4::4] = np.take_along_axis(pal, sel[..., None], 2)[..., 0] + rgba = np.full((h, w, 4), 255, dtype=np.uint8) + rgba[..., 3] = alpha[:h, :w] + return rgba + + +def astc_library(layout): + """runtime/tex_astc.c of this release as a library in data/cache, built once per source with the toolchain.""" + src = layout.app / 'runtime' / 'tex_astc.c' + if not src.is_file(): + raise ArtError(f'{src} is missing') + tag = hashlib.sha256(src.read_bytes() + (layout.app / 'runtime' / 'tex.h').read_bytes()).hexdigest()[:12] + dll = layout.cache / f'astc-{tag}.dll' + if not dll.is_file(): + clang = layout.msys / 'bin' / 'clang.exe' + if not clang.is_file(): + raise ArtError('the build tools are not installed (Rebuild the Game installs them)') + env = dict(os.environ, PATH=str(layout.msys / 'bin') + os.pathsep + os.environ.get('PATH', '')) + tmp = dll.with_name(dll.name + '.tmp') + r = subprocess.run([str(clang), '-shared', '-O2', '-static', '-w', '-I', str(layout.app / 'runtime'), + '-I', str(layout.msys / 'include' / 'vma'), str(src), '-o', str(tmp)], + capture_output=True, text=True, env=env, creationflags=NO_WINDOW) + if r.returncode or not tmp.is_file(): + raise ArtError(f'the texture decoder did not build: {(r.stderr or r.stdout).strip()[-300:]}') + os.replace(tmp, dll) + lib = ctypes.CDLL(str(dll)) + lib.astc_decode_ldr.restype = None + lib.astc_decode_ldr.argtypes = [ctypes.c_char_p, ctypes.c_uint32, ctypes.c_uint32, ctypes.c_uint32, + ctypes.c_uint32, ctypes.c_bool, ctypes.c_void_p] + return lib + + +def texture(bntx, name, lib): + """The named texture as an HxWx4 uint8 array (BC4 or ASTC 4x4), or None when the file has no such texture.""" + for t in bntx_textures(bntx): + if t['name'] != name: + continue + kind = t['fmt'] >> 8 + w, h = t['w'], t['h'] + wb, hb = (w + 3) // 4, (h + 3) // 4 + if kind == KIND_BC4: + return bc4(deswizzle(t, bntx, wb * 8, hb), wb, hb, w, h) + if kind == KIND_ASTC4: + lin = deswizzle(t, bntx, wb * 16, hb) + rgba = np.zeros((h, w, 4), dtype=np.uint8) + lib.astc_decode_ldr(lin.tobytes(), w, h, 4, 4, (t['fmt'] & 0xff) == 6, rgba.ctypes.data) + return rgba + raise ArtError(f'{name} has a texture format this launcher does not read ({t["fmt"]:#x})') + return None + + +# ---------------------------------------------------------------- the title movie + +_MASTERS = {0x18538067, 0x1F43B675, 0xA0, 0x1654AE6B, 0xAE, 0xE0} # Segment, Cluster, BlockGroup, Tracks, TrackEntry, Video + + +def _vint(b, at, keep_marker): + first = b[at] + n = 1 + while n <= 8 and not first & (0x80 >> (n - 1)): + n += 1 + if n > 8: + raise ArtError('the movie is not WebM') + v = first if keep_marker else first & (0xff >> n) + for i in range(1, n): + v = v << 8 | b[at + i] + return v, n + + +def webm_frames(b): + """(time in ms, key frame, data) of every block of a single-track WebM, and the video's width and height.""" + out, info = [], {} + + def walk(at, end, cluster): + while at < end: + eid, n1 = _vint(b, at, True) + size, n2 = _vint(b, at + n1, False) + body = at + n1 + n2 + if size == (1 << (7 * n2)) - 1: # unknown size: to the end of the parent + size = end - body + if eid in _MASTERS: + walk(body, min(body + size, end), cluster) + elif eid == 0xE7: + cluster[0] = int.from_bytes(b[body:body + size], 'big') + elif eid == 0xB0: + info['w'] = int.from_bytes(b[body:body + size], 'big') + elif eid == 0xBA: + info['h'] = int.from_bytes(b[body:body + size], 'big') + elif eid in (0xA3, 0xA1): # SimpleBlock (its flags say key frame), Block + _track, n = _vint(b, body, False) + rel = struct.unpack_from('>h', b, body + n)[0] + key = eid == 0xA3 and bool(b[body + n + 2] & 0x80) + out.append((cluster[0] + rel, key, b[body + n + 3:body + size])) + at = body + size + + walk(0, len(b), [0]) + if not out or 'w' not in info or 'h' not in info: + raise ArtError('the title movie has no video frames') + return out, info['w'], info['h'] + + +def movie_frame(layout, data, at_s, work): + """The picture of the movie `data` at at_s seconds as an HxWx3 uint8 array, decoded by the toolchain's dav1d.""" + dav1d = layout.msys / 'bin' / 'dav1d.exe' + if not dav1d.is_file(): + raise ArtError('the build tools are not installed (Rebuild the Game installs them)') + frames, w, h = webm_frames(data) + upto = [f for f in frames if f[0] <= at_s * 1000] or frames[:1] + k0 = max((i for i, f in enumerate(upto) if f[1]), default=0) + sel = upto[k0:] + ivf, y4m = Path(work) / 'title.ivf', Path(work) / 'title.y4m' + with open(ivf, 'wb') as f: + f.write(b'DKIF' + struct.pack('= 2 or fx >= 2: + fy, fx = max(fy, 1), max(fx, 1) + H, W = a.shape[0] // fy * fy, a.shape[1] // fx * fx + a = a[:H, :W].reshape(H // fy, fy, W // fx, fx, -1).mean(axis=(1, 3)) + ys = (np.arange(h) + 0.5) * a.shape[0] / h - 0.5 + xs = (np.arange(w) + 0.5) * a.shape[1] / w - 0.5 + y0 = np.clip(np.floor(ys).astype(int), 0, a.shape[0] - 1) + x0 = np.clip(np.floor(xs).astype(int), 0, a.shape[1] - 1) + y1, x1 = np.minimum(y0 + 1, a.shape[0] - 1), np.minimum(x0 + 1, a.shape[1] - 1) + wy = np.clip(ys - y0, 0, 1)[:, None, None] + wx = np.clip(xs - x0, 0, 1)[None, :, None] + top = a[y0][:, x0] * (1 - wx) + a[y0][:, x1] * wx + bottom = a[y1][:, x0] * (1 - wx) + a[y1][:, x1] * wx + return top * (1 - wy) + bottom * wy + + +def premultiplied_resize(rgba, w, h): + """Resize with the colours weighted by alpha, so transparent texels add no dark fringe; uint8 straight alpha out.""" + a = rgba.astype(np.float32) + a[..., :3] *= a[..., 3:4] / 255 + r = resize(a, w, h) + r[..., :3] /= np.maximum(r[..., 3:4], 1e-3) / 255 + return np.clip(r + 0.5, 0, 255).astype(np.uint8) + + +def trim(rgba, pad=8): + """rgba cut to the box of its visible texels, pad texels around.""" + ys, xs = np.nonzero(rgba[..., 3] > 4) + if not len(ys): + return rgba + y0, y1 = max(ys.min() - pad, 0), min(ys.max() + 1 + pad, rgba.shape[0]) + x0, x1 = max(xs.min() - pad, 0), min(xs.max() + 1 + pad, rgba.shape[1]) + return rgba[y0:y1, x0:x1] + + +def place(dst, src, x, y): + """src (uint8 RGBA, straight alpha) over dst (uint8 RGB) at x, y.""" + h, w = src.shape[:2] + s = src.astype(np.float32) + d = dst[y:y + h, x:x + w].astype(np.float32) + al = s[..., 3:4] / 255 + dst[y:y + h, x:x + w] = np.clip(d * (1 - al) + s[..., :3] * al + 0.5, 0, 255).astype(np.uint8) + + +def crop(frame, w, h, cx, cy): + """The largest w:h part of frame centred near (cx, cy) as fractions, scaled to w x h (uint8 RGB).""" + H, W = frame.shape[:2] + if W * h > H * w: + ch, cw = H, H * w // h + else: + cw, ch = W, W * h // w + x0 = int(np.clip(cx * W - cw / 2, 0, W - cw)) + y0 = int(np.clip(cy * H - ch / 2, 0, H - ch)) + return np.clip(resize(frame[y0:y0 + ch, x0:x0 + cw], w, h) + 0.5, 0, 255).astype(np.uint8) + + +def with_logo(bg, logo, width_share, cy): + """bg (RGB) with the logo scaled to width_share of its width, centred at height cy (a fraction).""" + H, W = bg.shape[:2] + lw = int(W * width_share) + lh = int(lw * logo.shape[0] / logo.shape[1]) + if lh > H * 0.8: + lh = int(H * 0.8) + lw = int(lh * logo.shape[1] / logo.shape[0]) + small = premultiplied_resize(logo, lw, lh) + out = bg.copy() + place(out, small, (W - lw) // 2, int(np.clip(cy * H - lh / 2, 0, H - lh))) + return out + + +def navi_icon(body, glow, size=128): + """Navi's HUD icon on her glow, size x size RGBA: the glow tinted the pale blue of her light.""" + out = np.zeros((size, size, 4), dtype=np.float32) + g = resize(glow, size, size) + g[..., :3] = np.array([200, 228, 255], dtype=np.float32) + g[..., 3] *= 0.85 + b = resize(body, int(size * 0.8), int(size * 0.8)) + for layer, at in ((g, 0), (b, (size - b.shape[0]) // 2)): + h, w = layer.shape[:2] + d = out[at:at + h, at:at + w] + sa, da = layer[..., 3:4] / 255, d[..., 3:4] / 255 + oa = sa + da * (1 - sa) + d[..., :3] = (layer[..., :3] * sa + d[..., :3] * da * (1 - sa)) / np.maximum(oa, 1e-6) + d[..., 3:4] = oa * 255 + return np.clip(out + 0.5, 0, 255).astype(np.uint8) + + +def rgba_of(rgb): + return np.concatenate([rgb, np.full(rgb.shape[:2] + (1,), 255, dtype=np.uint8)], -1) + + +def make(layout, game_dir='', dump=''): + """{name: HxWx4 uint8} for icon, logo, hero, portrait and wide, from the player's game. ArtError when the game's + files or the tools to read them are not there.""" + romfs = gamefiles.open_romfs(layout, game_dir, dump) + if romfs is None: + raise ArtError('the game\'s files were not found (the game folder\'s romfs, or the dump)') + zs = gamefiles.ZsReader(romfs) + title, common = zs.read(TITLE_ARCHIVE), zs.read(COMMON_ARCHIVE) + movie = romfs.read(TITLE_MOVIE) + if not title or not common or not movie: + raise ArtError('the game\'s title screen files were not found') + lib = astc_library(layout) + logo = texture(title, LOGO, lib) + body, glow = texture(common, NAVI, lib), texture(common, NAVI_GLOW, lib) + if logo is None or body is None or glow is None: + raise ArtError('the game\'s logo or Navi\'s icon was not found') + logo = trim(logo) + with tempfile.TemporaryDirectory(dir=str(layout.cache)) as work: + frame = movie_frame(layout, movie, TITLE_AT, work) + # the moon sits right of the middle and high: every cut keeps it, and the hero keeps the horizon too + hero = crop(frame, 1920, 620, 0.5, 0.42) + portrait = with_logo(crop(frame, 600, 900, 0.68, 0.5), logo, 0.88, 0.70) + wide = with_logo(crop(frame, 920, 430, 0.5, 0.45), logo, 0.52, 0.55) + return {'icon': navi_icon(body, glow), 'logo': logo, 'hero': rgba_of(hero), 'portrait': rgba_of(portrait), + 'wide': rgba_of(wide)} diff --git a/launcher/steam.py b/launcher/steam.py new file mode 100644 index 00000000..02397d35 --- /dev/null +++ b/launcher/steam.py @@ -0,0 +1,329 @@ +"""Add to Steam (Customize Launcher): the game as a non-Steam shortcut in the player's Steam library, with Navi as its +icon and the game's title screen and logo as its pictures (launcher/gameart.py). Windows only. + +Steam keeps the shortcuts of each account in /userdata//config/shortcuts.vdf, a binary key-value +file: a map is 0x00 name 0x00 ... 0x08, a string 0x01 name 0x00 value 0x00, a 32-bit integer 0x02 name 0x00 value +(0x07 is a 64-bit one). The file is the map "shortcuts" holding one map per shortcut, named "0", "1", ... Each holds +appid, AppName, Exe, StartDir, icon, LaunchOptions and Steam's own fields; the other shortcuts are read and written +back as they are, and a shortcut of ours already there (by its name or its id) is updated in place. Its pictures go +into config/grid named by the shortcut's id: p.png the cover, .png the wide tile, _hero.png the +background, _logo.png the logo, and the icon file the shortcut names. The id is CRC-32 of Exe and AppName as +Steam writes them, with the top bit set. + +Steam reads the file when it starts and writes it back when it closes, so a running Steam would neither see the new +shortcut nor keep it: Steam is asked to close (steam.exe -shutdown), the files are written, and Steam starts again. + +The shortcut starts the launcher's pythonw.exe on data/steam_play.pyw, which starts this launcher with --play: the game +runs from its own folder with its console hidden, and the Python process waits for it, so Steam shows the game as +running for as long as it is (the game's own Restart included). Starting ocarina.exe directly would open a console +window beside the game, and NX2navi.exe ends a few seconds after it started the launcher. +""" +import os +import shutil +import struct +import subprocess +import time +import zlib +from pathlib import Path + +NAME = 'The Legend of Zelda Ocarina of Time' +PLAY_SCRIPT = 'steam_play.pyw' +NO_WINDOW = 0x08000000 if os.name == 'nt' else 0 +STEAM64_BASE = 76561197960265728 + +PLAY_TEXT = '''"""Written by the NX2navi launcher (Customize Launcher > Add to Steam): Steam starts the game through this file. +It runs the launcher of app/current.txt with --play, which starts the game and waits for it.""" +import os +import runpy +import sys + +root = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +with open(os.path.join(root, 'app', 'current.txt'), encoding='utf-8') as f: + version = f.read().strip() +os.environ['NX2NAVI_LAUNCHER_ROOT'] = root +sys.argv = [os.path.join(root, 'app', version, 'launcher', 'start.pyw'), '--play'] + sys.argv[1:] +runpy.run_path(sys.argv[0], run_name='__main__') +''' + + +class SteamError(Exception): + pass + + +# ---------------------------------------------------------------- binary VDF + +T_MAP, T_STR, T_INT, T_FLOAT, T_PTR, T_WSTR, T_COLOR, T_U64, T_END = 0, 1, 2, 3, 4, 5, 6, 7, 8 + + +def vdf_load(b): + """A binary VDF as a list of [type, key, value]; a map's value is such a list. Raises SteamError when malformed.""" + def cstr(at): + end = b.index(b'\0', at) + return b[at:end].decode('utf-8', 'surrogateescape'), end + 1 + + def walk(at, top): + items = [] + while True: + if at >= len(b): + if top: + return items, at + raise SteamError('shortcuts.vdf ends inside a map') + t = b[at] + at += 1 + if t == T_END: + return items, at + key, at = cstr(at) + if t == T_MAP: + v, at = walk(at, False) + elif t == T_STR: + v, at = cstr(at) + elif t in (T_INT, T_FLOAT, T_PTR, T_COLOR): + v, at = b[at:at + 4], at + 4 + elif t == T_U64: + v, at = b[at:at + 8], at + 8 + else: + raise SteamError(f'shortcuts.vdf holds a value of type {t}') + items.append([t, key, v]) + + try: + items, at = walk(0, True) + except ValueError: + raise SteamError('shortcuts.vdf is cut short') from None + return items + + +def vdf_dump(items, top=True): + out = bytearray() + for t, key, v in items: + out += bytes([t]) + key.encode('utf-8', 'surrogateescape') + b'\0' + if t == T_MAP: + out += vdf_dump(v, False) + elif t == T_STR: + out += v.encode('utf-8', 'surrogateescape') + b'\0' + else: + out += v + if not top or items: + out += bytes([T_END]) + return bytes(out) + + +def _get(items, key): + for it in items: + if it[1].lower() == key.lower(): + return it + return None + + +def _set(items, t, key, v): + it = _get(items, key) + if it: + it[0], it[2] = t, v + else: + items.append([t, key, v]) + + +def appid(exe, name): + """The id Steam gives a shortcut: CRC-32 of its Exe and AppName fields with the top bit set (unsigned).""" + return (zlib.crc32((exe + name).encode('utf-8')) & 0xFFFFFFFF) | 0x80000000 + + +def add_shortcut(vdf_bytes, exe, start_dir, options, icon, name=NAME): + """shortcuts.vdf (b'' for none) with our shortcut added or updated; returns (new bytes, id).""" + items = vdf_load(vdf_bytes) if vdf_bytes else [] + root = _get(items, 'shortcuts') + if root is None or root[0] != T_MAP: + items = [[T_MAP, 'shortcuts', []]] + root = items[0] + sid = appid(exe, name) + entry = None + for it in root[2]: + if it[0] != T_MAP: + continue + n, a = _get(it[2], 'AppName'), _get(it[2], 'appid') + if (n and n[0] == T_STR and n[2] == name) or (a and a[0] == T_INT and struct.unpack('at || vk_exec_event_met(nvn_op_payload(c->at)) || vk_exec_submit_seq - c->since > CONT_MAX; } +// ---- the gate: a submission that begins with a wait for the CPU's signal (NX2NAVI_GATE_WAIT_MS, default 10; 0 off) +// Ocarina's main queue begins with a wait for an event the game signals from the CPU once it has written what the +// frame reads (vk_exec_xfer.c "events the CPU signals"). The render thread submits that queue without waiting for the +// signal, and polls the event afterwards, a millisecond at a time, before it presents. In Hyrule Field the signal +// came 1 to 2 ms after the submit call in about every other frame: the main thread first waits for the previous +// frame's fence and reads back what it needs. The submission then took its copies (the mirror sync and the uniform +// snapshot, 1.1 to 1.7 ms), met the wait unmet and was held; the render thread polled, presented, and the present +// replayed the frame (vk_backend.c held_resume) under a value the game's fence does not stand for, so the game's next +// wait for that fence returned with the GPU still at work and the frame after was replayed inside its submit call +// again: frames of 23 and 15 ms in turn where each takes 17. +// So when a submission would stop at such a wait before it has recorded anything, the submit call waits for the +// signal here, before the copies are taken, as the console's GPU waits at the wait itself: the copies are then +// taken after the game has finished writing, and the frame is replayed in the call the game's fence stands for. +// Only state may precede the wait in the command buffer, and the event must be one the CPU has signalled before; +// the wait ends after NX2NAVI_GATE_WAIT_MS at the latest, and the submission is then held as before. +static struct { uint64_t n, ns, late; } GW; // since the last perf line: waits, their time, waits that ran out + +static const void *gate_first_wait(const op_hdr *h) { + for (int k = 0; k < 256; k++) { + if (h->size < sizeof *h || (h->size & 7) || h->op >= OP_COUNT_) return NULL; + const void *a = nvn_op_payload(h); + switch (h->op) { + case OP_WAIT_EVENT: return a; + case OP_JUMP: h = (const op_hdr *)(uintptr_t)((const p_jump *)a)->target; continue; + // state the walk only stores: nothing is recorded and nothing reaches the CPU before the wait + case OP_BEGIN: case OP_SET_RT: case OP_VIEWPORTS: case OP_SCISSOR: case OP_DEPTH_RANGE: case OP_BIND_PROGRAM: + case OP_BIND_STATE: case OP_BIND_VATTR: case OP_BIND_VSTREAM: case OP_BIND_VBUF: case OP_BIND_UBO: + case OP_BIND_SSBO: case OP_BIND_TEX: case OP_BIND_SAMP: case OP_BIND_IMG: case OP_STENCIL: case OP_POLY_OFFSET: + case OP_SET_TEX_POOL: case OP_SET_SAMP_POOL: case OP_COPY_STRIDE: case OP_BLEND_COLOR: case OP_DEPTH_CLAMP: + case OP_RASTER_DISCARD: case OP_PRIM_RESTART: case OP_PUSH_CC: + break; + default: return NULL; // a draw, a copy, a signal, a call: the walk decides, as before + } + h = nvn_op_next(h); + } + return NULL; +} + +void vk_exec_gate_wait(const uint64_t *handles, int n) { + static int64_t cap_ns = -1; + if (cap_ns < 0) { + int ms = cfg_int("NX2NAVI_GATE_WAIT_MS"); + cap_ns = ms > 0 ? (int64_t)ms * 1000000 : 0; + } + if (!cap_ns || n < 1 || !handles[0] || !gfx_submit_queue || !evo_enabled()) return; + const void *w = gate_first_wait(nvn_op_first(handles[0])); + if (!w || vk_exec_event_holds_now(w) || !vk_exec_event_cpu_known((const void *)((const p_wait_event *)w)->val)) return; + uint64_t t0 = kern_now_ns(), now = t0; + for (unsigned i = 0; !vk_exec_event_holds_now(w); i++) { + if ((i & 15) == 15 && (now = kern_now_ns()) - t0 >= (uint64_t)cap_ns) { + GW.late++; + break; + } + if (i < 20000) plat_pause(); + else plat_yield(); + } + GW.n++, GW.ns += kern_now_ns() - t0; + if (GW.n >= 600) { + if (rt_trace & RT_TRACE_PERF) + rt_log("perf: gate: %llu submissions waited for the CPU's signal before their copies, %.2f ms on average; " + "%llu waited %lld ms without it and were held", (unsigned long long)GW.n, (double)GW.ns / (double)GW.n / 1e6, + (unsigned long long)GW.late, (long long)(cap_ns / 1000000)); + GW.n = GW.ns = GW.late = 0; + } +} + int vk_exec_held_ready(void) { for (int s = 0; evo_pending && s < EVO_QUEUES; s++) if (evq[s].head && cont_ready(evq[s].head)) return 1; diff --git a/runtime/vk_exec.h b/runtime/vk_exec.h index efb18c35..4758dda0 100644 --- a/runtime/vk_exec.h +++ b/runtime/vk_exec.h @@ -252,6 +252,11 @@ extern int vk_exec_signalled; // set by every replayed SignalEve // Events the CPU signals (vk_exec_xfer.c "events the CPU signals"). void vk_exec_event_cpu(const void *event); // any thread: nvnEventSignal is about to write the event's value void vk_exec_event_sync(int begin); // vk_exec_record: its mirror sync and uniform copy are taken now / it ends +int vk_exec_event_cpu_known(const void *event); // the CPU has signalled this event before +int vk_exec_event_holds_now(const void *payload); // an OP_WAIT_EVENT's condition on the value in memory now +// vk_backend.c submit_sync, before a submission is recorded: waits for the CPU's signal when the submission begins +// with a wait for it (vk_exec.c "the gate"). +void vk_exec_gate_wait(const uint64_t *handles, int n); int vk_exec_event_at_sync(void); // NX2NAVI_EVENT_AT_SYNC uint64_t vk_exec_event_late_take(void); // waits only a CPU signal after the copies would have passed, since the last call // Since the last call, which resets them: waits, waits reached before their signal, those of them a later replayed diff --git a/runtime/vk_exec_xfer.c b/runtime/vk_exec_xfer.c index f5cd3a6c..a0e83134 100644 --- a/runtime/vk_exec_xfer.c +++ b/runtime/vk_exec_xfer.c @@ -265,38 +265,103 @@ void vk_exec_oq_draw(VkExec *ex) { void vk_exec_oq_pass_end(VkExec *ex) { oq_close(ex); } +static void oq_batch_drop(void); // the results kept by the completion thread (oq_take) + void vk_exec_xfer_shutdown(void) { if (oq_state > 0) vkDestroyQueryPool(vk_ctx()->device, oq_pool, NULL); oq_pool = VK_NULL_HANDLE, oq_state = 0, oq_next = 0, oq_counting = oq_reported = oq_keep = 0, oq_open = -1; oq_nseg = 0, oq_lost = 0, oq_count = 0; memset((void *)oq_busy, 0, sizeof oq_busy); + oq_batch_drop(); +} + +// The completion thread reads the results in batches (NX2NAVI_OQ_BATCH, default 1; 0: every segment by itself, as +// before). The game tests one proxy per report, so a segment holds one query, and a frame of Hyrule Field has about a +// hundred of them: one vkGetQueryPoolResults and one vkResetQueryPool each. On an RX 7900 XT (26.10.1) each of those +// calls goes into the kernel, and the two hundred took 3.9 ms of every frame on the completion thread, which runs a +// submission's callbacks before it lets the submission's waiters go: the game's main thread waits for the previous +// frame's fence before it lets the next frame's GPU work start, so the frame after one replayed inside its submit +// call began 4 to 6 ms late with every thread idle. Slots come from the ring in order, so the slots of the segments +// that follow lie behind the one asked for: one call reads OQ_BATCH slots from there on with their availability, the +// available ones are kept (the results stay in the pool until the reset), and the segments after it take theirs from +// what is kept. A slot not available in the batch (its part had not run yet) is read again when its segment comes. +// The slots taken are reset in runs, one vkResetQueryPool for each: a run ends when a slot that does not continue it +// is taken or a new batch is read. Until then its slots stay busy for the replay, at most OQ_BATCH of the ring's +// OQ_N, also while the game reports nothing. +#define OQ_BATCH 256 +static struct { + uint32_t first, n; // the slots kept: [first, first + n) + uint64_t v[OQ_BATCH * 2]; // per slot: samples, available + uint8_t taken[OQ_BATCH]; + uint32_t r0, rn; // the run of taken slots still to be reset + uint64_t calls, slots, segs; // since the last perf line: result calls, slots they read, segments served +} OQB; +static int oq_batch_mode = -1; + +static void oq_batch_drop(void) { OQB.n = OQB.rn = 0; } + +static void oq_reset_flush(VkDevice dev) { + if (!OQB.rn) return; + vkResetQueryPool(dev, oq_pool, OQB.r0, OQB.rn); + for (uint32_t k = 0; k < OQB.rn; k++) __atomic_store_n(&oq_busy[OQB.r0 + k], 0, __ATOMIC_RELEASE); + OQB.rn = 0; +} + +static void oq_read(VkDevice dev, uint32_t first, uint32_t n) { + VkResult r = vkGetQueryPoolResults(dev, oq_pool, first, n, sizeof OQB.v[0] * 2 * n, OQB.v, 2 * sizeof OQB.v[0], + VK_QUERY_RESULT_64_BIT | VK_QUERY_RESULT_WITH_AVAILABILITY_BIT); + if (r != VK_SUCCESS && r != VK_NOT_READY) { + vk_device_fault_report(r); + rt_fatal("vk: occlusion query results failed (%d)", (int)r); + } + OQB.first = first, OQB.n = n; + memset(OQB.taken, 0, n); + OQB.calls++, OQB.slots += n; +} + +// The samples of slot q, whose submission has completed; *avail 0 when the query has no result. +static uint64_t oq_take(VkDevice dev, uint32_t q, int *avail) { + uint32_t k = q - OQB.first; + if (!(q >= OQB.first && k < OQB.n && !OQB.taken[k] && OQB.v[2 * k + 1])) { + // not kept, or kept as unavailable (read before its part ran): from this slot on, up to the replay's next slot + oq_reset_flush(dev); // before the kept results go: the run's slots are among them + uint32_t next = __atomic_load_n(&oq_next, __ATOMIC_RELAXED) % OQ_N, n = OQ_N - q; + if (next > q && next - q < n) n = next - q; + if (n > OQ_BATCH) n = OQ_BATCH; + if (oq_batch_mode == 0) n = 1; + oq_read(dev, q, n); + k = 0; + } + OQB.taken[k] = 1; + if (OQB.rn && q != OQB.r0 + OQB.rn) oq_reset_flush(dev); + if (!OQB.rn) OQB.r0 = q; + OQB.rn++; + *avail = OQB.v[2 * k + 1] != 0; + return OQB.v[2 * k]; } static void oq_cb(void *arg) { OqSegment *s = arg; VkDevice dev = vk_ctx()->device; - uint64_t sum = 0, v[64 * 2]; - for (uint32_t i = 0; i < s->n;) { - uint32_t j = i + 1; - while (j < s->n && j - i < 64 && s->q[j] == s->q[j - 1] + 1) j++; + if (oq_batch_mode < 0) oq_batch_mode = cfg_bool("NX2NAVI_OQ_BATCH"); + uint64_t sum = 0; + for (uint32_t i = 0; i < s->n; i++) { // The fence has passed, so a query that is still unavailable will not become available: waiting would block the // completion thread, and with it every later callback, for good. It counts as visible (one sample). - VkResult r = vkGetQueryPoolResults(dev, oq_pool, s->q[i], j - i, sizeof v, v, 2 * sizeof v[0], - VK_QUERY_RESULT_64_BIT | VK_QUERY_RESULT_WITH_AVAILABILITY_BIT); - if (r != VK_SUCCESS && r != VK_NOT_READY) { - vk_device_fault_report(r); - rt_fatal("vk: occlusion query results failed (%d)", (int)r); + int avail; + uint64_t v = oq_take(dev, s->q[i], &avail); + if (avail) sum += v; + else { + sum += 1; + LOG_ONCE("vk: occlusion query slot %u not available after its fence, counted as visible", s->q[i]); } - vkResetQueryPool(dev, oq_pool, s->q[i], j - i); - for (uint32_t k = i; k < j; k++) { - if (v[2 * (k - i) + 1]) sum += v[2 * (k - i)]; - else { - sum += 1; - LOG_ONCE("vk: occlusion query slot %u not available after its fence, counted as visible", s->q[k]); - } - __atomic_store_n(&oq_busy[s->q[k]], 0, __ATOMIC_RELEASE); - } - i = j; + } + if (s->n && ++OQB.segs >= 12000) { + if (rt_trace & RT_TRACE_PERF) + rt_log("perf: occlusion queries: %llu segments read with %llu result calls of %.1f slots on average", + (unsigned long long)OQB.segs, (unsigned long long)OQB.calls, + OQB.calls ? (double)OQB.slots / (double)OQB.calls : 0.0); + OQB.calls = OQB.slots = OQB.segs = 0; } if (s->reset) oq_count = 0; else { @@ -412,6 +477,22 @@ void vk_exec_event_cpu(const void *event) { plat_mutex_unlock(&cpu_ev_lock); } +// 1 when the CPU has signalled this event before (nvnEventSignal): its waits are gates the game opens from the CPU. +int vk_exec_event_cpu_known(const void *event) { + plat_mutex_lock(&cpu_ev_lock); + int i = 0; + while (i < cpu_ev_n && cpu_ev[i].val != event) i++; + int known = i < cpu_ev_n; + plat_mutex_unlock(&cpu_ev_lock); + return known; +} + +// The wait's condition on the value in memory now, whatever the replay has seen. Any thread. +int vk_exec_event_holds_now(const void *payload) { + const p_wait_event *p = payload; + return ev_holds(__atomic_load_n(p->val, __ATOMIC_ACQUIRE), p->mode, p->value); +} + void vk_exec_event_sync(int begin) { plat_mutex_lock(&cpu_ev_lock); ev_sync_seq = begin ? cpu_ev_seq : UINT64_MAX; diff --git a/runtime/vk_pipeline.c b/runtime/vk_pipeline.c index 327b128a..d1a26deb 100644 --- a/runtime/vk_pipeline.c +++ b/runtime/vk_pipeline.c @@ -87,6 +87,8 @@ static struct { VkShaderStageFlags sg32_stages; // stages that accept requiredSubgroupSize = 32 (0: not available) int reconverge; // modules with subgroup ops get MaximallyReconvergesKHR (spirv_reconverge) uint32_t nan_preserve; // float widths that get SignedZeroInfNanPreserve (spirv_nan_preserve): NP_16|NP_32|NP_64 + int nan_bits; // NaN tests as integer bit tests (spirv_nan_bits) + volatile uint32_t nan_bits_tests, nan_bits_selects, nan_bits_modules; // rewritten so far (the first log line) VkDynamicState dyn[32]; uint32_t n_dyn; PFN_vkCmdPushDescriptorSetKHR push; @@ -658,6 +660,12 @@ static void nan_preserve_init(VkCtx *c) { (fc.shaderSignedZeroInfNanPreserveFloat32 ? NP_32 : 0) | (fc.shaderSignedZeroInfNanPreserveFloat64 ? NP_64 : 0); P.nan_preserve = want && (can & NP_32) ? can : 0; + // NX2NAVI_VK_NAN_BITS: 1 every driver, 0 none; unset: AMD's driver for RDNA 1 and 2 (spirv_nan_bits) + const char *nb = cfg_get("NX2NAVI_VK_NAN_BITS"); + P.nan_bits = nb && *nb ? strcmp(nb, "0") != 0 : vk_caps.amd_sc; + if (P.nan_bits) + rt_log("vk_pipeline: game shaders test for NaN on the bits of the number, not as a float (%s)", + nb && *nb ? "NX2NAVI_VK_NAN_BITS=1" : "AMD's driver for RDNA 1 and 2"); if (P.nan_preserve) rt_log("vk_pipeline: game shaders keep NaN, infinity and signed zero (SignedZeroInfNanPreserve for %s32%s-bit " "floats%s)", can & NP_16 ? "16, " : "", can & NP_64 ? ", 64" : "", set ? ", NX2NAVI_VK_NAN_PRESERVE=1" : ""); @@ -1478,6 +1486,125 @@ static uint32_t *spirv_nan_preserve(const uint32_t *w, uint64_t n, uint32_t widt return o; } +// NaN tests that no compiler can drop (P.nan_bits). The game's shaders and the translator's own guards test for NaN with +// OpIsNan: 48,717 of an install's 54,959 translated modules, 26 tests each on average (the game's unordered +// comparisons, and the guard that keeps NaN out of every float colour output, patch 0027, so that one NaN texel does not +// spread through bloom and blur into blocks of black and white). SignedZeroInfNanPreserve asks the driver to keep NaN, +// but whether a compiler that reasons about floats keeps an isnan through every optimisation is up to it. The reports +// of black and white blocks, on Navi, on Link next to her and on bright surfaces seen from inside a house, come from +// RX 5000 and 6000 cards (#41, #110, #117), whose Windows driver compiles with "AMD proprietary shader compiler" +// (vk_caps.amd_sc); RDNA 3 and 4 (LLPC) draw the same scenes clean. It is not known whether that compiler keeps the +// mode: on it each float32 test becomes integer arithmetic on the bits, which means the same under any float +// assumption: +// isnan(x) -> (bitcast(x) & 0x7fffffff) > 0x7f800000 +// for scalars and vectors of 32-bit floats; tests of 16- and 64-bit floats stay. A float32 select with a constant +// operand whose bits are a NaN (an integer mask such as 0xffffffff that the translator types as float) becomes a +// select of the bits (issue #156: on an RX 9070 XT without the mode such a select between 1 and -1 gave 2^32). +// NULL: nothing to change (or no memory); *tests and *selects count what changed. +static uint32_t *spirv_nan_bits(const uint32_t *w, uint64_t n, uint64_t *out_n, uint32_t *tests, uint32_t *selects) { + enum { OP_UNDEF = 1, OP_TYPE_BOOL = 20, OP_SPEC_CONSTANT = 50, OP_SPEC_CONSTANT_COMPOSITE = 51, OP_STORE = 62, + OP_BITCAST = 124, OP_IS_NAN = 156, OP_SELECT = 169, OP_UGREATER = 172, OP_BITWISE_AND = 199 }; + *tests = *selects = 0; + if (n < 5 || w[3] >= (1u << 22)) return NULL; + uint32_t bound = w[3]; + uint32_t *rtype = calloc(bound, sizeof *rtype), *vec_comp = calloc(bound, sizeof *vec_comp); + uint8_t *f32 = calloc(bound, 1), *vec_n = calloc(bound, 1), *nan_const = calloc(bound, 1); + uint32_t *o = NULL; + if (!rtype || !vec_comp || !f32 || !vec_n || !nan_const) goto out; + uint32_t tu = 0, uvec[5] = {0}; // uint32 and its vectors by component count + uint64_t fn_at = 0; + int vec_used[5] = {0}; + // Ids that are not results of the instruction's second word, inside functions: first assignment wins anyway, as + // a definition comes before its uses (OpPhi operands aside, which this never reads). + for (uint64_t i = 5; i < n;) { + uint32_t op = w[i] & 0xffff, l = w[i] >> 16; + if (!l || i + l > n) goto out; + const uint32_t *a = w + i + 1; + if (!fn_at) { + if (op == SPV_OP_FUNCTION) fn_at = i; + else if (op == SPV_OP_TYPE_FLOAT && l >= 3 && a[0] < bound && a[1] == 32) f32[a[0]] = 1; + else if (op == SPV_OP_TYPE_INT && l >= 4 && a[1] == 32 && a[2] == 0 && !tu) tu = a[0]; + else if (op == SPV_OP_TYPE_VECTOR && l >= 4 && a[0] < bound && a[2] >= 2 && a[2] <= 4) { + vec_comp[a[0]] = a[1], vec_n[a[0]] = (uint8_t)a[2]; + if (tu && a[1] == tu && !uvec[a[2]]) uvec[a[2]] = a[0]; + } else if ((op == SPV_OP_CONSTANT || op == SPV_OP_CONSTANT_COMPOSITE || op == SPV_OP_CONSTANT_NULL || + op == OP_UNDEF || op == SPV_OP_VARIABLE || op == OP_SPEC_CONSTANT || + op == OP_SPEC_CONSTANT_COMPOSITE) && l >= 3 && a[1] < bound) { + rtype[a[1]] = a[0]; + if (op == SPV_OP_CONSTANT && l == 4 && a[0] < bound && f32[a[0]] && (a[2] & 0x7fffffffu) > 0x7f800000u) + nan_const[a[1]] = 1; + } + } + if (fn_at) { + int no_result = op == OP_STORE || op == 63 || op == 64 || op == 8 || op == 317 || (op >= 246 && op <= 255) || + op == 99 || op == 224 || op == 225 || op == 228 || op == 56 || op == 218 || op == 219 || + op == 5294 || op == 5295 || op == 4416 || op == 5380 || op == 5364 || op == 5365; + if (!no_result && op != SPV_OP_FUNCTION && l >= 3 && a[1] < bound && !rtype[a[1]]) rtype[a[1]] = a[0]; + if (op == OP_IS_NAN && l == 4 && a[2] < bound) { + uint32_t t = rtype[a[2]]; + if (t < bound && f32[t]) ++*tests; + else if (t < bound && vec_n[t] && vec_comp[t] < bound && f32[vec_comp[t]]) ++*tests, vec_used[vec_n[t]] = 1; + } else if (op == OP_SELECT && l == 6 && a[0] < bound && f32[a[0]] && a[3] < bound && a[4] < bound && + (nan_const[a[3]] || nan_const[a[4]])) { + ++*selects; + } + } + i += l; + } + if (!fn_at || (!*tests && !*selects)) goto out; + // the declarations go right before the first function: uint32 and its vectors where missing, then the constants + uint32_t next = bound, decl[64], nd = 0; + if (!tu) tu = next++, decl[nd++] = 4u << 16 | SPV_OP_TYPE_INT, decl[nd++] = tu, decl[nd++] = 32, decl[nd++] = 0; + uint32_t c_mag = next++, c_inf = next++, cv_mag[5] = {0}, cv_inf[5] = {0}; + for (int k = 2; k <= 4; k++) + if (vec_used[k] && !uvec[k]) + uvec[k] = next++, decl[nd++] = 4u << 16 | SPV_OP_TYPE_VECTOR, decl[nd++] = uvec[k], decl[nd++] = tu, decl[nd++] = (uint32_t)k; + decl[nd++] = 4u << 16 | SPV_OP_CONSTANT, decl[nd++] = tu, decl[nd++] = c_mag, decl[nd++] = 0x7fffffffu; + decl[nd++] = 4u << 16 | SPV_OP_CONSTANT, decl[nd++] = tu, decl[nd++] = c_inf, decl[nd++] = 0x7f800000u; + for (int k = 2; k <= 4; k++) { + if (!vec_used[k]) continue; + cv_mag[k] = next++, cv_inf[k] = next++; + decl[nd++] = (uint32_t)(3 + k) << 16 | SPV_OP_CONSTANT_COMPOSITE, decl[nd++] = uvec[k], decl[nd++] = cv_mag[k]; + for (int j = 0; j < k; j++) decl[nd++] = c_mag; + decl[nd++] = (uint32_t)(3 + k) << 16 | SPV_OP_CONSTANT_COMPOSITE, decl[nd++] = uvec[k], decl[nd++] = cv_inf[k]; + for (int j = 0; j < k; j++) decl[nd++] = c_inf; + } + uint64_t cap = n + nd + 10ull * *tests + 12ull * *selects; + o = malloc(cap * 4); + if (!o) goto out; + memcpy(o, w, fn_at * 4); + uint64_t p = fn_at; + memcpy(o + p, decl, nd * 4), p += nd; + for (uint64_t i = fn_at; i < n;) { + uint32_t op = w[i] & 0xffff, l = w[i] >> 16; + const uint32_t *a = w + i + 1; + uint32_t t = op == OP_IS_NAN && l == 4 && a[2] < bound ? rtype[a[2]] : 0; + int k = t && t < bound ? (f32[t] ? 1 : vec_n[t] && vec_comp[t] < bound && f32[vec_comp[t]] ? vec_n[t] : 0) : 0; + if (k) { + uint32_t ut = k == 1 ? tu : uvec[k], mag = k == 1 ? c_mag : cv_mag[k], inf = k == 1 ? c_inf : cv_inf[k]; + uint32_t bits = next++, low = next++; + o[p++] = 4u << 16 | OP_BITCAST, o[p++] = ut, o[p++] = bits, o[p++] = a[2]; + o[p++] = 5u << 16 | OP_BITWISE_AND, o[p++] = ut, o[p++] = low, o[p++] = bits, o[p++] = mag; + o[p++] = 5u << 16 | OP_UGREATER, o[p++] = a[0], o[p++] = a[1], o[p++] = low, o[p++] = inf; + } else if (op == OP_SELECT && l == 6 && a[0] < bound && f32[a[0]] && a[3] < bound && a[4] < bound && + (nan_const[a[3]] || nan_const[a[4]])) { + uint32_t x = next++, y = next++, s = next++; + o[p++] = 4u << 16 | OP_BITCAST, o[p++] = tu, o[p++] = x, o[p++] = a[3]; + o[p++] = 4u << 16 | OP_BITCAST, o[p++] = tu, o[p++] = y, o[p++] = a[4]; + o[p++] = 6u << 16 | OP_SELECT, o[p++] = tu, o[p++] = s, o[p++] = a[2], o[p++] = x, o[p++] = y; + o[p++] = 4u << 16 | OP_BITCAST, o[p++] = a[0], o[p++] = a[1], o[p++] = s; + } else { + memcpy(o + p, w + i, l * 4), p += l; + } + i += l; + } + o[3] = next; + *out_n = p; +out: + free(rtype), free(vec_comp), free(f32), free(vec_n), free(nan_const); + return o; +} + static int spirv_mark_list_test(const char *list); // texture pool marks, below // --selftest (main.c): spirv_nan_preserve on a small fragment module, without a device. Returns the number of failures. @@ -1523,6 +1650,42 @@ int vk_pipeline_spirv_selftest(void) { failures++; } printf("spirv: SignedZeroInfNanPreserve rewrite: %s\n", failures ? "FAIL" : "ok"); + { // spirv_nan_bits: a scalar and a vec2 isnan, a select on a NaN constant; no uint32 type in the module + static const uint32_t m[] = { + 0x07230203, 0x00010300, 0, 20, 0, + 2u << 16 | 17, 1, // OpCapability Shader + 3u << 16 | 14, 0, 1, // OpMemoryModel Logical GLSL450 + 5u << 16 | 15, 5, 1, 0x6e69616d, 0, // OpEntryPoint GLCompute %1 "main" + 2u << 16 | 19, 2, // %2 void + 3u << 16 | 33, 3, 2, // %3 fn + 3u << 16 | 22, 4, 32, // %4 float + 2u << 16 | 20, 5, // %5 bool + 4u << 16 | 23, 6, 4, 2, // %6 vec2 + 4u << 16 | 23, 7, 5, 2, // %7 bvec2 + 4u << 16 | 43, 4, 8, 0xffffffffu, // %8 = NaN bits + 4u << 16 | 43, 4, 9, 0x3f800000u, // %9 = 1.0 + 5u << 16 | 44, 6, 10, 8, 9, // %10 = vec2(%8, %9) + 3u << 16 | 41, 5, 11, // %11 = OpConstantTrue + 5u << 16 | 54, 2, 1, 0, 3, // %1 = OpFunction + 2u << 16 | 248, 12, // OpLabel + 4u << 16 | 156, 5, 13, 9, // %13 = OpIsNan bool %9 + 4u << 16 | 156, 7, 14, 10, // %14 = OpIsNan bvec2 %10 + 6u << 16 | 169, 4, 15, 11, 9, 8, // %15 = OpSelect float %11 %9 %8 + 1u << 16 | 253, 1u << 16 | 56}; // OpReturn, OpFunctionEnd + uint64_t on = 0; + uint32_t tests = 0, sels = 0, isnan_left = 0, ugt = 0, int32 = 0, bitcasts = 0; + uint32_t *o = spirv_nan_bits(m, sizeof m / 4, &on, &tests, &sels); + for (uint64_t i = 5; o && i < on;) { + uint32_t op = o[i] & 0xffff, l = o[i] >> 16; + if (!l) break; + isnan_left += op == 156, ugt += op == 172, int32 += op == 21 && o[i + 2] == 32, bitcasts += op == 124; + i += l; + } + int bad = !o || tests != 2 || sels != 1 || isnan_left || ugt != 2 || int32 != 1 || bitcasts != 5 || o[3] <= 20; + printf("spirv: integer NaN tests rewrite: %s\n", bad ? "FAIL" : "ok"); + failures += bad; + free(o); + } const char *list = cfg_get("NX2NAVI_SPV_MARK_TEST"); if (list && *list) failures += spirv_mark_list_test(list); return failures; @@ -1867,6 +2030,18 @@ static VkShaderModule make_module(const uint32_t *w, uint64_t words, int early_o free(fixed); w = fixed = np, words = np_n; } + uint64_t nb_n = 0; + uint32_t nb_tests = 0, nb_sels = 0; + uint32_t *nb = P.nan_bits ? spirv_nan_bits(w, words, &nb_n, &nb_tests, &nb_sels) : NULL; + if (nb) { + free(fixed); + w = fixed = nb, words = nb_n; + __atomic_fetch_add(&P.nan_bits_tests, nb_tests, __ATOMIC_RELAXED); + __atomic_fetch_add(&P.nan_bits_selects, nb_sels, __ATOMIC_RELAXED); + if (__atomic_add_fetch(&P.nan_bits_modules, 1, __ATOMIC_RELAXED) == 1) + rt_log("vk_pipeline: first module with NaN tests made integer: %u tests, %u selects of NaN bits", nb_tests, + nb_sels); + } if (P.feedback >= 2) mm_hash = fnv(FNV0, w, (size_t)words * 4); VkShaderModuleCreateInfo ci = {.sType = VK_STRUCTURE_TYPE_SHADER_MODULE_CREATE_INFO, .codeSize = (size_t)words * 4, .pCode = w}; VkShaderModule m = VK_NULL_HANDLE; diff --git a/tests/launcher_test.py b/tests/launcher_test.py index 55707c2a..597a3fe0 100644 --- a/tests/launcher_test.py +++ b/tests/launcher_test.py @@ -18,6 +18,7 @@ Run: python tests/launcher_test.py (Python 3.14 for the zstd tests; the window a source checkout's build/xlate/), the prompts step with a stand-in prompt tool, the object cache of the compile step with the system's C compiler, and its windres call under folders with spaces (a script in windres's place) - jobs: an install driven by a fake make_game.py, with and without the prompts row +- Add to Steam: the binary shortcuts file read and written back unchanged, a shortcut added and updated in place - the look: the colour schemes, the background picture's fitting and kept copy, the sound effects (low by default, only the game's own select and back sounds, decoded from a made-up .bwav) - the window: --selftest draws every screen (skipped without a display) @@ -3048,6 +3049,60 @@ def have_display(): @unittest.skipUnless(have_display(), 'needs tkinter, numpy and a display') +class SteamTest(Tmp): + """launcher/steam.py on made-up Steam files: the binary shortcuts file read and written back unchanged, our + shortcut added beside another and updated in place, its id as Steam computes it, the accounts that signed in.""" + + def other(self): + from launcher import steam + exe = '"C:\\Games\\Other\\other.exe"' + entry = [[steam.T_INT, 'appid', struct.pack('