End a run whose GPU device is lost in one place: one report, no callbacks of unfinished submissions, and exit code 72.

A lost device used to end the process from whichever submit or wait noticed it first, while the present only logged it
and went on and the other threads kept calling the driver. The completion thread could take the submissions' timeline
as reached and run the callbacks of submissions that never finished.

vk_device_lost_check now takes the result of every submit, present, acquire and wait. The first thread to see the loss
sets a flag, writes the report and ends the process as a fatal error does (the log written, a save commit under way
finished), with an exit code of its own; later threads stop in that call, and a watchdog ends the process after ten
seconds should the first thread hang. The replay, its submit, the uploads, the present and the lazy fetch read the flag
and return without touching the device. The completion thread asks the device with a call of its own before a new
value counts and runs no callback once the flag is set.

The executable's --probe-devlost and tests/device_lost_check.py check this without a lost device, and
device_lost_at in nx2navi.ini ends a real run the same way.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
chrg6
2026-11-11 04:35:30 -05:00
parent 84b0672ec2
commit 534591be04
16 changed files with 319 additions and 41 deletions

View File

@@ -59,14 +59,13 @@ void rt_fatal_notes(void) {
__attribute__((weak)) void fs_save_crash_wait(void) {} // fs_host.c; builds without it (the unit checks) have no saves
void rt_fatal(const char *fmt, ...) {
va_list ap;
va_start(ap, fmt);
// The fatal line, the report's notes and what has to be flushed, then the exit with `code`.
static void fatal_end(int code, const char *fmt, va_list ap) __attribute__((noreturn));
static void fatal_end(int code, const char *fmt, va_list ap) {
rt_log_sync(); // main.c: the lines queued for the log thread, then these, written here
rt_log_raw("FATAL: ", 7);
rt_vlogf(fmt, ap);
rt_log_raw("\n", 1);
va_end(ap);
// A failed allocation (malloc, realloc, VirtualAlloc) means the commit limit was reached: this line tells a leak of
// ours (process private bytes near the limit) from a machine short of page file.
double proc, used, limit;
@@ -81,7 +80,19 @@ void rt_fatal(const char *fmt, ...) {
rtss_limit_restore();
fflush(stdout);
fflush(stderr);
_Exit(70);
_Exit(code);
}
void rt_fatal(const char *fmt, ...) {
va_list ap;
va_start(ap, fmt);
fatal_end(RT_FATAL_EXIT, fmt, ap);
}
void rt_fatal_code(int code, const char *fmt, ...) {
va_list ap;
va_start(ap, fmt);
fatal_end(code, fmt, ap);
}
static const RtModule *module_at(uint64_t pc) {