Skip to content

2026-11-01 acnh-race-atomics (batch 2026-11-01-r3, Mac)

Branch claude/acnh-race-atomics-2026-11-01-r3 (on top of claude/sched-applets-2026-11-01-r1). Root cause of the ACNH base start crash found and fixed generically in the AOT rewrite: guest spinlocks without acquire ordering, which Apple cores reorder.

Root cause

ACNH 1.0.0's job system runs parallel-for jobs on the guest main thread (0x51, core 0) and workers (0x57/0x58, cores 1/2). Each executor claims items under the job's spinlock (main+0x19b65d4): 

ldr w9,[x8]; cmp w9,#1; b.eq <spin>          ; wait while locked
ldxr w10,[x8]; cbnz w10,<retry>; stxr w10,w9,[x8]; cbnz w10,<retry>   ; acquire: no acquire semantics, no barrier after
ldp w9,w26,[x21,#0xc0]                        ; total, next  (the protected data)
... str next+chunk ...
dmb ish; ldxr; stxr wzr                       ; release
ARMv8 lets the ldp run before the store-exclusive completes (the cbnz on the status orders only later stores), so a thread that wins the lock can use next as it was before the previous owner's release, and two threads claim the same items. On ACNH that is the same view recorded by two threads into one shared command-buffer slot (F main+0x1ae75c0), the overlap gfx-recorder-flake found and the main+0x1b61d3c null read. The Cortex-A57 practically never shows this reordering; Apple's cores do.

Evidence

  • tools/soak/litmus_excl_claim.c (new), the lock and claim above instruction for instruction, 3 threads, 41.5 M claims per run on this Mac (Apple M5): 5 333, 24 505, 8 171, 18 116 items claimed twice with ldxr; 0 (4 runs) with ldaxr.
  • NAPPLE_ARB_TRACE (new, runtime/src/threads.c) at a real collision (ACNH base, run with CBTRACE; collision at 459523.005886): the kernel side is clean. The manager's join is nn::os::WaitLightEvent(worker+0x178) per worker (stubs resolved: main+0x2210580..b0 = Wait / TimedWait / Signal / ClearLightEvent); workers receive kicks from an nn::os MessageQueue (ReceiveMessageQueue, condvar svcs). Every successful WaitForAddress exit of the main thread follows a SignalToAddress that names it as the woken thread, messages are received in order, and each done signal follows its message. Both threads were legitimately inside the same stage: the duplicate came from the claim.
  • runtime/tests/host/test_light_event.c (new): the SDK's auto-clear LightEvent (sdk+0xca500 / sdk+0xca690, 0/1/2 states, SignalToAddress ModifyByWaiterCountIfEqual count 1, WaitForAddress WaitIfEqual) in the manager's kick/join handshake, 3 pinned workers x 20 000 rounds: no early join; two waiters with count-1 signals: no double or lost wake. Passes on the current kernel, so suspect 2 (sync svc semantics) is ruled out.
  • Suspect 1 (exclusive-window emulation) ruled out for this path: every ldxr/stxr of the dispatcher, the claim, the pop and the SDK's LightEvent is native in main/sdk.patched.bin (windows are only copied when they contain an x18 site).
  • r1 had already ruled out scheduling (overlaps with NAPPLE_SLICE_MS=0).

Fix (crates/napple-rewrite)

ordering.rs (new): every exclusive load that starts a pair (an exclusive store within max_window_len instructions, neither word in the scanner's provably-data set) gets acquire ordering and its store release ordering, in place (bit 15: ldxr->ldaxr, stxr->stlxr, byte / halfword / pair forms too). It only adds ordering, so it is correct for any guest code. RewriteOptions::order_exclusives (default on), stats exclusive_loads_acquire / exclusive_stores_release; check.rs accepts exactly this one-bit upgrade in text and in window copies. ACNH base: main 4 267 words, sdk 325 changed, every one an ordering upgrade, nothing else. Title data must be rebuilt for the fix to apply.

Verified here (Mac)

Title data rebuilt with this branch into /Users/neto/NappleWork/games-r3/ (ACNH, LAS, SMO, MK8D); runtime = this branch; all runs headless, audio off, NAPPLE_GFX_CBTRACE=1, under lockf /Users/neto/NappleWork/.run.lock, one start per hold.

Check Result
cargo test -p napple-rewrite pass (3 new ordering tests)
make -C runtime GFX=napple test, make -C runtime test-host pass (light_event 3/3, threads_preempt 12/12)
ACNH base 40 x 30 s 40 ok, 0 crashes, 0 overlap dumps (was 11 of 40 with overlaps)
ACNH update+DLC 10 x 30 s 10 ok, 0 overlap dumps
LAS / SMO / MK8D 60 s ok / ok / ok (3547 / 3548 / 3308 frames)
Before (same conditions, old data): overlaps in 8 of 20 (base-commit scheduler) and 11 of 40 (r1 scheduler) starts, crashes main+0x1b61d3c.

Also here

  • NAPPLE_ARB_TRACE=<file>: ring of the last 1 M ArbitrateLock/Unlock, WaitProcessWideKeyAtomic/SignalProcessWideKey, WaitForAddress/SignalToAddress svcs (thread, address, type/value/count, word before/after, waiters, result, woken threads), written at quit and on a crash; tools/soak/arbtrace.py prints it (--tid, --addr, --from/--to in the CBTRACE clock).

Not done / follow-ups

  • The AArch32 lifter (runtime/src/a32jit, not this area) maps ldrex/strex to plain ldxr/stxr (docs/architecture/aarch32.md): the same reordering applies to 32-bit titles; it should emit ldaxr/stlxr too.
  • Plain (non-exclusive) guest locks such as ldr + stlr patterns are not touched; only exclusive pairs are.

Files

crates/napple-rewrite/src/{ordering.rs (new),a64.rs,image.rs,scan.rs,types.rs,rewrite.rs,check.rs,lib.rs}, runtime/src/threads.c, runtime/src/signals.c, runtime/tests/host/test_light_event.c (new), runtime/Makefile (one host test rule), tools/soak/{arbtrace.py,litmus_excl_claim.c} (new), docs/todo.md (ACNH section), docs/architecture/rewrite-format.md (one bullet), this note.