# Two ARM boards, one KVM, and a watchdog that failed

**Claim.** The KVM tooling was used to bring up two ODROID-M1S boards (RK3566, 8 GB, eMMC) with a rescue card, and it caught a failure that no network check could have named: on this board the hardware watchdog reset leaves the board dark until its power is cycled by hand.

**Method.** One Comet on one board at a time, HDMI plus USB keyboard. Each board got Armbian on its eMMC and a rescue card in the SD slot that hands over to the eMMC system when it is complete. A self-test script reboots the board, removes the kernel or the device tree on purpose, and checks that the rescue system comes up and repairs it. The watchdog experiment crashed the kernel with `kernel.panic=0`, so that only the hardware watchdog could reset it, and recorded the screen.

**Sample size.** Two boards (`orderly`, `matron`), one experiment on each, one Comet recording (matron).

**Result.**

| Test | Outcome | Record |
|---|---|---|
| Plain reboot hands over to the eMMC system | passed, back in 28 s | [selftest-orderly-full.txt](m1s-records/selftest-orderly-full.txt) |
| One-shot flag boots the rescue system | passed, rescue up at 35 s | [selftest-orderly-full.txt](m1s-records/selftest-orderly-full.txt) |
| Device tree moved aside | rescue system came up at 35 s and restored the file (115510 bytes) | [selftest-orderly-full.txt](m1s-records/selftest-orderly-full.txt) |
| Kernel moved aside | rescue system came up at 36 s and restored the kernel (35916288 bytes) | [selftest-orderly-full.txt](m1s-records/selftest-orderly-full.txt) |
| Kernel panic with `kernel.panic=10` | board back after 54 s, panic text received on haywire | [selftest-orderly-full.txt](m1s-records/selftest-orderly-full.txt) |
| Kernel hang with `kernel.panic=0`, hardware watchdog (44 s timeout) armed | no picture from 56 s into the recording to its end at 199.7 s: 8 frames with a picture, then 19 refused snapshots | [matron-watchdog-audit.jsonl](m1s-records/matron-watchdog-audit.jsonl), [matron-watchdog-transcript.txt](m1s-records/matron-watchdog-transcript.txt) |

The screen text shows the system started the watchdog with the 44 s timeout before the crash; the transcript has the frame. When the panic path resets the board, it boots. When the watchdog resets it, the recording shows a dark screen that does not recover. The watchdog is switched off in the baseline.

**What the tooling changed.** A timeout on the network side looks the same in every case: a stale name, a changed host key, an operating system with its network down, a dark board and a panic all read as "no answer". The `diagnose` command reads the video signal, whether the target still sees the KVM's keyboard, and the screen text together with ping, TCP and one ssh attempt, then names the cause and the next step. Run on matron while it was still dark, it answered "dark", with no signal and no keyboard seen as its evidence ([diagnose-matron-dark.txt](m1s-records/diagnose-matron-dark.txt)). The rules are tested against the scenarios on this page.

**A recording mistake, fixed.** The audit log of a session kept the text typed on the board's console, including a login password, because typed text was logged verbatim. Typed text is now redacted by default when the screen shows a password prompt or cannot be read, and `audit-scan` finds any that was kept. The scan of the matron recording above found none.

**Limits.**
- Two boards, one Comet recording. The orderly run is the self-test log, which stops with a timeout traceback in the watchdog step: the script waited for a board that never came back.
- The cause is not proven on our boards. Research on 2026-10-05 found a U-Boot patch series from June and July 2026 that describes the same symptom on the RK3566 and RK3568 chips: the watchdog is routed to the chip's second global reset, which hangs the chip, and routing it to the first global reset fixes it; the author names an ODROID-M1S as a board where the watchdog then reboots cleanly. That is a mailing-list report, not something we tested, and its merge status was not found. Our own earlier guesses (the PMIC rails and the SD card are not power-cycled, or the boot ROM comes up in an unexpected state) are not ruled out. A serial console would settle it; an HDMI KVM cannot see the bootloader. The research note is `docs/research/m1s-watchdog-reset.md` in the repository.
- The dark period is bounded by the recording (it ended while the board was still dark), not by the board recovering.
- The rescue tests ran on orderly; matron's rescue card passed the same self-test on a separate run that is not included here.