Unattended OS install over a KVM
Claim. An agent with only a GL.iNet Comet and ssh configured a real machine's BIOS and installed Debian 13 onto its SSD, booted it and verified it (21 of 26 checks first time), then ran a second install onto an SD card, with no one at the keyboard, no install stick and no sudo. Every screen, keystroke and OCR reading was recorded. The SD card first held a Debian rescue system, booted two ways (F11 menu, and a GRUB entry on the first system) and checked. It was later rewritten with the fleet's SystemRescue card, which was boot-tested in QEMU, booted once on the real machine through a one-shot GRUB entry, and checked over ssh. A deliberate kernel panic reached the receiving machine as text. The machine hung at the end of the second install with its power light on, and only a cold boot by a person recovered it, which is itself a finding. A second machine (section below) later booted the same SystemRescue card through the Comet. Two videos of these sessions are on the home page.
Method. Target: Foxconn NetBox NT-425/525 (Atom D525, 2 GB, legacy AMIBIOS v02.68 dated 2009, 60 GB SATA SSD, 8 GB SD card in the built-in reader). Comet firmware RM1 V1.10.1 release3, kvmd 4.82, 1280x720. Date 2026-10-01. The agent was Claude Code on a second machine on the same LAN, using bootscry from this repository (branch weirdo-debian-install). The steps, in order:
- Upload the Debian 13.7 netinst ISO to the Comet's storage (
msd upload), attach it as a DVD (msd mount). - Reboot, tap Delete through POST, walk every BIOS tab with arrow keys, change four settings, set the DVD first, save with F10.
- Reboot into the DVD. Stop the boot menu's countdown, clear the boot line, type
auto=true url=http://<haywire>:8099/p, press Enter. - Answer the three questions that come before the preseed loads (interface, hostname, domain); the rest is the preseed file.
- When the installer menu reappears after the final reboot, take the DVD away and reboot into the disk.
- Repeat from step 3 with a second preseed that targets only the SD card (matched by reader model and size), for a Debian rescue system.
- With root granted by the operator for that one machine, turn on netconsole, keys-only ssh and a GRUB entry for the rescue card.
- Build the fleet's SystemRescue card as an image on a loop device, boot-test that image in QEMU (legacy BIOS, an old Core 2 CPU model, read-only) and fix what the test finds, then write the card once.
- Boot the real machine from the card through the one-shot GRUB entry (
reboot-to-rescue), check it over ssh. - Trigger a kernel panic on purpose with
panic=10set and read what the receiver on the LAN got.
The screens are in docs/evidence/weirdo-install/shots/ (numbered, each with its OCR text), the action log in audit.jsonl, the readable story in transcript.md.
Sample size. One machine, one session, counted from audit.jsonl on 2026-10-02: 839 logged entries, of them 509 frames, 45 no-signal gaps, 244 actions (keys, key combinations, typed lines, repeats, and the ISO upload and mount) and 13 notes, over 7 h 23 min between the first and last entry (2026-10-01 17:26 UTC to 2026-10-02 00:49 UTC), most of it waiting on package downloads on a 2010 Atom and on this agent's own mistakes below. The last 9 entries are no-signal gaps: see the limits.
Result.
| Measured | Value |
|---|---|
| ISO upload to the Comet | 792,723,456 bytes in 61.3 s; the target saw an optical drive of exactly 792,723,456 bytes |
| Reached BIOS setup | third attempt: the first run crashed on the no-signal gap during POST (see below); the second resumed after POST was over and tapped Delete into the OS login prompt |
| BIOS settings changed through the KVM | 4, plus boot order; all read back on the same screens before saving |
| Installer started with a typed boot line | on the 3rd typed line; the first two were ignored, and two unplanned boots into the speech-synthesis installer came from missed countdowns (all listed below) |
| Preseed fetched from the target | 1 request, HTTP 200, from the target's address |
| Main system install | unattended after 3 prompts; SSD partitioned by model, not by name; 330 MB more RAM visible than under the old Ubuntu (it reserved memory for a crash kernel) |
| First rescue system (Debian 13) | booted from the SD card via the F11 menu and via the main system's GRUB entry; identified as weirdo-rescue on /dev/sdc1 by ssh |
| BIOS settings after a cold boot | every change read back on screen, intact |
| SystemRescue card, QEMU boot test | 11 PASS lines in rescue-qemu-rehearsal/report.txt: boots and takes the ssh key, label RESCUE1302, 16 keys installed, keys-only root login, watchdog running, all six repair tools present, mount-estate-disks finds a stand-in host root, password login refused. One more line, the netconsole receiver in QEMU, is informational only: QEMU's NAT does not carry netconsole frames. The same script found four defects in the card's builder while its file-level tests were green: ssh still offered passwords, the watchdog check looked for a wrong process name, root detection skipped a filesystem with no partition table, and netconsole lacked the local address |
| SystemRescue card on the real machine, one-shot | one GRUB entry, then the machine ran kernel 6.18.41-1-lts from /dev/sdc1 (rescue-on-real-weirdo.txt): 16 keys, keys-only root ssh, the watchdog, all six tools, mount-estate-disks mounted the host root (/dev/sda2, ext4), estate-diag ran. Read over ssh, not off the screen |
| Panic test | sysrq crash with panic=10 on the kernel line; panic-test-netconsole.txt holds the text the receiver got: the boot log replayed, then Kernel panic - not syncing: sysrq triggered crash, the call trace and Rebooting in 10 seconds. The next verify run (verify-7-after-netconsole-fix.txt) reported the machine up 0 minutes with 30 of 30 checks passing |
| Checks on the finished system, final run | verify-8-systemrescue-sd.txt: 32 PASS lines and no FAIL line, including that the SD card carries the fleet rescue |
Checks on the finished system (scripts/weirdo/verify.sh in the network repo) | 21 of 26 passed on the first run; 27 of 27 after the hostname repair and the check fixes. Of the 5 failures, 3 were mistakes in the checks (wrong PATH for non-root swapon and smartctl, a variable not exported, an unreadable grub.cfg), 1 was timing (clock still syncing), and the hostname was a real defect not yet covered by a check |
What went wrong, and what bootscry does about it now. Each row was seen on this run; the fix is in the tested code.
| What happened | Cause | Fix |
|---|---|---|
| Recorder crashed mid-POST | the Comet returns 503 for seconds while the target restarts its display | Recorder.shot logs no_signal and carries on |
Client crashed with AttributeError: 'NoneType' object has no attribute 'settimeout' | the KVM closed the kept-alive connection; http.client sets sock to None | _raw reopens the connection; test added |
| Attaching the ISO "failed" though it had attached | the Comet reports drive.image as an object, not a name | msd_mount accepts both; test uses the object form |
| Installer menu timed out into "Install with speech synthesis" (blank screen, keyboard grabbed, Ctrl+Alt+Del dead) | tapping Shift did not stop the countdown; only a real keystroke does | tap ArrowUp (no effect at the top of the menu); recovered with Alt+SysRq+B |
Installer ignored auto url=... | plain auto is not auto=true; and parameters after --- are for the installed kernel | auto=true url=... placed before any --- |
Boot line came out as http://198099/p | a BIOS-era keyboard buffer drops characters from fast typing | type_text(slow=True), then read the line back from the screen before pressing Enter |
| Boot line stopped after 65 characters | the boot loader's edit field has a length limit | short preseed path /p; Ctrl+U clears the field in one key |
| Installer asked for the disc back at "Finishing the installation" | the agent ejected the virtual DVD too early | eject only when the installer menu reappears after the reboot; install_guard.py does it |
Hostname became weirdoweirdo | the hostname prompt arrives before the preseed, pre-filled from DHCP; the agent typed over a filled field | set the name in the post-install script; repaired the first system from the installer's own root shell on VT2 |
| Rescue install stopped on a swap question | the rescue recipe has no swap | partman-basicfilesystems/no_swap in the preseed |
| The GRUB step failed twice on the rescue install | the package list asked for grub-efi-amd64-bin, whose dependencies (shim-signed, grub-efi-amd64-signed) block the BIOS bootloader step | removed from the list, installed after GRUB in the post-install script; repaired live by purging the packages from the installer's VT2 shell and re-running the step |
| Post-install script hung for 40 minutes | apt-get inside it waited for the install CD through apt's cdrom method | the script now comments the deb cdrom line and wraps apt in timeout 300; the stuck process was killed from VT2 |
| The kill did nothing, twice | the agent read the process IDs from OCR: 14041 and 14046 came back as 14641 and 14646 (the console font's 0 reads as 6) | checked the digits against the picture; digits from OCR are never used unverified |
| The machine stopped answering after "Unmounting file systems" | it was hung with its power light on (operator, in person): no video, no DHCP in 2 h, no reply to ping, a USB key, the USB Power key or Wake-on-LAN. Wake-on-LAN could not have worked anyway: the BIOS had it disabled in a menu that only appears once Deep Sleep is off | a cold boot by a person; a power accessory on the Comet's USB-A port (ATX board or Fingerbot) would have done it remotely; Wake-on-LAN is now enabled in the BIOS |
Text channels next to the screen. OCR garbled digits and identifiers throughout this run: process IDs (14041 read as 14641, which made two kill commands hit nothing), an IP octet (88 read as 86), and a disk UUID (8cf2644b-4be6b-4286f -9387-..., shots/0003-*.txt). Two non-OCR channels were built and tried, and the second was later proven by the panic test above:
| Channel | Result |
|---|---|
The installer's syslog, streamed as TCP lines to the agent's machine (tail -F /var/log/syslog piped to nc <host> 6667 in the installer's shell; its busybox nc has no UDP) | 401 lines of the installer's log arrived byte-exact, kernel USB IDs included; it is now in both preseed files as preseed/early_command |
Linux netconsole (UDP, extended format with sequence numbers) received by bootscry.netcon.NetConsole | the receiver is built (24 tests, a deliberate break caught) and running as a service; the target side needs root; it was switched on later in the session and carried the panic test |
What no text channel reaches on this machine: the BIOS, the boot loader menu and the installer's first screens. Its BIOS has no serial redirect and the Comet exposes no serial endpoint (/api/serial is 404, GPIO has no outputs), so those stay on OCR, with the picture as the check.
What bootscry gained from it. msd (upload, mount, eject, remove, with size verification and progress), Recorder (numbered frames, OCR text, audit log, transcript, typed text redacted), phase.classify (post, bios-setup, bootloader, installer, kernel-boot, login, tested on 15 real frames), reboot-to --expect PHASE (retry until the screen is where it should be), netcon (kernel and installer logs as exact text, over UDP or TCP, one file per sending machine, lost-message detection), the video tool (kvm_tool.py video, turns a recorded session into a captioned video: near-duplicate frames folded into a held frame shown for at most 1.6 s, chapters from the notes, result cards, addresses and host names that OCR finds are painted out) and the fixes above. 204 tests, all against the mock or local sockets.
Second machine. On 2026-10-02 the same card was booted on psycho, the Comet's usual test machine, through the Comet only (record in docs/evidence/psycho-rescue/: audit.jsonl, transcript.md, rescue-on-psycho.txt; the frames are not published because they show LAN addresses). The run took 14 min 12 s from the first to the last entry: 109 entries, 25 frames, 77 key presses, 1 key combination and 6 notes. The agent restarted the installed Ubuntu with Ctrl+Alt+Del, tapped Delete into the BIOS, walked the Advanced, Chipset, Security and Boot tabs read-only, and opened the Save and Exit tab's boot override without changing a setting. It picked the card from that list and took the default entry. Per the session account, the agent then reached the card by ssh with its 16 mesh keys and ran estate-diag and mount-estate-disks; the saved output (rescue-on-psycho.txt) shows the card's kernel 6.18.41-1-lts, the watchdog running, seven estate hosts answering ping, and the machine's Ubuntu LVM root mounted at /mnt/target. It ended with reboot-to-os from the card, and the last frame is the installed system's login prompt. The files record the tool output but not the ssh login itself. This run has no verify PASS count: no verify script was run on it.
The video. weirdo-install.mp4 (also .webm and a poster, in site/assets/) was rendered from this record by kvm_tool.py video: 5 min 15 s and 6.6 MB. Of the 509 frames, 341 were near-duplicates of the frame before and were folded into it (the video keeps 168 and holds a frame on screen for at most 1.6 s), with 244 actions, 13 chapters from the notes and 7 misses kept in and labelled. Addresses and host names that OCR finds in a frame are painted out. The second machine's video, psycho-rescue.mp4, is 56 s. No secret was typed in the clear on either machine and the installer's password was never typed; the frames were not otherwise checked for secrets beyond the address and host-name painting.
Limits.
- One machine and one BIOS. The typing losses are an observation, not a rate: both fast attempts at the boot line were damaged (one cut at 65 characters, one lost about a quarter of its characters) and all four slow attempts were exact.
- OCR misreads digits on small bootloader text (
88.71read as86.71); the agent checked the typed line by looking at the picture, not the OCR. - The Comet's virtual media needs a running streamer and a USB port that the target enumerates; nothing here covers a target without USB boot.
- The BIOS cannot fall back from a dead SSD to the SD card by itself (the card is not in the priority list, only in the F11 menu and the hard-disk list); that was read from the BIOS screens, not tested by breaking the SSD.
- The last 9 entries of the weirdo record (the one-shot boot of the rescue card) are no-signal gaps after two kernel-boot frames: the record ends without a login prompt, so the weirdo video closes on the last login prompt on record, not on a clean final take. What the card did on the real machine is shown by the ssh output, not by those frames.
- The QEMU boot test cannot check netconsole delivery, and no QEMU result stands for real hardware; the one-shot run covers that on one machine.
- The panic test was run once, on one kernel (6.12.111). The return to service is inferred from the next verify run reporting 0 minutes of uptime, not from a recorded reboot frame.
- The root shell used once (installer VT2) is a property of Debian's installer, not of the KVM.
Raw data. docs/evidence/weirdo-install/ (frames, audit.jsonl, transcript.md, verify-*.txt, panic-test-netconsole.txt, rescue-on-real-weirdo.txt, rescue-qemu-rehearsal/); the second machine is in docs/evidence/psycho-rescue/; scripts that drove it: scripts/weirdo/ in the network repository.