Repository navigation
Conversation
bc0357d to
628c6e0
Compare
|
Status as of 2026-11-11: this stays a draft. JetPack 7.2.1 does not boot reliably on the bench PAB, and JetPack 6.2.2.8 on the same board does. Everything below is from one PAB with an Orin NX 16GB on a test station, flashed over USB with the debug console captured. What works The image builds and flashes. On the boots that came up, the bundle test's post-flash checks passed, and the Jetson half of the carrier test passed from a cold boot apart from its USB section. That section fails the same way on JetPack 6.2.2.8, so it is the fixture. What blocks
Fixed here One fault was ours. The SSH host key drop-in ran Why this waits The NVMe stall and the first-boot hang are in NVIDIA's R39.2.1 kernel and firmware. JetPack 7.2.1 came out on 2026-08-11 and NVIDIA has not announced the next release. The reset loop needs a look at the bench before we can say whose it is. Next
The real carrier test with the flight controller, the bundle test's flash step, the Argus relaunch test, and JAJ and PAB_V3 have not run. The branch is rebased onto main. |
JetPack 7 / kernel-noble dropped gpio_cansleep, changed i2c_driver.probe to take only the client, and made i2c_driver.remove return void. Match NVIDIA's nv_imx219 compat macros so the overlay still builds on 5.15 and 6.8.
R39 brings Linux 6.8 and an Ubuntu 24.04 rootfs, and changes enough of the BSP layout that the R36 build cannot stage or compile it: a Crosstool-NG toolchain, the kernel-noble source tree, a second GPU display tree, a new DTB output directory and a new headers path. The Noble rootfs reads module indexes with kmod 31 and the 22.04 build host ships kmod 29, so the container builds kmod 31 and every host now builds through it. The camera stack installs at the BSP stamp. JetPack 6 pinned it below the BSP for an Argus regression, and those debs do not install on 39.x. R39 generates SSH host keys with nvfb-ssh-keygen.service instead of nvfb.sh, so the nvfb.sh patch is dropped with nothing in its place yet. Co-authored-by: alexklimaj <alex@arkelectron.com>
Noble socket-activates sshd, so port 22 accepts connections before nvfb-ssh-keygen.service has written the host keys. An open port no longer proves the board is past the window where a power cut truncates them. Co-authored-by: alexklimaj <alex@arkelectron.com>
Noble packages the WiFi and Bluetooth firmware as .zst, which the stock defconfig's firmware loader cannot read. Co-authored-by: alexklimaj <alex@arkelectron.com>
The Noble rootfs needs depmod from kmod 31. Compiling kmod 31 into a 22.04 container worked, but it sent every 22.04 host through the container, CI included. NVIDIA supports 24.04 hosts for R39.2.1 and 24.04 ships kmod 31, so the host now matches the rootfs as it did on JetPack 6, and CI builds natively on a 24.04 runner.
v1.4.0 is the first ARK-OS release with an ark-os-jetson-noble package, which the JetPack 7 rootfs needs.
R39 builds UAS as a module and lists uas.ko in its initrd, so a USB rootfs on a UAS drive mounts without our override. Building it in removed the module, and nv-update-initrd then failed on the missing file while flash.sh generated its images.
R39's nvidia-preferred-oot-modules.conf will not load btusb while rtk_btusb is installed, so the Intel AX210 on the bundles had no Bluetooth driver and hci0 never appeared. modprobe runs the first install command it finds, so a drop-in that sorts ahead of NVIDIA's file overrides the rule without editing a file their package owns. Co-authored-by: alexklimaj <alex@arkelectron.com>
On R39 the PMIC RTC driver is a module. The kernel sets the system clock from it when it registers, after systemd-timesyncd has already restored the build time. A board whose RTC was never set went back to 1970, snapd rejected its signing keys as expired, and snapd.seeded held multi-user.target, gdm and jtop forever. timesyncd now starts after /dev/rtc0 exists.
On R39 cloud-init writes the host keys on the first boot and then disables itself, and nvfb-ssh-keygen.service is skipped while cloud-init is installed. A power cut before the keys reach the disk left zero-length files that nothing regenerated. The JetPack 6 fix patched nvfb.sh, which R39 no longer ships. ssh.service now runs ssh-keygen -A before it starts.
…that keeps UEFI output The line-based capture dropped everything UEFI prints, because UEFI redraws with carriage returns and leaves its prompts unterminated. That hid a module sitting in the UEFI shell behind what looked like a silent boot. The skill said flashing needs a human. A running Linux and the UEFI shell can both set the forced-recovery bit, and flash.sh runs unattended on a station.
The stock ssh.service runs sshd -t first, and that fails while no host key exists, so the drop-in's ssh-keygen -A never ran. A first boot that the watchdog reset before cloud-init's keys reached the disk came back with no keys at all: cloud-init had already marked its key generation done, sshd failed five times and ssh.socket gave up. The drop-in now clears the pre-start list and runs ssh-keygen -A ahead of sshd -t.
Main reworded this page while the branch rewrote three of its sections for the Ubuntu 24.04 host. The rebase kept the branch's text there, which left those sections in the old style beside the reworded ones. The cache key example also still named R36.5.0.
NVIDIA's UEFI serves its variables from OP-TEE. A runtime call that releases an OP-TEE lock cannot wake a Linux OP-TEE call waiting on it, because OP-TEE sends the wake-up through the UEFI client, which drops it. On R39.2.1 the fTPM loads while nv-l4t-bootloader-config writes a UEFI variable, and first boots hung there until the watchdog reset the board. A runtime call now waits for the OP-TEE calls in flight and holds new ones off until it returns. Changes to the kernel's own files go in kernel_patches/, which build.sh applies and will not build without.
On R39.2.1 an NVMe completion interrupt can arrive before the completion it announces is visible in memory: the Tegra root port raises an MSI inside the controller, ahead of the device's earlier writes. The handler found nothing, no other interrupt came, and the command sat until the I/O timeout. Early in boot that stalled systemd for 30 s at a time. The NVMe handler now reads CSTS and looks again when it finds nothing. One reader on the bench PAB lost an interrupt about once per 700 GB before; with the fix it read 4.5 TB without one.
2122326 to
3055187
Compare
Nothing in the image uses snaps. On an offline first boot snapd spends about 45 s timing out on the Snap Store while gdm and jtop wait for it, and it was what turned the RTC's 1970 clock into a boot that never finished. The firefox and thunderbird packages are only stubs that install their snaps.
R39 indexes the core clock table with speed - 1 after the table gained a dummy first entry, so every controller runs one generation slow. The SSD's Gen3 x4 link runs at 125 MHz instead of 250 MHz, which leaves no margin. R36 and mainline index it by the speed.
Whether the core clock drives the datapath is unverified, so the claim that 125 MHz leaves no margin is dropped.
R39 takes PCIe MSIs in each controller's DesignWare receiver, which raises an MSI ahead of the device's earlier memory writes. An NVMe completion can then be missed and its command waits out the 30 s timeout. The receiver also sends every MSI to CPU 0. JetPack 5 sent them to the GIC, and this restores that: NVIDIA's R35 code as ported to R39 on the NVIDIA forum, with every host controller routed through the frame.
The re-check read CSTS when an interrupt found no completion, to catch one that the MSI had overtaken. With MSIs through the GIC's v2m frame they no longer arrive ahead of the data: stock NVMe read 13.6 TB on the PAB without a timeout, where the internal receiver gave one per 324 GB.
Without #address-cells on the GIC, UEFI's PCIe driver reads the PCIe interrupt maps with two parent address cells, finds none, asserts and reboots: the first flash of the GICv2m image never came back. The GIC now has two address cells, as mainline describes a v2m frame under its GIC, and the stock PCIe interrupt maps are restated with them.
…amera stamp The BSP already installs the camera and multimedia packages, so provisioning downloaded three debs that apt then found installed. With JetPack 6's pin gone the stamp is the BSP's own, so it is read from the installed camera package rather than kept in step by hand in versions.env.
ssh.service now regenerates missing or empty host keys when it starts, so losing power during the first boot no longer leaves a unit whose sshd never starts.
0002 went when GICv2m replaced the NVMe change. build.sh matches a patch by its content, so a staged tree reports the renamed patches as applied.
They are bench tooling, not part of the JetPack 7 move.
R39's stock camera stack runs the page's repro clean. It ran 72 launches on each release candidate, then 300 alternating launches, with no dirty teardown, no failed launch and no nvargus-daemon crash. 36.5.0 left nearly every teardown dirty and crashed the daemon about once in 96 launches. The jetpack6 branch keeps the page with its pin.
A tree that has flashed a board keeps bootloader/system.img and the sparse system.img.raw. tar packed both, and the raw image extracts at its full 59 GB, so a package made from such a tree needed 85 GB free to unpack. The flasher writes both again from the rootfs at flash time.
When the GPU first powers on inside an open() of its device, nvgpu loads its firmware inside that open's path walk, and nested lookups share its 40-symlink budget. NVIDIA's firmware_class.path=/etc/firmware does not exist on R39, so every load fell through the /lib/firmware paths across /lib -> usr/lib, the ACR firmware failed with ELOOP, and the GPU stayed down until reboot. With something opening the GPU before nvpmodel, that happened on 8 of 8 boots.
nvidia-jetpack pulls in nvidia-container-toolkit-base, whose nvidia-cdi-refresh runs early in boot. It failed on 5 of 5 reboots with the GPU driver not ready, and its NVML probe brought the GPU up before nvpmodel set the power-gating masks, failing nvpmodel. Ordered after nvpmodel, both passed on 10 of 10.
jetson-stats 4.3.2 has no entry for L4T 39.2.1, so jtop and the ARK UI show an empty JetPack version. Version 7 installs from a wheel, which never runs its service installer, so provisioning now runs `jtop --install-service` itself.
jetson-stats 7 opens the GPU through NVML on JetPack 7, and jtop started before nvpmodel. On boots where nvpmodel has to apply a mode, such as the first, nvpmodel then failed with status 234 because the GPU's golden context already existed. jtop also reported no power mode for the whole first boot, because nvpower.sh had not yet linked /etc/nvpmodel.conf when jtop read it.
R39's rootfs has no /etc/machine-id, so systemd treats the first boot as one and enables every unit no preset excludes. That started an iperf3 server and saned listening on every interface, and systemd-networkd beside NetworkManager, which NVIDIA's own preset disables. saned@.service carries Also=saned.socket, so the template needs its own line.
v1.5.0 encodes the camera with NVENC, installs jetson-stats 7.3.0 and starts jtop after nvpmodel, and keeps its opt-in services off through JetPack 7's first-boot preset pass.
Summary
Moves the build from JetPack 6.2.2 (L4T R36.5.0) to JetPack 7.2.1 (L4T R39.2.1): Linux 6.8 and an Ubuntu 24.04 rootfs. This is the JetPack half of #109, without its fast-boot work. It includes #162, which fixes R39.2.1's OP-TEE boot hang and NVMe stalls.
Draft until the PAB's full carrier and bundle tests and the JAJ and PAB_V3 bring-up have run. See Open.
Problem
#109 carries the JetPack 7 migration and the fast-boot port in one change, so neither can be tested or merged alone.
Solution
The tree builds one BSP, so the R36 paths are replaced instead of kept beside the R39 ones. JetPack 6 continues on a
jetpack6branch, to be cut from main immediately before this merges.The build host moves to Ubuntu 24.04 to match the rootfs, which keeps the host's kmod the same as the device's. CI builds natively on a 24.04 runner.
Provisioning installs
ark-os-jetson-noblefrom ARK-OS v1.5.0. It carries the jetson-stats and preset changes below, and streams through the hardware H.264 encoder.JetPack 6's camera stack pin and its regression page go. R39's stock camera stack passes the relaunch test that found the regression.
Five R39 regressions showed up on a PAB with an Orin NX 16GB and are fixed here:
uas.koand our config built UAS in.btusbout whenever their Realtek driver is installed.multi-user.target.open()of its device, nvgpu loads its firmware inside that open's path walk, and the kernel's nested lookups share one 40-symlink budget. NVIDIA'sfirmware_class.path=/etc/firmwaredoes not exist on R39, so every load crossed/lib -> usr/liband the ACR firmware failed with ELOOP. The firmware path is now/usr/lib/firmware. https://forums.developer.nvidia.com/t/intermittent-missing-gpu-devfreq-node-on-cold-boot-jetson-agx-orin-jetpack-7-2-l4t-39-2/384170nvfb.sh, which R39 does not ship. A drop-in now runsssh-keygen -Aahead of the stocksshd -t.Once a customer installs
nvidia-jetpack, NVIDIA's container toolkit startsnvidia-cdi-refreshearly in boot. It fails while the GPU driver is not ready, and it brings the GPU up before nvpmodel can set the power-gating masks, failing nvpmodel. A drop-in orders it after nvpmodel. https://forums.developer.nvidia.com/t/jetson-r39-2-package-ships-stale-nvidia-cdi-refresh-service/377209The flash package leaves out
bootloader/system.imgand its sparse raw image. A tree that has flashed a board keeps them, and the raw one extracts at 59 GB.jetson-stats moves from 4.3.2 to 7.3.0. 4.3.2 has no entry for L4T 39.2.1, so jtop and the ARK UI showed an empty JetPack version. Version 7 installs from a wheel, which never runs its service installer, so provisioning runs
jtop --install-service. On JetPack 7, version 7 also opens the GPU through NVML, so a drop-in starts jtop after nvpmodel. Started first, jtop created the GPU's golden context, and nvpmodel then failed with status 234 when it had to apply a mode. On a first boot jtop also found no/etc/nvpmodel.conf, whichnvpower.shlinks, and reported no power mode for the boot. ARK-Electronics/ARK-OS#127 moves its pin and installer to match.R39's rootfs has no
/etc/machine-id, so systemd treats the first boot as one and enables every unit no preset excludes. On a flashed image that started an iperf3 server and saned listening on every interface, and systemd-networkd beside NetworkManager, which NVIDIA's own preset disables. A preset frombuild.shkeeps them off. ARK-Electronics/ARK-OS#129 does the same for ARK-OS's opt-in services, which also came up running. The pass also enablesssh.serviceandnv-ftpm-device-provision, which every bench result here ran with, so it stays.Provisioning purges snapd, with the firefox and thunderbird stubs that only install their snaps. Nothing in the image uses snaps, and on an offline first boot gdm and jtop waited about 45 s for snapd to give up on the Snap Store.
Two R39.2.1 faults need kernel patches, which
build.shapplies from the newkernel_patches/and stops on if one no longer applies:completion polled, 15 times in 4.9 TB read and early in boot. R39 takes PCIe MSIs in each controller's DesignWare receiver, which raises an MSI ahead of the SSD's earlier memory writes, so the handler can find no completion and get no further interrupt. The receiver also sends every PCIe MSI to CPU 0. MSIs now go to the GIC's v2m frame, as on JetPack 5, for every host controller. A Tegra PCIe device cannot write to the GIC itself, so the controller rewrites the MSI into a window at0x54000000that the IOMMU maps, one page per SPI. That is NVIDIA's R35 code as ported to R39 on the NVIDIA forum, with two fixes, plus the frame and anmsi-mapper controller in each product's device-tree overrides. Others report the stall on R39.2.1: https://forums.developer.nvidia.com/t/nvme-timeout-completion-polled/380388A third patch fixes R39's PCIe core clock. NVIDIA's tree indexes the clock table with the link speed minus one, so every controller runs one generation slow. R36 and mainline index it by the speed. On the PAB's SSD the fix changes neither throughput nor latency, and it does not stop the NVMe stalls.
Bench results on that PAB, on a test station, from full builds with provisioning. A rebuild with this head's provisioning installed the same 2,054 packages at the same versions, and passed the first-boot, host key, carrier and Argus checks again:
pci=nomsi, every device falls back to INTx through the restated interrupt maps.generate_flash_package.shand flashed byflash_from_package.shfrom its cache, answered SSH 33 s into its first boot. A reflash from the cache took the--flash-onlypath in about 5 minutes.apt install nvidia-jetpackinstalled 7.2.1 with CUDA 13.2, and a CUDA program built on the device ran correctly.apt upgradethen upgraded 398 Ubuntu packages and kept every boot-chain hold. The upgraded image booted with the same kernel, GICv2m, CUDA and Argus working./usr/lib/firmwareon 0 of 8. Withnvidia-jetpackinstalled,nvidia-cdi-refreshfailed on 5 of 5 reboots and passed on 10 of 10 with the drop-in.rtsp-serverand go2rtc served 1080p30 H.264 to another machine for 30 and 10 minutes without errors.docs/gpio.md./etc/nvpmodel.confand nvpmodel's state removed, as on a first boot, the stockjtop.serviceleft the power mode unknown on 6 of 8 and nvpmodel failed on 4 of 8. With the drop-in, 0 of 8. The image's real first boot read 40W with nvpmodel passing, and verify passed./etc/machine-iddeleted to recreate a first boot, iperf3, saned, rsync, the networkd socket and ARK-OS's six opt-in services came back enabled without the presets, and stayed off with both. Only ssh, nginx, rtsp-server and go2rtc then listen on every interface.l2pinglost nothing. The hotspot pull ran at 0.3 to 0.9 MB/s, and JetPack 6.2.2.8 on the same board pulled 0.67 MB/s, with the hotspot sharing channel 1 with an office AP.flash_from_package.shflashed the releasedpab-6.2.2.8package from its cache, and it booted in 46 s.pcie4w: write @0x000000000f410040: EMEM address decode error, so the Tegra rewrite cannot be avoided.The power-on reset loop reported earlier was the bench, not JetPack 7. The fixture ran from one of its three supply rails and browned out on USB load steps.
Open: