Skip to content

fix(kernel): fix the R39.2.1 OP-TEE boot hang and NVMe stalls - #162

Merged
dakejahl merged 2 commits into
dakejahl/jetpack-7from
dakejahl/jetpack-7-boot-fixes
Nov 16, 2026
Merged

dakejahl merged 2 commits into
dakejahl/jetpack-7from
dakejahl/jetpack-7-boot-fixes

Conversation

@dakejahl

@dakejahl dakejahl commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Fixes the two JetPack 7 faults found on the bench PAB: the boot hang in OP-TEE and the 30 s NVMe stalls. Stacked on #157.

Problem

On L4T R39.2.1 the first boot after a flash hung on 3 boots in 5, and the fTPM module load hung once in 8 warm boots. NVIDIA's UEFI serves its variables from OP-TEE. When a UEFI runtime call releases an OP-TEE lock that a Linux OP-TEE call is waiting on, OP-TEE sends the wake-up through the UEFI runtime driver, which drops it. The Linux call sleeps for good, and the next runtime call spins in the firmware until the watchdog resets the board. nv-tee-supplicant.service loads the fTPM just as nv-l4t-bootloader-config writes a UEFI variable.

NVMe commands also waited out the 30 s I/O timeout, logged as I/O tag N QID M timeout, completion polled, early in boot and under load. The Tegra root port raises an MSI inside the controller, ahead of the memory writes the SSD sent before it. The NVMe interrupt can arrive before its completion entry is visible, the handler finds nothing, and no other interrupt follows. Drivers that read a device register in their handler are safe, because the read cannot pass the device's earlier writes. NVMe's handler reads only host memory.

Solution

Two kernel patches, applied from the new kernel_patches/ by build.sh, which stops if one no longer applies:

  • A UEFI runtime call waits for the Linux OP-TEE calls in flight and holds new ones off until it returns. The wait is bounded at ten seconds.
  • When an NVMe interrupt finds no completion, the handler reads CSTS and looks again.

Bench results on the PAB with an Orin NX 16GB:

  • Loading and unloading the fTPM beside UEFI variable traffic hung the stock kernel in both of two one-minute runs. The patched kernel ran three runs with no hang. Two flashes with the OP-TEE patch reached SSH on the first boot, after 28 and 33 s.
  • With the PCIe and GIC registers read during five stalls, no interrupt was pending or in flight anywhere, so each had already been taken with nothing to find. A build that logged each time the extra read made a completion visible logged it during the flash and 29 times in the first minute of boot.
  • One sequential reader lost an interrupt about once per 700 GB on the stock kernel. With the fix it read 4.5 TB without one.

The power-on reset loop reported in #157 was the bench supply, not JetPack 7.

On a flash of this branch:

  • 15 of 15 boots clean, 10 warm and 5 cold, each about 9.3 s, with the fTPM loading every time.
  • The bundle checks pass on the second boot, and the storage flush passes.
  • The Jetson half of the carrier test passes from a cold boot, except the fixture's bottom SuperSpeed port, which JetPack 6 also fails.
  • The Argus relaunch test ran 72 of 72 launches clean, so R39's camera stack needs no pin.

NVIDIA's UEFI serves its variables from OP-TEE. A runtime call that releases
an OP-TEE lock cannot wake a Linux OP-TEE call waiting on it, because OP-TEE
sends the wake-up through the UEFI client, which drops it. On R39.2.1 the
fTPM loads while nv-l4t-bootloader-config writes a UEFI variable, and first
boots hung there until the watchdog reset the board.

A runtime call now waits for the OP-TEE calls in flight and holds new ones
off until it returns. Changes to the kernel's own files go in
kernel_patches/, which build.sh applies and will not build without.
On R39.2.1 an NVMe completion interrupt can arrive before the completion it
announces is visible in memory: the Tegra root port raises an MSI inside the
controller, ahead of the device's earlier writes. The handler found nothing,
no other interrupt came, and the command sat until the I/O timeout. Early in
boot that stalled systemd for 30 s at a time.

The NVMe handler now reads CSTS and looks again when it finds nothing. One
reader on the bench PAB lost an interrupt about once per 700 GB before; with
the fix it read 4.5 TB without one.
@dakejahl dakejahl changed the title fix(kernel): keep UEFI runtime calls and OP-TEE calls apart fix(kernel): fix the R39.2.1 OP-TEE boot hang and NVMe stalls Nov 12, 2026
@dakejahl
dakejahl merged commit 2122326 into dakejahl/jetpack-7 Nov 16, 2026
1 check passed
@dakejahl
dakejahl deleted the dakejahl/jetpack-7-boot-fixes branch November 16, 2026 21:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant