Repository navigation
fix(kernel): fix the R39.2.1 OP-TEE boot hang and NVMe stalls - #162
Merged
Merged
Conversation
NVIDIA's UEFI serves its variables from OP-TEE. A runtime call that releases an OP-TEE lock cannot wake a Linux OP-TEE call waiting on it, because OP-TEE sends the wake-up through the UEFI client, which drops it. On R39.2.1 the fTPM loads while nv-l4t-bootloader-config writes a UEFI variable, and first boots hung there until the watchdog reset the board. A runtime call now waits for the OP-TEE calls in flight and holds new ones off until it returns. Changes to the kernel's own files go in kernel_patches/, which build.sh applies and will not build without.
On R39.2.1 an NVMe completion interrupt can arrive before the completion it announces is visible in memory: the Tegra root port raises an MSI inside the controller, ahead of the device's earlier writes. The handler found nothing, no other interrupt came, and the command sat until the I/O timeout. Early in boot that stalled systemd for 30 s at a time. The NVMe handler now reads CSTS and looks again when it finds nothing. One reader on the bench PAB lost an interrupt about once per 700 GB before; with the fix it read 4.5 TB without one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the two JetPack 7 faults found on the bench PAB: the boot hang in OP-TEE and the 30 s NVMe stalls. Stacked on #157.
Problem
On L4T R39.2.1 the first boot after a flash hung on 3 boots in 5, and the fTPM module load hung once in 8 warm boots. NVIDIA's UEFI serves its variables from OP-TEE. When a UEFI runtime call releases an OP-TEE lock that a Linux OP-TEE call is waiting on, OP-TEE sends the wake-up through the UEFI runtime driver, which drops it. The Linux call sleeps for good, and the next runtime call spins in the firmware until the watchdog resets the board.
nv-tee-supplicant.serviceloads the fTPM just asnv-l4t-bootloader-configwrites a UEFI variable.NVMe commands also waited out the 30 s I/O timeout, logged as
I/O tag N QID M timeout, completion polled, early in boot and under load. The Tegra root port raises an MSI inside the controller, ahead of the memory writes the SSD sent before it. The NVMe interrupt can arrive before its completion entry is visible, the handler finds nothing, and no other interrupt follows. Drivers that read a device register in their handler are safe, because the read cannot pass the device's earlier writes. NVMe's handler reads only host memory.Solution
Two kernel patches, applied from the new
kernel_patches/bybuild.sh, which stops if one no longer applies:Bench results on the PAB with an Orin NX 16GB:
The power-on reset loop reported in #157 was the bench supply, not JetPack 7.
On a flash of this branch: