Can't boot Pi 5 via NVMe behind PCIe switch/bridge unless NVMe is sole device.

[jreed1701]

Describe the bug
I am unable to boot a Raspberry Pi 5 from an NVMe SSD mounted to a POE M.2 HAT+ if used behind a PCIe switch AND that switch has a second non-SSD device attached to it. If the SSD is the sole device on the PCIe switch then it Pi 5 will boot.

To reproduce

1. Ensure you have and NVMe SSD that can boot the Raspberry Pi 5. (BOOT_ORDER=0xf25416)
2. Connect a PCI Express switch to the external PCIe connection on the Raspberry Pi 5.
3. Move the NVMe SSD you previously used dto boot the Pi 5 to a port behind teh PCIe switch, and simultaneously connect a second non-SSD device.
4. Attempt to boot the Raspberry Pi 5 off the NVME SSD with the second peripheral also reconized.

PCIe Switch Device is a Pi52 Quad B14 with an Asmedia ASM1184e switcher chip.

1. Asmedia: https://www.asmedia.com.tw/product/556yQ9dSX7gP9Tuf/b7FyQBCxz2URbzg0
2. 52Pi Quad PCIe HAT - https://www.asmedia.com.tw/product/556yQ9dSX7gP9Tuf/b7FyQBCxz2URbzg0

Expected behaviour

I would expect the NVMe SSD to be selecdted for boot even if the PCIe switch has an NVMe SSD and more than one additional peripheral device attached to it.

Actual behaviour

The Raspberry Pi 5 bootloader attempts to load the NVME when behind the PCIe switch as the sole device and suceeds, but fails if there is a second peripheral PCIe device connected to the PCIe switch. Hangs and eventually ends up in intrafms prompt.

System

• Which model of Raspberry Pi? - Pi5
• Which OS and version (cat /etc/rpi-issue)?
• Which firmware version (vcgencmd version)? - version 4b019946
• Which kernel version (uname -a)? 6.6.51-rpt-rpi-2712

Logs
No aparent errors in dmesg output. Output can be made available upon demand.

Additional context

To be clear, from Issue 1833, the fix does work. It's only when a second peripheral is introduced does the boot fail. The goal of this ticket is to ascertain why.

This issue is related to:

1. Can't boot Pi 5 via NVMe behind PCIe switch / bridge #1833
2. Can't boot Pi 5 behind PEX 8619 16-lane 16-port PCIe Switch #1927

[konsim83]

Hello, is there any news on that issue? It also happens with an ASM1182e switch. I am using this PCIe swtich together with this bundle. Please see the discussion around this comment in issue #1833

[peterharperuk]

From your description it sounds like Linux is having problems booting. What's the second non SSD device?

[konsim83]

The bundle in my previous comment consists of an nvme ssd + this Hailo 8L accelerator - sorry, I did not find the data sheet. This bundle is currently the only pcie board attached to the switch. The nvme is able to boot behind the switch if it is mounted on a pcie board like this one. Just mounted in conjunction with the AI accelerator it does not boot.
Hope that helps a bit.

[jreed1701]

From your description it sounds like Linux is having problems booting. What's the second non SSD device?

That is correct, it doesn't boot. Just plain old Raspbian. To clarify it's ANY second non SSD device. Could be an ethernet card, a USB hub. Doesn't matter.

[peterharperuk]

I need to know if the bootloader is actually the issue. If you boot off the sd card (boot order 1 is first) is Linux able to boot from nvme. This is a bit tricky as you need to get the partition id right on the sd card in cmdline.txt. If boot still fails then it's probably not a bootloader problem.

[jreed1701]

I need to know if the bootloader is actually the issue. If you boot off the sd card (boot order 1 is first) is Linux able to boot from nvme. This is a bit tricky as you need to get the partition id right on the sd card in cmdline.txt. If boot still fails then it's probably not a bootloader problem.

I had tried several boot orders and no matter what I seet BOOT_ORDER too in terms of sequencing mattered. If I had SD card before NMVe, noting, if I set it to be only NVMe, same thing. If there's a specific value you want me to check over again, please let me know.