Bug Report: Unable to Boot from NVME on Orange Pi 5 v 3.2

[tim273]

What happened?

Trying to boot from NVME, I followed the commands on the wiki

sudo u-boot-install-mtd
sudo ubuntu-rockchip-install /dev/nvme0n1

Then when I run fdisk -l I see this:

Disk /dev/mtdblock0: 16 MiB, 16777216 bytes, 32768 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 5F83A195-5CAC-4127-B7DC-6CAC436E1235

Device           Start   End Sectors  Size Type
/dev/mtdblock0p1    64  7167    7104  3.5M Linux filesystem
/dev/mtdblock0p2  7168  7679     512  256K Linux filesystem
/dev/mtdblock0p3  7680  8063     384  192K Linux filesystem
/dev/mtdblock0p4  8064  8127      64   32K Linux filesystem
/dev/mtdblock0p5  8128  8191      64   32K Linux filesystem
/dev/mtdblock0p6  8192 16383    8192    4M Linux filesystem
/dev/mtdblock0p7 16384 32734   16351    8M Linux filesystem


Disk /dev/mmcblk1: 117.75 GiB, 126437294080 bytes, 246947840 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 92F34E70-349D-43DB-B27D-C0F3FBFFB4B2

Device         Start       End   Sectors   Size Type
/dev/mmcblk1p1 32768     40959      8192     4M Microsoft basic data
/dev/mmcblk1p2 40960 246947806 246906847 117.7G EFI System


Disk /dev/nvme0n1: 238.47 GiB, 256060514304 bytes, 500118192 sectors
Disk model: KBG50ZNS256G NVMe KIOXIA 256GB          
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 2A50CF80-2A7E-4154-A385-1E7E47964961

Device         Start       End   Sectors   Size Type
/dev/nvme0n1p1 32768     40959      8192     4M Microsoft basic data
/dev/nvme0n1p2 40960 500117503 500076544 238.5G EFI System

I power off, remove the microSD card and nothing. Then when I put the microSD card back again and run fdisk -l the NVME drive disappears:

Disk /dev/mtdblock0: 16 MiB, 16777216 bytes, 32768 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 5F83A195-5CAC-4127-B7DC-6CAC436E1235

Device           Start   End Sectors  Size Type
/dev/mtdblock0p1    64  7167    7104  3.5M Linux filesystem
/dev/mtdblock0p2  7168  7679     512  256K Linux filesystem
/dev/mtdblock0p3  7680  8063     384  192K Linux filesystem
/dev/mtdblock0p4  8064  8127      64   32K Linux filesystem
/dev/mtdblock0p5  8128  8191      64   32K Linux filesystem
/dev/mtdblock0p6  8192 16383    8192    4M Linux filesystem
/dev/mtdblock0p7 16384 32734   16351    8M Linux filesystem


Disk /dev/mmcblk1: 117.75 GiB, 126437294080 bytes, 246947840 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 92F34E70-349D-43DB-B27D-C0F3FBFFB4B2

Device         Start       End   Sectors   Size Type
/dev/mmcblk1p1 32768     40959      8192     4M Microsoft basic data
/dev/mmcblk1p2 40960 246947806 246906847 117.7G EFI System

ubuntu@ubuntu:~$ ls /dev/nvme*
ls: cannot access '/dev/nvme*': No such file or directory

Sometimes it's there after this, but sometimes it's not. But if it's not there and I use parted to remove all the partitions on mtdblock0 and boot with the microSD card, it appears again.

Kernel version

6.1.0-1025-rockchip

SBC model

Orange Pi 5

What operating system are you seeing this problem on?

Ubuntu 24.04 LTS (Noble Nombat)

Relevant logs

No response

[Deiiji]

Hello.
After mtd erase and write to it, download image, decompress and use dd to write to nvme. If regular way don't work, ths one will do

[AaronNGray]

I booted from SDCard, then fdisked, then made an ext4 partition, downloaded the image, then use Ubuntu Image Writer by clicking on the download in file manager, then chose the NVMe ext4 partition then wrote to that.

[ewaldc]

@tim273,
ubuntu-rockchip-install is a bash script located in /usr/bin. Hence you can fully trace how it works by executing it as sudo bash -x ubuntu-rockchip-install /dev/nvme0n1.

The parts that seems strange are those Microsoft basic data partitions at mmcblk1p1 and /dev/nvme0n1p1. Normally the EFI System partition (containing an ext4 file system) should be the first partition. Could it be that your SD card was not fully formatted and erased? Also, when you log into the OS that is on de SD card and execute fdisk -l you should always see the NVME drive if it is properly connected and there are no short circuits (e.g. the heatsink on the NVME touching the board somehow). Also, you might be running a very old u-boot version. Try updating u-boot using the command /usr/bin/u-boot-install /dev/mmcblk0 from the OS on the SD card. This will flash the latest u-boot, one that understands how to scan the PCI bus and look for NVME devices.

If you are familiar with bash, then the script is easy to read. Essentially you first configure the OS on the SD card and the script just clones the (installed and configured) OS as-is to EMMC or NVME via rsysnc. At the very end, it creates the boot entries for u-boot to find, using the command line chroot ${mount_point}/desktop-rootfs/u-boot-update >/dev/null 2>&1. Everything else is determined by u-boot and it's (default) boot sequence, that is: sd card first, then emmc and finally NVME.

[tim273]

Thanks for the comments from everyone. For whatever reason, I couldn't get it to work, so I tried Armbian instead and used armbian-config to do the same thing and it worked. I'm now able to boot Armbian from the NVME drive. What I noticed is that I tried using the commands detailed in the wiki for booting to the NVME on a couple of existing Orange Pi 5's that I had it worked for both of them. Those were older models that purchased about a year ago and had put multiple other OS's on them also booting from the NVME.

However, this time I tried it with a brand new Orange Pi 5 and no dice. Then it worked with Armbian, so I'm not sure what is going on.

But thank you all again for your help!

[AaronNGray]

On Sun, 22 Dec 2024 at 18:01, Tim Urberg ***@***.***> wrote: Thanks for the comments from everyone. For whatever reason, I couldn't get it to work, so I tried Armbian instead and used armbian-config to do the same thing and it worked. I'm now able to boot Armbian from the NVME drive. What I noticed is that I tried using the commands detailed in the wiki for booting to the NVME on a couple of existing Orange Pi 5's that I had it worked for both of them. Those were older models that purchased about a year ago and had put multiple other OS's on them also booting from the NVME. However, this time I tried it with a brand new Orange Pi 5 and no dice. Then it worked with Armbian, so I'm not sure what is going on.

If you are using Armbian with NVMe, I would suggest you do a test with 'fio' for 2 minute or longer to make sure its functioning correctly as I had an issue with a Samsung EVO 970 plus on BananaPi-M7 for some strange reason.

But thank you all again for your help!

Hope things are easier now !

…

— Reply to this email directly, view it on GitHub <#1140 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAU4E3VZWWAFJ5QPTZQVUD2G347NAVCNFSM6AAAAABTHZRA4OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNJYGUZTSMZVGM> . You are receiving this because you commented.Message ID: ***@***.***>

-- Aaron Gray - @***@***.*** | @***@***.*** | @***@***.*** Independent Open Source Software Engineer, Computer Language Researcher and Designer, Amateur Type Theorist, Amatuer Computer Scientist, Environmentalist and Climate Science Researcher and Disseminator.

[p3t188]

Same problem here.
With Orange Pi 5 v3.2 unable to install nvme:

orangepi@orangepi-desktop:~$ dmesg | grep pcie
[ 10.780708] reg-fixed-voltage vcc3v3-pcie2x1l2: Looking up vin-supply from device tree
[ 10.780712] vcc3v3_pcie2x1l2: supplied by vcc5v0_sys
[ 10.837663] vcc3v3_pcie2x1l2: 1800 mV, enabled
[ 10.837739] reg-fixed-voltage vcc3v3-pcie2x1l2: vcc3v3_pcie2x1l2 supplying 1800000uV
[ 11.721135] rk-pcie fe190000.pcie: invalid prsnt-gpios property in node
[ 11.721141] rk-pcie fe190000.pcie: Looking up vpcie3v3-supply from device tree
[ 11.721508] rk-pcie fe190000.pcie: IRQ msi not found
[ 11.721523] rk-pcie fe190000.pcie: use outband MSI support
[ 11.721526] rk-pcie fe190000.pcie: Missing config reg space
[ 11.721537] rk-pcie fe190000.pcie: host bridge /pcie@fe190000 ranges:
[ 11.721553] rk-pcie fe190000.pcie: err 0x00f4000000..0x00f40fffff -> 0x00f4000000
[ 11.721559] rk-pcie fe190000.pcie: IO 0x00f4100000..0x00f41fffff -> 0x00f4100000
[ 11.721567] rk-pcie fe190000.pcie: MEM 0x00f4200000..0x00f4ffffff -> 0x00f4200000
[ 11.721572] rk-pcie fe190000.pcie: MEM 0x0a00000000..0x0a3fffffff -> 0x0a00000000
[ 11.721598] rk-pcie fe190000.pcie: Missing config reg space
[ 11.721626] rk-pcie fe190000.pcie: invalid resource
[ 11.926700] rk-pcie fe190000.pcie: PCIe Linking... LTSSM is 0x3
[ 11.952254] rk-pcie fe190000.pcie: PCIe Linking... LTSSM is 0x3
[ 11.978921] rk-pcie fe190000.pcie: PCIe Linking... LTSSM is 0x3
[ 12.008918] rk-pcie fe190000.pcie: PCIe Linking... LTSSM is 0x3
[ 12.035620] rk-pcie fe190000.pcie: PCIe Linking... LTSSM is 0x3
[ 12.062262] rk-pcie fe190000.pcie: PCIe Linking... LTSSM is 0x3
[ 12.088922] rk-pcie fe190000.pcie: PCIe Linking... LTSSM is 0x3
[ 12.115587] rk-pcie fe190000.pcie: PCIe Linking... LTSSM is 0x3
[ 12.142254] rk-pcie fe190000.pcie: PCIe Linking... LTSSM is 0x3
[ 12.168921] rk-pcie fe190000.pcie: PCIe Linking... LTSSM is 0x3
[ 12.659823] rockchip-pm-domain fd8d8000.power-management:power-controller: Looking up pcie-supply from device tree
[ 12.659864] rockchip-pm-domain fd8d8000.power-management:power-controller: Looking up pcie-supply property in node /power-management@fd8d8000/power-controller failed
[ 14.622382] rk-pcie fe190000.pcie: PCIe Link Fail
[ 14.622482] rk-pcie fe190000.pcie: failed to initialize host

[ewaldc]

@p3t188, the above messages with PCIE errors are (unfortunately) quite common and can be ignored (even though PCIe Link Fail does not sound reassuring at all ...). While it's not so hard to fix these error messages, it's very time consuming and most of the folks with knowledge in the kernel area have moved on help on the more recent kernels (e.g on kernel 6.12.8 recently all these errors are gone).
Try with dmesg | grep -i nvme. It should reveal some messages such as nvme0n1: p1 ... showing that at least your NVME drive is present and accessable over PCIe. If so, and if the sudo ubuntu-rockchip-install /dev/nvme0n1 command was at least partially succesful, than you might be able to mount the NVME drive (e.g. mkdir -p /mnt/tmp; mount /dev/nvme0n1p1 /mnt/tmp).
You may also try to execute ubuntu-rockchip-install as sudo bash -x /usr/bin/ubuntu-rockchip-install /dev/nvme0n1 since it is a shell script. In this way you will see step-by-step what is going on.

PS. Since there is no more support for this distro, you may eventually have to consider moving to e.g. Armbian Bookworm (Debian 12) or Ubuntu 24.04 (server or desktop) such as the edge versions with kernel 6.12.

[p3t188]

@ewaldc
OFF:
Maybe I'm wrong, but I understand that the mainline kernel have no npu support, but i need npu in my project.
I can only use ubuntu 22.04 with 5.10.0-1007 kernel version, because this is the last working version for me.
Later versions has an USB bug with my camera: #1081

ON:
The nvme problem:
When I write the image above to SD card, and boot up, the pcie error is present:
rk-pcie fe190000.pcie: PCIe Link Fail
rk-pcie fe190000.pcie: failed to initialize host
So the system cannot see NVME (used lsblk to check it)
If this image is writen to nvme with another device (earlier than OPi 5 v1.3.2 board) and Opi 5 factory rkspi_loader is flashed to mtd: https://drive.google.com/file/d/1oiyXJZzgNUKNAgm1FLRgh3XEhGesxzET/view?usp=drive_link, the board boot up successfully!

If i download original opi5 ubuntu image: Orangepi5_1.2.0_ubuntu_jammy_desktop_xfce_linux5.10.160.7z, and i flash it to sd card, the board cannot boot up, but with: Orangepi5_1.1.8_ubuntu_jammy_desktop_xfce_linux6.1.43.7z image, it's boot up successfully.
So my workaround:
I flashed the original image with newer kernel, and i copied my prefered old Joshua Riek image to fs. With this sd the board will boot up normally, and also the nvme drive will work, so i can mount my prefered image and "copy" file-s to nvme, and then it will boot up from nvme drive.

[ewaldc]

I was not aware of the impossibility to use NPU on mainline. The re-engineering process had made it to release (based on this and this even though it has not yet made its way to mainline source tree (but it is planned). I will give it a try on my 6.12.8 kernel and report back.

With respect to the PCIE issues, I have the failed to initialize error message also on systems where NVME (boot) is working fine (e.g. Joshua Riek's 24.04/vendor 6.1), but I know that mainline u-boot has issues to see/boot NVME with vendor kernels (a vendor u-boot is required that is compatible with vendor kernel releases). Also how you boot from NVME (or what rootfs you have) can make a difference (e.g. extlinux/extlinux.conf versus boot.scr) as not all u-boot support the same boot options or rootfs (e.g. btrfs) And distro's can have either mainline or vendor u-boot, some have custom changes to enable more combinations (e.g. Armbian), and this may also yield different results (e.g. unable to boot NVME via extlinux/extlinux.conf but OK via boot.scr, but in that case lsblk should list NVME, it just can't boot from it).

Maybe that is what is happening when you flash an original Orange Pi image: a vendor u-boot gets installed. And from there you see your NVME and can do things linke change OS, upgrade 6.1 kernel, boot NVME etc. as long as you don't reflash u-boot...

That may explain why had success with Orangepi5_1.1.8_ubuntu_jammy_desktop_xfce_linux6.1.43.7z image as it has a vendor u-boot compable with 6.1 vendor kernels. Anyhow, great that you found a workaround for your issue!

[Deiiji]

It look like spi boot issue, have you cleaned that flash memory?