Blade 3 node no longer boots after eMMC fsck

After having some trouble with the eMMC on my nodes, I was instructed to run an fsck on one of them during boot. Something must have gone wrong because it no longer boots! See this bunch of error messages I received on its serial console:

[    1.230497] OF: fdt: Reserved memory: failed to reserve memory for node 'drm-logo@0': base 0x0000000000000000, size 0 MiB
[    1.230516] OF: fdt: Reserved memory: failed to reserve memory for node 'drm-cubic-lut@0': base 0x0000000000000000, size 0 MiB
[    2.657428] fiq_debugger fiq_debugger.0: error -ENXIO: IRQ fiq not found
[    2.657445] fiq_debugger fiq_debugger.0: error -ENXIO: IRQ wakeup not found
[    2.657455] fiq_debugger_probe: could not install nmi irq handler
[    2.732421] dw-pcie fe180000.pcie: invalid resource
[    2.732439] dw-pcie fe180000.pcie: Failed to initialize host
[    2.732541] dw-pcie fe170000.pcie: invalid resource
[    2.732550] dw-pcie fe170000.pcie: Failed to initialize host
[    2.798862] rk806 spi2.0: no sleep-setting state
[    2.798883] rk806 spi2.0: no reset-setting pinctrl state
[    2.798893] rk806 spi2.0: no dvs-setting pinctrl state
[    2.906630] arm-scmi firmware:scmi: Failed. SCMI protocol 17 not active.
[    2.907001] dwmmc_rockchip fe2c0000.mmc: could not set regulator OCR (-22)
[    2.907024] dwmmc_rockchip fe2c0000.mmc: failed to enable vmmc regulator
[    2.923114] debugfs: File 'Capture' in directory 'dapm' already present!
[    2.949075] mpp-iep2 fdbb0000.iep: allocate roi buffer failed
[    2.949750] mpp_rkvdec2 fdc38100.rkvdec-core: shared_niu_a is not found!
[    2.949759] rkvdec2_init:1197: No niu aclk reset resource define
[    2.949765] mpp_rkvdec2 fdc38100.rkvdec-core: shared_niu_h is not found!
[    2.949769] rkvdec2_init:1200: No niu hclk reset resource define
[    2.950301] mpp_rkvdec2 fdc48100.rkvdec-core: shared_niu_a is not found!
[    2.950309] rkvdec2_init:1197: No niu aclk reset resource define
[    2.950316] mpp_rkvdec2 fdc48100.rkvdec-core: shared_niu_h is not found!
[    2.950321] rkvdec2_init:1200: No niu hclk reset resource define
[    2.950840] mpp_rkvenc2 fdbd0000.rkvenc-core: rockchip_init_read_margin: no regulator (venc) found: -19
[    2.951067] mpp_rkvenc2 fdbd0000.rkvenc-core: error -ENODEV: _opp_set_regulators: no regulator (venc) found
[    2.951084] mpp_rkvenc2 fdbd0000.rkvenc-core: failed to set opp config
[    2.951089] mpp_rkvenc2 fdbd0000.rkvenc-core: failed to init opp info
[    2.951093] mpp_rkvenc2 fdbd0000.rkvenc-core: failed to init_opp_table
[    2.951098] rkvenc_init:2147: failed to add venc devfreq
[    2.951407] mpp_rkvenc2 fdbe0000.rkvenc-core: rockchip_init_read_margin: no regulator (venc) found: -19
[    2.951612] mpp_rkvenc2 fdbe0000.rkvenc-core: error -ENODEV: _opp_set_regulators: no regulator (venc) found
[    2.951627] mpp_rkvenc2 fdbe0000.rkvenc-core: failed to set opp config
[    2.951632] mpp_rkvenc2 fdbe0000.rkvenc-core: failed to init opp info
[    2.951637] mpp_rkvenc2 fdbe0000.rkvenc-core: failed to init_opp_table
[    2.951641] rkvenc_init:2147: failed to add venc devfreq
[    2.958721] rockchip-vop2 fdd90000.vop: error -ENODEV: _opp_set_regulators: no regulator (vop) found
[    2.958742] rockchip-vop2 fdd90000.vop: failed to set opp config
[    2.958747] rockchip-vop2 fdd90000.vop: failed to init opp info
[    2.963881] debugfs: Directory 'fb000000.gpu-mali' with parent 'vdd_gpu_s0' already present!
[    2.995980] rockchip-dmc dmc: failed to get vop bandwidth to dmc rate
[    2.996002] rockchip-dmc dmc: failed to get vop pn to msch rl
[    2.996865] debugfs: File 'Capture' in directory 'dapm' already present!
[    3.001697] RKNPU fdab0000.npu: can't request region for resource [mem 0xfdab0000-0xfdabffff]
[    3.001741] RKNPU fdab0000.npu: can't request region for resource [mem 0xfdac0000-0xfdacffff]
[    3.001757] RKNPU fdab0000.npu: can't request region for resource [mem 0xfdad0000-0xfdadffff]
[    3.004935] debugfs: Directory 'fdab0000.npu-rknpu' with parent 'vdd_npu_s0' already present!
[    4.888149] rk_hdmirx fdee0000.hdmirx-controller: hdmirx_cancel_cpu_limit_freq freq qos nod add
[    5.620214] rk-pcie fe180000.pcie: PCIe Link Fail, LTSSM is 0x3, hw_retries=0
[    9.510207] rk-pcie fe180000.pcie: PCIe Link Fail, LTSSM is 0x3, hw_retries=1
[   10.530261] rk-pcie fe180000.pcie: failed to initialize host

After the last message line, the machine stops. Rebooting doesn’t help. Neither does removing the M.2 SSD. What’s wrong here?

Above these messages, I’ve found another bunch of strange system outputs:

Net:   No ethernet found.
Hit key to stop autoboot('CTRL+C'):  0 
MMC: no card present
mmc_init: -123, time 0

Device 0: unknown device
switch to partitions #0, OK
mmc0(part 0) is current device
** Unrecognized filesystem type **
fs_devread read outside partition 2
Failed to mount ext2 filesystem...
** Unrecognized filesystem type **
Scanning mmc 0:3...
Found /boot/extlinux/extlinux.conf
Retrieving file: /boot/extlinux/extlinux.conf

The kernel booted successfully, but the system failed to start properly because the system disk is damaged.

This is a normal log

If you know how to repair the system, you can use UMS to expose the system partition to another machine.

  1. Connect the USB-C port next to the HDMI port to a Linux machine using a USB data cable.

  2. Reboot the Blade3 and press Ctrl+C in the serial console before the following message appears:

    Hit key to stop autoboot('CTRL+C'):  0
    
  3. Once you reach the U-Boot command prompt, enter:

    ums mmc 0:3
    
  4. The system partition should then appear as a disk on the Linux machine. You can mount it and perform repairs.

mmc@fe2c0000: 1, mmc@fe2e0000: 0
Bootdev(atags): mmc 0
MMC0: HS200, 200Mhz
PartType: EFI
DM: v2
boot mode: None
FIT: No boot partition
No resource partition
No resource partition
Failed to load DTB, ret=-19
No find valid DTB, ret=-22
Failed to get kernel dtb, ret=-22
Model: Mixtile Blade 3
starting USB...
No working controllers found
No usb mass storage found
CLK: (sync kernel. arm: enter 1008000 KHz, init 1008000 KHz, kernel 0N/A)
  b0pll 24000 KHz
  b1pll 24000 KHz
  lpll 24000 KHz
  v0pll 24000 KHz
  aupll 24000 KHz
  cpll 1500000 KHz
  gpll 1188000 KHz
  npll 24000 KHz
  ppll 1100000 KHz
  aclk_center_root 702000 KHz
  pclk_center_root 100000 KHz
  hclk_center_root 396000 KHz
  aclk_center_low_root 500000 KHz
  aclk_top_root 750000 KHz
  pclk_top_root 100000 KHz
  aclk_low_top_root 396000 KHz
Net:   No ethernet found.
Hit key to stop autoboot('CTRL+C'):  0
=> <INTERRUPT>
=> <INTERRUPT>
=> <INTERRUPT>
=> ums mmc 0:3
UMS: LUN 0, dev 0, hwpart 0, sector 0x8000, count 0xe8edfdf
-

The disk that appears on the other linux machine.
image

So I’ll have to attach the left USB-C connector od the node to a USB jack on my PC, reboot the faulty node from the control board with nodectl reboot -n 3, and then open a serial connection by using nodectl console -n 3 very quickly in order to get into its boot prompt, right?

You need to open the serial console first, then reboot the node from another terminal window. You have to be quick, or you’ll miss the chance to enter the Boot Prompt.

1 Like

Exactly the same error has just occurred with a different node. Rebooting doesn’t help. Here is ther complete log:

DDR 9fffbe1e78 cym 24/02/04-10:09:20,fwver: v1.16
LPDDR4X, 2112MHz
channel[0] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=16 Size=4096MB
channel[1] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=16 Size=4096MB
channel[2] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=16 Size=4096MB
channel[3] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=16 Size=4096MB
Manufacturer ID:0xff
CH0 RX Vref:27.9%, TX Vref:20.8%,21.8%
CH1 RX Vref:27.5%, TX Vref:22.8%,21.8%
CH2 RX Vref:26.7%, TX Vref:22.8%,22.8%
CH3 RX Vref:29.3%, TX Vref:21.8%,21.8%
change to F1: 528MHz
change to F2: 1068MHz
change to F3: 1560MHz
change to F0: 2112MHz
out
U-Boot SPL board init
U-Boot SPL 2017.09-ge4e124926e-230922 #lxh (Sep 25 2023 - 10:58:38), fwver: v1.13
unknown raw ID 0 0 0
unrecognized JEDEC id bytes: 00, 00, 00
Trying to boot from MMC2
MMC: no card present
mmc_init: -123, time 0
spl: mmc init failed with error: -123
Trying to boot from MMC1
SPL: A/B-slot: _a, successful: 0, tries-remain: 7
Trying fit image at 0x4000 sector
## Verified-boot: 0
## Checking atf-1 0x00040000 ... sha256(a7d1d8d191...) + OK
## Checking uboot 0x00200000 ... sha256(52e089ce06...) + OK
## Checking fdt 0x0033bc38 ... sha256(fbf72ea309...) + OK
## Checking atf-2 0xff100000 ... sha256(4b2065349b...) + OK
## Checking atf-3 0x000f0000 ... sha256(aa71013e72...) + OK
Jumping to U-Boot(0x00200000) via ARM Trusted Firmware(0x00040000)
Total: 91.854/254.86 ms

INFO:    Preloader serial: 2
NOTICE:  BL31: v2.3():v2.3-682-g4ca8a8422:derrick.huang, fwver: v1.45
NOTICE:  BL31: Built : 10:11:21, Dec 27 2023
INFO:    spec: 0x1
INFO:    code: 0x88
INFO:    ext 32k is not valid
INFO:    ddr: stride-en 4CH
INFO:    GICv3 without legacy support detected.
INFO:    ARM GICv3 driver initialized in EL3
INFO:    valid_cpu_msk=0xff bcore0_rst = 0x0, bcore1_rst = 0x0
INFO:    l3 cache partition cfg-0
INFO:    system boots from cpu-hwid-0
INFO:    disable memory repair
INFO:    idle_st=0x21fff, pd_st=0x11fff9, repair_st=0xfff70001
INFO:    dfs DDR fsp_params[0].freq_mhz= 2112MHz
INFO:    dfs DDR fsp_params[1].freq_mhz= 528MHz
INFO:    dfs DDR fsp_params[2].freq_mhz= 1068MHz
INFO:    dfs DDR fsp_params[3].freq_mhz= 1560MHz
INFO:    BL31: Initialising Exception Handling Framework
INFO:    BL31: Initializing runtime services
WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK
ERROR:   Error initializing runtime service opteed_fast
INFO:    BL31: Preparing for EL3 exit to normal world
INFO:    Entry point address = 0x200000
INFO:    SPSR = 0x3c9


U-Boot 2017.09 (Apr 27 2025 - 03:56:27 +0000)

Model: Mixtile Blade 3
PreSerial: 2, raw, 0xfeb50000
DRAM:  16 GiB
Sysmem: init
Relocation Offset: eda23000
Relocation fdt: eb9f98e8 - eb9fecb0
CR: M/C/I
Using default environment

mmc@fe2c0000: 1, mmc@fe2e0000: 0
Bootdev(atags): mmc 0
MMC0: HS200, 200Mhz
PartType: EFI
DM: v2
boot mode: None
FIT: No boot partition
No resource partition
No resource partition
Failed to load DTB, ret=-19
No find valid DTB, ret=-22
Failed to get kernel dtb, ret=-22
Model: Mixtile Blade 3
starting USB...
No working controllers found
No usb mass storage found
CLK: (sync kernel. arm: enter 1008000 KHz, init 1008000 KHz, kernel 0N/A)
  b0pll 24000 KHz
  b1pll 24000 KHz
  lpll 24000 KHz
  v0pll 24000 KHz
  aupll 24000 KHz
  cpll 1500000 KHz
  gpll 1188000 KHz
  npll 24000 KHz
  ppll 1100000 KHz
  aclk_center_root 702000 KHz
  pclk_center_root 100000 KHz
  hclk_center_root 396000 KHz
  aclk_center_low_root 500000 KHz
  aclk_top_root 750000 KHz
  pclk_top_root 100000 KHz
  aclk_low_top_root 396000 KHz
Net:   No ethernet found.
Hit key to stop autoboot('CTRL+C'):  0 
MMC: no card present
mmc_init: -123, time 0

Device 0: unknown device
switch to partitions #0, OK
mmc0(part 0) is current device
** Unrecognized filesystem type **
fs_devread read outside partition 2
Failed to mount ext2 filesystem...
** Unrecognized filesystem type **
Scanning mmc 0:3...
Found /boot/extlinux/extlinux.conf
Retrieving file: /boot/extlinux/extlinux.conf
1044 bytes read in 6 ms (169.9 KiB/s)
1:      Ubuntu 24.04.2 LTS 6.1.0-1027-rockchip
Retrieving file: /boot/initrd.img-6.1.0-1027-rockchip
23601322 bytes read in 155 ms (145.2 MiB/s)
Retrieving file: /boot/vmlinuz-6.1.0-1027-rockchip
43295232 bytes read in 260 ms (158.8 MiB/s)
append: root=UUID=a5e5be9e-44a0-4d14-9ea4-83aed9de6634 rootwait rw console=ttyS2,1500000 console=tty1 cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory quiet splash plymouth.ignore-serial-consoles
Retrieving file: /lib/firmware/6.1.0-1027-rockchip/device-tree/rockchip/rk3588-mixtile-blade3.dtb
270451 bytes read in 143 ms (1.8 MiB/s)
Fdt Ramdisk skip relocation
## Flattened Device Tree blob at 0x08300000
   Booting using the fdt blob at 0x08300000
  'reserved-memory' cma: addr=10000000 size=10000000
  'reserved-memory' ramoops@110000: addr=110000 size=e0000
  'reserved-memory' miop_dma@0x0e000000: addr=e000000 size=2000000
  'reserved-memory' miop_tx_dma@0x06000000: addr=6000000 size=2000000
   Using Device Tree in place at 0000000008300000, end 0000000008345072
01:EP 23:RC
Adding bank: 0x00200000 - 0xf0000000 (size: 0xefe00000)
Adding bank: 0x100000000 - 0x3fc000000 (size: 0x2fc000000)
Adding bank: 0x3fc500000 - 0x3fff00000 (size: 0x03a00000)
Adding bank: 0x4f0000000 - 0x500000000 (size: 0x10000000)
Total: 878.657 ms

Starting kernel ...

[    1.200562] OF: fdt: Reserved memory: failed to reserve memory for node 'drm-logo@0': base 0x0000000000000000, size 0 MiB
[    1.200580] OF: fdt: Reserved memory: failed to reserve memory for node 'drm-cubic-lut@0': base 0x0000000000000000, size 0 MiB
[    2.612264] fiq_debugger fiq_debugger.0: error -ENXIO: IRQ fiq not found
[    2.612281] fiq_debugger fiq_debugger.0: error -ENXIO: IRQ wakeup not found
[    2.612291] fiq_debugger_probe: could not install nmi irq handler
[    2.683535] dw-pcie fe180000.pcie: invalid resource
[    2.683553] dw-pcie fe180000.pcie: Failed to initialize host
[    2.683658] dw-pcie fe170000.pcie: invalid resource
[    2.683666] dw-pcie fe170000.pcie: Failed to initialize host
[    2.747482] rk806 spi2.0: no sleep-setting state
[    2.747503] rk806 spi2.0: no reset-setting pinctrl state
[    2.747512] rk806 spi2.0: no dvs-setting pinctrl state
[    2.854391] arm-scmi firmware:scmi: Failed. SCMI protocol 17 not active.
[    2.854945] dwmmc_rockchip fe2c0000.mmc: could not set regulator OCR (-22)
[    2.854967] dwmmc_rockchip fe2c0000.mmc: failed to enable vmmc regulator
[    2.870855] debugfs: File 'Capture' in directory 'dapm' already present!
[    2.910315] dw-pcie fe160000.pcie: invalid resource
[    2.910343] dw-pcie fe160000.pcie: Failed to initialize host
[    2.914440] mpp-iep2 fdbb0000.iep: allocate roi buffer failed
[    2.916085] mpp_rkvdec2 fdc38100.rkvdec-core: shared_niu_a is not found!
[    2.916107] rkvdec2_init:1197: No niu aclk reset resource define
[    2.916123] mpp_rkvdec2 fdc38100.rkvdec-core: shared_niu_h is not found!
[    2.916137] rkvdec2_init:1200: No niu hclk reset resource define
[    2.917373] mpp_rkvdec2 fdc48100.rkvdec-core: shared_niu_a is not found!
[    2.917392] rkvdec2_init:1197: No niu aclk reset resource define
[    2.917408] mpp_rkvdec2 fdc48100.rkvdec-core: shared_niu_h is not found!
[    2.917422] rkvdec2_init:1200: No niu hclk reset resource define
[    2.918610] mpp_rkvenc2 fdbd0000.rkvenc-core: rockchip_init_read_margin: no regulator (venc) found: -19
[    2.918941] mpp_rkvenc2 fdbd0000.rkvenc-core: error -ENODEV: _opp_set_regulators: no regulator (venc) found
[    2.918984] mpp_rkvenc2 fdbd0000.rkvenc-core: failed to set opp config
[    2.919000] mpp_rkvenc2 fdbd0000.rkvenc-core: failed to init opp info
[    2.919014] mpp_rkvenc2 fdbd0000.rkvenc-core: failed to init_opp_table
[    2.919028] rkvenc_init:2147: failed to add venc devfreq
[    2.919798] mpp_rkvenc2 fdbe0000.rkvenc-core: rockchip_init_read_margin: no regulator (venc) found: -19
[    2.920078] mpp_rkvenc2 fdbe0000.rkvenc-core: error -ENODEV: _opp_set_regulators: no regulator (venc) found
[    2.920110] mpp_rkvenc2 fdbe0000.rkvenc-core: failed to set opp config
[    2.920125] mpp_rkvenc2 fdbe0000.rkvenc-core: failed to init opp info
[    2.920140] mpp_rkvenc2 fdbe0000.rkvenc-core: failed to init_opp_table
[    2.920153] rkvenc_init:2147: failed to add venc devfreq
[    2.942079] rockchip-vop2 fdd90000.vop: error -ENODEV: _opp_set_regulators: no regulator (vop) found
[    2.942130] rockchip-vop2 fdd90000.vop: failed to set opp config
[    2.942146] rockchip-vop2 fdd90000.vop: failed to init opp info
[    2.950488] debugfs: Directory 'fb000000.gpu-mali' with parent 'vdd_gpu_s0' already present!
[    2.952122] debugfs: Directory 'fb000000.gpu-mali' with parent 'vdd_gpu_s0' already present!
[    2.985037] rockchip-dmc dmc: failed to get vop bandwidth to dmc rate
[    2.985050] rockchip-dmc dmc: failed to get vop pn to msch rl
[    2.987209] debugfs: File 'Capture' in directory 'dapm' already present!
[    2.989267] debugfs: File 'Capture' in directory 'dapm' already present!
[    2.990912] debugfs: File 'Capture' in directory 'dapm' already present!
[    2.997494] RKNPU fdab0000.npu: can't request region for resource [mem 0xfdab0000-0xfdabffff]
[    2.997525] RKNPU fdab0000.npu: can't request region for resource [mem 0xfdac0000-0xfdacffff]
[    2.997537] RKNPU fdab0000.npu: can't request region for resource [mem 0xfdad0000-0xfdadffff]
[    3.000459] debugfs: Directory 'fdab0000.npu-rknpu' with parent 'vdd_npu_s0' already present!
[    3.012704] debugfs: Directory 'fdab0000.npu-rknpu' with parent 'vdd_npu_s0' already present!
[    4.839509] rk_hdmirx fdee0000.hdmirx-controller: hdmirx_cancel_cpu_limit_freq freq qos nod add
[    5.584849] rk-pcie fe180000.pcie: PCIe Link Fail, LTSSM is 0x3, hw_retries=0
[    9.494837] rk-pcie fe180000.pcie: PCIe Link Fail, LTSSM is 0x3, hw_retries=1
[   10.508169] rk-pcie fe180000.pcie: failed to initialize host

What’s wrong here?