Blade 3 node no longer boots after eMMC fsck

ProfessorFate · August 24, 2026, 3:35pm

After having some trouble with the eMMC on my nodes, I was instructed to run an fsck on one of them during boot. Something must have gone wrong because it no longer boots! See this bunch of error messages I received on its serial console:

[    1.230497] OF: fdt: Reserved memory: failed to reserve memory for node 'drm-logo@0': base 0x0000000000000000, size 0 MiB
[    1.230516] OF: fdt: Reserved memory: failed to reserve memory for node 'drm-cubic-lut@0': base 0x0000000000000000, size 0 MiB
[    2.657428] fiq_debugger fiq_debugger.0: error -ENXIO: IRQ fiq not found
[    2.657445] fiq_debugger fiq_debugger.0: error -ENXIO: IRQ wakeup not found
[    2.657455] fiq_debugger_probe: could not install nmi irq handler
[    2.732421] dw-pcie fe180000.pcie: invalid resource
[    2.732439] dw-pcie fe180000.pcie: Failed to initialize host
[    2.732541] dw-pcie fe170000.pcie: invalid resource
[    2.732550] dw-pcie fe170000.pcie: Failed to initialize host
[    2.798862] rk806 spi2.0: no sleep-setting state
[    2.798883] rk806 spi2.0: no reset-setting pinctrl state
[    2.798893] rk806 spi2.0: no dvs-setting pinctrl state
[    2.906630] arm-scmi firmware:scmi: Failed. SCMI protocol 17 not active.
[    2.907001] dwmmc_rockchip fe2c0000.mmc: could not set regulator OCR (-22)
[    2.907024] dwmmc_rockchip fe2c0000.mmc: failed to enable vmmc regulator
[    2.923114] debugfs: File 'Capture' in directory 'dapm' already present!
[    2.949075] mpp-iep2 fdbb0000.iep: allocate roi buffer failed
[    2.949750] mpp_rkvdec2 fdc38100.rkvdec-core: shared_niu_a is not found!
[    2.949759] rkvdec2_init:1197: No niu aclk reset resource define
[    2.949765] mpp_rkvdec2 fdc38100.rkvdec-core: shared_niu_h is not found!
[    2.949769] rkvdec2_init:1200: No niu hclk reset resource define
[    2.950301] mpp_rkvdec2 fdc48100.rkvdec-core: shared_niu_a is not found!
[    2.950309] rkvdec2_init:1197: No niu aclk reset resource define
[    2.950316] mpp_rkvdec2 fdc48100.rkvdec-core: shared_niu_h is not found!
[    2.950321] rkvdec2_init:1200: No niu hclk reset resource define
[    2.950840] mpp_rkvenc2 fdbd0000.rkvenc-core: rockchip_init_read_margin: no regulator (venc) found: -19
[    2.951067] mpp_rkvenc2 fdbd0000.rkvenc-core: error -ENODEV: _opp_set_regulators: no regulator (venc) found
[    2.951084] mpp_rkvenc2 fdbd0000.rkvenc-core: failed to set opp config
[    2.951089] mpp_rkvenc2 fdbd0000.rkvenc-core: failed to init opp info
[    2.951093] mpp_rkvenc2 fdbd0000.rkvenc-core: failed to init_opp_table
[    2.951098] rkvenc_init:2147: failed to add venc devfreq
[    2.951407] mpp_rkvenc2 fdbe0000.rkvenc-core: rockchip_init_read_margin: no regulator (venc) found: -19
[    2.951612] mpp_rkvenc2 fdbe0000.rkvenc-core: error -ENODEV: _opp_set_regulators: no regulator (venc) found
[    2.951627] mpp_rkvenc2 fdbe0000.rkvenc-core: failed to set opp config
[    2.951632] mpp_rkvenc2 fdbe0000.rkvenc-core: failed to init opp info
[    2.951637] mpp_rkvenc2 fdbe0000.rkvenc-core: failed to init_opp_table
[    2.951641] rkvenc_init:2147: failed to add venc devfreq
[    2.958721] rockchip-vop2 fdd90000.vop: error -ENODEV: _opp_set_regulators: no regulator (vop) found
[    2.958742] rockchip-vop2 fdd90000.vop: failed to set opp config
[    2.958747] rockchip-vop2 fdd90000.vop: failed to init opp info
[    2.963881] debugfs: Directory 'fb000000.gpu-mali' with parent 'vdd_gpu_s0' already present!
[    2.995980] rockchip-dmc dmc: failed to get vop bandwidth to dmc rate
[    2.996002] rockchip-dmc dmc: failed to get vop pn to msch rl
[    2.996865] debugfs: File 'Capture' in directory 'dapm' already present!
[    3.001697] RKNPU fdab0000.npu: can't request region for resource [mem 0xfdab0000-0xfdabffff]
[    3.001741] RKNPU fdab0000.npu: can't request region for resource [mem 0xfdac0000-0xfdacffff]
[    3.001757] RKNPU fdab0000.npu: can't request region for resource [mem 0xfdad0000-0xfdadffff]
[    3.004935] debugfs: Directory 'fdab0000.npu-rknpu' with parent 'vdd_npu_s0' already present!
[    4.888149] rk_hdmirx fdee0000.hdmirx-controller: hdmirx_cancel_cpu_limit_freq freq qos nod add
[    5.620214] rk-pcie fe180000.pcie: PCIe Link Fail, LTSSM is 0x3, hw_retries=0
[    9.510207] rk-pcie fe180000.pcie: PCIe Link Fail, LTSSM is 0x3, hw_retries=1
[   10.530261] rk-pcie fe180000.pcie: failed to initialize host

After the last message line, the machine stops. Rebooting doesn’t help. Neither does removing the M.2 SSD. What’s wrong here?

ProfessorFate · August 25, 2026, 10:43am

Above these messages, I’ve found another bunch of strange system outputs:

Net:   No ethernet found.
Hit key to stop autoboot('CTRL+C'):  0 
MMC: no card present
mmc_init: -123, time 0

Device 0: unknown device
switch to partitions #0, OK
mmc0(part 0) is current device
** Unrecognized filesystem type **
fs_devread read outside partition 2
Failed to mount ext2 filesystem...
** Unrecognized filesystem type **
Scanning mmc 0:3...
Found /boot/extlinux/extlinux.conf
Retrieving file: /boot/extlinux/extlinux.conf

adam · October 15, 2026, 3:52am

The kernel booted successfully, but the system failed to start properly because the system disk is damaged.

adam · October 15, 2026, 3:54am

This is a normal log

adam · October 15, 2026, 4:08am

If you know how to repair the system, you can use UMS to expose the system partition to another machine.

Connect the USB-C port next to the HDMI port to a Linux machine using a USB data cable.
Reboot the Blade3 and press Ctrl+C in the serial console before the following message appears:
```
Hit key to stop autoboot('CTRL+C'):  0
```
Once you reach the U-Boot command prompt, enter:
```
ums mmc 0:3
```
The system partition should then appear as a disk on the Linux machine. You can mount it and perform repairs.

mmc@fe2c0000: 1, mmc@fe2e0000: 0
Bootdev(atags): mmc 0
MMC0: HS200, 200Mhz
PartType: EFI
DM: v2
boot mode: None
FIT: No boot partition
No resource partition
No resource partition
Failed to load DTB, ret=-19
No find valid DTB, ret=-22
Failed to get kernel dtb, ret=-22
Model: Mixtile Blade 3
starting USB...
No working controllers found
No usb mass storage found
CLK: (sync kernel. arm: enter 1008000 KHz, init 1008000 KHz, kernel 0N/A)
  b0pll 24000 KHz
  b1pll 24000 KHz
  lpll 24000 KHz
  v0pll 24000 KHz
  aupll 24000 KHz
  cpll 1500000 KHz
  gpll 1188000 KHz
  npll 24000 KHz
  ppll 1100000 KHz
  aclk_center_root 702000 KHz
  pclk_center_root 100000 KHz
  hclk_center_root 396000 KHz
  aclk_center_low_root 500000 KHz
  aclk_top_root 750000 KHz
  pclk_top_root 100000 KHz
  aclk_low_top_root 396000 KHz
Net:   No ethernet found.
Hit key to stop autoboot('CTRL+C'):  0
=> <INTERRUPT>
=> <INTERRUPT>
=> <INTERRUPT>
=> ums mmc 0:3
UMS: LUN 0, dev 0, hwpart 0, sector 0x8000, count 0xe8edfdf
-

The disk that appears on the other linux machine.

ProfessorFate · October 15, 2026, 1:26pm

So I’ll have to attach the left USB-C connector od the node to a USB jack on my PC, reboot the faulty node from the control board with nodectl reboot -n 3, and then open a serial connection by using nodectl console -n 3 very quickly in order to get into its boot prompt, right?

adam · October 16, 2026, 1:26am

You need to open the serial console first, then reboot the node from another terminal window. You have to be quick, or you’ll miss the chance to enter the Boot Prompt.