I’ve recently had a couple of incidents with my Blade 3 in a Cluster Box: Node #3 (it’s always the same one) no longer reacts and drops out of the Ceph cluster. Even by using the serial console via control board, I can only type in the username, and after that, nothing happens. SSH access is dead. Grafana shows 100% CPU load.
The only possibility to get this node back to life, is to restart the whole cluster and to risk trouble with Ceph and other cluster apps.
You can take Node #3 out of the Cluster Box and boot it separately using a PD power supply. Then, check the serial console output to determine whether the node is able to boot and run normally.
I’ve just encountered the symptoms I had described on node #1, but this time I was able to read a handful of error messages from its serial console I don’t really understand (they are too low-level for me):
[193113.458173] mmc0: cqhci: timeout for tag 0, qcnt 31
[193113.458197] mmc0: cqhci: ============ CQHCI REGISTER DUMP ===========
[193113.458203] mmc0: cqhci: Caps: 0x000030c8 | Version: 0x00000510
[193113.458208] mmc0: cqhci: Config: 0x00001001 | Control: 0x00000000
[193113.458213] mmc0: cqhci: Int stat: 0x00000000 | Int enab: 0x00000006
[193113.458218] mmc0: cqhci: Int sig: 0x00000006 | Int Coal: 0x00000000
[193113.458222] mmc0: cqhci: TDL base: 0x030bd1e0 | TDL up32: 0x00000000
[193113.458227] mmc0: cqhci: Doorbell: 0x7fffffff | TCN: 0x00000000
[193113.458231] mmc0: cqhci: Dev queue: 0x00000001 | Dev Pend: 0x7fffffff
[193113.458236] mmc0: cqhci: Task clr: 0x00000000 | SSC1: 0x00011000
[193113.458240] mmc0: cqhci: SSC2: 0x00000001 | DCMD rsp: 0x00000800
[193113.458246] mmc0: cqhci: RED mask: 0xfdf9a080 | TERRI: 0x032f1f0d
[193113.458251] mmc0: cqhci: Resp idx: 0x0000000d | Resp arg: 0x00000001
[193113.458255] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
[193113.458261] mmc0: sdhci: Sys addr: 0x00000008 | Version: 0x00000005
[193113.458266] mmc0: sdhci: Blk size: 0x00007200 | Blk cnt: 0x00000000
[193113.458270] mmc0: sdhci: Argument: 0x00018000 | Trn mode: 0x00000023
[193113.458275] mmc0: sdhci: Present: 0x13f700f0 | Host ctl: 0x00000034
[193113.458280] mmc0: sdhci: Power: 0x0000000d | Blk gap: 0x00000000
[193113.458285] mmc0: sdhci: Wake-up: 0x00000000 | Clock: 0x00000007
[193113.458289] mmc0: sdhci: Timeout: 0x0000000e | Int stat: 0x00000000
[193113.458294] mmc0: sdhci: Int enab: 0x02ff4000 | Sig enab: 0x02ff4000
[193113.458299] mmc0: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000
[193113.458304] mmc0: sdhci: Caps: 0x226dc881 | Caps_1: 0x08000007
[193113.458309] mmc0: sdhci: Cmd: 0x00000d1e | Max curr: 0x00000000
[193113.458314] mmc0: sdhci: Resp[0]: 0x00000900 | Resp[1]: 0x00000001
[193113.458320] mmc0: sdhci: Resp[2]: 0x00000800 | Resp[3]: 0x000007cd
[193113.458324] mmc0: sdhci: Host ctl2: 0x0000000f
[193113.458329] mmc0: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x10718008
[193113.458334] mmc0: sdhci: ============================================
[193113.459623] I/O error, dev mmcblk0, sector 38995888 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 2
[193113.459645] Buffer I/O error on device mmcblk0p3, logical block 4870390
[193145.512183] systemd[1]: Failed to start systemd-journald.service - Journal Service.
I’ve also encountered some serious trouble with the Ceph cluster running oin all four nodes (daemons disappearing without obvious reason, SSD access gone etc.). I hope these issues are not caused by the failure I’m describing here.
Unfortunately, based on the current logs, the issue is most likely with the eMMC itself. It may be experiencing wear or media degradation, and its write endurance could be approaching its limit.
If you plan to build a storage cluster, it would be better to avoid using the eMMC for frequent or sustained workload I/O. Data volumes, logs, databases, and other write-intensive workloads should ideally be placed on SSDs, while the eMMC is used only for booting the system or low-frequency I/O. This should provide much better reliability.
As for the 100% CPU usage / high system load, it is also very likely related to the storage issue. When the eMMC experiences I/O timeouts, the filesystem and kernel may repeatedly wait, retry operations, or perform error handling, which can result in high system load.
I’m already using M.2 SSDs for my productive data. They are combined into one big datastore by using CephFS. Ceph is quite resilient when it comes to node or disk failure, but when several Ceph daemons exit without an obvious reason, something else must be wrong. What I’m afraid of, is that due to the eMMC failure, data write processes could be disturbed and then save corrupted data to disk.
The problem is: When a node is affected by the purported storage failure, it becomes pretty useless: You can’t even log into it anymore. and when too many Ceph monitor processes die, you will be left with a cluster, which no longer works. Please look into this urgently and tell me how to solve this problem:
Update:
I’ve just encountered the same issue once more. Node #3 hangs, but spits out a bunch of error messages on the serial console:
[19761.194157] mmc0: cqhci: timeout for tag 0, qcnt 31
[19761.194206] mmc0: cqhci: ============ CQHCI REGISTER DUMP ===========
[19761.194218] mmc0: cqhci: Caps: 0x000030c8 | Version: 0x00000510
[19761.194229] mmc0: cqhci: Config: 0x00001001 | Control: 0x00000000
[19761.194239] mmc0: cqhci: Int stat: 0x00000000 | Int enab: 0x00000006
[19761.194251] mmc0: cqhci: Int sig: 0x00000006 | Int Coal: 0x00000000
[19761.194261] mmc0: cqhci: TDL base: 0x0305c1e0 | TDL up32: 0x00000000
[19761.194271] mmc0: cqhci: Doorbell: 0x7fffffff | TCN: 0x00000000
[19761.194281] mmc0: cqhci: Dev queue: 0x00000001 | Dev Pend: 0x7fffffff
[19761.194292] mmc0: cqhci: Task clr: 0x00000000 | SSC1: 0x00011000
[19761.194302] mmc0: cqhci: SSC2: 0x00000001 | DCMD rsp: 0x00000800
[19761.194313] mmc0: cqhci: RED mask: 0xfdf9a080 | TERRI: 0x1e2f1f0d
[19761.194325] mmc0: cqhci: Resp idx: 0x0000000d | Resp arg: 0x00000001
[19761.194337] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
[19761.194349] mmc0: sdhci: Sys addr: 0x00000008 | Version: 0x00000005
[19761.194361] mmc0: sdhci: Blk size: 0x00007200 | Blk cnt: 0x00000000
[19761.194373] mmc0: sdhci: Argument: 0x00018000 | Trn mode: 0x00000023
[19761.194384] mmc0: sdhci: Present: 0x13f700f0 | Host ctl: 0x00000034
[19761.194395] mmc0: sdhci: Power: 0x0000000d | Blk gap: 0x00000000
[19761.194406] mmc0: sdhci: Wake-up: 0x00000000 | Clock: 0x00000007
[19761.194416] mmc0: sdhci: Timeout: 0x0000000e | Int stat: 0x00000000
[19761.194427] mmc0: sdhci: Int enab: 0x02ff4000 | Sig enab: 0x02ff4000
[19761.194437] mmc0: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000
[19761.194449] mmc0: sdhci: Caps: 0x226dc881 | Caps_1: 0x08000007
[19761.194460] mmc0: sdhci: Cmd: 0x00000d1e | Max curr: 0x00000000
[19761.194471] mmc0: sdhci: Resp[0]: 0x00000900 | Resp[1]: 0x00000001
[19761.194482] mmc0: sdhci: Resp[2]: 0x00000800 | Resp[3]: 0x000007cd
[19761.194492] mmc0: sdhci: Host ctl2: 0x0000000f
[19761.194502] mmc0: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x107f0008
[19761.194512] mmc0: sdhci: ============================================
[19763.441602] systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 3min)!
[19822.633148] mmc0: cqhci: timeout for tag 0, qcnt 31
[19822.633198] mmc0: cqhci: ============ CQHCI REGISTER DUMP ===========
[19822.633209] mmc0: cqhci: Caps: 0x000030c8 | Version: 0x00000510
[19822.633221] mmc0: cqhci: Config: 0x00001001 | Control: 0x00000000
[19822.633232] mmc0: cqhci: Int stat: 0x00000000 | Int enab: 0x00000006
[19822.633243] mmc0: cqhci: Int sig: 0x00000006 | Int Coal: 0x00000000
[19822.633253] mmc0: cqhci: TDL base: 0x0305c1d0 | TDL up32: 0x00000000
[19822.633263] mmc0: cqhci: Doorbell: 0x7fffffff | TCN: 0x00000000
[19822.633274] mmc0: cqhci: Dev queue: 0x00000001 | Dev Pend: 0x7fffffff
[19822.633285] mmc0: cqhci: Task clr: 0x00000000 | SSC1: 0x00011000
[19822.633295] mmc0: cqhci: SSC2: 0x00000001 | DCMD rsp: 0x00000800
[19822.633306] mmc0: cqhci: RED mask: 0xfdf9a080 | TERRI: 0x1d2e1f0d
[19822.633316] mmc0: cqhci: Resp idx: 0x0000000d | Resp arg: 0x00000001
[19822.633328] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
[19822.633340] mmc0: sdhci: Sys addr: 0x00000008 | Version: 0x00000005
[19822.633353] mmc0: sdhci: Blk size: 0x00007200 | Blk cnt: 0x00000000
[19822.633365] mmc0: sdhci: Argument: 0x00018000 | Trn mode: 0x00000033
[19822.633378] mmc0: sdhci: Present: 0x13f700f0 | Host ctl: 0x00000034
[19822.633390] mmc0: sdhci: Power: 0x0000000d | Blk gap: 0x00000000
[19822.633400] mmc0: sdhci: Wake-up: 0x00000000 | Clock: 0x00000007
[19822.633411] mmc0: sdhci: Timeout: 0x0000000e | Int stat: 0x00000000
[19822.633421] mmc0: sdhci: Int enab: 0x02ff4000 | Sig enab: 0x02ff4000
[19822.633432] mmc0: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000
[19822.633443] mmc0: sdhci: Caps: 0x226dc881 | Caps_1: 0x08000007
[19822.633453] mmc0: sdhci: Cmd: 0x00000d1e | Max curr: 0x00000000
[19822.633463] mmc0: sdhci: Resp[0]: 0x00000900 | Resp[1]: 0x00000001
[19822.633473] mmc0: sdhci: Resp[2]: 0x00000800 | Resp[3]: 0x000007cd
[19822.633484] mmc0: sdhci: Host ctl2: 0x0000000f
[19822.633496] mmc0: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x107e8008
[19822.633506] mmc0: sdhci: ============================================
[19884.072183] mmc0: cqhci: timeout for tag 0, qcnt 31
[19884.072233] mmc0: cqhci: ============ CQHCI REGISTER DUMP ===========
[19884.072245] mmc0: cqhci: Caps: 0x000030c8 | Version: 0x00000510
[19884.072257] mmc0: cqhci: Config: 0x00001001 | Control: 0x00000000
[19884.072267] mmc0: cqhci: Int stat: 0x00000000 | Int enab: 0x00000006
[19884.072278] mmc0: cqhci: Int sig: 0x00000006 | Int Coal: 0x00000000
[19884.072290] mmc0: cqhci: TDL base: 0x0305c1e0 | TDL up32: 0x00000000
[19884.072300] mmc0: cqhci: Doorbell: 0x7fffffff | TCN: 0x00000000
[19884.072312] mmc0: cqhci: Dev queue: 0x00000001 | Dev Pend: 0x7fffffff
[19884.072323] mmc0: cqhci: Task clr: 0x00000000 | SSC1: 0x00011000
[19884.072333] mmc0: cqhci: SSC2: 0x00000001 | DCMD rsp: 0x00000800
[19884.072344] mmc0: cqhci: RED mask: 0xfdf9a080 | TERRI: 0x1e2f1f0d
[19884.072355] mmc0: cqhci: Resp idx: 0x0000000d | Resp arg: 0x00000001
[19884.072367] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
[19884.072379] mmc0: sdhci: Sys addr: 0x00000008 | Version: 0x00000005
[19884.072391] mmc0: sdhci: Blk size: 0x00007200 | Blk cnt: 0x00000000
[19884.072402] mmc0: sdhci: Argument: 0x00018000 | Trn mode: 0x00000023
[19884.072413] mmc0: sdhci: Present: 0x13f700f0 | Host ctl: 0x00000034
[19884.072425] mmc0: sdhci: Power: 0x0000000d | Blk gap: 0x00000000
[19884.072435] mmc0: sdhci: Wake-up: 0x00000000 | Clock: 0x00000007
[19884.072445] mmc0: sdhci: Timeout: 0x0000000e | Int stat: 0x00000000
[19884.072456] mmc0: sdhci: Int enab: 0x02ff4000 | Sig enab: 0x02ff4000
[19884.072468] mmc0: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000
[19884.072479] mmc0: sdhci: Caps: 0x226dc881 | Caps_1: 0x08000007
[19884.072489] mmc0: sdhci: Cmd: 0x00000d1e | Max curr: 0x00000000
[19884.072499] mmc0: sdhci: Resp[0]: 0x00000900 | Resp[1]: 0x00000001
[19884.072510] mmc0: sdhci: Resp[2]: 0x00000800 | Resp[3]: 0x000007cd
[19884.072521] mmc0: sdhci: Host ctl2: 0x0000000f
[19884.072530] mmc0: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x107f0008
[19884.072540] mmc0: sdhci: ============================================
[19884.081405] I/O error, dev mmcblk0, sector 24548128 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 2
[19884.081459] Buffer I/O error on device mmcblk0p3, logical block 3064420
[19945.511103] mmc0: cqhci: timeout for tag 0, qcnt 31
[19945.511147] mmc0: cqhci: ============ CQHCI REGISTER DUMP ===========
[19945.511159] mmc0: cqhci: Caps: 0x000030c8 | Version: 0x00000510
[19945.511170] mmc0: cqhci: Config: 0x00001001 | Control: 0x00000000
[19945.511180] mmc0: cqhci: Int stat: 0x00000000 | Int enab: 0x00000006
[19945.511193] mmc0: cqhci: Int sig: 0x00000006 | Int Coal: 0x00000000
[19945.511203] mmc0: cqhci: TDL base: 0x0305c1e0 | TDL up32: 0x00000000
[19945.511213] mmc0: cqhci: Doorbell: 0x7fffffff | TCN: 0x00000000
[19945.511223] mmc0: cqhci: Dev queue: 0x00000001 | Dev Pend: 0x7fffffff
[19945.511234] mmc0: cqhci: Task clr: 0x00000000 | SSC1: 0x00011000
[19945.511244] mmc0: cqhci: SSC2: 0x00000001 | DCMD rsp: 0x00000800
[19945.511255] mmc0: cqhci: RED mask: 0xfdf9a080 | TERRI: 0x1e2f1f0d
[19945.511266] mmc0: cqhci: Resp idx: 0x0000000d | Resp arg: 0x00000001
[19945.511280] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
[19945.511291] mmc0: sdhci: Sys addr: 0x00000008 | Version: 0x00000005
[19945.511302] mmc0: sdhci: Blk size: 0x00007200 | Blk cnt: 0x00000000
[19945.511313] mmc0: sdhci: Argument: 0x00018000 | Trn mode: 0x00000023
[19945.511324] mmc0: sdhci: Present: 0x13f700f0 | Host ctl: 0x00000034
[19945.511334] mmc0: sdhci: Power: 0x0000000d | Blk gap: 0x00000000
[19945.511344] mmc0: sdhci: Wake-up: 0x00000000 | Clock: 0x00000007
[19945.511354] mmc0: sdhci: Timeout: 0x0000000e | Int stat: 0x00000000
[19945.511365] mmc0: sdhci: Int enab: 0x02ff4000 | Sig enab: 0x02ff4000
[19945.511375] mmc0: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000
[19945.511385] mmc0: sdhci: Caps: 0x226dc881 | Caps_1: 0x08000007
[19945.511395] mmc0: sdhci: Cmd: 0x00000d1e | Max curr: 0x00000000
[19945.511405] mmc0: sdhci: Resp[0]: 0x00000900 | Resp[1]: 0x00000001
[19945.511415] mmc0: sdhci: Resp[2]: 0x00000800 | Resp[3]: 0x000007cd
[19945.511425] mmc0: sdhci: Host ctl2: 0x0000000f
[19945.511435] mmc0: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x107f0008
[19945.511446] mmc0: sdhci: ============================================
[20006.950036] mmc0: cqhci: timeout for tag 0, qcnt 31
[20006.950086] mmc0: cqhci: ============ CQHCI REGISTER DUMP ===========
[20006.950098] mmc0: cqhci: Caps: 0x000030c8 | Version: 0x00000510
[20006.950109] mmc0: cqhci: Config: 0x00001001 | Control: 0x00000000
[20006.950120] mmc0: cqhci: Int stat: 0x00000000 | Int enab: 0x00000006
[20006.950131] mmc0: cqhci: Int sig: 0x00000006 | Int Coal: 0x00000000
[20006.950142] mmc0: cqhci: TDL base: 0x0305c1e0 | TDL up32: 0x00000000
[20006.950153] mmc0: cqhci: Doorbell: 0x7fffffff | TCN: 0x00000000
[20006.950164] mmc0: cqhci: Dev queue: 0x00000001 | Dev Pend: 0x7fffffff
[20006.950175] mmc0: cqhci: Task clr: 0x00000000 | SSC1: 0x00011000
[20006.950185] mmc0: cqhci: SSC2: 0x00000001 | DCMD rsp: 0x00000800
[20006.950196] mmc0: cqhci: RED mask: 0xfdf9a080 | TERRI: 0x1e2f1f0d
[20006.950206] mmc0: cqhci: Resp idx: 0x0000000d | Resp arg: 0x00000001
[20006.950220] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
[20006.950230] mmc0: sdhci: Sys addr: 0x00000008 | Version: 0x00000005
[20006.950241] mmc0: sdhci: Blk size: 0x00007200 | Blk cnt: 0x00000000
[20006.950253] mmc0: sdhci: Argument: 0x00018000 | Trn mode: 0x00000023
[20006.950264] mmc0: sdhci: Present: 0x13f700f0 | Host ctl: 0x00000034
[20006.950276] mmc0: sdhci: Power: 0x0000000d | Blk gap: 0x00000000
[20006.950285] mmc0: sdhci: Wake-up: 0x00000000 | Clock: 0x00000007
[20006.950296] mmc0: sdhci: Timeout: 0x0000000e | Int stat: 0x00000000
[20006.950307] mmc0: sdhci: Int enab: 0x02ff4000 | Sig enab: 0x02ff4000
[20006.950317] mmc0: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000
[20006.950327] mmc0: sdhci: Caps: 0x226dc881 | Caps_1: 0x08000007
[20006.950338] mmc0: sdhci: Cmd: 0x00000d1e | Max curr: 0x00000000
[20006.950348] mmc0: sdhci: Resp[0]: 0x00000900 | Resp[1]: 0x00000001
[20006.950358] mmc0: sdhci: Resp[2]: 0x00000800 | Resp[3]: 0x000007cd
[20006.950369] mmc0: sdhci: Host ctl2: 0x0000000f
[20006.950380] mmc0: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x107f0008
[20006.950391] mmc0: sdhci: ============================================
[20068.389139] mmc0: cqhci: timeout for tag 0, qcnt 31
[20068.389187] mmc0: cqhci: ============ CQHCI REGISTER DUMP ===========
[20068.389199] mmc0: cqhci: Caps: 0x000030c8 | Version: 0x00000510
[20068.389210] mmc0: cqhci: Config: 0x00001001 | Control: 0x00000000
[20068.389221] mmc0: cqhci: Int stat: 0x00000000 | Int enab: 0x00000006
[20068.389233] mmc0: cqhci: Int sig: 0x00000006 | Int Coal: 0x00000000
[20068.389243] mmc0: cqhci: TDL base: 0x0305c1e0 | TDL up32: 0x00000000
[20068.389254] mmc0: cqhci: Doorbell: 0x7fffffff | TCN: 0x00000000
[20068.389265] mmc0: cqhci: Dev queue: 0x00000001 | Dev Pend: 0x7fffffff
[20068.389275] mmc0: cqhci: Task clr: 0x00000000 | SSC1: 0x00011000
[20068.389285] mmc0: cqhci: SSC2: 0x00000001 | DCMD rsp: 0x00000800
[20068.389295] mmc0: cqhci: RED mask: 0xfdf9a080 | TERRI: 0x1e2f1f0d
[20068.389306] mmc0: cqhci: Resp idx: 0x0000000d | Resp arg: 0x00000001
[20068.389319] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
[20068.389329] mmc0: sdhci: Sys addr: 0x00000008 | Version: 0x00000005
[20068.389340] mmc0: sdhci: Blk size: 0x00007200 | Blk cnt: 0x00000000
[20068.389351] mmc0: sdhci: Argument: 0x00018000 | Trn mode: 0x00000023
[20068.389362] mmc0: sdhci: Present: 0x13f700f0 | Host ctl: 0x00000034
[20068.389373] mmc0: sdhci: Power: 0x0000000d | Blk gap: 0x00000000
[20068.389383] mmc0: sdhci: Wake-up: 0x00000000 | Clock: 0x00000007
[20068.389393] mmc0: sdhci: Timeout: 0x0000000e | Int stat: 0x00000000
[20068.389404] mmc0: sdhci: Int enab: 0x02ff4000 | Sig enab: 0x02ff4000
[20068.389414] mmc0: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000
[20068.389424] mmc0: sdhci: Caps: 0x226dc881 | Caps_1: 0x08000007
[20068.389434] mmc0: sdhci: Cmd: 0x00000d1e | Max curr: 0x00000000
[20068.389444] mmc0: sdhci: Resp[0]: 0x00000900 | Resp[1]: 0x00000001
[20068.389454] mmc0: sdhci: Resp[2]: 0x00000800 | Resp[3]: 0x000007cd
[20068.389466] mmc0: sdhci: Host ctl2: 0x0000000f
[20068.389478] mmc0: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x107f0008
[20068.389489] mmc0: sdhci: ============================================
[20068.391497] I/O error, dev mmcblk0, sector 129117664 op 0x0:(READ) flags 0x80700 phys_seg 13 prio class 2
[20129.828159] mmc0: cqhci: timeout for tag 0, qcnt 31
[20129.828208] mmc0: cqhci: ============ CQHCI REGISTER DUMP ===========
[20129.828220] mmc0: cqhci: Caps: 0x000030c8 | Version: 0x00000510
[20129.828232] mmc0: cqhci: Config: 0x00001001 | Control: 0x00000000
[20129.828242] mmc0: cqhci: Int stat: 0x00000000 | Int enab: 0x00000006
[20129.828255] mmc0: cqhci: Int sig: 0x00000006 | Int Coal: 0x00000000
[20129.828265] mmc0: cqhci: TDL base: 0x0305c1e0 | TDL up32: 0x00000000
[20129.828277] mmc0: cqhci: Doorbell: 0x7fffffff | TCN: 0x00000000
[20129.828288] mmc0: cqhci: Dev queue: 0x00000001 | Dev Pend: 0x7fffffff
[20129.828299] mmc0: cqhci: Task clr: 0x00000000 | SSC1: 0x00011000
[20129.828308] mmc0: cqhci: SSC2: 0x00000001 | DCMD rsp: 0x00000800
[20129.828318] mmc0: cqhci: RED mask: 0xfdf9a080 | TERRI: 0x1e2f1f0d
[20129.828328] mmc0: cqhci: Resp idx: 0x0000000d | Resp arg: 0x00000001
[20129.828341] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
[20129.828352] mmc0: sdhci: Sys addr: 0x00000008 | Version: 0x00000005
[20129.828363] mmc0: sdhci: Blk size: 0x00007200 | Blk cnt: 0x00000000
[20129.828375] mmc0: sdhci: Argument: 0x00018000 | Trn mode: 0x00000023
[20129.828386] mmc0: sdhci: Present: 0x13f700f0 | Host ctl: 0x00000034
[20129.828398] mmc0: sdhci: Power: 0x0000000d | Blk gap: 0x00000000
[20129.828409] mmc0: sdhci: Wake-up: 0x00000000 | Clock: 0x00000007
[20129.828419] mmc0: sdhci: Timeout: 0x0000000e | Int stat: 0x00000000
[20129.828430] mmc0: sdhci: Int enab: 0x02ff4000 | Sig enab: 0x02ff4000
[20129.828441] mmc0: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000
[20129.828451] mmc0: sdhci: Caps: 0x226dc881 | Caps_1: 0x08000007
[20129.828463] mmc0: sdhci: Cmd: 0x00000d1e | Max curr: 0x00000000
[20129.828475] mmc0: sdhci: Resp[0]: 0x00000900 | Resp[1]: 0x00000001
[20129.828486] mmc0: sdhci: Resp[2]: 0x00000800 | Resp[3]: 0x000007cd
[20129.828497] mmc0: sdhci: Host ctl2: 0x0000000f
[20129.828509] mmc0: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x107f0008
[20129.828521] mmc0: sdhci: ============================================
[20191.267058] mmc0: cqhci: timeout for tag 0, qcnt 31
[20191.267108] mmc0: cqhci: ============ CQHCI REGISTER DUMP ===========
[20191.267119] mmc0: cqhci: Caps: 0x000030c8 | Version: 0x00000510
[20191.267131] mmc0: cqhci: Config: 0x00001001 | Control: 0x00000000
[20191.267142] mmc0: cqhci: Int stat: 0x00000000 | Int enab: 0x00000006
[20191.267153] mmc0: cqhci: Int sig: 0x00000006 | Int Coal: 0x00000000
[20191.267165] mmc0: cqhci: TDL base: 0x0305c1e0 | TDL up32: 0x00000000
[20191.267176] mmc0: cqhci: Doorbell: 0x7fffffff | TCN: 0x00000000
[20191.267186] mmc0: cqhci: Dev queue: 0x00000001 | Dev Pend: 0x7fffffff
[20191.267197] mmc0: cqhci: Task clr: 0x00000000 | SSC1: 0x00011000
[20191.267207] mmc0: cqhci: SSC2: 0x00000001 | DCMD rsp: 0x00000800
[20191.267217] mmc0: cqhci: RED mask: 0xfdf9a080 | TERRI: 0x1e2e1f0d
[20191.267227] mmc0: cqhci: Resp idx: 0x0000000d | Resp arg: 0x00000001
[20191.267239] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
[20191.267250] mmc0: sdhci: Sys addr: 0x000000d8 | Version: 0x00000005
[20191.267262] mmc0: sdhci: Blk size: 0x00007200 | Blk cnt: 0x00000000
[20191.267273] mmc0: sdhci: Argument: 0x00018000 | Trn mode: 0x00000033
[20191.267285] mmc0: sdhci: Present: 0x13f700f0 | Host ctl: 0x00000034
[20191.267295] mmc0: sdhci: Power: 0x0000000d | Blk gap: 0x00000000
[20191.267305] mmc0: sdhci: Wake-up: 0x00000000 | Clock: 0x00000007
[20191.267315] mmc0: sdhci: Timeout: 0x0000000e | Int stat: 0x00000000
[20191.267324] mmc0: sdhci: Int enab: 0x02ff4000 | Sig enab: 0x02ff4000
[20191.267336] mmc0: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000
[20191.267346] mmc0: sdhci: Caps: 0x226dc881 | Caps_1: 0x08000007
[20191.267356] mmc0: sdhci: Cmd: 0x00000d1e | Max curr: 0x00000000
[20191.267367] mmc0: sdhci: Resp[0]: 0x00000900 | Resp[1]: 0x00000001
[20191.267377] mmc0: sdhci: Resp[2]: 0x00000800 | Resp[3]: 0x000007cd
[20191.267388] mmc0: sdhci: Host ctl2: 0x0000000f
[20191.267400] mmc0: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x107f00d8
[20191.267411] mmc0: sdhci: ============================================
You can first use mmc extcsd read /dev/mmcblk0 to read the eMMC EXT_CSD information. Pay particular attention to DEVICE_LIFE_TIME_EST_TYP_A/B and PRE_EOL_INFO to assess the eMMC’s lifetime and wear status.
If the eMMC is in good condition, replace the Blade 3 DTB file at /lib/firmware/6.1.0-1027-rockchip/device-tree/rockchip/rk3588-mixtile-blade3.dtb with rk3588-mixtile-blade3.dtb , and reboot
https://downloads.mixtile.com/cluster-box/rk3588-mixtile-blade3.dtb
Just tried it out /* the command you stated was not correct */:
mixtile@blade3n1:~$ sudo mmc extcsd read /dev/mmcblk0 | grep -E "LIFE_TIME|PRE_EOL"
eMMC Life Time Estimation A [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_A]: 0x01
eMMC Life Time Estimation B [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_B]: 0x01
eMMC Pre EOL information [EXT_CSD_PRE_EOL_INFO]: 0x01
At least according to Google, 0x01 means “all OK, plenty of reserves”. What I did find in the syslog, though, was a purported issue with the voltage regulator:
[ 3.012547] dwmmc_rockchip fe2c0000.mmc: Looking up vmmc-supply from device tree
[ 3.012609] sdhci-dwcmshc fe2e0000.mmc: Looking up vmmc-supply from device tree
[ 3.012619] sdhci-dwcmshc fe2e0000.mmc: Looking up vmmc-supply property in node /mmc@fe2e0000 failed
[ 3.012627] dwmmc_rockchip fe2c0000.mmc: Looking up vqmmc-supply from device tree
[ 3.012650] sdhci-dwcmshc fe2e0000.mmc: Looking up vqmmc-supply from device tree
[ 3.012667] sdhci-dwcmshc fe2e0000.mmc: Looking up vqmmc-supply property in node /mmc@fe2e0000 failed
[ 3.012847] dwmmc_rockchip fe2c0000.mmc: Failed getting OCR mask: -22
Update: I’ve just installed the DTB you’d sent me and rebooted the whole cluster, but I still get the voltage regulator issue message in the syslog.
Update #2: After decompiling your device tree blob, I’ve found these two lines, which could be problematic because they contradict with the warnings I sent you the last time:
vmmc-supply = <0x105>;
vqmmc-supply = <0x106>;
Additionally, the eMMC seems to be operated with maximum speed (400 MHz with enhanced strobe):
mmc-hs400-1_8v;
mmc-hs400-enhanced-strobe;
I’ve also found a recommendation to slow down the eMMC chip a bit, but before implementing it, I’d like to ask you whether this makes sense:
mmc-hs200-1_8v;
no-1-8-v;