One of my Blade 3 nodes (but only one!) suddenly refuses to start Docker due to a segfault. While investigating the cause, I stumbled upon some alarming error messages in the syslog, which were not there this morning:
mixtile@blade3n1:~$ sudo dmesg | tail -35
[ 136.070813] rockchip_canfd fea70000.can can0: rockchip_canfd_get_berr_counter RX_ERR_CNT=0x00000000, TX_ERR_CNT=0x00000000
[ 196.638569] rockchip_canfd fea70000.can can0: rockchip_canfd_get_berr_counter RX_ERR_CNT=0x00000000, TX_ERR_CNT=0x00000000
[ 198.920284] kauditd_printk_skb: 14 callbacks suppressed
[ 198.920288] audit: type=1400 audit(1779901034.770:26): apparmor="DENIED" operation="change_onexec" info="label not found" error=-2 profile="unconfined" name="ubuntu_pro_apt_news" pid=5817 comm="(python3)"
[ 198.946191] audit: type=1400 audit(1779901034.793:27): apparmor="DENIED" operation="change_onexec" info="label not found" error=-2 profile="unconfined" name="ubuntu_pro_esm_cache" pid=5821 comm="(python3)"
[ 219.461143] sdhci-dwcmshc fe2e0000.mmc: swiotlb buffer is full (sz: 65536 bytes), total 32768 (slots), used 32576 (slots)
[ 219.461168] cqhci_dma_map: sg-len: 20
[ 219.461187] mmc0: cqhci_prep_tran_desc: unable to map sg lists, -12
[ 219.461193] mmc0: cqhci: failed to setup tx desc: -12
[ 219.461200] I/O error, dev mmcblk0, sector 68165120 op 0x1:(WRITE) flags 0x4000 phys_seg 20 prio class 2
[ 219.464118] sdhci-dwcmshc fe2e0000.mmc: swiotlb buffer is full (sz: 65536 bytes), total 32768 (slots), used 32576 (slots)
[ 219.464136] cqhci_dma_map: sg-len: 20
[ 219.464149] mmc0: cqhci_prep_tran_desc: unable to map sg lists, -12
[ 219.464155] mmc0: cqhci: failed to setup tx desc: -12
[ 219.464161] I/O error, dev mmcblk0, sector 68170240 op 0x1:(WRITE) flags 0x4000 phys_seg 20 prio class 2
[ 219.489421] sdhci-dwcmshc fe2e0000.mmc: swiotlb buffer is full (sz: 65536 bytes), total 32768 (slots), used 32608 (slots)
[ 219.489462] cqhci_dma_map: sg-len: 20
[ 219.489490] mmc0: cqhci_prep_tran_desc: unable to map sg lists, -12
[ 219.489501] mmc0: cqhci: failed to setup tx desc: -12
[ 219.489515] I/O error, dev mmcblk0, sector 11416064 op 0x1:(WRITE) flags 0x4000 phys_seg 20 prio class 2
[ 219.594808] EXT4-fs warning (device mmcblk0p3): ext4_end_bio:343: I/O error 10 writing to inode 763 starting block 8521600)
[ 219.595209] Buffer I/O error on device mmcblk0p3, logical block 8515584
[ 219.595361] Buffer I/O error on device mmcblk0p3, logical block 8515585
[ 219.595384] Buffer I/O error on device mmcblk0p3, logical block 8515586
[ 219.595401] Buffer I/O error on device mmcblk0p3, logical block 8515587
[ 219.595419] Buffer I/O error on device mmcblk0p3, logical block 8515588
[ 219.595436] Buffer I/O error on device mmcblk0p3, logical block 8515589
[ 219.595450] Buffer I/O error on device mmcblk0p3, logical block 8515590
[ 219.595468] Buffer I/O error on device mmcblk0p3, logical block 8515591
[ 219.595485] Buffer I/O error on device mmcblk0p3, logical block 8515592
[ 219.595500] Buffer I/O error on device mmcblk0p3, logical block 8515593
[ 219.620064] EXT4-fs warning (device mmcblk0p3): ext4_end_bio:343: I/O error 10 writing to inode 763 starting block 1427328)
[ 404.810096] rockchip_canfd fea70000.can can0: rockchip_canfd_get_berr_counter RX_ERR_CNT=0x00000000, TX_ERR_CNT=0x00000000
[ 946.485048] rockchip_canfd fea70000.can can0: rockchip_canfd_get_berr_counter RX_ERR_CNT=0x00000000, TX_ERR_CNT=0x00000000
[ 1678.309911] rockchip_canfd fea70000.can can0: rockchip_canfd_get_berr_counter RX_ERR_CNT=0x00000000, TX_ERR_CNT=0x00000000
Especially the buffer I/O errors look quite alarming, as if the on-board eMMC storage was about to die within a day or two. What’s wrong here?
Update: I’ve just found the same error messages on all other nodes, too.