I’ve got a new type of error message on one of my Blade 3 nodes, which occurs once in a second while trying to copy the contents (some 1.2 TB) of an external hard drive attached to the Blade 3 to an M.2 NVMe, which is connected to the Blade 3 via Cluster Box backplane:
[ 3597.539034] miop-ep fe150000.pcie: failed to allocate new skb
Where does this error come from? And: Can it have negative consequences like data corruption? Google’s AI says that it’s probably lack of power:
Power Instability (Most Likely): The PCIe device (NVMe/Wi-Fi) may be consuming too much power during boot.
Hi, we now have a new driver that is still being optimized. invite you test it; it should be more stable.
Please confirm that your Blade3 kernel version is 6.1.0-1027-rockchip, and that the cluster box kernel version is 5.15.150.
After downloading and extracting the package from the link, you will get five .ko files. Replace the files in /lib/miop/ on each Blade3 with the following four files:
miop-ep.ko
miop-ep-net.ko
miop-reg.ko
pcie-ep-rk35.ko
After replacing them, you need to manually power off the Blade3. This is very important.
Then replace the miop.ko file on the cluster box at:
/lib/modules/5.15.150/miop.ko
After that, reboot the cluster box.
If the replacement is successful, you will see logs like peer[3] new-arch init printed on the Blade3.
Yes, after testing the new driver, there are no DMA timeout issues. Also, if no additional NVMe drive is connected, the iperf3 throughput is 16 Gbps; if an NVMe drive is connected, the speed is 10 Gbps
Do I have to leave /lib/miop/miop.ko on the four nodes untouched? The miop.ko, which comes with your zip file, seems to be made for the control board, right?
How do I unzip the zip file on the control board? OpenWRT seems not to come with the necessary unzip utility. Trying to get it, ends with such an error message:
mixtile@ClusterBox:~$ sudo opkg update
Downloading https://downloads.openwrt.org/releases/23.05-SNAPSHOT/targets/ramips/mt7620/packages/Packages.gz
*** Failed to download the package list from https://downloads.openwrt.org/releases/23.05-SNAPSHOT/targets/ramips/mt7620/packages/Packages.gz
[…]
mixtile@ClusterBox:~$ sudo opkg install unzip
Unknown package 'unzip'.
Collected errors:
* opkg_install_cmd: Cannot install package unzip.
Update: The transfer of the driver file did work via PC (unpack), Google Drive, and wget on the control board. Unfortunately, after replacing the file and rebooting (the nodes have already been updated and powered off), the nodes are no longer powered on! Even a manual nodectl poweron -n 1 ends in a segmentation fault. This is what the syslog says:
[ 40.652954] shpchp 0000:02:0c.0: card - bus=0x2, slot = 0x0 irq=4
[ 72.013184] export_store: invalid GPIO 499
[ 72.017829] do_page_fault(): sending SIGSEGV to nodectl for invalid read access from 0000004c
[ 72.026667] epc = 77dbe070 in libc.so[77d43000+ab000]
[ 72.031873] ra = 77dbe068 in libc.so[77d43000+ab000]
After replacing the driver on Blade 3, it must be powered off manually; it cannot be powered off through the control board. miop.ko is only for the control board.
You can try some recovery methods, such as removing Blade 3 separately, powering it on, and replacing the driver, or reflashing the latest firmware and then replacing it again.
If you are unable to replace it successfully, we expect to release a new firmware next week.
Yes, I have! In fact, I even made sure that the power LEDs on all four nodes were off before replacing the driver on the control board. Should I have removed the blades beforehand? You did not tell me to do that.
I’ve already tried that. No use. The problem must lie in the miop.ko code on the control board.
I’ve also done that. I’m glad that I made a backup of the control board just before replacing the driver, so that I could reset the board and then upload the backup. Unfortunately, the new (and presumably faulty) miop.ko is still there, so it was apparently not overwritten by the reset process:
mixtile@ClusterBox:~$ ls -al /lib/modules/5.15.150/miop.ko
-rw-r--r-- 1 root root 66090 May 17 17:01 /lib/modules/5.15.150/miop.ko
Shall I flash the board with the current firmware version from your website, or rather wait till you’ve got the new version ready? Please note that I originally wanted to put my cluster into productive use by the end of last week, so I’m losing time now.
And: Will the new image be based on the recent version of OpenWRT?
We have a 2025-year compiled OpenWrt, but we can’t find the image. We are compiling a new OpenWrt firmware, which will take 4 to 5 hours. Tomorrow we will provide you with the OpenWrt firmware link. And will fully verify it before sharing
24.10 was the stable version of OpenWRT by summer of 2025. By now (May 2026), 25.12.4 is the recent version. 23.05. has been declared end-of-life by its maintainers.
OK, I’ve just flashed the control board, as described by you. Unfortunately, I can’t install or update any software, as I get this error message:
Executing package manager
Downloading https://downloads.openwrt.org/releases/23.05-SNAPSHOT/targets/ramips/mt7620/packages/Packages.gz
*** Failed to download the package list from https://downloads.openwrt.org/releases/23.05-SNAPSHOT/targets/ramips/mt7620/packages/Packages.gz
Downloading https://downloads.openwrt.org/releases/23.05-SNAPSHOT/packages/mipsel_24kc/base/Packages.gz
*** Failed to download the package list from https://downloads.openwrt.org/releases/23.05-SNAPSHOT/packages/mipsel_24kc/base/Packages.gz
Downloading https://downloads.openwrt.org/releases/23.05-SNAPSHOT/packages/mipsel_24kc/luci/Packages.gz
*** Failed to download the package list from https://downloads.openwrt.org/releases/23.05-SNAPSHOT/packages/mipsel_24kc/luci/Packages.gz
Downloading https://downloads.openwrt.org/releases/23.05-SNAPSHOT/packages/mipsel_24kc/packages/Packages.gz
*** Failed to download the package list from https://downloads.openwrt.org/releases/23.05-SNAPSHOT/packages/mipsel_24kc/packages/Packages.gz
Errors
Collected errors:
* opkg_download: Failed to download https://downloads.openwrt.org/releases/23.05-SNAPSHOT/targets/ramips/mt7620/packages/Packages.gz, wget returned 8.
* opkg_download: Failed to download https://downloads.openwrt.org/releases/23.05-SNAPSHOT/packages/mipsel_24kc/base/Packages.gz, wget returned 8.
* opkg_download: Failed to download https://downloads.openwrt.org/releases/23.05-SNAPSHOT/packages/mipsel_24kc/luci/Packages.gz, wget returned 8.
* opkg_download: Failed to download https://downloads.openwrt.org/releases/23.05-SNAPSHOT/packages/mipsel_24kc/packages/Packages.gz, wget returned 8.
The opkg update command failed with code 4.
So your image still contains version 23.05 of OpenWRT, right? Where can I then get software for it? Do you have a repo I can use?
Neither does the internal network work correctly, as I can’t even ping the nodes from the control board:
Update: I think I’ve hunted down the cause. It’s outdated ARP entries! The only problem is: I can’t remove them, as the ip neigh flush command thinks there is nothing to remove:
Didn’t you flash the new Blade3 image? Because this OpenWrt firmware must be used together with the Blade3 firmware released at the same time in order to work properly. We tested it and it works fine.
Fix repo command
sed -i 's/23.05-SNAPSHOT/23.05.5/g' /etc/opkg/distfeeds.conf
opkg update
Due to the limited performance of the SoC used on our control board, upgrading to a newer OpenWrt release may introduce instability or additional compatibility risks. For this reason, we have decided to keep the control board on the most stable system version that has been fully validated on our hardware.