ClusterBox MIOP Driver Update Instructions

bobby · April 28, 2025, 9:08am

ClusterBox MIOP Driver Update:

Fix random driver collapse.
Resolve issue of full PCIe queue.

Below instructions describe how to install the MIOP driver on ClusterBox Controller Board and how to update the Blade3 firmware.

1. Installing the MIOP Driver on Controller Board

1.1 Firmware Download Link

You can download the latest OpenWrt ipk package from the following link:

miop_xx_mipsel_24kc.ipk Download Link

1.2 Upload the IPK File to the ClusterBox Controller Board

Transfer the miop_1.0_mipsel_24kc.ipk file to the /tmp directory on the Controller Board device over the network (for example, using the scp command).

Example command (executed on your PC):

scp miop_1.0_mipsel_24kc.ipk root@<Controller_Board_IP>:/tmp/

Or use ssh to transfer the file:

cat /your/path/miop_1.0_mipsel_24kc.ipk | ssh root@<Controller_Board_IP_IP> "cat > /tmp/miop_1.0_mipsel_24kc.ipk"

Replace <Controller_Board_IP> with the actual IP address of your Controller Board device.

1.3 Install the MIOP Driver on the Device

Log in to the Controller Board (using ssh), then execute the following commands:

cd /tmp
opkg install miop_1.0_mipsel_24kc.ipk

Once installation is complete, the MIOP driver will be successfully deployed.

2. Updating Blade3 Firmware

2.1 Firmware Download Link

You can download the latest Blade3 firmware from the following link:

Blade3 Firmware Download Link

Note: Please replace the link with the actual firmware URL as needed.

2.2 Firmware Flashing Method

Refer to the documentation in the following link for detailed flashing instructions:

Blade3 Firmware Flashing Guide

3. Restart and Verify

After installation, reboot the device and wait for a short while.
Then execute the following command to verify:

root@ClusterBox:/# nodectl list
If no device is found, run the rescan command and then the list command to view the device.
03:00.0 Network controller: Mixtile Limited Blade 3 (rev 01)
04:00.0 Network controller: Mixtile Limited Blade 3 (rev 01)
05:00.0 Network controller: Mixtile Limited Blade 3 (rev 01)
06:00.0 Network controller: Mixtile Limited Blade 3 (rev 01)


root@ClusterBox:/# arp
IP address       HW type     Flags       HW address            Mask     Device
10.20.0.22       0x1         0x2         02:0d:36:6d:66:0a     *        pci0
10.20.0.116      0x1         0x2         02:22:86:9a:fc:0a     *        pci0
10.20.0.71       0x1         0x2         02:b3:e8:44:62:0a     *        pci0
10.20.0.148      0x1         0x2         02:78:f7:9e:07:0a     *        pci0
...

4. Appendix

opkg is the package manager for OpenWrt-based systems.
All the operations above require root privileges.

roysbike · April 30, 2025, 8:04pm

Hi, is there any support for the miop driver in Talos OS?

I understand that Talos support is currently in alpha, but I haven’t seen any new commits on GitHub recently.

bobby · May 7, 2025, 3:28am

Hi, miop driver support for Talos is currently in active development. We’ll publish updates and fixes to GitHub once testing is complete.

FlUxIuS · May 20, 2025, 2:56pm

Hello,

I tried following these instruction, but I get the following error:

root@MixtileClusterBox:/home/mixtile# opkg install miop_1.0_mipsel_24kc.ipk
Unknown package 'miop'.
Collected errors:
 * pkg_hash_check_unresolved: cannot find dependency kernel (>= 5.15) for miop
 * pkg_hash_fetch_best_installation_candidate: Packages for miop found, but incompatible with the architectures configured
 * opkg_install_cmd: Cannot install package miop.

Was there any previous update?

Here is my build version :

root@MixtileClusterBox:/home/mixtile# uname -a
Linux MixtileClusterBox 5.10.161 #0 Tue Jan 3 00:24:21 2023 mips GNU/Linux

bobby · May 21, 2025, 10:10am

Hi there! We noticed you might be using a previous version of the control board. We’ve released an upgraded version and are offering free replacements . For details, please check Mixtile Cluster Box: Control Board Upgrade and Shipping Date. To request your new unit, kindly contact Mixtile Support directly via support@mixtile.com

srcshelton · June 2, 2025, 11:57am

@bobby I have upgraded 3 of the 4 nodes in my ClusterBox to the above image (using the physical switch on each blade to enter MaskRom mode and the ‘upgrade_tool uf’, and all have lost their hardware MAC address

1.) Please release an updated image which can correctly read the hardware MAC (as the immediately prior image could);
2.) Please test images for basic regressions such as this before releasing them;
3.) Please update Firmware Images | Mixtile, which currently has nothing post-2022
4.) Even though it’s not an LTS release (the next should be 26.04), it would be great to get an Ubuntu 25.04 image…
5.) What’s the roadmap for enabling firmware updates from the Cluster Box BMC? This has been promised since launch.
6.) It would be really helpful if firmware images were split or otherwise restricted so that the OS portion could be updated to a new release, but a user-data portion was maintained so that - for example - home directories and /etc weren’t wiped-out during upgrades.

Buyuliang · June 4, 2025, 3:26am

Regarding issue 1, here is provided an r8169 driver installation package to solve this problem. The usage method is sudo dpkg -i --force-overwrite r8169-replace-1.0.deb ```
http://downloads.mixtile.com/cluster-box/r8169-replace-1.0.deb

srcshelton · June 4, 2025, 8:38pm

Thanks for that - there is something weird going on:

On installing the driver, I see:

$ sudo dpkg -i --force-overwrite r8169-replace-1.0.deb
(Reading database ... 148634 files and directories currently installed.)
Preparing to unpack r8169-replace-1.0.deb ...
卸载原有 r8169 模块（如果已加载）...
Unpacking r8169-replace (1.0) over (1.0) ...
Setting up r8169-replace (1.0) ...
更新模块依赖...
加载新的 r8169 模块...
[  772.136766] r8169 0002:23:00.0: rk_get_eth_addr: mac address: c8:8x:xx:xx:xx:0a
[  772.166779] r8169 0002:24:00.0: rk_get_eth_addr: mac address: c8:8x:xx:xx:xx:0c

… and I can start dhcpcd, and everything works. But on reboot, the kernel log still reads:

$ dmesg | grep r8169
[    3.738869] r8169 0002:23:00.0: enabling device (0000 -> 0003)
[    3.752240] r8169 0002:23:00.0: can't read MAC address, setting random one
[    3.766088] r8169 0002:23:00.0 eth0: RTL8125B, 1e:ec:9f:68:c2:4c, XID 641, IRQ 137
[    3.766101] r8169 0002:23:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko]
[    3.766293] r8169 0002:24:00.0: enabling device (0000 -> 0003)
[    3.779005] r8169 0002:24:00.0: can't read MAC address, setting random one
[    3.788334] r8169 0002:24:00.0 eth1: RTL8125B, 3a:5d:59:b0:be:91, XID 641, IRQ 138
[    3.788364] r8169 0002:24:00.0 eth1: jumbo features [frames: 9194 bytes, tx checksumming: ko]
[    3.790249] r8169 0002:24:00.0 enP2p36s0: renamed from eth1
[    3.806235] r8169 0002:23:00.0 enP2p35s0: renamed from eth0
[    8.568529] RTL8226B_RTL8221B 2.5Gbps PHY r8169-2-2300:00: attached PHY driver (mii_bus:phy_addr=r8169-2-2300:00, irq=MAC)
[    8.758486] r8169 0002:23:00.0 enP2p35s0: Link is Down
[    8.801654] RTL8226B_RTL8221B 2.5Gbps PHY r8169-2-2400:00: attached PHY driver (mii_bus:phy_addr=r8169-2-2400:00, irq=MAC)
[    8.995047] r8169 0002:24:00.0 enP2p36s0: Link is Down
[   12.185462] r8169 0002:23:00.0 enP2p35s0: Link is Up - 2.5Gbps/Full - flow control rx/tx

However, after doing an apt-get dist-upgrade (… which, co-incidentally, included updating the package u-boot-mixtile-blade3, which might be significant if uboot is responsible for passing the MAC address data from the hardware to the kernel?), subsequent restarts do seem to have picked-up the correct MAC addresses.

Unfortuantely though, with this new image nodectl reboot -n {x} no longer seems to do anything. Once updated at least, nodes do appear to be able to come back up after a shutdown -r now.

(And if I shutdown -h now on a node and then nodectl poweron --all from the BMC then similarly nothing now seems to occur. nodectl poweroff --all still appears to be implemented but non-operational)

srcshelton · June 4, 2025, 9:02pm

Worth noting that I have one blade which I’ve not yet updated to the latest image (it’s on the prior 24.04 image without the updated MIOP module) and it’s also unresponsive to reboot/poweron/poweroff requests… although I guess this could be a consequence of only the BMC and three of the four nodes running the updated MIOP code? This does suggest, though, that it’s not the content of the most recent image which is preventing power-state control from working.

Unfortuantely, the remaining node is panicking on boot (Oops: Fatal exception in interrupt) and then locking-up such that it’s unresponsive even on the serial console – and so without the ability to reboot or change its power-state from the BMC, it looks as if the only option I have is to re-image it and lose its contents

Buyuliang · June 5, 2025, 8:06am

You can conduct the following tests. If possible, please provide the log to me:

Check the mac address of the vendor partition
vendor_storage -r VENDOR_LAN_MAC_ID -t hex -i /dev/null
Write the Mac address you want to set:

vendor_storage -w VENDOR_LAN_MAC_ID -t string -i AABBCCDDEEFFAABBCCDDEEFE

Reinstall the r8169 driver

dpkg-i --force-overwrite r8169-replace-1.0.deb

[772.136766] r8169 0002: 2300.0: rk_get_eth_addr: mac address: c8:8x:xx:xx:xx:0a
[772.166779] R816690002:2400.0: rk_get_eth_addr: mac address: c8:8x:xx:xx:xx:0c

Use the “ip addr” command to check if the mac addresses of the network port are consistent
Restart to check the mac address

srcshelton · June 5, 2025, 1:39pm

Ah, I think this might have been user-error: I’ve got so used to running nodectl console ... as the mixtile user that I think I may have forgotten to use sudo for power-state operations.

Although it would be a valuable feature-request for a future BMC firmware update to issue a warning if nodectl is executed with insufficient privileges, rather than silently failing (and leading to confusion such as the above).

srcshelton · June 5, 2025, 1:52pm

I’ve now done a do-release-upgrade on all nodes to bring them up to Ubuntu 25.04, but maintaining the original 6.1.0-1027-rockchip kernel, and the good news is that the MAC addresses have been maintained.

I definitely did see reboots with the updated driver where the correct MAC address was not used, but that was prior to the distro updates.

I still have the one problematic node which, after several power-downs, now seems to be responding correctly - so I’ll perform the update on that shortly and record the information you’ve requested as I go, as well as trying to note whether it is at all related to the u-boot upgrade when things start working.

I notice that with 6.1.0-1016-rockchip I had:

# lsmod | grep miop
miop_ep                16384  0
miop_ep_net            20480  0
miop_reg               16384  3 miop_ep_net,pcie_ep_rk35,miop_ep
miop                   73728  0

… whilst with 6.1.0-1027-rockchip I see:

$ lsmod | grep miop
miop_ep                16384  0
miop_ep_net            20480  0
miop_reg               16384  3 miop_ep_net,pcie_ep_rk35,miop_ep
$ sudo modprobe miop
modprobe: FATAL: Module miop not found in directory /lib/modules/6.1.0-1027-rockchip

I assume that 1016 was the prior Ubuntu 24.04 release and 1017 is the release on this page – has the miop module intentionally been built-in/removed/merged with the other three modules?

After a complete power-down for more than 30 seconds, node 1 is now booting but hasn’t yet been upgraded and is showing no carrier on interface pci0, node 2 (fully upgraded) is also showing no carrier on interface pci0, whilst nodes 3 and 4 are working as expected.

Buyuliang · June 6, 2025, 1:38am

The miop.ko driver is the one that needs to be installed on the RC end. There are a total of four drivers that need to be installed on the EP end, namely miop-reg.ko, pcie-ep-rk35.ko, miop-ep-net.ko, and miop-ep.ko. As for why miop.ko was seen installed on 6.1.0-1016-rockchip I’m also quite confused about this driver. Theoretically, it shouldn’t have been installed. Our service doesn’t install miop.ko either. I suggest manually deleting the miop.ko file

srcshelton · June 8, 2025, 12:22pm

Seems to be a file from your package!

mixtile@mixtile-blade3:~$ dpkg -S /lib/modules/miop/miop.ko 
miop: /lib/modules/miop/miop.ko
mixtile@mixtile-blade3:~$ dpkg -l | grep miop
ii  miop                                          1.0.0-1                                                arm64        miop TCP/IP over PCIe device driver.
mixtile@mixtile-blade3:~$ modinfo /lib/modules/miop/miop.ko 
filename:       /lib/modules/miop/miop.ko
license:        MIXTILE
version:        1.0
description:    Mixtile TCP/IP over PCIe host driver
author:         Martin Liu <martin@mixtile.com>
srcversion:     6F8B498A16A68EE999F9633
alias:          pci:v00004586d0000B6F2sv*sd*bc*sc*i*
depends:        
name:           miop
vermagic:       6.1.0-1016-rockchip SMP mod_unload modversions aarch64