Pcie network interface not working with miop driver

Technical Support
#1

Hello,

I have installed 4 blade3 in a cluster box. n1-3 work fine using the steps to install the driver here: Installing Drivers for Blade 3 Boards | Mixtile

n4 does not show the pcie interface at all.

root@blade3:/home/mixtile# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enP2p35s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
    link/ether c8:8e:d1:90:91:1e brd ff:ff:ff:ff:ff:ff
3: enP2p36s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
    link/ether c8:8e:d1:90:91:20 brd ff:ff:ff:ff:ff:ff

installing the driver after flashing the firmware and writing the boot.img I get the following errors

root@blade3:/home/mixtile# dpkg -i miop-control-arm64.deb 
(Reading database ... 67308 files and directories currently installed.)
Preparing to unpack miop-control-arm64.deb ...
/lib/miop /
rmmod: ERROR: Module miop_ep is not currently loaded
rmmod: ERROR: Module miop_ep_net is not currently loaded
rmmod: ERROR: Module pcie_ep_rk35 is not currently loaded
rmmod: ERROR: Module miop_reg is not currently loaded
/
Unpacking miop-control (1.0.0) over (1.0.0) ...
Setting up miop-control (1.0.0) ...

I can see that the modules installed

root@blade3:/home/mixtile# dpkg -L miop-control             
/.
/._etc
/bin
/bin/miop-loader
/etc
/etc/modprobe.d
/etc/modprobe.d/miop-control.conf
/lib
/lib/miop
/lib/miop/miop-ep-net.ko
/lib/miop/miop-ep.ko
/lib/miop/miop-reg.ko
/lib/miop/miop.ko
/lib/miop/pcie-ep-rk35.ko
/lib/modules
/lib/modules/5.10.66
/lib/modules/5.10.66/miop-control.ko

I can insmod the drivers fine

root@blade3:/home/mixtile# insmod /lib/miop/miop-reg.ko 
root@blade3:/home/mixtile# insmod /lib/miop/miop-ep.ko  
root@blade3:/home/mixtile# insmod /lib/miop/miop-ep-net.ko 
root@blade3:/home/mixtile# insmod /lib/miop/pcie-ep-rk35.ko 
root@blade3:/home/mixtile# lsmod 
Module                  Size  Used by
pcie_ep_rk35           20480  0
miop_ep_net            20480  0
miop_ep                16384  0
miop_reg               16384  3 miop_ep_net,pcie_ep_rk35,miop_ep
miop_control           16384  0
bcmdhd               1380352  0

But the interface still does not show up. Any help is much appreciated.

I have reflashed the firmware found here:
image-release-blade3-debian11-20230505.img

and here: Boot problem cluster box blade3 node2 - #7 by bobby

None of the images get the pcie interface working. I have re-flashed and re-installed the drivers using these guides a few times and nothing seems to get it working.

#2

It appeared that my N4 was entering maskrom mode when booting the cluster with N1-3 connected. I disconnected N1-3 and rebooted the cluster only connecting N4. Initially, I was unable to connect via adb. I was able to confirm using the upgrade tool that the device was in maskrom mode.

I rebooted the cluster with only N4 and verified connectivity with adb. Using the upgrade tool, I confirmed that the device was no longer in maskrom mode. On this boot, I was able to see the device via cluster ssh and web gui. Powered off the cluster, plugged in N1-3 and viola! They all showed.

Sometimes turning it off and on again if the best answer… And plugging and unplugging all the other boards.

#3

The reason for this problem is that the PCIe device does not support hotplug function, and the failure of any node in the current N1~N4 nodes will affect the normal operation of other nodes. To solve this problem, it is necessary to power down and restart the system, and check the connection status of the device. We are actively working to improve this issue, please check the website for the latest status updates.