Nightly K8s Matrix CI 2023-10-08: Failed

[weizhoublue]

action url: https://github.com/spidernet-io/spiderpool/actions/runs/6449743980

[ty-dc]

In the use case N00006, in order to simulate the scenario where the node restarts and the vlan sub-interface is automatically created, the operation of deleting the vlan sub-interface of the node and then restarting the Pod resulted in the Pod being unable to restart, and the following error was reported:

-------- kubectl get pod -A -o wide
NAMESPACE            NAME                                                         READY   STATUS        RESTARTS   AGE    IP              NODE                                 NOMINATED NODE   READINESS GATES   LABELS
...
local-path-storage   local-path-provisioner-74567d47b4-7zrhp                      1/1     Running       0          26m    172.18.40.179   spiderpool1008202421-worker          <none>           <none>            app=local-path-provisioner,pod-template-hash=74567d47b4
ns4740-192363920     ds-4bfa5c3fb5-5nwws                                          1/1     Running       0          5m3s   172.18.40.47    spiderpool1008202421-worker          <none>           <none>            app=ds-4bfa5c3fb5,controller-revision-hash=59c4654969,pod-template-generation=1
ns4740-192363920     ds-4bfa5c3fb5-8pwcf                                          0/1     Terminating   0          5m8s   172.18.40.6     spiderpool1008202421-control-plane   <none>           <none>            app=ds-4bfa5c3fb5,controller-revision-hash=59c4654969,pod-template-generation=1

Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-dv2pb:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason          Age                   From               Message
  ----     ------          ----                  ----               -------
  Normal   Scheduled       5m8s                  default-scheduler  Successfully assigned ns4740-192363920/ds-4bfa5c3fb5-8pwcf to spiderpool1008202421-control-plane
  Normal   AddedInterface  5m8s                  multus             Add eth0 [fc00:f853:ccd:e793:f::c7/64 172.18.40.6/16] from kube-system/macvlan-vlan0
  Normal   AddedInterface  5m6s                  multus             Add net1 [fd00:95b2::3/120 10.185.192.3/24] from ns4740-192363920/test-multus-6d8e8be07a
  Normal   Pulled          5m5s                  kubelet            Container image "alpine" already present on machine
  Normal   Created         5m5s                  kubelet            Created container samplepod
  Normal   Started         5m5s                  kubelet            Started container samplepod
  Normal   Killing         5m3s                  kubelet            Stopping container samplepod
  Warning  FailedKillPod   32s (x21 over 4m33s)  kubelet            error killing pod: failed to "KillPodSandbox" for "5c8fe27b-afa4-4ff2-9399-93773a4e1f5e" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"402c1fae47b140592e12cd11f3f5f1927788e6196e6f8d313ab09eee9cd4fedc\": delegateDel: error invoking ConflistDel - \"test-multus-6d8e8be07a\": conflistDel: error in getting result from DelNetworkList: Link not found"

[ty-dc]

1. In the local simulation use case scenario, when the vlan sub-interface does not exist on the host, the Pod can be terminated correctly and restarted without the following error:

   Warning  FailedKillPod   32s (x21 over 4m33s)  kubelet            error killing pod: failed to "KillPodSandbox" for "5c8fe27b-afa4-4ff2-9399-93773a4e1f5e" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"402c1fae47b140592e12cd11f3f5f1927788e6196e6f8d313ab09eee9cd4fedc\": delegateDel: error invoking ConflistDel - \"test-multus-6d8e8be07a\": conflistDel: error in getting result from DelNetworkList: Link not found"
   

2. After checking the coordinator log again, the Pod: ns4740-192363920/ds-4bfa5c3fb5-8pwcf only has add logs and no del logs. Is it more likely an occasional bug on macvlan?

e2edebugLog.txt

[ty-dc]

containernetworking/plugins#953

[cyclinder]

refer to containernetworking/plugins#954, still waiting for it.

[cyclinder]

Fixed by #2515