Symptoms

Some of the kubernetes system pods are not ready:

#  kubectl get pods -n kube-system
NAME                                         READY   STATUS      RESTARTS   AGE
coredns-86c58d9df4-ffngx                     0/1     Pending     0          3h22m
coredns-86c58d9df4-x866g                     0/1     Completed   14         376d
etcd-link8s.cloud.local                      1/1     Running     1          3h24m
kube-apiserver-link8s.cloud.local            1/1     Running     1          3h24m
kube-controller-manager-link8s.cloud.local   1/1     Running     1          3h24m
kube-proxy-q686h                             1/1     Running     1          3h22m
kube-scheduler-link8s.cloud.local            1/1     Running     1          3h24m
tiller-deploy-778f674bf5-8qmcg               0/1     Pending     0          3h22m

Kube node is in NotReady state:

# kubectl get nodes
NAME                 STATUS     ROLES    AGE     VERSION
link8s.cloud.local   NotReady   master   2y48d   v1.19.3

Node description indicates the error message "KubeletNotReady CSINode is not yet initialized" :

# kubect describe node link8s.cloud.local

Taints:             node.kubernetes.io/not-ready:NoSchedule
Unschedulable:      false
Lease:              Failed to get lease: the server could not find the requested resource
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Sun, 01 Nov 2020 04:48:21 +0200   Sat, 06 Oct 2018 13:20:51 +0300   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Sun, 01 Nov 2020 04:48:21 +0200   Sat, 06 Oct 2018 13:20:51 +0300   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Sun, 01 Nov 2020 04:48:21 +0200   Fri, 14 Sep 2018 22:39:18 +0300   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            False   Sun, 01 Nov 2020 04:48:21 +0200   Sat, 31 Oct 2020 23:59:14 +0200   KubeletNotReady              CSINode is not yet initialized
Addresses:



Cause

Kubernetes was upgraded to the 1.19.3 version that supports new feature: CSI node object

https://kubernetes-csi.github.io/docs/csi-node-object.html



Resolution

Disable the feature by adding the following lines to the /var/lib/kubelet/config.yaml file:

# tail -2 /var/lib/kubelet/config.yaml
featureGates:
  CSIMigration: false

Restart kubelet service:

# systemctl restart kubelet



Internal