Symptoms
Some of the kubernetes system pods are not ready:
# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-86c58d9df4-ffngx 0/1 Pending 0 3h22m coredns-86c58d9df4-x866g 0/1 Completed 14 376d etcd-link8s.cloud.local 1/1 Running 1 3h24m kube-apiserver-link8s.cloud.local 1/1 Running 1 3h24m kube-controller-manager-link8s.cloud.local 1/1 Running 1 3h24m kube-proxy-q686h 1/1 Running 1 3h22m kube-scheduler-link8s.cloud.local 1/1 Running 1 3h24m tiller-deploy-778f674bf5-8qmcg 0/1 Pending 0 3h22m
Kube node is in NotReady state:
# kubectl get nodes NAME STATUS ROLES AGE VERSION link8s.cloud.local NotReady master 2y48d v1.19.3
Node description indicates the error message "KubeletNotReady CSINode is not yet initialized" :
# kubect describe node link8s.cloud.local Taints: node.kubernetes.io/not-ready:NoSchedule Unschedulable: false Lease: Failed to get lease: the server could not find the requested resource Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- MemoryPressure False Sun, 01 Nov 2020 04:48:21 +0200 Sat, 06 Oct 2018 13:20:51 +0300 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Sun, 01 Nov 2020 04:48:21 +0200 Sat, 06 Oct 2018 13:20:51 +0300 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Sun, 01 Nov 2020 04:48:21 +0200 Fri, 14 Sep 2018 22:39:18 +0300 KubeletHasSufficientPID kubelet has sufficient PID available Ready False Sun, 01 Nov 2020 04:48:21 +0200 Sat, 31 Oct 2020 23:59:14 +0200 KubeletNotReady CSINode is not yet initialized Addresses:
Cause
Kubernetes was upgraded to the 1.19.3 version that supports new feature: CSI node object
https://kubernetes-csi.github.io/docs/csi-node-object.html
Resolution
Disable the feature by adding the following lines to the /var/lib/kubelet/config.yaml file:
# tail -2 /var/lib/kubelet/config.yaml featureGates: CSIMigration: false
Restart kubelet service:
# systemctl restart kubelet