Symptoms

Multiple microservices is in CrashLoopBackOff status.

On kube-system namespace, kube-router pod is in ErrImagePullBackOff status.


Checking the event of the kube-router pod using kubectl describe pod we can see that the following message:

Warning  Failed     5m41s (x3 over 6m24s)  kubelet, link8s.cloud.local  Failed to pull image "cloudnativelabs/kube-router": rpc error: code = Unknown desc = manifest for docker.io/cloudnativelabs/kube-router:latest not found
  Warning  Failed     5m2s (x5 over 6m23s)   kubelet, link8s.cloud.local  Error: ImagePullBackOff
  Normal   Pulling    4m47s (x4 over 6m25s)  kubelet, link8s.cloud.local  pulling image "cloudnativelabs/kube-router"
  Warning  Failed     4m45s (x4 over 6m24s)  kubelet, link8s.cloud.local  Error: ErrImagePull
  Normal   BackOff    84s (x19 over 6m23s)   kubelet, link8s.cloud.local  Back-off pulling image "cloudnativelabs/kube-router

Cause

Kubernetes cluster was not able to get the manifest of the latest kube-router.


Resolution

Downgrade kube-router to the last working version.

Firstly verify which version was working by using docker to pull the image:

docker image pull cloudnativelabs/kube-router:v1.5.3

Then set the working version in daemonset:

kubectl -n kube-system set image ds/kube-router kube-router=cloudnativelabs/kube-router:v1.5.3

Restart kube-router pod afterwards if necessary.