I'm testing the latest Kubernetes (1.12.0) on 3 nodes (using Digital Ocean VMs) - running latest version of CentOS 7.5. Everything was working fine until the master was rebooted.
Then, kubectl get nodes showed this error:
"The connection to the server 68.183.18.37:6443 was refused - did you specify the right host or port?"
Upon further investigation, the relevant Docker containers have stopped, i.e. only this container is running. (docker ps)
k8s_kube-scheduler_kube-scheduler-master_kube-system_
Prior to the reboot, the following Docker containers are running:
k8s_kube-proxy_kube-proxy-
k8s_kube-scheduler_kube-scheduler-master_
k8s_kube-apiserver_kube-apiserver-master_kube-system_
k8s_kube-controller-manager_kube-controller
k8s_etcd_etcd-master_kube-system_
Docker daemon itself is runing, although with errors, i.e.
------------------------------------------------------------------------------------------
Main PID: 1138 (dockerd-current)
CGroup: /system.slice/docker.service
├─1138 /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json --selinux-enabled --log-driver=journald --signature-verification=false --storage-driver overlay2
├─1166 /usr/bin/docker-containerd-current -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc --runtime-args --systemd-cgroup=true
├─4188 /usr/bin/docker-containerd-shim-current 07c507b88a4cf25f72dc4bb08ec3e305876a025d4ea7e821ec77b64ea2dc2030 /var/run/docker/libcontainerd/07c507b88a4cf25f72dc4bb08ec3e305876a025d4ea7e821ec77b64ea2dc2030 /usr/libexec/docker/docker-runc-current
├─4199 /usr/bin/docker-containerd-shim-current 70954656660d5858fdb3289863bd0246332ea1359eaba002f0a264634db90b44 /var/run/docker/libcontainerd/70954656660d5858fdb3289863bd0246332ea1359eaba002f0a264634db90b44 /usr/libexec/docker/docker-runc-current
├─4212 /usr/bin/docker-containerd-shim-current 8db54bf24e92b02677772688842b0b646ca99dd53a54845c106fba88f7f20127 /var/run/docker/libcontainerd/8db54bf24e92b02677772688842b0b646ca99dd53a54845c106fba88f7f20127 /usr/libexec/docker/docker-runc-current
├─4227 /usr/bin/docker-containerd-shim-current 1d8a9f77e69de87af5f7835282343747936f0986073b7549b924b80553e6bf6b /var/run/docker/libcontainerd/1d8a9f77e69de87af5f7835282343747936f0986073b7549b924b80553e6bf6b /usr/libexec/docker/docker-runc-current
└─4400 /usr/bin/docker-containerd-shim-current 6faa817ee3df1f19672d749d4135c04969eb42cd99f1512fdc1cceb7db979324 /var/run/docker/libcontainerd/6faa817ee3df1f19672d749d4135c04969eb42cd99f1512fdc1cceb7db979324 /usr/libexec/docker/docker-runc-current
Sep 30 09:39:26 master dockerd-current[1138]: E0930 09:39:26.506346 1 reflector.go:134] k8s.io/client-go/informers/factory.go:131: Failed to list *v1.ReplicaSet: Get https://68.183.18.37:6443/apis/apps/v1/replicasets?limit=500&resourceVersion=0: dial tcp 68.183.18.37:6443: connect: connection refused
Sep 30 09:39:26 master dockerd-current[1138]: E0930 09:39:26.508218 1 reflector.go:134] k8s.io/client-go/informers/factory.go:131: Failed to list *v1.PersistentVolume: Get https://68.183.18.37:6443/api/v1/persistentvolumes?limit=500&resourceVersion=0: dial tcp 68.183.18.37:6443: connect: connection refused
Sep 30 09:39:26 master dockerd-current[1138]: E0930 09:39:26.540676 1 reflector.go:134] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:178: Failed to list *v1.Pod: Get https://68.183.18.37:6443/api/v1/pods?fieldSelector=status.phase%21%3DFailed%2Cstatus.phase%21%3DSuc...: dial tcp 68.183.18.37:6443: connect: connection refused
Sep 30 09:39:26 master dockerd-current[1138]: E0930 09:39:26.542009 1 reflector.go:134] k8s.io/client-go/informers/factory.go:131: Failed to list *v1.Node: Get https://68.183.18.37:6443/api/v1/nodes?limit=500&resourceVersion=0: dial tcp 68.183.18.37:6443: connect: connection refused
Sep 30 09:39:26 master dockerd-current[1138]: E0930 09:39:26.543070 1 reflector.go:134] k8s.io/client-go/informers/factory.go:131: Failed to list *v1beta1.PodDisruptionBudget: Get https://68.183.18.37:6443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0: dial tcp 68.183.18.37:6443: connect: connection refused
Sep 30 09:39:26 master dockerd-current[1138]: E0930 09:39:26.576774 1 reflector.go:134] k8s.io/client-go/informers/factory.go:131: Failed to list *v1.PersistentVolumeClaim: Get https://68.183.18.37:6443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0: dial tcp 68.183.18.37:6443: connect: connection refused
Sep 30 09:39:26 master dockerd-current[1138]: E0930 09:39:26.578579 1 reflector.go:134] k8s.io/client-go/informers/factory.go:131: Failed to list *v1.ReplicationController: Get https://68.183.18.37:6443/api/v1/replicationcontrollers?limit=500&resourceVersion=0: dial tcp 68.183.18.37:6443: connect: connection refused
Sep 30 09:39:26 master dockerd-current[1138]: E0930 09:39:26.579668 1 reflector.go:134] k8s.io/client-go/informers/factory.go:131: Failed to list *v1.Service: Get https://68.183.18.37:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 68.183.18.37:6443: connect: connection refused
Sep 30 09:39:26 master dockerd-current[1138]: E0930 09:39:26.595324 1 reflector.go:134] k8s.io/client-go/informers/factory.go:131: Failed to list *v1.StorageClass: Get https://68.183.18.37:6443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0: dial tcp 68.183.18.37:6443: connect: connection refused
Sep 30 09:39:26 master dockerd-current[1138]: E0930 09:39:26.596878 1 reflector.go:134] k8s.io/client-go/informers/factory.go:131: Failed to list *v1.StatefulSet: Get https://68.183.18.37:6443/apis/apps/v1/statefulsets?limit=500&resourceVersion=0: dial tcp 68.183.18.37:6443: connect: connection refused
------------------------------------------------------------------------------------------
Any idea why this is happening? Any settings that should be set on the CentOS ?
I tried the same setup on Ubuntu 16.04 - everything works perfectly even after a reboot. All the docker containers are up and running.
Thanks!
