はじめに
Kubernetesのクラスタの証明書が切れていないので更新する。
環境
- ホスト：Ubuntu 20.04
- Kubernetes v1.28.3
問題、課題
Kubernetesのクラスタの証明書が切れて、kubectlコマンドが操作できなかった。その解決方法を記載する。
$ kubectl get pod E1127 00:19:14.796944 2753782 memcache.go:265] couldn't get current server API group list: Get "https://172.31.0.18:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-11-27T00:19:14Z is after 2024-11-26T09:50:08Z E1127 00:19:14.800772 2753782 memcache.go:265] couldn't get current server API group list: Get "https://172.31.0.18:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-11-27T00:19:14Z is after 2024-11-26T09:50:08Z
解決
証明書の期限を確認する。
$ date Wed Nov 27 01:15:04 UTC 2024 $ sudo kubeadm certs check-expiration failed to load kubeConfig file /etc/kubernetes/admin.conf: open /etc/kubernetes/admin.conf: permission denied To see the stack trace of this error execute with --v=5 or higher ubuntu@k8s-pub-master:~$ sudo kubeadm certs check-expiration [check-expiration] Reading configuration from the cluster... [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [check-expiration] Error reading configuration from the Cluster. Falling back to default configuration CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED admin.conf Nov 26, 2024 09:50 UTC <invalid> ca no apiserver Nov 26, 2024 09:50 UTC <invalid> ca no apiserver-etcd-client Nov 26, 2024 09:50 UTC <invalid> etcd-ca no apiserver-kubelet-client Nov 26, 2024 09:50 UTC <invalid> ca no controller-manager.conf Nov 26, 2024 09:50 UTC <invalid> ca no etcd-healthcheck-client Nov 26, 2024 09:50 UTC <invalid> etcd-ca no etcd-peer Nov 26, 2024 09:50 UTC <invalid> etcd-ca no etcd-server Nov 26, 2024 09:50 UTC <invalid> etcd-ca no front-proxy-client Nov 26, 2024 09:50 UTC <invalid> front-proxy-ca no scheduler.conf Nov 26, 2024 09:50 UTC <invalid> ca no CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED ca Nov 24, 2033 09:50 UTC 8y no etcd-ca Nov 24, 2033 09:50 UTC 8y no front-proxy-ca Nov 24, 2033 09:50 UTC 8y no
期限が切れていることが確認できた。
masterノードで証明書を更新する。
$ sudo kubeadm certs renew all [renew] Reading configuration from the cluster... [renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [renew] Error reading configuration from the Cluster. Falling back to default configuration certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed certificate for serving the Kubernetes API renewed certificate the apiserver uses to access etcd renewed certificate for the API server to connect to kubelet renewed certificate embedded in the kubeconfig file for the controller manager to use renewed certificate for liveness probes to healthcheck etcd renewed certificate for etcd nodes to communicate with each other renewed certificate for serving etcd renewed certificate for the front proxy client renewed certificate embedded in the kubeconfig file for the scheduler manager to use renewed Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates. #証明書の期限を確認する。 $ sudo kubeadm certs check-expiration [check-expiration] Reading configuration from the cluster... [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED admin.conf Nov 27, 2025 01:22 UTC 364d ca no apiserver Nov 27, 2025 01:22 UTC 364d ca no apiserver-etcd-client Nov 27, 2025 01:22 UTC 364d etcd-ca no apiserver-kubelet-client Nov 27, 2025 01:22 UTC 364d ca no controller-manager.conf Nov 27, 2025 01:22 UTC 364d ca no etcd-healthcheck-client Nov 27, 2025 01:22 UTC 364d etcd-ca no etcd-peer Nov 27, 2025 01:22 UTC 364d etcd-ca no etcd-server Nov 27, 2025 01:22 UTC 364d etcd-ca no front-proxy-client Nov 27, 2025 01:22 UTC 364d front-proxy-ca no scheduler.conf Nov 27, 2025 01:22 UTC 364d ca no CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED ca Nov 24, 2033 09:50 UTC 8y no etcd-ca Nov 24, 2033 09:50 UTC 8y no front-proxy-ca Nov 24, 2033 09:50 UTC 8y no
証明書の再読み込みするために、 kube-apiserver、kube-scheduler、kube-controller-manager、etcd の各コンテナを再起動する
次にクラスタを再起動する。
$ C_NAME=kube-apiserver $ sudo crictl ps | grep $C_NAME 29ca50447c3f9 53743472912306d2d17cab3f458cc57f2012f89ed0e9372a2d2b1fa1b20a8076 12 months ago Running kube-apiserver 3 fe51470d3dca5 kube-apiserver-k8s-pub-master $ sudo crictl stop 29ca50447c3f9 #自動的に再起動していることを確認 $ sudo crictl ps | grep $C_NAME dd4fc502033ef 53743472912306d2d17cab3f458cc57f2012f89ed0e9372a2d2b1fa1b20a8076 1 second ago Running kube-apiserver 4 fe51470d3dca5 kube-apiserver-k8s-pub-master 以下、 kube-scheduler、kube-controller-manager、etcdに対して実施する。
接続性が復元されたことを確認する
$ sudo kubectl get nodes --kubeconfig=/etc/kubernetes/admin.conf k8s-pub-master Ready control-plane 365d v1.28.3 k8s-pub-worker Ready <none> 362d v1.28.3
kubectlの証明書を更新する。
$ rm -rf $HOME/.kube $ mkdir -p $HOME/.kube $ sudo cp -i /etc/kubernetes/admin.conf ~/.kube/config $ sudo chown $(id -u):$(id -g) $HOME/.kube/config
これでpodへのアクセスを確認できた。
$ kubectl get po NAME READY STATUS RESTARTS AGE <pod名> 1/1 Running 0 249d
メモ
以下の記載から、本作業は非推奨のように感じ、Kubernetesのversionのアップグレードが推奨されているようである。
kubeadmはコントロールプレーンのアップグレード時にすべての証明書を更新します。 この機能は、最もシンプルなユースケースに対応するために設計されています。 証明書の更新に特別な要件がなく、Kubernetesのバージョンアップを定期的に行う場合(各アップグレードの間隔が1年未満)、kubeadmがクラスターを最新かつ適度に安全に保つための処理を行います。
