Kubernetes 업그레이드 (K8S v1.21.x → v1.22.x)

이번 포스트에서는 수동으로 Kubernetes 클러스터 업그레이드 하는 방법을 살펴보고자 합니다.

kubernetes.io 공식 가이드는 맨 하단 참고의 링크를 확인하시기 바랍니다.

 

테스트 환경 정보

## Ubuntu 버전
root@k8s-m0:~# cat /etc/os-release | grep PRETTY
PRETTY_NAME="Ubuntu 20.04.3 LTS"
root@k8s-m0:~# uname -a
Linux k8s-m0 5.4.0-96-generic #109-Ubuntu SMP Wed Jan 12 16:49:16 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

## 3 node HA Control plane 과 1 대의 worker 
root@k8s-m0:~# kubectl get no
NAME     STATUS   ROLES                  AGE     VERSION
k8s-m0   Ready    control-plane,master   68m     v1.21.7
k8s-m1   Ready    control-plane,master   43m     v1.21.7
k8s-m2   Ready    control-plane,master   20m     v1.21.7
k8s-w1   Ready    <none>                 5m27s   v1.21.7

## 주요 바이너리의 버전 확인
root@k8s-m0:~# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.7", GitCommit:"1f86634ff08f37e54e8bfcd86bc90b61c98f84d4", GitTreeState:"clean", BuildDate:"2021-11-17T14:40:08Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}
root@k8s-m0:~# kubelet --version
Kubernetes v1.21.7
root@k8s-m0:~# kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.7", GitCommit:"1f86634ff08f37e54e8bfcd86bc90b61c98f84d4", GitTreeState:"clean", BuildDate:"2021-11-17T14:41:19Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.7", GitCommit:"1f86634ff08f37e54e8bfcd86bc90b61c98f84d4", GitTreeState:"clean", BuildDate:"2021-11-17T14:35:38Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}

 

Kubernetes 업그레이드 절차 요약

1. primary control plane 을 업그레이드 한다.
2. additional control plane 을 업그레이드 한다.
3. worker 노드를 업그레이드 한다.

 

업그레이드 가능 버전 확인

먼저 apt update 를 수행하고 apt-cache 에서 업그레이드 가능한 버전을 확인해 봅니다.

테스트에서는 가장 최신 버전인 1.23.3 을 사용합니다.

root@k8s-m0:~# apt update
Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Hit:2 https://download.docker.com/linux/ubuntu focal InRelease
Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease
Get:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9383 B]
Fetched 345 kB in 23s (14.8 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
19 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@k8s-m0:~# apt-cache madison kubeadm
   kubeadm |  1.23.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.23.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.23.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.23.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.22.6-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   <생략>

 

이제 primary control plane 을 먼저 업그레이드 해보겠습니다.

 

Primary Control Plane 업그레이드

프라이머리 컨트롤 플레인의 업그레이드 절차는 간단히 아래와 같습니다.

1. kubeadm 바이너리 업그레이드

2. kubeadm upgrade plan 으로 사전 확인

3. kubeadm upgrade apply 로 업그레이드 (이때 kubectl version 으로 확인한 Server Version 이 업그레이드 됩니다.)

4. 노드 drain

5. kubectl, kubelet 바이너리 업그레이드

6. kubelet daemon-reload, restart

7. 노드 uncordon

 

수행 결과를 살펴 보겠습니다.

# 1. kubeadm 바이너리 업그레이드
root@k8s-m0:~# apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm=1.22.3-00 && apt-mark hold kubeadm
Canceled hold on kubeadm.
Hit:1 https://download.docker.com/linux/ubuntu focal InRelease
<생략>
Fetched 336 kB in 24s (14.2 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  kubeadm
1 upgraded, 0 newly installed, 0 to remove and 18 not upgraded.
Need to get 8712 kB of archives.
After this operation, 971 kB of additional disk space will be used.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.22.3-00 [8712 kB]
Fetched 8712 kB in 23s (372 kB/s)
(Reading database ... 64135 files and directories currently installed.)
Preparing to unpack .../kubeadm_1.22.3-00_amd64.deb ...
Unpacking kubeadm (1.22.3-00) over (1.21.7-00) ...
Setting up kubeadm (1.22.3-00) ...
kubeadm set on hold.
root@k8s-m0:~#

[참고] apt-mark unhold 와 hold 의 의미
업그레이드 전 apt-mark showhold 를 수행해보면 kubeadm, kubectl, kubelet 은 자동 업데이트, 제거가 되지 않도록 보류(hold) 상태인 것을 알 수 있습니다. 바이너리를 업데이트 하기 위해 unhold -> 패키지 업데이트 -> hold 를 수행하는 것을 알 수 있습니다.

 

# 2. kubeadm upgrade plan 으로 사전 확인
root@k8s-m0:~# kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.21.7
[upgrade/versions] kubeadm version: v1.22.3
W0205 10:07:29.475001   87282 version.go:103] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable.txt": Get "https://storage.googleapis.com/kubernetes-release/release/stable.txt": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
W0205 10:07:29.476348   87282 version.go:104] falling back to the local client version: v1.22.3
[upgrade/versions] Target version: v1.22.3
W0205 10:07:39.796859   87282 version.go:103] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.21.txt": Get "https://storage.googleapis.com/kubernetes-release/release/stable-1.21.txt": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
W0205 10:07:39.797158   87282 version.go:104] falling back to the local client version: v1.22.3
[upgrade/versions] Latest version in the v1.21 series: v1.22.3

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       TARGET
kubelet     4 x v1.21.7   v1.22.3

Upgrade to the latest version in the v1.21 series:

COMPONENT                 CURRENT    TARGET
kube-apiserver            v1.21.7    v1.22.3
kube-controller-manager   v1.21.7    v1.22.3
kube-scheduler            v1.21.7    v1.22.3
kube-proxy                v1.21.7    v1.22.3
CoreDNS                   v1.8.0     v1.8.4
etcd                      3.4.13-0   3.5.0-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.22.3

_____________________________________________________________________


The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.

API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
kubelet.config.k8s.io     v1beta1           v1beta1             no
_____________________________________________________________________

 

# 3. kubeadm upgrade apply 로 업그레이드 (이때 kubectl version 으로 확인한 Server Version 이 업그레이드 됩니다.)
# [upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y  ## <--- y 입력

root@k8s-m0:~# kubeadm upgrade apply v1.22.3
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.22.3"
[upgrade/versions] Cluster version: v1.21.7
[upgrade/versions] kubeadm version: v1.22.3
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
[upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.22.3"...
Static pod: kube-apiserver-k8s-m0 hash: d8693cdc9fb5dd3fefac2a158d5685e8
Static pod: kube-controller-manager-k8s-m0 hash: c3b44cc1fd7121c166d49b72c6a30be6
Static pod: kube-scheduler-k8s-m0 hash: 341d83fb10c50335ea71f49ea327305c
[upgrade/etcd] Upgrading to TLS for etcd
Static pod: etcd-k8s-m0 hash: 39ec3d381c0aa337ca22b0433ae46cf4
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Renewing etcd-server certificate
[upgrade/staticpods] Renewing etcd-peer certificate
[upgrade/staticpods] Renewing etcd-healthcheck-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2022-02-05-10-21-19/etcd.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: etcd-k8s-m0 hash: 39ec3d381c0aa337ca22b0433ae46cf4
<생략>
Static pod: etcd-k8s-m0 hash: 39ec3d381c0aa337ca22b0433ae46cf4
Static pod: etcd-k8s-m0 hash: a6bfdd1cf0b0bfcde91c69798c3c2f61
[apiclient] Found 3 Pods for label selector component=etcd
[upgrade/staticpods] Component "etcd" upgraded successfully!
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests139287456"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2022-02-05-10-21-19/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-apiserver-k8s-m0 hash: d8693cdc9fb5dd3fefac2a158d5685e8
<생략>
Static pod: kube-apiserver-k8s-m0 hash: d8693cdc9fb5dd3fefac2a158d5685e8
Static pod: kube-apiserver-k8s-m0 hash: dff5e288e0f790320cd77e7e1a9aa3db
[apiclient] Found 3 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2022-02-05-10-21-19/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-controller-manager-k8s-m0 hash: c3b44cc1fd7121c166d49b72c6a30be6
<생략>
Static pod: kube-controller-manager-k8s-m0 hash: c3b44cc1fd7121c166d49b72c6a30be6
Static pod: kube-controller-manager-k8s-m0 hash: aa3b83543a7ff5ed8c12c460f2ca4750
[apiclient] Found 3 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2022-02-05-10-21-19/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-scheduler-k8s-m0 hash: 341d83fb10c50335ea71f49ea327305c
<생략>
Static pod: kube-scheduler-k8s-m0 hash: 341d83fb10c50335ea71f49ea327305c
Static pod: kube-scheduler-k8s-m0 hash: 3d3feecdbe65a70d9d6a0cae39754db9
[apiclient] Found 3 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upgrade/postupgrade] Applying label node-role.kubernetes.io/control-plane='' to Nodes with label node-role.kubernetes.io/master='' (deprecated)
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.22" in namespace kube-system with the configuration for the kubelets in the cluster
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[addons] Applied essential addon: kube-proxy

[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.22.3". Enjoy!

[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.

# 버전 확인 (primary control palne 만 업그레이드가 되었으므로, Server Version이 업그레이드 되지 않은 노드가 결과를 보낼 수 있음)
root@k8s-m0:~# kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.7", GitCommit:"1f86634ff08f37e54e8bfcd86bc90b61c98f84d4", GitTreeState:"clean", BuildDate:"2021-11-17T14:41:19Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:35:25Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}

 

[참고] kubeadm upgrade 를 수행하면 관련된 인증서를 자동으로 리뉴얼 합니다. 만약 인증서 리뉴얼을 제외하고자 한다면 --certificate-renewal=false 옵션을 추가해서 실행해야 합니다.

 

# 4. 노드 drain
root@k8s-m0:~# kubectl drain k8s-m0 --ignore-daemonsets
node/k8s-m0 cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-flannel-ds-589g7, kube-system/kube-proxy-hqmll
node/k8s-m0 drained
root@k8s-m0:~# kubectl get no
NAME     STATUS                     ROLES                  AGE     VERSION
k8s-m0   Ready,SchedulingDisabled   control-plane,master   3h48m   v1.21.7
k8s-m1   Ready                      control-plane,master   3h23m   v1.21.7
k8s-m2   Ready                      control-plane,master   3h      v1.21.7
k8s-w1   Ready                      <none>                 165m    v1.21.7

 

[참고] kubelet 과 kubectl 을 업그레이드 하기 전에는 노드를 drain 시켜 노드가 파드 스케줄링을 받지 않도록 하고, 실행 중인 워크로드를 퇴출(evict)합니다.

 

# 5. kubectl, kubelet 바이너리 업그레이드
root@k8s-m0:~# apt-mark unhold kubelet kubectl && apt-get update && apt-get install -y kubelet=1.22.3-00 kubectl=1.22.3-00 && apt-mark hold kubelet kubectl
Canceled hold on kubelet.
Canceled hold on kubectl.
Hit:1 https://download.docker.com/linux/ubuntu focal InRelease
Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease
Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9383 B]
Fetched 345 kB in 24s (14.5 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 17 not upgraded.
Need to get 28.2 MB of archives.
After this operation, 3028 kB of additional disk space will be used.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.22.3-00 [9039 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.22.3-00 [19.1 MB]
Fetched 28.2 MB in 28s (999 kB/s)
(Reading database ... 64135 files and directories currently installed.)
Preparing to unpack .../kubectl_1.22.3-00_amd64.deb ...
Unpacking kubectl (1.22.3-00) over (1.21.7-00) ...
Preparing to unpack .../kubelet_1.22.3-00_amd64.deb ...
Unpacking kubelet (1.22.3-00) over (1.21.7-00) ...
Setting up kubectl (1.22.3-00) ...
Setting up kubelet (1.22.3-00) ...
kubelet set on hold.
kubectl set on hold.


# 6. kubelet daemon-reload, restart
root@k8s-m0:~# systemctl daemon-reload && systemctl restart kubelet

# 7. 노드 uncordon
root@k8s-m0:~# kubectl uncordon k8s-m0
node/k8s-m0 uncordoned
root@k8s-m0:~# kubectl get no
NAME     STATUS   ROLES                  AGE     VERSION
k8s-m0   Ready    control-plane,master   4h16m   v1.22.3
k8s-m1   Ready    control-plane,master   3h51m   v1.21.7
k8s-m2   Ready    control-plane,master   3h28m   v1.21.7
k8s-w1   Ready    <none>                 3h13m   v1.21.7

프라이머리 컨트롤 플레인이 v1.22.3 으로 업그레이드 되었습니다.

 

[참고] kubeadm upgrade 명령은 static pod 매니페스트를 업그레이드 한다. 기존 매니페스트와 비교해보면 아래와 같이 버전 정보가 변경된 것을 알 수 있다. 
root@k8s-m0:~/manifests_bk# diff kube-scheduler.yaml /etc/kubernetes/manifests/kube-scheduler.yaml
20c20
< image: k8s.gcr.io/kube-scheduler:v1.21.7
---
> image: k8s.gcr.io/kube-scheduler:v1.22.3

 

이제 나머지 컨트롤 플레인도 업그레이드를 진행하겠습니다.

 

Additional Control Plane 업그레이드

추가 컨트롤 플레인의 업그레이드 절차는 아래와 같습니다.

1. kubeadm 바이너리 업그레이드

2. kubeadm upgrade node 로 업그레이드 (이 부분이 kubeadm upgrade apply 에서  kubeadm upgrade node 로 다름)

3. 노드 drain

4. kubectl, kubelet 바이너리 업그레이드

5. kubelet daemon-reload, restart

6. 노드 uncordon

 

수행 결과를 살펴 보겠습니다.

# 1. kubeadm 바이너리 업그레이드
root@k8s-m1:~# apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm=1.22.3-00 && apt-mark hold kubeadm
Canceled hold on kubeadm.
Hit:1 https://download.docker.com/linux/ubuntu focal InRelease
Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease
Get:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9383 B]
Fetched 345 kB in 23s (14.8 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  kubeadm
1 upgraded, 0 newly installed, 0 to remove and 18 not upgraded.
Need to get 8712 kB of archives.
After this operation, 971 kB of additional disk space will be used.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.22.3-00 [8712 kB]
Fetched 8712 kB in 24s (366 kB/s)
(Reading database ... 64135 files and directories currently installed.)
Preparing to unpack .../kubeadm_1.22.3-00_amd64.deb ...
Unpacking kubeadm (1.22.3-00) over (1.21.7-00) ...
Setting up kubeadm (1.22.3-00) ...
kubeadm set on hold.


# 2. kubeadm upgrade node 로 업그레이드 (이 부분이 kubeadm upgrade apply 에서  kubeadm upgrade node 로 다름)
(admin-k8s:default) root@k8s-m1:~# kubeadm upgrade node
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[upgrade] Upgrading your Static Pod-hosted control plane instance to version "v1.22.3"...
Static pod: kube-apiserver-k8s-m1 hash: 121c5f0b29db118a71b4afb666436a28
Static pod: kube-controller-manager-k8s-m1 hash: 6c9d416769ffcfda2b9b457c2278465b
Static pod: kube-scheduler-k8s-m1 hash: 5b40ca2fc4b25745dd3e0a5fbb7cd7f1
[upgrade/etcd] Upgrading to TLS for etcd
Static pod: etcd-k8s-m1 hash: 52933cc98a0e0db8d2740d5d66b76a50
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Renewing etcd-server certificate
[upgrade/staticpods] Renewing etcd-peer certificate
[upgrade/staticpods] Renewing etcd-healthcheck-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2022-02-05-12-09-38/etcd.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: etcd-k8s-m1 hash: 52933cc98a0e0db8d2740d5d66b76a50
Static pod: etcd-k8s-m1 hash: 52933cc98a0e0db8d2740d5d66b76a50
<생략>
Static pod: etcd-k8s-m1 hash: 52933cc98a0e0db8d2740d5d66b76a50
Static pod: etcd-k8s-m1 hash: 52933cc98a0e0db8d2740d5d66b76a50
Static pod: etcd-k8s-m1 hash: 572bcedba634f1d0a840c50caf2c2c89
[apiclient] Found 3 Pods for label selector component=etcd
[upgrade/staticpods] Component "etcd" upgraded successfully!
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests160626215"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2022-02-05-12-09-38/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-apiserver-k8s-m1 hash: 121c5f0b29db118a71b4afb666436a28
Static pod: kube-apiserver-k8s-m1 hash: 121c5f0b29db118a71b4afb666436a28
<생략>
Static pod: kube-apiserver-k8s-m1 hash: 121c5f0b29db118a71b4afb666436a28
Static pod: kube-apiserver-k8s-m1 hash: 121c5f0b29db118a71b4afb666436a28
Static pod: kube-apiserver-k8s-m1 hash: ff13ada2027cc50e6a9db21ef2dd0678
[apiclient] Found 3 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2022-02-05-12-09-38/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-controller-manager-k8s-m1 hash: 6c9d416769ffcfda2b9b457c2278465b
Static pod: kube-controller-manager-k8s-m1 hash: 6c9d416769ffcfda2b9b457c2278465b
<생략>
Static pod: kube-controller-manager-k8s-m1 hash: 6c9d416769ffcfda2b9b457c2278465b
Static pod: kube-controller-manager-k8s-m1 hash: 6c9d416769ffcfda2b9b457c2278465b
Static pod: kube-controller-manager-k8s-m1 hash: 9894d6b6867df03a62434e48ca7456c3
[apiclient] Found 3 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2022-02-05-12-09-38/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-scheduler-k8s-m1 hash: 5b40ca2fc4b25745dd3e0a5fbb7cd7f1
Static pod: kube-scheduler-k8s-m1 hash: 5b40ca2fc4b25745dd3e0a5fbb7cd7f1
<생략>
Static pod: kube-scheduler-k8s-m1 hash: 5b40ca2fc4b25745dd3e0a5fbb7cd7f1
Static pod: kube-scheduler-k8s-m1 hash: 5b40ca2fc4b25745dd3e0a5fbb7cd7f1
Static pod: kube-scheduler-k8s-m1 hash: 6e1d9a4253856a2c13140b36a8811869
[apiclient] Found 3 Pods for label selector component=kube-scheduler
[apiclient] Error getting Pods with label selector "component=kube-scheduler" [Get "https://192.168.100.100:16443/api/v1/namespaces/kube-system/pods?labelSelector=component%3Dkube-scheduler": context deadline exceeded (Client.Timeout exceeded while awaiting headers)]
[apiclient] Error getting Pods with label selector "component=kube-scheduler" [Get "https://192.168.100.100:16443/api/v1/namespaces/kube-system/pods?labelSelector=component%3Dkube-scheduler": net/http: request canceled (Client.Timeout exceeded while awaiting headers)]
[apiclient] Error getting Pods with label selector "component=kube-scheduler" [Get "https://192.168.100.100:16443/api/v1/namespaces/kube-system/pods?labelSelector=component%3Dkube-scheduler": net/http: request canceled (Client.Timeout exceeded while awaiting headers)]
[apiclient] Error getting Pods with label selector "component=kube-scheduler" [Get "https://192.168.100.100:16443/api/v1/namespaces/kube-system/pods?labelSelector=component%3Dkube-scheduler": net/http: request canceled (Client.Timeout exceeded while awaiting headers)]
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upgrade] The control plane instance for this node was successfully updated!
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.


# 3. 노드 drain
root@k8s-m1:~# kubectl drain k8s-m1 --ignore-daemonsets
node/k8s-m1 cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-flannel-ds-p2r67, kube-system/kube-proxy-xvrqf
evicting pod kube-system/coredns-78fcd69978-njbnw
pod/coredns-78fcd69978-njbnw evicted
node/k8s-m1 evicted


# 4. kubectl, kubelet 바이너리 업그레이드
root@k8s-m1:~# apt-mark unhold kubelet kubectl && apt-get update && apt-get install -y kubelet=1.22.3-00 kubectl=1.22.3-00 && apt-mark hold kubelet kubectl
Canceled hold on kubelet.
Canceled hold on kubectl.
Hit:1 https://download.docker.com/linux/ubuntu focal InRelease
Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Hit:4 http://archive.ubuntu.com/ubuntu focal InRelease
Get:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Hit:3 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Fetched 336 kB in 23s (14.9 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 17 not upgraded.
Need to get 28.2 MB of archives.
After this operation, 3028 kB of additional disk space will be used.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.22.3-00 [9039 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.22.3-00 [19.1 MB]
Fetched 28.2 MB in 29s (970 kB/s)
(Reading database ... 64135 files and directories currently installed.)
Preparing to unpack .../kubectl_1.22.3-00_amd64.deb ...
Unpacking kubectl (1.22.3-00) over (1.21.7-00) ...
Preparing to unpack .../kubelet_1.22.3-00_amd64.deb ...
Unpacking kubelet (1.22.3-00) over (1.21.7-00) ...
Setting up kubectl (1.22.3-00) ...
Setting up kubelet (1.22.3-00) ...
kubelet set on hold.
kubectl set on hold.


# 5. kubelet daemon-reload, restart
root@k8s-m1:~# systemctl daemon-reload && systemctl restart kubelet


# 6. 노드 uncordon
root@k8s-m1:~# kubectl uncordon k8s-m1
node/k8s-m1 uncordoned
root@k8s-m1:~# kubectl get no
NAME     STATUS   ROLES                  AGE     VERSION
k8s-m0   Ready    control-plane,master   4h58m   v1.22.3
k8s-m1   Ready    control-plane,master   4h34m   v1.22.3
k8s-m2   Ready    control-plane,master   4h10m   v1.22.3
k8s-w1   Ready    <none>                 3h55m   v1.21.7

 

 

Worker 노드 업그레이드

워커 노드 업그레이드 절차는 추가 컨트롤 플레인의 업그레이드 절차와 동일합니다.

수행 결과를 살펴 보겠습니다.

 

root@k8s-w1:~# apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm=1.22.3-00 && apt-mark hold kubeadm
Canceled hold on kubeadm.
Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease
Hit:3 https://download.docker.com/linux/ubuntu focal InRelease
Get:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9383 B]
Fetched 345 kB in 23s (15.2 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  kubeadm
1 upgraded, 0 newly installed, 0 to remove and 18 not upgraded.
Need to get 8712 kB of archives.
After this operation, 971 kB of additional disk space will be used.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.22.3-00 [8712 kB]
Fetched 8712 kB in 22s (390 kB/s)
(Reading database ... 63715 files and directories currently installed.)
Preparing to unpack .../kubeadm_1.22.3-00_amd64.deb ...
Unpacking kubeadm (1.22.3-00) over (1.21.7-00) ...
Setting up kubeadm (1.22.3-00) ...
kubeadm set on hold.
root@k8s-w1:~# kubeadm upgrade node
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks
[preflight] Skipping prepull. Not a control plane node.
[upgrade] Skipping phase. Not a control plane node.
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.

kubeadm upgrade node 작업은 컨트롤 플레인의 static pod의 image pulling 이 없기 때문에 빨리 끝납니다.

아래의 노드 drain 은 컨트롤 플레인에서 수행합니다.

# Control Plane 에서 수행
root@k8s-m0:~# kubectl drain k8s-w1 --ignore-daemonsets
node/k8s-w1 cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-flannel-ds-4jqg8, kube-system/kube-proxy-rhzmv
evicting pod kube-system/coredns-78fcd69978-nwf2f
evicting pod kube-system/coredns-78fcd69978-j6p7g
pod/coredns-78fcd69978-nwf2f evicted
pod/coredns-78fcd69978-j6p7g evicted
node/k8s-w1 evicted
root@k8s-m0:~# kubectl get no
NAME     STATUS                     ROLES                  AGE     VERSION
k8s-m0   Ready                      control-plane,master   5h2m    v1.22.3
k8s-m1   Ready                      control-plane,master   4h38m   v1.22.3
k8s-m2   Ready                      control-plane,master   4h14m   v1.22.3
k8s-w1   Ready,SchedulingDisabled   <none>                 3h59m   v1.21.7

이어서 워커노드에서 나머지 업그레이드 작업을 수행합니다.

root@k8s-w1:~# apt-mark unhold kubelet kubectl && apt-get update && apt-get install -y kubelet=1.22.3-00 kubectl=1.22.3-00 && apt-mark hold kubelet kubectl

Canceled hold on kubelet.
Canceled hold on kubectl.
Hit:1 https://download.docker.com/linux/ubuntu focal InRelease
Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease
Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Hit:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Fetched 336 kB in 22s (15.0 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 17 not upgraded.
Need to get 28.2 MB of archives.
After this operation, 3028 kB of additional disk space will be used.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.22.3-00 [9039 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.22.3-00 [19.1 MB]
Fetched 28.2 MB in 25s (1105 kB/s)
(Reading database ... 63715 files and directories currently installed.)
Preparing to unpack .../kubectl_1.22.3-00_amd64.deb ...
Unpacking kubectl (1.22.3-00) over (1.21.7-00) ...
Preparing to unpack .../kubelet_1.22.3-00_amd64.deb ...
Unpacking kubelet (1.22.3-00) over (1.21.7-00) ...
Setting up kubectl (1.22.3-00) ...
Setting up kubelet (1.22.3-00) ...
kubelet set on hold.
kubectl set on hold.
root@k8s-w1:~# systemctl daemon-reload && systemctl restart kubelet
root@k8s-w1:~#

마지막으로 노드 uncordon 작업을 컨트롤 플레인에서 수행합니다.

# Control Plane 에서 수행
root@k8s-m0:~# kubectl uncordon k8s-w1
node/k8s-w1 uncordoned
root@k8s-m0:~# kubectl get no
NAME     STATUS   ROLES                  AGE     VERSION
k8s-m0   Ready    control-plane,master   5h7m    v1.22.3
k8s-m1   Ready    control-plane,master   4h42m   v1.22.3
k8s-m2   Ready    control-plane,master   4h18m   v1.22.3
k8s-w1   Ready    <none>                 4h4m    v1.22.3

이로써 Kubernetes 클러스터의 전체 노드 업그레이드가 완료 되었습니다.

 

Kubernetes 업그레이드 확인

업그레이드를 마치고 최종 상태를 확인하였습니다.

root@k8s-m0:~# kubectl get no
NAME     STATUS   ROLES                  AGE     VERSION
k8s-m0   Ready    control-plane,master   5h8m    v1.22.3
k8s-m1   Ready    control-plane,master   4h44m   v1.22.3
k8s-m2   Ready    control-plane,master   4h20m   v1.22.3
k8s-w1   Ready    <none>                 4h5m    v1.22.3
root@k8s-m0:~# kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:41:28Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:35:25Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}
root@k8s-m0:~# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:40:11Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}
root@k8s-m0:~# kubelet --version
Kubernetes v1.22.3

 

참고

https://kubernetes.io/ko/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/

https://wnw1005.tistory.com/363