Kubernetes가 컨테이너 런타임으로써의 docker 지원 중단을 발표 한지 1년여가 지났고 1.20 버전부터 docker를 런타임으로 사용 할 수 없다는 경고가 표시되고 있습니다

Using dockershim is deprecated, please consider using a full-fledged CRI implementation

작년 지원 중단을 발표할 때 1.20 부터 deprecation되며 2021 하반기에 출시할 1.22부터 deprecated 된다고 된다고 발표 되었습니다

해당 내용에 대해 잘 모르시는 분은 아래 블로그를 참고하시면 됩니다

당황하지 마세요. 쿠버네티스와 도커 

https://kubernetes.io/ko/blog/2020/12/02/dont-panic-kubernetes-and-docker/

그리고 드디어 2021.08.04 V 1.22가 릴리즈 되었습니다

1.22 릴리즈노트를 확인 한 결과 docker 가 아직 deprecated되지 않는 것처럼 보입니다 아직 시간을 더 준걸까요?

릴리즈노트상의 별다른 언급이 없어 혹시나? 하고 설치 해봤는데 아직 까지는 dockershim 사용이 가능합니다

분명 작년에 발표 할때는 1.22부터 deprecated 된다고 봤던 기억이 있어 관련 문서를 찾아보니 완전히 deprecated(kubelet에서 제거되는) 버전은 1.24로 예정되어 있습니다

Step 2: Release kubelet without dockershim

Target releases: 1.24 (assuming 3 release a year or after April 2022)

출처 : https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2221-remove-dockershim

준비성이 철저한 사용자라면 이미 containerd나 cri-o로 전환을 했을 테지만 저처럼 대부분의 사용자는 아직 Docker를 사용하고 있을 겁니다

사실 최근의 docker-engine은 내부적으로 containerd로 돌고 있기 때문에 containerd로의 전환은 그렇게 어렵지 않습니다

해서 쿠버네티스 클러스터 컨테이너 런타임을 docker → containerd로 전화하는 방법을 소개하고자 합니다

아래는 시험 환경에 사용한 클러스터 환경 입니다

# kubectl get node -o wide
NAME         STATUS   ROLES    AGE    VERSION    INTERNAL-IP     EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION                CONTAINER-RUNTIME
yb-master1   Ready    master   5d6h   v1.19.7    172.20.200.11   <none>        CentOS Linux 8   4.18.0-240.1.1.el8_3.x86_64   docker://20.10.8
yb-master2   Ready    master   5d6h   v1.19.7    172.20.200.12   <none>        CentOS Linux 8   4.18.0-240.1.1.el8_3.x86_64   docker://20.10.8
yb-master3   Ready    master   5d6h   v1.19.7    172.20.200.13   <none>        CentOS Linux 8   4.18.0-240.1.1.el8_3.x86_64   docker://20.10.8
yb-worker1   Ready    <none>   5d6h   v1.19.7    172.20.200.14   <none>        CentOS Linux 8   4.18.0-240.1.1.el8_3.x86_64   docker://20.10.8
yb-worker2   Ready    <none>   5d6h   v1.19.7    172.20.200.15   <none>        CentOS Linux 8   4.18.0-240.1.1.el8_3.x86_64   docker://20.10.8
yb-worker3   Ready    <none>   5d6h   v1.19.7    172.20.200.16   <none>        CentOS Linux 8   4.18.0-240.1.1.el8_3.x86_64   docker://20.10.8
yb-worker4   Ready    <none>   5d6h   v1.19.7    172.20.200.17   <none>        CentOS Linux 8   4.18.0-240.1.1.el8_3.x86_64   docker://20.10.8

# rpm -qa |grep docker-ce
docker-ce-rootless-extras-20.10.8-3.el8.x86_64
docker-ce-cli-20.10.8-3.el8.x86_64
docker-ce-20.10.8-3.el8.x86_64

OS는 CentOS 8버전이며

docker-ce 20.10 버전을 사용하고 있습니다

# rpm -qa |grep containerd

containerd.io-1.4.9-3.1.el8.x86_64

위와같이 containerd도 사전에 설치가 되어 있습니다

containerd는 별도로 설치 하지 않았습니다 위에 설명드렸죠? 최근 도커엔진은 내부적으로 containerd로 동작하며 docker-ce설치시 아래와 같이 자동으로 설치 되어 집니다

# yum install docker-ce 
Last metadata expiration check: 0:08:57 ago on Wed 11 Aug 2021 08:00:08 PM KST.
Dependencies resolved.
==========================================================================================================================================================
 Package                                 Architecture         Version                                                Repository                      Size
==========================================================================================================================================================
Installing:
 docker-ce                               x86_64               3:20.10.8-3.el8                                        docker-ce-stable                22 M
Installing dependencies:
 container-selinux                       noarch               2:2.162.0-1.module_el8.4.0+830+8027e1c4                appstream                       52 k
 containerd.io                           x86_64               1.4.9-3.1.el8                                          docker-ce-stable                30 M
 docker-ce-cli                           x86_64               1:20.10.8-3.el8                                        docker-ce-stable                29 M
 docker-ce-rootless-extras               x86_64               20.10.8-3.el8                                          docker-ce-stable               4.6 M
 docker-scan-plugin                      x86_64               0.8.0-3.el8                                            docker-ce-stable               4.2 M
 fuse-overlayfs                          x86_64               1.4.0-3.module_el8.4.0+830+8027e1c4                    appstream                       72 k
 fuse3                                   x86_64               3.2.1-12.el8                                           baseos                          50 k
 fuse3-libs                              x86_64               3.2.1-12.el8                                           baseos                          94 k
 libcgroup                               x86_64               0.41-19.el8                                            baseos                          70 k
 libslirp                                x86_64               4.3.1-1.module_el8.4.0+575+63b40ad7                    appstream                       69 k
 slirp4netns                             x86_64               1.1.8-1.module_el8.4.0+641+6116a774                    appstream                       51 k

Transaction Summary
==========================================================================================================================================================
Install  12 Packages

이제 실제로 docker 에서 containerd로 런타임을 전환해 봅니다

워커노드 먼저 수행 하겠습니다

먼저 워커노드를 drain시킵니다

테스트 환경에는 daemonset이 있으므로 daemonset은 무시하는 옵션을 주었습니다(데몬셋은 비워지지 않습니다)

또한 여러가지 테스트에 사용하는 환경이고 local storage도 사용중이라 local data를 삭제하는 옵션도 같이 주고 수행했습니다

# kubectl drain yb-worker3 --ignore-daemonsets --delete-local-data
node/yb-worker3 cordoned
WARNING: ignoring DaemonSet-managed Pods: default/csi-cephfsplugin-r55zm, default/csi-rbdplugin-wz59b, default/my-ds-ljz7t, kube-system/calico-node-pgj5t, kube-system/kube-proxy-v6b5w, metallb-system/speaker-tk566, monitoring/node-exporter-gl49w
evicting pod nginx-example/nginx-deployment-5cb8fd57b4-xxkcc
evicting pod ingress-nginx/ingress-nginx-controller-84896f5f66-9tqfx
evicting pod default/csi-rbdplugin-provisioner-6c7745cb5c-l4wmx
evicting pod default/exam1-99f7d48fb-zjhkv
evicting pod default/jenkins-0
evicting pod kube-system/metrics-server-64d9658cb-jsvpw
evicting pod kube-system/kube-state-metrics-f7f94b544-sk4sz
evicting pod monitoring/grafana-676646798f-qvbz6
I0811 20:24:58.458808 3774601 request.go:645] Throttling request took 1.080540435s, request: GET:https://172.20.200.10:6443/api/v1/namespaces/kube-system/pods/metrics-server-64d9658cb-jsvpw
pod/csi-rbdplugin-provisioner-6c7745cb5c-l4wmx evicted
pod/ingress-nginx-controller-84896f5f66-9tqfx evicted
pod/grafana-676646798f-qvbz6 evicted
pod/jenkins-0 evicted
pod/exam1-99f7d48fb-zjhkv evicted
I0811 20:25:11.658644 3774601 request.go:645] Throttling request took 1.031166879s, request: GET:https://172.20.200.10:6443/api/v1/namespaces/nginx-example/pods/nginx-deployment-5cb8fd57b4-xxkcc
pod/nginx-deployment-5cb8fd57b4-xxkcc evicted
pod/kube-state-metrics-f7f94b544-sk4sz evicted
pod/metrics-server-64d9658cb-jsvpw evicted
node/yb-worker3 evicted

노드의 상태를 확인 합니다

# kubectl get node 
NAME         STATUS                     ROLES    AGE    VERSION
yb-master1   Ready                      master   5d7h   v1.19.7
yb-master2   Ready                      master   5d7h   v1.19.7
yb-master3   Ready                      master   5d7h   v1.19.7
yb-worker1   Ready                      <none>   5d7h   v1.19.7
yb-worker2   Ready                      <none>   5d7h   v1.19.7
yb-worker3   Ready,SchedulingDisabled   <none>   5d7h   v1.19.7
yb-worker4   Ready                      <none>   5d7h   v1.19.13
# kubectl get pod -o wide |grep worker3
csi-cephfsplugin-r55zm                        3/3     Running             0          5d5h   172.20.200.16    yb-worker3   <none>           <none>
csi-rbdplugin-wz59b                           3/3     Running             0          5d5h   172.20.200.16    yb-worker3   <none>           <none>
my-ds-ljz7t                                   1/1     Running             0          5d6h   10.233.7.3       yb-worker3   <none>           <none>

SchedulingDisabled 되었으며 daemonset외에 모든 pod가 비어졌음을 확인 할 수 있습니다

이제 kubelet과 docker 서비스를 중지 합니다 (Warning 메시지는 무시해도 됩니다)

# systemctl stop kubelet
# systemctl stop docker
Warning: Stopping docker.service, but it can still be activated by:
  docker.socket

더이상 사용되지 않을 docker를 삭제 해줍니다

# yum remove docker-ce docker-ce-cli -y
Dependencies resolved.
==============================================================================================================================================================================================================================
 Package                                                    Architecture                            Version                                                                 Repository                                   Size
==============================================================================================================================================================================================================================
Removing:
 docker-ce                                                  x86_64                                  3:20.10.8-3.el8                                                         @docker-ce                                   95 M
 docker-ce-cli                                              x86_64                                  1:20.10.8-3.el8                                                         @docker-ce                                  139 M
Removing unused dependencies:
 docker-ce-rootless-extras                                  x86_64                                  20.10.8-3.el8                                                           @docker-ce                                   16 M
 docker-scan-plugin                                         x86_64                                  0.8.0-3.el8                                                             @docker-ce                                   13 M
 fuse-common                                                x86_64                                  3.2.1-12.el8                                                            @baseos                                     4.7 k
 fuse-overlayfs                                             x86_64                                  1.4.0-3.module_el8.4.0+830+8027e1c4                                     @appstream                                  145 k
 fuse3                                                      x86_64                                  3.2.1-12.el8                                                            @baseos                                      90 k
 fuse3-libs                                                 x86_64                                  3.2.1-12.el8                                                            @baseos                                     279 k
 libcgroup                                                  x86_64                                  0.41-19.el8                                                             @baseos                                     136 k
 libslirp                                                   x86_64                                  4.3.1-1.module_el8.4.0+575+63b40ad7                                     @appstream                                  129 k
 slirp4netns                                                x86_64                                  1.1.8-1.module_el8.4.0+641+6116a774                                     @appstream                                   98 k

Transaction Summary
==============================================================================================================================================================================================================================
Remove  11 Packages

Freed space: 265 M
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                      1/1 
  Running scriptlet: docker-ce-3:20.10.8-3.el8.x86_64                                                                                                                                                                     1/1 
  Running scriptlet: docker-ce-3:20.10.8-3.el8.x86_64                                                                                                                                                                    1/11 
  Erasing          : docker-ce-3:20.10.8-3.el8.x86_64                                                                                                                                                                    1/11 
  Running scriptlet: docker-ce-3:20.10.8-3.el8.x86_64                                                                                                                                                                    1/11 
  Running scriptlet: docker-ce-rootless-extras-20.10.8-3.el8.x86_64                                                                                                                                                      2/11 
  Erasing          : docker-ce-rootless-extras-20.10.8-3.el8.x86_64                                                                                                                                                      2/11 
  Running scriptlet: docker-ce-rootless-extras-20.10.8-3.el8.x86_64                                                                                                                                                      2/11 
  Erasing          : fuse-overlayfs-1.4.0-3.module_el8.4.0+830+8027e1c4.x86_64                                                                                                                                           3/11 
  Erasing          : slirp4netns-1.1.8-1.module_el8.4.0+641+6116a774.x86_64                                                                                                                                              4/11 
  Erasing          : fuse3-3.2.1-12.el8.x86_64                                                                                                                                                                           5/11 
  Erasing          : docker-ce-cli-1:20.10.8-3.el8.x86_64                                                                                                                                                                6/11 
  Running scriptlet: docker-scan-plugin-0.8.0-3.el8.x86_64                                                                                                                                                               7/11 
  Erasing          : docker-scan-plugin-0.8.0-3.el8.x86_64                                                                                                                                                               7/11 
  Running scriptlet: docker-scan-plugin-0.8.0-3.el8.x86_64                                                                                                                                                               7/11 
  Erasing          : fuse-common-3.2.1-12.el8.x86_64                                                                                                                                                                     8/11 
  Erasing          : libslirp-4.3.1-1.module_el8.4.0+575+63b40ad7.x86_64                                                                                                                                                 9/11 
  Erasing          : fuse3-libs-3.2.1-12.el8.x86_64                                                                                                                                                                     10/11 
  Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64                                                                                                                                                                     10/11 
  Erasing          : libcgroup-0.41-19.el8.x86_64                                                                                                                                                                       11/11 
  Running scriptlet: libcgroup-0.41-19.el8.x86_64                                                                                                                                                                       11/11 
  Verifying        : docker-ce-3:20.10.8-3.el8.x86_64                                                                                                                                                                    1/11 
  Verifying        : docker-ce-cli-1:20.10.8-3.el8.x86_64                                                                                                                                                                2/11 
  Verifying        : docker-ce-rootless-extras-20.10.8-3.el8.x86_64                                                                                                                                                      3/11 
  Verifying        : docker-scan-plugin-0.8.0-3.el8.x86_64                                                                                                                                                               4/11 
  Verifying        : fuse-common-3.2.1-12.el8.x86_64                                                                                                                                                                     5/11 
  Verifying        : fuse-overlayfs-1.4.0-3.module_el8.4.0+830+8027e1c4.x86_64                                                                                                                                           6/11 
  Verifying        : fuse3-3.2.1-12.el8.x86_64                                                                                                                                                                           7/11 
  Verifying        : fuse3-libs-3.2.1-12.el8.x86_64                                                                                                                                                                      8/11 
  Verifying        : libcgroup-0.41-19.el8.x86_64                                                                                                                                                                        9/11 
  Verifying        : libslirp-4.3.1-1.module_el8.4.0+575+63b40ad7.x86_64                                                                                                                                                10/11 
  Verifying        : slirp4netns-1.1.8-1.module_el8.4.0+641+6116a774.x86_64                                                                                                                                             11/11 

Removed:
  docker-ce-3:20.10.8-3.el8.x86_64          docker-ce-cli-1:20.10.8-3.el8.x86_64                               docker-ce-rootless-extras-20.10.8-3.el8.x86_64                  docker-scan-plugin-0.8.0-3.el8.x86_64         
  fuse-common-3.2.1-12.el8.x86_64           fuse-overlayfs-1.4.0-3.module_el8.4.0+830+8027e1c4.x86_64          fuse3-3.2.1-12.el8.x86_64                                       fuse3-libs-3.2.1-12.el8.x86_64                
  libcgroup-0.41-19.el8.x86_64              libslirp-4.3.1-1.module_el8.4.0+575+63b40ad7.x86_64                slirp4netns-1.1.8-1.module_el8.4.0+641+6116a774.x86_64         

Complete!

아래 명령어를 통해 containerd의 default values가 들어있는 파일을 생성 합니다

# containerd config default > /etc/containerd/config.toml

 

containerd를 재시작 합니다

# systemctl restart containerd

 

또한 containerd를 재부팅 이후 자동 구동 되도록 enable 해줍니다 기존에는 docker로 인해 자동으로 구동 되었고 disabled상태 입니다

# systemctl is-enabled containerd 
disabled 

# systemctl enable containerd
Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /usr/lib/systemd/system/containerd.service.

다음으로는 kubelet에서 런타임을 containerd로 인식하도록 설정을 수정 합니다

수정할 파일의 경로는 /var/lib/kubelet/kubeadm-flags.env 입니다

args에 아래와 같은 값을 추가하여 주시면 됩니다

--container-runtime=remote

--container-runtime-endpoint=unix:///run/containerd/containerd.sock

실제 적용한 파일 내용입니다 이는 환경마다 다를 수 있습니다

# cat /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS="--network-plugin=cni --pod-infra-container-image=172.20.7.100:5000/pause:3.2 --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock"

중지되었던 kubelet을 실행 합니다

# systemctl start kubelet

 

실행된 kubelet이 클러스터와 통신 하길 기다린 후 node의 상태를 확인 합니다

# kubectl get node -o wide
NAME         STATUS                     ROLES    AGE    VERSION    INTERNAL-IP     EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION                CONTAINER-RUNTIME
yb-master1   Ready                      master   5d8h   v1.19.7    172.20.200.11   <none>        CentOS Linux 8   4.18.0-240.1.1.el8_3.x86_64   docker://20.10.8
yb-master2   Ready                      master   5d8h   v1.19.7    172.20.200.12   <none>        CentOS Linux 8   4.18.0-240.1.1.el8_3.x86_64   docker://20.10.8
yb-master3   Ready                      master   5d8h   v1.19.7    172.20.200.13   <none>        CentOS Linux 8   4.18.0-240.1.1.el8_3.x86_64   docker://20.10.8
yb-worker1   Ready                      <none>   5d7h   v1.19.7    172.20.200.14   <none>        CentOS Linux 8   4.18.0-240.1.1.el8_3.x86_64   docker://20.10.8
yb-worker2   Ready                      <none>   5d7h   v1.19.7    172.20.200.15   <none>        CentOS Linux 8   4.18.0-240.1.1.el8_3.x86_64   docker://20.10.8
yb-worker3   Ready,SchedulingDisabled   <none>   5d7h   v1.19.7    172.20.200.16   <none>        CentOS Linux 8   4.18.0-240.1.1.el8_3.x86_64   containerd://1.4.9
yb-worker4   Ready                      <none>   5d7h   v1.19.7   172.20.200.17   <none>        CentOS Linux 8   4.18.0-240.1.1.el8_3.x86_64   docker://20.10.8

해당 워커의 런타임이 containerd://1.4.9로 잘 변경 되었습니다

이제 해당노드의 SchedulingDisabled 플래그를 해제하고 pod를 배치 해 봅니다

# kubectl uncordon yb-worker3
node/yb-worker3 uncordoned

kubectl get pod -A -o wide |grep worker3
default                csi-cephfsplugin-provisioner-99d8c67f-jbvvs   0/6     ImagePullBackOff        0          5m23s   10.233.7.25      yb-worker3   <none>           <none>
default                csi-cephfsplugin-r55zm                        0/3     ImagePullBackOff        0          5d5h    172.20.200.16    yb-worker3   <none>           <none>
default                csi-rbdplugin-provisioner-6c7745cb5c-nmvd5    0/7     ImagePullBackOff        0          5m23s   10.233.7.24      yb-worker3   <none>           <none>
default                csi-rbdplugin-wz59b                           0/3     ImagePullBackOff        0          5d5h    172.20.200.16    yb-worker3   <none>           <none>
default                my-ds-ljz7t                                   0/1     ImagePullBackOff        0          5d6h    10.233.7.20      yb-worker3   <none>           <none>
kube-system            calico-node-pgj5t                             0/1     Init:ImagePullBackOff   0          5d8h    172.20.200.16    yb-worker3   <none>           <none>
kube-system            kube-proxy-v6b5w                              0/1     ImagePullBackOff        0          5d8h    172.20.200.16    yb-worker3   <none>           <none>
metallb-system         speaker-tk566                                 0/1     ImagePullBackOff        0          5d6h    172.20.200.16    yb-worker3   <none>           <none>
monitoring             node-exporter-gl49w                           0/1     ImagePullBackOff        0          5d5h    10.233.7.21      yb-worker3   <none>           <none>

저의 경우 imagepull 에러가 발생햇는데요

사내에서사용하는 registry가 insecure형태이기 때문입니다

기존 도커는 daemon.json에 insecure를 등록하였지만 containerd는 아래와 같이 inseucre registry를 설정 합니다

/etc/containerd/config.toml

[plugins."io.containerd.grpc.v1.cri".registry]

[plugins.”io.containerd.grpc.v1.cri”.registry.mirrors]

[plugins.”io.containerd.grpc.v1.cri”.registry.mirrors.”docker.io”]

endpoint = [“https://registry-1.docker.io”]

[plugins.”io.containerd.grpc.v1.cri”.registry.mirrors.”172.20.7.100:5000″]

endpoint = [“http://172.20.7.100:5000”]

[plugins.”io.containerd.grpc.v1.cri”.registry.configs]

[plugins.”io.containerd.grpc.v1.cri”.registry.configs.”172.20.7.100:5000″.tls]

insecure_skip_verify = true

[plugins.”io.containerd.grpc.v1.cri”.registry.mirrors.”172.20.7.100:5000″] 부터 추가한 줄이며 레지스트리 주소를 적어 줍니다

수정된 insecure설정을 적용하기 위해 containerd를 재시작 합니다

# systemctl restart containerd

 

일정 시간이 경과되면 다시 이미지를 pull하기 시작합니다

# kubectl get pod -A -o wide |grep worker3
default                csi-cephfsplugin-provisioner-99d8c67f-jbvvs   6/6     Running             4          17m     10.233.7.25      yb-worker3   <none>           <none>
default                csi-cephfsplugin-r55zm                        3/3     Running             0          5d5h    172.20.200.16    yb-worker3   <none>           <none>
default                csi-rbdplugin-provisioner-6c7745cb5c-nmvd5    7/7     Running             5          17m     10.233.7.24      yb-worker3   <none>           <none>
default                csi-rbdplugin-wz59b                           3/3     Running             0          5d5h    172.20.200.16    yb-worker3   <none>           <none>
default                my-ds-ljz7t                                   1/1     Running             0          5d7h    10.233.7.20      yb-worker3   <none>           <none>
kube-system            calico-node-pgj5t                             1/1     Running             0          5d8h    172.20.200.16    yb-worker3   <none>           <none>
kube-system            kube-proxy-v6b5w                              1/1     Running             0          5d8h    172.20.200.16    yb-worker3   <none>           <none>
metallb-system         speaker-tk566                                 1/1     Running             0          5d6h    172.20.200.16    yb-worker3   <none>           <none>
monitoring             node-exporter-gl49w                           1/1     Running             0          5d5h    10.233.7.21      yb-worker3   <none>           <none>
# kubectl get node -o wide |grep worker3
yb-worker3   Ready      <none>   5d8h   v1.19.7    172.20.200.16   <none>        CentOS Linux 8   4.18.0-240.1.1.el8_3.x86_64   containerd://1.4.9

최종적으로 runtime이 변경되었고 모든 pod가 정상 작동 되고 있습니다

이러한 방식으로 모든노드에 적용을 해주시면 됩니다

최종적으로 완료된 상태 입니다

# kubectl get node -o wide
NAME         STATUS   ROLES    AGE     VERSION    INTERNAL-IP     EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION                CONTAINER-RUNTIME
yb-master1   Ready    master   5d21h   v1.19.7    172.20.200.11   <none>        CentOS Linux 8   4.18.0-240.1.1.el8_3.x86_64   containerd://1.4.9
yb-master2   Ready    master   5d21h   v1.19.7    172.20.200.12   <none>        CentOS Linux 8   4.18.0-240.1.1.el8_3.x86_64   containerd://1.4.9
yb-master3   Ready    master   5d21h   v1.19.7    172.20.200.13   <none>        CentOS Linux 8   4.18.0-240.1.1.el8_3.x86_64   containerd://1.4.9
yb-worker1   Ready    <none>   5d21h   v1.19.7    172.20.200.14   <none>        CentOS Linux 8   4.18.0-240.1.1.el8_3.x86_64   containerd://1.4.9
yb-worker2   Ready    <none>   5d21h   v1.19.7    172.20.200.15   <none>        CentOS Linux 8   4.18.0-240.1.1.el8_3.x86_64   containerd://1.4.9
yb-worker3   Ready    <none>   5d21h   v1.19.7    172.20.200.16   <none>        CentOS Linux 8   4.18.0-240.1.1.el8_3.x86_64   containerd://1.4.9
yb-worker4   Ready    <none>   5d20h   v1.19.7   172.20.200.17   <none>        CentOS Linux 8   4.18.0-240.1.1.el8_3.x86_64   containerd://1.4.9

모든 노드의 container runtime이 docker 에서 containerd로 변경 되었습니다

추가로 kubernetes node정보에 있는 container runtime 정보도 변경 해줘야 합니다

아래 명령어로 현재 설정된 annotations을 삭제하고 추가 해 줍니다

# kubectl annotate node/{nodename} kubeadm.alpha.kubernetes.io/cri-socket-
node/{nodename} annotated
# kubectl annotate node/{nodename} kubeadm.alpha.kubernetes.io/cri-socket=unix:///run/containerd/containerd.sock
node/{nodename} annotated


원래값은 아래와 같이 dockershime 입니다

kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock

이제 더 이상 docker 명령어는 사용되지 않습니다

개발 환경에서는 여전히 docker를 이용하여 개발하고 build하겠지만 containerd에서는 ctr명령어를 이용하여 확인 합니다

몇가지 명령어를 소개 시켜 드리겠습니다

먼저 k8s에서 익숙한 namespace를 사용합니다

# ctr namespaces list
NAME    LABELS 
default        
k8s.io         
moby

위 명령어를 통해 ns를 확인하고 사용합니다

container list는 기존 docker ps와 동일 하다고 보시면 됩니다

ctr -n k8s.io container list
CONTAINER                                                           IMAGE                                                                                                                    RUNTIME                  
0361f2ccdc58e0d1b60584f2f3971791aa8b39940c2d2a594d904edf56713e84    k8s.gcr.io/pause:3.2                                                                                                     io.containerd.runc.v2    
0b99ef42759cc905a562b03a83ee4691c3b2bc98b29bbe7e4f156ce3746d5bd3    k8s.gcr.io/pause:3.2                                                                                                     io.containerd.runc.v2    
0e4cee03df6709dd4b62f1af86e6eb926620a0f0a7ca1c6e4c1e62eceac16bae    172.20.7.100:5000/calico/cni:v3.20.0                                                                                     io.containerd.runc.v2    
1fe0b9b7182101fcef78eb49546252725ce88b762da1aa5287ef192ffb7893c6    k8s.gcr.io/pause:3.2                                                                                                     io.containerd.runc.v2    
2527c130cd77a855ec7bd52d856a12632a3bd3786b01a6cb99465ebde0c78a9b    172.20.7.100:5000/sig-storage/csi-provisioner@sha256:3b465cbcadf7d437fc70c3b6aa2c93603a7eef0a3f5f1e861d91f303e4aabdee    io.containerd.runc.v2    
.....

docker images 명령어와 대입되는 는 아래와 같습니다

ctr -n k8s.io image list    
REF                                                                                                                             TYPE                                                      DIGEST                                                                  SIZE      PLATFORMS                                                      LABELS                          
172.20.7.100:5000/calico/cni:v3.20.0                                                                                            application/vnd.docker.distribution.manifest.v2+json      sha256:7c43c152cdf589ed789528c4055503d2f2b0ee943d7815c820fd8e99793f36b3 46.1 MiB  linux/amd64                                                    io.cri-containerd.image=managed 
172.20.7.100:5000/calico/cni@sha256:7c43c152cdf589ed789528c4055503d2f2b0ee943d7815c820fd8e99793f36b3                            application/vnd.docker.distribution.manifest.v2+json      sha256:7c43c152cdf589ed789528c4055503d2f2b0ee943d7815c820fd8e99793f36b3 46.1 MiB  linux/amd64                                                    io.cri-containerd.image=managed 
172.20.7.100:5000/calico/node:v3.20.0                                                                                           application/vnd.docker.distribution.manifest.v2+json      sha256:913955a36179a53d36a0df7f26319a5278991eea2514eab339cce15df24c220c 57.9 MiB  linux/amd64                                                    io.cri-containerd.image=managed 
.....

기존 dokcer 명령어와 입력이나 출력이 달라 당황 스러우시죠? ctr은 docker명령어 처럼 containerd에만 작동하는 명령어 입니다

사실 예전부터 쿠버네티스에서는 cri표준을 준수하는 컨테이너 런타임을 통합하여 관리하기 위해 crictl이란 명령어 툴을 제공하고 있습니다

crictl은 docker,ctr과 달리 cri를 준수하는 모든 컨테이너 런타임에 사용이 가능합니다

일반적인 패키지 매니저를 통해 kubeadm kubelet등을 설치했다면 자동으로 설치 되어 집니다

바이너리로 설치하셨다면 cri-tools을 설치 하셔야 작동 합니다

먼저 crictl이 containerd로 작동할 수 있도록 아래 설정파일을 만들어 줍니다

cat /etc/crictl.yaml 
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock

crictl 도움말을 확인 합니다

# crictl
NAME:
   crictl - client for CRI

USAGE:
   crictl [global options] command [command options] [arguments...]

VERSION:
   v1.13.0

COMMANDS:
     attach        Attach to a running container
     create        Create a new container
     exec          Run a command in a running container
     version       Display runtime version information
     images        List images
     inspect       Display the status of one or more containers
     inspecti      Return the status of one or more images
     inspectp      Display the status of one or more pods
     logs          Fetch the logs of a container
     port-forward  Forward local port to a pod
     ps            List containers
     pull          Pull an image from a registry
     runp          Run a new pod
     rm            Remove one or more containers
     rmi           Remove one or more images
     rmp           Remove one or more pods
     pods          List pods
     start         Start one or more created containers
     info          Display information of the container runtime
     stop          Stop one or more running containers
     stopp         Stop one or more running pods
     update        Update one or more running containers
     config        Get and set crictl options
     stats         List container(s) resource usage statistics
     completion    Output bash shell completion code
     help, h       Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --config value, -c value            Location of the client config file (default: "/etc/crictl.yaml") [$CRI_CONFIG_FILE]
   --debug, -D                         Enable debug mode
   --image-endpoint value, -i value    Endpoint of CRI image manager service [$IMAGE_SERVICE_ENDPOINT]
   --runtime-endpoint value, -r value  Endpoint of CRI container runtime service (default: "unix:///var/run/dockershim.sock") [$CONTAINER_RUNTIME_ENDPOINT]
   --timeout value, -t value           Timeout of connecting to the server (default: 10s)
   --help, -h                          show help
   --version, -v                       print the version

# crictl ps
CONTAINER ID        IMAGE               CREATED             STATE               NAME                        ATTEMPT             POD ID
f04f4408e17cb       35036a0cd23a8       22 minutes ago      Running             kube-scheduler              1                   6ae463313ab52
09115790bcb05       90f4ff69a0bf9       22 minutes ago      Running             kube-controller-manager     1                   2e3329a303dfb
9a721c09d0725       48d79e554db69       24 minutes ago      Running             dashboard-metrics-scraper   0                   0aa2dc0bfa7c7
849202a25a780       76ba70f4748f9       24 minutes ago      Running             calico-kube-controllers     0                   3dea2a33b574d
f823a99ae8549       5ef66b403f4f0       28 minutes ago      Running             calico-node                 0                   de53af8e19dd6
064ec7a833b5d       76696340d7993       28 minutes ago      Running             kube-apiserver              1                   207d0374f2c90
8fd8537f6b190       e9f480f8f070f       28 minutes ago      Running             speaker                     0                   c0ff09a3cd6ac
fe390399e5d68       0369cf4303ffd       28 minutes ago      Running             etcd                        0                   580583d722d98
946355db2d8d3       046ec6b49f0b9       28 minutes ago      Running             kube-proxy                  0                   e5bd6804fc05f

어때요 ctr과는 달리 굉장히 익숙하죠? docker와 모든것이 동일 합니다(bulid나 tag push등의 개발 명령어만 없습니다)

crictl로는 pod의 상태도 볼 수 있습니다

crictl pods
POD ID              CREATED             STATE               NAME                                        NAMESPACE              ATTEMPT
0aa2dc0bfa7c7       11 minutes ago      Ready               dashboard-metrics-scraper-88b485d79-wxv27   kubernetes-dashboard   0
3dea2a33b574d       11 minutes ago      Ready               calico-kube-controllers-784b5b48f4-45nnn    kube-system            0
580583d722d98       15 minutes ago      Ready               etcd-yb-master1                             kube-system            0
2e3329a303dfb       15 minutes ago      Ready               kube-controller-manager-yb-master1          kube-system            0
207d0374f2c90       15 minutes ago      Ready               kube-apiserver-yb-master1                   kube-system            0
e5bd6804fc05f       15 minutes ago      Ready               kube-proxy-8sf8f                            kube-system            0
6ae463313ab52       15 minutes ago      Ready               kube-scheduler-yb-master1                   kube-system            0
c0ff09a3cd6ac       15 minutes ago      Ready               speaker-6lmn8                               metallb-system         0
de53af8e19dd6       15 minutes ago      Ready               calico-node-6rd8v                           kube-system            0

이외에 images rm rmi exec run등 모두 도커(docker-cli)와 동일하게 동작 합니다

ctr보다 crictl을 사용하시면 조금 더 빠르게 적응 하실 수 있습니다

지금까지 쿠버네티스 컨테이너 런타임을 docker 에서 containerd로 변경하는 방법과 crictl 명령어 사용 방법을 안내 해 드렸습니다

생각보다는 간단하죠? dockershim이 완전히 사용 불가능해 지기전에 미리 준비하고 이전하여 새로운 기능에 익숙해 지는것을 추천 합니다

감사합니다

오픈소스컨설팅의 마스코트, 열린이입니다! :)

Leave a Reply

Your email address will not be published.