Kubernetes가 컨테이너 런타임으로써의 docker 지원 중단을 발표 한지 1년여가 지났고 1.20 버전부터 docker를 런타임으로 사용 할 수 없다는 경고가 표시되고 있습니다
Using dockershim is deprecated, please consider using a full-fledged CRI implementation작년 지원 중단을 발표할 때 1.20 부터 deprecation되며 2021 하반기에 출시할 1.22부터 deprecated 된다고 된다고 발표 되었습니다
해당 내용에 대해 잘 모르시는 분은 아래 블로그를 참고하시면 됩니다
https://kubernetes.io/ko/blog/2020/12/02/dont-panic-kubernetes-and-docker/
그리고 드디어 2021.08.04 V 1.22가 릴리즈 되었습니다
1.22 릴리즈노트를 확인 한 결과 docker 가 아직 deprecated되지 않는 것처럼 보입니다 아직 시간을 더 준걸까요?
릴리즈노트상의 별다른 언급이 없어 혹시나? 하고 설치 해봤는데 아직 까지는 dockershim 사용이 가능합니다
분명 작년에 발표 할때는 1.22부터 deprecated 된다고 봤던 기억이 있어 관련 문서를 찾아보니 완전히 deprecated(kubelet에서 제거되는) 버전은 1.24로 예정되어 있습니다
Step 2: Release kubelet without dockershim
Target releases: 1.24 (assuming 3 release a year or after April 2022)출처 : https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2221-remove-dockershim
준비성이 철저한 사용자라면 이미 containerd나 cri-o로 전환을 했을 테지만 저처럼 대부분의 사용자는 아직 Docker를 사용하고 있을 겁니다
사실 최근의 docker-engine은 내부적으로 containerd로 돌고 있기 때문에 containerd로의 전환은 그렇게 어렵지 않습니다
해서 쿠버네티스 클러스터 컨테이너 런타임을 docker → containerd로 전화하는 방법을 소개하고자 합니다
아래는 시험 환경에 사용한 클러스터 환경 입니다
# kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
yb-master1 Ready master 5d6h v1.19.7 172.20.200.11 <none> CentOS Linux 8 4.18.0-240.1.1.el8_3.x86_64 docker://20.10.8
yb-master2 Ready master 5d6h v1.19.7 172.20.200.12 <none> CentOS Linux 8 4.18.0-240.1.1.el8_3.x86_64 docker://20.10.8
yb-master3 Ready master 5d6h v1.19.7 172.20.200.13 <none> CentOS Linux 8 4.18.0-240.1.1.el8_3.x86_64 docker://20.10.8
yb-worker1 Ready <none> 5d6h v1.19.7 172.20.200.14 <none> CentOS Linux 8 4.18.0-240.1.1.el8_3.x86_64 docker://20.10.8
yb-worker2 Ready <none> 5d6h v1.19.7 172.20.200.15 <none> CentOS Linux 8 4.18.0-240.1.1.el8_3.x86_64 docker://20.10.8
yb-worker3 Ready <none> 5d6h v1.19.7 172.20.200.16 <none> CentOS Linux 8 4.18.0-240.1.1.el8_3.x86_64 docker://20.10.8
yb-worker4 Ready <none> 5d6h v1.19.7 172.20.200.17 <none> CentOS Linux 8 4.18.0-240.1.1.el8_3.x86_64 docker://20.10.8
# rpm -qa |grep docker-ce
docker-ce-rootless-extras-20.10.8-3.el8.x86_64
docker-ce-cli-20.10.8-3.el8.x86_64
docker-ce-20.10.8-3.el8.x86_64OS는 CentOS 8버전이며
docker-ce 20.10 버전을 사용하고 있습니다
# rpm -qa |grep containerd
containerd.io-1.4.9-3.1.el8.x86_64위와같이 containerd도 사전에 설치가 되어 있습니다
containerd는 별도로 설치 하지 않았습니다 위에 설명드렸죠? 최근 도커엔진은 내부적으로 containerd로 동작하며 docker-ce설치시 아래와 같이 자동으로 설치 되어 집니다
# yum install docker-ce
Last metadata expiration check: 0:08:57 ago on Wed 11 Aug 2021 08:00:08 PM KST.
Dependencies resolved.
==========================================================================================================================================================
Package Architecture Version Repository Size
==========================================================================================================================================================
Installing:
docker-ce x86_64 3:20.10.8-3.el8 docker-ce-stable 22 M
Installing dependencies:
container-selinux noarch 2:2.162.0-1.module_el8.4.0+830+8027e1c4 appstream 52 k
containerd.io x86_64 1.4.9-3.1.el8 docker-ce-stable 30 M
docker-ce-cli x86_64 1:20.10.8-3.el8 docker-ce-stable 29 M
docker-ce-rootless-extras x86_64 20.10.8-3.el8 docker-ce-stable 4.6 M
docker-scan-plugin x86_64 0.8.0-3.el8 docker-ce-stable 4.2 M
fuse-overlayfs x86_64 1.4.0-3.module_el8.4.0+830+8027e1c4 appstream 72 k
fuse3 x86_64 3.2.1-12.el8 baseos 50 k
fuse3-libs x86_64 3.2.1-12.el8 baseos 94 k
libcgroup x86_64 0.41-19.el8 baseos 70 k
libslirp x86_64 4.3.1-1.module_el8.4.0+575+63b40ad7 appstream 69 k
slirp4netns x86_64 1.1.8-1.module_el8.4.0+641+6116a774 appstream 51 k
Transaction Summary
==========================================================================================================================================================
Install 12 Packages이제 실제로 docker 에서 containerd로 런타임을 전환해 봅니다
워커노드 먼저 수행 하겠습니다
먼저 워커노드를 drain시킵니다
테스트 환경에는 daemonset이 있으므로 daemonset은 무시하는 옵션을 주었습니다(데몬셋은 비워지지 않습니다)
또한 여러가지 테스트에 사용하는 환경이고 local storage도 사용중이라 local data를 삭제하는 옵션도 같이 주고 수행했습니다
# kubectl drain yb-worker3 --ignore-daemonsets --delete-local-data
node/yb-worker3 cordoned
WARNING: ignoring DaemonSet-managed Pods: default/csi-cephfsplugin-r55zm, default/csi-rbdplugin-wz59b, default/my-ds-ljz7t, kube-system/calico-node-pgj5t, kube-system/kube-proxy-v6b5w, metallb-system/speaker-tk566, monitoring/node-exporter-gl49w
evicting pod nginx-example/nginx-deployment-5cb8fd57b4-xxkcc
evicting pod ingress-nginx/ingress-nginx-controller-84896f5f66-9tqfx
evicting pod default/csi-rbdplugin-provisioner-6c7745cb5c-l4wmx
evicting pod default/exam1-99f7d48fb-zjhkv
evicting pod default/jenkins-0
evicting pod kube-system/metrics-server-64d9658cb-jsvpw
evicting pod kube-system/kube-state-metrics-f7f94b544-sk4sz
evicting pod monitoring/grafana-676646798f-qvbz6
I0811 20:24:58.458808 3774601 request.go:645] Throttling request took 1.080540435s, request: GET:https://172.20.200.10:6443/api/v1/namespaces/kube-system/pods/metrics-server-64d9658cb-jsvpw
pod/csi-rbdplugin-provisioner-6c7745cb5c-l4wmx evicted
pod/ingress-nginx-controller-84896f5f66-9tqfx evicted
pod/grafana-676646798f-qvbz6 evicted
pod/jenkins-0 evicted
pod/exam1-99f7d48fb-zjhkv evicted
I0811 20:25:11.658644 3774601 request.go:645] Throttling request took 1.031166879s, request: GET:https://172.20.200.10:6443/api/v1/namespaces/nginx-example/pods/nginx-deployment-5cb8fd57b4-xxkcc
pod/nginx-deployment-5cb8fd57b4-xxkcc evicted
pod/kube-state-metrics-f7f94b544-sk4sz evicted
pod/metrics-server-64d9658cb-jsvpw evicted
node/yb-worker3 evicted노드의 상태를 확인 합니다
# kubectl get node
NAME STATUS ROLES AGE VERSION
yb-master1 Ready master 5d7h v1.19.7
yb-master2 Ready master 5d7h v1.19.7
yb-master3 Ready master 5d7h v1.19.7
yb-worker1 Ready <none> 5d7h v1.19.7
yb-worker2 Ready <none> 5d7h v1.19.7
yb-worker3 Ready,SchedulingDisabled <none> 5d7h v1.19.7
yb-worker4 Ready <none> 5d7h v1.19.13
# kubectl get pod -o wide |grep worker3
csi-cephfsplugin-r55zm 3/3 Running 0 5d5h 172.20.200.16 yb-worker3 <none> <none>
csi-rbdplugin-wz59b 3/3 Running 0 5d5h 172.20.200.16 yb-worker3 <none> <none>
my-ds-ljz7t 1/1 Running 0 5d6h 10.233.7.3 yb-worker3 <none> <none>SchedulingDisabled 되었으며 daemonset외에 모든 pod가 비어졌음을 확인 할 수 있습니다
이제 kubelet과 docker 서비스를 중지 합니다 (Warning 메시지는 무시해도 됩니다)
# systemctl stop kubelet
# systemctl stop docker
Warning: Stopping docker.service, but it can still be activated by:
docker.socket더이상 사용되지 않을 docker를 삭제 해줍니다
# yum remove docker-ce docker-ce-cli -y
Dependencies resolved.
==============================================================================================================================================================================================================================
Package Architecture Version Repository Size
==============================================================================================================================================================================================================================
Removing:
docker-ce x86_64 3:20.10.8-3.el8 @docker-ce 95 M
docker-ce-cli x86_64 1:20.10.8-3.el8 @docker-ce 139 M
Removing unused dependencies:
docker-ce-rootless-extras x86_64 20.10.8-3.el8 @docker-ce 16 M
docker-scan-plugin x86_64 0.8.0-3.el8 @docker-ce 13 M
fuse-common x86_64 3.2.1-12.el8 @baseos 4.7 k
fuse-overlayfs x86_64 1.4.0-3.module_el8.4.0+830+8027e1c4 @appstream 145 k
fuse3 x86_64 3.2.1-12.el8 @baseos 90 k
fuse3-libs x86_64 3.2.1-12.el8 @baseos 279 k
libcgroup x86_64 0.41-19.el8 @baseos 136 k
libslirp x86_64 4.3.1-1.module_el8.4.0+575+63b40ad7 @appstream 129 k
slirp4netns x86_64 1.1.8-1.module_el8.4.0+641+6116a774 @appstream 98 k
Transaction Summary
==============================================================================================================================================================================================================================
Remove 11 Packages
Freed space: 265 M
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Running scriptlet: docker-ce-3:20.10.8-3.el8.x86_64 1/1
Running scriptlet: docker-ce-3:20.10.8-3.el8.x86_64 1/11
Erasing : docker-ce-3:20.10.8-3.el8.x86_64 1/11
Running scriptlet: docker-ce-3:20.10.8-3.el8.x86_64 1/11
Running scriptlet: docker-ce-rootless-extras-20.10.8-3.el8.x86_64 2/11
Erasing : docker-ce-rootless-extras-20.10.8-3.el8.x86_64 2/11
Running scriptlet: docker-ce-rootless-extras-20.10.8-3.el8.x86_64 2/11
Erasing : fuse-overlayfs-1.4.0-3.module_el8.4.0+830+8027e1c4.x86_64 3/11
Erasing : slirp4netns-1.1.8-1.module_el8.4.0+641+6116a774.x86_64 4/11
Erasing : fuse3-3.2.1-12.el8.x86_64 5/11
Erasing : docker-ce-cli-1:20.10.8-3.el8.x86_64 6/11
Running scriptlet: docker-scan-plugin-0.8.0-3.el8.x86_64 7/11
Erasing : docker-scan-plugin-0.8.0-3.el8.x86_64 7/11
Running scriptlet: docker-scan-plugin-0.8.0-3.el8.x86_64 7/11
Erasing : fuse-common-3.2.1-12.el8.x86_64 8/11
Erasing : libslirp-4.3.1-1.module_el8.4.0+575+63b40ad7.x86_64 9/11
Erasing : fuse3-libs-3.2.1-12.el8.x86_64 10/11
Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 10/11
Erasing : libcgroup-0.41-19.el8.x86_64 11/11
Running scriptlet: libcgroup-0.41-19.el8.x86_64 11/11
Verifying : docker-ce-3:20.10.8-3.el8.x86_64 1/11
Verifying : docker-ce-cli-1:20.10.8-3.el8.x86_64 2/11
Verifying : docker-ce-rootless-extras-20.10.8-3.el8.x86_64 3/11
Verifying : docker-scan-plugin-0.8.0-3.el8.x86_64 4/11
Verifying : fuse-common-3.2.1-12.el8.x86_64 5/11
Verifying : fuse-overlayfs-1.4.0-3.module_el8.4.0+830+8027e1c4.x86_64 6/11
Verifying : fuse3-3.2.1-12.el8.x86_64 7/11
Verifying : fuse3-libs-3.2.1-12.el8.x86_64 8/11
Verifying : libcgroup-0.41-19.el8.x86_64 9/11
Verifying : libslirp-4.3.1-1.module_el8.4.0+575+63b40ad7.x86_64 10/11
Verifying : slirp4netns-1.1.8-1.module_el8.4.0+641+6116a774.x86_64 11/11
Removed:
docker-ce-3:20.10.8-3.el8.x86_64 docker-ce-cli-1:20.10.8-3.el8.x86_64 docker-ce-rootless-extras-20.10.8-3.el8.x86_64 docker-scan-plugin-0.8.0-3.el8.x86_64
fuse-common-3.2.1-12.el8.x86_64 fuse-overlayfs-1.4.0-3.module_el8.4.0+830+8027e1c4.x86_64 fuse3-3.2.1-12.el8.x86_64 fuse3-libs-3.2.1-12.el8.x86_64
libcgroup-0.41-19.el8.x86_64 libslirp-4.3.1-1.module_el8.4.0+575+63b40ad7.x86_64 slirp4netns-1.1.8-1.module_el8.4.0+641+6116a774.x86_64
Complete!아래 명령어를 통해 containerd의 default values가 들어있는 파일을 생성 합니다
# containerd config default > /etc/containerd/config.toml
containerd를 재시작 합니다
# systemctl restart containerd
또한 containerd를 재부팅 이후 자동 구동 되도록 enable 해줍니다 기존에는 docker로 인해 자동으로 구동 되었고 disabled상태 입니다
# systemctl is-enabled containerd
disabled
# systemctl enable containerd
Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /usr/lib/systemd/system/containerd.service.다음으로는 kubelet에서 런타임을 containerd로 인식하도록 설정을 수정 합니다
수정할 파일의 경로는 /var/lib/kubelet/kubeadm-flags.env 입니다
args에 아래와 같은 값을 추가하여 주시면 됩니다
--container-runtime=remote
--container-runtime-endpoint=unix:///run/containerd/containerd.sock실제 적용한 파일 내용입니다 이는 환경마다 다를 수 있습니다
# cat /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS="--network-plugin=cni --pod-infra-container-image=172.20.7.100:5000/pause:3.2 --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock"중지되었던 kubelet을 실행 합니다
# systemctl start kubelet
실행된 kubelet이 클러스터와 통신 하길 기다린 후 node의 상태를 확인 합니다
# kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
yb-master1 Ready master 5d8h v1.19.7 172.20.200.11 <none> CentOS Linux 8 4.18.0-240.1.1.el8_3.x86_64 docker://20.10.8
yb-master2 Ready master 5d8h v1.19.7 172.20.200.12 <none> CentOS Linux 8 4.18.0-240.1.1.el8_3.x86_64 docker://20.10.8
yb-master3 Ready master 5d8h v1.19.7 172.20.200.13 <none> CentOS Linux 8 4.18.0-240.1.1.el8_3.x86_64 docker://20.10.8
yb-worker1 Ready <none> 5d7h v1.19.7 172.20.200.14 <none> CentOS Linux 8 4.18.0-240.1.1.el8_3.x86_64 docker://20.10.8
yb-worker2 Ready <none> 5d7h v1.19.7 172.20.200.15 <none> CentOS Linux 8 4.18.0-240.1.1.el8_3.x86_64 docker://20.10.8
yb-worker3 Ready,SchedulingDisabled <none> 5d7h v1.19.7 172.20.200.16 <none> CentOS Linux 8 4.18.0-240.1.1.el8_3.x86_64 containerd://1.4.9
yb-worker4 Ready <none> 5d7h v1.19.7 172.20.200.17 <none> CentOS Linux 8 4.18.0-240.1.1.el8_3.x86_64 docker://20.10.8해당 워커의 런타임이 containerd://1.4.9로 잘 변경 되었습니다
이제 해당노드의 SchedulingDisabled 플래그를 해제하고 pod를 배치 해 봅니다
# kubectl uncordon yb-worker3
node/yb-worker3 uncordoned
kubectl get pod -A -o wide |grep worker3
default csi-cephfsplugin-provisioner-99d8c67f-jbvvs 0/6 ImagePullBackOff 0 5m23s 10.233.7.25 yb-worker3 <none> <none>
default csi-cephfsplugin-r55zm 0/3 ImagePullBackOff 0 5d5h 172.20.200.16 yb-worker3 <none> <none>
default csi-rbdplugin-provisioner-6c7745cb5c-nmvd5 0/7 ImagePullBackOff 0 5m23s 10.233.7.24 yb-worker3 <none> <none>
default csi-rbdplugin-wz59b 0/3 ImagePullBackOff 0 5d5h 172.20.200.16 yb-worker3 <none> <none>
default my-ds-ljz7t 0/1 ImagePullBackOff 0 5d6h 10.233.7.20 yb-worker3 <none> <none>
kube-system calico-node-pgj5t 0/1 Init:ImagePullBackOff 0 5d8h 172.20.200.16 yb-worker3 <none> <none>
kube-system kube-proxy-v6b5w 0/1 ImagePullBackOff 0 5d8h 172.20.200.16 yb-worker3 <none> <none>
metallb-system speaker-tk566 0/1 ImagePullBackOff 0 5d6h 172.20.200.16 yb-worker3 <none> <none>
monitoring node-exporter-gl49w 0/1 ImagePullBackOff 0 5d5h 10.233.7.21 yb-worker3 <none> <none>저의 경우 imagepull 에러가 발생햇는데요
사내에서사용하는 registry가 insecure형태이기 때문입니다
기존 도커는 daemon.json에 insecure를 등록하였지만 containerd는 아래와 같이 inseucre registry를 설정 합니다
/etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins.”io.containerd.grpc.v1.cri”.registry.mirrors]
[plugins.”io.containerd.grpc.v1.cri”.registry.mirrors.”docker.io”]
endpoint = [“https://registry-1.docker.io”]
[plugins.”io.containerd.grpc.v1.cri”.registry.mirrors.”172.20.7.100:5000″]
endpoint = [“http://172.20.7.100:5000”]
[plugins.”io.containerd.grpc.v1.cri”.registry.configs]
[plugins.”io.containerd.grpc.v1.cri”.registry.configs.”172.20.7.100:5000″.tls]
insecure_skip_verify = true
[plugins.”io.containerd.grpc.v1.cri”.registry.mirrors.”172.20.7.100:5000″] 부터 추가한 줄이며 레지스트리 주소를 적어 줍니다
수정된 insecure설정을 적용하기 위해 containerd를 재시작 합니다
# systemctl restart containerd
일정 시간이 경과되면 다시 이미지를 pull하기 시작합니다
# kubectl get pod -A -o wide |grep worker3
default csi-cephfsplugin-provisioner-99d8c67f-jbvvs 6/6 Running 4 17m 10.233.7.25 yb-worker3 <none> <none>
default csi-cephfsplugin-r55zm 3/3 Running 0 5d5h 172.20.200.16 yb-worker3 <none> <none>
default csi-rbdplugin-provisioner-6c7745cb5c-nmvd5 7/7 Running 5 17m 10.233.7.24 yb-worker3 <none> <none>
default csi-rbdplugin-wz59b 3/3 Running 0 5d5h 172.20.200.16 yb-worker3 <none> <none>
default my-ds-ljz7t 1/1 Running 0 5d7h 10.233.7.20 yb-worker3 <none> <none>
kube-system calico-node-pgj5t 1/1 Running 0 5d8h 172.20.200.16 yb-worker3 <none> <none>
kube-system kube-proxy-v6b5w 1/1 Running 0 5d8h 172.20.200.16 yb-worker3 <none> <none>
metallb-system speaker-tk566 1/1 Running 0 5d6h 172.20.200.16 yb-worker3 <none> <none>
monitoring node-exporter-gl49w 1/1 Running 0 5d5h 10.233.7.21 yb-worker3 <none> <none>
# kubectl get node -o wide |grep worker3
yb-worker3 Ready <none> 5d8h v1.19.7 172.20.200.16 <none> CentOS Linux 8 4.18.0-240.1.1.el8_3.x86_64 containerd://1.4.9최종적으로 runtime이 변경되었고 모든 pod가 정상 작동 되고 있습니다
이러한 방식으로 모든노드에 적용을 해주시면 됩니다
최종적으로 완료된 상태 입니다
# kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
yb-master1 Ready master 5d21h v1.19.7 172.20.200.11 <none> CentOS Linux 8 4.18.0-240.1.1.el8_3.x86_64 containerd://1.4.9
yb-master2 Ready master 5d21h v1.19.7 172.20.200.12 <none> CentOS Linux 8 4.18.0-240.1.1.el8_3.x86_64 containerd://1.4.9
yb-master3 Ready master 5d21h v1.19.7 172.20.200.13 <none> CentOS Linux 8 4.18.0-240.1.1.el8_3.x86_64 containerd://1.4.9
yb-worker1 Ready <none> 5d21h v1.19.7 172.20.200.14 <none> CentOS Linux 8 4.18.0-240.1.1.el8_3.x86_64 containerd://1.4.9
yb-worker2 Ready <none> 5d21h v1.19.7 172.20.200.15 <none> CentOS Linux 8 4.18.0-240.1.1.el8_3.x86_64 containerd://1.4.9
yb-worker3 Ready <none> 5d21h v1.19.7 172.20.200.16 <none> CentOS Linux 8 4.18.0-240.1.1.el8_3.x86_64 containerd://1.4.9
yb-worker4 Ready <none> 5d20h v1.19.7 172.20.200.17 <none> CentOS Linux 8 4.18.0-240.1.1.el8_3.x86_64 containerd://1.4.9모든 노드의 container runtime이 docker 에서 containerd로 변경 되었습니다
추가로 kubernetes node정보에 있는 container runtime 정보도 변경 해줘야 합니다
아래 명령어로 현재 설정된 annotations을 삭제하고 추가 해 줍니다
# kubectl annotate node/{nodename} kubeadm.alpha.kubernetes.io/cri-socket-
node/{nodename} annotated
# kubectl annotate node/{nodename} kubeadm.alpha.kubernetes.io/cri-socket=unix:///run/containerd/containerd.sock
node/{nodename} annotated
원래값은 아래와 같이 dockershime 입니다
kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock이제 더 이상 docker 명령어는 사용되지 않습니다
개발 환경에서는 여전히 docker를 이용하여 개발하고 build하겠지만 containerd에서는 ctr명령어를 이용하여 확인 합니다
몇가지 명령어를 소개 시켜 드리겠습니다
먼저 k8s에서 익숙한 namespace를 사용합니다
# ctr namespaces list
NAME LABELS
default
k8s.io
moby위 명령어를 통해 ns를 확인하고 사용합니다
container list는 기존 docker ps와 동일 하다고 보시면 됩니다
ctr -n k8s.io container list
CONTAINER IMAGE RUNTIME
0361f2ccdc58e0d1b60584f2f3971791aa8b39940c2d2a594d904edf56713e84 k8s.gcr.io/pause:3.2 io.containerd.runc.v2
0b99ef42759cc905a562b03a83ee4691c3b2bc98b29bbe7e4f156ce3746d5bd3 k8s.gcr.io/pause:3.2 io.containerd.runc.v2
0e4cee03df6709dd4b62f1af86e6eb926620a0f0a7ca1c6e4c1e62eceac16bae 172.20.7.100:5000/calico/cni:v3.20.0 io.containerd.runc.v2
1fe0b9b7182101fcef78eb49546252725ce88b762da1aa5287ef192ffb7893c6 k8s.gcr.io/pause:3.2 io.containerd.runc.v2
2527c130cd77a855ec7bd52d856a12632a3bd3786b01a6cb99465ebde0c78a9b 172.20.7.100:5000/sig-storage/csi-provisioner@sha256:3b465cbcadf7d437fc70c3b6aa2c93603a7eef0a3f5f1e861d91f303e4aabdee io.containerd.runc.v2
.....docker images 명령어와 대입되는 는 아래와 같습니다
ctr -n k8s.io image list
REF TYPE DIGEST SIZE PLATFORMS LABELS
172.20.7.100:5000/calico/cni:v3.20.0 application/vnd.docker.distribution.manifest.v2+json sha256:7c43c152cdf589ed789528c4055503d2f2b0ee943d7815c820fd8e99793f36b3 46.1 MiB linux/amd64 io.cri-containerd.image=managed
172.20.7.100:5000/calico/cni@sha256:7c43c152cdf589ed789528c4055503d2f2b0ee943d7815c820fd8e99793f36b3 application/vnd.docker.distribution.manifest.v2+json sha256:7c43c152cdf589ed789528c4055503d2f2b0ee943d7815c820fd8e99793f36b3 46.1 MiB linux/amd64 io.cri-containerd.image=managed
172.20.7.100:5000/calico/node:v3.20.0 application/vnd.docker.distribution.manifest.v2+json sha256:913955a36179a53d36a0df7f26319a5278991eea2514eab339cce15df24c220c 57.9 MiB linux/amd64 io.cri-containerd.image=managed
.....기존 dokcer 명령어와 입력이나 출력이 달라 당황 스러우시죠? ctr은 docker명령어 처럼 containerd에만 작동하는 명령어 입니다
사실 예전부터 쿠버네티스에서는 cri표준을 준수하는 컨테이너 런타임을 통합하여 관리하기 위해 crictl이란 명령어 툴을 제공하고 있습니다
crictl은 docker,ctr과 달리 cri를 준수하는 모든 컨테이너 런타임에 사용이 가능합니다
일반적인 패키지 매니저를 통해 kubeadm kubelet등을 설치했다면 자동으로 설치 되어 집니다
바이너리로 설치하셨다면 cri-tools을 설치 하셔야 작동 합니다
먼저 crictl이 containerd로 작동할 수 있도록 아래 설정파일을 만들어 줍니다
cat /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sockcrictl 도움말을 확인 합니다
# crictl
NAME:
crictl - client for CRI
USAGE:
crictl [global options] command [command options] [arguments...]
VERSION:
v1.13.0
COMMANDS:
attach Attach to a running container
create Create a new container
exec Run a command in a running container
version Display runtime version information
images List images
inspect Display the status of one or more containers
inspecti Return the status of one or more images
inspectp Display the status of one or more pods
logs Fetch the logs of a container
port-forward Forward local port to a pod
ps List containers
pull Pull an image from a registry
runp Run a new pod
rm Remove one or more containers
rmi Remove one or more images
rmp Remove one or more pods
pods List pods
start Start one or more created containers
info Display information of the container runtime
stop Stop one or more running containers
stopp Stop one or more running pods
update Update one or more running containers
config Get and set crictl options
stats List container(s) resource usage statistics
completion Output bash shell completion code
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS:
--config value, -c value Location of the client config file (default: "/etc/crictl.yaml") [$CRI_CONFIG_FILE]
--debug, -D Enable debug mode
--image-endpoint value, -i value Endpoint of CRI image manager service [$IMAGE_SERVICE_ENDPOINT]
--runtime-endpoint value, -r value Endpoint of CRI container runtime service (default: "unix:///var/run/dockershim.sock") [$CONTAINER_RUNTIME_ENDPOINT]
--timeout value, -t value Timeout of connecting to the server (default: 10s)
--help, -h show help
--version, -v print the version
# crictl ps
CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT POD ID
f04f4408e17cb 35036a0cd23a8 22 minutes ago Running kube-scheduler 1 6ae463313ab52
09115790bcb05 90f4ff69a0bf9 22 minutes ago Running kube-controller-manager 1 2e3329a303dfb
9a721c09d0725 48d79e554db69 24 minutes ago Running dashboard-metrics-scraper 0 0aa2dc0bfa7c7
849202a25a780 76ba70f4748f9 24 minutes ago Running calico-kube-controllers 0 3dea2a33b574d
f823a99ae8549 5ef66b403f4f0 28 minutes ago Running calico-node 0 de53af8e19dd6
064ec7a833b5d 76696340d7993 28 minutes ago Running kube-apiserver 1 207d0374f2c90
8fd8537f6b190 e9f480f8f070f 28 minutes ago Running speaker 0 c0ff09a3cd6ac
fe390399e5d68 0369cf4303ffd 28 minutes ago Running etcd 0 580583d722d98
946355db2d8d3 046ec6b49f0b9 28 minutes ago Running kube-proxy 0 e5bd6804fc05f어때요 ctr과는 달리 굉장히 익숙하죠? docker와 모든것이 동일 합니다(bulid나 tag push등의 개발 명령어만 없습니다)
crictl로는 pod의 상태도 볼 수 있습니다
crictl pods
POD ID CREATED STATE NAME NAMESPACE ATTEMPT
0aa2dc0bfa7c7 11 minutes ago Ready dashboard-metrics-scraper-88b485d79-wxv27 kubernetes-dashboard 0
3dea2a33b574d 11 minutes ago Ready calico-kube-controllers-784b5b48f4-45nnn kube-system 0
580583d722d98 15 minutes ago Ready etcd-yb-master1 kube-system 0
2e3329a303dfb 15 minutes ago Ready kube-controller-manager-yb-master1 kube-system 0
207d0374f2c90 15 minutes ago Ready kube-apiserver-yb-master1 kube-system 0
e5bd6804fc05f 15 minutes ago Ready kube-proxy-8sf8f kube-system 0
6ae463313ab52 15 minutes ago Ready kube-scheduler-yb-master1 kube-system 0
c0ff09a3cd6ac 15 minutes ago Ready speaker-6lmn8 metallb-system 0
de53af8e19dd6 15 minutes ago Ready calico-node-6rd8v kube-system 0이외에 images rm rmi exec run등 모두 도커(docker-cli)와 동일하게 동작 합니다
ctr보다 crictl을 사용하시면 조금 더 빠르게 적응 하실 수 있습니다
지금까지 쿠버네티스 컨테이너 런타임을 docker 에서 containerd로 변경하는 방법과 crictl 명령어 사용 방법을 안내 해 드렸습니다
생각보다는 간단하죠? dockershim이 완전히 사용 불가능해 지기전에 미리 준비하고 이전하여 새로운 기능에 익숙해 지는것을 추천 합니다
감사합니다