Openstack Helm을 Centos 환경에서 Ceph RBD로 구축 하는 방법에 대하여 이야기 합니다.
공식 매뉴얼이 Ubuntu 및 Swift기반이기 때문에 커스텀하게 수정한 부분이 다소 존재 합니다. 하지만, 기본 배경은 공식 문서( https://docs.openstack.org/openstack-helm/latest/ )를 기반으로 작성 되었습니다.
Openstack helm ?
openstack helm은 이름에서 알수 있듯이 openstack을 kubernetes환경에서 helm chart를 이용하여 배포할 수 있는 프로젝트입니다. Kubernetes의 기본 기능인 Self-Healing 을 비롯하여 Kubernetes의 많은 장점들을 이용하여 Openstack을 관리 하기 때문에 확장등의 라이프 사이클 관리에 도움이 되며 을 구축, 업그레이드, 확장등의 관리를 손쉽게 할 수 있습니다. 대부분 AT&T를 주도적으로 99cloud , Suse등과 함께 국내에서는 SK Telecom이 많은 기여 하고 있는 프로젝트입니다.
환경 설정
아래의 환경과 같이 Centos7의 동일한 운영체제를 사용하며, 총 5개의 NIC이 용도별로 설정 되어 있어야 합니다. 이중 001노드에서는 Kubernetes Master및 Ceph Monitor의 역할과 배포의 역할을 담당하게 됩니다.
| hostname | OS | IP address | ETC |
|---|---|---|---|
| kube-cy4-kube001 | CentOS Linux release 7.8.2003 | eth0 : 10.4.10.21 (API, Deploy )eth1 : 10.4.20.21 (Ceph Storage Public )eth2 : 10.4.30.21 (Ceph Storage Replication )eth3 : 10.4.40.21 (Openstack Tanent Network )eth4 : 192.168.193.21 (Openstack External Netowkr: Provider Network ) | Kubernetes MasterCeph Monitor |
| kube-cy4-kube002 | CentOS Linux release 7.8.2003 | eth0 : 10.4.20.22 (API, Deploy )eth1 : 10.4.20.22 (Ceph Storage Public )eth2 : 10.4.30.22 (Ceph Storage Replication )eth3 : 10.4.40.22 (Openstack Tanent Network )eth4 : 192.168.193.22 (Openstack External Netowkr: Provider Network ) | Kubernetes WorkerCeph OSD |
| kube-cy4-kube003 | CentOS Linux release 7.8.2003 | eth0 : 10.4.10.23 (API, Deploy )eth1 : 10.4.20.23 (Ceph Storage Public )eth2 : 10.4.30.23 (Ceph Storage Replication )eth3 : 10.4.40.23 (Openstack Tanent Network )eth4 : 192.168.193.23 (Openstack External Netowkr: Provider Network ) | Kubernetes WorkerCeph OSD |
| kube-cy4-kube004 | CentOS Linux release 7.8.2003 | eth0 : 10.4.10.24 (API, Deploy )eth1 : 10.4.20.24 (Ceph Storage Public )eth2 : 10.4.30.24 (Ceph Storage Replication )eth3 : 10.4.40.24 (Openstack Tanent Network )eth4 : 192.168.193.24 (Openstack External Netowkr: Provider Network ) | Kubernetes WorkerCeph OSD |
Deploy Ceph
Ceph는 RBD형태로 Openstack상에서 볼륨을 사용하거나 인스턴스를 이용할 경우 사용되는 스토리지로 사용되며, Kubernetes상에 배포되는 Openstack-helm-infra의 Mariadb,RabbitMQ의 PV로 사용 됩니다.
배포에 필요한 스크립트는 “/hoem/deploy” 에서 관리하기 위하여 해당 디렉토리를 생성하며, 프로젝트를 클론 받기 위하여 git과 배포시 필요한 파이선 패키지 관리를 위하여 pip 패키지를 설치합니다.
[root@kube-cy4-kube001 ~]# mkdir /home/deploy ; cd /home/deploy [root@kube-cy4-kube001 deploy]# yum install -y git epel* [root@kube-cy4-kube001 deploy]# yum install -y python-pip
ceph 배포를 위하여 ceph-ansible을 클론받고, v4.0.20 로 checkout합니다. 그리고 배포를 위한 파이선 패키지 설치를 위하여 requirements.txt 파일을 기반으로 파이선 패키지를 설치 합니다.
[root@kube-cy4-kube001 deploy]# git clone https://github.com/ceph/ceph-ansible.git [root@kube-cy4-kube001 deploy]# cd ceph-ansible/ [root@kube-cy4-kube001 ceph-ansible]# git checkout v4.0.20 [root@kube-cy4-kube001 ceph-ansible]# pip install -r requirements.txt
ceph은 ansible을 통하여 배포하기 때문에 deploy 인터페이스 아이피로 “/etc/hosts” 파일을 수정 합니다.
[root@kube-cy4-kube001 ceph-ansible]# tee /etc/hosts << EOF 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 10.4.10.21 kube-cy4-kube001 kube1 10.4.10.22 kube-cy4-kube003 kube2 10.4.10.23 kube-cy4-kube002 kube3 10.4.10.24 kube-cy4-kube004 kube4 EOF
ceph-ansible에서 각 역할별로 구분하여 배포 하기 때문에 Group 별로 나눠 ceph-ansible의 inventory파일을 생성하며, 해당 inventory파일로 통신이 되는지 확인 합니다.
[root@kube-cy4-kube001 ceph-ansible]# tee ./ceph-hosts << EOF
[mons]
kube-cy4-kube001
[osds]
kube-cy4-kube002
kube-cy4-kube003
kube-cy4-kube004
[mdss]
[rgws]
[nfss]
[rbdmirrors]
[clients]
kube-cy4-kube001
[mgrs]
kube-cy4-kube001
[iscsigws]
[iscsi-gws]
[grafana-server]
[rgwloadbalancers]
[all:vars]
ansible_become=true
#### 설정된 User로 변경 필요
ansible_user=centos
#### 설정된 Password로 변경 필요
ansible_ssh_pass=password
EOF
[root@kube-cy4-kube001 ceph-ansible]# yum install -y sshpass
[root@kube-cy4-kube001 ceph-ansible]# ansible -i ceph-hosts -m ping all
kube-cy4-kube003 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python"
},
"changed": false,
"ping": "pong"
}
kube-cy4-kube002 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python"
},
"changed": false,
"ping": "pong"
}
...
group_vars/all.yml 파일은 ceph-ansible을 이용하여 배포시 설정할 수 있는 전반적인 배포 변수를 설정합니다.
[root@kube-cy4-kube001 ceph-ansible]# tee group_vars/all.yml << EOF
osd_scenario: lvm
osd_objectstore: bluestore
## monitor로 통신할 인터페이스로 실제 rbd를 사용할 경우 해당 인터페이스를 사용하게 됩니다.
monitor_interface: eth1
## monitor_interface 에 설정되어 있는 ip 대역으로 설정
public_network: 10.4.20.0/24
## 각 osd마다 복제를 위해 사용하는 인터페이스의 ip 대역으로 설정 합니다.
cluster_network: 10.4.30.0/24
ceph_stable_release: nautilus
ceph_origin: repository
ceph_repository: community
ceph_mirror: http://download.ceph.com
ceph_stable_key: https://download.ceph.com/keys/release.asc
ntp_service_enabled: true
osd_auto_discovery: false
dashboard_enabled: false
cluster: ceph
ceph_conf_overrides:
global:
mon_allow_pool_delete: false
mon_osd_down_out_subtree_limit: host
osd_pool_default_size: 2
osd_pool_default_min_size: 1
osd:
osd_min_pg_log_entries: 10
osd_max_pg_log_entries: 10
osd_pg_log_dups_tracked: 10
osd_pg_log_trim_min: 10
EOF
osds.yml 에서는 실제 osd로 사용할 디바이스를 지정합니다. osd 노드의 연결되어 있는 디바이스를 확인후 아래와 같이 osds.yml파일을 설정합니다.
[root@kube-cy4-kube001 ceph-ansible]# ansible -i ceph-hosts -ba 'lsblk' osds kube-cy4-kube004 | CHANGED | rc=0 >> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 366K 0 rom vda 253:0 0 68G 0 disk └─vda1 253:1 0 68G 0 part / vdb 253:16 0 40G 0 disk vdc 253:32 0 40G 0 disk kube-cy4-kube002 | CHANGED | rc=0 >> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 366K 0 rom vda 253:0 0 68G 0 disk └─vda1 253:1 0 68G 0 part / vdb 253:16 0 40G 0 disk vdc 253:32 0 40G 0 disk kube-cy4-kube003 | CHANGED | rc=0 >> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 366K 0 rom vda 253:0 0 68G 0 disk └─vda1 253:1 0 68G 0 part / vdb 253:16 0 40G 0 disk vdc 253:32 0 40G 0 disk [root@kube-cy4-kube001 ceph-ansible]# tee group_vars/osds.yml << EOF --- devices: - /dev/vdb - /dev/vdc EOF
clients.yml 파일에 admin key를 복사하여 ceph client명령이 수행할 수 있도록 설정 합니다.
[root@kube-cy4-deploy ceph-ansible]# tee group_vars/clients.yml << EOF --- copy_admin_key: true EOF
site.yml.sample 을 이용하여 설정한 group_vars의 변수를 기반으로 ansible-playbook 명령을 이용하여 ceph를 배포합니다.
[[root@kube-cy4-kube001 ceph-ansible]# ansible-playbook -i ceph-hosts site.yml.sample
만약 아래와 같은 에러가 발생하여 정상적으로 설치가 안되는 경우가 발생할 수 있습니다.
... TASK [check for python] ******************************************************************************* Saturday 01 August 2020 15:46:14 +0900 (0:00:00.058) 0:00:00.058 ******* fatal: [kube-cy4-kube001]: FAILED! => msg: The ips_in_ranges filter requires python's netaddr be installed on the ansible controller. fatal: [kube-cy4-kube002]: FAILED! => msg: The ips_in_ranges filter requires python's netaddr be installed on the ansible controller. fatal: [kube-cy4-kube003]: FAILED! => msg: The ips_in_ranges filter requires python's netaddr be installed on the ansible controller. fatal: [kube-cy4-kube004]: FAILED! => msg: The ips_in_ranges filter requires python's netaddr be installed on the ansible controller. ...
배포하는 호스트에서 netaddr 패키지를 설치한뒤 다시 배포를 진행 하면 정상적으로 배포가 진행됩니다.
[root@kube-cy4-kube001 ceph-ansible]# yum install python-netaddr.noarch -y
만약, ceph cluster를 삭제 하기 위해서는 아래와 같이 purge-cluster.yml 플레이북을 실행하여 삭제할 수 있습니다.
[root@kube-cy4-kube001 ceph-ansible]# ansible-playbook -i ceph-hosts infrastructure-playbooks/purge-cluster.yml
배포가 완료 되면 “ceph -s “명령을 이용하여 ” health: HEALTH_OK” 상태를 확인 합니다.
[root@kube-cy4-kube001 ceph-ansible]# ceph -s
cluster:
id: f9b17cb6-b38c-455b-b10d-5c44d7bcc36b
health: HEALTH_OK
services:
mon: 1 daemons, quorum kube-cy4-kube001 (age 3m)
mgr: kube-cy4-kube001(active, since 2m)
osd: 6 osds: 6 up (since 85s), 6 in (since 98s)
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 6.0 GiB used, 234 GiB / 240 GiB avail
pgs:
Install Docker
Kubernetes를 배포하기전 Container Runtime으로 사용될 Docker를 설치 합니다.
아래와 같이 docker-ce repository를 등록한뒤 docker-ce를 yum으로 설치 합니다. 해당 과정은 모든 노드에서 진행 하게 됩니다.
[root@kube-cy4-kube001 ~]# yum install -y yum-utils
[root@kube-cy4-kube001 ~]# yum-config-manager \
--add-repo \
https://download.docker.com/linux/centos/docker-ce.repo
[root@kube-cy4-kube001 ~]# yum install docker-ce docker-ce-cli containerd.io
[root@kube-cy4-kube001 ~]# systemctl enable --now docker
[root@kube-cy4-kube001 ~]# systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2020-08-01 13:56:13 KST; 2h 34min ago
Docs: https://docs.docker.com
Main PID: 1368 (dockerd)
Tasks: 15
Memory: 147.2M
CGroup: /system.slice/docker.service
└─1368 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
...
Deploy Kubernetes
kubernetes를 설치하기 위해서 kubeadm을 사용할것 입니다. kubeadm설치와 kubelet설치를 위하여 kubernetes upstream repository를 등록 하여 kubeadm,kubelet를 설치 합니다.
해당 설치는 모든 호스트에서 진행 합니다.
[root@kube-cy4-kube001 ~]# cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
[root@kube-cy4-kube001 ~]# yum install kubeadm -y
[root@kube-cy4-kube001 ~]# systemctl enable kubelet --no
“kubeadm init ” 명령을 이용하여 master노드로 사용할 kube-cy4-kube001 호스트에서 초기화 과정을 진행 합니다.
이때 kubernetes api를 사용할 인터페이스인 eth0의 ip로 ” –apiserver-advertise-address” 옵션을 추가 하며, pod들이 통신하기 위하여 사용되는 네트워크 cidr를 172.16.0.0/16으로 진행 합니다.
[root@kube-cy4-kube001 ~]# echo 1 > /proc/sys/net/ipv4/ip_forward [root@kube-cy4-kube001 ~]# kubeadm init --apiserver-advertise-address=10.4.10.21 --pod-network-cidr=172.16.0.0/16
정상적으로 진행 되었습니다면 아래와 같이 “initialized successfully” 메시지를 확인할 수 있습니다.
마지막에 나온 hash 값을 통하여 다른 노드들이 join할 수 있도록 메시지를 출력한것을 확인 합니다.
...
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 10.4.10.21:6443 --token 26amyi.687200qzjh5lkkxw \
--discovery-token-ca-cert-hash sha256:e1a4959da94c40d0d21aaf8fb39878608c0002a4a6be6122bc8fa3d116b5db9f
위에서 중간에 나온 메시지 처럼 master노드인 kube-cy4-kube001 에서 kubernetes clinet를 위해 사용되는 kube config파일을 복사하도록 하여 kubernetes clinet(kubectl)을 실행하여 봅니다.
[root@kube-cy4-kube001 ~]# mkdir -p $HOME/.kube [root@kube-cy4-kube001 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config [root@kube-cy4-kube001 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config [root@kube-cy4-kube001 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION kube-cy4-kube001 NotReady master 2m45s v1.18.6
kube-cy4-kube002 노드에서 master 노드로 join을 합니다.
[root@kube-cy4-kube002 ~]# echo 1 > /proc/sys/net/ipv4/ip_forward [root@kube-cy4-kube002 ~]# kubeadm join 10.4.10.21:6443 --token 26amyi.687200qzjh5lkkxw \ > --discovery-token-ca-cert-hash sha256:e1a4959da94c40d0d21aaf8fb39878608c0002a4a6be6122bc8fa3d116b5db9f
kube-cy4-kube003 노드에서 master 노드로 join을 합니다.
[root@kube-cy4-kube003 ~]# echo 1 > /proc/sys/net/ipv4/ip_forward [root@kube-cy4-kube003 ~]# kubeadm join 10.4.10.21:6443 --token 26amyi.687200qzjh5lkkxw \ > --discovery-token-ca-cert-hash sha256:e1a4959da94c40d0d21aaf8fb39878608c0002a4a6be6122bc8fa3d116b5db9f
kube-cy4-kube004 노드에서 master 노드로 join을 합니다.
[root@kube-cy4-kube004 ~]# echo 1 > /proc/sys/net/ipv4/ip_forward [root@kube-cy4-kube004 ~]# kubeadm join 10.4.10.21:6443 --token 26amyi.687200qzjh5lkkxw \ > --discovery-token-ca-cert-hash sha256:e1a4959da94c40d0d21aaf8fb39878608c0002a4a6be6122bc8fa3d116b5db9f
다시 master노드로 들어와서 “kubectl get nodes -o wide” 명령을 수행하여 join된 노드를 확인할 수 있습니다.
하지만 아직 CNI(Container Network Interface)가 설정이 안되어 있기 때문에 STATUS 가 아직 NotReady 상태입니다.
[root@kube-cy4-kube001 ~]# kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME kube-cy4-kube001 NotReady master 6m8s v1.18.6 10.4.10.21 <none> CentOS Linux 7 (Core) 3.10.0-1127.13.1.el7.x86_64 docker://19.3.12 kube-cy4-kube002 NotReady <none> 40s v1.18.6 10.4.20.22 <none> CentOS Linux 7 (Core) 3.10.0-1127.13.1.el7.x86_64 docker://19.3.12 kube-cy4-kube003 NotReady <none> 38s v1.18.6 10.4.20.23 <none> CentOS Linux 7 (Core) 3.10.0-1127.13.1.el7.x86_64 docker://19.3.12 kube-cy4-kube004 NotReady <none> 36s v1.18.6 10.4.20.24 <none> CentOS Linux 7 (Core) 3.10.0-1127.13.1.el7.x86_64 docker://19.3.12
Deploy Calico
calico배포를 위한 매니패스트를 다운로드 한뒤 pod 네트워크를 위한 인터페이스 추가 및 IP in IP 모드를 Never로 변경하여 L2 통신간에 encapsulation없이 통신하도록 합니다.
[root@kube-cy4-kube001 ~]# mkdir /home/deploy/calico ; cd /home/deploy/calico
[root@kube-cy4-kube001 calico]# yum install -y wget
[root@kube-cy4-kube001 calico]# wget https://docs.projectcalico.org/manifests/calico.yaml
[root@kube-cy4-kube001 calico]# vi calico.yaml
...
containers:
# Runs calico-node container on each Kubernetes node. This
# container programs network policy and routes on each
# host.
- name: calico-node
image: calico/node:v3.15.1
env:
### pod네트워크 인터페이스 추가
- name: IP_AUTODETECTION_METHOD
value: "interface=eth0"
# Use Kubernetes API as the backing datastore.
- name: DATASTORE_TYPE
value: "kubernetes"
# Wait for the datastore.
- name: WAIT_FOR_DATASTORE
value: "true"
# Set based on the k8s node name.
....
### IP in IP 모드를 사용하지 않도록 하기 위하여 Always를 Never로 변경
- name: CALICO_IPV4POOL_IPIP
# value: "Always"
value: "Never"
[root@kube-cy4-kube001 calico]# kubectl create -f calico.yaml
이제, calico-system namespace에 pod들이 정상적으로 동작하는것을 확인할 수 있으며, ” kubectl get nodes” 명령 수행시 Ready 상태로 변경된것을 확인할 수 있습니다.
[root@kube-cy4-kube001 calico]# kubectl get pods -n calico-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES calico-kube-controllers-5687f44fd5-nj49z 1/1 Running 0 2m8s 172.16.10.130 kube-cy4-kube004 <none> <none> calico-node-48lw6 1/1 Running 0 2m8s 10.4.20.24 kube-cy4-kube004 <none> <none> calico-node-6kt28 1/1 Running 0 2m8s 10.4.10.21 kube-cy4-kube001 <none> <none> calico-node-jpqf5 1/1 Running 0 2m8s 10.4.20.23 kube-cy4-kube003 <none> <none> calico-node-lh7fp 1/1 Running 0 2m8s 10.4.20.22 kube-cy4-kube002 <none> <none> calico-typha-7648bcdddb-4cblz 1/1 Running 0 60s 10.4.20.22 kube-cy4-kube002 <none> <none> calico-typha-7648bcdddb-4mjvp 1/1 Running 0 60s 10.4.20.24 kube-cy4-kube004 <none> <none> calico-typha-7648bcdddb-5qz8m 1/1 Running 0 2m8s 10.4.20.23 kube-cy4-kube003 <none> <none> calico-typha-7648bcdddb-kq5q6 1/1 Running 0 60s 10.4.10.21 kube-cy4-kube001 <none> <none> [root@kube-cy4-kube001 calico]# kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME kube-cy4-kube001 Ready master 24m v1.18.6 10.4.10.21 <none> CentOS Linux 7 (Core) 3.10.0-1127.13.1.el7.x86_64 docker://19.3.12 kube-cy4-kube002 Ready <none> 19m v1.18.6 10.4.20.22 <none> CentOS Linux 7 (Core) 3.10.0-1127.13.1.el7.x86_64 docker://19.3.12 kube-cy4-kube003 Ready <none> 19m v1.18.6 10.4.20.23 <none> CentOS Linux 7 (Core) 3.10.0-1127.13.1.el7.x86_64 docker://19.3.12 kube-cy4-kube004 Ready <none> 18m v1.18.6 10.4.20.24 <none> CentOS Linux 7 (Core) 3.10.0-1127.13.1.el7.x86_64 docker://19.3.12
calicoctl 바이너리를 다운로드 받아 calico cni 가 제어할 pod들의 ippool의 목록과 bgp peer정보를 확인 합니다.
[root@kube-cy4-kube001 calico]# curl -O -L https://github.com/projectcalico/calicoctl/releases/download/v3.15.1/calicoctl [root@kube-cy4-kube001 calico]# chmod +x calicoctl [root@kube-cy4-kube001 calico]# sudo mv calicoctl /usr/local/bin/ [root@kube-cy4-kube001 calico]# tee /root/calico.rc << EOF export KUBECONFIG=/root/.kube/config export DATASTORE_TYPE=kubernetes EOF [root@kube-cy4-kube001 calico]# source ~/calico.rc [root@kube-cy4-kube001 calico]# calicoctl get ippools NAME CIDR SELECTOR default-ipv4-ippool 172.16.0.0/16 all() [root@kube-cy4-kube001 calico]# calicoctl get nodes -o wide NAME ASN IPV4 IPV6 kube-cy4-kube001 (64512) 10.4.10.21/24 kube-cy4-kube002 (64512) 10.4.10.22/24 kube-cy4-kube003 (64512) 10.4.10.23/24 kube-cy4-kube004 (64512) 10.4.10.24/24
실제 pod통신은 아래와 같이 eth0인터페이스를 통하여 172.16.0.0/16 대역에서 각 노드별 block 대역으로 직접 라우팅 되는것을 확인할 수 있습니다.
[root@kube-cy4-kube001 calico]# route -n | grep 172.16 172.16.10.128 10.4.10.24 255.255.255.192 UG 0 0 0 eth0 172.16.39.128 10.4.10.22 255.255.255.192 UG 0 0 0 eth0 172.16.43.192 10.4.10.23 255.255.255.192 UG 0 0 0 eth0
Install Helm
openstack helm과 openstack helm infra는 helm기반으로 배포가 되기 때문에 helm을 설치 합니다.
[root@kube-cy4-kube001 ~ ]# curl -LO https://git.io/get_helm.sh
[root@kube-cy4-kube001 ~ ]# chmod 700 get_helm.sh
[root@kube-cy4-kube001 ~ ]# ./get_helm.sh
[root@kube-cy4-kube001 ~ ]# kubectl create serviceaccount --namespace kube-system tiller
[root@kube-cy4-kube001 ~ ]# kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
[root@kube-cy4-kube001 ~ ]# kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'
[root@kube-cy4-kube001 ~ ]# helm init --service-account tiller --upgrade
[root@kube-cy4-kube001 ~ ]# helm version
Client: &version.Version{SemVer:"v2.16.9", GitCommit:"8ad7037828e5a0fca1009dabe290130da6368e39", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.16.9", GitCommit:"8ad7037828e5a0fca1009dabe290130da6368e39", GitTreeState:"clean"}
local helm repository를 등록 합니다.
[root@kube-cy4-kube001 ~ ]# tee /etc/systemd/system/helm-serve.service <<EOF [Unit] Description=Helm Server After=network.target [Service] User=root Restart=always ExecStart=/usr/local/bin/helm serve [Install] WantedBy=multi-user.target EOF [root@kube-cy4-kube001 ~ ]# systemctl daemon-reload ; systemctl enable helm-serve --now [root@kube-cy4-kube001 calico]# helm repo list NAME URL local http://localhost:8879/charts
Setting CSI(Ceph RBD)
앞에서 배포한 ceph cluster 를 kubernetes에서 storage class를 총하여 pv를 생성하기 위하여 csi를 설정 합니다.
“ceph mon dump” 명령을 이용하여 cluster fsid를 확인합니다.
[root@kube-cy4-kube001 ~]# ceph mon dump dumped monmap epoch 1 epoch 1 fsid f9b17cb6-b38c-455b-b10d-5c44d7bcc36b last_changed 2020-08-01 16:23:11.258185 created 2020-08-01 16:23:11.258185 min_mon_release 14 (nautilus) 0: [v2:10.4.20.21:3300/0,v1:10.4.20.21:6789/0] mon.kube-cy4-kube001
storage class로 사용할 “kubernetes”이름의 pool을 생성하며, kubernetes pool 인증에 사용할 kubernetes유저를 생성합니다.
[root@kube-cy4-kube001 ~]# ceph osd pool create kubernetes 64 64 pool 'kubernetes' created [root@kube-cy4-kube001 ~]# rbd pool init kubernetes [root@kube-cy4-kube001 ~]# ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes' [client.kubernetes] key = AQBMeiVf1CKrHBAAYeIVScZlRiDo6D58xvPM4Q==
ceph-csi 프로젝트를 클론 받습니다.
[root@kube-cy4-kube001 ~]# cd /home/deploy/ [root@kube-cy4-kube001 deploy]# git clone https://github.com/ceph/ceph-csi.git ; cd ceph-csi/
csi-config-map에서는 앞에서 확인한 fsid와 초기 ceph구성시 monitor로 지정한 노드의 public network 대역의 아이피로 수정 한뒤 configmap을 적용합니다.
[root@kube-cy4-kube001 ceph-csi]# vi deploy/rbd/kubernetes/csi-config-map.yaml
---
apiVersion: v1
kind: ConfigMap
data:
config.json: |-
[
{
##### fsid
"clusterID": "f9b17cb6-b38c-455b-b10d-5c44d7bcc36b",
"monitors": [
##### Monitor 호스트의 아이피
"10.4.20.21:6789"
]
}
]
metadata:
name: ceph-csi-config
[root@kube-cy4-kube001 ceph-csi]# kubectl create -f deploy/rbd/kubernetes/csi-config-map.yaml
secret에는 앞에서 생성한 pool에 인증하기 위한 userkey와 id를 추가 한뒤 secret을 생성 합니다.
[root@kube-cy4-kube001 ceph-csi]# vi examples/rbd/secret.yaml --- apiVersion: v1 kind: Secret metadata: name: csi-rbd-secret namespace: default stringData: # Key values correspond to a user name and its key, as defined in the # ceph cluster. User ID should have required access to the 'pool' # specified in the storage class #userID: <plaintext ID> userID: kubernetes #userKey: <Ceph auth key corresponding to ID above> userKey: AQBMeiVf1CKrHBAAYeIVScZlRiDo6D58xvPM4Q== # Encryption passphrase encryptionPassphrase: test_passphrase [root@kube-cy4-kube001 ceph-csi]# kubectl create -f examples/rbd/secret.yaml
pool은 앞서 생성한 pool의 이름으로, clusterID는 ceph fsid를 입력합니다.
[root@kube-cy4-kube001 ceph-csi]# vi examples/rbd/storageclass.yaml ... #clusterID: <cluster-id> clusterID: f9b17cb6-b38c-455b-b10d-5c44d7bcc36b # If you want to use erasure coded pool with RBD, you need to create # two pools. one erasure coded and one replicated. # You need to specify the replicated pool here in the `pool` parameter, it is # used for the metadata of the images. # The erasure coded pool must be set as the `dataPool` parameter below. # dataPool: ec-data-pool #pool: rbd pool: kubernetes ... [root@kube-cy4-kube001 ceph-csi]# kubectl create -f examples/rbd/storageclass.yaml
ceph-csi가 vault를 이용한 키관리를 하기 때문에 해당부분을 적용합니다.
[root@kube-cy4-kube001 ceph-csi]# kubectl create -f examples/kms/vault/kms-config.yaml
이제 적용한 내용으로 plugin을 배포 합니다.
[root@kube-cy4-kube001 ceph-csi]# cd examples/rbd/ [root@kube-cy4-kube001 rbd]# ./plugin-deploy.sh
정상적으로 배포가 완료 되었습니다면 아래와 같이 pod를 확인할 수 있으며, storage class를 이용하여 pvc를 정상적으로 생성할 수 있습니다.
[root@kube-cy4-kube001 rbd]# kubectl get pod NAME READY STATUS RESTARTS AGE csi-rbdplugin-2m68m 3/3 Running 0 19s csi-rbdplugin-8xfpd 3/3 Running 0 19s csi-rbdplugin-provisioner-b77dfc64c-469b6 6/6 Running 0 20s csi-rbdplugin-provisioner-b77dfc64c-lwgg9 6/6 Running 0 20s csi-rbdplugin-provisioner-b77dfc64c-wnxkt 6/6 Running 0 20s csi-rbdplugin-r9v28 3/3 Running 0 19s [root@kube-cy4-kube001 rbd]# kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE csi-rbd-sc rbd.csi.ceph.com Delete Immediate true 79s [root@kube-cy4-kube001 rbd]# cd /home/deploy/ceph-csi/ [root@kube-cy4-kube001 ceph-csi]# kubectl create -f examples/rbd/pvc.yaml [root@kube-cy4-kube001 ceph-csi]# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE rbd-pvc Bound pvc-a33aeeec-e51a-463a-a708-f6ede4dbbc8a 1Gi RWO csi-rbd-sc 3s
ceph cluster에서 rbd명령을 이용하여 해당 이미지를 확인할 수 있습니다.
[root@kube-cy4-kube001 ceph-csi]# rbd ls -p kubernetes csi-vol-91ae5b24-d477-11ea-8fdb-1a270cdb0b8f
Deploy Openstack Helm/Helm infra
openstack helm,helm infra배포를 하기전 배포를 진행하는 노드에서 필요한 패키지를 설치 합니다.
[root@kube-cy4-kube001 ~]# yum install git jq python2-pip gcc python-devel -y
openstack helm,helm infra 프로젝트를 clone합니다.
[root@kube-cy4-kube001 deploy]# git clone https://opendev.org/openstack/openstack-helm.git [root@kube-cy4-kube001 deploy]# git clone https://opendev.org/openstack/openstack-helm-infra.git
각 프로젝트에서 make all 명령을 수행합니다. 수행이 완료되면 앞서 생성한 local helm repository에 각 차트들이 업로드 됩니다.
[root@kube-cy4-kube001 deploy]# cd openstack-helm ; make all [root@kube-cy4-kube001 openstack-helm]# cd ../openstack-helm-infra ; make all [root@kube-cy4-kube001 openstack-helm-infra]# helm search NAME CHART VERSION APP VERSION DESCRIPTION local/aodh 0.1.0 Openstack-Helm Aodh local/barbican 0.1.0 OpenStack-Helm Barbican local/ca-issuer 0.1.0 1.0 Certificate Issuer chart for OSH local/calico 0.1.0 OpenStack-Helm Calico local/ceilometer 0.1.0 OpenStack-Helm Ceilometer local/ceph-client 0.1.0 OpenStack-Helm Ceph Client ...
배포 전 OSH_INFRA_PATH 환경변수에 openstack-helm-infra 프로젝트의 경로를 선언해준다.
[root@kube-cy4-kube001 openstack-helm-infra]# cd /home/deploy/openstack-helm [root@kube-cy4-kube001 openstack-helm-infra]# export OSH_INFRA_PATH="/home/deploy/openstack-helm-infra"
각 노드에 대한 label을 지정합니다. 002~4 노드는 control plane의 역할을 할것입니다.
[root@kube-cy4-kube001 openstack-helm]# kubectl get nodes NAME STATUS ROLES AGE VERSION kube-cy4-kube001 Ready master 15h v1.18.6 kube-cy4-kube002 Ready <none> 15h v1.18.6 kube-cy4-kube003 Ready <none> 15h v1.18.6 kube-cy4-kube004 Ready <none> 15h v1.18.6 [root@kube-cy4-kube001 openstack-helm]# kubectl label node kube-cy4-kube002 openstack-control-plane=enabled [root@kube-cy4-kube001 openstack-helm]# kubectl label node kube-cy4-kube003 openstack-control-plane=enabled [root@kube-cy4-kube001 openstack-helm]# kubectl label node kube-cy4-kube004 openstack-control-plane=enabled
1. ingress
ingress는 openstack component 간의 통신시 도메인 기반으로 통신시 사용 됩니다. 기본 chart의 value에 override하여 helm chart를 배포하기 위하여 배포전 value파일을 만들어 변경사항만 추가하여 배포하도록합니다.
[root@kube-cy4-kube001 openstack-helm]# tee /tmp/ingress-kube-system.yaml << EOF
pod:
replicas:
error_page: 2
deployment:
mode: cluster
type: DaemonSet
network:
host_namespace: true
EOF
kube-system namespace에 ingress 차트를 배포합니다.
[root@kube-cy4-kube001 openstack-helm]# helm upgrade --install ingress-kube-system ${OSH_INFRA_PATH}/ingress --namespace=kube-system --values=/tmp/ingress-kube-system.yaml
[root@kube-cy4-kube001 openstack-helm]# ./tools/deployment/common/wait-for-pods.sh kube-system
openstack namespace에 ingress를 배포합니다.
[root@kube-cy4-kube001 openstack-helm]# tee /tmp/ingress-openstack.yaml << EOF
pod:
replicas:
ingress: 2
error_page: 2
EOF
[root@kube-cy4-kube001 openstack-helm]# helm upgrade --install ingress-openstack ${OSH_INFRA_PATH}/ingress --namespace=openstack --values=/tmp/ingress-openstack.yaml
[root@kube-cy4-kube001 openstack-helm]# ./tools/deployment/common/wait-for-pods.sh openstack
2. MariaDB
MariaDB 차트를 배포합니다. 이때, storage class의 이름을 csi-rbd-sc로 앞서 생성한 storage class이름으로 변경합니다.
[root@kube-cy4-kube001 openstack-helm]# tee /tmp/mariadb.yaml << EOF
pod:
replicas:
server: 3
ingress: 3
volume:
class_name: csi-rbd-sc
size: 5Gi
EOF
[root@kube-cy4-kube001 openstack-helm]# helm upgrade --install mariadb ${OSH_INFRA_PATH}/mariadb --namespace=openstack --values=/tmp/mariadb.yaml
[root@kube-cy4-kube001 openstack-helm]# ./tools/deployment/common/wait-for-pods.sh openstack
3. RabbitMQ
rabbitmq도 mariadb와 동일하게 배포 storage class이름을 수정하여 배포합니다.
[root@kube-cy4-kube001 openstack-helm]# tee /tmp/rabbitmq.yaml << EOF
volume:
size: 10Gi
class_name: csi-rbd-sc
[root@kube-cy4-kube001 openstack-helm]# helm upgrade --install rabbitmq ${OSH_INFRA_PATH}/rabbitmq --namespace=openstack --values=/tmp/rabbitmq.yaml
[root@kube-cy4-kube001 openstack-helm]# ./tools/deployment/common/wait-for-pods.sh openstack
4. Memcached
memcached는 사용할 pod selector를 지정하여 value 파일을 생성뒤 배포합니다.
[root@kube-cy4-kube001 openstack-helm]# tee /tmp/memcached.yaml <<EOF
manifests:
network_policy: true
network_policy:
memcached:
ingress:
- from:
- podSelector:
matchLabels:
application: keystone
- podSelector:
matchLabels:
application: heat
- podSelector:
matchLabels:
application: glance
- podSelector:
matchLabels:
application: cinder
- podSelector:
matchLabels:
application: horizon
- podSelector:
matchLabels:
application: nova
- podSelector:
matchLabels:
application: neutron
ports:
- protocol: TCP
port: 11211
EOF
[root@kube-cy4-kube001 openstack-helm]# helm upgrade --install memcached ${OSH_INFRA_PATH}/memcached --namespace=openstack --values=/tmp/memcached.yaml
[root@kube-cy4-kube001 openstack-helm]# ./tools/deployment/common/wait-for-pods.sh openstack
5. Keystone
keystone도 동일한 방법으로 배포합니다.
[root@kube-cy4-kube001 openstack-helm]# tee /tmp/keystone.yaml << EOF
pod:
replicas:
api: 2
EOF
[root@kube-cy4-kube001 openstack-helm]# helm upgrade --install keystone ./keystone --namespace=openstack --values=/tmp/keystone.yaml
[root@kube-cy4-kube001 openstack-helm]# ./tools/deployment/common/wait-for-pods.sh openstack
openstack 인증정보를 파일로 작성하고 openstack client 컨테이너를 이용하여 keystone이 정상적으로 작동하는지 확인합니다.
client는 host 에 직접 설치하지 않고 아래 방법과 같이 컨테이너를 이용하여 사용합니다. keystone의 ingress도메인은 호스트파일에 worker로 사용되는 노드의 아이피로 설정합니다.
[root@kube-cy4-kube001 openstack-helm-infra]# mkdir -p /etc/openstack [root@kube-cy4-kube001 openstack-helm-infra]# tee /etc/openstack/openrc.env << EOF OS_AUTH_URL=http://keystone.openstack.svc.cluster.local:80/v3 OS_IDENTITY_API_VERSION=3 OS_IMAGE_API_VERSION=2 OS_PROJECT_DOMAIN_NAME=default OS_USER_DOMAIN_NAME=default OS_PROJECT_NAME=admin OS_USERNAME=admin OS_PASSWORD=password EOF [root@kube-cy4-kube001 openstack-helm]# echo "10.4.10.22 keystone.openstack.svc.cluster.local" >> /etc/hosts [root@kube-cy4-kube001 openstack-helm-infra]# docker run -it --network host -v /images:/images --env-file /etc/openstack/openrc.env docker.io/sktdev/openstackclient:stein bash openstackclient@kube-cy4-kube001:~$ openstack endpoint list +----------------------------------+-----------+--------------+--------------+---------+-----------+---------------------------------------------------------+ | ID | Region | Service Name | Service Type | Enabled | Interface | URL | +----------------------------------+-----------+--------------+--------------+---------+-----------+---------------------------------------------------------+ | 0b33f6a61fdb4860b49ab2278e6ff50c | RegionOne | keystone | identity | True | internal | http://keystone-api.openstack.svc.cluster.local:5000/v3 | | 24103bb6eacb403facc31812019e6fbf | RegionOne | keystone | identity | True | public | http://keystone.openstack.svc.cluster.local/v3 | | 52edf255656c421f978bea28fd22f023 | RegionOne | keystone | identity | True | admin | http://keystone.openstack.svc.cluster.local/v3 | +----------------------------------+-----------+--------------+--------------+---------+-----------+---------------------------------------------------------+
6. Glance
glance는 ceph의 rbd pool을 자동적으로 생성하여 사용 됩니다. 그러기 위해서는 glance에서 ceph cluster에 접속할 정보가 있는 ceph.conf 파일을 configmap형태로 필요하기 때문에 config map을 생성합니다. 이때, ceph의 monitor버전이 nautilus버전 부터 v1,v2를 선택적으로 사용할 수 있도록 리스트 형태로 생성되지만 glance에서 해당 버전을 인식하지 못하기 때문에 아래와 같이 버전을 v1만 명시 합니다.
[root@kube-cy4-kube001 openstack-helm]# vi /etc/ceph/ceph.conf ... # Please do not change this file directly since it is managed by Ansible and will be overwritten [global] cluster network = 10.4.30.0/24 fsid = f9b17cb6-b38c-455b-b10d-5c44d7bcc36b #mon host = [v2:10.4.20.21:3300,v1:10.4.20.21:6789] mon host = 10.4.20.21:6789 ... [root@kube-cy4-kube001 openstack-helm]# kubectl create configmap ceph-etc -n openstack --from-file=/etc/ceph/ceph.conf
ceph의 admin 인증정보가 필요하기 때문에 admin keyring정보를 추가한뒤 glance를 배포 합니다.
[root@kube-cy4-kube001 openstack-helm]# ceph auth get client.admin | grep key
exported keyring for client.admin
key = AQBgGCVfjOayKBAAT4iPx2CSDEMU60aSQtgBXg==
[root@kube-cy4-kube001 openstack-helm]# tee /tmp/glance.yaml << EOF
storage: rbd
pod:
replicas:
api: 2
registry: 2
conf:
ceph:
enabled: true
admin_keyring: AQBgGCVfjOayKBAAT4iPx2CSDEMU60aSQtgBXg==
glance:
DEFAULT:
enable_v1_api: true
enable_v2_registry: true
EOF
[root@kube-cy4-kube001 openstack-helm]# helm upgrade --install glance ./glance --namespace=openstack --values=/tmp/glance.yaml
[root@kube-cy4-kube001 openstack-helm]# ./tools/deployment/common/wait-for-pods.sh openstack
정상적으로 pod가 배포 된것을 확인 합니다.
[root@kube-cy4-kube001 openstack-helm]#kubectl get pod -n openstack | grep glance glance-api-ff94f9577-ph6fx 1/1 Running 0 2m50s glance-api-ff94f9577-scs69 1/1 Running 0 2m50s glance-bootstrap-csjd4 0/1 Completed 0 2m49s glance-db-init-8lfws 0/1 Completed 0 2m50s glance-db-sync-24t8f 0/1 Completed 0 2m50s glance-ks-endpoints-fjczv 0/3 Completed 0 2m50s glance-ks-service-d59gp 0/1 Completed 0 2m50s glance-ks-user-q2jv6 0/1 Completed 0 2m50s glance-metadefs-load-tgtwn 0/1 Completed 0 2m50s glance-rabbit-init-sq4k4 0/1 Completed 0 2m50s glance-storage-init-d68nf 0/1 Completed 0 2m50s
glance ingress 도메인을 hosts파일에 추가후 openstack client에 접속하면 bootstrap과정에서 업로드된 cirros이미지를 확인할 수 있습니다.
[root@kube-cy4-kube001 openstack-helm]# echo "10.4.10.22 glance.openstack.svc.cluster.local" >> /etc/hosts [root@kube-cy4-kube001 openstack-helm]# docker run -it --network host -v /images:/images --env-file /etc/openstack/openrc.env docker.io/sktdev/openstackclient:stein bash openstackclient@kube-cy4-kube001:~$ openstack image list +--------------------------------------+---------------------+--------+ | ID | Name | Status | +--------------------------------------+---------------------+--------+ | 8869f634-9f67-4990-9e9a-84c110d816f4 | Cirros 0.3.5 64-bit | active | +--------------------------------------+---------------------+--------+ openstackclient@kube-cy4-kube001:~$
7. Cinder
cinder는 glance와 동일하게 ceph의 rbd pool을 사용하기 때문에 admin keyring정보를 추가한뒤 배포 합니다.
기본적으로 cinder volume과 cinder volume backup 두가지 서비스를 위한 pool을 각각 ceph의 pool로 자동 생성합니다.
[root@kube-cy4-kube001 openstack-helm]# ceph auth get client.admin | grep key
exported keyring for client.admin
key = AQBgGCVfjOayKBAAT4iPx2CSDEMU60aSQtgBXg==
[root@kube-cy4-kube001 openstack-helm]# tee /tmp/cinder.yaml << EOF
pod:
replicas:
api: 2
volume: 1
scheduler: 1
backup: 1
conf:
ceph:
admin_keyring: AQBgGCVfjOayKBAAT4iPx2CSDEMU60aSQtgBXg==
enabled: true
cinder:
DEFAULT:
backup_driver: cinder.backup.drivers.ceph.CephBackupDriver
EOF
[root@kube-cy4-kube001 openstack-helm]# helm upgrade --install cinder ./cinder --namespace=openstack --values=/tmp/cinder.yaml
[root@kube-cy4-kube001 openstack-helm]# ./tools/deployment/common/wait-for-pods.sh openstack
완료되면 cinder의 pod를 확인할 수 있습니다.
[root@kube-cy4-kube001 openstack-helm]# kubectl get pod -n openstack | grep cinder cinder-api-64f59cbcb-5jjzq 1/1 Running 0 112s cinder-api-64f59cbcb-jsjjp 1/1 Running 0 112s cinder-backup-6c47fff559-2w2xm 1/1 Running 0 112s cinder-backup-storage-init-cjlb4 0/1 Completed 0 112s cinder-bootstrap-h7bbj 0/1 Completed 0 112s cinder-create-internal-tenant-52s8p 0/1 Completed 0 112s cinder-db-init-6gpws 0/1 Completed 0 113s cinder-db-sync-xt9kq 0/1 Completed 0 113s cinder-ks-endpoints-mqb9c 0/9 Completed 0 112s cinder-ks-service-d4bdf 0/3 Completed 0 113s cinder-ks-user-jx8wn 0/1 Completed 0 113s cinder-rabbit-init-x7659 0/1 Completed 0 113s cinder-scheduler-f8b98c7b4-p42jm 1/1 Running 0 113s cinder-storage-init-6rz8c 0/1 Completed 0 113s cinder-volume-5d67df7bdd-sq2hx 1/1 Running 0 112s
cinder ingress 도메인 정보를 hosts파일에 추가후 openstack client를 통하여 볼륨 서비스 확인 및 테스트 볼륨을 생성하여 확인 합니다.
[root@kube-cy4-kube001 openstack-helm]# echo "10.4.10.22 cinder.openstack.svc.cluster.local" >> /etc/hosts [root@kube-cy4-kube001 openstack-helm]# docker run -it --network host -v /images:/images --env-file /etc/openstack/openrc.env docker.io/sktdev/openstackclient:stein bash openstackclient@kube-cy4-kube001:~$ openstack voluem lki openstackclient@kube-cy4-kube001:~$ openstack volume service list +------------------+---------------------------+------+---------+-------+----------------------------+ | Binary | Host | Zone | Status | State | Updated At | +------------------+---------------------------+------+---------+-------+----------------------------+ | cinder-scheduler | cinder-volume-worker | nova | enabled | up | 2020-08-02T05:56:41.000000 | | cinder-backup | cinder-volume-worker | nova | enabled | up | 2020-08-02T05:56:40.000000 | | cinder-volume | cinder-volume-worker@rbd1 | nova | enabled | up | 2020-08-02T05:56:41.000000 | +------------------+---------------------------+------+---------+-------+----------------------------+ openstackclient@kube-cy4-kube001:~$ openstack volume create --size 1 test openstackclient@kube-cy4-kube001:~$ cinder list +--------------------------------------+-----------+------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+------+------+-------------+----------+-------------+ | d47b5120-3d57-465f-aeb2-c655aceb565a | available | test | 1 | rbd1 | false | | +--------------------------------------+-----------+------+------+-------------+----------+-------------+
8. Openvswitch
openvswitch 차트는 daemonset으로 “openvswitch=enabled ” label이 설정된 노드에만 배포 됩니다. worker노드에 openvswitch label을 설정 합니다.
[root@kube-cy4-kube001 openstack-helm]# kubectl label node kube-cy4-kube002 openvswitch=enabled node/kube-cy4-kube002 labeled [root@kube-cy4-kube001 openstack-helm]# kubectl label node kube-cy4-kube003 openvswitch=enabled node/kube-cy4-kube003 labeled [root@kube-cy4-kube001 openstack-helm]# kubectl label node kube-cy4-kube004 openvswitch=enabled node/kube-cy4-kube004 labeled
openvswitch 차트를 배포 합니다.
[root@kube-cy4-kube001 openstack-helm]# helm upgrade --install openvswitch ${OSH_INFRA_PATH}/openvswitch --namespace=openstack
배포가 완료된 openvswitch pod를 확인 합니다.
[root@kube-cy4-kube001 openstack-helm]# kubectl get pod -n openstack | grep openv openvswitch-db-8llk2 1/1 Running 0 3m29s openvswitch-db-gw9w5 1/1 Running 0 3m33s openvswitch-db-q86zr 1/1 Running 0 3m37s openvswitch-vswitchd-2chg8 1/1 Running 0 3m37s openvswitch-vswitchd-lvntw 1/1 Running 0 3m29s openvswitch-vswitchd-vdwmx 1/1 Running 0 3m33s
9. Libvirt,Neutron,Nova
Libvirt,Neutron,Nova 차트는 서로 의존성이 있기 때문에 모두 정상적으로 동작해야 차트 배포가 완료 됩니다.
각 노드에 label을 지정합니다.003,004노드는 compute노드로 사용 됩니다.
[root@kube-cy4-kube001 openstack-helm]# kubectl label node kube-cy4-kube002 openstack-helm-node-class=primary node/kube-cy4-kube002 labeled [root@kube-cy4-kube001 openstack-helm]# kubectl label node kube-cy4-kube003 openstack-compute-node=enabled node/kube-cy4-kube003 labeled [root@kube-cy4-kube001 openstack-helm]# kubectl label node kube-cy4-kube004 openstack-compute-node=enabled node/kube-cy4-kube004 labeled
ceph admin과 cinder 유저 인증 정보를 추가한뒤 libvirt 차트를 배포 합니다. 아직 다른 component가 정상적으로 안올라와서 pod가 생성되지 않는다.
[root@kube-cy4-kube001 openstack-helm]# ceph auth get client.admin | grep key
exported keyring for client.admin
key = AQBgGCVfjOayKBAAT4iPx2CSDEMU60aSQtgBXg==
[root@kube-cy4-kube001 openstack-helm]# ceph auth get client.cinder | grep key
exported keyring for client.cinder
key = AQDHVCZfithVDBAALjJxP9UZob3Y0IC3KhGsrA==
[root@kube-cy4-kube001 openstack-helm]# tee /tmp/libvirt.yaml << EOF
network:
backend:
- openvswitch
conf:
ceph:
enabled: true
admin_keyring: AQBgGCVfjOayKBAAT4iPx2CSDEMU60aSQtgBXg==
cinder:
keyring: AQDHVCZfithVDBAALjJxP9UZob3Y0IC3KhGsrA==
secret_uuid: 582393ff-9a5c-4a2e-ae0d-86ec18c36afc
EOF
[root@kube-cy4-kube001 openstack-helm]# helm upgrade --install libvirt ${OSH_INFRA_PATH}/libvirt --namespace=openstack --values=/tmp/libvirt.yaml
[root@kube-cy4-kube001 openstack-helm]# kubectl get pod -n openstack | grep libvirt
libvirt-libvirt-default-4vxp5 0/1 Init:0/3 0 27s
libvirt-libvirt-default-5spwb 0/1 Init:0/3 0 27s
nova에서 ceph의 admin정보와 cinder 인정 정보를 차트에 추가 해주며 , virt type을 qemu로 설정 하여 배포 합니다.
[root@kube-cy4-kube001 openstack-helm]# ceph auth get client.admin | grep key
exported keyring for client.admin
key = AQBgGCVfjOayKBAAT4iPx2CSDEMU60aSQtgBXg==
[root@kube-cy4-kube001 openstack-helm]# ceph auth get client.cinder | grep key
exported keyring for client.cinder
key = AQDHVCZfithVDBAALjJxP9UZob3Y0IC3KhGsrA==
[root@kube-cy4-kube001 openstack-helm]# tee /tmp/nova.yaml << EOF
labels:
api_metadata:
node_selector_key: openstack-helm-node-class
node_selector_value: primary
conf:
ceph:
enabled: true
admin_keyring: AQBgGCVfjOayKBAAT4iPx2CSDEMU60aSQtgBXg==
cinder:
user: cinder
keyring: AQDHVCZfithVDBAALjJxP9UZob3Y0IC3KhGsrA==
nova:
libvirt:
images_type: rbd
rbd_user: cinder
rbd_secret_uuid: 582393ff-9a5c-4a2e-ae0d-86ec18c36afc
virt_type: qemu
pod:
replicas:
api_metadata: 1
placement: 2
osapi: 2
conductor: 2
consoleauth: 2
scheduler: 1
novncproxy: 1
EOF
[root@kube-cy4-kube001 openstack-helm]# helm upgrade --install nova ./nova --namespace=openstack --values=/tmp/nova.yaml
앞서 이야기 한것 처럼 compute노드들의 eth3인터페이스는 tanent 네트워크로 사용 하기 때문에 명시 하며, eth4 인터페이스는 자동으로 ovs 브릿지를 만들기 위해서 auto_bridge_add 변수에 추가 합니다.br-ex로 만든 인터페이스는 provider 이름의 flat network로 floating ip를 사용하기 위해서 생성 됩니다.
[root@kube-cy4-kube001 openstack-helm]# tee /tmp/neutron.yaml << EOF
network:
interface:
tunnel: eth3
pod:
replicas:
server: 1
conf:
auto_bridge_add:
br-ex: eth4
neutron:
DEFAULT:
l3_ha: False
max_l3_agents_per_router: 1
l3_ha_network_type: vxlan
dhcp_agents_per_network: 1
plugins:
ml2_conf:
ml2_type_flat:
flat_networks: provider
openvswitch_agent:
agent:
tunnel_types: vxlan
l2_population: True
arp_responder: True
ovs:
bridge_mappings: provider:br-ex
EOF
[root@kube-cy4-kube001 openstack-helm]# helm upgrade --install neutron ./neutron --namespace=openstack --values=/tmp/neutron.yaml
모든 pod와 서비스 job들이 종료 될때 까지 스크립트를 통하여 대기 합니다.
[root@kube-cy4-kube001 openstack-helm]# ./tools/deployment/common/wait-for-pods.sh openstack
정상적으로 종료 되었습니다면 client에 접속 하여 nova, neutron서비스를 확인 합니다.
[root@kube-cy4-kube001 openstack-helm]# echo "10.4.10.22 nova.openstack.svc.cluster.local" >> /etc/hosts 10.4.10.22 nova.openstack.svc.cluster.local /etc/hosts [root@kube-cy4-kube001 openstack-helm]# echo "10.4.10.22 neutron.openstack.svc.cluster.local" >> /etc/hosts [root@kube-cy4-kube001 openstack-helm]# docker run -it --network host -v /images:/images --env-file /etc/openstack/openrc.env docker.io/sktdev/openstackclient:stein bash openstackclient@kube-cy4-kube001:~$ openstack compute service list +----+------------------+-----------------------------------+----------+---------+-------+----------------------------+ | ID | Binary | Host | Zone | Status | State | Updated At | +----+------------------+-----------------------------------+----------+---------+-------+----------------------------+ | 34 | nova-consoleauth | nova-consoleauth-5468477744-qlr5d | internal | enabled | up | 2020-08-02T07:10:37.000000 | | 37 | nova-consoleauth | nova-consoleauth-5468477744-d27wr | internal | enabled | up | 2020-08-02T07:10:37.000000 | | 40 | nova-conductor | nova-conductor-54f649d6bd-nznqv | internal | enabled | up | 2020-08-02T07:10:38.000000 | | 43 | nova-scheduler | nova-scheduler-c5f45fb88-whbr5 | internal | enabled | up | 2020-08-02T07:10:29.000000 | | 58 | nova-conductor | nova-conductor-54f649d6bd-9w5hg | internal | enabled | up | 2020-08-02T07:10:29.000000 | | 61 | nova-compute | kube-cy4-kube004 | nova | enabled | up | 2020-08-02T07:10:38.000000 | | 64 | nova-compute | kube-cy4-kube003 | nova | enabled | up | 2020-08-02T07:10:37.000000 | +----+------------------+-----------------------------------+----------+---------+-------+----------------------------+ openstackclient@kube-cy4-kube001:~$ openstack network agent list +--------------------------------------+--------------------+------------------+-------------------+-------+-------+---------------------------+ | ID | Agent Type | Host | Availability Zone | Alive | State | Binary | +--------------------------------------+--------------------+------------------+-------------------+-------+-------+---------------------------+ | 261a37c4-58fc-4512-aafc-81bba3519003 | Metadata agent | kube-cy4-kube004 | None | :-) | UP | neutron-metadata-agent | | 2f015c71-9243-4774-bb2a-5d0d070ef4f3 | Open vSwitch agent | kube-cy4-kube004 | None | :-) | UP | neutron-openvswitch-agent | | 39f2dcf4-fbf3-46cd-b712-13d808b38dd6 | L3 agent | kube-cy4-kube002 | nova | :-) | UP | neutron-l3-agent | | 4a1266f9-0182-462b-9e8f-3424337483f7 | DHCP agent | kube-cy4-kube002 | nova | :-) | UP | neutron-dhcp-agent | | 4e1bac9f-577a-48d2-b0f7-f981cad85440 | DHCP agent | kube-cy4-kube003 | nova | :-) | UP | neutron-dhcp-agent | | 675ee208-2f49-4b58-9540-8de865fb3865 | Open vSwitch agent | kube-cy4-kube003 | None | :-) | UP | neutron-openvswitch-agent | | 7d6056bf-9dbb-4e55-99b4-84a056042449 | Open vSwitch agent | kube-cy4-kube002 | None | :-) | UP | neutron-openvswitch-agent | | 8ba71881-7367-4874-a41a-46f8d81cd0c2 | Metadata agent | kube-cy4-kube003 | None | :-) | UP | neutron-metadata-agent | | 97c7da9e-1a12-4cef-bbdf-e4c021b1345d | DHCP agent | kube-cy4-kube004 | nova | :-) | UP | neutron-dhcp-agent | | d0a5085e-d3a4-408c-bab8-a458d32d047b | Metadata agent | kube-cy4-kube002 | None | :-) | UP | neutron-metadata-agent | | d856ab20-547e-481f-857f-50a0b7a87e87 | L3 agent | kube-cy4-kube003 | nova | :-) | UP | neutron-l3-agent | | decd265a-9ea0-41a4-9516-c7467f2d7cad | L3 agent | kube-cy4-kube004 | nova | :-) | UP | neutron-l3-agent | +--------------------------------------+--------------------+------------------+-------------------+-------+-------+---------------------------+
openstack client에서 기본 network 생성하고 , 인스턴스를 생성해봅니다.
openstackclient@kube-cy4-kube001:~$ openstack network create --share --external \ --provider-physical-network provider \ --provider-network-type flat provider openstackclient@kube-cy4-kube001:~$ o openstack subnet create --network provider \ --allocation-pool start=192.168.193.210,end=192.168.193.240 \ --dns-nameserver 8.8.4.4 --gateway 192.168.0.1 \ --subnet-range 192.168.0.1/16 provider openstackclient@kube-cy4-kube001:~$ o openstack network create selfservice openstackclient@kube-cy4-kube001:~$ o openstack subnet create --network selfservice \ --dns-nameserver 8.8.4.4 --gateway 11.11.1.1 \ --subnet-range 11.11.1.0/24 selfservice openstackclient@kube-cy4-kube001:~$ o openstack router create router openstackclient@kube-cy4-kube001:~$ o neutron router-interface-add router selfservice openstackclient@kube-cy4-kube001:~$ o neutron router-gateway-set router provider openstackclient@kube-cy4-kube001:~$ openstack network list +--------------------------------------+-------------+--------------------------------------+ | ID | Name | Subnets | +--------------------------------------+-------------+--------------------------------------+ | 3e37ecae-fed8-432d-a7ca-0de991623717 | provider | 360e99c6-5bdc-43e3-8275-3336a0d6ef80 | | 9364f2bb-58ea-4ce5-a867-308a0115e3ba | selfservice | 69079fee-decb-41d6-9da2-f2cfca4cc9ca | +--------------------------------------+-------------+--------------------------------------+ openstackclient@kube-cy4-kube001:~$ openstack flavor list +--------------------------------------+-----------+-------+------+-----------+-------+-----------+ | ID | Name | RAM | Disk | Ephemeral | VCPUs | Is Public | +--------------------------------------+-----------+-------+------+-----------+-------+-----------+ | 0a866d33-ad39-45c7-8461-e90b21d37524 | m1.large | 8192 | 80 | 0 | 4 | True | | 17b234fc-ff37-493e-a96a-02df7e4cf574 | m1.tiny | 512 | 1 | 0 | 1 | True | | 401af6df-2c9a-4771-803d-f847b4c37d33 | m1.medium | 4096 | 40 | 0 | 2 | True | | 7ffcb940-fd02-46e9-9d63-9556210b31d1 | m1.xlarge | 16384 | 160 | 0 | 8 | True | | fe9146fa-a62c-41c6-a45c-02931fdedc5a | m1.small | 2048 | 20 | 0 | 1 | True | +--------------------------------------+-----------+-------+------+-----------+-------+-----------+ openstackclient@kube-cy4-kube001:~$ openstack security group list +--------------------------------------+---------+------------------------+----------------------------------+------+ | ID | Name | Description | Project | Tags | +--------------------------------------+---------+------------------------+----------------------------------+------+ | 8c210974-5d4c-4a8a-ac62-669846fb7ded | default | Default security group | d24347196d1a42999290eadba5c51151 | [] | | ad3441b9-eb4e-475a-a979-517ef556936c | default | Default security group | | [] | +--------------------------------------+---------+------------------------+----------------------------------+------+ openstackclient@kube-cy4-kube001:~$ openstack security group rule create --ingress --dst-port 22 8c210974-5d4c-4a8a-ac62-669846fb7ded openstackclient@kube-cy4-kube001:~$ openstack image list +--------------------------------------+---------------------+--------+ | ID | Name | Status | +--------------------------------------+---------------------+--------+ | 8869f634-9f67-4990-9e9a-84c110d816f4 | Cirros 0.3.5 64-bit | active | +--------------------------------------+---------------------+--------+ openstackclient@kube-cy4-kube001:~$ ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/home/openstackclient/.ssh/id_rsa): Created directory '/home/openstackclient/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/openstackclient/.ssh/id_rsa. Your public key has been saved in /home/openstackclient/.ssh/id_rsa.pub. The key fingerprint is: SHA256:rmKh6dug9CYW0bCIIcuWjSPLhrcD+woRDLYJoPmm8m0 openstackclient@kube-cy4-kube001 The key's randomart image is: +---[RSA 2048]----+ |=. | |Ooo | |O*B | |=@ o | |*.= S | |+B. . . | |==o+ . . | |==*=E . | |++B*o.. | +----[SHA256]-----+ openstackclient@kube-cy4-kube001:~$ openstack server create --image 8869f634-9f67-4990-9e9a-84c110d816f4 --security-group 8c210974-5d4c-4a8a-ac62-669846fb7ded --flavor m1.tiny --key-name admin_client_key --network 9364f2bb-58ea-4ce5-a867-308a0115e3ba test-cirros-vm
provider network 에서 floating ip 생성뒤 앞서 생성한 인스턴스에 붙이고 ssh를 이용하여 접속 합니다.
openstackclient@kube-cy4-kube001:~$ openstack floating ip create provider
+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| created_at | 2020-08-02T07:19:31Z |
| description | |
| dns_domain | None |
| dns_name | None |
| fixed_ip_address | None |
| floating_ip_address | 192.168.193.235 |
| floating_network_id | 3e37ecae-fed8-432d-a7ca-0de991623717 |
| id | 7cfd6a27-4bfb-46fa-b32b-2ce5c0c021e5 |
| location | Munch({'cloud': '', 'region_name': '', 'zone': None, 'project': Munch({'id': 'd24347196d1a42999290eadba5c51151', 'name': 'admin', 'domain_id': None, 'domain_name': 'default'})}) |
| name | 192.168.193.235 |
| port_details | None |
| port_id | None |
| project_id | d24347196d1a42999290eadba5c51151 |
| qos_policy_id | None |
| revision_number | 0 |
| router_id | None |
| status | DOWN |
| subnet_id | None |
| tags | [] |
| updated_at | 2020-08-02T07:19:31Z |
+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
openstackclient@kube-cy4-kube001:~$ openstack server add floating ip test-cirros-vm 192.168.193.235
openstackclient@kube-cy4-kube001:~$ ssh cirros@192.168.193.235
The authenticity of host '192.168.193.235 (192.168.193.235)' can't be established.
RSA key fingerprint is SHA256:45KMfL6+lSzqdN2fLLkd9vvxnfvfUg+h0kZUFF411uY.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.193.235' (RSA) to the list of known hosts.
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast qlen 1000
link/ether fa:16:3e:8b:86:9f brd ff:ff:ff:ff:ff:ff
inet 11.11.1.198/24 brd 11.11.1.255 scope global eth0
inet6 fe80::f816:3eff:fe8b:869f/64 scope link
valid_lft forever preferred_lft forever
10. Horizon
horizon 배포전 브라우저를 통하여 접속 하기 위하여 node port설정이 필요 하다. 31000포트로 지정 하여 loca_settings 변수에 필요한 기능만 활성화 한뒤 배포를 합니다.
[root@kube-cy4-kube001 openstack-helm]# tee /tmp/horizon.yaml << EOF
network:
node_port:
enabled: true
port: 31000
conf:
horizon:
local_settings:
config:
openstack_neutron_network:
enable_router: "True"
enable_quotas: "True"
enable_ipv6: "False"
enable_ha_router: "True"
enable_lb: "True"
enable_firewall: "False"
enable_vpn: "False"
enable_fip_topology_check: "True"
EOF
[root@kube-cy4-kube001 openstack-helm]# helm upgrade --install horizon ./horizon --namespace=openstack --values=/tmp/horizon.yaml
[root@kube-cy4-kube001 openstack-helm]# ./tools/deployment/common/wait-for-pods.sh openstack
배포가 완료되면 , http://{worker노드 ip}:31000 를 브라우저를 통하여 접속 확인 합니다.
앞서 생성한 인스턴스를 확인할 수 있습니다.
참고 사이트
오픈소스컨설팅의 마스코트, 열린이입니다! :)
