Openshift 의 HPA 를 이용한 Auto-Scaling 구현
이번 포스팅에서는 Horizontal Pod Autoscaler 에 대해 알아보도록 하겠습니다.
Openshift 에서 Horizontal Pod Autoscaler(이하 HPA ) 를 이용하여 설정한 CPU 사용률을 기반으로 Replicaset , Deployment 의 Pod 수를 자동으로 Scaling 할 수 있습니다.
HPA 를 하기 위해서는 Pod 의 부하에 대해 모니터링 및 수집을 하는 Metrics-Server 가 필요합니다.
아래에서 Openshift 에 Metrics-Server 를 배포하고 성능 수집을 해보도록 하겠습니다.
Metrics-Server 배포
기본적으로 Openshift 설치 과정에서는 metric-server 는 기본 배포로 설정이 안되어 있습니다.
추가로 설치를 진행 해야됩니다.
아래와 같이 이전 포스팅에서 사용한 Openshift Playbook 을 이용합니다.
# ansible-playbook -i inventory/hosts.localhost /usr/share/ansible/openshift-ansible/playbooks/metrics-server/config.yml -e openshift_metrics_server_install=true
... < 중략 > ...
PLAY RECAP ********************************************************************************************************************************************************************************************************************************************
localhost : ok=119 changed=13 unreachable=0 failed=0
INSTALLER STATUS **************************************************************************************************************************************************************************************************************************************
Initialization : Complete (0:00:59)
metrics-server Install : Complete (0:00:56)
Thursday 06 June 2019 13:28:05 +0200 (0:00:00.105) 0:01:56.691 *********
===============================================================================
Run variable sanity checks -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 23.51s
Gathering Facts -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 4.03s
Gather Cluster facts --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 3.00s
metrics_server : Ensure metrics-server namespace is present ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2.80s
metrics_server : slurp ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.44s
Initialize openshift.node.sdn_mtu -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.28s
get openshift_current_version ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2.09s
metrics_server : generate metrics-server keys -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.64s
metrics_server : generate metrics-server secret template --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.26s
metrics_server : Applying /tmp/openshift-metrics-server-ansible-56a8WR/templates/metrics-server-certs.yaml ------------------------------------------------------------------------------------------------------------------------------------- 1.17s
metrics_server : Applying /tmp/openshift-metrics-server-ansible-56a8WR/templates/metrics-server-sa.yaml ---------------------------------------------------------------------------------------------------------------------------------------- 1.14s
metrics_server : Applying /tmp/openshift-metrics-server-ansible-56a8WR/templates/metrics-server-resource-reader-rolebinding.yaml --------------------------------------------------------------------------------------------------------------- 1.14s
metrics_server : Applying /tmp/openshift-metrics-server-ansible-56a8WR/templates/metrics-server-service.yaml ----------------------------------------------------------------------------------------------------------------------------------- 1.13s
metrics_server : generate ca certificate chain ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.13s
metrics_server : Applying /tmp/openshift-metrics-server-ansible-56a8WR/templates/metrics-server-auth-delegator-rolebinding.yaml ---------------------------------------------------------------------------------------------------------------- 1.12s
metrics_server : Applying /tmp/openshift-metrics-server-ansible-56a8WR/templates/extension-apiserver-authentication-reader-metrics-server-rolebinding.yaml ------------------------------------------------------------------------------------- 1.09s
metrics_server : Applying /tmp/openshift-metrics-server-ansible-56a8WR/templates/metrics-server-apiservice.yaml -------------------------------------------------------------------------------------------------------------------------------- 1.08s
metrics_server : Applying /tmp/openshift-metrics-server-ansible-56a8WR/templates/metrics-server-deployment.yaml -------------------------------------------------------------------------------------------------------------------------------- 1.06s
metrics_server : Checking generation of Service metrics-server --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.03s
metrics_server : Determine change status of ClusterRoleBinding system:metrics-server ----------------------------------------------------------------------------------------------------------------------------------------------------------- 1.03s
[root@master openshift-ansible]#
위와 같이 배포가 완료되면, oc 명령을 통해 metrics-server 가 정상 작동하는지 확인 할 수 있습니다.
[root@master ~]# oc adm top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
master.example.com 907m 11% 6146Mi 38%
//openshift-metrics-server namespaces 에 metrics-server 가 배포 된 것을 확인 할 수 있습니다.
[root@master ~]# oc adm top pod --all-namespaces
NAMESPACE NAME CPU(cores) MEMORY(bytes)
default docker-registry-1-9t2zp 2m 32Mi
default registry-console-1-n4v9t 0m 1Mi
default router-1-58cpz 4m 45Mi
kube-service-catalog apiserver-pnsnf 1m 51Mi
kube-service-catalog controller-manager-qsq5m 10m 22Mi
kube-system master-api-master.example.com 96m 1061Mi
kube-system master-controllers-master.example.com 99m 532Mi
kube-system master-etcd-master.example.com 36m 408Mi
openshift-ansible-service-broker asb-1-k9nww 2m 19Mi
openshift-console console-5677c7c58d-n5z5h 2m 9Mi
openshift-metrics-server metrics-server-7bf4cf7dd4-svfs7 1m 30Mi
openshift-monitoring alertmanager-main-0 4m 25Mi
openshift-monitoring alertmanager-main-1 7m 25Mi
openshift-monitoring alertmanager-main-2 6m 26Mi
openshift-monitoring cluster-monitoring-operator-6465f8fbc7-f4w4t 0m 35Mi
openshift-monitoring grafana-6b9f85786f-z9gpj 6m 42Mi
openshift-monitoring kube-state-metrics-7449d589bc-qsrvh 7m 67Mi
openshift-monitoring node-exporter-hjh46 2m 21Mi
openshift-monitoring prometheus-k8s-0 64m 495Mi
openshift-monitoring prometheus-k8s-1 47m 485Mi
openshift-monitoring prometheus-operator-6644b8cd54-cv2hf 0m 25Mi
openshift-node sync-fnd6m 0m 7Mi
openshift-sdn ovs-ssckw 5m 80Mi
openshift-sdn sdn-77shb 12m 37Mi
openshift-template-service-broker apiserver-2mpk8 4m 30Mi
openshift-web-console webconsole-7df4f9f689-mxfkj 15m 25Mi
sample-project chhanz-hello-example 0m 0Mi
sample-project http-example-1-5dhj5 3m 9Mi
sample-project sampleapp-2-hbmzs 7m 222Mi
test-project load-generator-779c5f458c-g4kff 0m 0Mi
test-project mysample-3-cr8cp 1m 215Mi
test-project mysample-3-dvrlp 1m 206Mi
test-project mysample-3-npgq7 1m 228Mi
test-project mysample-3-t9dwg 1m 214Mi
test-project mysample-3-vtm5j 1m 217Mi
test-project nginx-1-rjd86 0m 1Mi
test-project php-apache-b5b7bd9c8-hpcw8 0m 9Mi
test-project ruby-ex-1-nkfxn 1m 83Mi
[root@master ~]#
참고로 $ oc top nodes 명령을 통해 성능 수집이 정상적으로 동작하려면 배포 후, 일정 시간이 지나야 됩니다.
HPA 구현
HPA 를 이용한 Auto-Scaling 을 구현하기 위해 아래와 같이 신규 Project 를 생성합니다.
[root@master ~]# oc new-project hpa-project
Now using project "hpa-project" on server "https://master.example.com:8443".
You can add applications to this project with the 'new-app' command. For example, try:
oc new-app centos/ruby-25-centos7~https://github.com/sclorg/ruby-ex.git
to build a new example application in Ruby.
[root@master ~]#
[root@master ~]# oc get po
No resources found.
[root@master ~]#
[root@master ~]#
부하(load)를 감지하고 응답 할 수 있는 App 을 생성합니다.
Dockerfile 생성 및 Build
# ls
Dockerfile index.php
# cat Dockerfile
FROM php:5-apache
ADD index.php /var/www/html/index.php
RUN chmod a+rx index.php
# cat index.php
<?php
$x = 0.0001;
for ($i = 0; $i <= 1000000; $i++) {
$x += sqrt($x);
}
echo "OK!";
?>
# docker build -t hpa-example .
Sending build context to Docker daemon 3.072kB
Step 1/3 : FROM php:5-apache
5-apache: Pulling from library/php
5e6ec7f28fb7: Pull complete
cf165947b5b7: Pull complete
7bd37682846d: Pull complete
99daf8e838e1: Pull complete
ae320713efba: Pull complete
ebcb99c48d8c: Pull complete
9867e71b4ab6: Pull complete
936eb418164a: Pull complete
bc298e7adaf7: Pull complete
ccd61b587bcd: Pull complete
b2d4b347f67c: Pull complete
56e9dde34152: Pull complete
9ad99b17eb78: Pull complete
Digest: sha256:0a40fd273961b99d8afe69a61a68c73c04bc0caa9de384d3b2dd9e7986eec86d
Status: Downloaded newer image for php:5-apache
---> 24c791995c1e
Step 2/3 : ADD index.php /var/www/html/index.php
---> bcc3ff35ceb8
Step 3/3 : RUN chmod a+rx index.php
---> Running in d1e173fb27a3
Removing intermediate container d1e173fb27a3
---> f4bb43246866
Successfully built f4bb43246866
Successfully tagged hpa-example:latest
# docker push han0495/hpa-example
The push refers to repository [docker.io/han0495/hpa-example]
b42dc42a2d18: Pushed
056e15bb815c: Pushed
1aab22401f12: Mounted from library/php
13ab94c9aa15: Mounted from library/php
588ee8a7eeec: Mounted from library/php
bebcda512a6d: Mounted from library/php
5ce59bfe8a3a: Mounted from library/php
d89c229e40ae: Mounted from library/php
9311481e1bdc: Mounted from library/php
4dd88f8a7689: Mounted from library/php
b1841504f6c8: Mounted from library/php
6eb3cfd4ad9e: Mounted from library/php
82bded2c3a7c: Mounted from library/php
b87a266e6a9c: Mounted from library/php
3c816b4ead84: Mounted from library/php
latest: digest: sha256:07c8ebfbe5b8084b878d0ed4ebafe5baad124b0b932e4f56d81480fbf715f03f size: 3449
App 배포
php 로 만들어진 App 을 Openshift 에 배포합니다.
[root@master ~]# oc login -u admin
Logged into "https://master.example.com:8443" as "admin" using existing credentials.
You have access to the following projects and can switch between them with 'oc project <projectname>':
default
* hpa-project
kube-public
kube-service-catalog
kube-system
management-infra
openshift
openshift-ansible-service-broker
openshift-console
openshift-infra
openshift-logging
openshift-metrics-server
openshift-monitoring
openshift-node
openshift-sdn
openshift-template-service-broker
openshift-web-console
sample-project
test-project
Using project "hpa-project".
[root@master ~]# oc run hpa-example-pod --image=han0495/hpa-example --expose --port 80 --requests=cpu=200m
service/hpa-example-pod created
deploymentconfig.apps.openshift.io/hpa-example-pod created
[root@master ~]# oc get all
NAME READY STATUS RESTARTS AGE
pod/hpa-example-pod-1-deploy 1/1 Running 0 9s
pod/hpa-example-pod-1-ngj64 0/1 ContainerCreating 0 6s
NAME DESIRED CURRENT READY AGE
replicationcontroller/hpa-example-pod-1 1 1 0 9s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/hpa-example-pod ClusterIP 172.30.193.122 <none> 80/TCP 9s
NAME REVISION DESIRED CURRENT TRIGGERED BY
deploymentconfig.apps.openshift.io/hpa-example-pod 1 1 1 config
위와 같이 배포가 완료 되었습니다.
해당 Pod 은 CPU 리소스를 200m 까지 요청 수 있도록 설정 하였습니다.
HPA 생성
생성된 Pod 에 HPA 를 만들고 연결합니다.
[root@master ~]# oc autoscale deploymentconfig.apps.openshift.io/hpa-example-pod --min 1 --max 8 --cpu-percent=70
horizontalpodautoscaler.autoscaling/hpa-example-pod autoscaled
[root@master ~]# oc describe hpa
Name: hpa-example-pod
Namespace: hpa-project
Labels: <none>
Annotations: <none>
CreationTimestamp: Mon, 24 Jun 2019 08:20:40 +0200
Reference: DeploymentConfig/hpa-example-pod
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): 0% (0) / 70%
Min replicas: 1
Max replicas: 8
DeploymentConfig pods: 1 current / 1 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale the last scale time was sufficiently old as to warrant a new scale
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
ScalingLimited True TooFewReplicas the desired replica count is increasing faster than the maximum scale rate
Events: <none>
[root@master ~]#
생성된 HPA 의 상세 정보를 보도록 하겠습니다.
부하(load)가 발생되면 해당 Pod 의 평균 CPU 사용량이 70% 을 유지하도록 설정 되어 있습니다. 최소 1개의 Pod 을 만들 수 있고, 최대 8개까지 Pod 을 증가 시킬 수 있습니다.
부하 테스트
Openshift 에서 제공하는 Grafana 를 이용하여 실제 Pod 에 발생되는 부하(load)를 모니터링 하도록 하겠습니다.
CPU 의 사용량은 0% 로 거의 부하(load)가 없습니다.
해당 Pod 에 부하(load)를 발생 시키도록 하겠습니다.
[root@master ~]# oc run -it load-generator --image=busybox
[root@master ~]# oc get all
NAME READY STATUS RESTARTS AGE
pod/hpa-example-pod-1-ngj64 1/1 Running 0 16m
pod/load-generator-1-tl2s4 1/1 Running 0 29s
NAME DESIRED CURRENT READY AGE
replicationcontroller/hpa-example-pod-1 1 1 1 16m
replicationcontroller/load-generator-1 1 1 1 32s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/hpa-example-pod ClusterIP 172.30.193.122 <none> 80/TCP 16m
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/hpa-example-pod DeploymentConfig/hpa-example-pod 0%/70% 1 8 1 13m
NAME REVISION DESIRED CURRENT TRIGGERED BY
deploymentconfig.apps.openshift.io/hpa-example-pod 1 1 1 config
deploymentconfig.apps.openshift.io/load-generator 1 1 1 config
[root@master ~]# oc exec load-generator-1-f9xrd -ti /bin/sh
/ #
/ # while true; do wget -q -O- http://hpa-example-pod.hpa-project.svc.cluster.local; done
위와 같이 busybox 를 이용하여 load-generator 를 만들었습니다.# while true; do wget -q -O- http://hpa-example-pod.hpa-project.svc.cluster.local; done 명령을 통해 App 에 부하(load)를 유발합니다.
점점 HPA 에서 부하(load)를 감지합니다.
본격적으로 HPA 에서 부하(load)를 감지하고 CPU 사용률 평균 70%를 유지하기 위해 Pod 를 Auto-Scaling 합니다.
지속된 부하(load)로 인해 Pod이 7개까지 증가 되었습니다.
Grafana에서 Pod 상태를 보도록 하겠습니다.
다량의 부하(load)로 인해 설정된 최대 Pod 수 8개까지 증가 된 것을 확인 할 수 있습니다.
Grafana 에서도 Pod 8개가 Auto-Scaling 되어 작동하는 것을 확인 하였습니다.
이처럼 HPA 에서 부하(load)를 감지하고 Auto-Scaling 을 하여 정상적인 서비스를 유지 할 수 있습니다.
부하 테스트 종료
부하(load) 발생을 중지하면 HPA 에서 감지하고 다시 Pod 의 수를 조절합니다.
이번 포스팅에서는 HPA 를 이용하여 Auto-Scaling 을 구현 하였습니다.
Openshift의 HPA 을 이용하면 유연하고 지속적인 서비스를 구현하고 운영 할 수 있습니다.
참고 자료
오픈소스컨설팅의 마스코트, 열린이입니다! :)
