개요
CloudNET@ 가시다님이 진행하는 AWES 4기 스터디 내용을 정리합니다.
워크샵을 제공해주신 최영락님 감사드립니다.
AWS 워크샵 (GitOps를 사용하여 Amazon EKS에서 SaaS 애플리케이션 구축)을 실습하고 정리하였습니다.
해당 워크샵은 SaaS멀티네넌트 환경에서 DevOps 프로세스를 간소화할 수 있는 플랫폼 경험을 제공하는 것을 목표로 구성되었습니다.
워크샵 살펴보기
구성 아키텍처
자동화를 위해 Flux, ArgoCD, Helm 기술 스택을 사용합니다.
SaaS 플랫폼 내 비즈니스 레벨 분리
SaaS 플랫폼 운영자로서 비즈니스 레벨에 맞게 테넌트 환경(하나의 서비스 안에서 분리된 사용자 조직/계정 단위)을 제공할 예정입니다.
비즈니스 레벨이 낮은 단계인 Pool은 동일한 테넌트 환경을 사용하며 Hybrid에는 서비스 분리, Silo는 Web App 레벨이 분리됩니다.
SaaS 플랫폼 R&R 패러다임 변화
SaaS 플랫폼 운영자을 제공하면 조직간 R&R 패러다임도 변경할 수 있습니다.
기존 환경에서 개발팀의 요청이 증가함에 따라 인프라 엔지니어의 일이 그만큼 많아져 병목 현상을 일으킬 수 있으며, 지속적인 요구로 인해 최적화 업무를 할 수 없게 됩니다.
개발자 주도형 인프라
개발자 주도형 인프라는 개발자가 사전 정의된 보안 및 거버넌스 프레임워크 내에서 인프라 구성 요소를 직접 프로비저닝하고 관리할 수 있도록 지원하는 패러다임을 지향합니다. 이 모델은 자동화 및 IaC(Infrastructure as Code) 도구를 활용하여 개발자에게 셀프 서비스 기능을 제공하고, 각 애플리케이션의 특정 요구 사항에 따라 인프라를 신속하게 프로비저닝할 수 있도록 합니다.
환경 구성
구성 내용 참고 : https://github.com/aws-solutions-library-samples/eks-saas-gitops/tree/main
모든 구성 내용은 AWS 환경 위에서 구성됩니다.
(구성 1) 테라폼 모듈 파악하기
├── modules
│ ├── flux_cd
│ ├── gitea
│ ├── gitops-saas-infra
│ └── tenant-apps
│ ├── data.tf
│ ├── main.tf
│ ├── outputs.tf
│ ├── README.md
│ ├── variables.tf
│ └── versions.tf- flux_cd: EKS에 Flux를 설치하는 데 필요한 리소스가 포함되어 있습니다.
- Gitea: Gitea 저장소에 필요한 리소스를 포함합니다.
- gitops-saas-infra: 이 워크숍을 위한 인프라를 구축하는 데 필요한 리소스를 포함하고 있습니다.
- tenant-apps : 테넌트 애플리케이션을 위한 인프라 구성 요소를 포함합니다.
모듈 설치는 아래 스크립트로 동작합니다. 동작 원리를 확인하겠습니다.
install.sh : 모듈별 순차 프로비저닝 진행
# 순차 함수 실행으로 내부 자원 프로비저닝 진행
main() {
check_prerequisites
deploy_terraform_infra # 1
create_gitea_repositories # 2
echo "Proceeding with Flux setup..."
apply_flux # 3
apply_remaining_resources # 4
echo "=============================="
echo "Flux Setup Complete!"
echo "=============================="
echo "You can now check the status of Flux with:"
echo "kubectl get pods -n flux-system"
echo "=============================="
print_setup_info
clone_gitea_repos
}스크립트 전체보기
- deploy_terraform_infra : VPC, EKS, IAM Role, EC2(gitta 용도), ssm_paramerter(gitea 패스워드), vpc peerinng (VSCODE <> gitta 연결 용도), eks 네임스페이스 생성(flux)
- create_gitea_repositories : git Repo(eks-saas-gitops ) 설정
- apply_flux : GitOps용 AWS 리소스 생성 → Git repo 초기화 → Flux 연결 → Git 기반 배포 시작
Flux CD 동기화 객체 확인
|
리소스 유형
|
역할
|
|
gitrepository
|
Flux가 변경 사항을 감시하는 Git 저장소
|
|
helmrepository
|
Helm 차트가 저장된 저장소 위치 (ECR 포함)
|
|
helmchart
|
각 소스에서 가져온 Helm 차트
|
|
helmrelease
|
실제 배포 단위. 하나의 차트를 여러 테넌트에 배포 가능
|
|
kustomization
|
GitRepository를 가리키는 포인터, Kubernetes 구성 관리
|
|
imagerepository / imagepolicy
|
새 컨테이너 이미지 태그 자동 감지 및 정책 적용
|
|
imageupdateautomation
|
새 이미지 감지 시 Git에 자동 커밋 (Image Automation)
|
# NAME: 리소스 종류/이름
# REVISION: 현재 적용된 버전 (Git commit / Helm chart version / OCI digest)
# SUSPENDED: 동기화 중지 여부 (False = 정상 동작 중)
# READY: 정상 동작 여부 (True = 성공)
# MESSAGE: 상태 메시지
$ flux get all
NAME REVISION SUSPENDED READY MESSAGE
ocirepository/capacitor v0.4.8@sha256:1efcb443 False True stored artifact for digest 'v0.4.8@sha256:1efcb443'
NAME REVISION SUSPENDED READY MESSAGE
gitrepository/flux-system refs/heads/main@sha1:8a0c4024 False True stored artifact for revision 'refs/heads/main@sha1:8a0c4024'
gitrepository/terraform-v0-0-1 v0.0.1@sha1:2d19a84a False True stored artifact for revision 'v0.0.1@sha1:2d19a84a'
NAME REVISION SUSPENDED READY MESSAGE
helmrepository/argo sha256:d8865b2e False True stored artifact: revision 'sha256:d8865b2e'
helmrepository/eks-charts sha256:d5d7cd31 False True stored artifact: revision 'sha256:d5d7cd31'
helmrepository/helm-application-chart False True Helm repository is Ready
helmrepository/helm-tenant-chart False True Helm repository is Ready
helmrepository/karpenter False True Helm repository is Ready
helmrepository/kubecost False True Helm repository is Ready
helmrepository/metrics-server sha256:ba69c5bb False True stored artifact: revision 'sha256:ba69c5bb'
helmrepository/tf-controller sha256:1fcad0f6 False True stored artifact: revision 'sha256:1fcad0f6'
NAME REVISION SUSPENDED READY MESSAGE
helmchart/flux-system-argo-events 2.4.3 False True pulled 'argo-events' chart with version '2.4.3'
helmchart/flux-system-argo-workflows 0.40.11 False True pulled 'argo-workflows' chart with version '0.40.11'
helmchart/flux-system-aws-load-balancer-controller 1.6.2 False True pulled 'aws-load-balancer-controller' chart with version '1.6.2'
helmchart/flux-system-karpenter 1.4.0 False True pulled 'karpenter' chart with version '1.4.0'
helmchart/flux-system-kubecost 2.1.0 False True pulled 'cost-analyzer' chart with version '2.1.0'
helmchart/flux-system-metrics-server 3.11.0 False True pulled 'metrics-server' chart with version '3.11.0'
helmchart/flux-system-onboarding-service 0.0.1 False True pulled 'application-chart' chart with version '0.0.1'
helmchart/flux-system-pool-1 0.0.1 False True pulled 'helm-tenant-chart' chart with version '0.0.1'
helmchart/flux-system-tf-controller 0.16.0-rc.4 False True pulled 'tf-controller' chart with version '0.16.0-rc.4'
NAME LAST SCAN SUSPENDED READY MESSAGE
imagerepository/consumer-image-repository 2026-04-23T12:53:41Z False True successful scan: found 2 tags with checksum 1927939810
imagerepository/payments-image-repository 2026-04-23T12:53:41Z False True successful scan: found 2 tags with checksum 1929119460
imagerepository/producer-image-repository 2026-04-23T12:53:41Z False True successful scan: found 2 tags with checksum 1925842655
NAME IMAGE TAG READY MESSAGE
imagepolicy/consumer-image-policy 601519041915.dkr.ecr.us-west-2.amazonaws.com/consumer prd-20260420T082452Z True Latest image tag for 601519041915.dkr.ecr.us-west-2.amazonaws.com/consumer resolved to prd-20260420T082452Z
imagepolicy/payments-image-policy 601519041915.dkr.ecr.us-west-2.amazonaws.com/payments prd-20260420T082436Z True Latest image tag for 601519041915.dkr.ecr.us-west-2.amazonaws.com/payments resolved to prd-20260420T082436Z
imagepolicy/producer-image-policy 601519041915.dkr.ecr.us-west-2.amazonaws.com/producer prd-20260420T082512Z True Latest image tag for 601519041915.dkr.ecr.us-west-2.amazonaws.com/producer resolved to prd-20260420T082512Z
NAME LAST RUN SUSPENDED READY MESSAGE
imageupdateautomation/consumer-update-automation-pooled-envs 2026-04-23T12:51:22Z False True repository up-to-date
imageupdateautomation/consumer-update-automation-tenants 2026-04-23T12:49:46Z False True repository up-to-date
imageupdateautomation/payments-update-automation-pooled-envs 2026-04-23T12:51:23Z False True repository up-to-date
imageupdateautomation/payments-update-automation-tenants 2026-04-23T12:49:46Z False True repository up-to-date
imageupdateautomation/producer-update-automation-pooled-envs 2026-04-23T12:51:21Z False True repository up-to-date
imageupdateautomation/producer-update-automation-tenants 2026-04-23T12:49:46Z False True repository up-to-date
NAME REVISION SUSPENDED READY MESSAGE
helmrelease/argo-events 2.4.3 False True Helm install succeeded for release argo-events/argo-events.v1 with chart argo-events@2.4.3
helmrelease/argo-workflows 0.40.11 False True Helm install succeeded for release argo-workflows/argo-workflows.v1 with chart argo-workflows@0.40.11
helmrelease/aws-load-balancer-controller 1.6.2 False True Helm install succeeded for release aws-system/aws-load-balancer-controller.v1 with chart aws-load-balancer-controller@1.6.2
helmrelease/karpenter 1.4.0 False True Helm install succeeded for release karpenter/karpenter.v1 with chart karpenter@1.4.0
helmrelease/kubecost 2.1.0 False True Helm install succeeded for release kubecost/kubecost.v1 with chart cost-analyzer@2.1.0
helmrelease/metrics-server 3.11.0 False True Helm install succeeded for release kube-system/metrics-server.v1 with chart metrics-server@3.11.0
helmrelease/onboarding-service 0.0.1 False True Helm install succeeded for release onboarding-service/onboarding-service.v1 with chart application-chart@0.0.1
helmrelease/pool-1 0.0.1 False True Helm upgrade succeeded for release pool-1/pool-1.v2 with chart helm-tenant-chart@0.0.1
helmrelease/tf-controller 0.16.0-rc.4 False True Helm install succeeded for release flux-system/tf-controller.v1 with chart tf-controller@0.16.0-rc.4
NAME REVISION SUSPENDED READY MESSAGE
kustomization/capacitor v0.4.8@sha256:1efcb443 False True Applied revision: v0.4.8@sha256:1efcb443
kustomization/controlplane refs/heads/main@sha1:8a0c4024 False True Applied revision: refs/heads/main@sha1:8a0c4024
kustomization/dataplane-pooled-envs refs/heads/main@sha1:8a0c4024 False True Applied revision: refs/heads/main@sha1:8a0c4024
kustomization/dataplane-tenants refs/heads/main@sha1:8a0c4024 False True Applied revision: refs/heads/main@sha1:8a0c4024
kustomization/dependencies refs/heads/main@sha1:8a0c4024 False True Applied revision: refs/heads/main@sha1:8a0c4024
kustomization/flux-system refs/heads/main@sha1:8a0c4024 False True Applied revision: refs/heads/main@sha1:8a0c4024
kustomization/infrastructure refs/heads/main@sha1:8a0c4024 False True Applied revision: refs/heads/main@sha1:8a0c4024
kustomization/sources refs/heads/main@sha1:8a0c4024 False True Applied revision: refs/heads/main@sha1:8a0c4024
(구성 2) GitOps로 Terraform 실행하기
flux 내부파드 TF Controller를 통해 테라폼 코드도 gitops 로 자동화할 수 있습니다.
동작원리는 아래 그림과 같습니다. github 코드를 감지하고 CRD에 정의된 구성에 내용에 따라 runner를 실행하여 정의된 코드의 자원을 프로비저닝합니다.
# 컨트롤러 확인
kubectl get pod -n flux-system -l app.kubernetes.io/instance=tf-controller
NAME READY STATUS RESTARTS AGE
tf-controller-7b8cb5d4-2qgbv 1/1 Running 0 3d7h
# 예제 테라폼 CRD 배포
cat << EOF > /home/ec2-user/environment/gitops-gitea-repo/application-plane/production/tenants/example-tenant-terraform-crd.yaml
---
apiVersion: infra.contrib.fluxcd.io/v1alpha2
kind: Terraform
metadata:
name: example-tenant
namespace: flux-system
spec:
path: ./terraform/modules/tenant-apps
interval: 1m
approvePlan: auto
destroyResourcesOnDeletion: true
sourceRef:
kind: GitRepository
name: terraform-v0-0-1
vars:
- name: tenant_id
value: example-tenant
- name: "enable_producer"
value: true
- name: "enable_consumer"
value: true
writeOutputsToSecret:
name: example-tenant-infra-output
EOF
# 태그 버전 확인
$ kubectl get GitRepository terraform-v0-0-1 -nflux-system -oyaml | grep -i spec -C10
finalizers:
- finalizers.fluxcd.io
generation: 1
labels:
kustomize.toolkit.fluxcd.io/name: sources
kustomize.toolkit.fluxcd.io/namespace: flux-system
name: terraform-v0-0-1
namespace: flux-system
resourceVersion: "3813"
uid: c657091f-2791-4724-9964-6eb2b2ee6916
spec:
interval: 300s
ref:
tag: v0.0.1
secretRef:
name: flux-system
timeout: 60s
url: http://10.35.48.244:3000/admin/eks-saas-gitops.git
status:
artifact:
digest: sha256:00cbf55dd85a7516b9a2cf826182830cfa25a8c9317670066588a8fb431921eakustomization.yaml에 예제 파일을 등록하여 Flux가 인식
cat << EOF > /home/ec2-user/environment/gitops-gitea-repo/application-plane/production/tenants/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- basic
- advanced
- premium
- example-tenant-terraform-crd.yaml # 추가하여 감지
EOF변경 사항 PUSH
cd /home/ec2-user/environment/gitops-gitea-repo/
git pull origin main
git status
git add .
git commit -am "Added example terraform CRD for testing"
git push origin main동기화
flux reconcile source git flux-systemtf-runner 파드 로그 확인 (테라폼 프로비저닝 확인)
kubectl logs -n flux-system -l app.kubernetes.io/name=tf-runner -f
2026/04/23 16:18:07 Starting the runner... version sha
{"level":"info","ts":"2026-04-23T16:18:21.080Z","logger":"runner.terraform","msg":"preparing for Upload and Extraction","instance-id":""}
{"level":"info","ts":"2026-04-23T16:18:21.096Z","logger":"runner.terraform","msg":"write backend config","instance-id":"","path":"/tmp/flux-system-example-tenant/terraform/modules/tenant-apps","config":"backend_override.tf"}
{"level":"info","ts":"2026-04-23T16:18:21.096Z","logger":"runner.terraform","msg":"write config to file","instance-id":"","filePath":"/tmp/flux-system-example-tenant/terraform/modules/tenant-apps/backend_override.tf"}
{"level":"info","ts":"2026-04-23T16:18:21.097Z","logger":"runner.terraform","msg":"looking for path","instance-id":"","file":"terraform"}
{"level":"info","ts":"2026-04-23T16:18:21.098Z","logger":"runner.terraform","msg":"creating new terraform","instance-id":"6ba8d14a-524d-4d2c-9570-e0b27d564a87","workingDir":"/tmp/flux-system-example-tenant/terraform/modules/tenant-apps","execPath":"/usr/local/bin/terraform"}
{"level":"info","ts":"2026-04-23T16:18:21.104Z","logger":"runner.terraform","msg":"setting envvars","instance-id":"6ba8d14a-524d-4d2c-9570-e0b27d564a87"}
..
{"level":"info","ts":"2026-04-23T16:19:19.200Z","logger":"runner.terraform","msg":"creating a plan","instance-id":"1f5dabc0-2fb8-4120-89ce-e73f11196af6"}
{"level":"info","ts":"2026-04-23T16:18:55.783Z","logger":"runner.terraform","msg":"loading plan from secret","instance-id":"6ba8d14a-524d-4d2c-9570-e0b27d564a87"}
{"level":"info","ts":"2026-04-23T16:18:57.422Z","logger":"runner.terraform","msg":"setting up the input variables","instance-id":"def263dc-f4bb-4fa8-81c5-7a8e43a8293d"}
{"level":"info","ts":"2026-04-23T16:18:55.804Z","logger":"runner.terraform","msg":"running apply","instance-id":"6ba8d14a-524d-4d2c-9570-e0b27d564a87"}
random_string.random_suffix: Creating...
random_string.random_suffix: Creation complete after 0s [id=gc1]
{"level":"info","ts":"2026-04-23T16:18:57.422Z","logger":"runner.terraform","msg":"mapping the Spec.Values","instance-id":"def263dc-f4bb-4fa8-81c5-7a8e43a8293d"}
{"level":"info","ts":"2026-04-23T16:18:57.422Z","logger":"runner.terraform","msg":"mapping the Spec.Vars","instance-id":"def263dc-f4bb-4fa8-81c5-7a8e43a8293d"}
{"level":"info","ts":"2026-04-23T16:18:57.422Z","logger":"runner.terraform","msg":"mapping the Spec.VarsFrom","instance-id":"def263dc-f4bb-4fa8-81c5-7a8e43a8293d"}
{"level":"info","ts":"2026-04-23T16:18:57.432Z","logger":"runner.terraform","msg":"generating the template founds"}
aws_sqs_queue.consumer_sqs[0]: Creating...
{"level":"info","ts":"2026-04-23T16:18:57.432Z","logger":"runner.terraform","msg":"main.tf.tpl not found, skipping"}
{"level":"info","ts":"2026-04-23T16:18:57.443Z","logger":"runner.terraform","msg":"initializing","instance-id":"def263dc-f4bb-4fa8-81c5-7a8e43a8293d"}
{"level":"info","ts":"2026-04-23T16:18:57.443Z","logger":"runner.terraform","msg":"mapping the Spec.BackendConfigsFrom","instance-id":"def263dc-f4bb-4fa8-81c5-7a8e43a8293d"}
{"level":"info","ts":"2026-04-23T16:19:30.783Z","logger":"runner.terraform","msg":"workspace select"}
{"level":"info","ts":"2026-04-23T16:19:30.791Z","logger":"runner.terraform","msg":"creating a plan","instance-id":"def263dc-f4bb-4fa8-81c5-7a8e43a8293d"}
module.producer_irsa_role[0].aws_iam_role.this[0]: Creating...
aws_dynamodb_table.consumer_ddb[0]: Creating...
module.consumer_irsa_role[0].aws_iam_role.this[0]: Creating...
module.producer_irsa_role[0].aws_iam_role.this[0]: Creation complete after 1s [id=producer-role-example-tenant]
module.consumer_irsa_role[0].aws_iam_role.this[0]: Creation complete after 1s [id=consumer-role-example-tenant]
aws_sqs_queue.consumer_sqs[0]: Still creating... [10s elapsed]
aws_dynamodb_table.consumer_ddb[0]: Still creating... [10s elapsed]
aws_dynamodb_table.consumer_ddb[0]: Creation complete after 11s [id=consumer-example-tenant-gc1]
aws_ssm_parameter.dedicated_consumer_ddb[0]: Creating...
aws_ssm_parameter.dedicated_consumer_ddb[0]: Creation complete after 0s [id=/example-tenant/consumer_ddb].구성 검증
$ aws dynamodb list-tables
{
"TableNames": [
"consumer-example-tenant-gc1", # 신규 생성 확인
"consumer-pool-1-wcv",
]
}(구성 3) Helm 차트와 Flux 통합
Flux가 관리하는 Helm 배포는 ECR에 저장된 패키지 차트를 기반으로 동작합니다.
# Get values from configmap
AWS_ACCOUNT_ID=$(kubectl get configmap saas-infra-outputs -n flux-system -o jsonpath='{.data.account_id}')
ECR_HELM_CHART_URL=$(kubectl get configmap saas-infra-outputs -n flux-system -o jsonpath='{.data.ecr_helm_chart_url}')
ECR_REGISTRY=$(echo $ECR_HELM_CHART_URL | cut -d'/' -f1)
ECR_REPOSITORY=$(echo $ECR_HELM_CHART_URL | cut -d'/' -f2-)
AWS_REGION=$(echo $ECR_HELM_CHART_URL | cut -d'.' -f4)
# Authenticate Docker to ECR
aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $ECR_REGISTRY
# List artifacts (images) in the ECR repository
aws ecr list-images --repository-name $ECR_REPOSITORY --region $AWS_REGION
..
Login Succeeded
{
"imageIds": [
{
"imageDigest": "sha256:1f6cfdb61e14b4d7cc30f208bbefd4c4a12e0fb54a4987e10fd149d5ad6dc4b1",
"imageTag": "0.0.1"
}
]
#
Flux 에서는 HelmRelease를 통해 helm를 자동 관리할 수 있음
HelmRelease는 Flux가 제공하는 CRD로, Helm 릴리스를 선언적으로 관리할 수 있게 해줌. YAML 파일로 Helm 릴리스를 정의하면 Flux가 배포와 업데이트를 자동으로 처리함
# 구성 배포
cat << EOF > /home/ec2-user/environment/gitops-gitea-repo/application-plane/production/tenants/example-tenant-helmrelease.yaml
apiVersion: v1
kind: Namespace
metadata:
name: example-tenant
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: example-tenant-premium
namespace: flux-system
spec:
releaseName: example-tenant-premium
targetNamespace: example-tenant
storageNamespace: example-tenant
interval: 1m0s
chart:
spec:
chart: helm-tenant-chart
version: "0.x"
sourceRef:
kind: HelmRepository
name: helm-tenant-chart
values:
tenantId: example-tenant
apps:
producer:
enabled: true
consumer:
enabled: true
EOF
# kustomization.yaml 파일에서 추가된 파일 reference 하도록 설정
cat << EOF >> /home/ec2-user/environment/gitops-gitea-repo/application-plane/production/tenants/kustomization.yaml
- example-tenant-helmrelease.yaml
EOF
# 변경 사항 PUSH
cd /home/ec2-user/environment/gitops-gitea-repo/
git pull origin main
git add .
git commit -m "Added HelmRelease for example-tenant"
git push origin main
# flux 동기화
flux reconcile source git flux-system
# 로그 확인
kubectl logs po/example-tenant-tf-runner -n flux-system -f
..
stance-id":"deba16e4-1dc5-4c1b-89eb-980ee352fde1"}
{"level":"info","ts":"2026-04-23T16:31:03.021Z","logger":"runner.terraform","msg":"mapping the Spec.BackendConfigsFrom","instance-id":"deba16e4-1dc5-4c1b-89eb-980ee352fde1"}
{"level":"info","ts":"2026-04-23T16:31:26.661Z","logger":"runner.terraform","msg":"workspace select"}
{"level":"info","ts":"2026-04-23T16:31:26.663Z","logger":"runner.terraform","msg":"creating a plan","instance-id":"deba16e4-1dc5-4c1b-89eb-980ee352fde1"}
{"level":"info","ts":"2026-04-23T16:32:41.309Z","logger":"runner.terraform","msg":"creating outputs","instance-id":"deba16e4-1dc5-4c1b-89eb-980ee352fde1"}
{"level":"info","ts":"2026-04-23T16:32:49.563Z","logger":"runner.terraform","msg":"write outputs to secret","instance-id":"deba16e4-1dc5-4c1b-89eb-980ee352fde1"}
{"level":"info","ts":"2026-04-23T16:32:49.595Z","logger":"runner.terraform","msg":"cleanup TmpDir","instance-id":"deba16e4-1dc5-4c1b-89eb-980ee352fde1","tmpDir":"/tmp/flux-system-example-tenant"}
# 구성 확인 (HelmRelease 에 정의한 네임스페이스에 배포 자원 확인)
kubectl get all -n example-tenant
구성 확인
gitea 환경 접속
# 아래 명령어 실행으로 URL & 패스워드 기입
# Get Gitea IPs from the configuration
export GITEA_PRIVATE_IP=$(kubectl get configmap saas-infra-outputs -n flux-system -o jsonpath='{.data.gitea_url}')
export GITEA_PUBLIC_IP=$(kubectl get configmap saas-infra-outputs -n flux-system -o jsonpath='{.data.gitea_public_url}')
export GITEA_PORT="3000"
# Get Gitea admin password from Systems Manager Parameter Store
export GITEA_ADMIN_PASSWORD=$(aws ssm get-parameter --name "/eks-saas-gitops/gitea-admin-password" --with-decryption --query 'Parameter.Value' --output text)
# Display access information for web browser login
echo "=== Gitea Web Interface Access ==="
echo "Public URL (for browser access): $GITEA_PUBLIC_IP"
echo "Username: admin"
echo "Password: $GITEA_ADMIN_PASSWORD"
echo "=================================="
echo ""
echo "Use the PUBLIC URL above to access Gitea from your web browser."
저장소 접근 변수 설정 & 저장소 복제
# Extract Gitea configuration from ConfigMap
export GITEA_TOKEN=$(kubectl get configmap saas-infra-outputs -n flux-system -o jsonpath='{.data.gitea_token}')
# Set up the repository paths used throughout the workshop
export REPO_PATH="/home/ec2-user/environment/microservice-repos"
export GITOPS_REPO_PATH="/home/ec2-user/environment/gitops-gitea-repo"
mkdir -p $REPO_PATH
cd $REPO_PATH
# Clone the microservice repositories
git clone http://admin:${GITEA_TOKEN}@${GITEA_PRIVATE_IP}:${GITEA_PORT}/admin/producer.git
git clone http://admin:${GITEA_TOKEN}@${GITEA_PRIVATE_IP}:${GITEA_PORT}/admin/consumer.git
git clone http://admin:${GITEA_TOKEN}@${GITEA_PRIVATE_IP}:${GITEA_PORT}/admin/payments.git
.
├── consumer
│ ├── Dockerfile
│ ├── consumer.py
│ └── requirements.txt
├── payments
│ ├── Dockerfile
│ ├── payments.py
│ └── requirements.txt
└── producer
├── Dockerfile
├── producer.py
└── requirements.txt
샘플 테넌트 앱 구조
애플리케이션 간, Amazon SQS 및 Amazon DynamoDB와 같은 외부 서비스 상호 작용 애플리케이션 구조 (payments는 실습 추후 제공)
- Producer : API 호출을 수신하고, 메시지를 생성하여 SQS 큐로 전송하는 역할을 담당
- Cosumer: SQS 큐에서 메시지를 가져와 특정 DynamoDB 테이블에 저장하는 역할을 담당
SaaS 애플리케이션 티어 전략
SaaS 애플리케이션은 일반적으로 다양한 고객군을 지원하기 위해 티어(Tier) 구조를 분리하여 관리하며 helm 차트를 통해 차등을 줍니다.
tree .
.
├── pooled-envs
│ ├── kustomization.yaml
│ └── pool-1.yaml
├── tenants
│ ├── advanced
│ │ ├── dummy-configmap.yaml
│ │ └── kustomization.yaml
│ ├── basic
│ │ ├── dummy-configmap.yaml
│ │ └── kustomization.yaml
│ ├── kustomization.yaml
│ └── premium
│ ├── dummy-configmap.yaml
│ └── kustomization.yaml
└── tier-templates
├── advanced_tenant_template.yaml
├── basic_env_template.yaml
├── basic_tenant_template.yaml
└── premium_tenant_template.yaml동일 헬름 차트는 다음 경로를 통해 확인 가능
$ pwd
/home/ec2-user/environment/eks-saas-gitops/helm-charts
$ tree .
.
├── application-chart
│ ├── Chart.yaml
│ ├── templates
│ │ ├── NOTES.txt
│ │ ├── _helpers.tpl
│ │ ├── deployment.yaml
│ │ ├── hpa.yaml
│ │ ├── ingress.yaml
│ │ ├── service.yaml
│ │ ├── serviceaccount.yaml
│ │ └── tests
│ │ └── test-connection.yaml
│ └── values.yaml
└── helm-tenant-chart
├── Chart.yaml
├── templates
│ ├── deployment.yaml
│ ├── hpa.yaml
│ ├── ingress.yaml
│ ├── service.yaml
│ ├── serviceaccount.yaml
│ └── terraform.yaml
└── values.yaml.template
기본 등급 (basic_env_template.yaml)
단일 테넌트 환경으로 마이크로서비스(Producer, Consumer)가 공용 Pool에서 구성합니다.
ingress false는 내부 전용으로 외부에 오픈하지 않습니다.
$ cat basic_env_template.yaml
..
values:
tenantId: {ENVIRONMENT_ID}
apps:
producer:
enabled: true
ingress:
enabled: false
image:
tag: "0.1" # {"$imagepolicy": "flux-system:producer-image-policy:tag"}
consumer:
enabled: true
ingress:
enabled: false
image:
tag: "0.1" # {"$imagepolicy": "flux-system:consumer-image-policy:tag"}
$ tree /home/ec2-user/environment/gitops-gitea-repo/application-plane/production/pooled-envs
/home/ec2-user/environment/gitops-gitea-repo/application-plane/production/pooled-envs
├── kustomization.yaml
└── pool-1.yaml
$ cat /home/ec2-user/environment/gitops-gitea-repo/application-plane/production/pooled-envs/pool-1.yaml
apiVersion: v1
kind: Namespace
metadata:
name: pool-1
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: pool-1
namespace: flux-system
spec:
releaseName: pool-1
targetNamespace: pool-1
interval: 1m0s
chart:
spec:
chart: helm-tenant-chart
version: "0.0.x"
sourceRef:
kind: HelmRepository
name: helm-tenant-chart
values:
tenantId: pool-1
apps:
producer:
enabled: true
ingress:
enabled: false
image:
tag: "prd-20260420T082512Z" # {"$imagepolicy": "flux-system:producer-image-policy:tag"}
consumer:
enabled: true
ingress:
enabled: false
image:
tag: "prd-20260420T082452Z" # {"$imagepolicy": "flux-system:consumer-image-policy:tag"이후 구성된 각 테넌트들은 공용된 Pool를 사용합니다.
프리미엄 등급 (premium_tenant_template.yaml)
프리미엄 등급은 각 테넌드가 독립된 구성 환경을 가집니다.
cat premium_tenant_template.yaml
apiVersion: v1
kind: Namespace
metadata:
name: {TENANT_ID}
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: {TENANT_ID}-premium
namespace: flux-system
spec:
releaseName: {TENANT_ID}-premium
targetNamespace: {TENANT_ID}
storageNamespace: {TENANT_ID}
interval: 1m0s
chart:
spec:
chart: helm-tenant-chart
version: "{RELEASE_VERSION}.x"
sourceRef:
kind: HelmRepository
name: helm-tenant-chart
values:
tenantId: {TENANT_ID}
apps:
producer:
enabled: true # Silo deployment -- premium tier has a dedicated deployment for each tenant
ingress:
enabled: true
image:
tag: "0.1" # {"$imagepolicy": "flux-system:producer-image-policy:tag"}
consumer:
enabled: true # Silo deployment -- premium tier has a dedicated deployment for each tenant
ingress:
enabled: true
image:
tag: "0.1" # {"$imagepolicy": "flux-system:consumer-image-policy:tag"}기본 등급과 비교했을 때 동일한 Helm 차트가 사용되며, Helm 릴리스 템플릿은 배포에 대한 티어별 구성을 지정합
고급 등급 정의
공용 리소스와 공유 리소스가 혼합된 형태여야 하며, 생산자는 공유되지만 소비자는 전용 리소스를 사용하는 등급을 정의합니다.
- releaseNamepremiumadvance 을 advacnded라고 명칭하여 새로운 등급 정의
- 프로듀서는 다른 테넌트와 공용이니 False로 지정하고 공용 풀로 설정
cat << EOF > /home/ec2-user/environment/gitops-gitea-repo/application-plane/production/tier-templates/advanced_tenant_template.yaml
apiVersion: v1
kind: Namespace
metadata:
name: {TENANT_ID}
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: {TENANT_ID}-advanced
namespace: flux-system
spec:
releaseName: {TENANT_ID}-advanced
targetNamespace: {TENANT_ID} # Deploying into the tenant-specific namespace
interval: 1m0s
chart:
spec:
chart: helm-tenant-chart
version: "{RELEASE_VERSION}.x"
sourceRef:
kind: HelmRepository
name: helm-tenant-chart
values:
tenantId: {TENANT_ID}
apps:
producer:
envId: pool-1
enabled: false # Pool deployment -- advanced tier shares resources with other tenants
ingress:
enabled: true
consumer:
enabled: true # Silo deployment -- advanced tier has a dedicated deployment for each tenant
ingress:
enabled: true
image:
tag: "0.1" # {"\$imagepolicy": "flux-system:consumer-image-policy:tag"}
EOF고급 등급 테넌트를 추가하여 리소스 확인
mkdir -p /home/ec2-user/environment/gitops-gitea-repo/application-plane/production/tenants/advanced
export TENANT_ID=tenant-t1d6c
export RELEASE_VERSION=0.0
cd /home/ec2-user/environment/gitops-gitea-repo/application-plane/production/
cp tier-templates/advanced_tenant_template.yaml tenants/advanced/$TENANT_ID.yaml
# 특정 테넌트 생성
sed -i "s|{TENANT_ID}|$TENANT_ID|g" "tenants/advanced/$TENANT_ID.yaml"
sed -i "s|{RELEASE_VERSION}|$RELEASE_VERSION|g" "tenants/advanced/$TENANT_ID.yaml"
cat << EOF > tenants/advanced/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- $TENANT_ID.yaml
EOF
# 변경사항 PUSH
cd /home/ec2-user/environment/gitops-gitea-repo/
git pull origin main
git add .
git commit -am "Adding tenant-t1d6c with Advanced Tier"
git push origin main
# flux 동기화
flux reconcile source git flux-system
# 파드 구성 확인
kubectl get deployment -n tenant-t1d6c
NAME READY UP-TO-DATE AVAILABLE AGE
tenant-t1d6c-consumer 3/3 3 3 3m1s
# 테스트 확인
APP_LB=http://$(kubectl get ingress -n tenant-t1d6c -o json | jq -r .items[0].status.loadBalancer.ingress[0].hostname)
curl -s -H "tenantID: tenant-t1d6c" $APP_LB/producer | jq
curl -s -H "tenantID: tenant-t1d6c" $APP_LB/consumer | jq
{
"environment": "pool-1", # 공용
"microservice": "producer",
"tenant_id": "tenant-t1d6c",
"version": "0.0.1"
}
{
"environment": "tenant-t1d6c", # 전용 테넌트
"microservice": "consumer",
"tenant_id": "tenant-t1d6c",
"version": "0.0.1"
}
테넌트 온보딩/오프보딩 자동화
Argo Workflow를 활용하여 위 설정 과정을 자동화하는 과정을 확인하겠습니다.
자동화는 이벤트 트리거 구조로 SQS 메세지에 의해 시작되어 전체 파이프라인이 구성됩니다.
여기서 Argo Workflow 는 위 티어 정의 과정(템플릿 복사, 변수 치환, git 커밋, 푸쉬)를 자동화합니다.
프리미엄 등급 자동화
# 워크플로우 템플릿 확인
tree /home/ec2-user/environment/gitops-gitea-repo/control-plane/production/workflows/
/home/ec2-user/environment/gitops-gitea-repo/control-plane/production/workflows/
├── event-bus.yaml
├── kustomization.yaml
├── tenant-deployment-sensor.yaml
├── tenant-deployment-workflow-template.yaml
├── tenant-offboarding-sensor.yaml
├── tenant-offboarding-workflow-template.yaml
├── tenant-onboarding-sensor.yaml
└── tenant-onboarding-workflow-template.yaml
# 배포 확인
kubectl get workflowtemplates -n argo-workflows
NAME AGE
tenant-deployment-template 3d7h
tenant-offboarding-template 3d7h
tenant-onboarding-template 3d7h|
워크플로우 템플릿
|
역할
|
|
onboarding
|
새 테넌트를 환경에 프로비저닝
|
|
offboarding
|
테넌트를 환경에서 제거
|
|
deployment
|
테넌트 HelmRelease 버전 업데이트
|
onboarding 동작 이해
- EventSource 객체 : 이벤트 감지
- Sensor : 이벤트 감지 후 메세지 내용을 추출하고 Argo Workflow 실행
- WorkflowTemplate : 실제 동작 원리
$ cat tenant-onboarding-sensor.yaml
---
apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
name: aws-sqs-onboarding
namespace: argo-events
spec:
template:
serviceAccountName: argo-events-sa
sqs:
tenant-provisioning:
jsonBody: true
region: "${aws_region}"
queue: "argoworkflows-onboarding-queue"
waitTimeSeconds: 20
---
apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
name: aws-sqs-onboarding
namespace: argo-events
spec:
template:
serviceAccountName: argo-events-sa
dependencies:
- name: tenant-provisioning-dep
eventSourceName: aws-sqs-onboarding
eventName: tenant-provisioning
triggers:
- template:
name: tenant-onboarding-template
k8s:
operation: create
source:
resource:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: tenant-onboarding-
namespace: argo-workflows
spec:
serviceAccountName: argoworkflows-sa
entrypoint: tenant-provisioning
synchronization:
mutex:
name: workflow
arguments:
parameters:
- name: TENANT_ID
value: "" # ID of your tenant, use this patter eg. tenant-xx (tenant-10, tenant-11)
- name: TENANT_TIER
value: "" # Valid values are: silo, pool, hybrid
- name: RELEASE_VERSION
value: "" # I.E. 0.0 or 1.0
- name: REPO_URL
value: "http://${gitea_url}:3000/admin/eks-saas-gitops.git"
- name: GIT_USER_EMAIL
value: "admin@example.com"
- name: GIT_USERNAME
value: "admin"
- name: GIT_TOKEN
value: "${gitea_token}"
- name: GIT_BRANCH
value: "main" # Can change based on your configs
templates:
- name: tenant-provisioning
steps:
- - name: clone-repository
templateRef:
name: tenant-onboarding-template
template: clone-repository
- - name: validate-if-tenant-exists
templateRef:
name: tenant-onboarding-template
template: validate-if-tenant-exists
- - name: create-tenant-helm-release
templateRef:
name: tenant-onboarding-template
template: create-tenant-helm-release
volumeClaimTemplates:
- metadata:
name: workdir
spec:
storageClassName: gp2
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1Gi
parameters:
- src:
dependencyName: tenant-provisioning-dep
dataKey: body.tenant_id
dest: spec.arguments.parameters.0.value
- src:
dependencyName: tenant-provisioning-dep
dataKey: body.tenant_tier
dest: spec.arguments.parameters.1.value
- src:
dependencyName: tenant-provisioning-dep
dataKey: body.release_version
dest: spec.arguments.parameters.2.value
$ cat tenant-onboarding-workflow-template.yaml
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: tenant-onboarding-template
namespace: argo-workflows
spec:
serviceAccountName: argoworkflows-sa # SA with IRSA permissions
templates:
- name: validate-if-tenant-exists
container:
image: "${ecr_argoworkflow_container}:0.1"
command: ["/bin/sh","-c"]
args: ['./00-validate-tenant.sh {{workflow.parameters.TENANT_ID}}']
volumeMounts:
- name: workdir
mountPath: /mnt/vol
env:
- name: GIT_USERNAME
value: "{{workflow.parameters.GIT_USERNAME}}"
- name: GIT_TOKEN
value: "{{workflow.parameters.GIT_TOKEN}}"
- name: clone-repository
container:
image: "${ecr_argoworkflow_container}:0.1"
command: ["/bin/sh","-c"]
args: ['./01-tenant-clone-repo.sh {{workflow.parameters.REPO_URL}} {{workflow.parameters.GIT_BRANCH}} {{workflow.parameters.GIT_USERNAME}} {{workflow.parameters.GIT_TOKEN}} && cp -r eks-saas-gitops /mnt/vol/eks-saas-gitops']
volumeMounts:
- name: workdir
mountPath: /mnt/vol
env:
- name: GIT_USERNAME
value: "{{workflow.parameters.GIT_USERNAME}}"
- name: GIT_TOKEN
value: "{{workflow.parameters.GIT_TOKEN}}"
- name: create-tenant-helm-release
container:
image: "${ecr_argoworkflow_container}:0.1"
command: ["/bin/sh","-c"]
args: ['./02-tenant-onboarding.sh {{workflow.parameters.TENANT_ID}} {{workflow.parameters.RELEASE_VERSION}} {{workflow.parameters.TENANT_TIER}} {{workflow.parameters.GIT_USER_EMAIL}} {{workflow.parameters.GIT_USERNAME}} {{workflow.parameters.GIT_BRANCH}} {{workflow.parameters.GIT_TOKEN}}']
volumeMounts:
- name: workdir
mountPath: /mnt/vol
env:
- name: GIT_USERNAME
value: "{{workflow.parameters.GIT_USERNAME}}"
- name: GIT_TOKEN
value: "{{workflow.parameters.GIT_TOKEN}}"
volumeClaimTemplates:
- metadata:
name: workdir
spec:
storageClassName: gp2
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1Gi프리미엄 등급 온보딩 확인 : SQS 메세지를 생성하여 workflow 템플릿 구성 동작 확인
export ARGO_WORKFLOWS_ONBOARDING_QUEUE_SQS_URL=$(kubectl get configmap saas-infra-outputs -n flux-system -o jsonpath='{.data.argoworkflows_onboarding_queue_url}')
aws sqs send-message \
--queue-url $ARGO_WORKFLOWS_ONBOARDING_QUEUE_SQS_URL \
--message-body '{
"tenant_id": "tenant-1",
"tenant_tier": "premium",
"release_version": "0.0"
}'
# 구성 워크플로우 확인
kubectl -n argo-workflows get workflow
NAME STATUS AGE MESSAGE
tenant-onboarding-fcwwd Running 7s
# URL 확인
ARGO_WORKFLOW_URL=$(kubectl -n argo-workflows get service/argo-workflows-server -o json | jq -r '.status.loadBalancer.ingress[0].hostname')
echo http://$ARGO_WORKFLOW_URL:2746/workflows
→ workflow 정의에 따라 git clone, 테넌트 ID 중복 확인, 신규 헬름 템플릿 생성
{{workflow.parameters.TENANT_ID}} 를 통해 티어 주입
- name: create-tenant-helm-release
container:
image: "${ecr_argoworkflow_container}:0.1"
command: ["/bin/sh","-c"]
args: ['./02-tenant-onboarding.sh {{workflow.parameters.TENANT_ID}} {{workflow.parameters.RELEASE_VERSION}} {{workflow.parameters.TENANT_TIER}} {{workflow.parameters.GIT_USER_EMAIL}} {{workflow.parameters.GIT_USERNAME}} {{workflow.parameters.GIT_BRANCH}} {{workflow.parameters.GIT_TOKEN}}']
volumeMounts:
- name: workdir
mountPath: /mnt/vol
env:
- name: GIT_USERNAME
value: "{{workflow.parameters.GIT_USERNAME}}"
- name: GIT_TOKEN
value: "{{workflow.parameters.GIT_TOKEN}}"리소스 확인
$ flux get helmreleases
NAME REVISION SUSPENDED READY MESSAGE
argo-events 2.4.3 False True Helm install succeeded for release argo-events/argo-events.v1 with chart argo-events@2.4.3
argo-workflows 0.40.11 False True Helm install succeeded for release argo-workflows/argo-workflows.v1 with chart argo-workflows@0.40.11
aws-load-balancer-controller 1.6.2 False True Helm install succeeded for release aws-system/aws-load-balancer-controller.v1 with chart aws-load-balancer-controller@1.6.2
karpenter 1.4.0 False True Helm install succeeded for release karpenter/karpenter.v1 with chart karpenter@1.4.0
kubecost 2.1.0 False True Helm install succeeded for release kubecost/kubecost.v1 with chart cost-analyzer@2.1.0
metrics-server 3.11.0 False True Helm install succeeded for release kube-system/metrics-server.v1 with chart metrics-server@3.11.0
onboarding-service 0.0.1 False True Helm install succeeded for release onboarding-service/onboarding-service.v1 with chart application-chart@0.0.1
pool-1 0.0.1 False True Helm upgrade succeeded for release pool-1/pool-1.v2 with chart helm-tenant-chart@0.0.1
tenant-1-premium 0.0.1 False True Helm upgrade succeeded for release tenant-1/tenant-1-premium.v2 with chart helm-tenant-chart@0.0.1
tenant-t1d6c-advanced 0.0.1 False True Helm upgrade succeeded for release tenant-t1d6c/tenant-t1d6c-advanced.v2 with chart helm-tenant-chart@0.0.1
tf-controller 0.16.0-rc.4 False True Helm install succeeded for release flux-system/tf-controller.v1 with chart tf-controller@0.16.0-rc.4
$ kubectl -n tenant-1 get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
tenant-1-consumer 3/3 3 3 10m
tenant-1-producer 3/3 3 3 10m'Cloud' 카테고리의 다른 글
| EKS 버전 업그레이드 (0) | 2026.05.03 |
|---|---|
| Vault VSO 맛보기 (0) | 2025.12.07 |
| HashiCorp Vault 맛보기 (0) | 2025.11.29 |
| KeyCloak SSO 실습 (0) | 2025.11.23 |
| ArgoCD로 멀티클러스터 관리하기 (0) | 2025.11.23 |