무중단 배포 작업 과정4

[hyeonji1220]

github actions

• bash 파일과 함께 docker-compose 를 실행하도록 변경
• 수동으로 할 수 있게 (트리거) 변경

실전

• 개발 서버에 적용을 해본다. 
• 배포 하는 순간에 Jmeter 로 요청을 보내서 확인을 한다.
  : dev 서버에 어플리케이션을 하나 더 띄워보고 얼마나 CPU, 메모리가 치솟나 보면 되지 않을까?
  어차피 micro이고, 깔려있는 것도 prod보다 더 많으니 dev에서 괜찮은 수치를 보인다면 prod도 안심할 수 있을 것 같다.
  여기에, 단순히 견디는지가 아니라 tps에 얼마나 영향을 주는지까지 확인해보고 싶다면, dev에 실험해보는 동안 tps 부하를 줘보면 알 수 있겠다.

추가로 해야할 것

• prod-cd에 max-parallel 설정하기

고민해볼 점

• 만약 포트가 8081 → 8082로 바뀌거나, 어플리케이션을 2대가 아닌 1대(8083) 더 띄우도록 바뀐다면 deploy.sh에 변경해줘야하는 부분이 많고, 파악하기 어렵다.

[hyeonji1220]

1. shell script

#!/bin/bash

# 환경 변수 확인
if [ -z "$PROFILE_VAR" ]; then
    echo "Error: PROFILE_VAR is not set"
    exit 1
fi

# .env.prod 또는 .env.dev 파일의 내용을 환경변수로 저장
if [ ! -f "/home/ubuntu/review-me/app/.env.$PROFILE_VAR" ]; then
    echo "Error: .env.$PROFILE_VAR file not found"
    exit 1
fi

# 현재 활성화된 포트 확인
get_current_server_port() {
    local current_port=$(docker ps --format '{{.Ports}}' | 
                         grep -E ':::8080->8080|:::8081->8081' | 
                         sed -E 's/.*:::(8080|8081).*/\1/' | 
                         head -n1)
    if [ -z "$current_port" ]; then
        echo "No active app container found running on port 8080 or 8081"
        exit 1
    fi
    echo "$current_port is current port!"
}

CURRENT_SERVER_PORT=$(get_current_server_port)

# 새로운 서버 포트, actuator 포트 설정
if [ "$CURRENT_SERVER_PORT" = "8080" ]; then 
   NEW_SERVER_PORT=8081
   NEW_ACTUATOR_PORT=5960
   CURRENT_ACTUATOR_PORT=5959
elif [ "$CURRENT_SERVER_PORT" = "8081" ]; then
   NEW_SERVER_PORT=8080
   NEW_ACTUATOR_PORT=5959
   CURRENT_ACTUATOR_PORT=5960
else
   echo "Error: Unexpected CURRENT_SERVER_PORT value: $CURRENT_SERVER_PORT"
   exit 1
fi

# 기존 활성화된 포트로 포워딩 유지
echo "Forwarding 80 -> $CURRENT_SERVER_PORT"
echo "Forwarding 5959 -> $CURRENT_ACTUATOR_PORT"

# iptables 규칙 확인: 기존 80 -> CURRENT_SERVER_PORT 규칙이 있는지 확인
if ! sudo iptables -t nat -C PREROUTING -p tcp --dport 80 -j REDIRECT --to-port $CURRENT_SERVER_PORT &> /dev/null; then
    # 규칙이 없으면 추가
    sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port $CURRENT_SERVER_PORT
    sudo iptables -t nat -A PREROUTING -p tcp --dport 5959 -j REDIRECT --to-port $CURRENT_ACTUATOR_PORT
else
    echo "iptables rule for port $CURRENT_SERVER_PORT already exists, skipping addition."
fi

# 새 버전 배포
docker-compose --env-file .env."$PROFILE_VAR"  up -d app-$NEW_SERVER_PORT 

# 새 컨테이너 상태 확인
check_new_container() {
    if ! docker ps --format '{{.Names}}:{{.Ports}}' | grep -q "^app.*:::80->$NEW_SERVER_PORT/tcp"; then
        echo "New container is not running on port $NEW_SERVER_PORT"
        return 1
    fi

    local health_status=$(curl -s --max-time 10 "http://localhost:$NEW_ACTUATOR_PORT/actuator/health" | jq -r '.status')
    if [ "$health_status" != "UP" ]; then
        echo "Actuator health check failed. Status: $health_status"
        return 1
    fi

    echo "New container is running and healthy"
    return 0
}

# 반복하며 새 컨테이너 상태 확인
MAX_RETRIES=5
RETRY_INTERVAL=5

for i in $(seq 1 $MAX_RETRIES)
do
   if check_new_container; then
       echo "New version is up and running on port $NEW_SERVER_PORT"
       break
   fi
    if [ $i -eq $MAX_RETRIES ]; then
        echo "New version failed to start, rolling back"
        docker-compose --env-file .env."$PROFILE_VAR" down app-$NEW_SERVER_PORT
        if [ $? -ne 0 ]; then
            echo "Error: Failed to stop new container during rollback"
        fi
      # 기존 iptables 규칙이 여전히 유효한지 확인
      if ! sudo iptables -t nat -C PREROUTING -p tcp --dport 80 -j REDIRECT --to-port $CURRENT_SERVER_PORT &> /dev/null; then
          echo "Error: Original iptables rule no longer exists"
      else 
          sudo iptables -t nat -D PREROUTING -p tcp --dport 80 -j REDIRECT --to-port $CURRENT_SERVER_PORT
          sudo iptables -t nat -D PREROUTING -p tcp --dport 5959 -j REDIRECT --to-port $CURRENT_ACTUATOR_PORT
      fi
      exit 1
  fi
   echo "Waiting for the new version to start... (Attempt $i/$MAX_RETRIES)"
   sleep $RETRY_INTERVAL
done

# 새 버전으로 포워딩 업데이트
echo "Updating forwarding: 80 -> $NEW_SERVER_PORT"
sudo iptables -t nat -R PREROUTING 1 -p tcp --dport 80 -j REDIRECT --to-port $NEW_SERVER_PORT
sudo iptables -t nat -R PREROUTING 1 -p tcp --dport 5959 -j REDIRECT --to-port $NEW_ACTUATOR_PORT

# 이전 버전 종료 (graceful shutdown)
OLD_PORT=$([ "$NEW_SERVER_PORT" == "8080" ] && echo "8081" || echo "8080")
docker-compose --env-file .env."$PROFILE_VAR" stop -t 10 app-$OLD_PORT

[hyeonji1220]

is_new_container_healthy() {
    local health_response=$(curl -s http://localhost:$NEW_PORT/actuator/health)
    local overall_status=$(echo "$health_response" | jq -r '.status')

    if [ "$overall_status" == "UP" ]; then
        echo "Overall health status is UP"
        return 0
    else
        echo "Overall health status is DOWN"
        return 1
    fi
}

[hyeonji1220]

NEW_SERVER_PORT=8081
NEW_ACTUATOR_PORT =5960

이런 변수들이 매직넘버일까? 모아서 관리할 필요는? 방법은?

[hyeonji1220]

2. CD (prod)

실험 결과 순서대로 하나씩 runs-on을 실행함!

name: "[RELEASE] CD using Github self-hosted runner"

on:
  workflow_dispatch:
  push:
    branches:
      - release
    paths:
      - 'backend/**'

env:
  APPLICATION_DIRECTORY: /home/ubuntu/review-me

jobs:
  build:
    name: Build Dockerfile and push to DockerHub
    runs-on: ubuntu-latest

    steps:
      - name: Checkout to current repository
        uses: actions/checkout@v4

      - name: Setup JDK Corretto using cached gradle dependencies
        uses: actions/setup-java@v4
        with:
          distribution: 'corretto'
          java-version: 17
          cache: 'gradle'

      - name: Setup Gradle
        uses: gradle/actions/setup-gradle@v3
        with:
          gradle-version: 8.8

      - name: Build and test with gradle
        run: |
          cd ./backend
          ./gradlew clean bootJar

      - name: Login to DockerHub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_ID }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Build and push
        uses: docker/build-push-action@v6
        with:
          context: ./backend
          platforms: linux/amd64,linux/arm64
          push: true
          tags: ${{ secrets.DOCKERHUB_ID }}/review-me-app:release

  deploy:
    name: Deploy via self-hosted runner
    needs: build
    strategy:
      max-parallel: 1 # 동시에 하나의 runner만 실행한다.
      matrix:
        runner: [prod-a, prod-b]
    runs-on: [ self-hosted, "${{ matrix.runner }}" ]

    steps:
      - name: Checkout to secret repository
        uses: actions/checkout@v4
        with:
          repository: ${{ secrets.PRIVATE_REPOSITORY_URL }}
          token: ${{ secrets.PRIVATE_REPOSITORY_TOKEN }}

      - name: Move application-related files to local
        run: |
          mkdir -p ${{ env.APPLICATION_DIRECTORY }}/app
          mv ./app/* ./app/.* ${{ env.APPLICATION_DIRECTORY }}/app

      - name: Login to DockerHub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_ID }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Deploy new version # 변경 부분
        env:
          PROFILE_VAR: "prod"
        run: |
          chmod +x ./deploy.sh
          sudo ./deploy.sh

      working-directory: ${{ env.APPLICATION_DIRECTORY }}/app

[nayonsoso]

cf. GitHub Actions 에서 환경 변수를 설정하는 방법

env:
  GLOBAL_VAR: "This is a global variable"

jobs:
  build:
    runs-on: ubuntu-latest
    env:
      JOB_VAR: "This is a job-specific variable"
    
    steps:
    - uses: actions/checkout@v2
    - name: Run a script
      env:
        STEP_VAR: "This is a step-specific variable"
      run: |
        ./my_script.sh

이렇게 설정한 환경 변수는 각 scope 마다 어떤 곳에서든 (shell script 포함) 다 사용을 할 수 있다.
단, docker-compose.yml 은 같은 scope 에서 쓰는 것이라고 하더라도 GitHub Actions 의 환경 변수를 바로 사용할수는 없고,
.env 파일로 넣어줘야 한다.

[hyeonji1220]

이번 작업과는 관련없지만 삭제한 내용

도커를 사용하기 때문에 더이상 artifact 관련 변수는 필요없어서 겸사겸사 삭제하겠습니다~

[hyeonji1220]

2. CD (dev)

name: "[DEVELOP] CD using Github self-hosted runner"

on:
  workflow_dispatch:
  push:
    branches:
      - develop
    paths:
      - 'backend/**'

env:
  APPLICATION_DIRECTORY: /home/ubuntu/review-me

jobs:
  build:
    name: Build Dockerfile and push to DockerHub
    runs-on: ubuntu-latest

    steps:
      - name: Checkout to current repository
        uses: actions/checkout@v4

      - name: Setup JDK Corretto using cached gradle dependencies
        uses: actions/setup-java@v4
        with:
          distribution: 'corretto'
          java-version: 17
          cache: 'gradle'

      - name: Setup Gradle
        uses: gradle/actions/setup-gradle@v3
        with:
          gradle-version: 8.8

      - name: Build and test with gradle
        run: |
          cd ./backend
          ./gradlew clean bootJar

      - name: Login to DockerHub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_ID }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Build and push
        uses: docker/build-push-action@v6
        with:
          context: ./backend
          platforms: linux/amd64,linux/arm64
          push: true
          tags: ${{ secrets.DOCKERHUB_ID }}/review-me-app:develop

  deploy:
    name: Deploy via self-hosted runner
    needs: build
    runs-on: [self-hosted, dev]

    steps:
      - name: Checkout to secret repository
        uses: actions/checkout@v4
        with:
          repository: ${{ secrets.PRIVATE_REPOSITORY_URL }}
          token: ${{ secrets.PRIVATE_REPOSITORY_TOKEN }}

      - name: Move application-related files to local
        run: |
          mkdir -p ${{ env.APPLICATION_DIRECTORY }}/app
          mv ./app/* ./app/.* ${{ env.APPLICATION_DIRECTORY }}/app

      - name: Login to DockerHub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_ID }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Deploy new version # 변경 부분
        env:
          PROFILE_VAR: "dev"
        run: |
          chmod +x ./deploy.sh
          sudo ./deploy.sh

        working-directory: ${{ env.APPLICATION_DIRECTORY }}/app

[hyeonji1220]

docker-compose.yml

version: '3'
services:
  app-8080:
    image: reviewme/review-me-app:${DOCKER_TAG}
    container_name: review-me-app-8080
    ports:
      - "8080:8080"
      - "5959:5959"
    volumes:
      - ./application-${SPRING_PROFILE}.yml:/application.yml
      - /home/ubuntu/logs/:/logs
    environment:
      - SERVER_PORT=8080
      - ACTUATOR_PORT=5959
      - TZ=Asia/Seoul

  app-8081:
    image: reviewme/review-me-app:${DOCKER_TAG}
    container_name: review-me-app-8081
    ports:
      - "8081:8081"
      - "5960:5960"
    volumes:
      - ./application-${SPRING_PROFILE}.yml:/application.yml
      - /home/ubuntu/logs/:/logs
    environment:
      - SERVER_PORT=8081
      - ACTUATOR_PORT=5960
      - TZ=Asia/Seoul

[hyeonji1220]

질문

container_name을 기존 review-me-app → review-me-app-8080 or review-me-app-8081로 변경하였는데
다른 곳에서 container_name으로 식별하는 곳이 있을까요?

[hyeonji1220]

application-dev.yml

spring:
 profiles:
   active: dev

    config:
    import:
        - classpath:logback.yml
        - classpath:ports.yml

[nayonsoso]

ports.yml

server:
    port: ${SERVER_PORT}

management:
    server:
        port: ${ACTUATOR_PORT}