Skip to content

fix: 배포 헬스체크를 로그 기반 완료 신호 감지 방식으로 변경 #196

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
unam98 wants to merge 1 commit into
base: dev
Choose a base branch
from
Open

fix: 배포 헬스체크를 로그 기반 완료 신호 감지 방식으로 변경 #196

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 35 additions & 12 deletions scripts/deploy.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,28 +30,51 @@ else
echo "> Grafana OTel agent 미설정, 모니터링 없이 실행"
fi

LOG_FILE=/home/ec2-user/app/nohup.out
START_LINE=$(wc -l < "$LOG_FILE" 2>/dev/null || echo 0)

nohup java $JAVA_AGENT_OPTS -jar \
-Duser.timezone=Asia/Seoul \
$JAR_PATH \
>> /home/ec2-user/app/nohup.out 2>&1 &
>> "$LOG_FILE" 2>&1 &
APP_PID=$!
echo "> 프로세스 시작됨 (PID: $APP_PID)"

echo "> 15초 후 헬스체크 시작"
sleep 15
echo "> 앱 준비 신호 대기 중 (최대 300초, 프로세스 생존 여부로 실패 판단)"
MAX_WAIT=300
ELAPSED=0
READY=false

for i in {1..10}; do
RESPONSE=$(curl -s http://localhost:8080/actuator/health || true)
if echo "$RESPONSE" | grep -q '"status":"UP"'; then
echo "> 헬스체크 성공"
break
fi
echo "> 헬스체크 실패($i/10): $RESPONSE"
if [ $i -eq 10 ]; then
while [ $ELAPSED -lt $MAX_WAIT ]; do
if ! kill -0 $APP_PID 2>/dev/null; then
echo "> 프로세스가 예기치 않게 종료됨 (크래시)"
echo "> 최근 로그:"
tail -n 40 "$LOG_FILE"
echo "> 배포 실패"
exit 1
fi
sleep 10

if tail -n +"$((START_LINE + 1))" "$LOG_FILE" | grep -q "Started ServerApplication"; then
echo "> 앱 준비 완료 신호 감지 (${ELAPSED}초 소요)"
READY=true
break
fi

sleep 2
ELAPSED=$((ELAPSED + 2))
done

if [ "$READY" != "true" ]; then
echo "> ${MAX_WAIT}초 내에 준비 신호 없음 — 안전장치 발동"
echo "> 최근 로그:"
tail -n 40 "$LOG_FILE"
echo "> 배포 실패"
exit 1
fi
Comment on lines +43 to +73

@coderabbitai coderabbitai Bot Jul 21, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

MAX_WAIT=300 matches the AppSpec hook's own 300s timeout, leaving no margin.

appspec.yml runs this script with timeout: 300 for the whole AfterInstall hook. This script's own readiness loop can consume the full 300s (line 44), on top of the earlier sleep 5 (line 14) for killing the previous process and the subsequent health curl (line 75) and nginx start. If the app is slow enough to approach MAX_WAIT, CodeDeploy will kill the entire script via its outer timeout before — or right around when — this loop's own graceful timeout branch (lines 67-73) would fire, so the intended "print log tail, exit 1 cleanly" diagnostics may never actually run. The PR's stated intent to keep the "5-minute limit for hang situations" while leaving CI/CD unchanged means this exact overlap should be accounted for.

Reduce MAX_WAIT to leave real margin under the 300s hook budget (e.g., 240-270s) so the script's own failure path reliably completes before CodeDeploy's outer timeout intervenes.

🛡️ Suggested fix 
-MAX_WAIT=300
+MAX_WAIT=240

As per PR objectives, the 5-minute (300s) limit is intentionally retained for hang protection, but this needs coordination with the AppSpec hook's own 300s budget rather than an exact 1:1 match.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
echo "> 앱 준비 신호 대기 중 (최대 300초, 프로세스 생존 여부로 실패 판단)"
MAX_WAIT=300
ELAPSED=0
READY=false
for i in {1..10}; do
RESPONSE=$(curl -s http://localhost:8080/actuator/health || true)
if echo "$RESPONSE" | grep -q '"status":"UP"'; then
echo "> 헬스체크 성공"
break
fi
echo "> 헬스체크 실패($i/10): $RESPONSE"
if [ $i -eq 10 ]; then
while [ $ELAPSED -lt $MAX_WAIT ]; do
if ! kill -0 $APP_PID 2>/dev/null; then
echo "> 프로세스가 예기치 않게 종료됨 (크래시)"
echo "> 최근 로그:"
tail -n 40 "$LOG_FILE"
echo "> 배포 실패"
exit 1
fi
sleep 10
if tail -n +"$((START_LINE + 1))" "$LOG_FILE" | grep -q "Started ServerApplication"; then
echo "> 앱 준비 완료 신호 감지 (${ELAPSED}초 소요)"
READY=true
break
fi
sleep 2
ELAPSED=$((ELAPSED + 2))
done
if [ "$READY" != "true" ]; then
echo "> ${MAX_WAIT}초 내에 준비 신호 없음 — 안전장치 발동"
echo "> 최근 로그:"
tail -n 40 "$LOG_FILE"
echo "> 배포 실패"
exit 1
fi
echo "> 앱 준비 신호 대기 중 (최대 300초, 프로세스 생존 여부로 실패 판단)"
MAX_WAIT=240
ELAPSED=0
READY=false
while [ $ELAPSED -lt $MAX_WAIT ]; do
if ! kill -0 $APP_PID 2>/dev/null; then
echo "> 프로세스가 예기치 않게 종료됨 (크래시)"
echo "> 최근 로그:"
tail -n 40 "$LOG_FILE"
echo "> 배포 실패"
exit 1
fi
if tail -n +"$((START_LINE + 1))" "$LOG_FILE" | grep -q "Started ServerApplication"; then
echo "> 앱 준비 완료 신호 감지 (${ELAPSED}초 소요)"
READY=true
break
fi
sleep 2
ELAPSED=$((ELAPSED + 2))
done
if [ "$READY" != "true" ]; then
echo "> ${MAX_WAIT}초 내에 준비 신호 없음 — 안전장치 발동"
echo "> 최근 로그:"
tail -n 40 "$LOG_FILE"
echo "> 배포 실패"
exit 1
fi
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/deploy.sh` around lines 43 - 73, Reduce MAX_WAIT in the readiness
loop to leave sufficient time below the AppSpec hook’s 300-second timeout for
the earlier shutdown delay, health check, and cleanup diagnostics; use a margin
such as 240–270 seconds while preserving the existing timeout failure path and
5-minute overall hang-protection intent.


RESPONSE=$(curl -s http://localhost:8080/actuator/health || true)
echo "> 헬스체크 확인: $RESPONSE"

Comment on lines +75 to +77

@coderabbitai coderabbitai Bot Jul 21, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Health curl has no timeout and its result doesn't gate success.

curl -s http://localhost:8080/actuator/health || true can hang indefinitely if the port isn't accepting connections yet, further eating into the already-tight 300s hook budget (see readiness-loop comment above). It also never fails the deploy even if the response is empty or reports DOWN, since only the raw string is echoed.

Add a request timeout so this can't block indefinitely, regardless of whether you decide to gate success on the response.

🛡️ Suggested fix 
-RESPONSE=$(curl -s http://localhost:8080/actuator/health || true)
+RESPONSE=$(curl -s --max-time 10 http://localhost:8080/actuator/health || true)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
RESPONSE=$(curl -s http://localhost:8080/actuator/health || true)
echo "> 헬스체크 확인: $RESPONSE"
RESPONSE=$(curl -s --max-time 10 http://localhost:8080/actuator/health || true)
echo "> 헬스체크 확인: $RESPONSE"
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/deploy.sh` around lines 75 - 77, Update the health-check curl
invocation in the deployment script to include an explicit request timeout,
ensuring an unavailable or unresponsive actuator endpoint cannot block
indefinitely. Keep the existing health response logging behavior and do not
change deployment success gating.

echo "> Nginx 시작"
sudo systemctl start nginx || true
sudo systemctl enable nginx || true
Expand Down
Loading