모니터링 인프라 구축: ELK + Prometheus/Grafana + Slack 알림#193
Conversation
- MDC traceId 필터 + Logback 롤링/gzip 압축 설정 추가 - Docker화: Dockerfile, docker-compose.yml (app+postgres+redis) - ELK 스택 추가 (Elasticsearch/Logstash/Kibana) - 로그 중앙 검색 - Actuator 보안설정(base-path 변경, 노출 엔드포인트 제한) + Micrometer/Prometheus 연동 - Grafana 대시보드 및 서버 다운 감지 Alert Rule 프로비저닝 (Discord 웹훅 연동, URL은 플레이스홀더) 로컬 docker-compose 검증 완료. 실제 배포/CI-CD 반영 전 리뷰 필요.
- contact-points.yml.tpl(플레이스홀더만 포함)을 커밋, 실제 URL은 .env(gitignore)로 로컬 주입 - docker-compose grafana 서비스에서 컨테이너 기동 시 sed로 치환 후 contact-points.yml 생성(비커밋) - .env.example로 필요한 환경변수 문서화 - 이전 커밋에 있던 실제 웹훅 URL은 push 전에 브랜치에서 제거됨(GitHub push protection에 의해 원격 유출은 없었음)
|
Warning Review limit reached
Next review available in: 44 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (9)
📝 WalkthroughWalkthroughAdds a Docker Compose monitoring stack around the Spring Boot application, including Prometheus metrics, Grafana dashboards and alerts, Logstash/Elasticsearch log aggregation, MDC request tracing, rolling logs, and local Slack webhook configuration. ChangesRunnect observability stack
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant RunnectApp
participant Prometheus
participant Grafana
participant Slack
Prometheus->>RunnectApp: Scrape /internal-monitor/prometheus
Grafana->>Prometheus: Evaluate server availability
Prometheus-->>Grafana: Return up metric
Grafana->>Slack: Deliver server-down notification
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
- src/main/resources 디렉토리가 이미 존재할 때 mkdir이 실패하던 문제 수정 (mkdir -p로 변경) - 이번 PR에서 logback-spring.xml을 추가하며 resources 디렉토리가 최초로 생겨서 노출된 버그 - CODEOWNERS에서 더 이상 팀에 참여하지 않는 인원(funnysunny08, RinRinPARK, YuSuhwa-ve) 제거
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
grafana/provisioning/alerting/contact-points.yml (1)
1-12: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winRemove or update this leftover Discord configuration.
The PR objectives indicate this stack uses Slack alerts, and your
notification-policies.ymlis correctly configured to route alerts toslack-alert. However, this static file defines adiscord-alertwith a dummy URL.If this file is dynamically overwritten at runtime by a script using
contact-points.yml.tpl, it should be removed from version control and added to.gitignore. Otherwise, Grafana provisioning will fail if it expectsslack-alertbut only finds this Discord configuration.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@grafana/provisioning/alerting/contact-points.yml` around lines 1 - 12, Remove the static discord-alert contact point from contact-points.yml and ensure the runtime-generated contact-points.yml is ignored by version control if contact-points.yml.tpl overwrites it. Otherwise, replace the Discord receiver with the expected slack-alert configuration so it matches notification-policies.yml.
🧹 Nitpick comments (6)
build.gradle (1)
34-36: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueOmit the explicit Micrometer version.
Spring Boot's dependency management automatically resolves the correct, compatible version of Micrometer for your specific Spring Boot version. Hardcoding the version can lead to compatibility conflicts during future Spring Boot upgrades.
♻️ Proposed fix
// Monitoring - Prometheus - implementation 'io.micrometer:micrometer-registry-prometheus:1.9.14' + implementation 'io.micrometer:micrometer-registry-prometheus'🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@build.gradle` around lines 34 - 36, Remove the explicit version from the Micrometer Prometheus dependency in the build configuration, leaving Spring Boot dependency management to select the compatible version..gitignore (1)
47-48: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueStop ignoring
contact-points.ymlif migrating to native variable interpolation.If you adopt Grafana's native
$__env{SLACK_WEBHOOK_URL}interpolation incontact-points.yml(as suggested in thedocker-compose.ymlreview), the file will no longer contain the actual webhook secret. It can then be safely versioned and committed, eliminating the need to ignore it.♻️ Proposed fix
.env -grafana/provisioning/alerting/contact-points.yml🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.gitignore around lines 47 - 48, Remove grafana/provisioning/alerting/contact-points.yml from .gitignore so the configuration can be versioned when it uses Grafana’s native $__env{SLACK_WEBHOOK_URL} interpolation instead of storing the webhook secret.docker-compose.yml (1)
11-26: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winAdd a healthcheck to Postgres to prevent the app from crashing on initial startup.
Using
depends_onsimply waits for the container to start, but not for the database inside it to be ready. If Postgres takes longer to initialize (e.g., on first run creating tables), the Spring Boot application will crash trying to connect.Adding a
healthcheckto Postgres and tyingdepends_ontoservice_healthyensures reliable startup orchestration.♻️ Proposed modifications
depends_on: - - postgres + postgres: + condition: service_healthy - redis volumes: - ./logs:/app/logs networks: - runnect-net postgres: image: postgres:15 container_name: runnect-postgres environment: POSTGRES_DB: runnect POSTGRES_USER: runnect POSTGRES_PASSWORD: runnect_local_password + healthcheck: + test: ["CMD-SHELL", "pg_isready -U runnect -d runnect"] + interval: 5s + timeout: 5s + retries: 5 ports: - "5432:5432"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docker-compose.yml` around lines 11 - 26, Add a Postgres healthcheck under the postgres service using pg_isready with the configured database and user, including suitable interval, timeout, and retry settings. Update the application service’s depends_on entry for postgres to require the service_healthy condition while preserving its existing Redis dependency.Dockerfile (2)
21-21: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low valuePrevent potential Docker build failures due to multiple JAR files.
Using the
*.jarwildcard to copy a source into a specific filename (app.jar) will cause a Docker build error (When using COPY with more than one source file, the destination must be a directory and end with a /) if the directory contains multiple.jarfiles. While safe here because onlybootJaris executed, if standardbuildis run later (producing a-plain.jar), this step will break.Consider being explicit with the target JAR name or excluding the plain jar.
♻️ Proposed fix to restrict the wildcard
-COPY --from=build /app/build/libs/*.jar app.jar +COPY --from=build /app/build/libs/*-SNAPSHOT.jar app.jar🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@Dockerfile` at line 21, Update the Dockerfile COPY source to select exactly one executable JAR from the build output, excluding the plain JAR or using the explicit boot JAR name, while preserving the destination filename app.jar.
17-19: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winRun the container as a non-root user.
By default, the
eclipse-temurin:11-jreimage runs as therootuser. Running the application as a non-root user is a security best practice that limits the potential impact of vulnerabilities within the container.🛡️ Proposed refactor to use a restricted user
FROM eclipse-temurin:11-jre + +RUN groupadd -r spring && useradd -r -g spring spring +USER spring:spring + WORKDIR /app🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@Dockerfile` around lines 17 - 19, Configure the Dockerfile run stage to execute the application as a dedicated non-root user instead of the image default root user. Create or reuse a restricted user with ownership of /app, then set the Docker USER before the application starts while preserving the existing WORKDIR.Source: Linters/SAST tools
src/main/java/org/runnect/server/config/logging/MdcLoggingFilter.java (1)
7-40: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse Spring's
OncePerRequestFilterinstead of standardFilter.Extending
OncePerRequestFilterguarantees that the filter is only executed once per request, properly handles asynchronous dispatches, and eliminates the need to manually cast theServletRequesttoHttpServletRequest(preventing a potentialClassCastException).♻️ Proposed refactor
-import javax.servlet.Filter; import javax.servlet.FilterChain; import javax.servlet.ServletException; -import javax.servlet.ServletRequest; -import javax.servlet.ServletResponse; import javax.servlet.http.HttpServletRequest; +import javax.servlet.http.HttpServletResponse; import java.io.IOException; import java.util.UUID; +import org.springframework.web.filter.OncePerRequestFilter; /** * 요청마다 traceId를 발급해 MDC에 담아둔다. * 로그 패턴에 %X{traceId}를 포함시키면(logback-spring.xml), 여러 요청이 뒤섞인 로그에서도 * 같은 traceId로 특정 요청의 흐름만 추적할 수 있다. */ `@Slf4j` `@Component` -public class MdcLoggingFilter implements Filter { +public class MdcLoggingFilter extends OncePerRequestFilter { private static final String TRACE_ID_KEY = "traceId"; private static final int TRACE_ID_LENGTH = 8; `@Override` - public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) + protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain chain) throws IOException, ServletException { String traceId = UUID.randomUUID().toString().substring(0, TRACE_ID_LENGTH); try { MDC.put(TRACE_ID_KEY, traceId); - HttpServletRequest httpRequest = (HttpServletRequest) request; - log.info("{} {}", httpRequest.getMethod(), httpRequest.getRequestURI()); + log.info("{} {}", request.getMethod(), request.getRequestURI()); chain.doFilter(request, response); } finally { MDC.remove(TRACE_ID_KEY); } } }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/main/java/org/runnect/server/config/logging/MdcLoggingFilter.java` around lines 7 - 40, Update MdcLoggingFilter to extend Spring’s OncePerRequestFilter and move its trace-ID, request logging, and MDC cleanup logic into the framework’s HTTP request method without manually casting ServletRequest. Preserve the existing TRACE_ID_KEY handling, generated ID length, logging, and finally-based cleanup.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docker-compose.yml`:
- Around line 96-101: Remove the Grafana service’s custom entrypoint and command
that run sed and overwrite the provisioning file. Update the contact-points
provisioning configuration to reference $__env{SLACK_WEBHOOK_URL} directly,
rename or use it as contact-points.yml instead of the template, and delete the
obsolete .tpl file if it is tracked.
In `@logstash.conf`:
- Around line 1-6: Update the file input configuration to add a multiline codec
that groups stack-trace continuation lines with each leading timestamped log
entry, ensuring exception records are passed as complete events to the existing
parsing pipeline.
- Around line 23-25: Update the index format in the Elasticsearch output
configuration to use calendar-year formatting by replacing the week-based YYYY
token with yyyy in the runnect-logs date pattern, while preserving the existing
month and day components.
- Around line 4-5: Update the file input configuration using start_position and
sincedb_path so Logstash stores the file offset on persistent storage instead of
/dev/null, preserving offsets across restarts and preventing old events from
being reindexed.
- Around line 16-18: Update the date filter parsing log_timestamp to explicitly
use the Asia/Seoul timezone by adding the timezone setting alongside match and
target, preserving the existing timestamp format and `@timestamp` target.
---
Outside diff comments:
In `@grafana/provisioning/alerting/contact-points.yml`:
- Around line 1-12: Remove the static discord-alert contact point from
contact-points.yml and ensure the runtime-generated contact-points.yml is
ignored by version control if contact-points.yml.tpl overwrites it. Otherwise,
replace the Discord receiver with the expected slack-alert configuration so it
matches notification-policies.yml.
---
Nitpick comments:
In @.gitignore:
- Around line 47-48: Remove grafana/provisioning/alerting/contact-points.yml
from .gitignore so the configuration can be versioned when it uses Grafana’s
native $__env{SLACK_WEBHOOK_URL} interpolation instead of storing the webhook
secret.
In `@build.gradle`:
- Around line 34-36: Remove the explicit version from the Micrometer Prometheus
dependency in the build configuration, leaving Spring Boot dependency management
to select the compatible version.
In `@docker-compose.yml`:
- Around line 11-26: Add a Postgres healthcheck under the postgres service using
pg_isready with the configured database and user, including suitable interval,
timeout, and retry settings. Update the application service’s depends_on entry
for postgres to require the service_healthy condition while preserving its
existing Redis dependency.
In `@Dockerfile`:
- Line 21: Update the Dockerfile COPY source to select exactly one executable
JAR from the build output, excluding the plain JAR or using the explicit boot
JAR name, while preserving the destination filename app.jar.
- Around line 17-19: Configure the Dockerfile run stage to execute the
application as a dedicated non-root user instead of the image default root user.
Create or reuse a restricted user with ownership of /app, then set the Docker
USER before the application starts while preserving the existing WORKDIR.
In `@src/main/java/org/runnect/server/config/logging/MdcLoggingFilter.java`:
- Around line 7-40: Update MdcLoggingFilter to extend Spring’s
OncePerRequestFilter and move its trace-ID, request logging, and MDC cleanup
logic into the framework’s HTTP request method without manually casting
ServletRequest. Preserve the existing TRACE_ID_KEY handling, generated ID
length, logging, and finally-based cleanup.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: c13f36a8-c991-418a-a529-8b967f50d444
📒 Files selected for processing (16)
.env.example.gitignoreDockerfilebuild.gradledocker-compose.ymlgrafana/provisioning/alerting/contact-points.ymlgrafana/provisioning/alerting/contact-points.yml.tplgrafana/provisioning/alerting/notification-policies.ymlgrafana/provisioning/alerting/rules.ymlgrafana/provisioning/dashboards/dashboard.ymlgrafana/provisioning/dashboards/json/runnect-overview.jsongrafana/provisioning/datasources/prometheus.ymllogstash.confprometheus.ymlsrc/main/java/org/runnect/server/config/logging/MdcLoggingFilter.javasrc/main/resources/logback-spring.xml
- Grafana 웹훅 주입을 커스텀 entrypoint(sed) 대신 네이티브 $__env{} 문법으로 단순화 (bind mount 권한 이슈 회피)
- Logstash: 멀티라인 코덱 추가로 스택트레이스가 leading 로그 라인과 분리되지 않도록 수정
- Logstash: sincedb_path를 /dev/null에서 기본 영구 경로로 변경 + 볼륨 추가 (재시작 시 전체 재색인 방지)
- Logstash: date filter에 timezone => Asia/Seoul 명시 (Logback이 KST naive 타임스탬프를 기록하므로)
- Logstash: 인덱스 패턴을 YYYY(week-based year)에서 yyyy(calendar year)로 수정 (연말/연초 오분류 방지)
모든 수정사항은 로컬 docker-compose로 재검증 완료:
- $__env{} 방식으로도 실제 장애 시뮬레이션 → Slack 알림 firing 확인
- 스택트레이스 포함 500 에러가 하나의 이벤트(6868자)로 정상 색인됨 확인
- 인덱스명이 runnect-logs-2026.07.21 형식(소문자)으로 생성됨 확인
2 participants
작업 배경
변경 사항
Dockerfile,docker-compose.ymlconfig/logging/MdcLoggingFilterlogback-spring.xmldocker-compose.yml(ELK)logstash.conf로 로그 grok 파싱 후 날짜별 인덱스 색인application.yml(관리형, 외부)build.gradlemicrometer-registry-prometheus추가prometheus.yml,docker-compose.yml(Prometheus/Grafana)grafana/provisioning/alerting/*up==0) Alert Rule, Slack Contact Point(웹훅은 커밋하지 않고.env+런타임 sed 치환으로 주입)영향 범위
/actuator/*를 직접 참조하는 외부 연동(있다면)은 확인 필요docker-compose up으로 실행 시에만 영향, 실제 배포 파이프라인(CI/CD, EC2)은 이번 PR에서 변경하지 않음검증 매트릭스
(테스트 코드 변경 없음 — 아래는 로컬 docker-compose 환경에서 수행한 수동 검증 내역)
/internal-monitor/health)runnect-server→up상태 확인docker stop으로 장애 시뮬레이션 → Prometheusup=0감지 → Grafana Alertpending→firing전환 → Slack 채널 실제 수신 확인Test Plan
docker-compose up으로 전체 스택(8개 컨테이너) 기동 확인