Skip to content

feat(dbi): Add severity parsing and fix multiline/timestamp for PostgreSQL server logs#2209

Open
JayPolanco wants to merge 1 commit into
mainfrom
dbi-server-log-fix
Open

feat(dbi): Add severity parsing and fix multiline/timestamp for PostgreSQL server logs#2209
JayPolanco wants to merge 1 commit into
mainfrom
dbi-server-log-fix

Conversation

@JayPolanco

Copy link
Copy Markdown
Contributor

Description of the issue

DBI (Database Insights) server-logs filelog receiver had three issues:

  1. Multiline log entries were being fragmented into separate log records
  2. Timestamp parsing was not working (timeUnixNano was 0)
  3. Severity levels (LOG/ERROR/WARNING/etc.) were not being parsed

Additionally, there was a gosnowflake/dbus dependency leak from contrib.

Description of changes

  • Added WithSeverityPattern option to filelog translator
  • DBI translator now configures multiline, timestamp, and severity parsing for PostgreSQL stderr logs
  • Updated contrib version to 84054ba42340 to fix gosnowflake/dbus leak

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

  • Added TestTranslator_WithTimestampAndSeverity for filelog translator
  • Added TestDbiTranslateServerLogs_FilelogConfig for DBI translator
  • Updated golden YAML files
  • Verified on EC2 instance - logs now include severityNumber, severityText, and attributes.severity

Leak fixed

Created a bash script that does:

  #!/bin/bash
  for i in {1..5}; do
      sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a stop
      sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a start
      count=$(ps aux | grep 'dbus-daemon.*session' | grep -v grep | wc -l)
      echo "After restart $i: $count orphaned dbus-daemon processes"
  done
  $ bash test.sh
  After restart 1: 0 orphaned dbus-daemon processes
  After restart 2: 0 orphaned dbus-daemon processes
  After restart 3: 0 orphaned dbus-daemon processes
  After restart 4: 0 orphaned dbus-daemon processes
  After restart 5: 0 orphaned dbus-daemon processes

  $ ps aux | grep 'dbus-daemon.*session'
  ec2-user 21349  0.0  0.0   6756  2176 pts/0    S+   23:03   0:00 grep --color=auto dbus-daemon.*session

Also manually ran:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:config.json -s

CloudWatch Log Output

ERROR log:

{
    "timeUnixNano": 1784329491691000000,
    "severityNumber": 17,
    "severityText": "ERROR",
    "body": "2026-07-17 23:04:51.691 UTC testdb postgres [21991]ERROR:  relation \"nonexistent_table_12345\" does not exist at character 15",
    "attributes": {
        "timestamp": "2026-07-17 23:04:51.691 UTC",
        "severity": "ERROR"
    }
}

LOG (regular) log:

  {
      "timeUnixNano": 1784329520159000000,
      "severityNumber": 9,
      "severityText": "LOG",
      "body": "2026-07-17 23:05:20.159 UTC testdb postgres [22079]LOG:  duration: 200.869 ms  statement: SELECT pg_sleep(0.2);",
      "attributes": {
          "timestamp": "2026-07-17 23:05:20.159 UTC",
          "severity": "LOG"
      }
  }

Requirements

  1. Run make fmt and make fmt-sh
  2. Run make lint

@JayPolanco
JayPolanco requested a review from a team as a code owner July 17, 2026 23:46
@JayPolanco JayPolanco added the ready for testing Indicates this PR is ready for integration tests to run label Jul 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Binary Size Report

linux/amd64

Binary PR vs main (5d99671) vs v1.300070.0
amazon-cloudwatch-agent 193.2 MB ${\color{green}▼}$ 40.1 MB (-17.2%) ${\color{green}▼}$ 38.1 MB (-16.5%)
amazon-cloudwatch-agent-config-wizard 2.0 MB +0 B ${\color{red}▲}$ +86.0 KB (+4.5%)
config-downloader 2.0 MB +0 B ${\color{red}▲}$ +86.0 KB (+4.5%)
config-translator 2.0 MB +0 B ${\color{red}▲}$ +86.0 KB (+4.5%)
start-amazon-cloudwatch-agent 2.5 MB +0 B ${\color{red}▲}$ +81.9 KB (+3.4%)
workload-discovery 3.0 MB +0 B ${\color{red}▲}$ +86.0 KB (+2.9%)
Total 204.7 MB ${\color{green}▼}$ 40.1 MB (-16.4%) ${\color{green}▼}$ 37.7 MB (-15.6%)
linux/amd64 amazon-cloudwatch-agent (last 9 main commits + this PR)

 235 ┤                                        
     ┤▄▄▄ ███ ███ ███ ███ ███ ███ ███ ███     
     ┤███ ███ ███ ███ ███ ███ ███ ███ ███     
     ┤███ ███ ███ ███ ███ ███ ███ ███ ███     
     ┤███ ███ ███ ███ ███ ███ ███ ███ ███     
     ┤███ ███ ███ ███ ███ ███ ███ ███ ███     
     ┤███ ███ ███ ███ ███ ███ ███ ███ ███     
     ┤███ ███ ███ ███ ███ ███ ███ ███ ███     
 191 ┤███ ███ ███ ███ ███ ███ ███ ███ ███ ▄▄▄ 
 MB  └────────────────────────────────────────
      a53d263                             PR

Notable changes:

  • +1.6 MB at 98e058f
  • -40.1 MB at PR
linux/arm64
Binary PR vs main (5d99671) vs v1.300070.0
amazon-cloudwatch-agent 176.9 MB ${\color{green}▼}$ 35.3 MB (-16.6%) ${\color{green}▼}$ 34.5 MB (-16.3%)
amazon-cloudwatch-agent-config-wizard 2.0 MB +0 B ${\color{green}▼}$ 22 B (-0.0%)
config-downloader 2.0 MB +0 B ${\color{green}▼}$ 22 B (-0.0%)
config-translator 2.0 MB +0 B ${\color{green}▼}$ 22 B (-0.0%)
start-amazon-cloudwatch-agent 2.4 MB +0 B ${\color{red}▲}$ +65.5 KB (+2.8%)
workload-discovery 2.9 MB +0 B ${\color{red}▲}$ +65.5 KB (+2.3%)
Total 188.2 MB ${\color{green}▼}$ 35.3 MB (-15.8%) ${\color{green}▼}$ 34.4 MB (-15.5%)
windows/amd64
Binary PR vs main (5d99671) vs v1.300070.0
amazon-cloudwatch-agent.exe 196.0 MB ${\color{green}▼}$ 40.3 MB (-17.0%) ${\color{green}▼}$ 38.2 MB (-16.3%)
amazon-cloudwatch-agent-config-wizard.exe 2.1 MB +0 B ${\color{red}▲}$ +130.6 KB (+6.5%)
config-downloader.exe 2.1 MB +0 B ${\color{red}▲}$ +130.0 KB (+6.5%)
config-translator.exe 2.1 MB +0 B ${\color{red}▲}$ +130.6 KB (+6.5%)
start-amazon-cloudwatch-agent.exe 2.3 MB +0 B ${\color{red}▲}$ +112.1 KB (+5.2%)
workload-discovery.exe 2.9 MB +0 B ${\color{red}▲}$ +91.6 KB (+3.3%)
Total 207.6 MB ${\color{green}▼}$ 40.3 MB (-16.2%) ${\color{green}▼}$ 37.6 MB (-15.3%)
Investigating size changes

Use go-size-analyzer to compare binaries:

GOEXPERIMENT=jsonv2 go install github.com/Zxilly/go-size-analyzer/cmd/gsa@latest
gsa diff --old <baseline-binary> --new <new-binary>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready for testing Indicates this PR is ready for integration tests to run

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant