Skip to content

feat: Adding a connection pool to reduce the time spent opening connection to Postgres#4851

Open
Ma77Ball wants to merge 21 commits intoapache:mainfrom
Ma77Ball:fix/connection-pool
Open

feat: Adding a connection pool to reduce the time spent opening connection to Postgres#4851
Ma77Ball wants to merge 21 commits intoapache:mainfrom
Ma77Ball:fix/connection-pool

Conversation

@Ma77Ball
Copy link
Copy Markdown
Contributor

@Ma77Ball Ma77Ball commented May 3, 2026

What changes were proposed in this PR?

This PR adds a HikariCP connection pool to SqlServer.scala so that jOOQ queries borrow pre-authenticated connections from a pool instead of opening a new TCP connection and performing SCRAM-SHA-256 authentication on every database call.

What changed in SqlServer.scala:

  • Added a HikariConfig with maximumPoolSize=10, minimumIdle=2, connectionTimeout=30s, idleTimeout=10min, maxLifetime=30min (chosen to stay below typical PostgreSQL idle and load-balancer connection-reaping windows)
  • Initialized a HikariDataSource and built the jOOQ DSLContext via DSL.using(dataSource, SQL_DIALECT) so queries draw from the pool
  • Added a close() method to shut down the pool on application or test teardown, and updated clearInstance() to call it, so test classes that replace the singleton don't leak pool threads
  • Added a documented warning against caching the DSLContext in a val or lazy val. MockTexeraDB swaps the singleton between test classes, and a cached context would hold a stale reference to a dead pool

Any related issues, documentation, discussions?

closes: #4852

How was this PR tested?

  • Existing unit and integration tests pass — the pool-backed DSLContext is a drop-in replacement for the previous direct-driver context
  • Verified the MockTexeraDB flow still works across test classes by running the full test suite and confirming no "Connection refused" failures from stale contexts

Was this PR authored or co-authored using generative AI tooling?

Co-authored with Claude opus 4.6 (Anthropic) in compliance with ASF

@github-actions github-actions Bot added dependencies Pull requests that update a dependency file fix common labels May 3, 2026
@Ma77Ball Ma77Ball changed the title Adding a connection pool to reduce the time spent opening connection to Postgres refactor: Adding a connection pool to reduce the time spent opening connection to Postgres May 3, 2026
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented May 3, 2026

Codecov Report

❌ Patch coverage is 93.33333% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 43.21%. Comparing base (deb0e2c) to head (dd50017).

Files with missing lines Patch % Lines
...c/main/scala/org/apache/texera/dao/SqlServer.scala 93.33% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #4851      +/-   ##
============================================
+ Coverage     42.72%   43.21%   +0.49%     
+ Complexity     2185     1224     -961     
============================================
  Files          1031      764     -267     
  Lines         38152    30584    -7568     
  Branches       4004     2992    -1012     
============================================
- Hits          16301    13218    -3083     
+ Misses        20831    16692    -4139     
+ Partials       1020      674     -346     
Flag Coverage Δ *Carryforward flag
access-control-service 39.53% <ø> (ø)
agent-service 33.72% <ø> (ø) Carriedforward from deb0e2c
amber 45.57% <93.33%> (+2.36%) ⬆️
computing-unit-managing-service 0.00% <ø> (ø)
config-service 0.00% <ø> (ø)
file-service 32.18% <ø> (ø)
frontend 33.08% <ø> (ø) Carriedforward from deb0e2c
python 88.90% <ø> (ø) Carriedforward from deb0e2c
workflow-compiling-service 47.72% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Copy Markdown
Contributor

@Yicong-Huang Yicong-Huang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! Left inline comments

Comment thread common/dao/src/main/scala/org/apache/texera/dao/SqlServer.scala
Comment thread common/dao/src/main/scala/org/apache/texera/dao/SqlServer.scala Outdated
Comment thread common/dao/build.sbt Outdated
Comment thread common/dao/src/main/scala/org/apache/texera/dao/SqlServer.scala Outdated
@Yicong-Huang
Copy link
Copy Markdown
Contributor

please also change the title. this is not a refactor. we are adding a new feature.

@Ma77Ball Ma77Ball changed the title refactor: Adding a connection pool to reduce the time spent opening connection to Postgres feat: Adding a connection pool to reduce the time spent opening connection to Postgres May 4, 2026
@github-actions github-actions Bot added the platform Non-amber Scala service paths label May 4, 2026
@Ma77Ball Ma77Ball requested a review from Yicong-Huang May 5, 2026 00:03
Comment thread common/dao/build.sbt Outdated
@Ma77Ball Ma77Ball requested a review from Yicong-Huang May 7, 2026 01:59
Copy link
Copy Markdown
Contributor

@Yicong-Huang Yicong-Huang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks

@Yicong-Huang Yicong-Huang enabled auto-merge (squash) May 7, 2026 15:28
@Ma77Ball
Copy link
Copy Markdown
Contributor Author

Ma77Ball commented May 8, 2026

@Yicong-Huang, can you approve the workflows again?

auto-merge was automatically disabled May 8, 2026 21:43

Head branch was pushed to by a user without write access

@aicam aicam enabled auto-merge (squash) May 8, 2026 23:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common dependencies Pull requests that update a dependency file fix platform Non-amber Scala service paths

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add HikariCP connection pool to SqlServer

3 participants