Connectors: Add source and target adapters for Delta Lake

[amotl]

About

Add support for reading from and writing to Delta Lake tables.

Backlog

• Documentation
• Software tests

References

• Delta Lake #104

[read-the-docs-community]

Documentation build overview

📚 omniload | 🛠️ Build #33898640 | 📁 Comparing b43360f against latest (b48a7f5)

  🔍 Preview build  

65 files changed · + 39 added · ± 20 modified · - 6 deleted

+ Added

• decisions/ADR-001-stage-remote-file-databases.html
• supported-sources/amqp.html
• supported-sources/azure-storage.html
• supported-sources/bson.html
• supported-sources/cbor.html
• supported-sources/databricks-files.html
• supported-sources/databricks-warehouse.html
• supported-sources/dropbox.html
• supported-sources/file.html
• supported-sources/filesystem.html
• and 29 more...

± Modified

• backlog.html
• changelog.html
• index.html
• commands/ingest.html
• getting-started/incremental-loading.html
• getting-started/python-api.html
• supported-sources/athena.html
• supported-sources/bigquery.html
• supported-sources/cratedb.html
• supported-sources/duckdb.html
• and 10 more...

- Deleted

• supported-sources/custom_queries.html
• supported-sources/databricks.html
• supported-sources/gcs.html
• supported-sources/google_analytics.html
• supported-sources/gsheets.html
• supported-sources/postgres.html

[codecov]

Codecov Report

❌ Patch coverage is 37.50000% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.18%. Comparing base (a9b8429) to head (b43360f).

Files with missing lines	Patch %	Lines
src/omniload/target/deltalake/__init__.py	40.00%	9 Missing ⚠️
src/omniload/source/deltalake/api.py	33.33%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #109      +/-   ##
==========================================
- Coverage   60.23%   60.18%   -0.05%     
==========================================
  Files         236      238       +2     
  Lines       11157    11181      +24     
==========================================
+ Hits         6720     6729       +9     
- Misses       4437     4452      +15     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[coderabbitai]

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one) (1)

• coderabbit-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 53adfcc2-b621-4d5e-afe4-c4214391b3d7

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch deltalake

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[amotl]

Hi @hampsterx. I will be travelling starting tomorrow. If you like this feature, feel free to complete this patch, or let your clanker do the job. It looks like I was not able to complete it in time.

NB: GH-104 might include more hints or thoughts about what's left to do, other than the backlog enumeration in the original post.