update: kafka

[parmesant]

• add rayon to kafka ingestion path
• update deps

Fixes #XXXX.

Description

---

This PR has:

• been tested to ensure log ingestion and log query works.
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added documentation for new or modified features or behaviors.

Summary by CodeRabbit

• Improvements
  • Updated the Kafka connector for improved compatibility with current Kafka capabilities.
  • Added support for zstd-compressed Kafka records.
  • Improved Kafka event processing efficiency, helping reduce processing overhead for larger workloads.
• Bug Fixes
  • Corrected validation coverage for stream names containing path separators.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 002a102e-7480-4196-ba8c-efaf308c57a3

📥 Commits

Reviewing files that changed from the base of the PR and between 044274d and d48ae93.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (3)

• Cargo.toml
• src/connectors/kafka/processor.rs
• src/handlers/http/users/filters.rs

---

Walkthrough

Kafka dependencies and zstd feature wiring were updated. Kafka event processing now separates stream preparation from CPU-bound event construction, using Rayon and a oneshot channel. A stream-name validation test expectation was reformatted.

Changes

Kafka connector processing

Layer / File(s)	Summary
Kafka dependency and feature wiring Cargo.toml	Kafka, SASL, and AWS MSK IAM dependencies were updated, and the rdkafka/zstd feature was enabled.
Stream preparation and event processing src/connectors/kafka/processor.rs	Stream metadata resolution was separated from event construction, which now runs in Rayon and returns through a oneshot channel.

Stream-name validation test

Layer / File(s)	Summary
Invalid stream-name assertion src/handlers/http/users/filters.rs	The path-like stream-name test now matches the InvalidStreamName payload for /.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

• parseablehq/parseable#1727: Updates related stream-name validation behavior and its test expectations.

Suggested reviewers: nikhilsinhaparseable

Poem

I’m a rabbit hopping through Kafka’s stream,
With zstd tucked neatly in the dream.
Metadata first, then events take flight,
Rayon helps them process right.
A slash-shaped test now lands just right.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description is largely just template text and lacks the required goal, rationale, key changes, testing, and documentation details.	Fill in the Description section with the PR goal, chosen approach, key changes, and complete the checklist items or remove irrelevant template text.
Title check	❓ Inconclusive	The title is too vague and does not clearly describe the main Kafka ingestion and dependency update changes.	Use a more specific title like "Optimize Kafka ingestion with Rayon and dependency updates".

✅ Passed checks (3 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}