Skip to content

fix: Iceberg lookup double-encodes user_meta (0.19.1)#400

Open
drernie wants to merge 3 commits into
mainfrom
fix/iceberg-user-meta-double-encoding
Open

fix: Iceberg lookup double-encodes user_meta (0.19.1)#400
drernie wants to merge 3 commits into
mainfrom
fix/iceberg-user-meta-double-encoding

Conversation

@drernie

@drernie drernie commented Jul 14, 2026

Copy link
Copy Markdown
Member

Fixes #399.

The bucketless Iceberg lookup projected the metadata column as json_format(CAST(m.metadata AS JSON)), but m.metadata is a VARCHAR that already contains a JSON document — and Trino's VARCHAR→JSON cast wraps the string as a JSON string value instead of parsing it. Every row returned by the Iceberg path therefore came back double-encoded: json.loads yielded a str, _parse_user_meta logged "Athena metadata JSON was not an object", and the result's metadata silently degraded to the matched {key: value} fallback. Deterministic on every lookup, first observed during live verification of quiltdata/deployment#2502 on tf-dev-bucketless.

Canvas rendering was unaffected (it only consumes bucket + package name), so this is log noise plus a latent trap for future metadata consumers — hence a patch release, not a re-pin of the 0.19.0 deployment.

Changes

  • _build_iceberg_union_query: project m.metadata AS user_meta raw, matching the legacy _packages-view path (which selects user_meta directly and never had this problem). The WHERE json_extract_scalar(...) filter was always correct — json_extract_scalar does parse VARCHAR — which is why matching worked while the projection double-encoded.
  • _parse_user_meta: if the first decode yields a str, defensively decode once more before warning, so any double-encoded values still round-trip to the full dict.
  • Tests: the union-query test now asserts the raw projection (it previously asserted the buggy json_format(CAST(...)) form); new test covers double-encoded user_meta round-tripping to the full metadata dict.
  • Version bump 0.19.0 → 0.19.1 + changelog.

Test plan

  • docker/: 468 Python tests pass, including the new double-encode round-trip test and test_versions_match
  • Existing non-object-metadata fallback test ('["EXP-1"]') still passes — genuinely non-object metadata keeps the warning + fallback behavior

🤖 Generated with Claude Code

Greptile Summary

This PR fixes Iceberg lookup metadata decoding for bucketless package searches. The main changes are:

  • Projects Iceberg m.metadata as raw user_meta instead of wrapping it with json_format.
  • Adds a defensive second JSON decode when user_meta arrives double-encoded.
  • Updates Iceberg lookup tests for raw projection and double-encoded metadata recovery.
  • Bumps the app, Python package, lockfile, and npm package versions to 0.19.1.
  • Adds the 0.19.1 changelog entry.

Confidence Score: 5/5

This looks safe to merge.

  • No blocking issues found in the changed code.

Important Files Changed

Filename Overview
docker/src/package_query.py Changes Iceberg lookup projection to return raw metadata and adds double-encoded JSON recovery.
docker/tests/test_package_query.py Updates Iceberg SQL assertions and adds coverage for double-encoded metadata parsing.
CHANGELOG.md Documents the 0.19.1 metadata decoding fix.
docker/app-manifest.yaml Bumps the app manifest version to 0.19.1.
docker/pyproject.toml Bumps the Python package version to 0.19.1.
docker/uv.lock Aligns the locked editable package version with 0.19.1.
package.json Bumps the npm package version to 0.19.1.

Reviews (1): Last reviewed commit: "docs: changelog for 0.19.1" | Re-trigger Greptile

drernie and others added 3 commits July 13, 2026 20:54
…#399)

json_format(CAST(m.metadata AS JSON)) wraps the already-JSON varchar as a
JSON string value instead of parsing it, so every Iceberg-path row warned
"Athena metadata JSON was not an object" and degraded result metadata to
the {key: value} fallback. Select the column raw like the parquet
_packages-view path, and decode a double-encoded string defensively in
_parse_user_meta.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Iceberg lookup double-encodes user_meta, degrading result metadata to the fallback on every row

1 participant