Add support for JSON database columns backed by non-string .NET types

[chullybun]

Summary

Adds first-class support for JSON-typed database columns that convert to/from a .NET type other than the default string mapping — proving the pattern end-to-end across both a simple CRUD domain and a DDD aggregate domain, and documenting it so it's repeatable.

What's included

• Naming convention: a JSON-backed column is identified by a Json (SQL Server, PascalCase) / _json (Postgres, snake_case) suffix, or by using the database's native JSON type. dbex.yaml wires the column to a typed CLR property via TypeToJsonStringEfConverter<T>, which serializes/deserializes through System.Text.Json at the EF Core layer.
• Default column typing: JSON columns default to bounded text (e.g. NVARCHAR(n)/VARCHAR(n)) consistent with every other string column in the schema. Unbounded (MAX) or native database JSON types (e.g. Postgres JSONB) are supported as an explicit, opt-in override — demonstrated in the Products sample.
• Contract/persistence/mapper pattern: a JSON-backed value is always represented as a matched contract type + persistence type pair with a corresponding mapper. For DDD aggregates, an intermediate Domain value object sits between the two, requiring a three-type chain (Contract ↔ Domain value object ↔ Persistence model) and two mappers — one per layer boundary. Demonstrated with Basket.ShippingAddress in the Shopping sample.
• Two worked examples:
  • Products (CRUD, PostgreSQL) — a List<string>? tags collection stored as native JSONB.
  • Shopping (DDD aggregate, SQL Server) — a ShippingAddress value object stored as bounded JSON text.
• AI instructions/skills updated so this pattern is discoverable and repeatable going forward: DbEx tooling guidance (including the new Inspect output Json: Yes|No flag), Domain layer conventions for JSON-backed value objects, Application-layer mapping conventions for the Domain↔Contract mapper half, and the corresponding coreex-db-migration, coreex-aggregate, and coreex-app-service skills.
• A core library correctness fix and accompanying regression tests in the JSON string converter used by the EF Core value-conversion pipeline.

Validation

• dotnet build CoreEx.sln — succeeds with 0 errors (net8.0/net9.0/net10.0).
• CoreEx.Test.Unit — full suite passes (719/719) across all target frameworks.
• Integration tests requiring live SQL Server/Postgres/Service Bus infra were not run in this environment; recommend running those once local dependency containers are available.

[Copilot]

Pull request overview

This PR extends CoreEx/DbEx/EF Core to support JSON-backed database columns mapped to non-string CLR types via a new TypeToJsonStringEfConverter<T> (and underlying TypeToJsonStringConverter<T>), demonstrates the end-to-end pattern in the Products (CRUD) and Shopping (DDD aggregate) samples, and updates skills/instructions to document the approach. It also includes RuntimeMetadata cycle-safety improvements and request-body handling changes in WebApiRequestOptions.

Changes:

• Add typed JSON↔string converters and wire them into DbEx EF model generation for JSON columns.
• Demonstrate JSON-backed properties in samples: Products Tags (List<string>? via Postgres JSONB) and Shopping Basket.ShippingAddress (DDD value object via SQL Server NVARCHAR(2000) JSON).
• Improve deep graph safety in RuntimeMetadata (cycle detection) and update tests; adjust web API request-body validation/assertion helpers.

Reviewed changes

Copilot reviewed 76 out of 79 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/CoreEx.Test.Unit/Runtime/RuntimeMetadataTests.cs	Adds regression tests for cycles/cleaning.
tests/CoreEx.Test.Unit/Mapping/Converters/TypeToJsonStringConverterTests.cs	New unit tests for JSON converter.
tests/CoreEx.RefData.Test.Unit/ReferenceDataValidationTests.cs	Adds case-sensitivity validation coverage.
tests/CoreEx.AspNetCore.Test.Unit/PersonApi_QueryTestsBase.cs	Updates query tests for casing changes.
tests/CoreEx.AspNetCore.Test.Unit/PersonApi_MutateTestsBase.cs	Updates missing-body error assertions.
src/CoreEx/Metadata/RuntimeMetadata.IsDefault.cs	Adds cycle detection to IsDefault.
src/CoreEx/Metadata/RuntimeMetadata.GetHashCode.cs	Adds cycle detection to GetHashCode.
src/CoreEx/Metadata/RuntimeMetadata.Clean.cs	Adds cycle detection + value-type handling.
src/CoreEx/Metadata/RuntimeMetadata.AreEqual.cs	Adds cycle detection to deep equals.
src/CoreEx/Mapping/Converters/TypeToJsonStringConverter.cs	Introduces typed JSON string converter.
src/CoreEx.Validation/ValidatorT.cs	Formatting-only change.
src/CoreEx.Validation/Rules/ReferenceDataRule.cs	Formatting-only change.
src/CoreEx.UnitTesting/UnitTestExExtensions.Assert.cs	Replaces title-only ProblemDetails assert with callback-based assert.
src/CoreEx.Template/content/CoreEx.Core/_Directory.Packages.props	Updates DbEx package versions in template.
src/CoreEx.RefData/ReferenceDataCollectionT2.cs	Documents new default code comparer behavior.
src/CoreEx.RefData/ReferenceDataCollectionT.cs	Documents new default code comparer behavior.
src/CoreEx.RefData/Abstractions/ReferenceDataCollectionCore.cs	Changes default refdata code comparer to Ordinal.
src/CoreEx.EntityFrameworkCore/Converters/TypeToJsonStringEfConverter.cs	Adds EF ValueConverter bridge for typed JSON.
src/CoreEx.EntityFrameworkCore/Converters/JsonElementStringEfConverter.cs	Formatting-only change.
src/CoreEx.Database/Templates/EfModelBuilder_cs.hbs	Auto-wires typed JSON conversion for JSON columns.
src/CoreEx.AspNetCore/WebApiRequestResponseOptions.cs	Adds auto-cleaning + new required-body behavior.
src/CoreEx.AspNetCore/WebApiRequestOptions.cs	Adds auto-cleaning + new required-body behavior.
src/CoreEx.AspNetCore/Abstractions/WebApiBase.cs	Centralizes request-body ProblemDetails constants.
src/CoreEx.AspNetCore/Abstractions/WebApi.cs	Adjusts request-body detection and invalid-body error.
src/CoreEx.AspNetCore/Abstractions/IWebApiRequestOptions.cs	Adds AutoCleanValue to request options contract.
samples/tests/Contoso.Shopping.Test.Unit/Validators/AddressValidatorTests.cs	Adds validator unit tests for Address.
samples/tests/Contoso.Shopping.Test.Unit/Domains/BasketTests.cs	Adds domain tests for shipping address behavior.
samples/tests/Contoso.Shopping.Test.Common/Data/read-data.seed.yaml	Seeds shopping baskets with ShippingAddressJson.
samples/tests/Contoso.Shopping.Test.Common/Data/mutate-data.seed.yaml	Seeds mutate baskets with ShippingAddressJson.
samples/tests/Contoso.Shopping.Test.Api/Resources/Basket_Get_Found.res.json	Adds shippingAddress to expected basket response.
samples/tests/Contoso.Shopping.Test.Api/MutateTests.BasketItem.cs	Updates ProblemDetails assertion style.
samples/tests/Contoso.Shopping.Test.Api/MutateTests.Basket.cs	Adds shipping-address endpoint tests + checkout guard test.
samples/tests/Contoso.Products.Test.Common/Data/read-data.seed.yaml	Seeds products with tags_json JSON.
samples/tests/Contoso.Products.Test.Common/Data/mutate-data.seed.yaml	Seeds products with tags_json JSON.
samples/tests/Contoso.Products.Test.Api/Resources/ReadTests/Product_Get_Found.res.json	Adds tags to expected product response.
samples/tests/Contoso.Products.Test.Api/Resources/ProductMutateTests/Create_WithTags.res.json	New expected JSON for tagged product create.
samples/tests/Contoso.Products.Test.Api/ReadTests.ProductQuery.cs	Updates filters for casing changes.
samples/tests/Contoso.Products.Test.Api/ReadTests.MovementQuery.cs	Updates filters for casing changes.
samples/tests/Contoso.Products.Test.Api/ProductMutateTests.Delete.cs	Updates ProblemDetails assertion style.
samples/tests/Contoso.Products.Test.Api/ProductMutateTests.Create.cs	Adds create-with-tags API test + casing updates.
samples/tests/Contoso.Products.Test.Api/MovementMutateTests.Reserve.cs	Updates ProblemDetails assertion style.
samples/tests/Contoso.E2E.Runner/Scenarios/ShoppingBasketScenario.cs	Adds shipping-address step to E2E scenario.
samples/src/Contoso.Shopping.Infrastructure/Repositories/ShoppingDbContext.g.cs	Generated EF mapping for ShippingAddress JSON column.
samples/src/Contoso.Shopping.Infrastructure/Persistence/Basket.g.cs	Generated persistence property for ShippingAddress.
samples/src/Contoso.Shopping.Infrastructure/Persistence/Address.cs	Adds persistence POCO for JSON storage.
samples/src/Contoso.Shopping.Infrastructure/Mapping/BasketMapper.cs	Maps ShippingAddress Persistence→Domain.
samples/src/Contoso.Shopping.Infrastructure/Mapping/BasketIntoMapper.cs	Maps ShippingAddress Domain→Persistence.
samples/src/Contoso.Shopping.Infrastructure/Mapping/AddressMapper.cs	Adds persistence↔domain mapper for Address.
samples/src/Contoso.Shopping.Domain/ValueObjects/Address.cs	Adds Address domain value object.
samples/src/Contoso.Shopping.Domain/GlobalUsing.cs	Adjusts global usings for domain project.
samples/src/Contoso.Shopping.Domain/Basket.cs	Adds ShippingAddress + checkout guard + mutation method.
samples/src/Contoso.Shopping.Database/Migrations/20260622-194111-alter-shopping-basket-table.sql	Adds ShippingAddressJson column (SQL Server).
samples/src/Contoso.Shopping.Database/dbex.yaml	Configures typed JSON column mapping for basket.
samples/src/Contoso.Shopping.Contracts/Basket.cs	Adds ShippingAddress to contract.
samples/src/Contoso.Shopping.Contracts/Address.cs	Adds Address contract DTO.
samples/src/Contoso.Shopping.Application/Validators/AddressValidator.cs	Adds Address validator.
samples/src/Contoso.Shopping.Application/Mapping/BasketMapper.cs	Maps ShippingAddress Domain→Contract.
samples/src/Contoso.Shopping.Application/Mapping/AddressMapper.cs	Adds domain↔contract mapper for Address.
samples/src/Contoso.Shopping.Application/Interfaces/IBasketService.cs	Adds UpdateShippingAddressAsync contract.
samples/src/Contoso.Shopping.Application/BasketService.cs	Implements UpdateShippingAddressAsync with validation + mapping.
samples/src/Contoso.Shopping.Api/Controllers/BasketController.cs	Adds PUT /shipping-address endpoint.
samples/src/Contoso.Products.Infrastructure/Repositories/ProductsDbContext.g.cs	Generated EF mapping for tags_json column.
samples/src/Contoso.Products.Infrastructure/Persistence/Product.g.cs	Generated persistence property for Tags.
samples/src/Contoso.Products.Infrastructure/Mapping/ProductMapper.cs	Maps Tags contract↔persistence.
samples/src/Contoso.Products.Database/Migrations/20260731-120000-alter-products-product-add-tags-json.pgsql	Adds tags_json column (Postgres JSONB).
samples/src/Contoso.Products.Database/dbex.yaml	Configures typed JSON column mapping for product.
samples/src/Contoso.Products.Contracts/ProductBase.cs	Adds Tags to product contract base.
Directory.Packages.props	Updates DbEx package versions.
.github/skills/coreex-repository/references/workflow.md	Documents repository mapping rules for JSON columns.
.github/skills/coreex-db-migration/SKILL.md	Adds JSON columns guidance.
.github/skills/coreex-db-migration/references/workflow.md	Adds detailed JSON column workflow and examples.
.github/skills/coreex-app-service/SKILL.md	Documents JSON-backed value object mapping guidance.
.github/skills/coreex-app-service/references/workflow.md	Documents JSON-backed value object mapping workflow.
.github/skills/coreex-aggregate/SKILL.md	Documents JSON-backed value object conventions.
.github/skills/coreex-aggregate/references/workflow.md	Adds JSON-backed value object persistence guidance.
.github/instructions/coreex-tooling.instructions.md	Adds dbex.yaml JSON column conventions + Inspect flag note.
.github/instructions/coreex-repositories.instructions.md	Documents JSON-typed property mapping conventions.
.github/instructions/coreex-domain.instructions.md	Documents JSON-backed value object conventions.
.github/instructions/coreex-application-services.instructions.md	Documents application-layer mapping for JSON-backed value objects.

[Copilot]

Pull request overview

Copilot reviewed 99 out of 102 changed files in this pull request and generated no new comments.

Suppressed comments (6)

src/CoreEx/Metadata/RuntimeMetadata.GetHashCode.cs:45

• If a property getter throws while computing a hash, the current implementation can leave the thread-static visited set polluted (because set.Remove(id) is not in a finally). That can cause subsequent GetHashCode calls on the same thread to incorrectly treat unrelated objects as part of a cycle. Also, using the object reference rather than an identity hash avoids false positives due to hash collisions.
  src/CoreEx/Metadata/RuntimeMetadata.GetHashCode.cs:96
• Same visited-set leak/collision concerns as the IRuntimeMetadataCore branch above: the thread-static visited set should be cleaned up in a finally, and cycle detection should use object references (reference equality) rather than identity hash-code integers to avoid collisions.
  src/CoreEx/Metadata/RuntimeMetadata.IsDefault.cs:76
• Same visited-set cleanup issue as the IRuntimeMetadataCore path above: if reflection-based property access throws, set.Remove(value) might not run, leaving the thread-static set in a bad state. Ensure removal happens in a finally when this frame actually added the object.
  src/CoreEx/Metadata/RuntimeMetadata.GetHashCode.cs:7
• GetHashCode cycle detection tracks RuntimeHelpers.GetHashCode integers. These identity hash codes are not guaranteed unique, so collisions can produce false cycle detection and incorrect hash codes. Tracking object references with ReferenceEqualityComparer avoids collisions and matches the approach used elsewhere (e.g. IsDefault).

This issue also appears in the following locations of the same file:

• line 27
• line 77
  src/CoreEx/Metadata/RuntimeMetadata.IsDefault.cs:50

• If a property getter throws during IsDefault, the current implementation can leave the thread-static visited set polluted because set.Remove(value) is not guaranteed to run. That can make subsequent IsDefault calls on the same thread incorrectly report cycles. Moving the removal into the finally (when this frame actually added the value) makes the guard robust.

This issue also appears on line 61 of the same file.
src/CoreEx.AspNetCore/Abstractions/WebApi.cs:158

• hasBody detection relies on Transfer-Encoding, which is not used by HTTP/2+ and can result in a request with a body but no Content-Length being treated as body-less (silently skipping deserialization). Using ContentLength == 0 as the only fast-path avoids false negatives for unknown-length bodies while preserving the intended “no body” behavior for the common case.

[Copilot]

Pull request overview

Copilot reviewed 100 out of 103 changed files in this pull request and generated no new comments.

Suppressed comments (7)

src/CoreEx/Metadata/RuntimeMetadata.IsDefault.cs:65

• Same issue as the IRuntimeMetadataCore path: the reflection-based ThreadStatic visited set can leak entries when an exception occurs before set.Remove(value), causing false cycle detections later on the same thread.
  src/CoreEx/Metadata/RuntimeMetadata.GetHashCode.cs:83
• Same issue as the IRuntimeMetadataCore branch: the reflection-based ThreadStatic set can leak id entries on exceptions before set.Remove(id), leading to false cycle detection on subsequent calls on the same thread.
  src/CoreEx/Metadata/RuntimeMetadata.IsDefault.cs:50
• The cycle-detection HashSet is ThreadStatic, but set.Remove(value) is not protected by a finally. If any property getter/metadata evaluation throws inside the try, and this call is not the root invocation, the set can retain stale entries and break subsequent IsDefault calls on the same thread.

This issue also appears on line 61 of the same file.
samples/tests/Contoso.Products.Test.Api/ProductMutateTests.Create.cs:107

• ExpectIdentifier(), ExpectETag(), and ExpectChangeLogCreated() already assert presence and auto-exclude those fields from the JSON resource compare. Passing "id", "etag", and "changeLog" exclusions to AssertJsonFromResource is redundant and conflicts with the established test pattern for volatile fields.
  src/CoreEx.UnitTesting/UnitTestExExtensions.Assert.cs:15
• Renaming AssertProblemDetailsTitle to AssertProblemDetails is a breaking change for any downstream test suites using CoreEx.UnitTesting. Consider keeping AssertProblemDetailsTitle as a backward-compatible shim (optionally [Obsolete]) that forwards to the new API.
  src/CoreEx/Metadata/RuntimeMetadata.GetHashCode.cs:33
• The ThreadStatic cycle-detection set removes id only on the success path. If a property getter/metadata evaluation throws after set.Add(id), and this call is not the root invocation, the set can retain stale ids and cause incorrect hash computation in later calls on the same thread.

This issue also appears on line 77 of the same file.
samples/src/Contoso.Shopping.Infrastructure/Repositories/ShoppingDbContext.g.cs:88

• The EF model config declares ShippingAddressJson as NVARCHAR(MAX), but the migration adds it as NVARCHAR(2000). This mismatch can lead to confusing drift (and potentially runtime failures if larger JSON is written via EF but the DB column is bounded). The generated model/migration/dbex config should agree on the intended size.

[Copilot]

Pull request overview

Copilot reviewed 100 out of 103 changed files in this pull request and generated 1 comment.

Suppressed comments (4)

src/CoreEx/Metadata/RuntimeMetadata.GetHashCode.cs:40

• Within the IRuntimeMetadataCore hashing path, prefer tracking visited object references (with ReferenceEqualityComparer) instead of identity hash codes to avoid collisions causing incorrect early termination.
  src/CoreEx/Metadata/RuntimeMetadata.GetHashCode.cs:90
• Same issue as the IRuntimeMetadataCore branch: the reflection-based hashing path should use a visited set of object references rather than RuntimeHelpers.GetHashCode integers to avoid collisions causing incorrect cycle detection.
  src/CoreEx.UnitTesting/UnitTestExExtensions.Assert.cs:15
• This change removes/renames AssertProblemDetailsTitle(...), which is a breaking public API change for CoreEx.UnitTesting consumers. Since existing call sites likely exist outside this repo, consider keeping AssertProblemDetailsTitle as a wrapper (optionally obsolete) and adding AssertProblemDetails(Action<ProblemDetails>?) as the new flexible API.
  src/CoreEx/Metadata/RuntimeMetadata.GetHashCode.cs:7
• Cycle detection uses HashSet<int> of RuntimeHelpers.GetHashCode(object). RuntimeHelpers.GetHashCode is not guaranteed unique, so collisions can cause false cycle detection and return the sentinel hash (0) for non-cyclic graphs, producing unstable/incorrect hash codes.

This issue also appears in the following locations of the same file:

• line 27
• line 77