fix: Fix buckets creation not being read from schema file, update sch…

CHANGELOG.md

@@ -34,6 +34,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **writer/api**: one-call **`WriteReplay`** pipeline (#30) -- new header **`sdk/include/vtx/writer/core/vtx_write_replay.h`**.  `WriteReplay(config, source, format)` runs the whole recording pipeline in one call: creates the writer (schema missing/invalid -> `SchemaInvalid`), initializes the `IFrameDataSource` (failure -> `Internal`), drains every frame through `TryRecordFrame`, finalizes, and returns a **`WriteReplayResult`** (`ok`, `error`, `warnings`, `frames_written`, `frames_dropped`, `total_frames`, `output_path`, `elapsed_seconds`).  Frames rejected by finalization/timer are counted in `frames_dropped` and surfaced one-per-`VtxWarning`; the call still produces a valid replay of the accepted frames.  `SerializationFormat` selects FlatBuffers (default) or Protobuf
 - **writer/api**: automatic output-directory creation + destructor finalize (#29) -- `WriterFacadeConfig::create_output_dirs` (default **on**) creates any missing parent directories of `output_filepath` before the sink opens the file (set `false` to require the directory to pre-exist and keep the previous throw-on-missing behavior).  The writer facade destructor now finalizes the file as a best-effort fallback (`Stop()` wrapped in `try/catch`) when the caller forgot to call `Stop()`, so a dropped writer still yields a readable `.vtx`
 - **writer/api**: in-memory schema sources -- `WriterFacadeConfig` / `NetworkWriterFacadeConfig` (and the internal `ReplayWriter::Config`) gain `schema_json_content` (a raw JSON string) and `schema_registry` (a pre-built `std::shared_ptr<VTX::SchemaRegistry>`, copied in with no re-parse) alongside `schema_json_path`.  Source precedence is **registry > content > path**, applied consistently in the create-time schema probe and in `ReplayWriter`.  The writer no longer requires the schema to live on disk
+- **common/schema**: the schema's top-level **`"buckets"` array is now parsed and is the source of truth for the frame bucket layout** -- `SchemaRegistry::GetBucketNames()` returns the declared names in order (`"buckets"[i]` names Frame bucket index `i`), and `PropertyAddressCache` carries them (`bucket_names`) so the reader can use them too.  A new `Buckets` validation rule rejects a malformed key when present (non-array, non-string entries, empty or duplicate names); schemas without the key stay valid (legacy)
+- **writer/api**: **schema-driven bucket normalization** -- before finalization the writer rearranges every frame's buckets into the schema-declared layout: buckets are reordered to schema order, declared buckets missing from the frame are created empty, and a bucket the schema does not declare rejects the frame with the new **`VtxErrorCode::BucketUnresolved`** (observable via `TryRecordFrame`).  Frames built positionally (no `bucket_map`) adopt the schema layout as long as they do not exceed the declared bucket count.  Schemas without a `"buckets"` array skip the normalization entirely
+- **reader/api**: **bucket names restored on read** -- bucket names never hit the wire (both formats serialize buckets positionally), so deserialized frames used to come back with an empty `bucket_map`.  The reader now stamps `bucket_map` from the embedded schema's `"buckets"` array when chunks are deserialized, so by-name lookups (`Frame::GetBucket(name)` const) and `bucket_map` iteration work on read frames.  Replays written without a `"buckets"` array keep the old positional-only behavior
 
 ### Changed
 
@@ -56,6 +59,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **differ**: map fields were never diffed correctly on either backend.  The FlatBuffers and Protobuf view adapters' `GetMapSize` / `GetMapKey` / `GetMapValueAsStruct` treated the outer `map_properties` vector (one `MapContainer` per map-field slot) as if each element were a single key/value pair -- so a multi-entry map reported size 1, and added / removed keys were never seen.  They now iterate the entries **within** the field's `MapContainer` (slot 0, matching the writer/loader convention), so `DefaultTreeDiff::DiffMapContainers` emits correct add / remove / value-change operations.  The Protobuf adapter's "Case 2" also looked for capitalised `Keys` / `Values` fields on the wrong message; it now descends into `map_properties` -> `MapContainer.keys` / `.values`.  A single map field per struct is assumed (documented in both adapters)
 - **writer/reader (Protobuff -> Protobuf rename)** (#28): the misspelled "Protobuff" was removed from the public API -- **breaking, no aliases kept**.  `SerializationFormat::Protobuffs` -> **`SerializationFormat::Protobuf`**; `CreateProtobuffWriterFacade` -> **`CreateProtobufWriterFacade`**; `CreateProtobuffNetworkWriterFacade` -> **`CreateProtobufNetworkWriterFacade`**; reader `CreateProtobuffFacade` -> **`CreateProtobufFacade`** (internal `ProtobuffFacadeImpl` -> `ProtobufFacadeImpl`).  Migrate call sites with a literal `Protobuff` -> `Protobuf` rename
 - **common/types**: `FlatArray::GetSubArray` & `FlatArray::GetMutableSubArray` did not have a check for StartIndex being greater or equal to EndIndex. In exceptional cases, like an empty array this can cause a crash or undefined behaviour in the best case scenario.
+- **writer/flatbuffers**: `FlatBuffersVtxPolicy::FromNative` hardcoded exactly two bucket slots -- `buckets[0]` was renamed to `"data"`, `buckets[1]` to `"bone_data"`, and **any bucket at index >= 2 was silently dropped** from the file.  It now iterates all buckets generically (same shape as the Protobuf policy, shared `Serialization::SortBucketByTypeId` helper in `bucket_type_sort.h`), preserving every bucket and the frame's `bucket_map`.  Bucket naming is schema-driven (see the `"buckets"` entries above), not serializer-driven
+- **reader/schema**: the reader-side `PropertyAddressCache` built from the embedded schema (`PopulateCacheFromJsonString`) hand-copied the registry cache and **omitted `type_max_indices`**, so `ValidateEntity` on read frames treated every per-type max as 0 (any populated property would flag `FieldIndexOutOfRange` once frame validation ran).  It now reuses `SchemaRegistry::GetPropertyCache()` verbatim.  Also makes `ValidateReplay`'s per-frame pass effective: it iterates `frame.bucket_map`, which was always empty on read frames before bucket-name restoration
+- **samples/benchmarks**: `basic_write.cpp` and `bench_writer.cpp` created a `"Players"` bucket while writing against the arena schema, which declares `"buckets": ["entity"]` -- under schema-driven normalization those frames would now be rejected.  Both use `"entity"`
 
 
 ### Notes

benchmarks/bench_writer.cpp

@@ -62,7 +62,7 @@ static void BM_WriterThroughput(benchmark::State& state) {
 
         for (int i = 0; i < kFramesPerIteration; ++i) {
             VTX::Frame frame;
-            auto& bucket = frame.CreateBucket("Players");
+            auto& bucket = frame.CreateBucket("entity");
 
             VTX::PropertyContainer entity;
             entity.entity_type_id = 0;

docs/FILE_FORMAT.md

@@ -145,6 +145,8 @@ Frame
 
 Bucket names are user-defined and typically represent logical categories in the game world (e.g., `"Players"`, `"Vehicles"`, `"World"`, `"Projectiles"`). A frame can have any number of buckets.
 
+Bucket names never hit the wire: both formats serialize buckets as an ordered, nameless list. The schema's top-level `"buckets"` array is the source of truth for the layout -- `"buckets"[i]` names the bucket stored at index `i`. The writer normalizes every frame to that layout before serialization (rejecting buckets the schema does not declare), and the reader stamps the names back onto `bucket_map` when frames are deserialized. Schemas without a `"buckets"` array skip both steps; their frames are purely positional.
+
 ### Bucket
 
 A `Bucket` groups entities of related types. It stores two parallel arrays:

docs/POST_PROCESSING.md

@@ -300,7 +300,7 @@ Gotchas worth knowing:
 
 - **`RemoveIf` invalidates `type_ranges`**: the bucket's typed-range index is wiped to zero after a bulk filter. The serializer rebuilds them from `entity_type_id`, so on-disk output is correct; in-memory tooling that reads `type_ranges` after the processor runs must recompute them or rely on `entity_type_id` directly.
 
-- **Writer bucket limits**: the FlatBuffers serializer only persists `buckets[0]` (renamed to `"data"` on disk) and `buckets[1]` (renamed to `"bone_data"`). Additional buckets created by a processor via `view.raw()->CreateBucket(...)` are **silently dropped**. The Protobuf serializer preserves all buckets but applies the type-id reordering only to bucket 0.
+- **Buckets come from the schema**: when the schema declares a top-level `"buckets"` array, the writer normalizes every frame to that layout before serialization -- buckets are reordered to schema order, declared buckets missing from the frame are created empty, and a bucket **not** declared in the schema rejects the whole frame with `VtxErrorCode::BucketUnresolved`. A bucket created by a processor via `view.raw()->CreateBucket(...)` must therefore be declared in the schema. Both serializers persist all buckets positionally and apply the type-id reordering only to bucket 0. Schemas without a `"buckets"` array skip the normalization entirely (legacy behavior).
 
 ## Error handling
 

docs/SDK_API.md

@@ -183,11 +183,13 @@ config.schema_registry     = my_shared_registry;   // OR a pre-built std::shared
 
 Missing parent directories of `output_filepath` are created automatically (`config.create_output_dirs`, default `true`; set it `false` to require the directory to pre-exist).
 
+The schema's top-level `"buckets"` array declares the frame bucket layout (`SchemaRegistry::GetBucketNames()`), e.g. `"buckets": ["World"]`. When declared, every recorded frame is normalized to that layout: buckets are reordered to schema order, declared buckets missing from the frame are created empty, and a bucket the schema does not declare rejects the frame (`VtxErrorCode::BucketUnresolved`). The reader uses the same array to restore bucket names on deserialized frames. Schemas without a `"buckets"` array skip the normalization (legacy behavior).
+
 ### Recording Frames
 
 ```cpp
 VTX::Frame frame;
-auto& bucket = frame.CreateBucket("World");
+auto& bucket = frame.CreateBucket("World"); // must be declared in the schema's "buckets" array
 
 // Add entities
 bucket.unique_ids.push_back("player_001");
@@ -219,7 +221,8 @@ if (!r.IsWritten()) {
 }
 ```
 
-Before a frame enters the chunk pipeline the writer **finalizes** it: validates that every
+Before a frame enters the chunk pipeline the writer **finalizes** it: normalizes the bucket
+layout to the schema's `"buckets"` array (rejecting undeclared buckets), validates that every
 entity type resolves to a schema struct, recomputes each entity's `content_hash` after all
 schema fields and post-processor overrides, then **freezes** the frame (any mutation handle a
 post-processor stashed is revoked).

samples/basic_write.cpp

@@ -45,8 +45,9 @@ int main(int argc, char* argv[]) {
     for (int i = 0; i < frame_count; ++i) {
         VTX::Frame frame;
 
-        // Create a bucket and add one entity per frame.
-        VTX::Bucket& bucket = frame.CreateBucket("Players");
+        // Create the bucket declared by the schema's "buckets" array and add
+        // one entity per frame. Buckets not declared there reject the frame.
+        VTX::Bucket& bucket = frame.CreateBucket("entity");
 
         VTX::PropertyContainer entity;
         entity.entity_type_id = 0;

sdk/include/vtx/common/readers/schema_reader/schema_registry.h

@@ -90,6 +90,13 @@ namespace VTX {
 
         const VTX::PropertyAddressCache& GetPropertyCache() const { return property_cache_; }
 
+        /**
+         * @brief Bucket names declared by the schema's top-level "buckets" array.
+         * @details Order matters: name at position i corresponds to Frame bucket index i.
+         * Empty if the schema declares no buckets (legacy schemas).
+         */
+        const std::vector<std::string>& GetBucketNames() const { return bucket_names_; }
+
         /**
          * @brief Validate a schema document without loading it.
          * @details Runs the full SchemaValidator rule set over @p raw_json and
@@ -106,6 +113,7 @@ namespace VTX {
         std::string json_content_;
         std::unordered_map<std::string, SchemaStruct> structs_; ///< Storage map: Struct Name -> Definition.
         std::unordered_map<std::string, int32_t> struct_type_ids_;
+        std::vector<std::string> bucket_names_; ///< Declared bucket layout, in schema order.
         VTX::PropertyAddressCache property_cache_;
         int32_t current_type_id_ = 0;
         bool b_is_valid_;                        ///< Internal validity flag.

sdk/include/vtx/common/readers/schema_reader/schema_validator.h

@@ -51,6 +51,11 @@ namespace VTX {
      */
     struct RawSchema {
         std::vector<RawSchemaStruct> structs;
+
+        bool has_buckets = false;          ///< Document has a top-level "buckets" key.
+        bool buckets_is_array = false;     ///< That key is an array (valid shape).
+        std::vector<std::string> buckets;  ///< String entries of "buckets", in order.
+        int non_string_bucket_entries = 0; ///< Entries of "buckets" that were not strings.
     };
 
     /**

sdk/include/vtx/common/vtx_diagnostics.h

@@ -48,6 +48,7 @@ namespace VTX {
         SchemaMissing,
 
         EntityTypeUnresolved,
+        BucketUnresolved,
         FieldIndexOutOfRange,
         TypeMismatch,
         ContainerMismatch,
@@ -80,6 +81,8 @@ namespace VTX {
             return "SchemaMissing";
         case VtxErrorCode::EntityTypeUnresolved:
             return "EntityTypeUnresolved";
+        case VtxErrorCode::BucketUnresolved:
+            return "BucketUnresolved";
         case VtxErrorCode::FieldIndexOutOfRange:
             return "FieldIndexOutOfRange";
         case VtxErrorCode::TypeMismatch:

sdk/include/vtx/common/vtx_property_cache.h

@@ -122,9 +122,12 @@ namespace VTX {
         // Value: The schema cache containing its specific properties
         std::unordered_map<int32_t, StructSchemaCache> structs;
         std::unordered_map<std::string, int32_t> name_to_id;
+        // Bucket names from the schema's "buckets" array; position i names Frame bucket index i.
+        std::vector<std::string> bucket_names;
         void Clear() {
             structs.clear();
             name_to_id.clear();
+            bucket_names.clear();
         }
     };
 } // namespace VTX

sdk/include/vtx/reader/core/vtx_reader.h

@@ -665,6 +665,7 @@ namespace VTX {
                 cc.index = idx;
                 SerializerPolicy::ProcessChunkData(idx, compressed_blob, stop_token, cc.native_frames,
                                                    cc.decompressed_blob, cc.raw_frames_spans);
+                RestoreBucketNames(cc.native_frames);
                 return cc;
             } catch (const std::exception& e) {
                 VTX_ERROR("[READER] Chunk {} deserialization failed: {}", idx, e.what());
@@ -675,6 +676,25 @@ namespace VTX {
             }
         }
 
+        // Bucket names never hit the wire (buckets serialize positionally), so
+        // freshly deserialized frames carry an empty bucket_map. The schema's
+        // "buckets" array names bucket index i with its i-th entry; stamp that
+        // back so by-name lookups and per-bucket validation work on read frames.
+        // property_address_cache_ is built once in the constructor (ReadHeader)
+        // before any chunk load starts, so reading it here is thread-safe.
+        void RestoreBucketNames(std::vector<VTX::Frame>& frames) const {
+            const auto& bucket_names = property_address_cache_.bucket_names;
+            if (bucket_names.empty())
+                return;
+
+            for (auto& frame : frames) {
+                const size_t count = std::min(bucket_names.size(), frame.GetBuckets().size());
+                for (size_t i = 0; i < count; ++i) {
+                    frame.bucket_map[bucket_names[i]] = i;
+                }
+            }
+        }
+
     private:
         std::string filepath_;
 

sdk/include/vtx/writer/core/writer.h

@@ -109,6 +109,12 @@ namespace VTX {
                 view.Freeze();
             }
 
+            std::string bucket_detail;
+            if (!NormalizeBucketsToSchema(native_frame, bucket_detail)) {
+                timer_.Rollback();
+                return RecordResult::MadeRejected(VtxErrorCode::BucketUnresolved, std::move(bucket_detail));
+            }
+
             std::string validation_detail;
             if (!FinalizeFrame(native_frame, validation_detail)) {
                 timer_.Rollback();
@@ -220,6 +226,60 @@ namespace VTX {
         }
 
     private:
+        // Rearranges the frame's buckets into the schema-declared layout so that
+        // bucket index i always holds the bucket named by the schema's "buckets"[i].
+        // Declared buckets missing from the frame are created empty; a bucket the
+        // schema does not declare rejects the frame. Schemas without a "buckets"
+        // array leave the frame untouched (legacy behavior).
+        bool NormalizeBucketsToSchema(VTX::Frame& frame, std::string& out_detail) {
+            const std::vector<std::string>& schema_buckets = registry_.GetBucketNames();
+            if (schema_buckets.empty()) {
+                return true;
+            }
+
+            auto& buckets = frame.GetMutableBuckets();
+
+            if (frame.bucket_map.empty() && !buckets.empty()) {
+                // Positionally built frame (no names): adopt the schema layout as-is.
+                if (buckets.size() > schema_buckets.size()) {
+                    out_detail = "frame has " + std::to_string(buckets.size()) +
+                                 " unnamed buckets but the schema declares only " +
+                                 std::to_string(schema_buckets.size());
+                    return false;
+                }
+                buckets.resize(schema_buckets.size());
+                for (size_t i = 0; i < schema_buckets.size(); ++i) {
+                    frame.bucket_map[schema_buckets[i]] = i;
+                }
+                return true;
+            }
+
+            if (frame.bucket_map.size() != buckets.size()) {
+                out_detail = "frame contains buckets with no name; cannot map them onto the schema 'buckets' layout";
+                return false;
+            }
+
+            for (const auto& [name, idx] : frame.bucket_map) {
+                if (std::find(schema_buckets.begin(), schema_buckets.end(), name) == schema_buckets.end()) {
+                    out_detail = "bucket '" + name + "' is not declared in the schema 'buckets' array";
+                    return false;
+                }
+            }
+
+            std::vector<VTX::Bucket> ordered(schema_buckets.size());
+            std::map<std::string, size_t> ordered_map;
+            for (size_t i = 0; i < schema_buckets.size(); ++i) {
+                auto it = frame.bucket_map.find(schema_buckets[i]);
+                if (it != frame.bucket_map.end()) {
+                    ordered[i] = std::move(buckets[it->second]);
+                }
+                ordered_map[schema_buckets[i]] = i;
+            }
+            buckets = std::move(ordered);
+            frame.bucket_map = std::move(ordered_map);
+            return true;
+        }
+
         bool FinalizeFrame(VTX::Frame& frame, std::string& out_detail) {
             for (auto& bucket : frame.GetMutableBuckets()) {
                 for (auto& entity : bucket.entities) {