measure(profiler): decode/removal/parse split for DELETE - removal 59%, parse 19%, decode 11%

Items 1 and 2 of the agreed sequence, and item 1's diagnosis inverted by the measurement it required.

Item 2 - two new stages (Parse, IndexDecode) plus the remaining stamps:
  - Parse around the batch dispatcher's statement classification: statement parsing is ~1.4us per statement and
    was invisible, which is why a stage report could look complete while a fifth to a half of the wall time
    sat in the dispatcher. Also stamped the batch commit (Commit).
  - IndexDecode, split out of IndexMaintenance, because "is it the decode or the removal?" decides the fix.
  - IndexMaintenance around the bulk-update path's per-row hash-index Remove/Add.
  - still uncovered and recorded in the plan: the second batch dispatcher, parser internals, and
    WalAppend/WalFlush (stages that exist but nothing writes).

Item 1 - DELETE attributed with the new split (10K by PK, 50K rows, one batch transaction):
  TEXT-indexed, plaintext   97,528 ops/s  10.25 us/row
  TEXT-indexed, at-rest     91,917 ops/s  10.88 us/row
  INT-indexed,  plaintext  193,001 ops/s   5.18 us/row
  INT-indexed,  at-rest    154,004 ops/s   6.49 us/row
  stages (at-rest): index-maint 58.6% (42.2 ms in SEVEN calls: PK DeleteBulk + one RemoveBatchKeys per loaded
  index) | parse 19.3% (13.9 ms in 10,000 calls) | index-decode 11.3% | row-locate 7.1% | commit 2.6% |
  engine-write 1.2%.
The decode hypothesis (arena reads to build hash keys) is WRONG: decode is 11%. The cost is the REMOVAL, and a
TEXT key costs ~3x an INTEGER key there (42.2 vs 14.4 ms) - string hashing/equality on the key. The tombstone
is noise for the third time. And SQL parsing turned out to be 19-45% of a DELETE batch: the same per-statement
cost the harness pays for every row of its UPDATE/DELETE phases, which makes a prepared/canonical DELETE path
the more promising of the two remaining levers.

Gate: SharpCoreDB.slnx 0 errors; SharpCoreDB.Tests 1857 / 0 failed / 16 skipped. One full-suite run showed a
single failure in HashIndexPerformanceTests (a timing-ratio test) which passes 2/2 in isolation - recorded as a
flake, not a regression.
Docs: plan section 7 (the split, the findings, and the honest instrumentation-coverage list).

Author: MPCoreDeveloper

docs/performance/INSERT_UPDATE_PERFORMANCE_PLAN.md

@@ -1095,11 +1095,35 @@ Stage shares at-rest: **`index-maintenance` 81.9%** (42.2 ms of 10K deletes), `r
   key. So the lever is **not** "make the delete cheaper" but "do the index removal in bulk / deferred", which
   is what `DeferredIndexUpdater` exists for. The tombstone (the part the plan expected to matter) is noise.
 
-**Instrumentation coverage (2026-09-14, §2).** Added: `Table.InsertBatch` (Validate — covering validation and
-serialization — plus RowLocate around the batch PK probes; the path had none) and the fixed-width bulk-delete
-fast path (RowLocate / IndexMaintenance / EngineWrite). Still uncovered, recorded honestly: the bulk **UPDATE**
-path's index-maintenance block and the SQL/engine overhead outside the table (parse, plan cache, commit) —
-the last column above is exactly that share, and it is why the totals in a stage report are not wall time.
+**Follow-up (2026-09-14, decode vs removal vs parse).** With `IndexDecode` and `Parse` now separate stages,
+the same workload (10K deletes by PK, 50K rows, one batch transaction) splits as follows — TEXT column
+indexed vs INTEGER column indexed, which isolates the cost of a *variable-length* index key:
+
+| arm | ops/s | wall µs/row |
+|---|---:|---:|
+| TEXT indexed, plaintext | 97,528 | 10.25 |
+| TEXT indexed, at-rest | 91,917 | 10.88 |
+| INTEGER indexed, plaintext | 193,001 | 5.18 |
+| INTEGER indexed, at-rest | 154,004 | 6.49 |
+
+Stage shares (at-rest): `index-maintenance` **58.6%** (42.2 ms, only **7 calls** — the PK `DeleteBulk` plus one
+`RemoveBatchKeys` per loaded index), `parse` **19.3%** (13.9 ms, **10,000 calls** — 1.4 µs per statement),
+`index-decode` **11.3%**, `row-locate` 7.1%, `commit` 2.6%, `engine-write` **1.2%**. So:
+- the public decode hypothesis was **wrong**: decoding the indexed columns (arena reads, decrypts at rest) is
+  11%, not the bulk — the removal itself is, and a TEXT key costs roughly **3× an INTEGER key** there
+  (42.2 ms vs 14.4 ms), i.e. string hashing/equality on the key is the expensive part;
+- **SQL statement parsing is 1.4 µs per statement and 19–45% of a DELETE batch** — a share that was completely
+  invisible before `Parse` existed, and it is the same cost the `--pk` harness pays for every row of its
+  UPDATE/DELETE phases;
+- the tombstone (1.2%) is confirmed as noise for the third time.
+
+**Instrumentation coverage (2026-09-14, §2).** Covered now: `Table.InsertBatch` (Validate — validation *and*
+serialization — plus RowLocate around the batch PK probes; the path had none), the fixed-width bulk-delete fast
+path (RowLocate / IndexMaintenance / IndexDecode / EngineWrite), bulk-update per-row hash-index maintenance
+(IndexMaintenance), the batch dispatcher's statement classification (Parse) and the batch commit (Commit).
+Still uncovered, recorded honestly: the *second* batch dispatcher path, parser internals below the dispatcher,
+and `WalAppend`/`WalFlush` — those stages exist in the enum but nothing writes them yet, so a stage report
+still cannot be read as wall time.
 
 ---
 

src/SharpCoreDB/DataStructures/Table.CRUD.cs

@@ -2209,10 +2209,18 @@ internal void UpdateMultiple(List<(string where, Dictionary<string, object> upda
 
                                         if (oldRow.TryGetValue(colName, out var oldVal) && oldVal is not null)
                                         {
+                                            long hashRemoveStart = Diagnostics.WritePathProfiler.Stamp();
                                             hashIdx.Remove(oldVal, rowPosition);
+                                            Diagnostics.WritePathProfiler.Add(
+                                                Diagnostics.WritePathProfiler.Stage.IndexMaintenance,
+                                                hashRemoveStart);
                                         }
 
+                                        long hashAddStart = Diagnostics.WritePathProfiler.Stamp();
                                         hashIdx.Add(newVal, rowPosition);
+                                        Diagnostics.WritePathProfiler.Add(
+                                            Diagnostics.WritePathProfiler.Stage.IndexMaintenance,
+                                            hashAddStart);
                                     }
                                 }
                             }
@@ -4074,8 +4082,9 @@ this.storage is null ||
         // loaded hash-index entry, decoding only the indexed columns from the raw fixed-width records
         // (no full-row deserialization). Variable values resolve through the overflow arena, mirroring
         // the fixed-width codec used by the generic path.
-        long deleteIndexStart = WritePathProfiler.Stamp();
+        long deletePkIndexStart = WritePathProfiler.Stamp();
         this.Index.DeleteBulk(keys);
+        WritePathProfiler.Add(WritePathProfiler.Stage.IndexMaintenance, deletePkIndexStart);
 
         var arena = GetOverflowArena();
         foreach (var (colName, hashIdx) in this.hashIndexes)
@@ -4097,6 +4106,10 @@ this.storage is null ||
 
             var type = this.ColumnTypes[colIdx];
             var decoded = new object?[count];
+
+            // §2 split: the DECODE (per row, per index — an arena read per variable value) is measured
+            // separately from the REMOVAL, because they have completely different fixes.
+            long decodeStart = WritePathProfiler.Stamp();
             for (int i = 0; i < count; i++)
             {
                 var payload = raw.AsSpan((int)(i * stride) + 4, layout.FixedSize);
@@ -4119,11 +4132,13 @@ this.storage is null ||
                 }
             }
 
+            WritePathProfiler.Add(WritePathProfiler.Stage.IndexDecode, decodeStart);
+
+            long removeStart = WritePathProfiler.Stamp();
             hashIdx.RemoveBatchKeys(decoded, positions);
+            WritePathProfiler.Add(WritePathProfiler.Stage.IndexMaintenance, removeStart);
         }
 
-        WritePathProfiler.Add(WritePathProfiler.Stage.IndexMaintenance, deleteIndexStart);
-
         if (this.storage is { IsInTransaction: true })
         {
             // Transactional delete: buffer the physical offsets so the in-place marker is applied

src/SharpCoreDB/Database/Execution/Database.Batch.cs

@@ -1021,6 +1021,11 @@ private bool TryParseDeleteForBatch(string sql, out string tableName, out string
 
         foreach (var sql in statements)
         {
+            // §2 instrumentation (2026-09-14): the batch dispatcher's own cost — statement classification and
+            // the per-table parse passes below — was invisible, so a stage report looked complete while a
+            // large share of the wall time was spent here. Only the classification, not the DML (which the
+            // table layer stamps itself), so nothing is double counted.
+            long batchParseStart = Diagnostics.WritePathProfiler.Stamp();
             if (IsInsertStatement(sql))
             {
                 // ✅ PHASE 3: Fast path — parse directly into column-ordered object[] rows
@@ -1064,6 +1069,8 @@ private bool TryParseDeleteForBatch(string sql, out string tableName, out string
             {
                 nonInserts.Add(sql);
             }
+
+            Diagnostics.WritePathProfiler.Add(Diagnostics.WritePathProfiler.Stage.Parse, batchParseStart);
         }
 
         lock (_walLock)
@@ -1175,8 +1182,10 @@ private bool TryParseDeleteForBatch(string sql, out string tableName, out string
                 // Only commit if we started the transaction
                 if (!isInTransactionBefore)
                 {
+                    long commitStart = Diagnostics.WritePathProfiler.Stamp();
                     storage.CommitSync();
                     storage.FlushTransactionBuffer();
+                    Diagnostics.WritePathProfiler.Add(Diagnostics.WritePathProfiler.Stage.Commit, commitStart);
                 }
 
                 // ✅ FIX: Force tables to refresh row count from disk to ensure visibility

src/SharpCoreDB/Diagnostics/WritePathProfiler.cs

@@ -57,16 +57,31 @@ public enum Stage
 
         /// <summary>Committing a transaction.</summary>
         Commit = 8,
+
+        /// <summary>
+        /// Parsing a SQL statement and resolving its execution plan. Instrumented on the batch dispatcher
+        /// (2026-09-14) because a stage report previously showed only the work *inside* the table, so its
+        /// total was never wall time — the missing share was parse/plan/dispatch.
+        /// </summary>
+        Parse = 9,
+
+        /// <summary>
+        /// Decoding the indexed columns of a row to compute the keys for index maintenance. Separated from
+        /// <see cref="IndexMaintenance"/> (2026-09-14) because a DELETE measured 82% in that bucket, and the
+        /// question "is it the decode or the removal?" decides the fix: the decode of a TEXT column resolves
+        /// an overflow-arena block (and decrypts it at rest), while the removal is a hashed bucket update.
+        /// </summary>
+        IndexDecode = 10,
     }
 
-    private const int StageCount = 9;
+    private const int StageCount = 11;
 
     private static readonly long[] ElapsedTicks = new long[StageCount];
     private static readonly long[] CallCounts = new long[StageCount];
     private static readonly string[] StageNames =
     [
         "validate", "encode", "index-maint", "row-locate", "in-place-patch",
-        "engine-write", "wal-append", "wal-flush", "commit",
+        "engine-write", "wal-append", "wal-flush", "commit", "parse", "index-decode",
     ];
 
     private static int _enabled;