Skip to content
This repository was archived by the owner on Aug 17, 2026. It is now read-only.

Bind a sized void payload as the bytes it is - #81

Merged
borisbat merged 7 commits into
borisbat:masterfrom
olegus8:pr-vk-emitter-net
Aug 14, 2026
Merged

Bind a sized void payload as the bytes it is#81
borisbat merged 7 commits into
borisbat:masterfrom
olegus8:pr-vk-emitter-net

Conversation

@olegus8

@olegus8 olegus8 commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

The generator skips a struct whole if any member is unmappable, and the most
common one left is a const void* payload with a len. Where that len means a
size in bytes
— the registry spells it *Size / size — it binds as
array<uint8> whose scratch pointer reinterprets into the raw void? slot, the
same move a flags array already makes.

That unskips 11 boost structs (57 → 46 in the ratchet report):
VkPipelineCacheCreateInfo, VkSpecializationInfo, VkShaderCreateInfoEXT,
VkPushConstantsInfo, VkValidationCacheCreateInfoEXT, VkCuModuleCreateInfoNVX,
VkWriteDescriptorSetInlineUniformBlock, VkHdrVividDynamicMetadataHUAWEI,
VkHostAddressRangeConstEXT and both ObjectTagInfo forms.

A count-shaped len is not bytes, and is refused by name. VkLayerSettingEXT
carries len="valueCount" — a count of typed values whose stride comes from a
sibling type field, and which array<uint8> cannot represent at all (its STRING
case is a char* per value). It re-skips with a named reason and takes
VkLayerSettingsCreateInfoEXT by cascade. On a single const void*, valueCount
is the registry's only count-shaped len; VkCuLaunchInfoNVX / VkCudaLaunchInfoNV
have four more, but each is a double pointer this arm never reaches.

Non-const is refused too: the call writes it, and a view's scratch never
reaches the caller back.

The command side. A two-call enumerate whose out array is void returns
array<uint8> against its byte count — get_pipeline_cache_data,
get_validation_cache_data_e_x_t, get_shader_info_a_m_d,
get_shader_binary_data_e_x_t. Those counts are size_t, which is what makes the
64-bit spelling load-bearing rather than cosmetic: das promotes nothing, so a
vkcount > 0u compare against a 0ul count does not compile, and
resize(int(vkcount)) truncates above 2^31. Both are now spelled from
plan.count_is_64. A command with an out VALUE beside an enumerated array is
refused with a named reason instead of emitting with the value dropped.

A borrowed opt-struct-ptr no longer costs its holder's copyability. Making
VkSpecializationInfo mappable gave PipelineShaderStageCreateInfo a
pSpecializationInfo field, and the old scratch embedded the boost pointee by
value
— dragging its arrays in and flipping the holder from copyable to
non-copyable. das copies a copyable element out of an array literal and MOVES a
non-copyable one, so gp.pStages <- [st_vert, st_frag] silently zeroed the
caller's locals and the second create-info got blank stages. The holder now keeps
only @do_not_delete pX : T?, the POD Vk* view and a _present flag; the view
marshals through the caller's pointee in place, and @do_not_delete is what keeps
the finalize off the caller's local — the job the clone was really doing. Consume
semantics are unchanged, as test_opt_ptr_view_semantics requires.

generator/copyability_report.txt is written beside the skip report on full
runs: every boost struct that is not copyable, with the array field responsible
(170 today). A flip like the one above now lands in a review diff rather than in a
driver crash.

A jagged argument's views build and release through one generic pair
(vk_views2_create / vk_views2_destroy in vulkan_structs.das) instead of a
copy inside each wrapper — 34 fewer generated lines, one definition of the
void?-interior-pointer rule, and it clears the STYLE040 the generated file
carried.

daslib/, generator/skip_report.txt and the new report are regenerated in the
same commit, and utils/vulkan2rst.das groups the creators the unskips add so the
doc job stays quiet.

Verified against a real driver (our Windows box, whole integration suite):
upstream master is 40/42 there — only test_gltf, which needs assets — and this
branch is 40/42 with the same single failure. Whole-tree lint: 150 files, 0
issues, 0 errors. Regeneration is deterministic (a second run produces no diff).

@olegus8

olegus8 commented Aug 13, 2026

Copy link
Copy Markdown
Contributor Author

The integration lane's lint step is red on the tree, not on this branch: the 7
findings are 4 × STYLE040 and 3 × PERF031, in daslib/vulkan_boost.das,
generator/vk_model.das and generator/vk_naming.das — two of those three files
this branch never touches, and the third's finding is at a shape master already
emits.

Linting the untouched base with the same tool reports the same 7, and master's
last green Integration tests run (2026-08-12, on b2d4269, this branch's base)
predates the daslang side that added those rules. So the step fails for anything
opened against master today, and this branch adds nothing to it — a first pass of
mine did add four LINT018 and three comment-cap hits, and those are fixed in the
emitter rather than in the generated output.

@olegus8

olegus8 commented Aug 13, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up to the lint note above: the one finding that WAS in a file this branch
rewrites is now gone, so the branch adds nothing to that lane and removes one
from it.

daslib/vulkan_cmds.das carried a STYLE040 — the jagged view build and release
emitted verbatim into both acceleration-structure build wrappers. Since the file
is generated, the fix is in the emitter: vk_views2_create / vk_views2_destroy
are now emitted once into vulkan_structs.das as a generic pair, and each
wrapper calls them. That is 34 fewer generated lines and one definition of the
void?-interior-pointer rule instead of two copies of it.

Measured with the branch's generated files in place so cross-module requires
resolve to them: 150 files, 6 issues, 0 errorsvulkan_boost.das (3),
vk_model.das (2), vk_naming.das (1), none of which this branch touches, and
all of which predate it (master's last green Integration tests run is from
before the daslang side added STYLE040 / PERF031). vulkan_cmds.das is
clean.

@borisbat

Copy link
Copy Markdown
Owner

Verified the mechanism locally (merge with master, correct in-tree rig): regen is deterministic — the skip report and all four generated daslib files match a fresh generator run byte-for-byte — the new named skip refuses vkGetPipelineBinaryDataKHR exactly as described, and the 64-bit spelling is right in the output (uint64(long_length(...)) for both size_t struct counts, all four byte-enumerate commands carry genuine byte counts). Your lint accounting also checks out: 150 files, 0 errors, and none of the findings are in files this branch touches. (My rig sees 7, not 6 — vulkan_imgui_app.das:256 carries a STYLE040 that CI's daslang can't compile-reach without dasImgui. All pre-existing; a PR fixing all of them on our side is coming, so nothing for this branch.)

One finding, and it gates merge: VkLayerSettingEXT breaks the premise the void arm rests on. "vk.xml always gives such a member's len in BYTES" is true for 12 of the 13 unskips — every one names its len dataSize / initialDataSize / codeSize / tagSize / dynamicMetadataSize / size — but pValues is:

<member noautovalidity="true" len="valueCount">const <type>void</type>* <name>pValues</name>
    <comment>Values to pass for a setting</comment></member>

with valueCount commented "Number of values of the setting" — a typed-value count whose stride comes from the sibling type : VkLayerSettingTypeEXT (4 for BOOL32/INT32/UINT32/FLOAT32, 8 for the 64-bit forms, and STRING is a char* per value, not flat bytes at all). The generated view writes the byte length into it:

vk.valueCount = uint(length(b.pValues))

so one FLOAT32 setting packed as 4 bytes advertises valueCount = 4, and the consuming layer reads one real value and three garbage ones off the end. A byte array can't even represent the STRING case.

The fix is the same one-gate class as the arm itself: bind void-as-bytes only when the len member actually denotes bytes (in the registry that's exactly the *Size / size spellings — valueCount is the registry's only count-shaped len on a const void*, I swept), and let VkLayerSettingEXT re-skip with a named reason ("len counts typed values, stride from a sibling field"), taking VkLayerSettingsCreateInfoEXT with it by cascade. Ratchet goes 44 → 46, which is the honest number; the comment above the arm and the PR body drop "always". Everything else here is merge-ready as far as I can verify without lavapipe.

@olegus8

olegus8 commented Aug 14, 2026

Copy link
Copy Markdown
Contributor Author

Caught, and the premise was wrong exactly as you say. 6d4cf80.

The arm now gates on the len MEANING bytes — len_is_bytes() accepts the
registry's *Size / size spellings and nothing else — so VkLayerSettingEXT
re-skips with a named reason and takes VkLayerSettingsCreateInfoEXT by cascade:

VkLayerSettingEXT: member pValues len valueCount counts typed values, stride from a sibling field
VkLayerSettingsCreateInfoEXT: member pSettings (VkLayerSettingEXT x1) not mappable

Ratchet is 44 → 46, and the comment above the arm no longer says "always" — it
says bytes only where the len means bytes. The other twelve are untouched
(PipelineCacheCreateInfo, SpecializationInfo, ShaderCreateInfoEXT,
PushConstantsInfo, both ObjectTagInfo, ValidationCacheCreateInfoEXT,
CuModuleCreateInfoNVX and the rest all name a *Size), and the four byte
enumerates are unaffected since they carry genuine byte counts.

I swept the registry independently before touching it, and it agrees with yours
for this arm's scope: on a single const void*, valueCount is the only
count-shaped len. There are four more count-shaped ones —
VkCuLaunchInfoNVX / VkCudaLaunchInfoNV, pParams/pExtras against
paramCount/extraCount — but each is const void* const*, a double pointer
the arm never reaches, so they were never in scope and stay skipped as before.
Worth knowing they exist if the double-pointer rule ever grows a void case.

Lint after the change: daslib/vulkan_boost.das (3), vk_model.das (2),
vk_naming.das (1) — the pre-existing set, none in files this branch touches.
My first cut of the gate carried a 5-line comment inside a def private and
tripped STYLE015; trimmed, so vk_emit_boost.das is clean again. Thanks for
the vulkan_imgui_app.das note — that one my rig can't reach either.

olegus8 pushed a commit to olegus8/dasVulkan-new that referenced this pull request Aug 14, 2026
The lane's lint step reports 7 findings on master, none newer than the
rules that added them (STYLE040 and PERF031 postdate every file's last
touch), so every PR opened today inherits a red step it didn't cause.

- vulkan_boost: set_extent_2d absorbs the four 2D-extent runs, and
  push_memory_management_extensions the extension trio both device
  creators carry.
- vulkan_imgui_app: the no-scene harness_end_frame delegates to the
  block overload with an empty scene - same frame, 11 fewer lines.
- vk_model / vk_naming: the dim-scan and vendor-tag loops walk a
  peek_data view, so nothing re-strlens the source per iteration.
  Regenerated output is byte-identical, so the rewrites are proven
  behavior-preserving.

The one finding left is vulkan_cmds' jagged-view STYLE040, which borisbat#81
removes in the emitter - the honest fix for a generated file.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@olegus8

olegus8 commented Aug 14, 2026

Copy link
Copy Markdown
Contributor Author

Merged master in (ff422fd) — with #82 landed, the lint lane is green: 150
files, 0 issues, 0 errors. Thanks for clearing that; this branch now adds nothing
to it and removes the one finding it used to carry.

The integration lane fails, and I think it is ours, but I cannot reproduce it
here.
tests/integration/test_pipeline_boost_structs.das dies at
daslib/vulkan_runtime.das:23: Vulkan call failed: ERROR_UNKNOWN; the other 40
tests pass, so the device is fine.

What I can establish from here:

  • The generated files are not stale. I merged master and regenerated: no
    delta at all, so your vk_model / vk_naming changes since our base do not
    move this output.
  • The change does alter EXISTING structs, not just add new ones. Making
    VkSpecializationInfo mappable gives PipelineShaderStageCreateInfo a new
    pSpecializationInfo : SpecializationInfo? field and flips its view onto the
    borrow-and-null path (the view now takes var b). A second struct gains
    pData : HostAddressRangeConstEXT? the same way. That is the blast radius I
    under-weighed when I called the arm a one-gate change.
  • The obvious suspect is not it. The test builds its stages as
    PipelineShaderStageCreateInfo(module_ = …, pName = "main") with no
    specialization info, and the generated view guards on
    if (b.pSpecializationInfo != null), so that path leaves the raw pointer null
    exactly as before.

Beyond that I am blind: this box has no Vulkan device (the lint and generator
work all runs headless), so I cannot bisect a runtime ERROR_UNKNOWN.

Two ways forward, and I do not mind either: you run it on your rig, which has
found the last three things faster than I would have — or I stand up lavapipe and
the native module on our sim box and bisect it there, which is a few hours of
setup before it tells us anything. Say which and I will do it. If it turns out the
answer is "unskipping a struct must not reshape an existing one", the arm can be
narrowed to structs nothing else embeds, at the cost of SpecializationInfo.

@olegus8

olegus8 commented Aug 14, 2026

Copy link
Copy Markdown
Contributor Author

Correcting myself: I said I was blind here because this box has no Vulkan device.
That was wrong — we have a GPU box, it runs this binding through a renderer every
day, and I should have gone there before writing that. I did now.

It does not settle it, for a reason worth stating: on that box
test_pipeline_boost_structs crashes with EXCEPTION_ACCESS_VIOLATION both
with this branch's boost and with the unmodified one
— same crash, same place —
so its Vulkan setup discriminates nothing. Its device is a real driver rather
than lavapipe, and the test dies before it can tell us anything about this
change.

What that trip did turn up is a gap in the baseline: master has not run the
integration lane since b2d4269 (2026-08-12)
— this branch's base. #82's run
stopped at the lint step, so nothing has executed those tests with #82 in the
tree. This branch's run is the first. #82's only change to hand-written boost is
the push_memory_management_extensions extraction, which reads inert to me, so I
am not claiming it is yours — I am saying "master is green on this test" is
currently an assumption on both our parts.

Two ways to cut it, and I will do whichever you prefer:

  1. Re-run master's integration lane once. If it fails the same way, the cause is
    in master and this branch is a bystander; if it passes, the cause is here and
    I will find it.
  2. I push a commit that narrows the arm so it never reshapes an EXISTING struct —
    VkSpecializationInfo stays skipped, so PipelineShaderStageCreateInfo keeps
    its current shape — and we read CI. That is a bisect through the only rig that
    reproduces, at the cost of that one struct.

The lint lane is green on this branch, and master is merged in as of ff422fd.

@borisbat

Copy link
Copy Markdown
Owner

Reproduced the integration-lane failure locally (WSL, lavapipe, daslang at master 2779541) and pinned the root cause — posting to save you the hunt.

Where it breaks: every creator up the test succeeds; the failing call is vkCreateGraphicsPipelines itself. All 5 non-merge commits fail, starting with b14a3c6 — the jagged-view generic pair and the byte-len gate revision are innocent.

Root cause: PipelineShaderStageCreateInfo now embeds the opt-ptr clone scratch by value:

_vk_view_pSpecializationInfo_value : SpecializationInfo

SpecializationInfo carries array<SpecializationMapEntry> + array<uint8>, so the embed flips PipelineShaderStageCreateInfo from copyable to non-copyable. A das array literal copies copyable elements but silently moves non-copyable ones. The test reuses its stage locals across the two batch entries:

gp_const.pStages <- [st_vert, st_frag]   // now MOVES st_vert/st_frag — zeroes them
gp_plain.pStages <- [st_vert, st_frag]   // builds from zeroed locals

Probe at the PR head: after the first literal, st_vert.pName goes 'main''' and stage(0); gp_plain gets two stages with null module + empty pName, and the batched create returns ERROR_UNKNOWN. Same probe at base 540b561: st_vert survives both literals and the create returns SUCCESS. Note the test never sets pSpecializationInfo — the null arm is inert at runtime; the breakage is purely the copyability flip.

Why it matters beyond this test: reusing a stage struct by value across create-infos is idiomatic Vulkan; any downstream caller doing it silently loses data. So the fix belongs in the emitter, not the test.

Fix shape: don't embed the non-copyable clone scratch by value. Either marshal through the user's pointee directly (its _vk_view_* scratch already lives inside SpecializationInfo; the stage struct then only needs the POD VkSpecializationInfo embed, which stays copyable), or put the scratch behind a heap pointer freed in vk_view_destroy. General invariant worth keeping: unskipping a struct must never flip an existing boost struct's copyability — a copyability report beside skip_report.txt would make such flips visible in review.

@olegus8

olegus8 commented Aug 14, 2026

Copy link
Copy Markdown
Contributor Author

Fixed at the emitter, and your agents' diagnosis was exactly right — the
copyability flip, not the null arm. a539bf0.

The fix. The scratch no longer embeds the boost pointee by value, so nothing
drags its arrays into the holder:

@do_not_delete pSpecializationInfo : SpecializationInfo?
_vk_view_pSpecializationInfo_vk : VkSpecializationInfo
_vk_view_pSpecializationInfo_present : bool

The view marshals through the caller's pointee in place
(vk_view_create_unsafe(*b.pX)), so the pointee's own scratch stays in the
pointee, and the holder embeds only the POD Vk* view. @do_not_delete on the
borrowed field is what now stops the finalize from freeing the caller's local —
that is what the clone was really working around. PipelineShaderStageCreateInfo
is a pointer + POD + bool again, so it is copyable and an array literal copies it.

The consume semantics are unchanged: the field is still nulled after viewing, and
vk_view_destroy still clears _present. My first cut dropped the nulling and
test_opt_ptr_view_semantics failed on "a re-view sees the consumed field as
absent" — that test earned its keep.

Verified on our Windows box against a real driver, since our first pass at
this was worthless (I compared two configurations that both carried our delta):

boost integration suite
upstream master 40/42 — only test_gltf (assets)
this branch before the fix + test_pipeline_boost_structs crashes in the driver
this branch after the fix 40/42 — identical to master

The copyability report is in, as you suggested: generator/copyability_report.txt,
written on full runs beside the skip report, listing every boost struct that is not
copyable and the array field responsible (170 today). GraphicsPipelineCreateInfo
is listed for pStages; PipelineShaderStageCreateInfo is not listed, which is
precisely the line that would have appeared in the diff when this branch first
broke it. Same ratchet shape as the skip report, so a flip shows up as a review
diff rather than as a crash three CI rounds later.

Emitter and driver lint clean, dasfmt --verify clean, regeneration deterministic.

@borisbat borisbat left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the fix end-to-end on the lavapipe rig (daslang at master 2779541): the copyability probe shows the stage locals surviving both pStages <- [...] literals again, the full integration suite is 42 tests / 41 passed / 1 skipped / 0 failed — identical to the green master baseline — and regeneration is deterministic and byte-identical across all six daslib files, the skip report, and the new copyability report.

The in-place scheme is a genuine improvement over what it replaces, not just a patch: net −144 generated lines, and @do_not_delete also closes a latent master hazard (delete on a holder with a populated borrowed addr(local) pointer would have freed a non-chunk pointer). I also checked the copyability report's direct-array-field criterion for completeness — it's the full truth, since FieldKind has no boost-struct-by-value embed now that the clone scratch is gone. Nice work, and thanks for the fast turnaround on the round-2 finding.

Two tiny items we'll take as follow-ups on our side (no action needed): wiring copyability_report.txt into the CI ratchet diff, and the now-stale "cloned into scratch" doc-comment on FieldKind.opt_struct_ptr.

@borisbat
borisbat merged commit 1dd795d into borisbat:master Aug 14, 2026
5 checks passed
olegus8 pushed a commit to olegus8/dasVulkan-new that referenced this pull request Aug 15, 2026
…comment

Two follow-ups from the borisbat#81 review ledger:

- The generator already writes copyability_report.txt beside the skip report;
  the ratchet step now diffs it against the committed copy, so a copyable ->
  non-copyable flip in the boost surface fails CI instead of relying on
  committed regen. Verified locally: today's regen matches both reports.
- FieldKind.opt_struct_ptr said "(caller's addr; cloned into scratch)" - both
  halves stale since the view rework: the pointee is borrowed and raw-viewed
  into the parent view's scratch, and the raw pointer aims at that scratch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants