Skip to content

c api: enable JIT from embedded hosts; fix the macOS in-memory JIT arm - #3889

Merged
borisbat merged 1 commit into
masterfrom
bbatkin/capi-jit-enable
Aug 28, 2026
Merged

c api: enable JIT from embedded hosts; fix the macOS in-memory JIT arm#3889
borisbat merged 1 commit into
masterfrom
bbatkin/capi-jit-enable

Conversation

@borisbat

Copy link
Copy Markdown
Collaborator

Fixes #3885. The C API had no way to turn JIT on. An embedded host can now do what the CLI's -jit does: DAS_POLICY_JIT_ENABLED and DAS_POLICY_JIT_DLL_MODE join das_bool_policy, das_set_root(_n) sets the daslang root for hosts that live outside the distribution, and das_fileaccess_add_extra_module(_n) injects daslib/just_in_time.das the way -jit does. The embedding doc gains an "Enabling JIT" recipe. Script-side options jit_enabled = true stays inert: JIT is a host policy.

Testing the recipe uncovered that in-memory MCJIT execution never worked on macOS. This is the arm a static-linked host uses, and also -jit-no-cache. Mach-O codegen lowers llvm.global_dtors into __cxa_atexit(.., extern_weak @__dso_handle). RuntimeDyld cannot resolve __dso_handle in-process, and that one failed lookup silently aborts external-symbol resolution for the whole module, so every extern's GOT slot stays null and the first ctor jumps to address 0. Had resolution succeeded, the registration would fire at process exit() into the disposed engine's memory instead. The darwin in-memory arm now emits no dtor list at all; dll and exe arms are real images and keep it. CI never sees this arm on mac: the mac lane's daslang is a dll build, so its JIT sweep uses the dll-cache path. The new C test now covers it on every static ctest lane with LLVM present.

Two full-AOT (nightly-lane) fixes ride along: aot_builtin_ast.h gains the addModulePostRewriteMacro / addModulePostCompileMacro declarations that generated AOT for the ast_verify port already calls, and dasGlsl registers geom_gen.das as an AOT module lib so tests/glsl/test_geom_gen_winding.das links its geom_gen instantiations (fresh from #3886, would have reddened tonight's nightly).

Where to look: generate_global_ctors_dtors in modules/dasLLVM/daslib/llvm_jit.das (the fix and its WHY comment), src/misc/daScriptC.cpp (new surface), tests-cpp/small/test_capi_jit.cpp (the end-to-end recipe).

Validation, claims, ledger

Validation

  • macOS in-memory arm (no CI lane exists for it): full tests/jit_tests swept through it, 362/362, via a temporary jit_dll_mode = false in dastest (not committed); dll arm swept green unchanged.
  • dasLLVM module-owned suite on macOS: 51/51 (the changed arm is darwin-gated, so the platform matters; per its REVIEW.md no other lane covers this suite).
  • Full AOT sweep (nightly-only gate): 13093/13094, 0 failed, 1 skipped, after the two AOT fixes; full JIT sweep 13921/13922; interpreter sweep 14053/14056 (3 skips). The chained preflight --full run showed suite reds that did not reproduce standalone - box contention tripping --max-file-time 30; each red gate was re-run targeted and is green.
  • Negative controls (TDD audit): the C test fails when the policy switch-arm is deleted, when the extra module is inert, and when the dtor skip is reverted - the last one crashes at exit() AFTER doctest prints SUCCESS, so the exit code is the signal and ctest reads it.
  • LLVM_JIT_CODEGEN_VERSION bumped 0x54 -> 0x55 per the dasLLVM checklist rule; the changed arm writes no cache artifact, so the bump costs one cold rebuild of every .jitted_scripts cache and protects nothing - kept because the rule as written demands it.

Claims - stated, not tested

  • DAS_POLICY_JIT_DLL_MODE's payload is covered only by the set_bool return-code check: a static build cannot observe the field (no dll cache exists there), so a wrong-field assignment would pass the test. Verified by reading the switch arm.
  • dll/exe emission is byte-identical: the skip gate is jit_mode && darwin, and jit_mode is false for dll/exe. Verified by reading the guard, plus the dll-arm probe and suite.
  • On lanes without lib/LLVM.dll the JIT test case self-skips with a message; LLVM-enabled lanes (darwin26, linux, windows nightly) run it.

Not done

  • The review round surfaced five REVIEW.md checklist defects (dead routing line in tests-cpp, pin-rule wording that cannot surface cross-folder edits, the codegen-version trigger being wider than the property it protects, a missing platform clause on the module-suite rule, and a proposed green-by-skip rule). All are semantic rule rewrites - queued for a ruling, not legislated here.
  • A defined hidden __dso_handle dummy also fixes the resolution abort on its own (proven by IR-level bisection) but leaves the exit()-time crash; it was dropped in favor of the dtor skip, and the comment names it as the fallback if another lowering ever references __dso_handle.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FkBKGJisiiHXEgSsGjpqLm

Copilot AI lite review requested due to automatic review settings August 27, 2026 22:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends daScript’s C embedding surface so hosts can enable the LLVM-backed JIT (mirroring the CLI’s -jit behavior), and fixes a macOS-specific crash in the in-memory JIT path by preventing emission of global dtors for that arm. It also includes two AOT-related build fixes needed for nightly/full-AOT lanes.

Changes:

  • Add C API support for JIT enablement via new boolean policies, root override APIs, and “extra module” injection to auto-require daslib/just_in_time.das.
  • Fix darwin in-memory JIT emission by skipping llvm.global_dtors generation to avoid __dso_handle/__cxa_atexit pitfalls in RuntimeDyld scenarios.
  • Wire missing AOT module inputs/decls (GLSL geom_gen module stub registration; missing AOT builtin AST macro declarations).

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
tests/aot/CMakeLists.txt Adds GLSL module-lib AOT sources to the module AOT suite list.
tests-cpp/small/test_capi_jit.cpp New end-to-end doctest validating C API JIT enable + root round-trip.
src/misc/daScriptC.cpp Implements new C API functions (set root, add extra module) and new policy switch cases.
modules/dasLLVM/daslib/llvm_jit.das Skips llvm.global_dtors emission for darwin in-memory JIT arm; threads skip flag through helper.
modules/dasLLVM/daslib/llvm_jit_run.das Bumps codegen version + emitter hash for cache invalidation / pinning.
modules/dasGlsl/CMakeLists.txt Registers geom_gen.das as runtime-relevant AOT module input.
include/daScript/simulate/aot_builtin_ast.h Adds missing declarations used by generated AOT for AST macros.
include/daScript/daScriptC.h Adds new C API declarations and new das_bool_policy entries for JIT enablement.
doc/source/reference/embedding/c_api.rst Documents the new embedding recipe (“Enabling JIT”) and new APIs/policies.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread include/daScript/daScriptC.h Outdated
Comment thread modules/dasLLVM/daslib/llvm_jit_run.das Outdated
Comment thread doc/source/reference/embedding/c_api.rst Outdated
Copilot AI review requested due to automatic review settings August 27, 2026 22:45
@borisbat
borisbat force-pushed the bbatkin/capi-jit-enable branch from 5442d8c to 378d30d Compare August 27, 2026 22:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.

Comment thread include/daScript/daScriptC.h Outdated
Comment thread doc/source/reference/embedding/c_api.rst Outdated
Comment thread tests-cpp/small/test_capi_jit.cpp
Comment thread tests-cpp/small/test_capi_jit.cpp
Copilot AI review requested due to automatic review settings August 27, 2026 22:56
@borisbat
borisbat force-pushed the bbatkin/capi-jit-enable branch from 378d30d to 94159cf Compare August 27, 2026 22:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Comment thread src/misc/daScriptC.cpp
@borisbat
borisbat force-pushed the bbatkin/capi-jit-enable branch from 94159cf to c750c36 Compare August 27, 2026 23:05
Copilot AI review requested due to automatic review settings August 27, 2026 23:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Comment thread modules/dasLLVM/daslib/llvm_jit.das
Issue #3885: the C API had no way to turn JIT on - DAS_POLICY_JIT_ENABLED and
DAS_POLICY_JIT_DLL_MODE join das_bool_policy, das_set_root(_n) sets the daslang
root for hosts living outside the distribution, and
das_fileaccess_add_extra_module(_n) exposes the extra-module injection the CLI's
-jit uses for daslib/just_in_time.das. The embedding doc gains the JIT recipe.

Chasing the new test uncovered that in-memory MCJIT execution (static-linked
hosts, -jit-no-cache) never worked on macOS: Mach-O codegen lowers
llvm.global_dtors into __cxa_atexit(.., extern_weak @__dso_handle), RuntimeDyld's
failed lookup of that one symbol silently aborts external resolution for the
whole module, and the first ctor jumps through a null GOT slot; had resolution
succeeded, the registration would fire at process exit() into the disposed
engine's memory. CI never sees the arm - the mac lane's daslang is a dll build,
so its JIT sweep exercises only the dll-cache path. The darwin in-memory arm now
emits no dtor list at all; dll/exe emission changes shape for no configuration
that caches, but LLVM_JIT_CODEGEN_VERSION is bumped per the dasLLVM checklist
rule and the emitter-source pin re-pinned.

tests-cpp/small/test_capi_jit.cpp drives the full embedded recipe through the C
API - set root, policies, dynamic modules, extra module, compile, simulate - and
asserts jit_enabled() and is_jit_function() script-side; on macOS its static link
exercises the fixed in-memory arm. Skips when lib/LLVM.dll is absent.

Two full-AOT (nightly-lane) fixes ride along: aot_builtin_ast.h gains the
addModulePostRewriteMacro/addModulePostCompileMacro declarations the ast_verify
port's generated AOT already calls, and dasGlsl registers geom_gen.das as an
AOT module lib so tests/glsl/test_geom_gen_winding.das links its geom_gen
instantiations instead of failing with error 50101.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FkBKGJisiiHXEgSsGjpqLm
@borisbat
borisbat force-pushed the bbatkin/capi-jit-enable branch from c750c36 to 0fab03a Compare August 27, 2026 23:41
Copilot AI review requested due to automatic review settings August 27, 2026 23:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

@borisbat
borisbat merged commit 88fb8d6 into master Aug 28, 2026
38 checks passed
@borisbat
borisbat deleted the bbatkin/capi-jit-enable branch August 28, 2026 00:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

How to turn on JIT mode when using the C interface?

2 participants