diff --git a/.gitignore b/.gitignore index 7d9aead..a52dd35 100644 --- a/.gitignore +++ b/.gitignore @@ -109,3 +109,7 @@ sync_report*.txt # Hypatia scan cache (local-only) .hypatia/ target/ + +# Zig build cache/output +**/.zig-cache/ +**/zig-out/ diff --git a/0-AI-MANIFEST.a2ml b/0-AI-MANIFEST.a2ml index 3f982ef..f0ac8ad 100644 --- a/0-AI-MANIFEST.a2ml +++ b/0-AI-MANIFEST.a2ml @@ -12,9 +12,14 @@ This is the AI manifest for **julianiser**. It declares: - Critical invariants (rules that must NEVER be violated) - Repository structure and organization -Julianiser analyses Python and R data science code, identifies -performance-critical array/dataframe operations, and generates -equivalent Julia modules that deliver 10-100x speedups via LLVM JIT. +Julianiser analyses Python and R data science code, identifies known +library calls, and generates equivalent Julia modules via the Rust +codegen pipeline (`src/codegen/`). Julia's LLVM JIT is well-documented +to deliver large (commonly cited as 10-100x) speedups on this class of +workload; julianiser generates benchmark scaffolding for users to +measure that on their own pipeline, but does not itself measure or +ship a speedup number today. See "HONESTY NOTE" below re: the +Idris2-ABI/Zig-FFI scaffolding. ## CANONICAL LOCATIONS (UNIVERSAL RULE) @@ -79,28 +84,53 @@ Policy enforcement contracts (k9, dust, lust, must, trust). ## PROJECT-SPECIFIC CONTEXT -### What Julianiser Does - -1. Parses Python source (via AST) or R scripts (via R parser) -2. Identifies array operations, dataframe transformations, numeric hot paths -3. Generates Julia equivalents with type annotations and broadcasting -4. Creates a Zig FFI bridge for calling Julia from existing code -5. Benchmarks original vs. generated code to verify speedup - -### Key Translation Patterns +### What Julianiser Does (operational today) + +1. Parses Python/R source with a line-based scanner (`src/codegen/parser.rs`) — + NOT a full AST parser; detects import statements and library-qualified calls +2. Resolves qualifiers ONLY through the known import-alias table; unresolved + qualifiers (local variables, e.g. `df.dropna()`) are skipped, never + fabricated into a fake "library" or a hard runtime error (see G31 fix) +3. Looks up each detected call in a source-to-Julia translation table + (`src/codegen/julia_gen.rs`) and emits a Julia module; unmapped calls get a + `# TODO` comment instead of a hard error +4. Generates benchmark *scaffolding* (`benchmark.jl`, `benchmark_runner.sh`, + `results.toml`) for the user to run and record their own timings — + julianiser does not measure or claim a speedup number itself + +### HONESTY NOTE — Idris2 ABI / Zig FFI is scaffolding, not operational + +`src/interface/abi/` (Idris2) and `src/interface/ffi/` (Zig) exist as source +files but are **not part of the working pipeline**: +- The Idris2 proofs are not built or checked by `cargo build`/`cargo test`. +- The Zig FFI's exported functions are TODO stubs: `julianiser_parse_python`/ + `julianiser_parse_r` are no-ops that only record the language tag, + `julianiser_codegen` returns a fixed placeholder module string, and + `julianiser_benchmark` hardcodes `speedup = 1.0`. +- The Rust CLI never dlopens or links against the Zig library. Codegen is + 100% Rust string generation in `src/codegen/`. +- Do NOT describe this layer as "the operational pipeline" in any doc. It is + tracked for future wiring. If you touch it, keep this note in sync. + +### Key Translation Patterns (via src/codegen/julia_gen.rs lookup tables) - `pandas.DataFrame` -> `DataFrames.jl` - `numpy` arrays -> native Julia arrays (with broadcasting) - `scipy` -> Julia stdlib / Optim.jl - R `data.frame` -> `DataFrames.jl` -- 0-based indexing -> 1-based indexing (formally verified remapping) ### Architecture -- **Rust CLI** (`src/main.rs`) — orchestration -- **Idris2 ABI** (`src/interface/abi/`) — SourceLanguage, DataFrameOperation, ArrayPattern, JuliaType, EquivalenceWitness -- **Zig FFI** (`src/interface/ffi/`) — C-ABI bridge with Python/R parse + Julia codegen -- **Codegen** (`src/codegen/`) — Julia code generation from typed IR +- **Rust CLI** (`src/main.rs`) — orchestration; the actual entry point for + `init`/`validate`/`generate`/`build`/`run` +- **Codegen** (`src/codegen/`) — the operational translation pipeline: + parser.rs (detection) + julia_gen.rs (translation) + benchmark.rs (scaffold + generation) +- **Idris2 ABI** (`src/interface/abi/`) — SCAFFOLDING: SourceLanguage, + DataFrameOperation, ArrayPattern, JuliaType, EquivalenceWitness declared but + not wired into the build +- **Zig FFI** (`src/interface/ffi/`) — SCAFFOLDING: TODO-stubbed C-ABI bridge, + never called by the Rust CLI ### Allowed Languages @@ -121,6 +151,7 @@ TypeScript, Node.js, npm, Python (as implementation language — Python is a *ta 7. **Container runtime** - Podman, never Docker. Files are `Containerfile`, never `Dockerfile` 8. **Container orchestration** - `selur-compose`, never `docker-compose` 9. **Julia is the target, not the implementation** - Julianiser is written in Rust/Idris2/Zig; it *generates* Julia +10. **No overclaiming the FFI seam** - Never describe the Idris2-ABI/Zig-FFI layer as operational or wired; it is scaffolding until the Rust CLI actually links/dlopens it and a benchmark shows a real, measured speedup (see HONESTY NOTE above) ## REPOSITORY STRUCTURE diff --git a/README.md b/README.md index 258bb9f..c205f40 100644 --- a/README.md +++ b/README.md @@ -6,49 +6,80 @@ SPDX-FileCopyrightText: 2025-2026 Jonathan D.A. Jewell # What Is This? Julianiser analyses existing Python and R data science code, identifies -performance-critical array and dataframe operations, and generates -equivalent Julia code with full type annotations — producing **drop-in -replacement modules** that deliver **10-100x speedups** through Julia’s -LLVM JIT compilation. - -Scientists keep their Python notebooks. Julia runs underneath. - -# How It Works - -Describe your data pipeline in a `julianiser.toml` manifest. Julianiser: - -1. **Parses** your Python functions (via AST) or R scripts (via R - parser) - -2. **Identifies** array operations, dataframe transformations, and - numeric hot paths - -3. **Generates** Julia equivalents with proper type annotations and - broadcasting - -4. **Creates** a Zig FFI bridge for calling Julia from your existing - code (zero-copy where possible) - -5. **Benchmarks** the original vs. generated code to verify speedup - -6. **Falls back** gracefully — if Julia is not installed, your Python/R - code still runs +common library calls (pandas/numpy/scipy/dplyr/ggplot2 patterns), and +generates a corresponding Julia module using known source-to-Julia +function mappings. The goal is drop-in replacement modules that let you +run the equivalent logic under Julia's LLVM JIT compilation. + +**Current status (honest, as of this writing):** the working, tested +pipeline is the **Rust codegen** in `src/codegen/` (line-based +Python/R parsing + template-based Julia generation), invoked by the +Rust CLI (`src/main.rs`). It does not (yet) measure or claim any actual +speedup — the generated `benchmarks/results.toml` ships with +`speedup = 0.0 # TODO: fill in`, to be measured by the user after +running the generated Julia code against the original. Any "10-100x" +figures elsewhere in this repository's docs describe the *general, +well-documented characteristics of Julia's JIT* on this class of +workload, not a number julianiser itself has measured. + +The **Idris2-ABI (`src/interface/abi/`) and Zig-FFI (`src/interface/ffi/`)** +layer described in earlier drafts of this README is **scaffolding**: +the Idris2 proofs and the Zig C-ABI bridge exist as source files, but +the Zig implementation is TODO-stubbed (parsing is a no-op, codegen +returns a hardcoded placeholder module, and the benchmark hook returns +a hardcoded `speedup = 1.0`), and the Rust CLI never links, dlopens, or +otherwise calls into it. It is tracked for future wiring, not part of +the operational translation path today. + +Scientists keep their Python notebooks. Julia runs underneath, once +generated. + +# How It Works (today) + +Describe your data pipeline in a `julianiser.toml` manifest. The Rust +CLI, via `src/codegen/`: + +1. **Parses** your Python or R source with a line-based scanner + (`src/codegen/parser.rs`) that detects import statements and + library-qualified calls + +2. **Looks up** each detected call in a table of known source-language + → Julia translations (`src/codegen/julia_gen.rs`) + +3. **Generates** a Julia module with `using` statements and translated + calls (or a `# TODO` comment where no mapping exists) + +4. **Generates** benchmark scaffolding (`benchmarks/benchmark.jl`, + `benchmark_runner.sh`, `results.toml`) so *you* can measure and + record the speedup after running both versions — julianiser does + not measure it for you yet + +5. **Falls back** gracefully — if Julia is not installed, your Python/R + code still runs; the generated Julia is optional, additive output + +The Idris2-ABI / Zig-FFI seam described below is planned future +infrastructure for a compiled, zero-copy bridge; it is not required for +(and is not currently used by) the codegen path above. # Key Value - **No Julia knowledge required** — Julianiser handles the translation + for the library calls it recognises -- **10-100x speedups** on numeric, array, and dataframe code without - manual rewrites +- **Julia's JIT is well-documented to deliver large speedups** on + numeric, array, and dataframe code — julianiser generates the scripts + to measure that for *your* workload; it does not yet ship a measured + number of its own - **Gradual adoption** — julianise one function at a time, benchmark each -- **Drop-in modules** — generated Julia code exposes the same API as - your Python/R functions +- **Drop-in modules** — generated Julia code exposes a `run_pipeline()` + entry point mirroring the detected calls in your Python/R script -- **Formally verified bridges** — Idris2 ABI proofs guarantee interface - correctness +- **Idris2 ABI + Zig FFI scaffolding exists** (`src/interface/`) for a + future formally-verified, compiled bridge — not yet wired into the + translation path (see Architecture, below) # Supported Patterns @@ -90,29 +121,50 @@ language. # Architecture Follows the hyperpolymath -iser pattern (same as -[Chapeliser](https://github.com/hyperpolymath/chapeliser)): +[Chapeliser](https://github.com/hyperpolymath/chapeliser)) in file +layout, but — to be explicit about what is operational today versus +what is scaffolding for later: + +**Operational (this is what `julianiser generate` actually runs):** - **Manifest** (`julianiser.toml`) — describe WHAT you want julianised -- **Source Parser** — Python AST analysis / R parser for identifying - translatable patterns +- **Source Parser** (`src/codegen/parser.rs`) — line-based Python/R + scanner that detects import statements and known library-qualified + calls + +- **Julia Codegen** (`src/codegen/julia_gen.rs`) — looks up each + detected call in a source-to-Julia translation table and emits a + Julia module; unmapped calls become a `# TODO` comment, not a runtime + error -- **Idris2 ABI** (`src/interface/abi/`) — formal proofs of equivalence - between source and generated code +- **Benchmark scaffold generator** (`src/codegen/benchmark.rs`) — + generates `benchmark.jl` / `benchmark_runner.sh` / `results.toml` so + the user can run and record original-vs-Julia timings themselves -- **Julia Codegen** (`src/codegen/`) — generates type-annotated Julia - with proper broadcasting +- **Rust CLI** (`src/main.rs`) — orchestrates manifest validation, + parsing, and generation (`init`, `validate`, `generate`, `build`, + `run`) -- **Zig FFI** (`src/interface/ffi/`) — C-ABI bridge between Julia - runtime and calling code +**Scaffolding — present in the tree, not yet wired into the CLI:** -- **Benchmark Harness** — compares original Python/R vs. generated Julia - performance +- **Idris2 ABI** (`src/interface/abi/`) — formal type/layout + declarations intended to eventually prove equivalence between source + and generated code. Exists as `.idr` source; not built or checked by + `cargo build`/`cargo test`. -- **Rust CLI** (`src/main.rs`) — orchestrates parsing, validation, - generation, building, and benchmarking +- **Zig FFI** (`src/interface/ffi/`) — a C-ABI bridge skeleton. The + exported functions are TODO-stubbed (parsing is a no-op that only + records the source language; codegen returns a fixed placeholder + module string; the benchmark hook returns a hardcoded + `speedup = 1.0`). The Rust CLI never calls into this library — no + `dlopen`, no linking. It is tracked for future wiring, not part of + today's translation pipeline. -User writes zero Julia code. Julianiser generates everything. +Today, the user writes zero Julia code and the Rust codegen generates +the translated module; the Idris2/Zig seam is where a future, +formally-verified, compiled bridge is planned to replace the current +pure-codegen approach. Part of the [-iser family](https://github.com/hyperpolymath/iseriser) of acceleration frameworks. @@ -150,9 +202,13 @@ julianiser run --benchmark # Status -**Codebase in progress.** Architecture defined, CLI scaffolded, codegen -pending. Phase 0 (scaffold) complete. Phase 1 (Python AST parser + Julia -codegen) underway. +**Codebase in progress.** The Rust CLI and codegen pipeline +(`src/codegen/`) are implemented and covered by an automated test suite +(`cargo test`). The Idris2-ABI / Zig-FFI layer (`src/interface/`) is +scaffolding — present in the tree but not yet built, checked, or called +by the Rust CLI. No speedup number has been measured by julianiser +itself; the generated benchmark scripts exist for the user to measure +their own workload. # License diff --git a/TOPOLOGY.md b/TOPOLOGY.md index 1dae26e..6629c55 100644 --- a/TOPOLOGY.md +++ b/TOPOLOGY.md @@ -3,9 +3,20 @@ ## Purpose -Julianiser analyses Python and R data science code, identifies performance-critical -array/dataframe operations, and generates equivalent Julia modules that deliver -10-100x speedups via LLVM JIT compilation. +Julianiser analyses Python and R data science code, identifies known +pandas/numpy/scipy/dplyr/ggplot2-style calls, and generates equivalent Julia +modules via the Rust codegen pipeline (`src/codegen/`). Julia's LLVM JIT is +well-documented to deliver large speedups on this class of workload; julianiser +generates benchmark scaffolding so users can measure that for their own +pipeline, but does not itself measure or claim a speedup number today. + +**Honesty note (see README.md for full detail):** the Idris2 ABI +(`src/interface/abi/`) and Zig FFI (`src/interface/ffi/`) below are +scaffolding — source files exist, but the Zig implementation is +TODO-stubbed and the Rust CLI never links or calls into it. The +operational translation pipeline today is the Rust codegen +(`src/codegen/parser.rs` + `src/codegen/julia_gen.rs`), invoked directly +by `src/main.rs`. ## Module Map @@ -16,17 +27,17 @@ julianiser/ │ ├── lib.rs # Library API │ ├── manifest/mod.rs # julianiser.toml parser and validator │ ├── codegen/mod.rs # Julia code generation engine -│ ├── abi/mod.rs # Rust-side ABI types mirroring Idris2 proofs -│ └── interface/ # Verified Interface Seams -│ ├── abi/ # Idris2 ABI (The Spec) +│ ├── abi/mod.rs # Rust-side domain types (not linked to Idris2 below) +│ └── interface/ # Interface Seam SCAFFOLDING (planned, not wired — see Data Flow) +│ ├── abi/ # Idris2 ABI (The Spec) — source only, not built by cargo │ │ ├── Types.idr # SourceLanguage, DataFrameOp, ArrayPattern, JuliaType, EquivalenceWitness │ │ ├── Layout.idr # AST node memory layout with alignment proofs │ │ └── Foreign.idr # FFI declarations for Python/R parsing and Julia codegen -│ ├── ffi/ # Zig FFI (The Bridge) +│ ├── ffi/ # Zig FFI (The Bridge) — TODO-stubbed, never called by the Rust CLI │ │ ├── build.zig # Zig build config (shared + static lib) -│ │ ├── src/main.zig # FFI implementation (parse, translate, codegen) -│ │ └── test/ # Integration tests verifying ABI compliance -│ └── generated/abi/ # Auto-generated C headers from Idris2 +│ │ ├── src/main.zig # FFI stub (parse/codegen/benchmark are placeholders) +│ │ └── test/ # Zig-side unit tests of the stub behaviour +│ └── generated/abi/ # Auto-generated C headers from Idris2 (placeholder dir) ├── verification/ # Proofs, benchmarks, fuzzing, safety cases │ ├── benchmarks/ # Python/R vs Julia performance comparisons │ ├── proofs/ # Formal verification artifacts @@ -47,27 +58,36 @@ julianiser/ └── integrations/ # proven, verisimdb, vexometer, feedback-o-tron ``` -## Data Flow +## Data Flow (operational today) ``` -julianiser.toml ──→ Manifest Parser ──→ Source Parser (Python AST / R parser) - │ - ▼ - Typed IR (operations, types, data flow) +julianiser.toml ──→ Manifest Parser ──→ Source Parser (line-based Python/R scan) + (src/manifest/) (src/codegen/parser.rs) │ ▼ - Idris2 ABI (equivalence proofs) + Detected library calls + (import aliases resolved; + local-variable calls skipped) │ ▼ - Julia Codegen (type-annotated, broadcasting) + Julia Codegen (template lookup per call) + (src/codegen/julia_gen.rs) │ ▼ - Zig FFI Bridge (C-ABI interop) - │ - ▼ - Benchmark Harness (original vs generated) + Benchmark scaffold generator + (src/codegen/benchmark.rs — emits scripts + for the user to run and record timings) ``` +## Data Flow (planned, not yet wired) + +The Idris2 ABI (`src/interface/abi/`) and Zig FFI (`src/interface/ffi/`) +are intended to eventually sit between source parsing and Julia codegen +as a formally-verified, compiled bridge. Today they are source-only +scaffolding: the Idris2 proofs are not built or checked by `cargo +build`/`cargo test`, and the Zig implementation's exported functions +are TODO stubs that the Rust CLI never calls. + ## Key Dependencies | Dependency | Purpose | diff --git a/src/codegen/parser.rs b/src/codegen/parser.rs index 6fbc89f..daa2b8f 100644 --- a/src/codegen/parser.rs +++ b/src/codegen/parser.rs @@ -217,6 +217,19 @@ pub fn parse_python_source(source_path: &str, content: &str) -> Result lib, + None => { + // Unresolved qualifier — local variable method call. + // Skip: do not record as a translatable library call. + i += 1; // move past the opening paren + continue; + } }; // Extract arguments (simplified: everything between parens). @@ -499,8 +523,13 @@ fn detect_r_bare_calls( /// Extract arguments from a parenthesised expression. /// /// Given a string and the position of the opening `(`, extracts the -/// comma-separated arguments as raw strings. Handles nested parentheses -/// at one level of depth. +/// comma-separated arguments as raw strings. Tracks nesting depth across +/// `()`, `[]`, and `{}` (all three bracket kinds share one depth counter, +/// since a comma inside any of them is not an argument separator), and +/// treats single- and double-quoted string literals as opaque — brackets +/// and commas inside a quoted string never affect depth or splitting. +/// This ensures calls like `np.array([1, 2, 3])` or `f("a, b", [1, 2])` +/// are split into the correct top-level arguments. fn extract_parenthesised_args(s: &str, open_paren_pos: usize) -> Vec { let chars: Vec = s.chars().collect(); if open_paren_pos >= chars.len() || chars[open_paren_pos] != '(' { @@ -512,16 +541,44 @@ fn extract_parenthesised_args(s: &str, open_paren_pos: usize) -> Vec { let mut current_arg = String::new(); let mut i = open_paren_pos; + // Tracks whether we're inside a quoted string literal, and which + // quote character opened it (so e.g. `'it\'s'` style nesting of the + // *other* quote type inside a string doesn't get misread). + let mut in_string: Option = None; + while i < chars.len() { let ch = chars[i]; + + if let Some(quote) = in_string { + // Inside a string literal: only watch for the closing quote + // (respecting a backslash escape) or an unterminated line end. + current_arg.push(ch); + if ch == '\\' && i + 1 < chars.len() { + // Copy the escaped character verbatim and skip past it so + // e.g. `\"` inside a double-quoted string doesn't close it. + i += 1; + current_arg.push(chars[i]); + } else if ch == quote { + in_string = None; + } + i += 1; + continue; + } + match ch { - '(' => { + '\'' | '"' => { + in_string = Some(ch); + if depth >= 1 { + current_arg.push(ch); + } + } + '(' | '[' | '{' => { depth += 1; if depth > 1 { current_arg.push(ch); } } - ')' => { + ')' | ']' | '}' => { depth -= 1; if depth == 0 { let trimmed = current_arg.trim().to_string(); @@ -650,4 +707,83 @@ result <- dplyr::filter(df, x > 0) let args = extract_parenthesised_args("func(a, inner(b, c), d)", 4); assert_eq!(args, vec!["a", "inner(b, c)", "d"]); } + + // -- G30: bracket/quote-aware argument extraction ---------------------- + + #[test] + fn test_extract_args_with_list_literal() { + // Commas inside a `[...]` list must not split the argument. + let args = extract_parenthesised_args("array([1,2,3])", 5); + assert_eq!(args, vec!["[1,2,3]"]); + } + + #[test] + fn test_extract_args_with_list_literal_and_spaces() { + let args = extract_parenthesised_args("array([1, 2, 3])", 5); + assert_eq!(args, vec!["[1, 2, 3]"]); + } + + #[test] + fn test_extract_args_with_dict_literal() { + // Commas inside a `{...}` dict must not split the argument. + let args = extract_parenthesised_args("DataFrame({\"a\": 1, \"b\": 2})", 9); + assert_eq!(args, vec!["{\"a\": 1, \"b\": 2}"]); + } + + #[test] + fn test_extract_args_with_nested_brackets_and_extra_arg() { + let args = extract_parenthesised_args("func([1, 2], 3)", 4); + assert_eq!(args, vec!["[1, 2]", "3"]); + } + + #[test] + fn test_extract_args_comma_inside_string_literal() { + // A comma embedded in a string literal argument must not split it. + let args = extract_parenthesised_args("func(\"a, b\", c)", 4); + assert_eq!(args, vec!["\"a, b\"", "c"]); + } + + #[test] + fn test_extract_args_bracket_inside_string_literal() { + // A bracket character embedded in a string must not affect depth. + let args = extract_parenthesised_args("func(\"[not, a, list]\", c)", 4); + assert_eq!(args, vec!["\"[not, a, list]\"", "c"]); + } + + #[test] + fn test_extract_args_single_quoted_string() { + let args = extract_parenthesised_args("func('a, b', c)", 4); + assert_eq!(args, vec!["'a, b'", "c"]); + } + + /// G30 (critical): translating `np.array([1,2,3])` must preserve the + /// full list literal in the generated Julia call, rather than stopping + /// at the first comma inside the brackets (which previously produced + /// unparseable output like `collect([1)`). + #[test] + fn test_numpy_array_full_list_preserved_in_translation() { + let code = "import numpy as np\narr = np.array([1,2,3])\n"; + let unit = parse_python_source("test.py", code).expect("parse should succeed"); + + let array_call = unit + .detected_calls + .iter() + .find(|c| c.function == "array") + .expect("array call should be detected"); + assert_eq!(array_call.arguments, vec!["[1,2,3]"]); + + let julia_code = + crate::codegen::julia_gen::generate_julia_module(&unit, &[]); + assert!( + julia_code.contains("[1,2,3]") || julia_code.contains("[1, 2, 3]"), + "Generated Julia must contain the full list literal, got:\n{}", + julia_code + ); + // The previous bug truncated at the first comma, producing + // `collect([1)` — assert that specific breakage is gone. + assert!( + !julia_code.contains("collect([1)"), + "Generated Julia must not contain the truncated/unparseable form" + ); + } } diff --git a/src/interface/ffi/bench/bench.zig b/src/interface/ffi/bench/bench.zig new file mode 100644 index 0000000..2c4d47e --- /dev/null +++ b/src/interface/ffi/bench/bench.zig @@ -0,0 +1,46 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) 2026 Jonathan D.A. Jewell (hyperpolymath) +// +// Julianiser FFI — minimal benchmark harness executable. +// +// STATUS: scaffolding. Exercises the current TODO-stubbed FFI lifecycle +// (init → parse → codegen → benchmark → free) purely to give `zig build +// bench` something real to run and time. The underlying `julianiser_codegen` +// / `julianiser_benchmark` calls are placeholders (see src/main.zig) — the +// numbers this prints measure the stub's overhead, not real Python/R-to- +// Julia translation performance. Do not cite this as a speedup measurement. + +const std = @import("std"); + +extern fn julianiser_init() ?*anyopaque; +extern fn julianiser_free(handle: ?*anyopaque) void; +extern fn julianiser_parse_python(handle: ?*anyopaque, path_ptr: ?[*]const u8, path_len: u32) c_int; +extern fn julianiser_codegen(handle: ?*anyopaque) c_int; +extern fn julianiser_benchmark(handle: ?*anyopaque, iterations: u32) c_int; +extern fn julianiser_get_speedup(handle: ?*anyopaque) f64; +extern fn julianiser_version() [*:0]const u8; + +pub fn main() !void { + const stdout = std.io.getStdOut().writer(); + + try stdout.print("julianiser FFI bench (scaffolding stub) — version {s}\n", .{julianiser_version()}); + + var timer = try std.time.Timer.start(); + + const handle = julianiser_init() orelse { + try stdout.print("init failed\n", .{}); + return error.InitFailed; + }; + defer julianiser_free(handle); + + const path = "bench.py"; + _ = julianiser_parse_python(handle, path.ptr, path.len); + _ = julianiser_codegen(handle); + _ = julianiser_benchmark(handle, 1000); + + const elapsed_ns = timer.read(); + const speedup = julianiser_get_speedup(handle); + + try stdout.print("lifecycle elapsed: {d}ns\n", .{elapsed_ns}); + try stdout.print("reported speedup (stub placeholder, not measured): {d}\n", .{speedup}); +} diff --git a/src/interface/ffi/build.zig b/src/interface/ffi/build.zig index 8298e75..f5020c6 100644 --- a/src/interface/ffi/build.zig +++ b/src/interface/ffi/build.zig @@ -9,15 +9,22 @@ pub fn build(b: *std.Build) void { const optimize = b.standardOptimizeOption(.{}); // Shared library (.so, .dylib, .dll) + // + // NOTE (Zig 0.13 compat): `.version` must be passed inline here, not + // assigned to `lib.version` after construction. When assigned after + // the fact, `Compile.zig` never populates `major_only_filename` / + // `name_only_filename`, and `InstallArtifact.create` unconditionally + // dereferences them once `artifact.version != null` — crashing + // `zig build` with "attempt to use null value". const lib = b.addSharedLibrary(.{ .name = "julianiser", .root_source_file = b.path("src/main.zig"), .target = target, .optimize = optimize, + .version = .{ .major = 0, .minor = 1, .patch = 0 }, }); - - // Set version - lib.version = .{ .major = 0, .minor = 1, .patch = 0 }; + // main.zig uses std.heap.c_allocator, which requires libc. + lib.linkLibC(); // Static library (.a) const lib_static = b.addStaticLibrary(.{ @@ -26,13 +33,14 @@ pub fn build(b: *std.Build) void { .target = target, .optimize = optimize, }); + lib_static.linkLibC(); // Install artifacts b.installArtifact(lib); b.installArtifact(lib_static); // Generate header file for C compatibility - const header = b.addInstallHeader( + const header = b.addInstallHeaderFile( b.path("include/julianiser.h"), "julianiser.h", ); @@ -44,6 +52,7 @@ pub fn build(b: *std.Build) void { .target = target, .optimize = optimize, }); + lib_tests.linkLibC(); const run_lib_tests = b.addRunArtifact(lib_tests); @@ -70,6 +79,7 @@ pub fn build(b: *std.Build) void { .target = target, .optimize = .Debug, }); + docs.linkLibC(); const docs_step = b.step("docs", "Generate documentation"); docs_step.dependOn(&b.addInstallDirectory(.{ diff --git a/src/interface/ffi/include/julianiser.h b/src/interface/ffi/include/julianiser.h new file mode 100644 index 0000000..0193b64 --- /dev/null +++ b/src/interface/ffi/include/julianiser.h @@ -0,0 +1,74 @@ +/* SPDX-License-Identifier: MPL-2.0 */ +/* Copyright (c) 2026 Jonathan D.A. Jewell (hyperpolymath) */ +/* + * Julianiser FFI — C header + * + * C-ABI declarations for the Zig FFI implementation in src/main.zig. + * This header mirrors the `export fn` surface of that file exactly; keep + * the two in sync when either changes. + * + * STATUS: scaffolding. The Zig implementation behind this header is + * TODO-stubbed (see src/main.zig) and is not currently linked into the + * julianiser Rust CLI. See 0-AI-MANIFEST.a2ml "HONESTY NOTE" for detail. + */ + +#ifndef JULIANISER_H +#define JULIANISER_H + +#include + +#ifdef __cplusplus +extern "C" { +#endif + +/* Opaque session handle. */ +typedef void julianiser_handle; + +/* Result codes — must match the `Result` enum in src/main.zig. */ +typedef enum { + JULIANISER_OK = 0, + JULIANISER_ERROR = 1, + JULIANISER_INVALID_PARAM = 2, + JULIANISER_OUT_OF_MEMORY = 3, + JULIANISER_NULL_POINTER = 4, + JULIANISER_PARSE_ERROR = 5, + JULIANISER_CODEGEN_ERROR = 6, +} julianiser_result; + +/* Lifecycle */ +julianiser_handle *julianiser_init(void); +void julianiser_free(julianiser_handle *handle); + +/* Source parsing */ +int julianiser_parse_python(julianiser_handle *handle, const uint8_t *path_ptr, uint32_t path_len); +int julianiser_parse_r(julianiser_handle *handle, const uint8_t *path_ptr, uint32_t path_len); + +/* AST query */ +uint32_t julianiser_node_count(julianiser_handle *handle); + +/* Julia code generation */ +int julianiser_codegen(julianiser_handle *handle); +const char *julianiser_get_julia_code(julianiser_handle *handle); + +/* Benchmark operations */ +int julianiser_benchmark(julianiser_handle *handle, uint32_t iterations); +double julianiser_get_speedup(julianiser_handle *handle); + +/* String operations */ +void julianiser_free_string(const char *str); + +/* Error handling */ +const char *julianiser_last_error(void); + +/* Version information */ +const char *julianiser_version(void); +const char *julianiser_build_info(void); + +/* Utility */ +uint32_t julianiser_is_initialized(julianiser_handle *handle); + +#ifdef __cplusplus +} +#endif + +#endif /* JULIANISER_H */ diff --git a/src/interface/ffi/test/integration_test.zig b/src/interface/ffi/test/integration_test.zig index 8302850..380c71a 100644 --- a/src/interface/ffi/test/integration_test.zig +++ b/src/interface/ffi/test/integration_test.zig @@ -32,7 +32,10 @@ test "create and destroy session" { const handle = julianiser_init() orelse return error.InitFailed; defer julianiser_free(handle); - try testing.expect(handle != null); + // `orelse` above already unwrapped the optional, so `handle` is a + // non-optional `*anyopaque` here — reaching this line at all proves + // julianiser_init() did not return null. + try testing.expect(@intFromPtr(handle) != 0); } test "session is initialized after creation" { diff --git a/tests/integration_tests.rs b/tests/integration_tests.rs index d32090d..86efc73 100644 --- a/tests/integration_tests.rs +++ b/tests/integration_tests.rs @@ -436,6 +436,72 @@ fn test_abi_types_serde_round_trip() { assert_eq!(unit.output_path, "my_data_pipeline.jl"); } +// ========================================================================= +// Test 9 (G31): Local-variable method calls must not become hard errors +// ========================================================================= + +/// G31 (critical): the shipped `examples/data-pipeline/pipeline.py` calls +/// several methods on local variables that are NOT import aliases — +/// `df.dropna()`, `filtered.groupby(...)`, `summary.sort_values(...)`, +/// `result.to_csv(...)`. Before the fix, the parser treated ANY +/// `qualifier.function(` as a library-qualified call, so these local +/// method chains were misdetected as calls to a (nonexistent) library +/// named e.g. "df" or "result", which codegen then turned into a hard +/// `error("Unmapped call: df.dropna")` INSIDE the generated +/// `run_pipeline()` — meaning the "drop-in replacement" would throw on +/// its first DataFrame method call. This test asserts the generated +/// module contains no such hard error for the example pipeline. +#[test] +fn test_local_method_calls_do_not_produce_unmapped_call_errors() { + let pipeline_source = std::fs::read_to_string(concat!( + env!("CARGO_MANIFEST_DIR"), + "/examples/data-pipeline/pipeline.py" + )) + .expect("examples/data-pipeline/pipeline.py should exist and be readable"); + + let unit = julianiser::parse_source_string( + "pipeline.py", + &pipeline_source, + SourceLanguage::Python, + ) + .expect("Parsing the example pipeline should succeed"); + + // Sanity: local-variable qualifiers must not leak in as fabricated + // "libraries" (df, filtered, summary, result are never imported). + let libraries: Vec<&str> = unit + .detected_calls + .iter() + .map(|c| c.library.as_str()) + .collect(); + for local_var in ["df", "filtered", "summary", "result"] { + assert!( + !libraries.contains(&local_var), + "Local variable '{}' must not be detected as a library", + local_var + ); + } + + let julia_code = julia_gen::generate_julia_module(&unit, &[]); + + assert!( + !julia_code.contains("error(\"Unmapped call"), + "Generated module must never hard-error on local method calls; got:\n{}", + julia_code + ); + + // Real library calls (pd.read_csv, np.array, np.mean, np.std, np.dot) + // must still be detected and translated normally — the fix must not + // over-correct and drop legitimate library calls too. + assert!( + libraries.contains(&"pandas"), + "Should still detect pandas calls (pd.read_csv)" + ); + assert!( + libraries.contains(&"numpy"), + "Should still detect numpy calls (np.array, np.mean, ...)" + ); +} + // ========================================================================= // Test 8: Manifest validation catches invalid inputs // =========================================================================