Skip to content

fix(codegen): parseable non-throwing Julia output; honest FFI relabel; Zig 0.13 build #48

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hyperpolymath merged 1 commit into from
Jul 1, 2026
Merged

fix(codegen): parseable non-throwing Julia output; honest FFI relabel; Zig 0.13 build #48

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,3 +109,7 @@ sync_report*.txt
# Hypatia scan cache (local-only)
.hypatia/
target/

# Zig build cache/output
**/.zig-cache/
**/zig-out/
65 changes: 48 additions & 17 deletions 0-AI-MANIFEST.a2ml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,14 @@ This is the AI manifest for **julianiser**. It declares:
- Critical invariants (rules that must NEVER be violated)
- Repository structure and organization

Julianiser analyses Python and R data science code, identifies
performance-critical array/dataframe operations, and generates
equivalent Julia modules that deliver 10-100x speedups via LLVM JIT.
Julianiser analyses Python and R data science code, identifies known
library calls, and generates equivalent Julia modules via the Rust
codegen pipeline (`src/codegen/`). Julia's LLVM JIT is well-documented
to deliver large (commonly cited as 10-100x) speedups on this class of
workload; julianiser generates benchmark scaffolding for users to
measure that on their own pipeline, but does not itself measure or
ship a speedup number today. See "HONESTY NOTE" below re: the
Idris2-ABI/Zig-FFI scaffolding.

## CANONICAL LOCATIONS (UNIVERSAL RULE)

Expand Down Expand Up @@ -79,28 +84,53 @@ Policy enforcement contracts (k9, dust, lust, must, trust).

## PROJECT-SPECIFIC CONTEXT

### What Julianiser Does

1. Parses Python source (via AST) or R scripts (via R parser)
2. Identifies array operations, dataframe transformations, numeric hot paths
3. Generates Julia equivalents with type annotations and broadcasting
4. Creates a Zig FFI bridge for calling Julia from existing code
5. Benchmarks original vs. generated code to verify speedup

### Key Translation Patterns
### What Julianiser Does (operational today)

1. Parses Python/R source with a line-based scanner (`src/codegen/parser.rs`) —
NOT a full AST parser; detects import statements and library-qualified calls
2. Resolves qualifiers ONLY through the known import-alias table; unresolved
qualifiers (local variables, e.g. `df.dropna()`) are skipped, never
fabricated into a fake "library" or a hard runtime error (see G31 fix)
3. Looks up each detected call in a source-to-Julia translation table
(`src/codegen/julia_gen.rs`) and emits a Julia module; unmapped calls get a
`# TODO` comment instead of a hard error
4. Generates benchmark *scaffolding* (`benchmark.jl`, `benchmark_runner.sh`,
`results.toml`) for the user to run and record their own timings —
julianiser does not measure or claim a speedup number itself

### HONESTY NOTE — Idris2 ABI / Zig FFI is scaffolding, not operational

`src/interface/abi/` (Idris2) and `src/interface/ffi/` (Zig) exist as source
files but are **not part of the working pipeline**:
- The Idris2 proofs are not built or checked by `cargo build`/`cargo test`.
- The Zig FFI's exported functions are TODO stubs: `julianiser_parse_python`/
`julianiser_parse_r` are no-ops that only record the language tag,
`julianiser_codegen` returns a fixed placeholder module string, and
`julianiser_benchmark` hardcodes `speedup = 1.0`.
- The Rust CLI never dlopens or links against the Zig library. Codegen is
100% Rust string generation in `src/codegen/`.
- Do NOT describe this layer as "the operational pipeline" in any doc. It is
tracked for future wiring. If you touch it, keep this note in sync.

### Key Translation Patterns (via src/codegen/julia_gen.rs lookup tables)

- `pandas.DataFrame` -> `DataFrames.jl`
- `numpy` arrays -> native Julia arrays (with broadcasting)
- `scipy` -> Julia stdlib / Optim.jl
- R `data.frame` -> `DataFrames.jl`
- 0-based indexing -> 1-based indexing (formally verified remapping)

### Architecture

- **Rust CLI** (`src/main.rs`) — orchestration
- **Idris2 ABI** (`src/interface/abi/`) — SourceLanguage, DataFrameOperation, ArrayPattern, JuliaType, EquivalenceWitness
- **Zig FFI** (`src/interface/ffi/`) — C-ABI bridge with Python/R parse + Julia codegen
- **Codegen** (`src/codegen/`) — Julia code generation from typed IR
- **Rust CLI** (`src/main.rs`) — orchestration; the actual entry point for
`init`/`validate`/`generate`/`build`/`run`
- **Codegen** (`src/codegen/`) — the operational translation pipeline:
parser.rs (detection) + julia_gen.rs (translation) + benchmark.rs (scaffold
generation)
- **Idris2 ABI** (`src/interface/abi/`) — SCAFFOLDING: SourceLanguage,
DataFrameOperation, ArrayPattern, JuliaType, EquivalenceWitness declared but
not wired into the build
- **Zig FFI** (`src/interface/ffi/`) — SCAFFOLDING: TODO-stubbed C-ABI bridge,
never called by the Rust CLI

### Allowed Languages

Expand All @@ -121,6 +151,7 @@ TypeScript, Node.js, npm, Python (as implementation language — Python is a *ta
7. **Container runtime** - Podman, never Docker. Files are `Containerfile`, never `Dockerfile`
8. **Container orchestration** - `selur-compose`, never `docker-compose`
9. **Julia is the target, not the implementation** - Julianiser is written in Rust/Idris2/Zig; it *generates* Julia
10. **No overclaiming the FFI seam** - Never describe the Idris2-ABI/Zig-FFI layer as operational or wired; it is scaffolding until the Rust CLI actually links/dlopens it and a benchmark shows a real, measured speedup (see HONESTY NOTE above)

## REPOSITORY STRUCTURE

Expand Down
156 changes: 106 additions & 50 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,49 +6,80 @@ SPDX-FileCopyrightText: 2025-2026 Jonathan D.A. Jewell <j.d.a.jewell@open.ac.uk>
# What Is This?

Julianiser analyses existing Python and R data science code, identifies
performance-critical array and dataframe operations, and generates
equivalent Julia code with full type annotations — producing **drop-in
replacement modules** that deliver **10-100x speedups** through Julia’s
LLVM JIT compilation.

Scientists keep their Python notebooks. Julia runs underneath.

# How It Works

Describe your data pipeline in a `julianiser.toml` manifest. Julianiser:

1. **Parses** your Python functions (via AST) or R scripts (via R
parser)

2. **Identifies** array operations, dataframe transformations, and
numeric hot paths

3. **Generates** Julia equivalents with proper type annotations and
broadcasting

4. **Creates** a Zig FFI bridge for calling Julia from your existing
code (zero-copy where possible)

5. **Benchmarks** the original vs. generated code to verify speedup

6. **Falls back** gracefully — if Julia is not installed, your Python/R
code still runs
common library calls (pandas/numpy/scipy/dplyr/ggplot2 patterns), and
generates a corresponding Julia module using known source-to-Julia
function mappings. The goal is drop-in replacement modules that let you
run the equivalent logic under Julia's LLVM JIT compilation.

**Current status (honest, as of this writing):** the working, tested
pipeline is the **Rust codegen** in `src/codegen/` (line-based
Python/R parsing + template-based Julia generation), invoked by the
Rust CLI (`src/main.rs`). It does not (yet) measure or claim any actual
speedup — the generated `benchmarks/results.toml` ships with
`speedup = 0.0 # TODO: fill in`, to be measured by the user after
running the generated Julia code against the original. Any "10-100x"
figures elsewhere in this repository's docs describe the *general,
well-documented characteristics of Julia's JIT* on this class of
workload, not a number julianiser itself has measured.

The **Idris2-ABI (`src/interface/abi/`) and Zig-FFI (`src/interface/ffi/`)**
layer described in earlier drafts of this README is **scaffolding**:
the Idris2 proofs and the Zig C-ABI bridge exist as source files, but
the Zig implementation is TODO-stubbed (parsing is a no-op, codegen
returns a hardcoded placeholder module, and the benchmark hook returns
a hardcoded `speedup = 1.0`), and the Rust CLI never links, dlopens, or
otherwise calls into it. It is tracked for future wiring, not part of
the operational translation path today.

Scientists keep their Python notebooks. Julia runs underneath, once
generated.

# How It Works (today)

Describe your data pipeline in a `julianiser.toml` manifest. The Rust
CLI, via `src/codegen/`:

1. **Parses** your Python or R source with a line-based scanner
(`src/codegen/parser.rs`) that detects import statements and
library-qualified calls

2. **Looks up** each detected call in a table of known source-language
→ Julia translations (`src/codegen/julia_gen.rs`)

3. **Generates** a Julia module with `using` statements and translated
calls (or a `# TODO` comment where no mapping exists)

4. **Generates** benchmark scaffolding (`benchmarks/benchmark.jl`,
`benchmark_runner.sh`, `results.toml`) so *you* can measure and
record the speedup after running both versions — julianiser does
not measure it for you yet

5. **Falls back** gracefully — if Julia is not installed, your Python/R
code still runs; the generated Julia is optional, additive output

The Idris2-ABI / Zig-FFI seam described below is planned future
infrastructure for a compiled, zero-copy bridge; it is not required for
(and is not currently used by) the codegen path above.

# Key Value

- **No Julia knowledge required** — Julianiser handles the translation
for the library calls it recognises

- **10-100x speedups** on numeric, array, and dataframe code without
manual rewrites
- **Julia's JIT is well-documented to deliver large speedups** on
numeric, array, and dataframe code — julianiser generates the scripts
to measure that for *your* workload; it does not yet ship a measured
number of its own

- **Gradual adoption** — julianise one function at a time, benchmark
each

- **Drop-in modules** — generated Julia code exposes the same API as
your Python/R functions
- **Drop-in modules** — generated Julia code exposes a `run_pipeline()`
entry point mirroring the detected calls in your Python/R script

- **Formally verified bridges** — Idris2 ABI proofs guarantee interface
correctness
- **Idris2 ABI + Zig FFI scaffolding exists** (`src/interface/`) for a
future formally-verified, compiled bridge — not yet wired into the
translation path (see Architecture, below)

# Supported Patterns

Expand Down Expand Up @@ -90,29 +121,50 @@ language.
# Architecture

Follows the hyperpolymath -iser pattern (same as
[Chapeliser](https://github.com/hyperpolymath/chapeliser)):
[Chapeliser](https://github.com/hyperpolymath/chapeliser)) in file
layout, but — to be explicit about what is operational today versus
what is scaffolding for later:

**Operational (this is what `julianiser generate` actually runs):**

- **Manifest** (`julianiser.toml`) — describe WHAT you want julianised

- **Source Parser** — Python AST analysis / R parser for identifying
translatable patterns
- **Source Parser** (`src/codegen/parser.rs`) — line-based Python/R
scanner that detects import statements and known library-qualified
calls

- **Julia Codegen** (`src/codegen/julia_gen.rs`) — looks up each
detected call in a source-to-Julia translation table and emits a
Julia module; unmapped calls become a `# TODO` comment, not a runtime
error

- **Idris2 ABI** (`src/interface/abi/`) — formal proofs of equivalence
between source and generated code
- **Benchmark scaffold generator** (`src/codegen/benchmark.rs`) —
generates `benchmark.jl` / `benchmark_runner.sh` / `results.toml` so
the user can run and record original-vs-Julia timings themselves

- **Julia Codegen** (`src/codegen/`) — generates type-annotated Julia
with proper broadcasting
- **Rust CLI** (`src/main.rs`) — orchestrates manifest validation,
parsing, and generation (`init`, `validate`, `generate`, `build`,
`run`)

- **Zig FFI** (`src/interface/ffi/`) — C-ABI bridge between Julia
runtime and calling code
**Scaffolding — present in the tree, not yet wired into the CLI:**

- **Benchmark Harness** — compares original Python/R vs. generated Julia
performance
- **Idris2 ABI** (`src/interface/abi/`) — formal type/layout
declarations intended to eventually prove equivalence between source
and generated code. Exists as `.idr` source; not built or checked by
`cargo build`/`cargo test`.

- **Rust CLI** (`src/main.rs`) — orchestrates parsing, validation,
generation, building, and benchmarking
- **Zig FFI** (`src/interface/ffi/`) — a C-ABI bridge skeleton. The
exported functions are TODO-stubbed (parsing is a no-op that only
records the source language; codegen returns a fixed placeholder
module string; the benchmark hook returns a hardcoded
`speedup = 1.0`). The Rust CLI never calls into this library — no
`dlopen`, no linking. It is tracked for future wiring, not part of
today's translation pipeline.

User writes zero Julia code. Julianiser generates everything.
Today, the user writes zero Julia code and the Rust codegen generates
the translated module; the Idris2/Zig seam is where a future,
formally-verified, compiled bridge is planned to replace the current
pure-codegen approach.

Part of the [-iser family](https://github.com/hyperpolymath/iseriser) of
acceleration frameworks.
Expand Down Expand Up @@ -150,9 +202,13 @@ julianiser run --benchmark

# Status

**Codebase in progress.** Architecture defined, CLI scaffolded, codegen
pending. Phase 0 (scaffold) complete. Phase 1 (Python AST parser + Julia
codegen) underway.
**Codebase in progress.** The Rust CLI and codegen pipeline
(`src/codegen/`) are implemented and covered by an automated test suite
(`cargo test`). The Idris2-ABI / Zig-FFI layer (`src/interface/`) is
scaffolding — present in the tree but not yet built, checked, or called
by the Rust CLI. No speedup number has been measured by julianiser
itself; the generated benchmark scripts exist for the user to measure
their own workload.

# License

Expand Down
62 changes: 41 additions & 21 deletions TOPOLOGY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,20 @@

## Purpose

Julianiser analyses Python and R data science code, identifies performance-critical
array/dataframe operations, and generates equivalent Julia modules that deliver
10-100x speedups via LLVM JIT compilation.
Julianiser analyses Python and R data science code, identifies known
pandas/numpy/scipy/dplyr/ggplot2-style calls, and generates equivalent Julia
modules via the Rust codegen pipeline (`src/codegen/`). Julia's LLVM JIT is
well-documented to deliver large speedups on this class of workload; julianiser
generates benchmark scaffolding so users can measure that for their own
pipeline, but does not itself measure or claim a speedup number today.

**Honesty note (see README.md for full detail):** the Idris2 ABI
(`src/interface/abi/`) and Zig FFI (`src/interface/ffi/`) below are
scaffolding — source files exist, but the Zig implementation is
TODO-stubbed and the Rust CLI never links or calls into it. The
operational translation pipeline today is the Rust codegen
(`src/codegen/parser.rs` + `src/codegen/julia_gen.rs`), invoked directly
by `src/main.rs`.

## Module Map

Expand All @@ -16,17 +27,17 @@ julianiser/
│ ├── lib.rs # Library API
│ ├── manifest/mod.rs # julianiser.toml parser and validator
│ ├── codegen/mod.rs # Julia code generation engine
│ ├── abi/mod.rs # Rust-side ABI types mirroring Idris2 proofs
│ └── interface/ # Verified Interface Seams
│ ├── abi/ # Idris2 ABI (The Spec)
│ ├── abi/mod.rs # Rust-side domain types (not linked to Idris2 below)
│ └── interface/ # Interface Seam SCAFFOLDING (planned, not wired — see Data Flow)
│ ├── abi/ # Idris2 ABI (The Spec) — source only, not built by cargo
│ │ ├── Types.idr # SourceLanguage, DataFrameOp, ArrayPattern, JuliaType, EquivalenceWitness
│ │ ├── Layout.idr # AST node memory layout with alignment proofs
│ │ └── Foreign.idr # FFI declarations for Python/R parsing and Julia codegen
│ ├── ffi/ # Zig FFI (The Bridge)
│ ├── ffi/ # Zig FFI (The Bridge) — TODO-stubbed, never called by the Rust CLI
│ │ ├── build.zig # Zig build config (shared + static lib)
│ │ ├── src/main.zig # FFI implementation (parse, translate, codegen)
│ │ └── test/ # Integration tests verifying ABI compliance
│ └── generated/abi/ # Auto-generated C headers from Idris2
│ │ ├── src/main.zig # FFI stub (parse/codegen/benchmark are placeholders)
│ │ └── test/ # Zig-side unit tests of the stub behaviour
│ └── generated/abi/ # Auto-generated C headers from Idris2 (placeholder dir)
├── verification/ # Proofs, benchmarks, fuzzing, safety cases
│ ├── benchmarks/ # Python/R vs Julia performance comparisons
│ ├── proofs/ # Formal verification artifacts
Expand All @@ -47,27 +58,36 @@ julianiser/
└── integrations/ # proven, verisimdb, vexometer, feedback-o-tron
```

## Data Flow
## Data Flow (operational today)

```
julianiser.toml ──→ Manifest Parser ──→ Source Parser (Python AST / R parser)
│
▼
Typed IR (operations, types, data flow)
julianiser.toml ──→ Manifest Parser ──→ Source Parser (line-based Python/R scan)
(src/manifest/) (src/codegen/parser.rs)
│
▼
Idris2 ABI (equivalence proofs)
Detected library calls
(import aliases resolved;
local-variable calls skipped)
│
▼
Julia Codegen (type-annotated, broadcasting)
Julia Codegen (template lookup per call)
(src/codegen/julia_gen.rs)
│
▼
Zig FFI Bridge (C-ABI interop)
│
▼
Benchmark Harness (original vs generated)
Benchmark scaffold generator
(src/codegen/benchmark.rs — emits scripts
for the user to run and record timings)
```

## Data Flow (planned, not yet wired)

The Idris2 ABI (`src/interface/abi/`) and Zig FFI (`src/interface/ffi/`)
are intended to eventually sit between source parsing and Julia codegen
as a formally-verified, compiled bridge. Today they are source-only
scaffolding: the Idris2 proofs are not built or checked by `cargo
build`/`cargo test`, and the Zig implementation's exported functions
are TODO stubs that the Rust CLI never calls.

## Key Dependencies

| Dependency | Purpose |
Expand Down
Loading
Loading