From 3ddd8d442c446d7f56e30700c456d46fdd5d2f11 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 1 Jul 2026 13:32:10 +0000 Subject: [PATCH] =?UTF-8?q?docs:=20convert=20README.md=20=E2=86=92=20READM?= =?UTF-8?q?E.adoc=20(estate=20.adoc-default=20policy)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per the estate docs standard, README is .adoc by default; only boj-server and the profile repo (hyperpolymath/hyperpolymath) are the .md exceptions. julianiser is neither, so its README.md was a prior-AI md-ification error. Converted to AsciiDoc and removed the .md (no duplicate formats). - Faithful conversion — every honesty disclaimer preserved verbatim in meaning (operational Rust codegen vs. Idris2/Zig scaffolding; no measured-speedup claim; the G38 sklearn "⚠ Planned" note). - Dropped the HTML-comment SPDX wrapper for a native AsciiDoc `//` header. - License section now states the project's actual licensing accurately — MPL-2.0 (code) + CC-BY-SA-4.0 (docs), matching LICENSES/ (which contains exactly those two texts). Not a relicensing; the previous section showed only the README's own CC-BY-SA-4.0 SPDX line. Render-validated with asciidoctor (--failure-level=WARN → 0 warnings). No V-lang token; README.adoc at repo root (not docs/), so the docs gate is unaffected. --- README.adoc | 186 +++++++++++++++++++++++++++++++++++++++++++++ README.md | 215 ---------------------------------------------------- 2 files changed, 186 insertions(+), 215 deletions(-) create mode 100644 README.adoc delete mode 100644 README.md diff --git a/README.adoc b/README.adoc new file mode 100644 index 0000000..2c1b3d0 --- /dev/null +++ b/README.adoc @@ -0,0 +1,186 @@ +// SPDX-License-Identifier: CC-BY-SA-4.0 +// SPDX-FileCopyrightText: 2025-2026 Jonathan D.A. Jewell += julianiser +:toc: macro +:toclevels: 2 + +Auto-generate Julia modules from existing Python/R data-science code. + +toc::[] + +== What Is This? + +Julianiser analyses existing Python and R data science code, identifies common +library calls (pandas/numpy/scipy/dplyr/ggplot2 patterns), and generates a +corresponding Julia module using known source-to-Julia function mappings. The +goal is drop-in replacement modules that let you run the equivalent logic under +Julia's LLVM JIT compilation. + +*Current status (honest, as of this writing):* the working, tested pipeline is +the *Rust codegen* in `src/codegen/` (line-based Python/R parsing + +template-based Julia generation), invoked by the Rust CLI (`src/main.rs`). It +does not (yet) measure or claim any actual speedup — the generated +`benchmarks/results.toml` ships with `speedup = 0.0 # TODO: fill in`, to be +measured by the user after running the generated Julia code against the +original. Any "10-100x" figures elsewhere in this repository's docs describe the +_general, well-documented characteristics of Julia's JIT_ on this class of +workload, not a number julianiser itself has measured. + +The *Idris2-ABI (`src/interface/abi/`) and Zig-FFI (`src/interface/ffi/`)* layer +described in earlier drafts of this README is *scaffolding*: the Idris2 proofs +and the Zig C-ABI bridge exist as source files, but the Zig implementation is +TODO-stubbed (parsing is a no-op, codegen returns a hardcoded placeholder +module, and the benchmark hook returns a hardcoded `speedup = 1.0`), and the +Rust CLI never links, dlopens, or otherwise calls into it. It is tracked for +future wiring, not part of the operational translation path today. + +Scientists keep their Python notebooks. Julia runs underneath, once generated. + +== How It Works (today) + +Describe your data pipeline in a `julianiser.toml` manifest. The Rust CLI, via +`src/codegen/`: + +. *Parses* your Python or R source with a line-based scanner + (`src/codegen/parser.rs`) that detects import statements and library-qualified + calls. +. *Looks up* each detected call in a table of known source-language → Julia + translations (`src/codegen/julia_gen.rs`). +. *Generates* a Julia module with `using` statements and translated calls (or a + `# TODO` comment where no mapping exists). +. *Generates* benchmark scaffolding (`benchmarks/benchmark.jl`, + `benchmark_runner.sh`, `results.toml`) so _you_ can measure and record the + speedup after running both versions — julianiser does not measure it for you + yet. +. *Falls back* gracefully — if Julia is not installed, your Python/R code still + runs; the generated Julia is optional, additive output. + +The Idris2-ABI / Zig-FFI seam described below is planned future infrastructure +for a compiled, zero-copy bridge; it is not required for (and is not currently +used by) the codegen path above. + +== Key Value + +* *No Julia knowledge required* — Julianiser handles the translation for the + library calls it recognises. +* *Julia's JIT is well-documented to deliver large speedups* on numeric, array, + and dataframe code — julianiser generates the scripts to measure that for + _your_ workload; it does not yet ship a measured number of its own. +* *Gradual adoption* — julianise one function at a time, benchmark each. +* *Drop-in modules* — generated Julia code exposes a `run_pipeline()` entry + point mirroring the detected calls in your Python/R script. +* *Idris2 ABI + Zig FFI scaffolding exists* (`src/interface/`) for a future + formally-verified, compiled bridge — not yet wired into the translation path + (see Architecture, below). + +== Supported Patterns + +Julianiser recognises and translates these common data science patterns: + +[cols="1,1,2",options="header"] +|=== +| Python | Julia Equivalent | Notes + +| `pandas.DataFrame` | `DataFrames.jl` | Column operations, groupby, joins +| `numpy` arrays | Native Julia arrays | Broadcasting, slicing, linear algebra +| `scipy.optimize` | `Optim.jl` / Julia stdlib | Minimisation, root-finding, curve fitting +| `matplotlib` / `seaborn` | `Plots.jl` / `Makie.jl` | Static and interactive plotting +| `scikit-learn` pipelines | `MLJ.jl` | ⚠ *Planned* — sklearn imports are _detected_ (the `sk` alias is recognised in the parser), but no sklearn→MLJ function conversions are mapped yet in the codegen +| R `data.frame` / `tibble` | `DataFrames.jl` | dplyr-style verbs mapped to Julia +| R `apply` / `sapply` / `lapply` | Julia broadcasting / `map` | Vectorised equivalents +|=== + +== Why Julia? + +Julia achieves C-like performance through: + +* *LLVM JIT compilation* — code is compiled to native machine code on first call. +* *Multiple dispatch* — the type system enables aggressive specialisation. +* *Type inference* — the compiler infers concrete types for fast code paths. +* *Broadcasting* — element-wise operations fuse into single loops (no temporary + arrays). +* *In-place operations* — `mul!`, `ldiv!`, and friends avoid allocation. + +The "two-language problem" (prototype in Python, rewrite in C) disappears. Julia +is both the prototyping language and the production language. + +== Architecture + +Follows the hyperpolymath -iser pattern (same as +https://github.com/hyperpolymath/chapeliser[Chapeliser]) in file layout, but — +to be explicit about what is operational today versus what is scaffolding for +later: + +*Operational (this is what `julianiser generate` actually runs):* + +* *Manifest* (`julianiser.toml`) — describe WHAT you want julianised. +* *Source Parser* (`src/codegen/parser.rs`) — line-based Python/R scanner that + detects import statements and known library-qualified calls. +* *Julia Codegen* (`src/codegen/julia_gen.rs`) — looks up each detected call in + a source-to-Julia translation table and emits a Julia module; unmapped calls + become a `# TODO` comment, not a runtime error. +* *Benchmark scaffold generator* (`src/codegen/benchmark.rs`) — generates + `benchmark.jl` / `benchmark_runner.sh` / `results.toml` so the user can run + and record original-vs-Julia timings themselves. +* *Rust CLI* (`src/main.rs`) — orchestrates manifest validation, parsing, and + generation (`init`, `validate`, `generate`, `build`, `run`). + +*Scaffolding — present in the tree, not yet wired into the CLI:* + +* *Idris2 ABI* (`src/interface/abi/`) — formal type/layout declarations intended + to eventually prove equivalence between source and generated code. Exists as + `.idr` source; not built or checked by `cargo build`/`cargo test`. +* *Zig FFI* (`src/interface/ffi/`) — a C-ABI bridge skeleton. The exported + functions are TODO-stubbed (parsing is a no-op that only records the source + language; codegen returns a fixed placeholder module string; the benchmark + hook returns a hardcoded `speedup = 1.0`). The Rust CLI never calls into this + library — no `dlopen`, no linking. It is tracked for future wiring, not part + of today's translation pipeline. + +Today, the user writes zero Julia code and the Rust codegen generates the +translated module; the Idris2/Zig seam is where a future, formally-verified, +compiled bridge is planned to replace the current pure-codegen approach. + +Part of the https://github.com/hyperpolymath/iseriser[-iser family] of +acceleration frameworks. + +== Use Cases + +* *Data science acceleration* — speed up pandas/numpy pipelines without rewriting. +* *Batch processing* — convert overnight Python ETL jobs to minutes with Julia. +* *Scientific computing migration* — gradually move research code from Python/R + to Julia. +* *HPC preparation* — Julia code can target GPU (CUDA.jl) and distributed + (Distributed.jl) backends. + +== Quick Start + +[source,bash] +---- +# Install julianiser +cargo install julianiser + +# Initialise a manifest in your project +julianiser init + +# Edit julianiser.toml to point at your Python/R code +# Then generate Julia replacements +julianiser generate + +# Benchmark original vs. generated +julianiser run --benchmark +---- + +== Status + +*Codebase in progress.* The Rust CLI and codegen pipeline (`src/codegen/`) are +implemented and covered by an automated test suite (`cargo test`). The +Idris2-ABI / Zig-FFI layer (`src/interface/`) is scaffolding — present in the +tree but not yet built, checked, or called by the Rust CLI. No speedup number +has been measured by julianiser itself; the generated benchmark scripts exist +for the user to measure their own workload. + +== License + +Code: `MPL-2.0`. Documentation (including this README): `CC-BY-SA-4.0`. Full +texts are in `LICENSES/`. diff --git a/README.md b/README.md deleted file mode 100644 index 80d3f15..0000000 --- a/README.md +++ /dev/null @@ -1,215 +0,0 @@ - - -# What Is This? - -Julianiser analyses existing Python and R data science code, identifies -common library calls (pandas/numpy/scipy/dplyr/ggplot2 patterns), and -generates a corresponding Julia module using known source-to-Julia -function mappings. The goal is drop-in replacement modules that let you -run the equivalent logic under Julia's LLVM JIT compilation. - -**Current status (honest, as of this writing):** the working, tested -pipeline is the **Rust codegen** in `src/codegen/` (line-based -Python/R parsing + template-based Julia generation), invoked by the -Rust CLI (`src/main.rs`). It does not (yet) measure or claim any actual -speedup — the generated `benchmarks/results.toml` ships with -`speedup = 0.0 # TODO: fill in`, to be measured by the user after -running the generated Julia code against the original. Any "10-100x" -figures elsewhere in this repository's docs describe the *general, -well-documented characteristics of Julia's JIT* on this class of -workload, not a number julianiser itself has measured. - -The **Idris2-ABI (`src/interface/abi/`) and Zig-FFI (`src/interface/ffi/`)** -layer described in earlier drafts of this README is **scaffolding**: -the Idris2 proofs and the Zig C-ABI bridge exist as source files, but -the Zig implementation is TODO-stubbed (parsing is a no-op, codegen -returns a hardcoded placeholder module, and the benchmark hook returns -a hardcoded `speedup = 1.0`), and the Rust CLI never links, dlopens, or -otherwise calls into it. It is tracked for future wiring, not part of -the operational translation path today. - -Scientists keep their Python notebooks. Julia runs underneath, once -generated. - -# How It Works (today) - -Describe your data pipeline in a `julianiser.toml` manifest. The Rust -CLI, via `src/codegen/`: - -1. **Parses** your Python or R source with a line-based scanner - (`src/codegen/parser.rs`) that detects import statements and - library-qualified calls - -2. **Looks up** each detected call in a table of known source-language - → Julia translations (`src/codegen/julia_gen.rs`) - -3. **Generates** a Julia module with `using` statements and translated - calls (or a `# TODO` comment where no mapping exists) - -4. **Generates** benchmark scaffolding (`benchmarks/benchmark.jl`, - `benchmark_runner.sh`, `results.toml`) so *you* can measure and - record the speedup after running both versions — julianiser does - not measure it for you yet - -5. **Falls back** gracefully — if Julia is not installed, your Python/R - code still runs; the generated Julia is optional, additive output - -The Idris2-ABI / Zig-FFI seam described below is planned future -infrastructure for a compiled, zero-copy bridge; it is not required for -(and is not currently used by) the codegen path above. - -# Key Value - -- **No Julia knowledge required** — Julianiser handles the translation - for the library calls it recognises - -- **Julia's JIT is well-documented to deliver large speedups** on - numeric, array, and dataframe code — julianiser generates the scripts - to measure that for *your* workload; it does not yet ship a measured - number of its own - -- **Gradual adoption** — julianise one function at a time, benchmark - each - -- **Drop-in modules** — generated Julia code exposes a `run_pipeline()` - entry point mirroring the detected calls in your Python/R script - -- **Idris2 ABI + Zig FFI scaffolding exists** (`src/interface/`) for a - future formally-verified, compiled bridge — not yet wired into the - translation path (see Architecture, below) - -# Supported Patterns - -Julianiser recognises and translates these common data science patterns: - -| Python | Julia Equivalent | Notes | -|----|----|----| -| `pandas.DataFrame` | `DataFrames.jl` | Column operations, groupby, joins | -| `numpy` arrays | Native Julia arrays | Broadcasting, slicing, linear algebra | -| `scipy.optimize` | `Optim.jl` / Julia stdlib | Minimisation, root-finding, curve fitting | -| `matplotlib` / `seaborn` | `Plots.jl` / `Makie.jl` | Static and interactive plotting | -| `scikit-learn` pipelines | `MLJ.jl` | ⚠ **Planned** — sklearn imports are *detected* (the `sk` alias is recognised in the parser), but no sklearn→MLJ function conversions are mapped yet in the codegen | -| R `data.frame` / `tibble` | `DataFrames.jl` | dplyr-style verbs mapped to Julia | -| R `apply` / `sapply` / `lapply` | Julia broadcasting / `map` | Vectorised equivalents | - -# Why Julia? - -Julia achieves C-like performance through: - -- **LLVM JIT compilation** — code is compiled to native machine code on - first call - -- **Multiple dispatch** — the type system enables aggressive - specialisation - -- **Type inference** — the compiler infers concrete types for fast code - paths - -- **Broadcasting** — element-wise operations fuse into single loops (no - temporary arrays) - -- **In-place operations** — `mul!`, `ldiv!`, and friends avoid - allocation - -The "two-language problem" (prototype in Python, rewrite in C) -disappears. Julia is both the prototyping language and the production -language. - -# Architecture - -Follows the hyperpolymath -iser pattern (same as -[Chapeliser](https://github.com/hyperpolymath/chapeliser)) in file -layout, but — to be explicit about what is operational today versus -what is scaffolding for later: - -**Operational (this is what `julianiser generate` actually runs):** - -- **Manifest** (`julianiser.toml`) — describe WHAT you want julianised - -- **Source Parser** (`src/codegen/parser.rs`) — line-based Python/R - scanner that detects import statements and known library-qualified - calls - -- **Julia Codegen** (`src/codegen/julia_gen.rs`) — looks up each - detected call in a source-to-Julia translation table and emits a - Julia module; unmapped calls become a `# TODO` comment, not a runtime - error - -- **Benchmark scaffold generator** (`src/codegen/benchmark.rs`) — - generates `benchmark.jl` / `benchmark_runner.sh` / `results.toml` so - the user can run and record original-vs-Julia timings themselves - -- **Rust CLI** (`src/main.rs`) — orchestrates manifest validation, - parsing, and generation (`init`, `validate`, `generate`, `build`, - `run`) - -**Scaffolding — present in the tree, not yet wired into the CLI:** - -- **Idris2 ABI** (`src/interface/abi/`) — formal type/layout - declarations intended to eventually prove equivalence between source - and generated code. Exists as `.idr` source; not built or checked by - `cargo build`/`cargo test`. - -- **Zig FFI** (`src/interface/ffi/`) — a C-ABI bridge skeleton. The - exported functions are TODO-stubbed (parsing is a no-op that only - records the source language; codegen returns a fixed placeholder - module string; the benchmark hook returns a hardcoded - `speedup = 1.0`). The Rust CLI never calls into this library — no - `dlopen`, no linking. It is tracked for future wiring, not part of - today's translation pipeline. - -Today, the user writes zero Julia code and the Rust codegen generates -the translated module; the Idris2/Zig seam is where a future, -formally-verified, compiled bridge is planned to replace the current -pure-codegen approach. - -Part of the [-iser family](https://github.com/hyperpolymath/iseriser) of -acceleration frameworks. - -# Use Cases - -- **Data science acceleration** — speed up pandas/numpy pipelines - without rewriting - -- **Batch processing** — convert overnight Python ETL jobs to minutes - with Julia - -- **Scientific computing migration** — gradually move research code from - Python/R to Julia - -- **HPC preparation** — Julia code can target GPU (CUDA.jl) and - distributed (Distributed.jl) backends - -# Quick Start - -```bash -# Install julianiser -cargo install julianiser - -# Initialise a manifest in your project -julianiser init - -# Edit julianiser.toml to point at your Python/R code -# Then generate Julia replacements -julianiser generate - -# Benchmark original vs. generated -julianiser run --benchmark -``` - -# Status - -**Codebase in progress.** The Rust CLI and codegen pipeline -(`src/codegen/`) are implemented and covered by an automated test suite -(`cargo test`). The Idris2-ABI / Zig-FFI layer (`src/interface/`) is -scaffolding — present in the tree but not yet built, checked, or called -by the Rust CLI. No speedup number has been measured by julianiser -itself; the generated benchmark scripts exist for the user to measure -their own workload. - -# License - -SPDX-License-Identifier: CC-BY-SA-4.0