Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
186 changes: 186 additions & 0 deletions README.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,186 @@
// SPDX-License-Identifier: CC-BY-SA-4.0
// SPDX-FileCopyrightText: 2025-2026 Jonathan D.A. Jewell <j.d.a.jewell@open.ac.uk>
= julianiser
:toc: macro
:toclevels: 2

Auto-generate Julia modules from existing Python/R data-science code.

toc::[]

== What Is This?

Julianiser analyses existing Python and R data science code, identifies common
library calls (pandas/numpy/scipy/dplyr/ggplot2 patterns), and generates a
corresponding Julia module using known source-to-Julia function mappings. The
goal is drop-in replacement modules that let you run the equivalent logic under
Julia's LLVM JIT compilation.

*Current status (honest, as of this writing):* the working, tested pipeline is
the *Rust codegen* in `src/codegen/` (line-based Python/R parsing +
template-based Julia generation), invoked by the Rust CLI (`src/main.rs`). It
does not (yet) measure or claim any actual speedup — the generated
`benchmarks/results.toml` ships with `speedup = 0.0 # TODO: fill in`, to be
measured by the user after running the generated Julia code against the
original. Any "10-100x" figures elsewhere in this repository's docs describe the
_general, well-documented characteristics of Julia's JIT_ on this class of
workload, not a number julianiser itself has measured.

The *Idris2-ABI (`src/interface/abi/`) and Zig-FFI (`src/interface/ffi/`)* layer
described in earlier drafts of this README is *scaffolding*: the Idris2 proofs
and the Zig C-ABI bridge exist as source files, but the Zig implementation is
TODO-stubbed (parsing is a no-op, codegen returns a hardcoded placeholder
module, and the benchmark hook returns a hardcoded `speedup = 1.0`), and the
Rust CLI never links, dlopens, or otherwise calls into it. It is tracked for
future wiring, not part of the operational translation path today.

Scientists keep their Python notebooks. Julia runs underneath, once generated.

== How It Works (today)

Describe your data pipeline in a `julianiser.toml` manifest. The Rust CLI, via
`src/codegen/`:

. *Parses* your Python or R source with a line-based scanner
(`src/codegen/parser.rs`) that detects import statements and library-qualified
calls.
. *Looks up* each detected call in a table of known source-language → Julia
translations (`src/codegen/julia_gen.rs`).
. *Generates* a Julia module with `using` statements and translated calls (or a
`# TODO` comment where no mapping exists).
. *Generates* benchmark scaffolding (`benchmarks/benchmark.jl`,
`benchmark_runner.sh`, `results.toml`) so _you_ can measure and record the
speedup after running both versions — julianiser does not measure it for you
yet.
. *Falls back* gracefully — if Julia is not installed, your Python/R code still
runs; the generated Julia is optional, additive output.

The Idris2-ABI / Zig-FFI seam described below is planned future infrastructure
for a compiled, zero-copy bridge; it is not required for (and is not currently
used by) the codegen path above.

== Key Value

* *No Julia knowledge required* — Julianiser handles the translation for the
library calls it recognises.
* *Julia's JIT is well-documented to deliver large speedups* on numeric, array,
and dataframe code — julianiser generates the scripts to measure that for
_your_ workload; it does not yet ship a measured number of its own.
* *Gradual adoption* — julianise one function at a time, benchmark each.
* *Drop-in modules* — generated Julia code exposes a `run_pipeline()` entry
point mirroring the detected calls in your Python/R script.
* *Idris2 ABI + Zig FFI scaffolding exists* (`src/interface/`) for a future
formally-verified, compiled bridge — not yet wired into the translation path
(see Architecture, below).

== Supported Patterns

Julianiser recognises and translates these common data science patterns:

[cols="1,1,2",options="header"]
|===
| Python | Julia Equivalent | Notes

| `pandas.DataFrame` | `DataFrames.jl` | Column operations, groupby, joins
| `numpy` arrays | Native Julia arrays | Broadcasting, slicing, linear algebra
| `scipy.optimize` | `Optim.jl` / Julia stdlib | Minimisation, root-finding, curve fitting
| `matplotlib` / `seaborn` | `Plots.jl` / `Makie.jl` | Static and interactive plotting
| `scikit-learn` pipelines | `MLJ.jl` | ⚠ *Planned* — sklearn imports are _detected_ (the `sk` alias is recognised in the parser), but no sklearn→MLJ function conversions are mapped yet in the codegen
| R `data.frame` / `tibble` | `DataFrames.jl` | dplyr-style verbs mapped to Julia
| R `apply` / `sapply` / `lapply` | Julia broadcasting / `map` | Vectorised equivalents
|===

== Why Julia?

Julia achieves C-like performance through:

* *LLVM JIT compilation* — code is compiled to native machine code on first call.
* *Multiple dispatch* — the type system enables aggressive specialisation.
* *Type inference* — the compiler infers concrete types for fast code paths.
* *Broadcasting* — element-wise operations fuse into single loops (no temporary
arrays).
* *In-place operations* — `mul!`, `ldiv!`, and friends avoid allocation.

The "two-language problem" (prototype in Python, rewrite in C) disappears. Julia
is both the prototyping language and the production language.

== Architecture

Follows the hyperpolymath -iser pattern (same as
https://github.com/hyperpolymath/chapeliser[Chapeliser]) in file layout, but —
to be explicit about what is operational today versus what is scaffolding for
later:

*Operational (this is what `julianiser generate` actually runs):*

* *Manifest* (`julianiser.toml`) — describe WHAT you want julianised.
* *Source Parser* (`src/codegen/parser.rs`) — line-based Python/R scanner that
detects import statements and known library-qualified calls.
* *Julia Codegen* (`src/codegen/julia_gen.rs`) — looks up each detected call in
a source-to-Julia translation table and emits a Julia module; unmapped calls
become a `# TODO` comment, not a runtime error.
* *Benchmark scaffold generator* (`src/codegen/benchmark.rs`) — generates
`benchmark.jl` / `benchmark_runner.sh` / `results.toml` so the user can run
and record original-vs-Julia timings themselves.
* *Rust CLI* (`src/main.rs`) — orchestrates manifest validation, parsing, and
generation (`init`, `validate`, `generate`, `build`, `run`).

*Scaffolding — present in the tree, not yet wired into the CLI:*

* *Idris2 ABI* (`src/interface/abi/`) — formal type/layout declarations intended
to eventually prove equivalence between source and generated code. Exists as
`.idr` source; not built or checked by `cargo build`/`cargo test`.
* *Zig FFI* (`src/interface/ffi/`) — a C-ABI bridge skeleton. The exported
functions are TODO-stubbed (parsing is a no-op that only records the source
language; codegen returns a fixed placeholder module string; the benchmark
hook returns a hardcoded `speedup = 1.0`). The Rust CLI never calls into this
library — no `dlopen`, no linking. It is tracked for future wiring, not part
of today's translation pipeline.

Today, the user writes zero Julia code and the Rust codegen generates the
translated module; the Idris2/Zig seam is where a future, formally-verified,
compiled bridge is planned to replace the current pure-codegen approach.

Part of the https://github.com/hyperpolymath/iseriser[-iser family] of
acceleration frameworks.

== Use Cases

* *Data science acceleration* — speed up pandas/numpy pipelines without rewriting.
* *Batch processing* — convert overnight Python ETL jobs to minutes with Julia.
* *Scientific computing migration* — gradually move research code from Python/R
to Julia.
* *HPC preparation* — Julia code can target GPU (CUDA.jl) and distributed
(Distributed.jl) backends.

== Quick Start

[source,bash]
----
# Install julianiser
cargo install julianiser

# Initialise a manifest in your project
julianiser init

# Edit julianiser.toml to point at your Python/R code
# Then generate Julia replacements
julianiser generate

# Benchmark original vs. generated
julianiser run --benchmark
----

== Status

*Codebase in progress.* The Rust CLI and codegen pipeline (`src/codegen/`) are
implemented and covered by an automated test suite (`cargo test`). The
Idris2-ABI / Zig-FFI layer (`src/interface/`) is scaffolding — present in the
tree but not yet built, checked, or called by the Rust CLI. No speedup number
has been measured by julianiser itself; the generated benchmark scripts exist
for the user to measure their own workload.

== License

Code: `MPL-2.0`. Documentation (including this README): `CC-BY-SA-4.0`. Full
texts are in `LICENSES/`.
Loading
Loading