An agentic data integration and lineage engine for auditable, explainable, and compliant ETL workflows.
DataWeave redefines how organizations merge heterogeneous datasets by combining LLM reasoning, human-in-the-loop validation, and enterprise-grade auditability.
Built during the EY Data Integration Challenge @ Hack The Valley X, the system demonstrates what next-generation ETL should look like — transparent, traceable, and intelligent.
DataWeave allows users to:
- Upload and analyze heterogeneous financial datasets
- Automatically align relational databases across sources using Gemini-assisted metadata reasoning
- Visually map and merge records with confidence scoring, human review, and full lineage tracking
- Export clean, API-ready datasets and audit reports
The platform consists of three core modules:
-
Upload Page — Upload and validate source datasets
-
Schema Visualizer — Inspect structure, compare schemas, and correct mismatched relationships
-
Mapping Page — Approve or edit AI-suggested mappings and export clean merges
“Can we prove where every merged value came from?”
Every change made to the data in DataWeave is fully traceable:
- Shows source dataset, applied transformation, confidence score, and final output field
“The AI suggests, the analyst approves.”
- Analysts can check and edit AI-generated mappings
- Every action updates the mapping JSON and informs future model suggestions
- All approvals and overrides are logged.
“Clean the data without letting sensitive data leak.”
- Automatically standardizes formats using regex + LLM classification (emails, SSNs, account numbers)
- Uses secure algorithms to clean data without sending important financial data to external LLMs
“Explain why you matched these fields.”
- Uses Gemini to generate rationales for every mapping:
“These columns share similar naming, type, and refer to the same concept (Customer ID).”
- Displays confidence scores and plain-English reasoning alongside each mapping
“What changed after merging?”
Automatically generates an detailed report after every merge, including:
- Number of rows merged
- % matched / unmatched
- Confidence distributions
- Validation summaries
- Frontend: React, TypeScript, Next, MUI
- Backend: FastAPI, Python, Supabase, PostgreSQL
- AI & Data Processing: Gemini API, Pandas
Clone the repo
- git clone https://github.com/yourusername/dataweave.git
- cd dataweave
Backend setup
- cd server
- pip install -r requirements.txt
- python app.py
Frontend setup
- cd client
- npm install
- npm run dev