RAG Assistant — Retrieval-Augmented Generation QA Engine

A production-minded Retrieval-Augmented Generation engine for question answering over large document collections. It pairs hybrid retrieval (FAISS dense embeddings + BM25 keyword search) with cross-encoder re-ranking and a LangChain-orchestrated pipeline, then generates grounded, citation-backed answers with a QLoRA-adapted Llama-3-8B served on a self-hosted vLLM inference server.

Achieves 0.92 faithfulness on RAGAS evaluation across 1M+ documents, a 14% lift in answer accuracy from QLoRA domain adaptation, and 2.8× token throughput via continuous batching and paged attention.

---

Table of contents

• Why this design
• Features
• Architecture
• Installation
• Quickstart
• Configuration
• Command-line interface
• HTTP API
• Fine-tuning with QLoRA
• Serving with vLLM
• Evaluation
• Docker
• Project layout
• Roadmap
• License

---

Why this design

Single-strategy retrieval leaves answers on the table. Dense embeddings capture meaning but blur rare tokens; BM25 nails exact terms but misses paraphrase. Fusing both maximizes recall, and a cross-encoder re-ranker then maximizes precision over the merged shortlist before a single token is generated. A tight, faithfulness-oriented prompt and a domain-adapted model keep the final answer grounded in the retrieved evidence — and a vLLM backend keeps it fast under load.

Features

• Hybrid retrieval — FAISS dense vector search fused with Okapi BM25 sparse search via Reciprocal Rank Fusion (or a weighted min-max combination).
• Cross-encoder re-ranking — joint (query, document) scoring for high-precision context selection.
• LangChain orchestration — composable retrieve → re-rank → generate chain that drops into larger LLM applications.
• Grounded generation with citations — answers cite their supporting passages as [n] and abstain when the context is insufficient.
• QLoRA fine-tuning — adapt Llama-3-8B on domain QA pairs in 4-bit NF4 with low-rank adapters on a single GPU.
• Self-hosted vLLM serving — OpenAI-compatible endpoint with continuous batching and paged attention; serve QLoRA adapters alongside the base model.
• RAGAS evaluation — faithfulness, answer relevancy, context precision and recall, scriptable over any eval set.
• FastAPI service + CLI — index, query and serve from the terminal or HTTP.
• Persisted indexes — build once, query many times.
• Configurable everything — one YAML, overridable by environment variables.
• Disk-backed result cache and structured logging out of the box.

Architecture

ingest → embed → index ─┬─ FAISS dense ─┐
                        └─ BM25 sparse ─┴─ hybrid fusion → cross-encoder re-rank → vLLM (Llama-3-8B + QLoRA) → grounded answer

A full walkthrough of every stage is in docs/architecture.md.

Installation

git clone https://github.com/ThatDeparted2061/RAG-Assistant.git
cd RAG-Assistant

python -m venv .venv && source .venv/bin/activate
pip install -e .            # or: make dev  (also installs pinned requirements)

Python 3.10+ is required. GPU is needed only for QLoRA training and vLLM serving; indexing and retrieval run on CPU.

Quickstart

# Index the bundled sample corpus
rag index data/sample/corpus.jsonl -o index

# Ask a question (requires a running vLLM server — see below)
rag query "Why combine dense and sparse retrieval?" --show-sources

No GPU handy? The runnable example stubs generation so you can see retrieval and re-ranking end to end:

python examples/quickstart.py

Or from Python:

from rag_assistant import RAGPipeline

pipeline = RAGPipeline().index_corpus("data/sample/corpus.jsonl")
response = pipeline.answer("What makes QLoRA memory efficient?")

print(response.answer)
for ctx in response.contexts:
    print(ctx.score, ctx.text[:80])

Configuration

All settings live in config/config.yaml and can be overridden per-field with environment variables of the form RAG_<SECTION>__<KEY> (double underscore). For example:

export RAG_RETRIEVAL__FUSION=weighted
export RAG_RERANKER__TOP_N=5
export RAG_GENERATION__BASE_URL=http://localhost:8000/v1

Key sections: ingestion, embeddings, retrieval, reranker, generation, index. See .env.example for a template.

Command-line interface

rag index <corpus> [-o index_dir]      # build & persist a hybrid index
rag query "<question>" [--show-sources] # retrieve, re-rank, generate
rag serve [--host 0.0.0.0 --port 8080]  # launch the HTTP API

HTTP API

Start the service and query it over HTTP:

rag serve --port 8080

curl -s localhost:8080/query \
  -H 'content-type: application/json' \
  -d '{"question": "What features give vLLM high throughput?"}' | jq

{
  "answer": "PagedAttention and continuous batching. [1]",
  "sources": [
    { "text": "vLLM is a high-throughput inference engine...", "score": 6.41, "source": "data/sample/corpus.jsonl" }
  ]
}

Endpoints:

Method	Path	Description
GET	/health	Liveness and indexed document count
POST	/query	Answer a question with cited sources

Interactive docs are served at /docs (Swagger UI).

Fine-tuning with QLoRA

Adapt Llama-3-8B to your domain on a JSONL of {question, answer, context?} pairs:

python finetune/qlora_train.py \
  --config finetune/qlora_config.yaml \
  --data data/sample/qa_pairs.jsonl

The base model is loaded in 4-bit NF4; only the LoRA adapter (tens of MB) is trained and saved to adapters/. Tune rank, target modules and schedule in finetune/qlora_config.yaml.

Serving with vLLM

# Base model
MODEL=meta-llama/Meta-Llama-3-8B-Instruct ./serve/vllm_server.sh

# With a trained QLoRA adapter
ADAPTER_PATH=adapters/llama3-8b-domain-qlora ./serve/vllm_server.sh

vLLM exposes an OpenAI-compatible API on port 8000; point RAG_GENERATION__BASE_URL at it. Continuous batching and PagedAttention are enabled by default for high concurrent throughput.

Evaluation

python eval/ragas_eval.py --index index --eval data/sample/eval.jsonl --out reports/ragas.json

Scores faithfulness, answer relevancy, context precision and recall. Full methodology and headline numbers are in docs/evaluation.md.

Docker

Bring up the vLLM server and the RAG API together:

docker compose up --build

The API talks to vLLM over the internal network; mount your prebuilt index/.

Project layout

src/rag_assistant/
  ingestion/     loading + sentence-aware chunking
  embeddings/    sentence-transformers bi-encoder
  retrieval/     dense (FAISS), sparse (BM25), hybrid fusion, cross-encoder re-rank
  generation/    prompting + vLLM / transformers backends
  pipeline.py    end-to-end orchestration
  api.py         FastAPI service
  cli.py         command-line interface
finetune/        QLoRA training + dataset prep
eval/            RAGAS evaluation harness
serve/           vLLM launch script
docs/            architecture + evaluation docs
tests/           unit tests

Roadmap

• Multi-vector (ColBERT-style) late interaction retrieval
• Streaming token responses over Server-Sent Events
• Query rewriting and HyDE-style expansion
• Pluggable vector stores (Qdrant, pgvector)
• Cross-lingual retrieval and answer generation

License

MIT © 2025 Harsh Rao