⚠️ Note: The notebook file is too large to preview directly on GitHub. Please use the buttons below to open it.
A multi-stage pipeline that generates, scores, votes and fuses image captions into one refined, grounded description
M.Sc. Data Science — Aligarh Muslim University, Aligarh · 2025–26
Standard image captioning models produce a single output with no quality check — often generic, incomplete, or hallucinated. This project proposes an Image Caption Refinement System that treats captioning as a multi-stage quality improvement process:
- Generates 5 diverse candidate captions from an input image
- Evaluates each candidate through 3 independent scoring signals
- Selects the top-2 by majority voting consensus
- Fuses them into one refined, comprehensive caption using a language model
The system was built and evaluated on the Flickr8k dataset (8,091 images · 40,455 captions) and deployed as a live web application on HuggingFace Spaces.
Upload any image and get a refined, detailed caption.
👉 Try the Live App on HuggingFace Spaces
Flickr8k provides 8,092 real-world images each paired with 5 independently written human captions — this multi-reference structure is the key reason. Having 5 references per image allowed the scoring signals (BLIP ITM, Jina, Cosine) to evaluate caption quality against a diverse human benchmark, and the evaluation metrics (BLEU, METEOR, CIDEr) to compare refined outputs against multiple valid descriptions rather than just one. It is also lightweight enough to run the full pipeline on free-tier GPU (Google Colab) without truncation.
| Model | Role | Type |
|---|---|---|
| Florence-2-Large | Caption generation | Local |
| BLIP (Salesforce) | Embedding extraction + ITM scoring | Local |
| Jina Reranker M0 | Cross-encoder relevance scoring | API |
| Qwen2.5-1.5B-Instruct | Caption fusion | Local |
Refined captions evaluated against 5 human reference captions per image.
| Method | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | METEOR | CIDEr |
|---|---|---|---|---|---|---|
| Random Caption | 1.000 | 1.000 | 1.000 | 0.999 | 0.999 | 2.641 |
| Best Cosine | 0.977 | 0.971 | 0.964 | 0.957 | 0.922 | 2.498 |
| Qwen Fused | 0.648 | 0.536 | 0.440 | 0.353 | 0.697 | 0.697 |
Note: Random Caption scores 1.0 because it IS one of the 5 references — a trivial baseline. Qwen Fused generates novel, independent descriptions — more complete and descriptive.
BLIP embedding cosine similarity computed between each image and its 5 captions across all 8,091 images.
| Score Type | Mean | Std |
|---|---|---|
| Best Caption | 0.5360 | 0.0412 |
| Worst Caption | 0.4452 | 0.0389 |
| Average | 0.4912 | — |
All 8,091 images processed · 0 skipped · Strong inter-signal consensus confirmed.
Example:
- Cap 1: a little girl in a pink dress going into a wooden cabin
- Cap 2: a little girl climbing the stairs to her playhouse
- Fused: A little girl in a pink dress climbs into a wooden playhouse near a tree, entering through its door.
| Embedding | Shape | Notes |
|---|---|---|
| Image Embeddings | (8091, 256) | L2-normalised |
| Caption Embeddings | (40455, 256) | L2-normalised |
| Category | Tools |
|---|---|
| Language | Python 3.10+ |
| Deep Learning | PyTorch · HuggingFace Transformers |
| Vision Models | Florence-2-Large · BLIP |
| Language Models | Qwen2.5-1.5B-Instruct |
| Reranking | Jina Reranker M0 |
| Scoring | scikit-learn · Cosine Similarity |
| Evaluation | NLTK · BLEU · METEOR · pycocoevalcap · CIDEr |
| Web App | Streamlit |
| Deployment | HuggingFace Spaces |
| Development | Google Colab · Google Drive |
image-captioning-refinement/
├── app.py # HuggingFace Streamlit application
├── requirements.txt # Python dependencies
├── Image_captioning_notebook.ipynb # Full research pipeline notebook
├── README.md # Documentation
├── LICENSE # Open-source license
└── results/ # Experimental charts & visualizations
├── cosine_distribution.png # Visualizing BLIP embedding similarity
├── baseline_comparison.png # Comparison against standard benchmarks
├── majority_voting.png # Consensus stage performance
├── Qwen_fusion.png # LLM refinement output analysis
└── combined_scores.png # BLEU, CIDEr, and METEOR metrics
git clone https://github.com/Afsha001/image-captioning-refinement
cd image-captioning-refinement
pip install -r requirements.txt
streamlit run app.pyAdd your
JINA_KEYas an environment variable before running.
| Field | Details |
|---|---|
| Degree | M.Sc. Data Science |
| University | Aligarh Muslim University, Aligarh |
| Department | Statistics & Operations Research |
| Internship Centre | Interdisciplinary Centre for Artificial Intelligence |
| Supervisor | Dr. Junaid Ali Reshi |
| Co-Supervisors | Dr. Ahmad Yusuf Adhami · Dr. Mohd. Faizan · Dr. Shazia Farhin |
| Session | 2025–26 |
This project is licensed under the MIT License — see the LICENSE file for details.
Made with ❤️ by Afsha Anjum · AMU Aligarh · 2025–26